3

I want to transfer particular types of files from directory on server. So here is the scene:

The directory structure is

companies/compA/filedir/
companies/compB/filedir/
companies/compC/filedir/

So I want to move *.csv files from filedir directories from each company to other server. I want to use ssh for file transfer. Has anyone used ssh for file transfer in shell script? How do you handle this situation?

yogsma
  • 245

4 Answers4

6

Use scp or rsync.

scp uses ssh to copy files to and from connected computers. Sample syntax of copying from the local computer to a remote host:

scp -r companies/compA/filedir username@remote.host.address:/some/remote/directory

You'll need to use public/private keys in order to facilitate logging in if you haven't done that already for ssh.

rsync can use ssh for it's ability to connect to another server. The sample syntax would be:

rsync -avz -e ssh remoteuser@remotehost:/remote/dir /companies/compA/filedir

It will also need public/private keys enabled.

Chealion
  • 5,753
1

You would actually want to use scp which uses SSH. If you want to use rsync instead, you would want to use the -e ssh argument.

As for scripting it you would probably create either 1) a for loop that iterates through an array that holds a list of all the locations and runs the corresponding scp/rsync command OR 2) you could use a find statement that goes through and executes each scp/rsync command for you.

The find command example would be something like (untested):

find /dir/companies/*/filedir/ -type f -name \*.csv -exec scp {} username@otherserver.tld \;

I suggest that if you are using SSH for any type of transfer you setup some sort of public-key authentication, so that you aren't prompted for the password.

I can go into more detail if need be, just trying to get you started and pointed into the right direction.

drewrockshard
  • 1,763
  • I was assuming that everything from /dir/companies/* was some sort of "compX" naming convention, so I globbed for "everything" - this may not be correct, but I did assume this. – drewrockshard Aug 24 '10 at 15:03
  • @drew - while your approach would technically work, it's far from ideal. Imagine the case where each CompA directory has 5000 files in it. That would mean that for each file, you'd need to go through all the overhead of starting and tearing down the scp connection. This would introduce a ton of extra time into the process. Solutions not involving find are much better in this respect. – EEAA Aug 24 '10 at 15:05
  • This is why I suggested the for loop +array. I wouldn't use the find command, but some people prefer it. – drewrockshard Aug 24 '10 at 15:07
  • how do you set up public-key authentication and use it in the script? – yogsma Aug 24 '10 at 15:12
  • yogsma: http://serverfault.com/questions/2429/how-do-you-setup-ssh-to-authenticate-using-keys-instead-of-a-username-password – EEAA Aug 24 '10 at 15:17
  • yogsma: While you could sift through the forums on here, like ErikA posted, you could go through the post. Sometimes these posts can be confusing if you are new to something and you have to put several "answers" together to formulate the "solution". Here's a link to a tutorial on how to setup "passwordless" SSH: http://everydaylht.com/howtos/system-administration/loggin-in-via-ssh-without-a-password/ . As long as the user that is running the script is setup for "passwordless login", then it will work inside a script (includes the scp/rsync commands that reference the logins also). – drewrockshard Aug 24 '10 at 15:26
  • @drew @Erik - Thanks for your responses. I am trying to set up public key authentication. – yogsma Aug 24 '10 at 15:46
  • Awesome! Post if you need help. – drewrockshard Aug 24 '10 at 16:01
1

You wouldn't use the ssh program - you should use scp or sftp. But having said that a better solution would be to run rsync over ssh to avoid copying files you already have received - but this is only an option if rsync is available (or can be installed on the remote system).

In order to avoid having to supply a password you'd need to set up a keypair first, then just:

#!/bin/bash

scp somuser@server:/companies/compA/filedir/*.csv /local/compa/filedir
scp somuser@server:/companies/compB/filedir/*.csv /local/compb/filedir
scp somuser@server:/companies/compC/filedir/*.csv /local/compC/filedir
symcbean
  • 22,376
0

scp companies/comp?/filedir/*.csv 192.168.1.200:/remotedir being 192.168.1.200 the remote server ip.

If you need user and pass you should check the scp manual (man scp), but I believe you can't specify those in the scp command. Instead you should use keys.