64

How to not copy but move files from one server to another (both Linux)?

man scp didn't give me anything useful. I cannot use 'scp' then 'rm' because I must make sure the file is successfully transferred. If there is any error during transfer, the file must not be deleted.

Perhaps I should use exit code somehow, but how? Also, there are a lot of files, and if the last file fails it would be not-so-good option keep the whole bunch of successfully transferred files.

Maybe there is something besides SCP?

Ilia Ross
  • 1,106
Putnik
  • 2,327

7 Answers7

62

rsync over ssh is probably your best bet with the --remove-source-files option

rsync -avz --remove-source-files -e ssh /this/dir remoteuser@remotehost:/remote/dir

a quick test gives;

[tomh@workstation001 ~]$ mkdir test1
[tomh@workstation001 ~]$ mkdir test2
[tomh@workstation001 ~]$ touch test1/testfile.1
[tomh@workstation001 ~]$ ls test1/
testfile.1
[tomh@workstation001 ~]$ rsync --remove-source-files -av -e ssh test1/testfile.1 tomh@localhost:/home/tomh/test2/
sending incremental file list

sent 58 bytes  received 12 bytes  10.77 bytes/sec
total size is 0  speedup is 0.00

[tomh@workstation001 ~]$ ls test1/
[tomh@workstation001 ~]$
[tomh@workstation001 ~]$ ls test2/
testfile.1

As @SvenW mentioned, -e ssh is the default so can be omitted.

Tom
  • 11,346
  • 5
  • 42
  • 63
  • 11
    -e ssh is the implicit default for many years now, there is usually no need to use this parameter anymore. – Sven Feb 26 '12 at 20:21
  • @SvenW ah good to know! I don't have much need for rsync day to day as scp supports recursive mode for most purposes. – Tom Feb 26 '12 at 20:29
  • After that use this to clean up any empty folders (rsync won't remove them): http://serverfault.com/a/95935/58568 – bmaupin Dec 06 '12 at 05:07
  • 2
    Just added smv() { rsync -az --remove-source-files "$@"; } to my toolbox. Thank you. – Rhys Ulerich Jun 03 '14 at 16:13
  • What if you don't have rsync? For example OpenWrt doesn't have rsync and I need to use scp to move files. What to do now? – valentt Oct 17 '16 at 20:42
  • -e ssh is still useful for passing a specific authentication key -e "ssh -i mykey.pem" – k_o_ Jun 05 '21 at 02:02
26

Use rsync instead of scp: 

rsync -avz --remove-source-files /sourcedir user@host:/targetdir

More info with man rsync.

Sven
  • 99,533
  • 15
  • 182
  • 228
25

This question's been answered just fine, and the answer accepted, but since it's floated to the top of the front page, I thought I'd at least try to answer it more precisely, if less elegantly. Yes, you can use the return code from scp, and I do it often. In bash:

scp foo user@server:/destination && rm foo

I take your point about multiple files to copy and handling failure down the stack correctly, so for multiple files:

for file in bar*; do scp "$file" user@server:/destination && rm "$file" ; done

This last is only practical if you're using ssh-agent, but I very much hope you are.

MadHatter
  • 80,590
  • 2
    I feel this should have been the answer. It answers the question asked precisely. – Travis Griggs May 10 '17 at 21:27
  • As SCP doesn't handle (AFAIK) bandwith limitation, I used your hanswer with rsync to move files one by one at a defined bandwith. Thanks ! – Moonchild Jan 14 '18 at 12:40
  • @Moonchild, scp actually can handle bandwidth limits using the -l flag. E.g., -l 8192 to limit transfers at 8192 kb/s – Spencer D Feb 05 '18 at 05:07
5

If doing it in two steps is not a problem, you can use scp to copy the file from the remote server, and then execute ssh -e "rm /path/to/file" to remove from disk. When moving files especially between machines things may go wrong, so it may be good to perform the copy and the delete separately, and only remove the file when you know for sure that it has been successfully copied first.

stambata
  • 1,718
Giannis
  • 61
3

in my situation ,ssh port is not 22, so 

rsync -avz --remove-source-files -e "ssh -p $portNumber" user@remoteip:/path/to/files/ /local/path/

works for me.

bigCat
  • 139
  • 6
    You are much better off putting custom ssh port numbers into your ssh config file (~/.ssh/config) than specifying them for every command. It can be specified globally, per-host, or for a regular expression matching the hostname. – sneak Sep 26 '14 at 18:42
  • How to set ports individually both for source and destination? – Brethlosze Jun 18 '17 at 17:59
3

if you have older target server as I do, you can't use

--remove-source-files

but you have to use

--remove-sent-files --protocol=29

instead.

andrej
  • 503
1

Thought I'd suggest an alternative to rsync I found, lftp, because rsync requires shell access which my sftp server blocks.

lftp -e "mirror --Remove-source-files ./remote/path/ /local/path/; quit" remoteuser@remotehost:

You can also pass in the password if you need to run this as part of a batch process / cron job (obviously insecure)

lftp --password hunter2 -e "mirror --Remove-source-files ./remote/path/ /local/path/; quit" remoteuser@remotehost:
Andy C
  • 111