Using wget to recursively download whole FTP directories

Question

I want to copy all of the files and folders from one host to another. The files on the old host sit at /var/www/html and I only have FTP access to that server, and I can't TAR all the files. Regular connection to the old host through FTP brings me to the /home/admin folder.

I tried running the following command form my new server:

wget -r ftp://username:password@ip.of.old.host

But all I get is a made up index.html file.

What the right syntax for using wget recursively over FTP?

Note that, by default, wget -r has a maximum recursion depth of 5; if you have an deeper subdirectories, they will be ignored unless you change this (see -l in the man page). — mikewaters, May 23 '11 at 19:21
What wget does also depends on wget version (or build) I came across one GNU Wget 1.10.2 (Red Hat modified) where when path ends without '/' wget considers it a file and a download fails, but when --timestamping (-N) is used, it would work. Then when path ended in '/' it would always create an index.html instead of downloading files. Ending in * would also work for a dir. In all cases also --recursive was used. — papo, Feb 18 '22 at 21:22

score 156 · Answer 1 · answered Jun 13 '09 at 10:04

156

Try -m for --mirror

wget -m ftp://username:password@ip.of.old.host

answered Jun 13 '09 at 10:04

Dave Cheney

18,635

26

For security reason, it would be good to use --ask-password to avoid password being kept in history. – checksum Oct 04 '14 at 02:49
This is so cool. Worked like a charm.. – Sambhav Sharma Oct 07 '15 at 21:11
This should be the accepted answer (for better visibility). – bgondy May 03 '16 at 15:52
it doesn't include path of directory to download, I guess @Formoda answer is better – Wasim A. Apr 24 '17 at 06:38
1

@WasimA., you can just add directory path here as well: wget -m ftp://username:password@ip.of.old.host/mydir/ – Alex Che May 16 '19 at 15:28

score 48 · Answer 2 · edited Jan 05 '17 at 11:33

48

You have it right, you just need a trailing * on the end:

wget -r ftp://username:password@1.2.3.4/dir/*

For shared servers, you can use like this:

wget -r ftp://1.2.3.4/dir/* --ftp-user=username --ftp-password=password

Because most shared servers has ftp-username something like username@hostname, so, the first wget command not works, and second command works fine.

edited Jan 05 '17 at 11:33

Banee Ishaque K

105

answered Jun 22 '09 at 12:35

2

This would only go one directory deep.Better to use the -m flag – SvennD Aug 13 '15 at 12:18
in some cases second command is not working because of space, You can use first command for share hosting like this. ** wget -r ftp:/ /username%40host:password@host/dir/ ** – Wasim A. Apr 24 '17 at 07:11
1

Note: you should use either -m or -r -l inf because -r has a default recursion depth of 5. See https://www.gnu.org/software/wget/manual/wget.html#Recursive-Retrieval-Options – ndemou Oct 10 '17 at 20:45

score 11 · Answer 3 · edited Jul 18 '17 at 21:02

11

Check the below wget command to download data from FTP recursively.

wget --user="<user-name>" --password="<password>" -r -np -nH --cut-dirs=1 --reject "index.html*" "<URL to download files>"

-r: Is for recursively download.

-np: Is for no parent ascending.

-nH: Is for disabling creation of directory having name same as URL i.e. abc.xyz.com

--cut-dirs: Is for ignoring number of parent directories. The value of this option will differ for your command.

You can check by executing the above command.

edited Jul 18 '17 at 21:02

Peter Mortensen

2,318

answered Jul 24 '15 at 04:48

AVJ

211

What does the --reject do? Is there any way to specify that it is not to overwrite anything? – bvargo Feb 28 '23 at 14:01

score 8 · Answer 4 · edited Jul 18 '17 at 21:00

Besides wget, you may also use lftp in script mode. The following command will mirror the content of a given remote FTP directory into the given local directory, and it can be put into the cron job:

lftp -c 'open <hostname>; user <username> <password>; mirror -e <remote-src-path> <local-dest-path>; quit'

It automatically handles recursion into directories and allows specifying the remote source starting directory from to download data from.

score 5 · Answer 5 · answered Jun 13 '09 at 21:22

5

You can use 'curlftpfs - mount a ftp host as a local directory' and, once mounted, you can use normal file tools like 'cp -r'.

answered Jun 13 '09 at 21:22

chmeee

7,440

curlftpfs has major performance problems right now unless you downgrade libcurl3-gnutls – Hengjie Feb 06 '15 at 05:40

score 4 · Answer 6 · answered Jun 13 '09 at 09:54

4

Should work try:

wget -r ftp://ftp:ftp@ftp.sunet.se/tst/

answered Jun 13 '09 at 09:54

rkthkr

8,678

score 3 · Answer 7 · edited Jun 07 '15 at 17:04

3

Use:

wget -m ftp://192.168.0.1

and it will mirror all the files and folders.

edited Jun 07 '15 at 17:04

kenorb

6,779

answered Jun 13 '09 at 14:07

Rajat

3,349

As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html So when I run the command you suggested, I only get the contents of /home/admin I tried running it with ftp://192.168.0.1/var/www/html, but then it tries to CWD /home/admin/var/www/html. How do I make it go to this folder from the root? – Jun 13 '09 at 17:23
I have no experience with this particular problem, but you could try making a symlink to /var/www/html under your home. Then you could use an address like ftp://192.168.0.1/html – prestomation Jun 13 '09 at 17:29
I tried creating a symlink, but it resulted with wget creating a similar symlink on my local. – Jun 13 '09 at 17:32

score 3 · Answer 8 · answered Jun 17 '13 at 11:56

3

Use -m rather than -r, because of the recursion depth.

http://www.editcorp.com/Personal/Lars_Appel/wget/wget_2.html#SEC11

answered Jun 17 '13 at 11:56

Bojan Hrnkas

143

score 1 · Answer 9 · answered Jun 13 '09 at 09:27

1

That's the right syntax. Not sure why you aren't getting the expected results.

As ever there is more than one way to do it. Try ncftp, in particular ncftpget

answered Jun 13 '09 at 09:27

goo

2,878

I can't install more software on my new server. Shouldn't I be telling wget to download all files from /var/www/html? I tried wget -r ftp://username:password@ip.of.old.host/var/www/html but I got a directory not found error. – Jun 13 '09 at 09:37

score 1 · Answer 10 · answered Jun 13 '09 at 13:37

1

I can understand if you're trying to dump this into cron or something, but why not simply ftp into the server with your normal client and mget *? This might be a quicker path to success.

answered Jun 13 '09 at 13:37

dr.pooter

399

I tried mget * but it didn't work with sub-folders, saying the local sub-folder doesn't exist. Is there a way to make him create the local folders automatically? – Jun 13 '09 at 17:16
Depending on your client, the -r switch usually does the trick. IE:
mget -r *
– dr.pooter Jun 15 '09 at 06:17
mmm. AFAIK standard ftp client in linux is not designed to retrieve directories recursively. I mean - there is no -r option. other clients like ncftp or lftp support recursive retrieval but they usually not available by default. – Stann Feb 13 '11 at 05:42

score 1 · Answer 11 · answered Jun 21 '09 at 21:25

1

As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html

I think this will work in your case:

wget -r ftp://192.168.0.1/../../var/www/html

answered Jun 21 '09 at 21:25

kubanczyk

13,982

score 0 · Answer 12 · edited Jul 18 '17 at 21:01

0

I came across Windows hosting with the username as brinkster/username, so wget would throw an error if you use the syntax:

wget -m ftp://brinkster/username:password@ip.of.the.host

To get past this, use

wget -m ftp://brinkster%2Fusername:password@ip.of.old.host

I had to face this with Windows hosting @ brinkster.

edited Jul 18 '17 at 21:01

Peter Mortensen

2,318

answered Nov 25 '12 at 21:28

Anto Joseph

11

score -3 · Answer 13 · answered Jun 19 '13 at 23:06

-3

wget --user username --password yourpassword ftp://example.com/ftpfiles/filename will do the job or you can add * instead of filename entring : makes wget think of port which is invalid in your case.

answered Jun 19 '13 at 23:06

Khushwant Singh

5

This answer to a four year old question with multiple upvoted answers doesn't offer anything new and even omits the requirement for a recursive copy. – Sven Jun 20 '13 at 00:10

Using wget to recursively download whole FTP directories

13 Answers13