Getting all files from a web page using curl

Question

I would like to get all files from the web page below using curl:

http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/

I tried:

curl http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/

It returned a bunch of lines in the terminal, but didn't get any file.

score 172 · Accepted Answer · edited Jun 15 '22 at 06:10

172

Use wget instead.

Install it with Homebrew: brew install wget or MacPorts: sudo port install wget

For downloading files from a directory listing, use -r (recursive), -np (don't follow links to parent directories), and -k to make links in downloaded HTML or CSS point to local files (credit @xaccrocheur).

wget -r -np -k http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/

Other useful options:

-nd (no directories): download all files to the current directory
-e robots=off: ignore restrictions in robots.txt file and don't download robots.txt files
-A png,jpg: accept only files with the extensions png or jpg
-m (mirror): -r --timestamping --level inf --no-remove-listing
-nc, --no-clobber: Skip download if files exist

edited Jun 15 '22 at 06:10

molf

103

answered Aug 31 '13 at 13:56

Lri

105,117

3

wget -r -np -k http://your.website.com/specific/directory. The trick is to use -k to convert the links (images, etc.) for local viewing. – yPhil Dec 11 '14 at 15:44
brew and port doesn't work for me to install wget. What should I do? – Hoseyn Heydari Jan 27 '16 at 15:58
@HoseynHeydari : you can use rudix.org for compiled binaries for osx. so you need to install rudix and then use : sudo rudix install wget – mamonu Jul 03 '16 at 21:29
The option -k does not always work. E.g., if you have two links pointing to the same file on the webpage you are trying to capture recursively, wget only seems to convert link of the first instance but not the second one. – Kun Jan 28 '17 at 19:45
Don't forget the / at the end. – khatchad Nov 25 '20 at 17:34
Strange that it doesn't download index.html. – khatchad Nov 10 '21 at 18:51

score 31 · Answer 2 · answered Aug 31 '13 at 14:09

31

curl can only read single web pages files, the bunch of lines you got is actually the directory index (which you also see in your browser if you go to that URL). To use curl and some Unix tools magic to get the files you could use something like

for file in $(curl -s http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/ |
                  grep href |
                  sed 's/.*href="//' |
                  sed 's/".*//' |
                  grep '^[a-zA-Z].*'); do
    curl -s -O http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/$file
done

which will get all the files into the current directory.

For more elaborated needs (including getting a bunch of files from a site with folders/directories), wget (as proposed in another answer already) is the better option.

answered Aug 31 '13 at 14:09

nohillside

100,768

Thank you. This is a nice solution and providing working example is great ! – egelev Oct 21 '15 at 10:21
3

xmllint --html --xpath '//a/@href' is probably a better parser than grep. – ceving Feb 05 '19 at 13:17
Thank you for answering the question that was asked, as I can not install wget on the server I need to use – StevenWernerCS Jan 29 '21 at 15:57

score 19 · Answer 3 · answered Jun 19 '16 at 04:22

Ref: http://blog.incognitech.in/download-files-from-apache-server-listing-directory/

You can use following command:

wget --execute="robots = off" --mirror --convert-links --no-parent --wait=5 <website-url>

Explanation with each options

wget: Simple Command to make CURL request and download remote files to our local machine.
--execute="robots = off": This will ignore robots.txt file while crawling through pages. It is helpful if you're not getting all of the files.
--mirror: This option will basically mirror the directory structure for the given URL. It's a shortcut for -N -r -l inf --no-remove-listing which means:
- -N: don't re-retrieve files unless newer than local
- -r: specify recursive download
- -l inf: maximum recursion depth (inf or 0 for infinite)
- --no-remove-listing: don't remove '.listing' files
--convert-links: make links in downloaded HTML or CSS point to local files
--no-parent: don't ascend to the parent directory
--wait=5: wait 5 seconds between retrievals. So that we don't thrash the server.
<website-url>: This is the website url from where to download the files.

Happy Downloading :smiley:

score 4 · Answer 4 · edited Jun 18 '17 at 13:37

4

You can use httrack available for Windows/MacOS and installable via Homebrew.

edited Jun 18 '17 at 13:37

nohillside

100,768

answered Jun 18 '17 at 13:08

user242053

41

something new for me, thanks. best is it is quick and fast: one -liner. just cut the fat and avoid all rubbish. – Peter Teoh May 09 '18 at 07:40

score 1 · Answer 5 · answered Feb 15 '14 at 16:05

For those of us who would rather use an application with a GUI, there is the inexpensive shareware program DeepVacuum for Mac OS X, which implements wget in a user-friendly manner, with a list of presets that can handle commonly-needed tasks. You can also save your own custom configurations as presets.

enter image description here

Getting all files from a web page using curl

5 Answers5

Explanation with each options

Linked

Related