How To Download Entire Websites Using wget.

An invaluable tool that I use a lot is wget. wget is a free terminal utility for downloading files from the Internet.

I do not have a reliable Internet connection and when I do get online, I am usually on a metered network. Not being able to connect to the Internet when I want to makes accessing online resources difficult. To solve this problem I use wget to download websites to my computer. I will show you how to do this yourself in this post.

The command to download a site is wget url
Here is an example using the Python Documentation Website wget
This will download everything on the python docs website’s first page. Adding an “r” option to the wget command will cause it to download the entire site.
$ wget --recursive

Other options you can make use of are:

  • –domains : don’t follow links outside the domain in url
  • –convert-links: convert links so that the work offline
  • –no-clobber: don’t overwrite any existing file. Use this option to resume an interrupted download

wget can download files over FTP, HTTPS and even retrieve files through proxies.