Introduction

Wget command is a Linux command line utility that helps us to download the files from the web. We can download the files from web servers using HTTP, HTTPS and FTP protocols. We can use wget in scripts and cronjobs.

Here non-interactive means that it can work in the background, while the user is not logged on. When we do the minimal installation of Linux distributions then wget command is not installed, So to install wget on Linux distributions, run

$ sudo yum install -y wget     for  CentOS 7 / RHEL 7
$ sudo dnf install -y wget     for CentOS 8/ RHEL 8/ Rocky Linux 8
$ sudo apt install -y wget     for Ubuntu / Debian
$ sudo pacman -S wget          for Arch Linux
$ sudo zypper install wget     for OpenSUSE

Downloading a single file with wget

[unixcop@rhel-pc ~]$ wget https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso
--2021-08-13 04:24:05--  https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso
Resolving releases.ubuntu.com (releases.ubuntu.com)... 91.189.91.124, 91.189.91.123, 2001:67c:1562::28, ...
Connecting to releases.ubuntu.com (releases.ubuntu.com)|91.189.91.124|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2877227008 (2.7G) [application/x-iso9660-image]
Saving to: ‘ubuntu-20.04.2.0-desktop-amd64.iso’

ubuntu-20.04.2.0-desktop-amd64.iso      47%[=======          ]   4.3G   2.13M/s    eta 1h 02m

This command will download Ubuntu ISO file in the user’s current working directory.

Resume the partially downloaded file

Using the option ‘-c’ option in wget command we can resume our download from where it got disconnected.

[unixcop@rhel-pc ~]$ wget -c https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso

 Download Files in the background

Use ‘-b’ option in wget command to download files in the background

[unixcop@rhel-pc ~]$ wget -b https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso
Continuing in background, pid 2412.
Output will be written to ‘wget-log’.
[unixcop@rhel-pc ~]$ 

Also use tail command to view status of download.

[unixcop@rhel-pc ~]$ tail -f wget-log
  2700K .......... .......... .......... .......... ..........  0% 85.8K 16h41m
  2750K .......... .......... .......... .......... ..........  0% 90.0K 16h32m
  2800K .......... .......... .......... .......... ..........  0% 78.8K 16h25m
  2850K .......... .......... .......... .......... ..........  0%  189K 16h13m
  2900K .......... .......... .......... .......... ..........  0% 92.4K 16h5m
  2950K .......... .......... .......... .......... ..........  0% 89.7K 15h57m
  3000K .......... .......... .......... .......... ..........  0% 92.8K 15h50m
  3050K .......... .......... .......... .......... ..........  0%  177K 15h39m
  3100K .......... .......... .......... .......... ..........  0% 91.2K 15h32m
  3150K .......... .......... .......... .......... ..........  0% 96.0K 15h25m
  3200K .......... .......... .......... .......... ..........  0% 79.6K 15h20m
  3250K .......... .......... .......... .......... ..........  0%  199K 15h9m
  3300K .......... .......... .......... .......... ..........  0% 97.7K 15h3m
  3350K .......... .......... .......... .......... ..........  0% 94.6K 14h57m
  3400K .......... .......... .......... .......... ..........  0% 95.8K 14h51m
  3450K .......... .......... .......... .......... ..........  0% 96.0K 14h45m
  3500K .......... .......... .......... .......... ..........  0%  102K 14h39m
  3550K .......... .......... .......... .......... ..........  0%  153K 14h31m
  3600K .......... .......... .......... .......... ..........  0% 62.1K 14h29m
  3650K .......... .......... .......... .......... ..........  0%  109K 14h23m
  3700K .......... .......... .......... .......... ..........  0% 83.1K 14h19m
  3750K .......... .......... .......... .......... ..........  0%  118K 14h13m
  3800K .......... .......... .......... .......... ..........  0%  104K 14h8m
  3850K .......... .......... .......... .......... ..........  0%  112K 14h2m





etc..

Limit the download speed while downloading the files

You can limit the download speed using ‘–limit-rate’ option.

[unixcop@rhel-pc ~]$ wget --limit-rate=350k https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso

we have set the download limit as 350K.

Download multiple files

First create a text file and add all URLs in that text file as shown below:

[unixcop@rhel-pc ~]$ cat wget-download-list.txt 
http://centos.mirror.server24.net/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso
https://download.rockylinux.org/pub/rocky/8/isos/x86_64/Rocky-8.4-x86_64-minimal.iso

[unixcop@rhel-pc ~]$ 

Now run :

[unixcop@rhel-pc ~]$ wget -i wget-download-list.txt 
--2021-08-13 04:36:46--  http://centos.mirror.server24.net/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
Resolving centos.mirror.server24.net (centos.mirror.server24.net)... 217.70.144.100, 2001:1a38:144::100
Connecting to centos.mirror.server24.net (centos.mirror.server24.net)|217.70.144.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9928966144 (9.2G) [application/octet-stream]
Saving to: ‘CentOS-8.4.2105-x86_64-dvd1.iso’

CentOS-8.4.2105-x86_64-dvd1.iso         4%[==                 ] 964M  1.9M/s    eta 3h 44m
[unixcop@rhel-pc ~]$ 

Increase Retry Attempts in wget

We can increase the retry attempts using ‘–tries’ option in wget. By default wget command retries 20 times to make the download successful.

# wget --tries=60 http://centos.mirror.server24.net/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso

The command above will make wget command try to download the file max 60 times .

Redirect wget command output to a File

We can redirect the output to a log file using ‘-o’ option as shown below:

$ wget -o download_file.log http://centos.mirror.server24.net/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso

You can show it with

# ls
# cat download_file.log

Mirror a complete website with wget command

We can do that with:

# wget --mirror -p --convert-links -P ./<Local-Folder> website-URL

Where

• –mirror : turn on options suitable for mirroring.
• –p : download all files to properly display a given HTML page.
• –convert-links : convert the links in document for local viewing.
• -P ./Local-Folder : save all the files and directories to the specified directory.

Reject file types while downloading

When you are planning to download full website , then we can force wget command not to download pdf files for example using ‘–reject‘ option .

# wget --reject=pdf <Website-To-Be-Downloaded>

Set the Download quota

Wget can quit downloading when download size exceeds certain size. Use ‘-Q‘ option in wget command to set download quota.

# wget -Q14m -i wget-download-list.txt

Allow wget to overwrite the files

If we specify the output file using the -o option, it will overwrite any existing file.

wget -q http://www.example.com/filename.txt -O /path/filename.txt

Download a file using FTP server

​Wget uses the anonymous FTP to download the files from the web. It does’t require FTP logins to download the files. 

# wget ftp://ftp.example.com/file.tar.gz

Downloading file from password protected site

# wget --ftp-user=<user-name> --ftp-password=<password> Download-URL
OR
# wget --user <user_name> --password <password> http://<url-path>/file_you_want_to_download

Another way to specify username and password is in the URL itself.

Download a file from untrusted secure URL

Want to download file from https ports and want to skip certificate checks?

so it can accomplished by using option “–no-check-certificate”

# wget https://example.com/file.tar.gz –no-check-certificate

Example : Downloading Oracle Java from https portal and skip certificate checks

# wget --header "Cookie: oraclelicense=accept-securebackup-cookie" https://download.oracle.com/otn-pub/java/jdk/12.0.1+12/69cfe15208a647278a19ef0990eea691/jdk-12.0.1_linux-x64_bin.rpm --no-check-certificate

Downloading Files via Proxy

First we need to set proxy and then we can download file from the internet. To set proxy on the command line use the following variables and export command

# export http_proxy=http://<Your-Proxy-IP>:<Proxy-Port>
# export https_proxy=http://<Your-Proxy-IP>:<Proxy-Port>
# export ftp_proxy=http://<Your-Proxy-IP>:<Proxy-Port>

In Case user name and password is required for proxy then use :

# export http_proxy=http://<user-name>:<password>@<Your-Proxy-IP>:<Proxy-Port>
# export https_proxy=http://<user-name>:<password>@<Your-Proxy-IP>:<Proxy-Port>
# export ftp_proxy=http://<user-name>:<password>@<Your-Proxy-IP>:<Proxy-Port>

Check the version of wget

We can check the version of Wget command as below 

# wget –version

Conclusion

Wget has huge number of options, We covered many of them .

By the way we can find more options from manual page.

# man wget