Complete Guide on How to Download Files Using cURL

cURL or Client URL is a tool developed for fetching files, interfacing with APIs and even for web resource testing. Also is available for Windows, Mac and Linux operating systems which makes it easy for any cURL user to download files. Client URL is appropriate for simple file downloading as well as for more advanced tasks involving data manipulation and even for script embedding.

In this article, we will cover how to download files using cURL, covering some useful cURL commands for downloading, specifying names, folder, and dealing with redirects, speed throttles, and much more. We will also cover commands specific to cURL on Windows, Linux, and Mac systems.

Managing Filenames

It is essential to choose the correct file name when saving the downloaded document. cURL doesn’t save the data but rather displays it on the terminal. Hence, if the user does not select a location for saving the file, it will not be found in the file system.

All commands with cURL are done on the command line: using Windows — cmd or PowerShell and Linux or macOS — terminal.

For manual file naming, the user is to use the command “-o” or alternatively for its verbose format “–output”.

Consider the following example:

curl -o test.html https://example.com

This command will fetch the page on example.com and save it as test.html to the current working directory.

cURL will, by default, save the file in the same directory which contains the executable file unless otherwise specified.

How does one using cURL save a file along with specifying the directory in which the file is to be stored:

curl -o ~/Downloads/test.html https://example.com

This option will place the file in the Downloads sub-folder in the user’s home directory.

In this configuration if a file already exists, the new file will replace the old file with no warnings. To bypass this, cURL -O so that the file can be saved with the name provided in the URL:

curl -O https://example.com/file.txt

If file.txt is what is provided in the URL, it will automatically be stored with that name. If such a file happens to exist already, it will be replaced.

To ensure there is no loss of data, one can set a verification prompt before downloading:

if [ -f file.txt ]; then
    echo "File already exists!"  
else  
    curl -O https://example.com/file.txt  
fi

This script first scans to see if the file is already present, only downloading the file if it is not.

Handling Redirects

Some files are not saved in specific locations, but rather obtained via a redirect. For example, a cURL user trying to download a large file may discover that the browser directs them to another page prior to commencing the download. cURL does not follow redirects by default a, but this can be changed.

To enable cURL to automatically follow redirects, add the flag -L:

curl -L -o test.zip https://example.com/download

If the server initially redirects the user, then cURL will do the new request first and only then will the file be downloaded from the URL.

Why this is important:

• Avoiding the need to pass through intervening pages. Some pages implement redirects for reasons such as limited downloads, authentication, or advertisements.
• Using shortened URLs. For instance, Bit.ly or TinyURL, and others, simply redirect the user to the actual URL, without -L the cURL will fetch an HTML page with the redirect instead of the actual file.
• HTTPS forward support. A number of sites will first forward all http:// requests to the https://. If -L is not used, downloading files will not work.

cURL’s method processes each redirect in order, following every link that leads to another URL in sequence until it reaches the target link. In the event that the server gets stuck in an endless loop of redirections, the server will get caught up. cURL has a command that can be used to manage the number of redirections, which is ‘--max-redirs’.

curl -L --max-redirs 5 -o file.zip https://example.com/download

This parameter places a restriction on cURL, so it stops the downloading process if too many redirects are processed and exceeds the specified limit.

An example cURL gives when the redirection limits are surpassed is the following:

curl: (47) Maximum redirection limit reached

Downloading Multiple Files

Using cURL you can download a multiple of files simultaneously, and with some commands will work across multiple operating systems, while others will be specific to Windows or Linux/macOS.

Ways that are common to all OS

• If a server contains multiple files, all you will need to do is add as many -O commands as needed:

  
  curl -O https://example.com/file1.zip -O https://example.com/file2.zip
  
  

• If the files are sequentially numbered like file1.zip, file2.zip, file3.zip, then you can wrap the file name in brackets:

  
  curl -O https://example.com/file{1,2,3}.zip
  
  

For Windows

• For Windows cURL, if a user has a text document containing the URLs like urls.txt, they can download the file using the command line:

  
  Get-Content urls.txt | ForEach-Object { curl -O $_ }
  
  

• If the files are numbered, you can use a loop:

  
  For ($i=1; $i -le 3; $i++) { curl -O "https://example.com/file$i.zip" }
  
  

For Linux and MacOS

• Downloading from a list.

  To start off, create a file urls.txt containing a list of links:

  
  https://example.com/file1.zip 
  https://example.com/file2.zip 
  https://example.com/file3.zip
  
  

  Then execute the command:

  
  xargs -n 1 curl -O < urls.txt
  
  

• Downloading in a loop (Bash).

  In the case when files are numbered, it is convenient to use for:

  
  for i in {1..5}; do 
   curl -O https://example.com/file$i.zip 
  done
  
  

Rate Limiting

When there is a need to download a large file using cURL and the expectation of a low bandwidth traffic is there, the --limit-rate flag is available, which helps in setting a cap on downloading speed in (k) kilobytes per second, (m) megabytes per second, or (g) gigabytes per second.

For cURL users, the command structure is the same on every OS:

curl --limit-rate 500k -O https://example.com/archive.zip

In this case, 500k signifies that the download speed is not to exceed 500 KB/s.

Other notes:

• --limit-rate 2m — limit of 2 MB/s;
• --limit-rate 100k — limit of 100 KB/s.

Here’s a sample script to download archive1.zip, archive2.zip, and archive3.zip all at 500 KB/s download speed.

Windows (PowerShell):

1..3 | ForEach-Object { curl --limit-rate 500k -O https://example.com/archive$_.zip }

Linux/macOS (Bash):

for i in {1..3}; do 
 curl --limit-rate 500k -O https://example.com/archive$i.zip 
done

The option --limit-rate is effective under conditions of a sustained network connection as it lowers the chances of network disruption. This is helpful while multiple files are being downloaded at once, or if other activities are being performed on the network. Additionally, --limit-rate helps mitigate the chances of getting blocked by a server that has restrictions for downloads. When the download is done via proxy, the parameter helps reduce the burden and minimize the chances of getting cut off. This command also helps manage traffic when working with mobile internet or tariffs with a limited amount of data.

Silent Downloads

In cURL, the progress of a download, along with its speed, is shown in a terminal window as a default. This is useful for some cURL users, however, it can be suppressed for scripts or automated processes. In these cases, the -s or --silent flag can be enabled.

Example of how to use cURL command line within “silent” mode to download files:

curl -s -O https://example.com/file.zip

As the command suggests, all files are to be downloaded with cURL with no information shown on the screen. In case there is a need for showing errors only, and putting all the other messages on mute, -S (--show-error) can be used and combined with the aforementioned arguments:

curl -s -S -O https://example.com/file.zip

Authentication

Downloading some files and using some APIs require authorization. cURL has an architecture that supports a good number of authentication procedures such as those that use a username and password, or even access tokens.

In case access requires a username and password, these can be sent as a single argument with the -u flag:

curl -u username:password -O https://example.com/protected-file.zip

If the password carries special components like, for instance, @, #, &, and so on, then it is advisable to put the password in quotation marks:

curl -u username:"your@password#123" -O https://example.com/protected-file.zip

In the event that only the username is sent, cURL prompts for the password.

Another method of authenticating to a issuers system can be done using an access token, in this instance, the token is included in the request header with the as shown below:

curl -H "Authorization: Bearer your_token_here" -O https://api.example.com/data.json

Some services may require that the token be included directly in the URL. Then the command will look like this:

curl -O https://api.example.com/data.json?access_token=your_token_here

In any case, putting the token in the URL can be problematic because the user's pathname could be stored in the browser history or server logs.

These authentication methods work well when one needs to fetch data from a protected server's files, requesting data from APIs, or accessing data from within a corporate environment.

Downloading via Proxy

Proxies are especially useful for cURL and its common use is in remote file access as cURL supports transmission of data over most protocols.

Setting a proxy is done with -x (or –proxy).

The general form of the command is the following:

curl -x [scheme://]proxy-server[:port] -o output.file URL

Example: downloading a file through an HTTP proxy:

curl -x http://proxy.testnet.com:80 -o file.zip https://example.com/archive.zip

Where:

• -x http://proxy.testnet.com:80 — designates the required HTTP proxy and the port it will use;
• -o archive.zip — indicates the file name to store it under;
• https://example.com/archive.zip — the location of the file that is being downloaded.

When a username and password are required for proxy servers, they can be used like this:

curl -x http://proxy.example.com:80 -U user:password -o archive.zip https://example.com/archive.zip

A different approach — passing authentication via the URL:

curl -x http://user:password@proxy.example.com:80 -o archive.zip https://example.com/archive.zip

To work with SOCKS5 or SOCKS4, set the --proxy parameter with the type, for example:

curl --proxy socks5h://proxy.example.com:1080 -o archive.zip https://example.com/archive.zip

Important nuance:

• socks5h:// — enables passing DNS queries through the proxy higher anonymity;
• socks5:// — transmits DNS queries directly (less secure).

The proxy will provide anonymity since the original IP address will not be available to the sites the users visits. Additionally, blocks placed on resources from certain regions can be bypassed if it is accessed from a different country. By using a proxy to download files, users from corporate networks can access closed resources and improve loading speeds as a result of restriction from the provider or data caching.

Tracking Progress

It is advisable to remain on the alert to properly supervise the process of downloading files using cURL, particularly with larger files. This can be accomplished by utilizing built in capabilities which include progress displays such as an interactive download bar as well as detailed.

The -# or --progress-bar flag replaces the basic percentage download indicator with a moving bar. This is useful if measurement of the system performance is required in a more compact and graphical form.

curl -# -o file.zip https://example.com/file.zip

A bar with a filling indicator moves from left to right:

This method is convenient if one wishes to keep track of the operation with a limit on the amount of information displayed.

For observing in greater detail the speed, time, and amount of data that is being downloaded, one can use the -v and -w options in conjunction.

• Example with -v (Some added details in the output):

  
  curl -v -o file.zip https://example.com/file.zip
  
  

• To display the major indicators after finishing the download, you have to set a particular form with -w:

  
  curl -o file.zip -w "Download speed: %{speed_download} bits per second\n" https://example.com/file.zip
  
  

This enables monitoring the download speed during the whole process.

For cases where the connection is unstable, the download speed can be closely judged. If there is obstruction, there will be a sudden dip in the speed of the process or a decline altogether. When the connection is stable, the speed, and the overall performance of the network can be monitored.

Request/Response Info

The cURL command allows for retrieval of a multitude of information that can help with identifying problems such as disconnections, ratifications, or even HTTP response codes parameters which might be crucial in diagnosing errors. For this purpose, the -v flag will be used which switches on the expanded output mode.

Example:

curl -v -o file.zip https://example.com/archive.zip

The output will look somewhat like this during execution of the command.

You will notice the following key data:

• server’s IP address and the connection establishment;
• original HTTP request;
• server’s response status;
• response headers which include Content-Type, date and some caching parameters.

The details of server requests and responses can be viewed with cURL by including the -v flag. This provides insight into how the client communicates with the server. For instance, in case of a redirect, if the server changes the location to a new URL, "-v" will indicate usage of the commands “status 301 Moved Permanently” or “302 Found”.

This can also be used to check errors, such as “403 Forbidden” or “404 Not Found”, which reference restricted access and file absence. Additionally, response headers can also be checked to analyze different file parameters prior to the downloading process.

cURL vs Wget

cURL and Wget are the two most prominent tools for file downloads on command line interfaces and cURL and Wget are widely used for that purpose, yet they differ in functionalities.

Furthermore, let us look at the major distinctions among them.

Wget and cURL have differences in command syntax as well:

• In cURL, it is necessary to indicate flags while using commands explicitly: set -O, when saving a file, -C - when continuing a download and -L when following redirects. If no option to save to a file is specified, cURL’s default is to print the file’s contents on the terminal screen.
• In Wget, commands are straightforward; the file is downloaded automatically without any additional flags being specified, resumption is done with -c, and redirects are automatic.

Wget appears to have been created primarily for downloading files while cURL was made for interacting with HTTP requests and APIs.

In terms of supported protocols, Wget is at a disadvantage compared to cURL in many aspects.

Conclusion

Indeed, cURL is admirable for its support of a multitude of protocols, file naming conventions, proxy use, and the ability to resume interrupted downloads.

cURL is apt for not only simple downloads becoming integrated with APIs or performing massive data fetching. However, its ability to redirect, set speed limits, authenticate, and track server interaction, allow server connection diagnosis and web resource testing makes it a great multipurpose tool. And with the flexible parameter customizations offered, cURL can be tailored to suit any and every task.