Making HTTP Requests from the Command Line

Often, when working on the command line, you need to retrieve data from a web server. This is commonly done using tools that send HTTP requests. While wget is a popular option on many Linux distributions, it’s not natively available on macOS. This tutorial explores how to make HTTP requests from the command line, focusing on tools available by default on macOS, and how to achieve similar functionality to wget.

The `curl` Command: A Versatile Tool

The curl command is a powerful and ubiquitous tool for transferring data with URLs. It’s pre-installed on most macOS systems, making it a convenient solution for making HTTP requests.

Basic Usage:

The simplest way to download a file using curl is:

curl <URL> -o <filename>

<URL>: The URL of the resource you want to download.
-o <filename>: This option specifies the filename to save the downloaded content as. If you omit this option, curl will print the content to your terminal’s standard output.

For example, to download the homepage of a website and save it as index.html:

curl https://www.example.com -o index.html

Retrieving Data to Standard Output:

If you want to see the content directly in your terminal, simply omit the -o option:

curl https://www.example.com

This will print the HTML source code of the website to your terminal.

Following Redirects:

Sometimes, a web server will redirect your request to another URL. By default, curl does not automatically follow these redirects. To enable this, use the -L option:

curl -L <URL> -o <filename>

This is especially useful when dealing with shortened URLs or websites that use redirects for tracking.

Achieving `wget`-like Functionality

wget and curl share many similarities. The following curl options can replicate common wget behaviors:

Downloading a file: As shown above, curl <URL> -o <filename> is equivalent to wget <URL>.
Following redirects: Use curl -L <URL> to replicate wget‘s default behavior of following redirects.

Creating a `wget` Alias (Optional)

If you’re accustomed to using wget and prefer to keep using that command, you can create an alias in your shell configuration file (e.g., .bashrc, .zshrc). This will tell the shell to execute curl whenever you type wget.

Add the following line to your shell configuration file:

alias wget='curl -O'

The -O option tells curl to save the downloaded file with the same name as the file on the server. After adding the alias, you’ll need to either restart your terminal or source your configuration file (e.g., source ~/.bashrc) for the changes to take effect.

Considerations

While curl is a powerful substitute for wget in most cases, there are a few differences to keep in mind. wget has more advanced features for resuming downloads and recursive retrieval of files, which curl may require more complex options to achieve. However, for simple HTTP requests, curl provides a convenient and readily available solution on macOS.