Inspecting HTTP Requests with Curl

Understanding HTTP Requests and Why Inspect Them

HTTP (Hypertext Transfer Protocol) is the foundation of data communication on the web. When your browser requests a webpage, or an application communicates with a server, it sends an HTTP request. These requests contain crucial information, including headers, which provide metadata about the request itself (e.g., content type, accepted encoding, user agent).

Sometimes, it’s necessary to inspect the exact HTTP request your application (or a command-line tool like curl) is sending. This is invaluable for debugging, understanding server behavior, and ensuring your application is communicating correctly. This tutorial focuses on how to inspect HTTP requests made by the powerful command-line tool curl.

Using Curl to View Request Headers

curl provides several options to reveal the details of the HTTP requests it makes. Let’s explore the most common and useful methods:

1. The --verbose or -v Option

The simplest way to see request headers is using the --verbose (or its short form, -v) option. This option provides a detailed log of the entire communication process, including the request headers sent to the server.

curl -v https://www.example.com

The output will include lines prefixed with > indicating the headers being sent in the request. You’ll see information like the User-Agent, Host, Accept and any custom headers you might have specified.

2. The --trace-ascii Option

The --trace-ascii option provides even more detailed tracing information and saves it to a file (or standard output if you use -). This option captures everything that curl sends and receives, making it excellent for in-depth analysis.

curl --trace-ascii curl.trace https://www.example.com

This command will create a file named curl.trace containing the complete communication log. You can then open this file in a text editor to examine the request and response headers. To output to the standard output, use:

curl --trace-ascii - https://www.example.com

3. Saving Response Headers with --dump-header

If you only want to save the response headers (not the entire request or body), you can use the --dump-header option.

curl --dump-header headers.txt https://www.example.com

This will create a file named headers.txt containing only the response headers from the server.

4. Custom Headers with Curl

curl allows you to specify custom headers to be included in the request. This is useful for testing API integrations or simulating different client environments. Use the -H option followed by the header name and value.

curl -H "X-Custom-Header: MyValue" https://www.example.com

Combine this with the -v option to verify that your custom header is being sent in the request.

Inspecting Requests in Code (Libcurl)

While the command-line options are great for quick inspection, you might need to examine requests within a program using the Libcurl library. Here’s a basic example in PHP:

<?php
$ch = curl_init();
curl_setopt_array($ch, array(
    CURLOPT_URL => 'https://www.example.com',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HEADER => 1, // Include headers in the response
    CURLOPT_VERBOSE => 1, // Enable verbose output (to STDERR)
));

$response = curl_exec($ch);

if (curl_errno($ch)) {
    echo 'Curl error: ' . curl_error($ch);
} else {
    echo $response; // The response headers and body will be printed
}

curl_close($ch);
?>

In this example:

• CURLOPT_HEADER => 1 tells curl to include the headers in the response.
• CURLOPT_VERBOSE => 1 enables verbose output, sending detailed debugging information to the standard error stream (stderr). You can redirect this output to a file for later analysis.

Best Practices

• Redirect Output: When using verbose options (like -v or CURLOPT_VERBOSE), consider redirecting the output to a file for easier analysis. For example: curl -v https://example.com > curl_log.txt
• Use --trace-ascii for Comprehensive Debugging: If you need a complete log of the communication, --trace-ascii is the most powerful option.
• Check for Errors: Always check for errors after executing curl or using Libcurl to ensure the request was successful.

By mastering these techniques, you can effectively inspect HTTP requests made by curl and diagnose communication issues quickly and efficiently.