Interacting with Web Services Using cURL in Linux

cURL is a powerful command-line tool for transferring data with URLs. It’s incredibly versatile and widely used for interacting with web services, retrieving data from APIs, and automating web-based tasks. This tutorial will guide you through using cURL in Linux to make both GET and POST requests to web services, along with techniques for formatting the responses.

Understanding HTTP Methods: GET and POST

Before diving into cURL commands, it’s important to understand the basic HTTP methods:

• GET: Used to retrieve data from a server. Think of it as asking the server for information. GET requests generally don’t modify data on the server.
• POST: Used to send data to a server to create or update a resource. POST requests can modify data on the server.

Making a GET Request

The simplest way to make a GET request using cURL is:

curl http://hostname/resource

This command will fetch the resource at the specified URL and print the response to your terminal.

Specifying Content and Accept Headers

Web services often communicate data in specific formats, such as JSON or XML. You can tell the server what format you accept and what format you are sending using the -H option to set HTTP headers.

• Accept: Specifies the media type(s) that the client is willing to accept.
• Content-Type: Specifies the media type of the body of the request.

Here’s how to specify headers for JSON and XML:

• JSON:

  curl -H "Accept: application/json" -H "Content-Type: application/json" http://hostname/resource
  

• XML:

  curl -H "Accept: application/xml" -H "Content-Type: application/xml" http://hostname/resource
  

Making a POST Request

To make a POST request, you need to provide data to the server. There are a few ways to do this with cURL:

• Sending Form Data: Use the --data option for simple key-value pairs, similar to HTML form submissions.

  curl --data "param1=value1&param2=value2" http://hostname/resource
  

• Sending JSON or XML Data: You can send a JSON or XML payload directly in the request body. This is commonly used for RESTful APIs.

  curl -X POST -H "Content-Type: application/json" -d '{"key1": "value1", "key2": "value2"}' http://hostname/resource
  

• Uploading Files: Use the --form option to upload files.

  curl --form "[email protected]" http://hostname/resource
  

  Replace filename.txt with the actual path to your file.

Formatting the Output

Raw JSON or XML output can be difficult to read. Here are some tools to help format the output:

• JSON Formatting (using Python, Ruby, or other tools):

  • Python: If you have Python installed, you can use json.tool:

    curl -i -H "Accept: application/json" http://hostname/resource | python -m json.tool
    

  • Ruby: If you have Ruby installed, you can use cjson:

    curl -i -H "Accept: application/json" http://hostname/resource | cjson
    

  • yajl-tools (Linux): sudo apt-get install yajl-tools then:

    curl -i -H "Accept: application/json" http://hostname/resource | json_reformat
    

• XML Formatting (Linux):

  • xmllint: sudo apt-get install libxml2-utils then:

    curl -H "Accept: application/xml" -H "Content-Type: application/xml" -X GET http://hostname/resource | xmllint --format -
    

  • tidy: sudo apt-get install tidy then:

    curl -H "Accept: application/xml" -H "Content-Type: application/xml" -X GET http://hostname/resource | tidy -xml -i -
    

Saving the Response to a File

You can redirect the output of cURL to a file using the > or >> operators:

• >: Overwrites the file if it exists.
• >>: Appends to the file if it exists.

curl http://hostname/resource > response.txt
curl http://hostname/resource >> response.txt

Or, use the -o option:

curl http://hostname/resource -o response.txt

Getting Help

For a complete list of cURL options, use:

curl -h

Or, for detailed documentation:

man curl