Using Proxies with Python's Requests Library

Python’s Requests library is a popular tool for making HTTP requests in Python. One of its features is support for proxies, which can be useful for routing requests through intermediate servers or services. In this tutorial, we’ll cover how to use proxies with the Requests library.

Introduction to Proxies

A proxy server acts as an intermediary between your application and the target server. When you make a request through a proxy, the request is first sent to the proxy server, which then forwards it to the target server. The response from the target server is also sent back through the proxy server before being returned to your application.

Specifying Proxies with Requests

To use a proxy with the Requests library, you need to specify the proxy URL in a dictionary that maps protocols to proxy URLs. The dictionary should have the following structure:

proxies = {
    "http": "http://proxy_url:port",
    "https": "https://proxy_url:port",
    # Optional: FTP proxy
    "ftp": "ftp://proxy_url:port"
}

You can then pass this dictionary to the requests.get() or other request methods using the proxies parameter:

import requests

url = "http://example.org"
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.10:1080"
}

response = requests.get(url, proxies=proxies)

Note that you can specify different proxy URLs for each protocol (HTTP, HTTPS, and FTP).

Using Environment Variables

Alternatively, you can set environment variables to specify the proxy URLs. The Requests library will automatically use these environment variables if they are set:

import os

os.environ["http_proxy"] = "http://10.10.1.10:3128"
os.environ["https_proxy"] = "https://10.10.1.10:1080"

response = requests.get("http://example.org")

On Linux, you can set these environment variables using the export command:

export http_proxy=http://10.10.1.10:3128
export https_proxy=https://10.10.1.10:1080

On Windows, you can use the set command:

set http_proxy=http://10.10.1.10:3128
set https_proxy=https://10.10.1.10:1080

Using Authentication with Proxies

If your proxy server requires authentication, you can specify the username and password in the proxy URL using the following format:

proxies = {
    "http": "http://username:password@proxy_url:port"
}

For example:

proxies = {
    "http": "http://user:[email protected]:3128"
}

Using Sessions with Proxies

If you want to persist cookies and session data across multiple requests, you can use the requests.Session class and set the proxies attribute:

import requests

proxies = {
    "http": "http://user:[email protected]:3128",
    "https": "https://user:[email protected]:3128"
}

session = requests.Session()
session.proxies = proxies

response = session.get("http://example.org")

Conclusion

Using proxies with the Requests library is a straightforward process that involves specifying the proxy URL in a dictionary and passing it to the request method. You can also use environment variables or authentication with your proxy server if needed. By following these examples, you should be able to use proxies effectively with the Requests library.