Downloading Files Over HTTP Using Python

Introduction

In this tutorial, we will explore how to download files over HTTP using Python. This is a common requirement when dealing with web scraping or automating tasks that involve downloading resources like images, documents, or media files. Python offers several built-in libraries and third-party packages that make this task straightforward.

Using urllib

Python’s standard library provides the urllib module for working with URLs. It includes several submodules:

• urllib.request: For opening and reading URLs (available in Python 3).
• urllib2: Used in Python 2 for similar purposes.

Example: Using urlretrieve

The urlretrieve function is a convenient way to download files. Here’s how you can use it:

import urllib.request

# Download the file from the specified URL
urllib.request.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

This code downloads an MP3 file and saves it locally as mp3.mp3.

Example: Using urlopen

For more control, such as reading the content directly, you can use urlopen:

import urllib.request

# Open a URL and read its content
with urllib.request.urlopen('http://www.example.com/') as response:
    html = response.read().decode('utf-8')

print(html)

This method opens the URL and reads its HTML content.

Using requests

The requests library is a popular third-party package for making HTTP requests. It simplifies many tasks compared to urllib.

Installing Requests

First, ensure you have the requests library installed:

pip install requests

Example: Downloading with requests

Here’s how you can download a file using requests:

import requests

# Specify the URL of the file to download
url = "http://download.thinkbroadband.com/10MB.zip"

# Send an HTTP GET request and get the response
response = requests.get(url)

# Print the content length (size in bytes)
print(len(response.content))

This code downloads a ZIP file and prints its size.

Example: Saving a File with Progress Bar

You can also download files with a progress bar using tqdm:

from tqdm import tqdm
import requests

url = "http://download.thinkbroadband.com/10MB.zip"
response = requests.get(url, stream=True)

# Open the file in binary write mode
with open("10MB", "wb") as handle:
    for data in tqdm(response.iter_content(chunk_size=1024), unit="kB"):
        handle.write(data)

This script downloads a large file while displaying a progress bar.

Best Practices

1. Error Handling: Always include error handling to manage network issues or invalid URLs.
2. Streaming Large Files: Use streaming for large files to avoid memory overload.
3. Binary Mode: When writing binary files, use 'wb' mode to prevent data corruption.
4. User-Agent Header: Consider setting a User-Agent header if the server requires it.

Conclusion

Downloading files over HTTP in Python can be accomplished using both built-in libraries like urllib and third-party packages like requests. Each method has its advantages, and choosing one depends on your specific needs and preferences. Whether you need simplicity or advanced features like progress tracking, Python provides robust solutions for file downloading tasks.