Managing Docker Container Logs: Rotation and Clearing

Docker containers generate logs that can grow significantly over time, potentially consuming substantial disk space. Understanding how to manage these logs – both in terms of preventing excessive growth and clearing existing logs – is crucial for maintaining a healthy and efficient Docker environment. This tutorial will cover common techniques for controlling and managing Docker container logs.

Understanding Docker Logging Drivers

Docker utilizes logging drivers to handle how container logs are collected and stored. The default driver, json-file, stores logs in a JSON format on the host machine. While convenient, this can lead to large log files if not managed correctly.

Automatic Log Rotation

The most effective approach to managing Docker logs is to configure automatic rotation. This ensures that log files are periodically rotated, compressed, and potentially removed, preventing them from growing indefinitely. You can configure this globally for the Docker daemon or on a per-container basis.

Global Configuration (Daemon-Level)

To configure global log rotation, modify the daemon.json file (typically located at /etc/docker/daemon.json on Linux systems). Add or modify the log-driver and log-opts sections:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

• max-size: Specifies the maximum size of a log file before it’s rotated (e.g., "10m" for 10 megabytes).
• max-file: Defines the number of rotated log files to keep (e.g., "3" will keep the current file and the three most recent rotated files).

After modifying daemon.json, reload the Docker daemon to apply the changes:

sudo systemctl reload docker

This configuration will be applied to all subsequently created containers.

Per-Container Configuration

You can override the global settings on a per-container basis using the --log-opt flags with the docker run command:

docker run --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 your_image

Alternatively, within a docker-compose.yml file, configure logging under the service definition:

version: '3.7'
services:
  your_service:
    image: your_image
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Clearing Existing Logs (Manual Approach)

While automatic rotation prevents excessive growth, you may occasionally need to clear existing logs manually. Caution: Directly manipulating log files can potentially corrupt them if done incorrectly while Docker is writing to the file. It’s generally best to rely on rotation, but if manual clearing is required, proceed with caution.

Using truncate

The truncate command can be used to clear the contents of a log file. However, be extremely careful when using this approach.

sudo truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

Replace <container_id> with the actual container ID. You can find this using docker ps.

Important: Running truncate while Docker is actively writing to the log file can lead to an invalid log file, causing errors when attempting to view logs with docker logs.

Alternate Logging Drivers

Consider the local logging driver as an alternative. It stores logs in a binary format, which is more compact than JSON.

To use the local driver, configure it in daemon.json:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Caveat: The local driver is not compatible with log forwarders that rely on parsing JSON formatted logs (e.g., Filebeat, Splunk’s universal forwarder).

Viewing Logs

Regardless of your logging strategy, you can always view container logs using the docker logs command:

docker logs <container_id>

You can follow the logs in real-time using the -f flag:

docker logs -f <container_id>

You can also limit the number of lines displayed with the --tail flag.

docker logs --tail 100 <container_id>