Tutorial: Hardening a Docker Compose File for Unattended Operation on a Raspberry Pi

A Compose file that starts a stack is written in ten minutes. A Compose file that keeps the stack running for a year takes about six more lines per service, and each of those lines exists because of a specific way things fall over.

What follows goes through them in the order they usually become necessary – which is roughly the order in which a machine left alone runs into them.

The Line That Decides Whether It Comes Back

A container that stops stays stopped. After a power cut, after a crash, after a daemon restart – unless a restart policy says otherwise.

services:
  app:
    restart: unless-stopped

Three values are worth telling apart. no is the default and means nothing comes back. always restarts the container in every case, including after it was stopped by hand and the machine then rebooted – which is how a container deliberately taken out of service reappears at three in the morning. unless-stopped does the same thing except in that one case, and is the right default for a machine nobody is watching.

Two details behind it. Docker backs off between restart attempts, doubling from 100 milliseconds up to a minute, so a container in a crash loop does not saturate the machine – but it also does not stop trying, and nothing draws attention to it. And the restart policy only applies to a container that exits. A process that hangs while its container stays up is exactly the case the next section covers.

A Healthcheck That Tests Something

A hung application is worse than a crashed one, because everything around it believes it is working. The container is running, the port is open, and the requests go nowhere.

    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s

The last line is the one most often missing and the one that causes the most confusion. During start_period, a failing check does not count towards retries. Without it, a service that needs more than three check intervals, here about 90 seconds, to come up is declared unhealthy on the way there, and services waiting for it via condition: service_healthy do not start. An unhealthy container is not restarted, though: without Swarm the result only changes its status, and the restart policy still applies only once the container exits.

What the check tests matters more than that it exists. A request to / proves the web server answers, which it will do while the database connection behind it has been gone for an hour. An endpoint that touches the parts that can fail is worth writing for exactly this purpose – and it should return quickly, because it runs every thirty seconds for the life of the machine.

A healthcheck also makes an ordering statement possible, which depends_on alone does not. Without the condition, Compose starts the dependency and moves on immediately; with it, it waits.

  app:
    depends_on:
      db:
        condition: service_healthy

One consequence for slim images: curl is often not installed, and a healthcheck that cannot run counts as a failure. Either the check uses something the image has – wget -q --spider, or a command the application itself brings – or the image gets a package it would otherwise not need.

Logs That Do Not Fill the Card

The default logging driver writes to a JSON file that grows without limit. On a server with a large disk that is untidy; on a Pi with a memory card it is a failure mode, because a full file system stops the whole machine rather than the noisy container.

    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Thirty megabytes per service is generous for anything that is not being actively debugged. The setting belongs on every service and not only on the chatty one, because the chatty one is usually the service that starts logging a stack trace once a second at some point.

The same limit can be set once for the whole machine in /etc/docker/daemon.json, which is the better place for it – though it only applies to containers created afterwards, so an existing stack has to be recreated for it to take effect.

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Where the writes themselves are the problem rather than the space, a memory card benefits from moving the logs out of the file system entirely – driver: journald hands them to systemd, which can be told to keep them in RAM.

Memory Limits, and the Pi-Specific Trap

A machine with four gigabytes and no swap has no room to absorb a leak. The process that gets killed when memory runs out is chosen by the kernel, and it is regularly not the one responsible.

    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.5"

A limit turns a machine-wide failure into a container-wide one: the offending container is killed and restarted by its restart policy, and everything else keeps running. That is a much better outcome than the alternative, and it is worth setting even when the number is a guess – a generous limit still bounds the damage.

And here is the part specific to the Raspberry Pi. On Raspberry Pi OS the memory cgroup controller is disabled by default, and a memory limit in the Compose file is then accepted without complaint and has no effect. A warning appears once at daemon start and never again.

# append to the single line in /boot/firmware/cmdline.txt, then reboot
cgroup_enable=memory cgroup_memory=1

# afterwards, this line has to be gone:
docker info 2>&1 | grep -i "no memory limit support"

The file contains exactly one line and a line break added by an editor makes the machine unbootable, so the addition goes at the end of the existing line, separated by a space.

Pinning the Image So a Restart Is Not an Upgrade

A tag is a moving pointer. postgres:16 means something different this month than last, and the moment it changes is not chosen by anybody – it is whenever the container happens to be recreated.

    image: postgres:16.4@sha256:f8cd4e0b9b0d15b6bbbcbbdf8c5f8c8b6c6b3e9b7cbb4f2d1e3a5c7d9f0b2a46

A digest is immutable, which turns “recreate the container” into an operation with a known result. The tag stays in front of it as documentation – the digest alone is unreadable, and a file full of hashes tells nobody what is running.

The digest for an image already in use is one command away.

docker image inspect postgres:16.4 --format '{{index .RepoDigests 0}}'

One warning about the combination with automatic updates. A tool that pulls new images on a schedule and a pinned digest are opposites, and running both means the pin quietly wins while the update tool reports success. Either the versions move on a schedule, or they move when somebody decides – both are defensible, and having one of them by accident is not.

Two more lines belong in the same block and cost nothing. A container that never writes to its own file system can say so, and one that does not need to become root can be told it never will.

    read_only: true
    tmpfs:
      - /tmp
    security_opt:
      - no-new-privileges:true
    user: "1000:1000"

Checking the File Before It Runs

Compose resolves variables, merges override files and applies defaults, and the result is often not what the file appears to say. One command prints what will actually be started.

docker compose config

Check the output for three things: empty variable values, images without a digest, and services missing required settings. An unset variable becomes an empty string without an error, and a missing password can therefore be interpreted as no password. Also check each service for restart, healthcheck and logging; these omissions are easy to overlook in a file with eight services.

The last one can be checked mechanically, which is worth doing because it is exactly the kind of omission that happens when a service is added quickly.

docker compose config --format json | python3 -c '
import json, sys
stack = json.load(sys.stdin)
for name, svc in stack["services"].items():
    fehlt = [f for f in ("restart", "healthcheck", "logging") if f not in svc]
    if "@sha256:" not in svc.get("image", ""):
        fehlt.append("digest")
    print(f"{name:20s} {\", \".join(fehlt) if fehlt else \"ok\"}")
'

After the first run, one last check belongs to the routine and needs no tooling: reboot the machine on purpose and see what comes back. A stack that has never survived a restart has not been tested, and the moment to find that out is not the morning after a power cut.