Container security: Docker walks straight past your firewall

The firewall collision

Here's the sequence that earned this article its title. Postgres went into a container with -p 5432:5432, temporarily, for convenience, the way every temporary thing happens. The box runs the nftables ruleset from TX·05: default-deny inbound, three ports open. Then I scanned myself from outside, and 5432 answered. The firewall hadn't failed. It had never been consulted.

The mechanism is worth understanding because it's permanent knowledge. When rootful Docker publishes a port, it writes DNAT rules into the nat table, and those are evaluated in PREROUTING. Your default-deny lives in the INPUT chain, and a DNAT'd packet never traverses INPUT; it goes through FORWARD into the container's network namespace. Docker's own packet-filtering docs concede the packets are diverted "effectively ignoring your firewall configuration." Every ufw tutorial that says ufw deny 5432 is describing a chain your container traffic does not visit.

The fix is three disciplines in priority order, and the first one solves most of it: publish nothing you don't have to. Containers on the same user-defined network reach each other by service name with no published ports at all, so the database gets no ports: entry, ever, and sits on an internal: true network that can't route out. Only the reverse proxy faces the world:

services:
  db:
    image: postgres:17-bookworm@sha256:...
    networks: [backend]
    # no ports: entry. nothing to DNAT, nothing to bypass the firewall.
  app:
    build: .
    expose: ["3000"]          # reachable by the proxy, not the internet
    networks: [backend, edge]
  proxy:
    image: caddy:2@sha256:...
    ports:
      - "443:443"             # the ONLY 0.0.0.0 binding on the host
    networks: [edge]

networks:
  backend:
    internal: true            # the db tier has no external connectivity
  edge: {}

Second: anything that must be published for local-only use, a metrics endpoint, an admin UI, binds to loopback explicitly, "127.0.0.1:9090:9090", so the DNAT rule only matches traffic that originated on the box. Third, belt and suspenders for rootful Docker: the DOCKER-USER chain is the one hook Docker guarantees to honor before its own accept rules and to leave alone across restarts, so a drop there closes the class of mistake rather than the single instance:

# order matters: the RETURN for established flows must precede the DROP
iptables -I DOCKER-USER -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
iptables -I DOCKER-USER -i eth0 -j DROP

Two closing notes on this mess. Rootless Podman has the exact opposite failure mode: it can't touch the host firewall at all, a published port is just an unprivileged process listening on the host, and your INPUT rules govern it like anything else. It under-connects rather than over-exposes, which is the direction you want to fail in. And the tempting nuclear option, setting "iptables": false in the daemon config, is a cargo-cult fix that breaks container networking outright and can still leave ports reachable from the local network. Don't disable the machinery; stop publishing things.

Rootful, rootless, or Podman

Three privilege models. Rootful Docker runs a persistent daemon as root, and container root is host root wearing a seatbelt of dropped capabilities and the default seccomp profile. Rootless Docker keeps the daemon architecture but pushes the whole thing into a user namespace, so container root maps to an unprivileged host UID. Podman drops the daemon entirely, forks the runtime per invocation, and treats rootless as the native mode, with systemd integration via Quadlet units that feel right at home after TX·05's sandboxing section.

The distro politics mirror the firewall table from TX·05. Debian packages both and blesses neither; the RHEL family blesses Podman outright, integrates it with SELinux, and doesn't ship Docker at all. The Debian-specific catch is version lag: trixie's docker.io package is Docker 26 while upstream is on 29, and its Podman is 5.4 against an upstream 6.0. Debian backports security fixes, but that gap is a live fact you have to manage, and it comes back with teeth in the escapes section below.

My call, for a new single-admin box with this profile: rootless Podman under Quadlet, because removing the root daemon removes the single juiciest escalation target on the host, and daemonless fits a box where systemd is already the supervisor. Rootless Docker is an entirely defensible second place if Compose fluency matters to you, same security posture where it counts. Rootful Docker is the option you accept only when a dependency forces it, and then the loopback binding and DOCKER-USER disciplines stop being advice and become requirements. The honest costs of rootless: binding ports below 1024 takes an extra step (irrelevant here, the proxy owns 443 and everything else is loopback), and traffic crosses a userspace network stack with measurable overhead that a reverse-proxied web app will never feel. What rootless does not do is patch the kernel. "Rootless fixes everything" is this article's first cargo-cult flag: it shrinks blast radius, and the shared-kernel escape surface stays exactly where it was.

The image: thin, pinned, and shell-free

The base-image debate for Node has a boring answer that the "use Alpine, it's smaller" folklore keeps obscuring. Alpine uses musl instead of glibc, and native modules, anything touching node-gyp, can build differently, resolve DNS differently, or fail in ways you discover in production. Fewer CVEs in the base image is not the same thing as a secure application, and the musl tax can cost more than the megabytes save. The default that wants no caveats: a Debian-based build stage, and a distroless runtime stage. Distroless ships no shell and no package manager, which means a popped app can't fetch a second stage, poke around, or drop into an interactive anything. It removes the attacker's toolbox instead of just watching them use it.

# build stage: full toolchain, never ships
FROM node:24-bookworm@sha256:... AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev
COPY . .

# runtime stage: glibc distroless, no shell, non-root baked in
FROM gcr.io/distroless/nodejs24-debian12:nonroot@sha256:...
WORKDIR /app
COPY --from=build --chown=nonroot:nonroot /app /app
USER nonroot
EXPOSE 3000
CMD ["server.js"]

Details that carry weight: npm ci against the lockfile, never npm install, so the build is reproducible; the cache mount keeps npm's cache out of the layers; the nonroot distroless variant means the non-root user exists without me creating it; and both base images are pinned by digest, not tag. :latest in production is the canonical anti-pattern, a standing invitation for an upstream change to ship itself to your host unreviewed, and even a version tag moves under you. A digest doesn't move. Updates then become deliberate: a bot like Renovate opens a pull request bumping the digest, and I read it before it lands, the same deliberate-update posture TX·05 took with unattended upgrades scoped to security only.

One scanner, one SBOM, no ceremony

Supply-chain tooling is where enterprise process most often gets transplanted onto solo operators without its context. What earns its keep at this scale is one scanner wired into the build, failing on what's actually fixable:

trivy image --exit-code 1 --ignore-unfixed --severity HIGH,CRITICAL app:release
trivy image --format cyclonedx --output sbom.json app:release

Trivy wins the solo-operator slot because one binary covers images, filesystems, Dockerfiles, and secrets, and emits the SBOM in the same breath. The SBOM's value for one person is concrete and small: when the next big CVE lands, I rescan the manifest in seconds without rebuilding anything. Grype is a sharper pure matcher and a fine second tool, someday. Signing is where I draw the line deliberately: cosign-verifying upstream base images where publishers sign them, yes, it's a flag on the pull. Standing up keyless signing infrastructure to sign images for myself, no. I am the only consumer of my own builds, there's no admission controller to convince, and the digest pin already provides the integrity property. That ceremony starts paying rent when there's a second operator or a Kubernetes cluster, and this bench has neither.

The runtime hardening block

TX·05 sandboxed systemd services with a drop-in of restrictions; this is the same move one layer up, and it goes on every service in the compose file:

  app:
    build: .
    user: "10001:10001"
    read_only: true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=64m
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
      - apparmor=docker-default    # SELinux labels on the RHEL family
    pids_limit: 200
    deploy:
      resources:
        limits:
          memory: 256M
          cpus: "0.50"
    healthcheck:
      test: ["CMD", "node", "-e",
        "require('http').get('http://127.0.0.1:3000/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"]
      interval: 30s
      timeout: 3s
      retries: 3

What each line buys, against this threat model. The non-root user removes the prerequisite most escape paths assume. read_only plus a small noexec tmpfs means a web shell has nowhere durable to land, the same category-elimination logic as key-only SSH. cap_drop: ALL strips every Linux capability, and this app needs none back; the dangerous ones like CAP_SYS_ADMIN are exactly what escape techniques shop for. no-new-privileges means a setuid binary lurking in some base layer can't elevate anything. Docker's default seccomp profile stays on, the AppArmor label keeps TX·05's MAC story intact inside the containers, and the pids and memory limits blunt fork bombs and memory exhaustion, the container-shaped cousin of the DoS budgeting from TX·04. The healthcheck isn't a security control; it's how a wedged or hijacked process becomes visible instead of quietly wrong.

The socket is the crown jewels

One compose line undoes everything above: /var/run/docker.sock mounted into a container. The socket is the daemon's API, the daemon runs as root, and any process holding the socket can ask for a privileged container with the host's root filesystem bind-mounted inside. That's not a vulnerability, it's the feature working as designed, and the auto-updater projects that need the socket say as much in their own documentation. Mounting it into a convenience container converts your entire blast-radius design into a single hop: pop the helper, or poison the helper's image upstream, and the host is gone.

So: no blanket auto-updaters. The alternatives cost almost nothing. Deliberate digest-bump PRs handle images; podman auto-update under Quadlet does controlled refreshes with no socket-mounted daemon at all; and if some dashboard genuinely needs the API, a filtering socket proxy that exposes only the read-only endpoints it uses turns "root on the host" back into "can list containers." Rootless shrinks the stakes here too, since the socket it exposes isn't root's. But the clean answer is that nothing on an internet-facing box holds the socket except the admin.

The boundary is the kernel, and it's shared

"Containers are a security boundary" needs its qualifier said out loud: they're namespaces, cgroups, and syscall filters wrapped around a process that shares the host kernel. Against automated attackers that's a useful boundary. Against kernel exploitation it's a polite suggestion. At this threat level, escapes come in exactly two flavors, and both are within your control.

Flavor one is misconfiguration, and it's what bots actually find: --privileged containers, mounted sockets, host PID or network namespaces, over-broad bind mounts, gratuitous added capabilities. Every item on that list is a choice, which is why the hardening block exists. Flavor two is runtime CVEs, and the case study is recent: a cluster of three runc vulnerabilities disclosed in November 2025 by one of runc's own maintainers, abusing proc masking and console bind-mount races to reach host root, with two of the three affecting every runc version ever shipped. Fixes landed in runc 1.2.8 and 1.3.3. The analysis that followed pointed at untrusted images and Dockerfiles as the likely delivery vehicle, which closes the loop neatly: the supply-chain hygiene from the image section is also your escape mitigation. The year before it was BuildKit's turn, the Leaky Vessels cluster, fixed in 0.12.5. Same lesson as regreSSHion in TX·05: config didn't save anyone, patch cadence did.

Secrets: where each method leaks

Secrets in containers leak in enumerable places, so enumerate them. Environment variables show up in docker inspect, in /proc/<pid>/environ, in crash dumps and error reporters, and in every child process that inherits them. Build args are worse: they bake into image layers permanently, and a COPY of a .env file deleted one layer later is still fully present in the earlier layer, because layers are additive and immutable; docker history and a layer browser read them back out years later. And a -e PASSWORD=... on the command line archives itself in shell history. The pattern that avoids all of it on one host, with no vault to run:

services:
  app:
    secrets: [db_password]
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password   # app reads the file

secrets:
  db_password:
    file: ./secrets/db_password.txt   # mode 600, gitignored, outside build context

The app reads the _FILE path at startup, a convention most libraries already understand, and the value never appears in an env listing or an image layer. Build-time credentials, a private registry token say, go through BuildKit's --secret mount, which exists precisely so they never persist to a layer. Two rules complete the posture. Anything secret-shaped that ever touched a git commit or a pushed image is burned: rotate it, because rewriting history un-publishes nothing from forks, clones, and CI caches. And keep the launching shell's environment empty of secrets entirely; a Podman bug fixed this cycle leaked host environment variables into containers via malformed image metadata, which is the kind of bug you shrug at when your shell holds nothing worth stealing.

Teaching the SOC about containers

TX·06 ended with twelve detections and a tuned alert path; the containers need to join it, and the honest constraint is CPU. The floor costs nothing: the auditd ruleset already shipping off-box grows a few container-shaped rules, watching the compose files and unit files that define the stack, exec calls into the runtime binaries, and the socket path. Auditd's blind spot is context, it sees syscalls and paths but not which container or image, which is exactly what Falco adds. Falco is the CNCF's eBPF runtime sensor, it speaks container-native, and its community shipped detection rules for the runc escape symptoms within days of the disclosure, which is the responsiveness you want in a sensor. Its price is real CPU on a small VPS, so the deployment that makes sense here is host-level Falco scoped to a handful of rules, JSON output into the same Wazuh pipeline, measured before trusting, exactly like every other resource decision in this series.

The handful, each mapped for the coverage review: a privileged container or one gaining dangerous capabilities appearing on this host (ATT&CK T1610, and the setup move for T1611, escape to host); any mount of the Docker socket into a container, same pair; an exec into a running container spawning an interactive shell (T1609); any shell process at all inside the distroless app container, which by construction has no shell, so a shell there is by definition someone else's (T1059); unexpected outbound from the backend network, the earliest possible sign the app is popped; and procfs symlink anomalies, the observable symptom of the runc escape class. Six rules, correlated against the host telemetry TX·06 already collects, and the "assume the app container is popped" scenario has eyes on every next move it could make.

The checklist

The whole article, compressed into what I actually enforce on the stack now:

• Database on an internal: true network with no ports: entry; the proxy's 443 is the only 0.0.0.0 binding on the host
• Local-only publishes bound to 127.0.0.1 explicitly
• Exposure verified from outside with nmap -Pn after every topology change; ufw status trusted for nothing
• DOCKER-USER drop on the external interface wherever rootful Docker runs, established flows returned first
• Rootless runtime, Podman with Quadlet preferred; rootful only under duress and then with the full discipline set
• Runtime patch level verified against the current escape CVEs, distro backports confirmed in the changelog or upstream repos used instead
• Multi-stage builds; glibc distroless runtime, non-root, no shell; Alpine only after the native modules pass tests on musl
• Base images pinned by digest; bumps arrive as reviewable PRs, never as :latest surprises
• Trivy gating CI on fixable HIGH/CRITICAL; SBOM emitted per release for rebuild-free rescans
• Hardening block on every service: non-root user, read-only rootfs plus noexec tmpfs, cap_drop: ALL, no-new-privileges, pids and memory limits, MAC label kept on
• Nothing holds the Docker socket; updates via digest PRs or podman auto-update; a filtering proxy if some tool truly needs the API
• Secrets as mounted files with the _FILE convention; BuildKit --secret at build time; anything leaked is rotated, not scrubbed
• Container detections wired into the TX·06 pipeline: auditd extensions as the floor, scoped host-level Falco where CPU allows, all of it ATT&CK-tagged

The interview question this whole stack answers is the one the job postings keep converging on: walk through a production Dockerfile and justify every line, why the user isn't root, why the base is a digest and not a tag, why there's no shell in the runtime image. As of this month, Digital Descent's backend runs exactly this way on the TX·05 box, the SOC from TX·06 watches it, and the firewall finally means what it says again. The database port that started this article is back behind default-deny where it belongs, and this time I have the outside-in scan output to prove it.