Surviving the bad day: attackers delete your backups first

The bad day, specified

Three failure modes, in priority order. First, a root-level compromise of the VPS by the same automated adversaries this whole series defends against, now including the ransomware playbook: mass encryption or deletion, with the documented adversary behavior of disabling versioning, deleting snapshots, and wiping any backup the compromised machine can reach. ATT&CK gives it a number, T1490, Inhibit System Recovery, and government ransomware advisories now repeat the same word in every bulletin: backups must be immutable. Second, the ordinary disasters: a disk dies, a provider account closes, or I fat-finger something destructive myself. Third, silent corruption found months after it happened, which is a retention problem wearing a trench coat.

One principle answers the worst case, and it's the same principle that shaped TX·06's log collector: the machine being protected must not hold the power to destroy its own history. The collector accepted journal entries and offered no verb for deleting them; the backup architecture needs the identical property, and everything below is arranged around it.

What's actually irreplaceable

Before tooling, the list. What gets backed up is the state that can't be rebuilt: the database (as a proper dump, never the raw file, section six is entirely about this), user-uploaded data in the app volumes, the compose and Quadlet files that define the stack, the Caddyfile, Caddy's data directory (it holds the ACME account key and issued certificates; losing it is survivable but buys you rate-limit pain), the Wazuh config and custom rules that TX·06 and TX·09 built, and the journald archive on the bench collector, which is the forensic timeline and deserves the same care as the data it describes. What deliberately doesn't get backed up: container images (rebuildable from registries and Dockerfiles), node_modules (that's what the lockfile is for), package caches, and the OS itself, which a provisioning script recreates faster than any file-level restore.

The failure mode of inventories is that they're written once and rot. Mine lives as a manifest file in the same git repo as the compose and Quadlet files, and the backup script reads its paths from the manifest instead of hardcoding them, so adding a volume without adding it to the manifest is a reviewable diff instead of a silent gap. Getting the list right doesn't make the capture method right, but it makes forgetting things a merge-request problem instead of a restore-day surprise.

Immutability has tiers

Here's where the marketing language needs sorting, because "immutable" gets stamped on five different mechanisms with five different strengths. At the top sits object storage with Object Lock in Compliance mode, where per AWS's own documentation a protected object "can't be overwritten or deleted by any user, including the root user." That's the real thing: even the account owner can't shorten the clock. Everything below it bends under enough credential compromise. Governance mode has a bypass permission, which means it's convenience rather than a wall. Plain versioning dies the moment an attacker with account keys disables it, which is precisely the behavior the advisories document. Provider snapshots share credentials and fate with the account that made them. And append-only modes deserve special suspicion: restic's own docs say plainly that an attacker holding the full-access credential renders the protection void, and Borg's append-only is weaker still, its own documentation noting that other tools' deletions still work against the repository. Append hyphen only is a speed bump with good marketing. Treat it as defense in depth, never as the load-bearing wall.

The pull model earns its place on the top tier by architecture rather than flags: the bench rig initiates the backup, reads from the VPS over a restricted SSH forced command, and stores locally. The VPS holds no credential to the backup store at all, so there is nothing on it for ransomware to steal or wield. My arrangement uses both top-tier mechanisms as two independent destinations: a push to Backblaze B2 with Object Lock in Compliance mode (the survives-anything copy), and a pull to the bench rig (the fast local restore copy the VPS can't touch). An attacker who owns the VPS completely can trash the live disk and nothing else.

3-2-1 needed a patch

The classic rule, three copies, two media, one offsite, predates an adversary who actively hunts backups, which is why the modern phrasing appends two more digits: 3-2-1-1-0, one copy immutable or offline, and zero unverified restores. My three copies: the production data itself on the VPS, the B2 bucket with Compliance lock (offsite, immutable, and for a stack whose irreplaceable data sits well under B2's permanently free first 10 GB, literally free; past that it's about six dollars a terabyte-month), and the bench rig's pulled copy (fast, local, structurally out of the VPS's reach).

The honest note about the home lab: it's a genuine second medium and the low-latency restore path, and it is not the offsite copy, no matter how convenient that accounting would be. It shares an operator with the VPS (one fat-finger, one compromised SSH agent), often shares credentials, and shares a physical house with its own fire, flood, and theft. The bench rig counts as copy three; it can never count as the "1". That's what the object storage leg is for, and it's the leg the cargo-cult version of 3-2-1 always skips: three reachable copies just means the attacker encrypts three things.

The tool: restic, scheduled by systemd

I compared the big three as of this month: restic 0.19.1, Borg 1.4.5, and Kopia 0.23.1. All three are competent, encrypted, deduplicating, and actively maintained, so the choice came down to fit. Borg is the odd one out for this design: it speaks SSH and local paths only, its append-only mode is the weakest of the graded list above, and Borg 2.0 has been in don't-use-in-production beta for years. Kopia is the genuine runner-up and arguably the pick if you want the client itself driving Object Lock Compliance with no server component at all. restic wins for this fleet: one static binary on both machines, native object-storage backends so the B2 lock needs no glue, a simpler passphrase model for escrow, and a real append-only server (rest-server) for the day I want a third destination. The schedule is a systemd timer, not cron, because Persistent=true runs a missed job after downtime instead of silently skipping a night:

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/b2.env
ExecStartPre=/usr/local/bin/db-dump.sh
ExecStart=/usr/bin/restic backup \
  --files-from /etc/restic/backup-paths.txt \
  --exclude-file /etc/restic/excludes.txt \
  --limit-upload 5120 \
  --tag nightly
ExecStartPost=/usr/bin/restic forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
ExecStartPost=/usr/bin/curl -fsS --retry 3 https://hc-ping.example/your-uuid

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=1200
Persistent=true

[Install]
WantedBy=timers.target

The pieces worth noticing: the database dump runs first as ExecStartPre so restic only ever sees consistent files, the paths come from the manifest, upload bandwidth is capped so a backup never starves the game's sockets, and the final line pings a dead man's switch only on success, which the detection section turns into an alert. Retention at seven dailies, four weeklies, six monthlies gives roughly six months of history, long enough to reach back past an attacker's dwell time or a slow-burn corruption. And the immutable-bucket caveat from the field note applies to the --prune: against the Compliance bucket it's bookkeeping, not deletion, until locks expire. The pull leg on the bench rig runs its own timer with its own schedule, because two destinations that fail independently is the entire point of having two.

Never copy a live database

The single most common self-hosting backup defect: pointing the backup tool at a live database file. A database mid-write is a moving target, and the copy captures pages from different moments, a corruption you discover only at restore time. SQLite's docs are blunt that a copy taken during a write may be corrupted after recovery, and WAL mode makes it worse: recent commits live in the -wal sidecar, so copying only the main file silently loses data while looking perfectly healthy. The fix costs nothing, because both databases ship consistent-snapshot tooling:

#!/usr/bin/env bash
set -euo pipefail
TS=$(date +%F)
OUT=/srv/backups/pg/app-${TS}.dump

# -Fc: compressed custom format, restorable selectively with pg_restore
docker exec -t app-postgres pg_dump -U app -d appdb -Fc > "${OUT}.tmp"
mv "${OUT}.tmp" "${OUT}"   # atomic rename: restic never sees a partial dump

docker exec app sqlite3 /data/app.db ".backup '/data/backup/app-$(date +%F).db'"
# keep the WAL sidecar from growing without bound:
docker exec app sqlite3 /data/app.db "PRAGMA wal_checkpoint(TRUNCATE);"

pg_dump rides Postgres's MVCC snapshot, so the dump is internally consistent without blocking writers, and the app never stops. SQLite's .backup does the same through the online backup API. Neither requires downtime, which is why stopping the container to copy its file is a solution to a problem the vendors already solved. On the question of going further, WAL archiving and point-in-time recovery would shrink the recovery point from a day to minutes, and my judgment is that for one small game database it's over-engineering: nightly dumps mean at most a day of writes lost, and the operational complexity of a PITR pipeline is real, standing cost for one operator. The honest caveat that retention has to cover instead: a dump is faithfully consistent even when the data it's dumping is already damaged. Point-in-time correctness isn't semantic correctness; the six months of history is what lets you reach back to before the damage began.

Restore is the product

A backup that has never been restored is a hypothesis with a progress bar. Verification runs at two depths. The cheap layer is weekly and automated: restic check with a rotating --read-data-subset so a slice of actual pack data gets re-read and re-hashed every week without paying to re-read the whole bucket. The layer that matters is quarterly and manual: a full restore drill to a throwaway VM on the bench rig, restore the latest snapshot, load the dump into a fresh Postgres container, bring the compose stack up against the restored data, and confirm the app answers on its health route. The drill follows a written runbook that lives in the same git repo, exact commands, where the passphrase is, expected sizes, ending in a checkbox that says the app served a page.

The honest numbers for one operator, measured rather than aspirational: recovery point of about 24 hours with nightly dumps, and recovery time of two to four hours for a full rebuild from the bench-rig copy, provisioning, restore, DNS and TLS re-issue included. From B2 alone, over a residential connection, add most of a day. That asymmetry is the two-destination design earning its keep: the local copy is the fast path, the locked bucket is the survives-anything path, and the drill is what makes either number real instead of a guess. One residual to rehearse eventually: the drill restores to a lab VM, and the actual provider-rebuild path, new VPS, new network, firewall from scratch, hides its own surprises. It gets one rehearsal too.

The key is the second single point of failure

Encrypting backups creates a new way to lose them: lose the passphrase and the repository is as gone as if ransomware had taken it. The key has to satisfy two opposing constraints at once, ransomware on the VPS must not be able to use it to reach the backup store's controls, and a house fire must not be able to destroy every copy of it. The arrangement: on the VPS, the passphrase sits in a root-only file for the backup unit to read, accepting that root can read it, which is exactly why the immutability design never depends on that secret staying safe. Outside the VPS, it lives in the password manager and on paper in a second physical location, because paper survives both ransomware and bit rot, and two unrelated locations mean no single fire, theft, or account takeover loses it.

Just as important is which credentials the VPS does not hold: the B2 application key it uses can write to one bucket and nothing else, and the account-level credentials, the lock-bypass powers, and the prune-capable key all live elsewhere. Decrypt, write, and destroy are three different privileges, and the machine most likely to be compromised gets only the ones it can't hurt itself with. And a scope note on what the encryption itself buys: confidentiality against the storage provider and a stolen bench disk, tamper evidence through authenticated encryption, and nothing whatsoever for availability. Encryption is why an attacker can't read your backups; Object Lock is why they can't delete them. Different controls, both required.

The SOC watches the backups

A backup system that fails silently is indistinguishable from no backup system, usually discovered on the worst possible day. So backup health joins the TX·06 pipeline as a first-class signal, in three parts. Failures are easy: the wrapper logs to a file the Wazuh agent tails, and a custom rule raises anything that matches a failure line:

<group name="backup,">
  <rule id="100200" level="12">
    <decoded_as>syslog</decoded_as>
    <match>restic backup FAILED</match>
    <description>Nightly backup job reported failure</description>
  </rule>
</group>

Staleness is the interesting one, because a SIEM has a structural blind spot here: correlation rules count events that arrive, and a backup that simply never ran produces no event to count. Wazuh cannot alert on silence. The answer is the same dead man's switch pattern TX·08 used for certificate renewal: the backup unit pings an external check only on success, and the check alerts when the ping doesn't come, which catches the timer that got disabled, the box that went down, and the job that hangs forever, all invisible from inside. A self-hosted healthchecks instance on the bench rig does this for free; a belt-and-suspenders script also compares the latest snapshot's age against a threshold and logs a stale-backup line, converting an absence into a presence the SIEM can match.

Third, the destination is itself a target, so the bench rig's backup directory gets file integrity monitoring with whodata, recording which process and user touched what. A child rule escalates any deletion under the backup path, and a frequency layer on top flags mass deletion, the T1490 signature itself. Honest weighting, in the spirit of every detection section in this series: that alert buys response time, and only the Compliance lock buys survival. If an attacker with root is deleting backups, the alarm matters because the copies it can't reach exist. Detection is the tripwire; immutability is the wall.

The checklist

The recovery posture, compressed into what I now hold the fleet to:

• Inventory lives as a manifest in git; the backup script reads paths from it; new volumes without manifest entries fail review
• Databases dumped, never file-copied: pg_dump -Fc or SQLite .backup, atomic rename, dump before backup in the same unit
• Two independent destinations: push to object storage with Object Lock in Compliance mode, pull to the bench rig the VPS holds no credentials for
• Object Lock enabled at bucket creation, Compliance mode, retention sized past the longest keep interval; storage growth until locks expire accepted as the cost of the word immutable
• Append-only treated as defense in depth, never as the load-bearing wall; provider snapshots counted as convenience, not as a copy
• 3-2-1-1-0 honestly scored: the home lab is copy three and the fast restore path, never the offsite leg
• restic on systemd timers with Persistent=true; upload bandwidth capped; retention seven daily, four weekly, six monthly
• Weekly restic check with a rotating read-data subset; quarterly full-restore drill to a lab VM against the written runbook, ending at a serving app
• Measured, not aspirational: recovery point about a day, recovery time two to four hours from the local copy, and both numbers re-earned at every drill
• Passphrase escrowed in the password manager and on paper in a second location; decrypt, write, and destroy privileges held by different parties; the VPS holds only scoped write keys
• Backup failures alert through the SIEM; staleness caught by a success-only dead man's switch plus a snapshot-age check
• File integrity monitoring with whodata on the backup destination; mass deletion escalates as T1490
• The provider-rebuild path rehearsed at least once, not just the lab-VM restore

The arc of this series has been prevent, detect, and now survive, and the honest summary of ten articles is that the last one is the safety net under all the others: hardening reduces how often the bad day comes, detection tells you it's here, and backups decide whether it's an incident or an ending. The SOC can now page me about a backup that didn't happen, which closes the last silent failure I knew about. What it can't do yet is tell me what to do in the first ten minutes after a real page. That's TX·11: incident response for a team of one, the 3 a.m. article, where every runbook this series has written gets opened in anger.