Incident response for one: you can't clean a rooted box

The standard changed while nobody was looking

Housekeeping first, because it dates every tutorial you'll find. The canonical incident response document, NIST SP 800-61, got its first revision in thirteen years in April 2025: revision 3 maps the old lifecycle onto the six functions of the Cybersecurity Framework and quietly withdraws revision 2, so anything still teaching "the four NIST phases" as current is citing a retired document. More interesting than the reshuffle is the admission inside it: the old model was written for a world where incidents were rare, narrow, and handled by a dedicated team. That world was never mine. So this article uses the new framing where it helps and leans on the practitioner's checklist that never went away, the SANS six steps, for muscle memory, all of it adapted down to one responder with no one to hand off to and nobody to page for a second opinion.

The first ten minutes are discipline

Minute zero is not a command, it's a file. Open the incident notes and timestamp everything in UTC from the first alert onward: what fired, what you checked, what you concluded, what you typed. Under adrenaline, one brain does not reliably reconstruct its own last hour, and the notes file is the only version of events that will survive contact with your memory:

2026-08-14T14:03:12Z  Wazuh 5710 (sshd) + sudo alert from 203.0.113.9
2026-08-14T14:04:01Z  No legit login from that ASN in 90 days. Treating as real.
2026-08-14T14:06:40Z  Scoping: edge log clean, journal shows shell for user deploy

Then verify before acting, because a single alert is a hypothesis, not an incident. Corroborate against a second, independent source: the alert plus the off-box journal, or the alert plus the edge log. Industry false-positive numbers are all over the place but uniformly high, half or more of queue volume by some measures, and the entire reason TX·06 tuned for three to five pages a day was so that a page still means something; honor that by confirming rather than reflexively believing. Next, scope: does the evidence stay inside the app boundary, or is there host-level activity, a new process, an unexpected outbound connection, a file change under /etc? Then severity, a scale agreed with yourself in advance: SEV-3 for an anomaly with no confirmed compromise, SEV-2 for a confirmed account or container compromise with host integrity intact, SEV-1 for anything host-level, anything touching data, backups, or secrets.

And the cardinal rule of the whole first phase: do not type destructive commands into a box you don't understand yet. No kill, no rm, no reboot. A reboot destroys the volatile memory you might want in twenty minutes and tells the attacker you noticed. The first ten minutes are read-only: ps, ss -tunp, journalctl, podman ps, observed and written down. What triage doesn't achieve, stated plainly: it tells you whether you have an incident and how bad, and nothing about how they got in, how long they've been there, or whether the backups are safe. That's investigation, and it comes after containment.

Four moves, one box

On a single VPS there are exactly four containment moves, and each preserves some evidence by destroying other evidence. Firewall isolation, dropping traffic while the box keeps running, preserves memory and disk and leaves an attacker able to act locally. Stopping the compromised service or container narrows the foothold while keeping the host alive. A provider snapshot followed by a halt captures the disk at a point in time and loses whatever was only in RAM. Pulling the plug closes the attacker's window instantly and pays for it with the memory and a small risk of filesystem inconsistency. The sequence that gets the most from all four, for most incidents: isolate first, which freezes the situation and buys thinking time; snapshot second, which banks durable evidence; then decide, with the notes file open, whether the box halts or keeps running under observation.

# freeze the box: only my admin address in, no new connections out
nft add rule inet firewall input ip saddr != $ADMIN_IP drop
nft add rule inet firewall output ct state new drop

The one exception inverts the order: active destruction in progress, ransomware chewing through files, a database being dropped, is the moment speed beats evidence. Halt now, snapshot the halted disk after, and be at peace with the lost RAM, because TX·10's locked bucket is the reason you can afford that trade. And know what containment is not: it stops the bleeding, and it removes no persistence, explains no root cause, and makes nothing safe to keep. Containment is not eradication.

Account or host: the trust decision

The fork in every response is one question: is this a compromised account on a trustworthy host, or is the host itself gone? The account case is where TX·09's unfashionable architecture pays its dividend. Server-side sessions revoke server-side: kill every session for the account, rotate the session secret if there's any doubt about scope (which invalidates every cookie in existence), force a credential reset, revoke the account's API keys, and read back through what the account touched. No rebuild, provided you can show the compromise never left the application boundary.

The host case deletes all of those options, because an attacker with the box reads the session store and the secrets directly, and every revocation tool you'd use runs on hardware they control. So write down the criteria that flip the switch, and hold to them when you're tired: any evidence of root or sudo escalation; unexplained kernel modules, users, or SSH keys; any binary or package whose hash you can't explain; an attacker shell as any user that can reach secrets; or the honest state of simply being unable to rule root out. That last one is the clause that matters. On a single box with no EDR and no second opinion, "I can't prove it's clean" is functionally identical to "it's dirty," and the section after next is what dirty means.

Evidence without the theater

The goal at this scale is not courtroom forensics. It's preserving enough state to answer the investigation's questions, and to re-answer them if your first theory turns out wrong. In priority order. First, always, the provider snapshot: the cheapest atomic disk image a hobbyist can get, labeled with a timestamp, never booted in place. Second, memory, with honest expectations: AVML is a static binary built for exactly this, trying /proc/kcore and friends and able to stream the capture off-box so nothing lands on the compromised disk, and its own docs concede that kernel lockdown defeats it entirely. The alternative, LiME, wants a module compiled for your exact kernel, which is friction you don't have at 3 a.m. Third, container state before anything is torn down, and the off-box journal archive, which is your highest-integrity evidence precisely because it left the box before the attacker could edit it:

podman inspect app  > /evidence/inspect.json   # config + runtime state
podman diff app     > /evidence/diff.txt       # A added, C changed, D deleted
podman export app -o /evidence/rootfs.tar      # NOTE: volumes not included
podman logs app     > /evidence/app.log
sha256sum /evidence/* > /evidence/SHA256SUMS   # hash at capture, work on copies

Soundness on a budget is three habits: hash everything the moment it's captured, work only on copies and verify the hash either side of any analysis, and write a one-paragraph custody note per artifact, what it is, when, from where, the hash, where it lives now. And the honest line about theater: write-blockers, memory-profile safaris, and courtroom-grade custody chains are cosplay at this scale. If it ever genuinely escalates, file with the authorities and let professionals image things; your energy belongs in the timeline and the rebuild.

The timeline you already own

The investigation's four questions are what happened, when, through what, and how far, and the stack built across this series already holds the answers in four places. The edge log is the front door: the exploit request, the odd user-agent, the path nobody legitimate ever hits. The off-box journals are the sequence: process starts, auth events, the shell appearing, trustworthy precisely because they shipped before tampering was possible. The auditd trail is the kernel's version: file writes to identity paths, executions, the things TX·05 started watching before there was anything to watch for. And the SIEM is the correlation layer with its file-integrity events and who-data attribution. Pull each source for a generous window around the alert, normalize to UTC, merge, and the story usually reads left to right: edge shows the entry, journal shows the shell, auditd shows the persistence write, FIM confirms it.

journalctl --directory=/var/log/journal/remote --since "2026-08-14 12:00" --until "2026-08-14 15:00"
ausearch -k cred_access -ts today          # keyed auditd hits (TX·05/·06 rules)
aureport -au                                # auth summary
aureport -x                                 # executables seen

Then the persistence sweep, which doubles as a review of whether the detections earned their keep: new or modified users (T1136, the identity FIM rule should have fired), every user's authorized_keys (T1098.004, real-time FIM), cron drops and spools (T1053.003), new systemd units and timers (T1053.006), rogue containers or modified entrypoints, and the shell profiles (T1546.004). Every row where something is found but no alert fired is a detection gap that goes straight into the post-incident review. The blind spots, stated plainly because pretending otherwise is how solo operators get overconfident: memory-only malware left nothing on disk and died with the RAM you maybe couldn't capture; a kernel rootkit lies to the very tools you're running on the live box, which is half of why rebuilds are the default; the window between compromise and the first shipped log line may simply be missing; and nothing you didn't instrument ever generated evidence at all. Investigation yields a best-effort reconstruction, not certainty, and absence of evidence of deeper compromise is not evidence of absence. Which is why confirmed host compromise ends one way regardless of how tidy the findings look.

Wipe, rebuild, rotate

Here's why cleaning a rooted host in place is the cargo cult this article's title is about: enumerating every possible backdoor on a live system requires trusting tools the attacker may have replaced, on a kernel they may own. Deleting the cron job you found addresses the persistence you found, and says nothing about the persistence you didn't. The only way to know the running system matches your intent is to rebuild it from trusted provisioning onto a fresh image and restore only data from a known-good backup, which is the moment every TX·10 decision either pays off or gets audited. Even the official ransomware guidance frames recovery this carefully, warning responders to "take care not to re-infect clean systems during recovery." The rebuild order matters: fresh image, firewalled shut; data-only restore, never binaries or system state from the old box; integrity check, no unexpected users, keys, cron, or units; monitoring reattached before exposure, so the SOC has eyes from second zero; and the public internet last.

Rotation is the step people skip because it's tedious, so it's a script, not a memory test: new SSH host and user keys with every old authorized_keys gone, the session secret (every cookie dies), all database credentials, the B2 write key, every API key the app held, and the TLS and ACME account material if it lived on the box. The rule is simple: everything the box could read, the attacker could read, so everything the box could read gets rotated.

And the subtle trap: the backup itself can be poisoned. Immutability protects against deletion, not against faithfully preserving an already-compromised state, so if the attacker's dwell time exceeds your retention, every restore point may carry the backdoor. This is why investigation precedes restore: the timeline's estimate of earliest compromise decides the restore point, from before that moment. If no clean point exists within retention, restore selectively, database dumps you can inspect rather than whole filesystems, and rebuild everything executable from source. One configuration detail decides the whole ransomware scenario and deserves its re-check here: the bucket must be in Compliance mode, where not even the account root or the provider's own support can unlock it early. Governance mode only holds if the stolen key lacks the bypass capability, which scoped single-bucket keys do by default, and "only holds if" is not a phrase you want load bearing. TX·10 chose Compliance; this is the article where that choice matters. Last, the closing trap of the whole phase: a clean rebuild of the same vulnerable app behind the same misconfiguration is a subscription, not a recovery. Root cause or you're back.

Decisions to make now, not at 3 a.m.

Three decisions go better decided in advance. Offline versus degraded: a solo hobby service has the luxury an enterprise lacks, you can just turn it off, and the default should be exactly that, a static maintenance page, whenever integrity is in question. Running degraded while possibly rooted risks ongoing theft and makes your box someone else's problem; save degraded mode for pure availability trouble.

Second, the legal clock, summarized as engineering and explicitly not legal advice. If the game has EU users, GDPR applies; there is no hobbyist exemption in the text. Article 33 gives 72 hours from awareness, and awareness means reasonable certainty a breach occurred, not the first alert, so a brief confirmation investigation is permitted; a breach includes destruction and loss, not just disclosure, so ransomware counts; Article 34's duty to tell affected users has a higher bar, likely high risk, with an exemption when the data was properly encrypted; and Article 33(5) requires documenting every breach and your reasoning even when you decide not to report. The preparation step costs five minutes today: write the competent supervisory authority and its reporting URL into the playbook repo, because decision time is not research time. Third, where to ask for help: the FBI's IC3 portal for a formal record (it will not investigate your VPS, and the record matters), your national CERT, and the hosting provider's abuse desk, which is essential the moment your box was used against anyone else and useful for snapshot preservation besides.

Five playbooks and a timer

The artifact that makes any of this real is embarrassingly small: five one-page playbooks, version controlled, one per scenario, every one the same shape so the shape itself becomes muscle memory:

# Playbook: <scenario>
## Trigger      (which alert / log pattern)
## Verify       (two independent sources)
## Contain      (the reversible-first move)
## Preserve     (snapshot + what else)
## Investigate  (which logs answer what)
## Recover      (rebuild? account-only? restore point?)
## Rotate       (which secrets)
## Report       (GDPR? IC3? provider?)
## What this does NOT fix

The five scenarios and their distinctive first moves: web app compromise (snapshot, isolate, host-untrusted on any hint of escalation), account takeover (revoke sessions and rotate the secret, rebuild only on escalation), ransomware or data destruction (speed over evidence, halt, verify the locked bucket, restore from a clean point), defacement or outbound abuse (isolate first, because the priority is to stop hurting others, then snapshot, rebuild, and tell the provider), and a leaked secret (rotate immediately, then investigate what it reached and whether it was used). Exercising them alone is not only possible, it's clarifying: set a thirty-minute timer, pick a scenario, and actually run the read-only steps against the live system, find the log, run the search, locate the snapshot button, find the key you'd revoke. Then trigger benign versions of the alerts, touch a FIM-watched file, fail an SSH login, and confirm the page fires and you can pivot from it. The goal is to find the broken step on a quiet evening instead of during the real thing.

Every incident, real or drilled, ends in the same small review: the timeline from the notes file, the actual root cause, what detection worked and what stayed silent, new hardening and new rules with dates, and what to do faster next time. One rule keeps it useful: blame systems, never people. "The user picked a weak password" is not a root cause; "the service didn't rate-limit or offer MFA" is. That's also the current standard's framing, improvement as a continuous activity rather than a meeting held once and archived, and the findings feed straight back into the TX·05 hardening and TX·06 rules they audited.

The checklist

Response readiness, compressed into what I now keep true:

• Notes template and five one-page playbooks reachable without the VPS: bench rig and phone, provider console access included
• Severity scale written in advance: SEV-1 means host, data, backups, or secrets; SEV-1 means rebuild
• First ten minutes read-only: two independent sources confirm before anything destructive; no reboot on an un-scoped box
• Containment sequence memorized: isolate, snapshot, then decide; active destruction inverts it to halt first
• Account-level response rehearsed: session purge, secret rotation, forced reset, API key revocation, blast-radius readback
• Host-trust criteria written down; inability to rule out root treated as root
• Provider snapshot procedure tested with current permissions; AVML pre-staged and its failure mode on this provider known
• Evidence hashed at capture, analyzed only as copies, each artifact carrying a one-paragraph custody note
• Timeline pivots rehearsed: edge log, off-box journal, keyed ausearch, FIM events, merged in UTC
• Rotation as a script covering everything the box could read; rebuild order ends with monitoring on before exposure
• Restore point chosen from before earliest compromise; bucket confirmed in Compliance mode; selective restore when retention can't prove clean
• GDPR authority, IC3, and provider abuse contacts written into the playbook repo; Article 33 reasoning documented either way
• One timed tabletop per quarter; every incident and drill ends in a blameless review that changes a rule or a hardening line

Honest framing to close, since this page doubles as a portfolio: responding alone on one VPS is not enterprise SOC work, no hand-offs, no SLAs, no tier structure. What it is, is the reasoning those jobs formalize, triage discipline, evidence before action, ATT&CK fluency, root cause over blame, which is also precisely the ground the blue-team certifications on my ladder examine, one of them as a 24-hour hands-on incident. The series arc now reads prevent, detect, survive, respond. One gap keeps surfacing in every drill and every article: the code I ship myself, the npm install that runs strangers' code on my machine, the pipeline that builds the image. That's TX·12, the supply chain. Meanwhile TX·05's auth log is still filling with guesses from the whole internet, and if one of them ever stops being a guess, there's a page for it, a playbook for the page, and a calm version of me on paper telling the scared one what to do next.