Hardening a Linux VPS: what stops the bots and what's just theater

The attacker is a for-loop

Everything in this article is me hardening and testing my own servers, and all of it assumes one specific adversary: the automated kind. Credential stuffers, botnet scanners, mass-exploit scripts that spray a fresh CVE at the whole IPv4 space within days of the patch shipping. The scale is not folklore. A measurement paper from July 2025 (Rieck et al., arXiv 2507.09022) puts the average at roughly 250,000 unauthorized access attempts per internet-facing server per month, and finds only about a third of exposed servers require anything stronger than a password. My little staging box gets its share of that quarter-million like everyone else. It is volume, not interest.

Optimizing for that adversary reorders your priorities. A for-loop doesn't care that you moved SSH to port 2222 or that your cipher list came from a very serious blog post. It cares whether password auth is on, whether you're running a version with a known exploit, and whether you left something listening that you forgot about. So the five controls that carry nearly all the weight are: key-only SSH with root login off, automatic security updates, a default-deny firewall, the shortest possible list of listening services, and your distro's mandatory access control left enforcing. Everything else in this article is refinement on those five. None of it is sufficient against a resourced, targeted adversary, but that's a different program with a different budget, and it isn't who is knocking on a hobby VPS at 3 a.m.

SSH first: keys, then leave the crypto alone

SSH is the most-attacked service on any public box, and the fix is almost boring: eliminate passwords, don't expose root, patch fast. The config file is the same path and syntax on every distro I run, which makes this the most portable section of the whole article. Modern releases read drop-ins from /etc/ssh/sshd_config.d/, so leave the vendor file alone:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
AllowUsers deploy

Order of operations matters here, because this is a change that can lock you out. Generate an Ed25519 key with ssh-keygen -t ed25519, install the public half, confirm a key login works, and only then turn passwords off. Validate before restarting, and keep a second session logged in while you do it:

sshd -t
sshd -T | grep -Ei 'permitrootlogin|passwordauth|pubkey'
systemctl restart sshd    # rc-service sshd restart on Alpine

With passwords off, credential stuffing is over as a category. The bots keep guessing, the guesses stop mattering, and the remaining SSH risks are a leaked private key, a compromised client, or a bug in sshd itself. AllowUsers is the allowlist version of account control, which beats blocklisting the same way default-deny beats default-allow everywhere else. If you need exceptions, a Match block scopes them: I allow password auth from my LAN's address range on one bench machine and nowhere else, and that's the entire list of exceptions I've ever needed.

Now the folklore. Every hardening listicle since 2015 ships a wall of Ciphers, MACs, and KexAlgorithms to paste in, and on a current distro that wall is somewhere between useless and harmful. OpenSSH's compiled-in defaults are already modern: curve25519, a post-quantum hybrid key exchange, ChaCha20-Poly1305 and AES-GCM, encrypt-then-MAC. A pasted list from years ago pins you to the past and quietly excludes better algorithms as they arrive. If a compliance scanner genuinely forces your hand, subtract the offending algorithm with the minus syntax, Ciphers -aes128-cbc, instead of replacing the whole list. Otherwise leave the crypto alone. That's not laziness, it's the engineering call: the defaults are maintained by the people who wrote the protocol, and my pasted list is maintained by nobody.

Changing the SSH port deserves an honest classification: it's log hygiene, not security. Moving off 22 cuts scan noise dramatically, one documented single-server test that moved to port 222 saw a 98 percent drop in attempts over a week, and quieter logs are genuinely worth something when you're the one reading them. But scanners fingerprint services by their banner, not their port, so it stops nothing that decides to look. If you do it for the log noise, fine, I did on one box. Pick a port below 1024 so an unprivileged process can't claim it after a crash, and know exactly what you bought.

The case for treating sshd patching as sacred has a name: regreSSHion, CVE-2024-6387, a pre-auth root RCE from a signal-handler race, affecting versions 8.5p1 through 9.7p1 on glibc systems and fixed in 9.8p1 back in July 2024. No config setting saved you. The workaround of LoginGraceTime 0 traded the RCE for a denial-of-service vector. The fix was the patch, which is exactly why the next section outranks this one.

Patch faster than the botnet

Unpatched known-CVE software is how mass exploitation actually happens, and the regreSSHion timeline is the case study: patches were available within hours, and boxes were still getting picked off weeks later. The same story played out with every WebSocket library CVE I walked through in TX·04. Speed is the control. A human remembering to run upgrades is not speed, so the real question per distro is what does automatic patching look like.

On Debian and Ubuntu it's unattended-upgrades, installed and pointed at the security origin by default once you enable it (dpkg-reconfigure -plow unattended-upgrades). Scope it to security updates and give it a reboot window, because a patched kernel on disk protects nothing while the vulnerable one keeps running:

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:30";

The RHEL family equivalent is dnf-automatic: install it, set security-only mode, enable the install timer:

[commands]
upgrade_type = security
apply_updates = yes
reboot = when-needed

openSUSE schedules zypper patch from a timer. Arch and Alpine have no unattended mechanism at all, and that's a deliberate design choice, not an oversight: on a rolling distro an unattended upgrade can pull a breaking change at 4 a.m. with nobody watching. The ArchWiki states it as an imperative: you must "keep up to date with changes in Arch Linux that require manual intervention" before upgrading. Tools like arch-audit (checks installed packages against the security tracker) and the informant pacman hook (blocks a transaction while news items sit unread) make the manual routine safer, but it stays a routine a human has to run. My call: if you won't commit to reading the news and upgrading on a schedule, don't put Arch or Alpine on a public address. Debian with unattended security updates is not exciting, and not exciting is the feature.

One trap left: kernel, glibc, and OpenSSL updates don't take effect until a reboot or service restart, so "auto-updates enabled" plus "999 days uptime" means you're accumulating patched-but-inactive kernels while the old one keeps serving exploits. Check /var/run/reboot-required on Debian-family, set the reboot window, and if reboots are genuinely scarce, kernel livepatching (Canonical Livepatch, kpatch, SUSE's live patching) covers the kernel piece. Only the kernel piece: glibc still wants its reboot.

Default-deny, whichever front-end you get

Every open port is surface, and a default-deny inbound policy means the service you forgot, or the one an intruder starts, isn't reachable just because it's listening. The part that surprised me when I started keeping cross-distro notes: underneath, everybody is running the same firewall. The kernel enforcement layer is netfilter programmed via nftables on every modern distro. The distros only disagree about which tool writes the rules: ufw on Debian and Ubuntu, firewalld on the RHEL family and openSUSE, awall on Alpine, and raw nft on Arch or anywhere you want to see the actual ruleset. Fighting your distro's blessed front-end buys you nothing; pick the one it ships and move on.

The ufw version is four lines and includes rate limiting, since limit tempbans an address that opens six or more connections in thirty seconds:

ufw default deny incoming
ufw default allow outgoing
ufw limit 22/tcp
ufw allow 80,443/tcp
ufw enable

On firewalld it's the same policy spelled as services and a rich rule for the SSH limit. On my Arch bench box I write the nftables ruleset directly, which is worth doing at least once anyway because it shows you what every front-end is generating. Default-deny input, loopback and established traffic accepted, a per-IP meter on new SSH connections:

flush ruleset
table inet firewall {
  set ssh_meter { type ipv4_addr; size 65535; flags dynamic,timeout; timeout 1m; }
  chain input {
    type filter hook input priority 0; policy drop;
    ct state vmap { established : accept, related : accept, invalid : drop }
    iifname lo accept
    ct state new tcp dport 22 add @ssh_meter { ip saddr limit rate 10/minute } accept
    tcp dport { 80, 443 } accept
    ip protocol icmp icmp type echo-request limit rate 5/second accept
  }
  chain forward { type filter hook forward priority 0; policy drop; }
}

Two principles hiding in there. Rate-limit only ct state new, so established sessions never feel the limiter, and meter per source IP rather than globally, so one noisy scanner can't consume the allowance for everyone. And one warning that saves real pain: don't mix managers. Old iptables scripts still circulate and mostly still work through the compatibility shim, but a hand-rolled script fighting ufw or firewalld over the same chains produces rules that half-apply and failures nobody can reproduce. One writer per firewall.

fail2ban, sshguard, CrowdSec: one real edge

Here's the reframe that took me too long: once SSH is key-only, banning brute-forcers isn't defense, it's janitorial. The guesses were already guaranteed to fail; the ban just keeps the log readable. So the honest pitch for this tool family is log hygiene for SSH, plus real protection for whatever else parses credentials, web logins and mail, and one genuinely distinct capability I'll get to.

fail2ban is the incumbent: log parsing, temporary firewall bans, filters for nearly everything, packaged everywhere (EPEL on the RHEL family). The modern backend rides the systemd journal and bans via nftables:

[DEFAULT]
banaction = nftables-multiport
backend = systemd

[sshd]
enabled = true
maxretry = 4
findtime = 10m
bantime = 1h

On Alpine there's no journal, so backend = auto plus an explicit logpath pointed at syslog's output. sshguard does the same job in a smaller C daemon with a narrower filter library. CrowdSec is the different one: its agent detects locally, a separate bouncer enforces, and the community blocklist means an IP that spent the morning hammering other people's servers can be blocked on yours before its first packet arrives. That's crowd-sourced IP reputation, the same species of signal I was building by hand in TX·02, delivered as a feed. It's also a multi-component stack where fail2ban is one Python process, so the complexity has to earn its keep; on a single box I file it under nice, not necessary.

Know what none of them can do. They're reactive log parsers: a distributed botnet making one attempt per IP never trips a per-IP threshold, a single-shot exploit is done before any threshold counts to two, and a login with valid stolen credentials looks like you. Layer, not foundation.

Fewer listeners, then sandbox the survivors

The service you don't run can't be exploited. Before hardening anything, I run ss -tulpen and read the output like an outsider would: every socket bound to 0.0.0.0 or :: is a public offer. Databases, metrics endpoints, and admin panels that only local processes need should be rebound to 127.0.0.1; services nobody needs should be disabled outright. This ten-minute audit is the highest ratio of risk-removed to effort-spent in the whole piece, the same argument I made about Express middleware surface in TX·01, one layer down the stack.

What survives the audit gets sandboxed. On systemd distros, which is everything here except Alpine, per-service confinement is nearly free: never edit the vendor unit, drop an override in via systemctl edit, and let systemd-analyze security <svc> score you. Fresh packaged services routinely score around 9.6 of a possible terrible 10; a working override lands in the 2-to-5 range:

[Service]
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
SystemCallFilter=@system-service
SystemCallArchitectures=native
RestrictSUIDSGID=yes
LockPersonality=yes
ReadWritePaths=/var/log/nginx /var/cache/nginx

Apply in phases, restart, re-score, watch the journal. ProtectSystem=strict makes the whole filesystem read-only except your declared ReadWritePaths, so the first run usually surfaces a path you forgot as a loud read-only-filesystem error naming the exact directory. That's the directive doing its job. The four that carry most of the score: NoNewPrivileges, ProtectSystem=strict, PrivateTmp, and a tight CapabilityBoundingSet.

Alpine deserves a straight answer here: it runs OpenRC, not systemd, so none of this machinery exists there. No Protect* directives, no syscall filters, no scorer. supervise-daemon gives you ulimits and cgroup caps, and beyond that you're reaching for AppArmor, Bubblewrap, or a container to get comparable isolation. I like Alpine a lot in containers; for a bare public VPS where per-service sandboxing is part of the plan, the systemd distros hand it to you and Alpine makes you build it.

sudo, doas, and accounts nobody uses

With root login off, daily driving happens as an unprivileged user, which makes the escalation tool part of the attack surface. sudo's track record here is humbling: CVE-2021-3156, Baron Samedit, was a heap overflow to full root that sat in the code for nearly a decade before anyone noticed. That's not an argument against sudo so much as a reminder that 177,000 lines of C is a lot of places for a bug to hide. doas, OpenBSD's alternative, is around 3,000 lines, and its entire config on my boxes is one line, permit persist :wheel. Alpine ships it as the default. The port isn't magically bug-free either (the Linux port has carried its own TTY-related CVE), but two orders of magnitude less code is a real difference in how much you're trusting. My split: doas on single-admin boxes, sudo where I actually need its policy granularity, scoped tightly in /etc/sudoers.d/ via visudo and never with blanket NOPASSWD: ALL on anything public.

The rest is account hygiene that takes five minutes and closes real doors: audit /etc/passwd, give system accounts nologin shells, lock anything unused with usermod -L, confirm no account except root carries UID 0, and grant admin through the wheel or sudo group instead of per-user scatter.

The sysctl short list

Kernel network knobs are where hardening guides go to bloat. The syntax and path are identical everywhere, a file in /etc/sysctl.d/ applied with sysctl --system, and the list that earns its place on an internet-facing host is short:

# spoofing and redirect games
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
# not a router
net.ipv4.ip_forward = 0
# info-leak hardening
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
kernel.yama.ptrace_scope = 1

Before crediting your new file with anything, check what was already set: tcp_syncookies and rp_filter are on by default in most modern distros, and Debian ships several of these values out of the box. And know the three that bite back. Strict rp_filter breaks asymmetric routing setups, use loose mode (2) if you're multi-homed. ip_forward = 0 breaks Docker and Podman networking, and the container runtime writes its own sysctl file, so on a container host leave forwarding to it rather than winning a file-ordering fight you didn't know you entered. ptrace_scope = 1 breaks cross-process debugging for non-root, which is correct on a server and infuriating on a dev box. Meanwhile the classics people still paste, like disabling TCP timestamps and ECN, buy nothing against bots and can cost real performance. Skip them.

Leave the MAC on

Mandatory access control is the backstop for the day a service gets popped anyway: the compromised daemon stays confined to what its policy allows even if it's running as root. Which one you have was decided by your distro. The RHEL family and Fedora ship SELinux enforcing; Debian and Ubuntu ship AppArmor; openSUSE, after years as AppArmor's flagship, flipped to SELinux (Tumbleweed in early 2025, Leap 16 at release). Arch and Alpine ship neither by default, which fits their you-assemble-it philosophy and is one more quiet argument for boring distros on public boxes.

The only advice most people need is negative: when something breaks, don't turn it off. setenforce 0 is the most-typed wrong answer in Linux administration, and disabling the LSM to fix one service strips protection from all of them. The right loop on SELinux is to read the actual denial with ausearch -m avc -ts recent, then fix the file context or flip the relevant boolean, reaching for audit2allow only when you understand what you're granting. On AppArmor, put the noisy profile in complain mode, reproduce, and let aa-logprof update it. Every denial has a specific cause, and the tools name it if you ask.

Logs the intruder can't reach

Nothing above prevents everything, so you want to see what happened, and you want the record to survive the person it describes. Step one is just making logs persistent: journald on most distros defaults to losing everything at reboot unless Storage=persistent is set in journald.conf, with SystemMaxUse=2G so it can't eat the disk. Alpine, again the outlier, has no journald: BusyBox syslogd writes /var/log/messages and every journalctl incantation in every guide silently doesn't apply.

Step two, for the things syslog never sees, is auditd: file watches and syscall rules that record someone reading /etc/shadow or exec-ing their way to euid 0. The RHEL family ships it by default for compliance reasons; it's a package away elsewhere. The trap is rule greed. Auditing every execve on a busy server produces volume you will never read and overhead you will definitely pay. Watch the files whose modification means something is already wrong:

-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/sudoers -p wa -k priv
-a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k setuid_exec

Step three is the one that actually changes the game: get a copy off the box. An intruder with root can stop auditd and edit anything local, and none of it helps them if rsyslog or syslog-ng has been forwarding over TLS to a collector that only accepts writes. Now hiding requires compromising a second machine that exposes exactly one inbound port. For one VPS the collector can be nearly anything, a container on my bench rig does it here; what matters is that the copy exists somewhere the first box can't reach back into.

The checklist

The whole article, compressed into the list I ran against my own staging box, roughly in order of risk removed:

• Ed25519 key installed and tested, then PasswordAuthentication no, PermitRootLogin no, AllowUsers allowlist
• sshd -t before every restart; second session open during lockout-capable changes
• Automatic security updates on: unattended-upgrades or dnf-automatic, with a reboot window set
• On Arch/Alpine: a scheduled manual upgrade routine plus arch-audit or equivalent, or a different distro
• Default-deny inbound firewall via the distro's front-end; only served ports open; one manager, never two
• SSH rate-limited at the firewall on ct state new, per source IP
• ss -tulpen audit: every listener public on purpose, local-only services bound to 127.0.0.1
• systemd drop-in sandboxes on internet-facing services, iterated with systemd-analyze security
• Unprivileged admin user; doas or tightly-scoped sudo; unused accounts locked, no stray UID 0
• sysctl short list applied, minus the knobs your workload needs (containers keep ip_forward)
• SELinux or AppArmor confirmed enforcing (getenforce / aa-status) and left that way
• journald persistent (or syslog configured on Alpine); narrow auditd ruleset; logs shipped off-box
• fail2ban/sshguard for log hygiene, CrowdSec if the community blocklist earns its complexity

The thread through all of it: against an adversary that is a script, the wins come from eliminating whole categories, passwords that can't be guessed because there are none, exploits that don't land because the patch beat the scanner, ports that don't answer because nothing listens. Everything here ran on my own bench and my own staging VPS, Debian 13 on the public box, AlmaLinux and Arch and Alpine VMs for the branches, and the auth log is still full of guesses. They just don't matter anymore.