A production Linux server hardening checklist that will not break your application

There are two ways to fail at server hardening.

The first is not doing it, which leaves a machine running with whatever the provider’s default image shipped: services nobody uses, password authentication enabled, and logging that records almost nothing you would want after an incident.

The second is running a hardening script that applies a hundred changes from a generic benchmark in one pass. Something breaks, nobody can tell which of the hundred did it, the whole thing gets reverted, and hardening is written off as disruptive. The server ends up less secure than a careful subset would have made it, and now there is organizational resistance to trying again.

This checklist is ordered to avoid the second failure. Each group is small enough to verify before moving on.

Step zero: make sure you can get back in

Before touching SSH configuration, confirm you have an independent way onto the machine.

Nearly every provider offers a serial or VNC console. Open it, log in, and leave that window open for the rest of the work. This takes two minutes and is the difference between a mistake and an incident.

# Also worth having: a second SSH session already open and authenticated.
# If you break sshd, the existing session survives and you can fix it.

Locking yourself out of a production server while securing it is a genuinely common outcome. It is also entirely avoidable.

Establish the baseline before you change anything

You cannot tell what is unusual without knowing what is normal. Capture the current state:

# What is listening and what owns it
ss -tulpanH > /root/baseline-sockets.txt

# What is installed
dpkg -l > /root/baseline-packages.txt        # or: rpm -qa

# What runs at boot
systemctl list-unit-files --state=enabled > /root/baseline-units.txt

# Who can log in
awk -F: '$7 !~ /(nologin|false)$/ {print $1": "$7}' /etc/passwd > /root/baseline-shells.txt

# Who has sudo
grep -rE '^[^#]' /etc/sudoers /etc/sudoers.d/ > /root/baseline-sudo.txt

Keep these somewhere off the server. They are the reference for every future “is this supposed to be here” question, and after an incident they are the difference between a diff and an investigation.

Group one: authentication

The highest-value changes, and the ones that need the console open.

Key-based SSH, passwords off

# Confirm your key works BEFORE disabling passwords
ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no you@server 'echo ok'

Only once that prints ok:

# /etc/ssh/sshd_config.d/10-hardening.conf
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
PermitEmptyPasswords no
MaxAuthTries 3

Then validate the configuration before reloading, which is the step that saves you:

sshd -t && systemctl reload ssh
sshd -T | grep -Ei 'passwordauthentication|permitrootlogin|pubkeyauthentication'

sshd -T prints the effective configuration including defaults, not just what is in your file. Read that, not the file you edited.

Audit existing keys

Every authorized_keys file is a standing grant of access. People leave, laptops get replaced, and nobody ever removes the key.

find / -xdev -name 'authorized_keys' -type f -exec sh -c \
  'echo "== $1"; cat "$1"' _ {} \; 2>/dev/null

Look at the comment field on each key. If you cannot name the person and confirm they still need access, remove it.

Also check whether sshd_config redirects where keys are read from, because if it does, the file you just audited may not be the one being used:

sshd -T | grep -Ei 'authorizedkeysfile|authorizedkeyscommand'

Named accounts, not shared logins

If four people share one login, your authentication log cannot tell you who did anything, and revoking access for one person means changing it for everyone. Give people their own accounts with their own keys and sudo where needed.

Verify before continuing: log out completely, log back in with a key, and run a sudo command.

Group two: network exposure

Most compromises begin with something reachable that did not need to be.

Inventory what is listening, and justify each one

ss -tulpanH | awk '{print $1, $5, $7}' | sort -u

For every line, answer: what is this, does it need to be reachable, and by whom? Anything you cannot justify gets bound to localhost or removed.

The frequent finding here is a database listening on 0.0.0.0. On a provider’s shared internal network that is meaningfully exposed even with no public firewall rule.

# /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address = 127.0.0.1

# /etc/redis/redis.conf
bind 127.0.0.1
protected-mode yes

Default-deny firewall

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp comment 'ssh'
ufw allow 80,443/tcp comment 'web'
ufw enable
ufw status numbered

If your administrative interfaces can be restricted by source address, do it. It is a much stronger control than any amount of rate limiting.

ufw allow from 203.0.113.0/24 to any port 22 proto tcp comment 'office'

Lock the origin if you use a CDN

If a CDN or WAF sits in front of your site, and your origin still accepts connections from anywhere, the edge protection is optional from an attacker’s point of view. Restrict port 80 and 443 to your edge provider’s published ranges.

This one change defeats the most common way that edge protection is bypassed.

Verify before continuing: load the site, log in over SSH from an allowed address, and confirm the application still reaches its database.

Group three: reduce what exists

Configuration can be wrong. Software that is not installed cannot be exploited.

# What is installed that nothing depends on
apt list --installed 2>/dev/null | wc -l
deborphan 2>/dev/null

# Common unnecessary services on a web server
systemctl list-unit-files --state=enabled | grep -Ei 'avahi|cups|rpcbind|nfs|bluetooth'

Compilers, development headers and debugging tools on a production web server are convenient for you and equally convenient for anyone who gets a shell.

Removing is more durable than configuring around. It also cannot drift back.

Group four: permissions and secrets

Find world-readable secrets

# Config files readable by everyone
find /var/www /etc -type f \( -name '*.env' -o -name 'wp-config.php' -o -name '*.conf' \) \
  -perm -o=r -ls 2>/dev/null

wp-config.php containing database credentials, readable by every user on the box, is extremely common on shared setups.

chown root:www-data /var/www/example.com/wp-config.php
chmod 640 /var/www/example.com/wp-config.php

Stop code execution in upload directories

This turns a file upload vulnerability into a nothing instead of into remote code execution.

location ~* ^/wp-content/uploads/.*\.(php|phtml|php[0-9])$ {
    deny all;
    return 403;
}

Look for the surprising things

# Files anyone can write
find / -xdev -type f -perm -o=w -not -path '/proc/*' -ls 2>/dev/null | head -50

# setuid binaries, which is a short list on a normal server
find / -xdev -perm -4000 -type f -ls 2>/dev/null

The setuid list is worth reading properly. Anything unexpected on it deserves an explanation.

Service isolation with systemd

If your services run under systemd, a few directives cost nothing and meaningfully constrain a compromised process:

# /etc/systemd/system/myapp.service.d/hardening.conf
[Service]
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
ProtectKernelTunables=true
RestrictSUIDSGID=true

Apply these one at a time and restart the service after each. ProtectSystem=strict in particular will break anything that writes outside its declared paths, which is the point, and which is also why you add ReadWritePaths deliberately rather than guessing.

Group five: visibility

Hardening without logging means that after an incident you have a reduced attack surface and no idea what happened.

Record the events that matter

Authentication, privilege escalation, and changes to accounts. On most distributions the defaults capture authentication; privilege escalation and account changes often need auditd:

apt install auditd
auditctl -w /etc/passwd -p wa -k identity
auditctl -w /etc/shadow -p wa -k identity
auditctl -w /etc/sudoers -p wa -k privilege
auditctl -w /root/.ssh/ -p wa -k ssh_keys

Send logs off the host

This is the single most important logging decision and the one most often skipped.

Logs held only on the affected machine are logs an intruder with root can edit. Remote logs are the only ones that still mean anything afterwards. It does not have to be elaborate: forwarding syslog to a second machine you control is enough to change what is possible during an investigation.

Brute force mitigation

apt install fail2ban
systemctl enable --now fail2ban
fail2ban-client status sshd

Add your own addresses to the ignore list. The first serious failure mode of any auto-blocking tool is locking out the administrator during an incident.

Group six: patching and recovery

Everything above is undone by a server that is not updated.

# Unattended security updates, with a record of what was applied
apt install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

Automatic updates are good. Automatic updates with nothing checking that services came back up afterwards are a gamble. Pair them with a health check.

Also: applying kernel updates without rebooting means running vulnerable code while the patch log looks healthy.

# Does this machine need a reboot to activate what has been installed
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required

Prove the backup

A backup that has never been restored is a hypothesis. Restore one to a throwaway machine, confirm the data is complete and the application starts, and record how long it took.

That last number is the one nobody has and everybody wants during an incident.

What a script cannot do for you

Automated hardening tools and benchmark scripts are useful starting points and poor finishing points, for three reasons.

They do not know your workload. A script cannot know that port 8080 serves an internal API your mobile app depends on, or that a service needs to write to a path outside its own directory.

They cannot tell you what they decided not to do. The most valuable output of a hardening engagement is often the list of controls deliberately not applied, with the reasoning. A script produces no such list.

They apply everything at once. Which is precisely the failure mode that gets hardening reverted.

Use the benchmarks as a source of ideas. Apply them in groups. Verify after each. Write down what you skipped and why.

The order, condensed

1. Confirm console access, and keep a second session open
2. Capture the baseline before changing anything
3. Authentication: keys on, passwords off, audit existing keys, named accounts
4. Exposure: inventory listeners, default-deny firewall, bind internal services to localhost, lock the origin to your edge provider
5. Reduce: remove packages and services nothing needs
6. Permissions: secrets not world-readable, no execution in upload directories, systemd sandboxing
7. Visibility: audit rules, remote log shipping, brute force mitigation
8. Sustain: patching cadence, reboot when required, a backup you have actually restored

Verify after each group. Write down what you skipped. The list of deliberate exceptions is part of the deliverable, not an admission of incomplete work.