How a long-lived server backdoor can survive routine security checks

A client runs a malware scan on a server they suspect was compromised. The scan comes back clean. Two weeks later, the same server is sending spam again.

Nothing went wrong with the scanner. It answered the question it was asked. The problem is that it was asked the wrong question.

A scanner answers “is there malware here”, not “does someone still have access”

Signature-based scanning looks for known-bad files. That works well for commodity web shells and mass-deployed cryptominers, which is most of what hits an exposed server.

It works badly for a targeted or patient intruder, for three reasons.

The persistence mechanism is often not a file with a signature. A public key appended to authorized_keys is not malware. It is a correctly formatted line in a legitimate configuration file. No signature will ever match it, because there is nothing to match. The same is true of a systemd unit that runs a normal binary with unusual arguments, or a cron entry that curls a payload at 04:00 and deletes it afterwards.

Signatures lag. A scanner recognizes what has been catalogued. Anything freshly built, or recompiled with a different flag, will not be in the set.

The scanner has no idea what your server is supposed to look like. It cannot know that /usr/local/bin/nginx-helper is not part of your stack, because it has no baseline. A human who knows the system spots that filename in about a second. A scanner does not know it should care.

This is why cleaning malware and closing a compromise are different jobs. The first removes what you found. The second establishes that nobody retained access, which is the question that actually determines whether you get reinfected.

Where persistence actually lives on a Linux host

MITRE catalogues persistence as its own tactic, TA0003, precisely because it is a distinct objective from initial access. In our experience on Linux hosts, most of what we find sits in a handful of places.

Authorized keys

The cheapest persistence available. One line appended to ~/.ssh/authorized_keys for any user with a shell, and the intruder no longer needs the credential they originally used. Rotating the password does nothing.

What people miss: sshd_config can point AuthorizedKeysFile somewhere other than the default, and AuthorizedKeysCommand can generate the list of accepted keys dynamically from a script. If either directive was modified, checking ~/.ssh/authorized_keys for every user still tells you nothing.

Check the effective configuration rather than the file you expect to be authoritative:

# What sshd is actually configured to do, including defaults
sshd -T | grep -Ei 'authorizedkeys|permitrootlogin|passwordauthentication'

# Every authorized_keys file on the box, not just the ones you thought of
find / -xdev -name 'authorized_keys' -type f -exec ls -l {} \; 2>/dev/null

Scheduled execution

cron is the obvious place, and it is checked often enough that a competent intruder frequently uses something else. Look at all of it:

# Per-user crontabs, which are not in /etc/crontab
for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done

# System-wide, including the drop-in directories people forget
cat /etc/crontab; ls -la /etc/cron.{d,daily,hourly,weekly,monthly}/

# systemd timers, which are not cron and are frequently overlooked
systemctl list-timers --all

systemd units and drop-ins

A service unit is a durable, restart-surviving, entirely legitimate-looking way to keep something running. Units load from several directories, and a drop-in file can modify an existing unit without changing the original.

That last point matters more than it first appears. An intruder does not need to create a suspicious new service. They can add /etc/systemd/system/some-real-service.service.d/override.conf containing an ExecStartPre= line. The service you trust now runs their command first, every time it starts, and the unit file you would think to inspect is untouched.

# Units that do not come from a package, listed with their source path
systemctl list-unit-files --state=enabled

# Drop-in overrides, which do not appear in the original unit file
find /etc/systemd/system /run/systemd/system -name '*.conf' -path '*.d/*'

# What a unit actually resolves to after drop-ins are applied
systemctl cat some-real-service.service

Shell initialization and profile files

~/.bashrc, ~/.bash_profile, ~/.profile and /etc/profile.d/*.sh all execute on login. A single line at the bottom of a long file that nobody reads is easy to miss and trivially effective.

Accounts

An added account is loud. A modified one is quiet. Check for accounts that gained a shell, accounts with UID 0 that are not root, and accounts added to sudo groups:

# Any account with UID 0 other than root
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Accounts with a real shell
awk -F: '$7 !~ /(nologin|false)$/ {print $1": "$7}' /etc/passwd

# Recently modified account state
ls -l /etc/passwd /etc/shadow /etc/group /etc/sudoers /etc/sudoers.d/

The application layer

On a WordPress or WooCommerce host, persistence often lives above the operating system entirely: an administrator account created through the application, a must-use plugin in wp-content/mu-plugins that loads automatically and never appears in the plugins list, a modified theme function, or a scheduled event in the application’s own cron table. None of that is an operating system artifact, and a host-level scan will not surface it.

Timeline reconstruction is what tells you the scope

The single most useful thing you can do early is establish when it started, because that determines which backups are trustworthy. Restoring from a backup taken after the compromise reinstalls it.

Log timestamps are evidence, but treat them as evidence rather than fact. An intruder with root can edit logs. Corroboration across sources that would have to be tampered with separately is worth far more than any single source.

Useful anchors, roughly in order of how often they help:

• Package manager history. /var/log/dpkg.log or /var/log/yum.log records installs with timestamps, and is edited less often than auth logs.
• File modification times across configuration directories. Cluster the changes and look at what happened in the same window. find /etc -xdev -newermt '2026-08-01' -type f is blunt and frequently effective.
• Authentication logs. /var/log/auth.log or /var/log/secure. Look for successful authentication from an unexpected source, not just failures. Everyone has failures.
• Web server access logs. For an application-layer entry point, the request that first succeeded is usually visible, often a POST to an upload or plugin path.
• Shell history. Frequently cleared, occasionally forgotten. Costs nothing to check.
• Anything shipped off the host. Logs already sent to a remote collector are the only ones the intruder could not edit after the fact. This is the argument for remote logging in one sentence.

Egress tells you what a file listing cannot

Persistence has to talk to something eventually. A backdoor that never makes an outbound connection and never accepts an inbound one is not useful to anyone.

Most servers have a small and predictable set of outbound destinations: package repositories, an SMTP relay, a payment gateway, an API or two. Anything outside that set deserves an explanation.

# Listening sockets with the owning process
ss -tulpanH

# Established outbound connections
ss -tpn state established

# Processes whose executable has been deleted from disk, a strong signal
ls -l /proc/*/exe 2>/dev/null | grep -i deleted

That last check is worth running on any host you are unsure about. A running process whose binary has been unlinked is doing something that ordinary software does not do.

The catch: if the host is genuinely compromised at root level, these commands are running on the system you are investigating, and their output cannot be fully trusted. A rootkit can hide a process from ss and ps. This is why comparing what the host reports against what your firewall or network layer observed is more reliable than either alone.

Deciding between cleaning and rebuilding

This is the decision people put off, and delay is expensive.

NIST’s incident handling guidance frames eradication and recovery around restoring systems to a known-good state. On a single-tenant Linux server, the honest reading of “known-good” is usually a rebuild.

Rebuild when any of these are true:

• The intruder had root, or you cannot establish that they did not.
• You cannot date the initial compromise, so no backup is provably clean.
• Persistence was found in more than one distinct mechanism, which indicates deliberate effort.
• System binaries or kernel modules were modified.
• The host holds credentials that reach other systems, and rebuilding is cheaper than proving containment.

Cleaning in place is defensible when:

• The compromise was clearly confined to the application layer, for example a web shell in an uploads directory on a host where the web user cannot escalate.
• You have a reliable timeline and it is short.
• You have file integrity data, or a package manager that can verify the installed files against their expected checksums.

The pattern we see most often is a middle path chosen for the wrong reason: nobody wants the downtime of a rebuild, so the host gets cleaned, and three weeks later it is reinfected. The second incident costs more than the rebuild would have, because now the timeline is longer and the trust question is harder.

Whichever path you take, rotate everything the host could see. Not just the compromised account: SSH keys, database credentials, API tokens, application secrets, mail credentials and anything in an environment file. Credentials on a compromised host should be assumed read.

What actually prevents the repeat

Preventing reinfection is mostly about making the next compromise visible rather than making it impossible.

• Send logs off the host. Local logs on a compromised machine are evidence the intruder controls. Remote logs are the ones that still mean something afterwards.
• Baseline the things that should not change. Knowing what authorized_keys, the enabled unit list and the package set looked like last month turns a two-day investigation into a diff.
• Alert on the shape of the change, not just on known-bad. A new enabled service, a new UID 0 account, a new outbound destination and a new authorized key are all low-frequency events on a stable server. They are far better alerting signals than a signature feed.
• Reduce what a single compromise reaches. Separate credentials per host, least-privilege database users and no shared keys across environments all mean one compromised host stays one compromised host.
• Verify a restore before you need it. A backup that has never been restored is a hypothesis.

None of this is exotic. It is the unglamorous work that turns a compromise into an incident with a known end date instead of an open question you revisit every few weeks.

The short version

A clean malware scan tells you no known-bad files were found. It does not tell you that nobody has a key, a timer, a drop-in override or an application-level account. Those are different questions, they are answered by different work, and skipping the second one is the reason servers get cleaned twice.