Containing a Server Compromise and Rebuilding It Clean

A routine maintenance audit uncovered a long-running cryptominer and backdoor on a client's production server. We contained it the same day, preserved the evidence, and moved the application to a rebuilt, hardened environment.

Request Malware Removal
BEFORE Internet open login One small server Production Staging Development hidden miner, running unnoticed AFTER VPN only Production dedicated server Staging + development separate server Encrypted backups, kept off the server
AWS certified Top Rated in DesignRush Clients Say We Deliver on Clutch Top Rated Plus in Upwork Certified by UpCity Top Web Design Company in Techreviewer Best Mobile App Development in Irvine on Expertise.com Top Web & Software Development Company on SelectedFirms
Client

A client running a customer-facing web application

Services

Malware removal, incident response, security audit and infrastructure hardening

Approach

Contain, preserve evidence, then rebuild clean instead of cleaning in place


The Challenge

What looked like a healthy application was running on a server that had been compromised for a long time. The problem was not one infected file. It was an unmonitored, outdated server with several weaknesses around it.

Compromise Challenge Server

An Unmonitored Server

The client's production server sat outside any maintenance plan and was not actively watched. Production, staging and development ran side by side on one small server that was already using swap memory.

Nothing looked broken from the outside. The application kept running, which is how an infection can go unnoticed.

Compromise Challenge Hidden

A Hidden Infection

During an audit for newly approved maintenance work, our DevOps team found a scheduled task running every few minutes under the database service account.

Triage identified a known Linux cryptominer and backdoor family (perfctl), along with a program that resold the server's bandwidth.

Compromise Challenge Stack

An End-of-Life Stack

Every core component was past end of life: the operating system, the language runtime, the database, the cache and the web framework. Several of the web framework's add-on libraries were no longer maintained.

Patching in place would have meant upgrading a server we could no longer trust.

Compromise Challenge Config

Weak Configuration Around It

Internal services listened on every network interface. A production database password sat in plain text in a scheduled task. Backups were stored on the same server, and code was deployed by pulling directly onto production.

Administrator access over SSH was reachable from the internet.


What We Found

Compromise Finding Known

Known

The malware files belonged to the database service account, and that account could open every database without a password.

Indicators included the scheduled task, a payload binary, a working folder holding Tor data, and the bandwidth-resale client. No system-wide rootkit file was present, and no root-level compromise was found.

Compromise Finding Inferred

Inferred

The database was configured to accept password logins from any address. A database superuser can run shell commands as the service account, which is the documented way this malware gets in.

Entry through that open database rule was judged very likely, at a time when the database port was reachable from the internet.

Compromise Finding Unknown

Not Yet Known

Whether the database port was ever reachable through the firewall, and since when.

Whether any data was actually read or taken. Perfctl is built mainly for mining and proxy resale, but because it could have reached the database, user data and every secret on the server were treated as exposed.

Five stages, in order. The priority at every step was to limit damage, keep the evidence and avoid trusting a server we knew was compromised.

On the day of discovery we blocked outbound traffic from the affected account, which cut off the miner, the Tor traffic and the proxy client. We disabled the open database login rule and restricted SSH to the company VPN.

The application stayed online throughout.

We took a provider snapshot of the server as the primary forensic copy, then collected processes, network state, scheduled tasks, services, accounts and logs. The archives were copied off the server and hashed.

The malware was left in place as evidence.

A read-only review inventoried the software stack, services, environment variables, scheduled jobs and credentials, and ranked the findings by risk.

Every secret on the server was treated as exposed and queued for rotation.

We chose to replace the server instead of cleaning it. A new hardened server was built the same evening: SSH through the VPN only, key-only logins, an intrusion-prevention service and automatic updates.

The database moves as a plain SQL dump, reviewed before restore, under new credentials. Nothing else is copied from the old server.

Contain

On the day of discovery we blocked outbound traffic from the affected account, which cut off the miner, the Tor traffic and the proxy client. We disabled the open database login rule and restricted SSH to the company VPN.

The application stayed online throughout.

Preserve

We took a provider snapshot of the server as the primary forensic copy, then collected processes, network state, scheduled tasks, services, accounts and logs. The archives were copied off the server and hashed.

The malware was left in place as evidence.

Scope

A read-only review inventoried the software stack, services, environment variables, scheduled jobs and credentials, and ranked the findings by risk.

Every secret on the server was treated as exposed and queued for rotation.

Build Clean

We chose to replace the server instead of cleaning it. A new hardened server was built the same evening: SSH through the VPN only, key-only logins, an intrusion-prevention service and automatic updates.

Cut Over

The database moves as a plain SQL dump, reviewed before restore, under new credentials. Nothing else is copied from the old server.

Beyond the infection, the audit found configuration weaknesses. Rebuilding on a clean server let us remove each one at the source instead of patching around it. Here is what the new environment fixes by design.

Finding Risk Fix on rebuild
Production, staging and development on one small server
Risk

A development issue or load spike can take production down; test code runs next to production data.

Fix on rebuild

A dedicated production server, with staging and development on a second server.

Internal services listening on every network interface
Risk

Exposed as soon as a firewall rule slips.

Fix on rebuild

Internal services bound to localhost, with a cloud firewall as a second layer.

Production database password in plain text in a scheduled task
Risk

Readable by anyone with access to that account.

Fix on rebuild

Credentials kept in a root-only environment file.

Backups stored on the same server
Risk

Loss or encryption of the server also destroys the backups.

Fix on rebuild

Nightly encrypted backups shipped off-server, plus provider backups.

SSH reachable from the internet
Risk

Exposure to password brute-force attacks.

Fix on rebuild

SSH restricted to the company VPN gateway.

Background workers logging at debug level
Risk

Logs may contain task data and fill the disk.

Fix on rebuild

Info-level logging in production, with log rotation.

Unused runtimes still installed
Risk

Unnecessary attack surface.

Fix on rebuild

Not installed on the new server.

Deployment by pulling code directly onto the server
Risk

No build artifact and no reliable record of what is running.

Fix on rebuild

Artifact-based deployment pipeline.


Lessons Learned

The same four points apply to most servers we review.

Malware Why Guarantee

Keep Every Server That Holds Data on a Maintenance Plan

A server outside regular maintenance went unwatched for years. Any server that holds data should be on a maintenance plan.

Malware Why Report

Databases Listen on Localhost by Default

Remote access should go through a VPN. A database that accepts logins from any address invites exactly this kind of compromise.

Malware Why Code

No Secrets in Scheduled Tasks or Plain-Text Config

Credentials belong in a root-only environment file or a secrets store, never in a crontab or other readable configuration.

Malware Why Root Cause

Maintenance Is How You Catch It Early

Constant monitoring is what regular maintenance is for: security updates, tested backups and someone looking at the logs. A server that is watched shows trouble early, and an infection like this is far less likely to run unnoticed for years.

Talk to us about ongoing maintenance

Services Used

Malware Removal Services Incident Response Security Audit Infrastructure Hardening

Get Your Site Cleaned Today

A malware infection can put your website, customer information, and business operations at risk. Removing the visible symptoms is only the beginning. You also need to identify what caused the compromise and make sure the affected systems are properly secured.

Tell us what's happening with your website or application, and our team will help determine the next steps toward recovery.

Request Malware Removal