Yarosoft

Home / Our work

Case study · Web design and hosting company

400 infected websites, rescued and rebuilt.

A web design company had grown for years, client by client, site by site. Then the sites were infected, and it turned out the business couldn't see its own estate.

Hosting control plane, simplified A control plane holds how every site should be set up. Agents on each web server reach out to it, report what they see and run recorded jobs. Each site runs on a home server with a standby copy on a second server. Backups go to a local tier and an off-site tier. Control plane Inventory · desired state Jobs, audit trail, alerts agents reach out only Web server A Home for some sites Standby for the others Each site walled off Web server B Home for some sites Standby for the others Each site walled off standby sync Local backups (encrypted) Off-site backups (encrypted)
Simplified system diagram, drawn from the published project details. Not a screenshot.
Challenge
Around 400 client websites on one server were defaced, the control panel was hijacked, and there was no inventory.
What we built
A rescue across the whole estate, then a control plane with agents on every web server.
Outcome
Every site known and walled off, a standby for each, backups in two places, and room to sell more.
Scale
Around 400 sites in the rescue; 406 sites under management in the system today.
Status
Live
Challenge

The day it went wrong

Around 400 client websites lived on one server, and most of them had been defaced. The hosting control panel itself had been taken over, and the attackers had installed a hidden service on the server's operating system that kept letting them back in.

Every client was exposed, and the company was the one they'd call. The server couldn't be trusted again. It couldn't be cleaned, only replaced.

Working blind

There was no list of which sites existed, where each one lived or what it ran. The answer to "where is this site?" was worked out on the fly, every time, and years of scripts held the business together.

You can't protect what you can't see. So the first job wasn't fixing. It was finding out.

Rescue

The rescue

  • Contain it. Every site was quarantined, so nothing infected could spread or be served to a visitor.
  • Start clean. A fresh server, built from scratch, instead of trying to clean a compromised one.
  • Clean every site. Our own malware scan went through each site, then WordPress itself was reinstalled from a known-good copy.
  • Keep what was theirs. Each site's own customisations were restored on top, so clients got their sites back as they knew them.

Back online before the cause was known

The company's clients needed their websites back straight away, but how the attackers had got in wasn't known yet. Bringing the sites back meant bringing them back into the line of fire.

So we put watchers in place first: monitoring that traced every attempt to inject code, where it came from and how it worked. Each way in was closed, until the attempts stopped landing and the estate was stable. Then the whole system was hardened.

With the control panel gone, a temporary tool took over its backup and management jobs while the permanent system was built. Then everything was migrated onto it, with every lesson from the attack designed in.

Result

From blind to in control

  • Every site known. A full inventory: what exists, where it runs, what it's built with.
  • Every site walled off. One hacked site can no longer reach the others.
  • A second server for every site, holding a recent copy, ready to take over.
  • Backups in two places, proven by actually restoring sites from each.
  • Anything out of line shows up. The system knows how every server should be set up, and flags anything that drifts.

Then, room to grow

Once the business could see every client and every site, it could sell to the clients it already had. New products and services are offered to the right clients, billing runs itself, and routine work that used to be done by hand is automated. With everything structured, AI can now be plugged in to take on more.

The same system that protects the business is the one that lets it grow.

Inside the system

What it looks like in use

Site inventory with lifecycle, kind, WordPress version and size columns; domains, clients and servers concealed.
Website inventory. Every site the estate knows about, with its lifecycle, whether it is live or staging, its WordPress version and its size.From the live system, with identifying details concealed. Select the image to see it full size.
Compared with the usual way

Off the shelf, or built for the business?

A typical hosting setup: a shared-hosting control panel on each server, with a backup plugin on each site.

AspectA typical hosting setupWhat we built
Knowing what you have Each server knows its own sites. Nothing gives a list of the whole estate. Every site, on every server, in one inventory: where it runs, what it runs, who it belongs to.
One site hacked Sites often share users and resources, so an infection can spread to its neighbours. Each site runs as its own user with its own resources. One hacked site can't reach another.
A server dies Rebuild the server, then restore each site from whatever backup exists. A standby copy of every site is already on a second server. Switch it over and carry on.
Backups Per-site plugins or a server snapshot, rarely tested. Two backup locations, encrypted, with restores proven from each.
If the management tool is breached The control panel runs with full control of the server it manages. The control plane has no way to log in to any server. Breaching it touches no website.
Quiet changes Manual edits pile up unnoticed until something breaks. How every server should be set up is recorded; anything that drifts is flagged and put back.
Selling more Client details, services and billing live in separate tools. Clients, sites and services in one place, so new products are offered to the right clients and billed automatically.
Cost Panel licences that often grow with every account or site, plus a paid plugin per site. No per-site licence fees: the system belongs to the business.
Technical detail

For your IT team: what we actually built

A control plane for a WordPress hosting estate, with agents on each server. The details below are the ones a hosting competitor wouldn't think of.

  • The control plane cannot log in to the serversIt holds no shell access. If it were ever broken into, an attacker still couldn't touch a single website.
  • Agents only reach outServers accept no inbound connections from the control plane; agents pull their work. No inbound firewall rule on any web server.
  • Desired state versus observed stateThe database holds how things should be; agents report how they are. Anything that disagrees is visible drift, and managed settings are re-applied on a schedule.
  • Every change is a recorded jobTyped steps, live logs and an audit trail. Nothing changes on a server outside the job engine.
  • Read-only until adoptedA newly found site is observed and recorded. Nothing is changed until it is deliberately taken under management.
  • Each site isolatedIts own system user and PHP pool, so one compromised site can't read another's files.
  • A home server and a standby for every siteScheduled copies of files and database; failover by repointing DNS; sites can move between servers.
  • Restores never written over a hacked siteA restore merged over a compromised site would quietly keep the attacker's files. Ours never do that.
  • Suspect files never travelFiles are compared between servers by fingerprint only, so a malware scan can't spread the infection to the standby.
  • Backups proven on both tiersEncrypted, deduplicated, local and off-site, with restores verified from each before launch.
  • The rescue, scriptedQuarantine, a malware scan, a clean WordPress core reinstall and a restore of each site's customisations, run across the whole estate rather than by hand.
  • Attack tracing in productionMonitoring that traced injection attempts to their method and source while the sites were live, so each way in was found and patched until the estate was stable.
  • Lessons built inA temporary backup and management tool bridged the gap after the control panel was retired; the permanent control plane was then built around what the attack had exposed.

Tell us what's slowing you down.

Describe how things run today, or what worries you. We'll come back with a plain-English view: what we see, what we'd fix first, and roughly what it would involve. It's free, and yours to keep.

Get a free review