Salvaging a Malware-infected Website

(Heavy sigh.)

I'm afraid this post reflects very badly on me as a system administrator. Server setup and maintenance is a substantial part of what I do at work these days, so I feel some trepidation admitting to what I'm about to admit to. But hell, let's write it anyway.

Set it and forget it 😚

Back in the waning days of the Obama administration I built a website for a small local business as a side project. The client was a mom-and-pop equipment rental place owned by a friend of a family member. The website was nothing fancy — it just needed to look good, be fast and show their product catalog and contact info. It also had a basic CMS where the owners could log in and add, update or remove products. I wrote the thing in Laravel and deployed it to a cheap VPS on DigitalOcean. They were happy with the result, I got paid a satisfactory amount and all was well.

I then proceeded to almost completely ignore this website for a decade. In my defense, your honor, the damn thing kept running just fine! The client never reported any problems or asked for changes, apart from a few tweaks to the contact info early on. Meanwhile I was busy making my 30's happen: working, building a house, starting a family, more work. It did occasionally cross my mind that I should probably check on that website and do some maintenance, but that concern never rose to the top of my to-do list — even though I really knew better at this point.

The situation suddenly changed a few weeks ago.

The Discovery 👨🏻‍💻

I've been doing some reviewing of my online credentials lately — going through 1Password to check for breaches, update credentials or throw them away if no longer needed. One of these was the SSH key for that DigitalOcean droplet, so I figured I might as well check if it still worked.

Oops, turns out I didn't load that key into my SSH agent properly … but that should just give me an error, not a password prompt? Nowadays the first step I take when configuring a web server is to disable password login, but apparently not in 2015. Luckily I still had that password saved too, so let's continue.

Oof, OK. The server is still running Ubuntu 14, which stopped receiving security updates in 2019. I should definitely make some time to upgrade that, and disable password logins while I'm at it.

That System information disabled due to load higher than 1.0 catches me off guard, though. I'm pretty sure this website rarely gets enough traffic to push CPU usage beyond single digits. So what's eating the resources? This is when I begin to feel a little uneasy.

Something is running on this server that uses nearly the whole CPU and a good chunk of RAM. A look at the charts on DigitalOcean confirms that's been the case continuously for at least two weeks. kauditd0 sounds like a Linux kernel thing, but it's running under my deploy user — which is only meant to run the website itself (that's the php5-fpm there, chugging along happily with its 0.3% sliver of CPU time).

Maybe it is an OS thing anyway that got stuck and is spinning its wheels for some reason? I reboot the server but kauditd0 comes right back, gnawing down on those processor cycles. Time to investigate.

(Cracks knuckles.)

Exposing the Crime 🔍

First let's find out where our CPU-eater lives and how it keeps coming back.

kauditd0's process ID leads to an executable file in a directory under my deploy user's home folder. It doesn't require a Torvaldian level of Linux knowledge to find out how it survives a reboot: there are a bunch of entries in deploy's cron config pointing to the same directory.

That directory contains several binaries and one readable shell script. The script base64-decodes a very long string of gibberish, writes the result to a file and runs it through … Perl? There's a language I hadn't heard of in a while. I grab the string from the script, decode it and take a look.

The Sudden Brazilian 🇧🇷

The script is over 900 lines and a lot of it is in Portuguese — Brazilian Portuguese, to be precise.

It's an IRC bot that enables remote control of the system: the attacker can run arbitrary commands, download files, whatever. The only limit on its power is that it runs as the deploy user, which has limited control and can only access certain directories — but that doesn't prevent the script from downloading and running anything it wants to (almost) fully exploit the CPU.

I won't go over the whole implementation in detail, but this is what I could find out about how it works:

• It starts off with some clever tricks to disguise itself and ensure it keeps running: it changes its own process name to edac0 (a standard Linux service), detaches from the terminal by forking itself, and ignores termination sysnals like SIGKILL and SIGTERM.
• The script then connects to an IRC server at an IP address hosted by Private Layer Inc, a Panama-based ISP with servers in Switzerland. How deliciously shady.
• An attacker can then send arbitrary shell commands to the server by logging in to that same IRC server as molly or polly and DM'ing the bot. It responds with the stdout and stderr output of the command, effectively turning a private IRC chat into a remote terminal session.
• The script also contains code to execute DDOS attacks by flooding targets with incoming traffic; and a set of port scanning tools, supposedly to map the network for more potential hosts to infect.

Apparently this kind of IRC bot malware has been around since the mid-2000's in various permutations, and it has its roots in the Brazilian and Romanian hacking scenes. Cool!

What's not cool is that somebody (or some automated thing) has clearly gained illicit control of my server — probably months or years ago — and is relentlessly exploiting its resources. I have a suspicion what for, but let's check to make sure.

Big Cash Money 💰

Inspecting the network traffic reveals a steady stream of activity, but not a lot of meaningful info at first glance. But if the CPU-eating process is what I suspect, it will probably start off with some kind of handshake or login call when it first starts up. Let's use tcpdump to look for signs of that, and then kill the kauditd0 process so we can see what happens as it automatically starts up again:

Aha! This is followed by a series of these, on a steady cadence:

What we're looking at is a handshake with a Monero crypto wallet, followed by a bunch of work assignments from a mining pool to compute hashes. I don't completely understand how this all works, and would honestly prefer to keep it that way (I'm still psychically recovering from the great NFT craze of 2022). But I know enough to know what's going on now: my server's CPU is being used to convert these hashes into fake Internet money.

By the way, this is an astonishingly ineffective way to mine crypto. Serious mining happens on big, expensive GPU's built for massively parallel number crunching — the kind of computational monsters that power AI workloads these days. These guys are stealing cycles from a single virtual CPU on a $10/month VPS that was allocated a small fraction of a real, physical server. Considering the algorithms listed in the login call and the amount of compute power on offer, my server is probably making them about $0.02 a day.

Then again — I don't know who or what is behind this, and how big their operation is. They could have thousands of poorly-protected servers like mine under their control, some of them much more powerful than mine — which could add up to serious money, I guess. I do have the wallet's address from the login call, but Monero is a "privacy coin", which means I can't access its total balance or find out who owns it. Sad.

In this case it's kind of a victimless crime — the software is clever enough not to hog 100% of the resources, and modest applications like my website work just fine on a handful of leftover processing power. The actual victim I guess is Digital Ocean, who has been footing the bill to power all these extra CPU cycles in one of their data centers just to generate some more heat and 2 cents a day for a criminal enterprise.

I do kind of feel bad about that, even if it doesn't amount to anything in the grand scheme of things. Sorry, Mr. and Mrs. Digital Ocean 🥺

Now It's a Rescue Mission 🙎🏻‍♂️

Well that was a fun ride, but now it's time to put a stop to this — and to make amends for a decade of shameful neglect. Let's rescue our website from the hands of those sinister Brazilian crypto miners and take measures so this kind of thing won't happen again.

(Sounds of knuckles cracking)

First step first: taking responsibility. I let the client know about the situation, admitted fault and gave them an estimate for how long it would take to fix. They were very understanding and told me to do what's needed and bill them for my time. Love those guys.

Step two: decide on a plan of action. I briefly consider just taking the website as-is, wrapping it in a Docker container and running that on new, properly secured Ubuntu system. But that would leave the website itself unchanged: an old piece of software running on fragile foundations that would likely still be vulnerable to exploits. But the thought of upgrading the tech stack, from Laravel 5 all the way to Laravel 13, doesn't fill me with joy either. It's likely the framework has changed a lot in ways both overt and subtle, and I don't feel like learning all of that — especially considering how little of the framework this website actually needs.

This thing is simple enough it doesn't need a full-featured web application framework. So I decide to rebuild the whole thing using simple, boring components. I go for plain PHP + a few modern, battle-tested dependencies: Eloquent for the database, Twig for templating and FastRoute for routing. Solid plan.

The rewrite goes surprisingly smooth and enjoyable — in a handful of evenings I have the website rewritten, looking and working as it should, running inside Docker on top of modern foundations. And I take advantage of the rewrite to make a few sorely-needed aesthetic improvements as well.

Nuke it from orbit ☢️

Now the easy part: rebuilding my Digital Ocean VPS and deploying the new version of the site! I log in to Digital Ocean, take one final backup of my compromised droplet just to be sure I don't lose anything, and go through the steps to rebuild my server as a brand new squeaky clean Ubuntu 24 system. As I click "Confirm", somewhere in Brazil a cyber criminal loses one of their income streams.

Time to secure this thing (for real this time).

Shut the front door 🗝️

I won't go over the full list of steps to get a fresh Ubuntu system securely configured; there are plenty of good guides online about how to do basic server security. Let's highlight what I did do this time that I neglected back in 2015:

• Disable password login 🚫 I can't say with 100% certainty but I'm pretty sure this is how my server was compromised: they just brute-forced the password for my deploy user until they got in. Always disable passwords and require strongly encrypted SSH authentication.
• Reject repeated login attempts 👮🏻‍♂️ A tool like fail2ban can detect failed logins and automatically blacklist IP's that are trying to brute force entry into the system.
• Set up monitoring and e-mail alerts for suspicious activity 🔔 I've got a calendar reminder set to log in to the server every few months to do a sanity check - but any other activity outside that (like logins or configuration changes) should be considered suspicious. The logwatch package can flag unexpected activity and send e-mail alerts; so if the crypto cartel ever comes knocking on my server's door again I'll know about it immediately.
• Automatically install security patches ❤️‍🩹 Just in case I get neglectful again — or in case a critical security issue happens to get patched inbetween maintenances, having the system install updates automatically is very handy. I use the unattended-upgrades package, configured to install patches on a weekly basis.

So there you have it. Thanks for joining me through this shameful ordeal all the way to the end; at the very least I feel less guilty and a little wiser.

🧘🏻‍♂️