Hacked
Ok, so life in the hosting business can truly suck sometimes.
On a daily basis, we all have to fend off about a billion passive attempts from worms and scripts, all trying to exploit our computers. People sitting at home on their Windows XP Home Ed computers sure cop a lot of it and the IIS servers of the world take the brunt but even those of us who chose a ’safer’ option in the open source world have an endless and, eventually, losing, battle here.
When you’re a web host, it’s a million times harder. I have to not only open up services that I want or need but I have to open up ports and protocols at the whims of people who suffer badly from the “I want this but have no idea what I’m asking for” syndrome. It’s so often that someone will want some stupid 3rd party app enabled that needs to run as root and opens every port under the sun and hasn’t been patched since 1992… then you find out they never even used it.
When you try to explain to them that perhaps, running PHP out of safe mode and enabling global variables might be a bad thing they just complain “but some crappy thing I downloaded from spammers-r-us needs it!”. I know, you’re going to say “don’t let them”… well, it’s never that easy when they’re they paying customer and I’m the guy who has to play rhoshambo with the miriad of worms out there. For once, I just wish people would get this… But they don’t get it and they don’t care. And if it costs me several days to clean up the mess… they still don’t care.
Anyhow, there’s a point to this… yes… I got a r00kit installed. Thanks fuckers. r0nin rootkit, to be precise, which came on in through a vulnerability in phpBB2, installed on a site by a client who, of course, wasn’t even using it. They just installed it and then decided, nah don’t need it. Did they patch it? Keep it up to date? No, they weren’t using it. Was it available to the world (worms too?) – hell yes!
Now, coz I *really* want to help the next poor bastard this happens to, I’m going to go through a little detail about how I found it, and what I can do about it.
I needed to change a setting on Apache, so i went into the conf file and made my change. Then I went to restart the service. Ok (yes, it’s RedHat): service httpd restart…..stoppping (OK)… restarting (FAILED). Something is already on port 443.
Crap. That’s not good. So i figured that maybe a PID had locked or the like. Let’s have a quick look:
root@halima]# netstat -lpn |grep ‘0.0.0.0:443′
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 20893/r0nin
-Fuck-
Nothing good has a mixture of alphanumerics when it comes to executables. And I sure as hell didn’t put it there. Let’s kill it with a kill 20893. Does nothing, it restarts in a new PID. Over and over again. Damn it…
Next up, updatedb + locate r0ninas well as find / -name r0nin turn nothing up, so they’re hiding it a little better than not at all.
Trolling the logs turns nothing up so it’s time to try something new… what’s hidding in /tmp? Nothing… what about /var/tmp? newbash. Owned by user Apache and world executable. Bingo. Take a copy to look at later, delete it and kill the process… it stops and Apache resumes conrol of potr 443.
Ok, the immediate problem’s passed but we’re not out of the woods yet. Hell, I’m not even sure what wood I’m in. But i have something nice – a file date/time stamp that’s actually believable (a week back). So let’s troll the logs for things that happened around then…
You’d never believe it but I actually found something at the same damn time, in a weblog for a client’s site, running, of course, phpBB2. A file called ‘viewtopic.php‘ was hit at the same time, from an IP at internode (Adelaide/Australia based ISP).
Google viewtopic.php and see what comes up.. Guess what? r0nin rootkit of course! Turns out this is a known exploit that’s able to exploit this file in phpBB and has been around since 2004. A frikkin year! But the person who uploaded phpBB uploaded an old version they had sitting on their hard disk and never even thought that maybe, just maybe, they should check to see if there’s a newer version around?!
This is what I mean about hosting. I take responsibility for my machine and I know some of this rests with me but imagine if all the people out there who didn’t know / care about security had the ability to write to your computer – and you just have to let them. At the end of the day, there’s only so much you can do to protect yourself and only so much you can know about.
The good news is that because the server is configured securly, the process installed wasn’t able to achieve any of it’s payload and the bash script they ran wasn’t able to be exploited. Ordinarily, r0nin tries to deface websites, etc but it couldn’t do that in this case.
So now I’m off to do what has to be done with a hacked server. Yes, it’s all working and the attack vector has been silenced but it’s still time to burn it to the ground and start again. You just have to.
So until next time, cya later.
-=Rodney=-
—————————–
Some related reading:
Superb Blog by someone else in my shoes.
)/
on November 29th, 2005 at 6:16 am
Pain felt… I too ran across r0nin, though it was in the midst of an entirely different procedure (good thing I scrolled through the output of ‘ps’, eh?) and killed it dead. As near as I can tell, nothing else of badness has transpired, but I’m going to have a long talk with one of the users I provide hosting for… argh.
on November 29th, 2005 at 8:02 pm
It’s the most annoying thing in the world, isn’t it? I was lucky like you and the payload of the rootkit was never fully executed. I was doubly lucky in that the server in question was due for retirement at the end of this year anyway and the replacement was already there to go. Still, did you have a look at my post on mod_security for apache?
Old news to many but if you’re not playing with it – you should. The URL for the post is http://www.ausweblogs.com/users/rodneyfletcher/?p=6 (or the one about hacking the web server). It’s really nice to know that anything that tries to fly ludicrous comments like “wget’ etc through a URL is going to fail.