Rootkit Detection and Removal

The other day I noticed something weird when I performed the command “ls -l”. Bash was reporting some strange output and the first thing I did was throw it into Google…. which revealed one thing; my Linux box had been comprised. Freaked out at first, I started researching everything I could; I was already aware of tools such as rkhunter and Chkrootkit and this was the first thing I did. RKHunter confirmed what I thought, my server had a rootkit, a SHV5 Rootkit to be correct. The compromised server was used in a web hosting environment and had all the usual services running such as; ftp, http, smtp, dns, pop3, imap etc. After some “intense” googling, I revealed that many sites and forums stated the three R’s to root kit removal;

Repartition, Reformat, Reinstall

And whilst I agree that the above actions are the only 100% way to ensure your system is completely rootkit free, I offer advice on how you can get get up and running without having to rebuild or disappoint your clients *gasp*.

Please take the following advice with caution – we cannot be held responsible for any further damage that you may cause. Further to that, this article is “always updating” – should you have further advice as to assist with the removal of Rootkits – please feel free to chip in and lend a hand.

Detection

The first step is to detect what Rootkit you have and there are two well known tools to do this;

- RKhunter
- Chkrootkit

Cleaning up

The first thing you should do once you KNOW what rootkit you have, is to Google any information you can. What you want to do is find out exactly what the rootkit does and often there are detailed posts or whitepapers which explain this.

In my case, the SHV5 rootkit replaced quite a few system binaries with its own “trojan’ed” version, designed to hide any suspicious activity from the system administrator. An example of commands that were replaced were; ps, ls, top, lsmod, find, netstat etc. (As you can see, these are essential tools to help clean out the rootkit! We need our originals back asap!)

The SHV5 rootkit also set the immutable attribute on core system commands, making it “undeletable”. To check what files the Rootkit may have infected, use the “lsattr” command.


root# lsattr /bin

You may see something like;

s---ia------- /bin/netstat
------------- /bin/zcat
s---ia------- /bin/mv
------------- /bin/date
s---ia------- /bin/cp
------------- /bin/grep
s---ia------- /bin/ls

The files that have the “sia” flags set have most likely been compromised and you are advised to replace them from your package maintainer. The flag “i” represents immutable, meaning it cannot be modified, renamed or deleted.

If you run a system that uses RPM’s, you’ll most likely have to delete the package that contains the above compromised tools, before you can replace them. As this is a dangerous move (deleting crucial commands), I suggest you get everything in place before you proceed.

Simple download the packages you need. In my case I needed the following;

- coreutils-x.i386.rpm
- findutils-x.i386.rpm
- net-tools-x.i386.rpm
- proccps-x.i386.rpm

Once downloaded I performed the following commands;

Because RPM won’t let me uninstall a package that has files with immutable set, I did the following;

chattr -ias /bin/ls
chattr -ias /bin/find

Then;

rpm -e coreutils.i386 --nodeps
rpm -i coreutils-x.i386.rpm

And I did this for all the files that I knew were compromised. Now that I had my everyday system commands back up and running, I could explore further.

Always check the /tmp folder, as this is most likely where the rootkit was first introduced. The /tmp folder is used by many web applications which is often the method of intrusion. In my case I found /tmp/r00t … interesting? I ziped up the folder and stored elsewhere for later viewing. (Oh and I also rm -rf’ed the prick :))

The SHV5 rootkit installs itself under /usr/lib/libsh – and removing this folder is probably a good idea.

Conclusion

Once you’ve got your core system commands back, you can perform an analysis of the situation, digging deeper to see what else has changed. For example;

- netstat -an
Use this command to verify which ports on your machine are open and awaiting information. SHV5 has the ability to run a hidden shell session, listening on a particular port.

- lsof
Cycle through the output this command generates and see what files are being used by the system.

Good luck!

Resources

Some handy sites to assist;

- Linux RootKits For Beginners – From Prevention to Removal
- Analysis of RedHat 8.0 Honeypot Compromise



Leave a Reply

You must be logged in to post a comment.