Tuesday, March 24, 2009

Internet Malware - Anatomy of a Forensic Investigation

I wanted to take a second and write down some notes on a recent malicious code attack and how Fortinet was able to help stamp out the culprit.

I first noticed some irregular activity when doing some routine reviews of our Cacti monitoring system. In a previous post I described how to monitor firewall statistics such as CPU, session counters and interface traffic with Cacti. One of our low volume web servers all of a sudden spiked to several thousand active sessions which is highly unusual.

Upon further investigation using our FortiAnalyzer I determined that the server was trying to access a particular website at a rapid pace.

Initially it looked like the server was participating in a denial of service attack. I dug a little deeper and manually connected to port 8080 on the destination server and was greeted with the following message:

:irc.dvast8.us NOTICE AUTH :*** Looking up your hostname...
:irc.dvast8.us NOTICE AUTH :*** Found your hostname
:irc.dvast8.us 451 GET :You have not registered
:irc.dvast8.us 451 Host: :You have not registered
:irc.dvast8.us 451 User-Agent: :You have not registered
:irc.dvast8.us 451 Accept: :You have not registered
:irc.dvast8.us 451 Accept-Language: :You have not registered
:irc.dvast8.us 451 Accept-Encoding: :You have not registered
:irc.dvast8.us 451 Accept-Charset: :You have not registered
:irc.dvast8.us 451 Keep-Alive: :You have not registered
:irc.dvast8.us 451 Connection: :You have not registered

irc.dvast8.us (devastate, they are so clever :) seemed a weird place for my system to connect to, further confirming my suspicions about bad things going on. Since I did not yet know why this machine was behaving strangely I installed various Linux rootkit detection systems to see if the machine had actually been penetrated. However all the systems came up blank.

As a next step I reviewed my FortiAnalyzer IDP attack logs but nothing there either even though I had cranked up the IDP settings to full force by enabling all signatures and setting them to log and block bad traffic.

Next I turned up Content Archiving to see what my system was accessing and more importantly what was being accessed on my server. By this point I was leaning in the direction that somewhere on the webserver I had a vulnerable script with a buffer overflow or similar exploit.
After sitting back and watching for a while I saw my server accessing an odd text file on a server in China. I attempted to download the text file to a machine specifically configured for forensics and found a nasty little script which opens various backdoors on an infected machine. My PC's anti-virus actually picked it up and quarantined it which means it made its way through the firewall's anti-virus scanners without the Fortinet picking it up.

An additional tool I installed on the server was OSSEC. OSSEC is an open source host intrusion detection system. Installing it was very straightforward and later the same evening I got the first alert via email. The server was attempting to download and save the file from the server in China again but failed due to the limited permissions of the web server process.
I then went back to the logs to see if I could find out what was triggering the download attempts. And sure enough just before my server went active there was a POST operation to a PHP script on the box. I immediately modified my firewall policies to deny all traffic to and from the machine to prevent any further exploit of the script.

I googled the php script that was being accessed and wouldn't you know it the developers had issued a security alert for a code injection vulnerability that I had missed. Obviously the safest thing to do was to entirely remove the application since it was only used for a proof-of-concept and then discontinued. I then re-enabled access for the server at the firewall level.

Follow-Up with Fortinet

To enhance security for other systems protected by Fortinet firewalls I went ahead and submitted the different pieces of the above investigation to Fortinet for dissemination:

I was very happy to see that within six hours I had received responses to all my requests. The virus is now being detected by a signature in the Fortinet A/V, the IDP signature will be able to block the code injection as soon as Fortinet releases their new signatures and the URL has now been rated as a malware hosting site. The obvious benefit here is that anyone else around the world using FortiGuard services will be able to profit from this experience.


Mutairy said...

great post
I have a fortigate3600 and 800
I love your blog

Sebastian said...

Glad you enjoy the blog :)

Anonymous said...

Hey, it is really usefull ... Best from Bosnia ... Damir