I first noticed some irregular activity when doing some routine reviews of our Cacti monitoring system. In a previous post I described how to monitor firewall statistics such as CPU, session counters and interface traffic with Cacti. One of our low volume web servers all of a sudden spiked to several thousand active sessions which is highly unusual.
Upon further investigation using our FortiAnalyzer I determined that the server was trying to access a particular website at a rapid pace.
Initially it looked like the server was participating in a denial of service attack. I dug a little deeper and manually connected to port 8080 on the destination server and was greeted with the following message:
:irc.dvast8.us NOTICE AUTH :*** Looking up your hostname...irc.dvast8.us (devastate, they are so clever :) seemed a weird place for my system to connect to, further confirming my suspicions about bad things going on. Since I did not yet know why this machine was behaving strangely I installed various Linux rootkit detection systems to see if the machine had actually been penetrated. However all the systems came up blank.
:irc.dvast8.us NOTICE AUTH :*** Found your hostname
:irc.dvast8.us 451 GET :You have not registered
:irc.dvast8.us 451 Host: :You have not registered
:irc.dvast8.us 451 User-Agent: :You have not registered
:irc.dvast8.us 451 Accept: :You have not registered
:irc.dvast8.us 451 Accept-Language: :You have not registered
:irc.dvast8.us 451 Accept-Encoding: :You have not registered
:irc.dvast8.us 451 Accept-Charset: :You have not registered
:irc.dvast8.us 451 Keep-Alive: :You have not registered
:irc.dvast8.us 451 Connection: :You have not registered
As a next step I reviewed my FortiAnalyzer IDP attack logs but nothing there either even though I had cranked up the IDP settings to full force by enabling all signatures and setting them to log and block bad traffic.
Next I turned up Content Archiving to see what my system was accessing and more importantly what was being accessed on my server. By this point I was leaning in the direction that somewhere on the webserver I had a vulnerable script with a buffer overflow or similar exploit.
After sitting back and watching for a while I saw my server accessing an odd text file on a server in China. I attempted to download the text file to a machine specifically configured for forensics and found a nasty little script which opens various backdoors on an infected machine. My PC's anti-virus actually picked it up and quarantined it which means it made its way through the firewall's anti-virus scanners without the Fortinet picking it up.
An additional tool I installed on the server was OSSEC. OSSEC is an open source host intrusion detection system. Installing it was very straightforward and later the same evening I got the first alert via email. The server was attempting to download and save the file from the server in China again but failed due to the limited permissions of the web server process.
I then went back to the logs to see if I could find out what was triggering the download attempts. And sure enough just before my server went active there was a POST operation to a PHP script on the box. I immediately modified my firewall policies to deny all traffic to and from the machine to prevent any further exploit of the script.
I googled the php script that was being accessed and wouldn't you know it the developers had issued a security alert for a code injection vulnerability that I had missed. Obviously the safest thing to do was to entirely remove the application since it was only used for a proof-of-concept and then discontinued. I then re-enabled access for the server at the firewall level.
Follow-Up with Fortinet
To enhance security for other systems protected by Fortinet firewalls I went ahead and submitted the different pieces of the above investigation to Fortinet for dissemination:
- Online Virus Sample Submission at FortiGuard Center - Since the firewall anti-virus scanners did not pick up the malicious script file I sent in a copy
- Report an Exploit/Vulnerability - Since the firewall IDP did not detect the code injection vulnerability in the PHP script I sent the team a URL describing the vulnerability
- Submit a URL for WebFiltering - The FortiGuard WebFilter did not have the site in China rated as hosting malware.
I was very happy to see that within six hours I had received responses to all my requests. The virus is now being detected by a signature in the Fortinet A/V, the IDP signature will be able to block the code injection as soon as Fortinet releases their new signatures and the URL has now been rated as a malware hosting site. The obvious benefit here is that anyone else around the world using FortiGuard services will be able to profit from this experience.
3 comments:
great post
I have a fortigate3600 and 800
I love your blog
Glad you enjoy the blog :)
Hey, it is really usefull ... Best from Bosnia ... Damir
Post a Comment