Tuesday, July 13, 2010

High CPU Utilization caused by IPS Engine

Over the past few weeks I have been seeing quite a number of CPU spikes for various types of firewalls ranging from FG60B to 310B to 800. In every instance the "ipsengine" process was consuming all available CPU resources on the firewall. After consulting with Fortinet there appears to be an issue related to the current IPS Engine. Ask your SE and they may be able to provide you with a pre-release version of IPS Engine 1.165.

In the meantime if you run into this problem you can run the following command to restart the IPS Engine:

# diag test app ipsmonitor 99

14 comments:

Sebastian said...

Mark,

depends on your model and how much traffic you are funneling through there. I would recommend formatting the flash memory and reloading the OS via tftp as described in Fortinet's upgrade notes.

mark said...

its the FortiWiFi-80CM. not sure how I would relate traffic use. the best example might be active session count. my 80CM shows that it was in conserve mode yesterday from 0800-1400. during that time active sessions peaked at 1663 and the average was 976. the minimum was 401. the specs on the 80CM state 100k concurrent sessions with 5k per second. i dont know how to find my sessions per second data. i am just using SNMP to record active sessions. thank you for your note btw. formatting and reloading an OS is never a bad suggestion.

johannes said...

Should be fixed with the latest rls of FortiOS4.
It's a confirmed bug and fortinet was working on it.
regards,
johannes

David said...

I have found that on systems running A-A HA it is helpful to run the command on all nodes in the cluster. Otherwise you are apparently only restarting the IPS engine on the cluster master.

Anonymous said...

Just a thank you as I had this issue after a DoS attack and it seemed to work. Although the next day the A-P cluster failed to the P node for no apparent reason. I rebooted the original A node and everything is back to normal.

David said...

If you are still having this issue contact support. There is a new IPS engine that fixes the bug. I have been running the new engine for a week now and no issues, yet.

M00sebyte said...

Disable or Renable IPSengine
-----------------------------------
# diag test application ipsmonitor
IPS Engine Test Usage:
1: Display IPS engine information
2: Toggle IPS engine enable/disable status
3: Display restart log
4: Clear restart log
5: Toggle bypass status
6: Submit attack characteristics now
97: Start all IPS engines
98: Stop all IPS engines
99: Restart all IPS engines and monitor

Sachin said...

1> The system has activated session fail mode

2> The system exited system conserve mode

System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.

Reason for High CPU utilization can also be because of -->

The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.

you can see this out put by entering CLI command "diag sys top" and in output you see below result:

sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).

Sachin B W said...

1> The system has activated session fail mode

2> The system exited system conserve mode


System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.

Memory/local disk can be freed by entering following command

"execute log delete-all"



Reason for High CPU utilization can also be because of -->

The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.

you can see this out put by entering CLI command "diag sys top" and in output you see below result:

sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).

Irfan Pathan said...

hello,
whenever i enable log & report > log settings > upload log remotely or disk then immediately CPU goes above 95%.
v4 MR3 patch 10.

Anonymous said...

Thanks

Minesh Medatwal said...
This comment has been removed by the author.
Rama Krishna Pinnamaneni said...

1> The system has activated session fail mode

2> The system exited system conserve mode

System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.

Reason for High CPU utilization can also be because of -->

The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.

you can see this out put by entering CLI command "diag sys top" and in output you see below result:

sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).

Rama Krishna Pinnamaneni said...

We have updated fortigate 800C appliance from 5.0.9 GA patch to 5.0.11 and
5.0.11 to 5.2.3 in offline mode.

After connecting the appliance to network we were unable to connect to the internet, we have outbound policies configured and default route to reach to internet.
Kindly suggest the solution and does 5.2.3 is the stable version.