Over the past few weeks I have been seeing quite a number of CPU spikes for various types of firewalls ranging from FG60B to 310B to 800. In every instance the "ipsengine" process was consuming all available CPU resources on the firewall. After consulting with Fortinet there appears to be an issue related to the current IPS Engine. Ask your SE and they may be able to provide you with a pre-release version of IPS Engine 1.165.
In the meantime if you run into this problem you can run the following command to restart the IPS Engine:
# diag test app ipsmonitor 99
Mark,
ReplyDeletedepends on your model and how much traffic you are funneling through there. I would recommend formatting the flash memory and reloading the OS via tftp as described in Fortinet's upgrade notes.
its the FortiWiFi-80CM. not sure how I would relate traffic use. the best example might be active session count. my 80CM shows that it was in conserve mode yesterday from 0800-1400. during that time active sessions peaked at 1663 and the average was 976. the minimum was 401. the specs on the 80CM state 100k concurrent sessions with 5k per second. i dont know how to find my sessions per second data. i am just using SNMP to record active sessions. thank you for your note btw. formatting and reloading an OS is never a bad suggestion.
ReplyDeleteShould be fixed with the latest rls of FortiOS4.
ReplyDeleteIt's a confirmed bug and fortinet was working on it.
regards,
johannes
I have found that on systems running A-A HA it is helpful to run the command on all nodes in the cluster. Otherwise you are apparently only restarting the IPS engine on the cluster master.
ReplyDeleteJust a thank you as I had this issue after a DoS attack and it seemed to work. Although the next day the A-P cluster failed to the P node for no apparent reason. I rebooted the original A node and everything is back to normal.
ReplyDeleteIf you are still having this issue contact support. There is a new IPS engine that fixes the bug. I have been running the new engine for a week now and no issues, yet.
ReplyDeleteDisable or Renable IPSengine
ReplyDelete-----------------------------------
# diag test application ipsmonitor
IPS Engine Test Usage:
1: Display IPS engine information
2: Toggle IPS engine enable/disable status
3: Display restart log
4: Clear restart log
5: Toggle bypass status
6: Submit attack characteristics now
97: Start all IPS engines
98: Stop all IPS engines
99: Restart all IPS engines and monitor
1> The system has activated session fail mode
ReplyDelete2> The system exited system conserve mode
System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.
Reason for High CPU utilization can also be because of -->
The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.
you can see this out put by entering CLI command "diag sys top" and in output you see below result:
sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).
1> The system has activated session fail mode
ReplyDelete2> The system exited system conserve mode
System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.
Memory/local disk can be freed by entering following command
"execute log delete-all"
Reason for High CPU utilization can also be because of -->
The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.
you can see this out put by entering CLI command "diag sys top" and in output you see below result:
sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).
hello,
ReplyDeletewhenever i enable log & report > log settings > upload log remotely or disk then immediately CPU goes above 95%.
v4 MR3 patch 10.
Thanks
ReplyDeleteThis comment has been removed by the author.
ReplyDelete1> The system has activated session fail mode
ReplyDelete2> The system exited system conserve mode
System entering into conserve mode is mainly because when memory is full (memory and local disk that is SDHC). Here your Fortigate AV will go into fail open mode when it can not scan the live network traffic. As memory is full traffic cannot be cached into the memory/local disk so traffic flows without being monitored by AV.
Reason for High CPU utilization can also be because of -->
The summary reports daemon ( sumreportsd ) is responsible for computing data for drill down widgets configured in the dashboard.
you can see this out put by entering CLI command "diag sys top" and in output you see below result:
sumreportsd (name of process),394(proc ID),R (Running state),99.9 (CPU used),0.4(system memory used).
We have updated fortigate 800C appliance from 5.0.9 GA patch to 5.0.11 and
ReplyDelete5.0.11 to 5.2.3 in offline mode.
After connecting the appliance to network we were unable to connect to the internet, we have outbound policies configured and default route to reach to internet.
Kindly suggest the solution and does 5.2.3 is the stable version.