There are a number of ways to resolve the problem, although they are all temporary until Fortinet comes up with a fix.
- Reboot the firewall
- In Firewall -> Policy -> Protocol Options modify your scan profile(s) and reduce the file Size Threshold down to 2MB from 10MB.
- From the CLI you can run "diag sys top 1" and figure
out which processes are using the most memory (right most column in the process list). You can then restart the
processes using "diag test app
99 ", so for example "diag test app ipsmonitor 99" if the IPS engine is running high.
7 comments:
Two of our fortigates have the same issue (FW 80CM) with this "cmdb add entry failed" dialog.
When this happens, our fortigates are in kernel conserve mode according to the eventlog. After restarting the IPsengine, the memory usage drops and an event with the message "Kernel leaves conserve mode" is written. But not long after this, it will enter kernel conserve mode again.
Also be aware of the following: A few times when I tried to commit a config change in this state the whole entry was truncated. (firewall rules and service groups)
more info about conserve mode over here:
http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD33103&sliceId=1&docTypeID=DT_KCARTICLE_1_1&dialogID=28321239&stateId=0%200%2028319382
Anyway, I've optimized the IPS and firewall as much as possible, but this did not solve the problem for us, however it occurs less often now. (Eventualy memory increases again until it enters conserve mode in a few weeks.)
According to the fortinet sizing tool, the fortigate model is sufficient for the environment in which it's used. While fortinet support tells me to get a bigger box with more RAM.
It seems to me that the sizing tool for this model is not accurate anymore when using the new bloated? firmwares. (4.2+)
Regards,
Jelle Stoel
Yup, good point Jelle. I noticed some odd behaviour after adding addresses/group objects after simply reducing the memory usage.
It appears a reboot is required to actually fix the problem until the next conserve mode incident.
"diag deb cli 8" can give you some more information why this message is shown.
This usually happen when the box enters conserve mode as you pointed out. But from experience you better reboot the box because even if the memory threshold goes back under 70%, the box might be just too unstable to take your command and execute it properly.
This has been a huge issue for me, and as mentioned above, is due to memory utilization. The problem I had was that I don't use ANY of the IPS features, yet the IPS processes were causing huge mem spikes. This has been fixed in MR2 Patch 8 and above.
I've been having similar problems too when editing policies. The FG80 enters conserve mode for a minute or two and after that it's impossible to edit a policy without rebooting first. The error we get is FG_CMDBAPI_ERR.
Our session and bandwidth load are massively below the specs of the FG80 (just as they were when we had a FG60B which also under-performed) but as soon as it flicks in and out of conserve mode we get the cmdb error when trying to save a policy.
Fortinet recently suggested removing the IPS rules that don't apply to our profiles. e.g. just add rules to protect against Windows IIS rather than Apache, and so on, to limit the amount of processing the FG80 needs to do. Hmm, CPU levels aren't our problem, it's the creeping memory consumption that seems to trigger conserve mode which then trips errors when editing policies.
Great help, thanks for sharing this one..
Post a Comment