There are a number of ways to resolve the problem, although they are all temporary until Fortinet comes up with a fix.
- Reboot the firewall
- In Firewall -> Policy -> Protocol Options modify your scan profile(s) and reduce the file Size Threshold down to 2MB from 10MB.
- From the CLI you can run "diag sys top 1" and figure
out which processes are using the most memory (right most column in the process list). You can then restart the
processes using "diag test app
99 ", so for example "diag test app ipsmonitor 99" if the IPS engine is running high.
13 comments:
Two of our fortigates have the same issue (FW 80CM) with this "cmdb add entry failed" dialog.
When this happens, our fortigates are in kernel conserve mode according to the eventlog. After restarting the IPsengine, the memory usage drops and an event with the message "Kernel leaves conserve mode" is written. But not long after this, it will enter kernel conserve mode again.
Also be aware of the following: A few times when I tried to commit a config change in this state the whole entry was truncated. (firewall rules and service groups)
more info about conserve mode over here:
http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD33103&sliceId=1&docTypeID=DT_KCARTICLE_1_1&dialogID=28321239&stateId=0%200%2028319382
Anyway, I've optimized the IPS and firewall as much as possible, but this did not solve the problem for us, however it occurs less often now. (Eventualy memory increases again until it enters conserve mode in a few weeks.)
According to the fortinet sizing tool, the fortigate model is sufficient for the environment in which it's used. While fortinet support tells me to get a bigger box with more RAM.
It seems to me that the sizing tool for this model is not accurate anymore when using the new bloated? firmwares. (4.2+)
Regards,
Jelle Stoel
Yup, good point Jelle. I noticed some odd behaviour after adding addresses/group objects after simply reducing the memory usage.
It appears a reboot is required to actually fix the problem until the next conserve mode incident.
"diag deb cli 8" can give you some more information why this message is shown.
This usually happen when the box enters conserve mode as you pointed out. But from experience you better reboot the box because even if the memory threshold goes back under 70%, the box might be just too unstable to take your command and execute it properly.
This has been a huge issue for me, and as mentioned above, is due to memory utilization. The problem I had was that I don't use ANY of the IPS features, yet the IPS processes were causing huge mem spikes. This has been fixed in MR2 Patch 8 and above.
I've been having similar problems too when editing policies. The FG80 enters conserve mode for a minute or two and after that it's impossible to edit a policy without rebooting first. The error we get is FG_CMDBAPI_ERR.
Our session and bandwidth load are massively below the specs of the FG80 (just as they were when we had a FG60B which also under-performed) but as soon as it flicks in and out of conserve mode we get the cmdb error when trying to save a policy.
Fortinet recently suggested removing the IPS rules that don't apply to our profiles. e.g. just add rules to protect against Windows IIS rather than Apache, and so on, to limit the amount of processing the FG80 needs to do. Hmm, CPU levels aren't our problem, it's the creeping memory consumption that seems to trigger conserve mode which then trips errors when editing policies.
Great help, thanks for sharing this one..
I have noticed alot of issues with deploying 60c's to remote offices regarding memory, and did like this forum since I often found answers here and wanted to help contribute. The UTM options we use are web / App Control / Antivirus.
I did have this error yesterday with an 80CM in our lab, and had to do a few things in order to get it going again, no reboot needed. (restarted IPS Monitor / killed the cmdbsrv and forticron proccess) After this was done I was able to edit firewall policies and add new addresses with out errors.
I typically see that a few process just get maxed out and the processes need to be killed. I also found out that restarting the forticron process seems to get back a bunch of memory, even if its not always using alot and the system is in conserve mode. (diag sys top 1 50 / diag sys kill 11 PID) will kill the processes and they will restart automatically. Some process could be slightly service affecting, so just be careful.
Another few things I noticed were that if your using APP Control, in your sensor use as little applications as you can, and dont overlap. Every time you call for something, and your not using it, its loading those apps in memory. I have seen my ram go up by 10 megs just by adjusting the app control sensor, and removing sensors im not using. Also in the app sensor if your are doing "packet logging" this can take up memory as well, and unneeded space on the analyzer.
#diagnose test application ipsmonitor 99: Same error
#A harder way to restart "ipsengine": diagnose sys kill 9 6085 (6085 is the PID of ipsengine): Same error
#Restarting "forticron": diagnose sys kill 9 63 (63 is the PID of forticron): Used memory decreases from 73% to 51%. Now we can creat new policies again without errors. THANK YOU ALL!!! Case closed :)
What affect does killing the Forticron process have?
Can I run the diagnose sys kill command on this process without affecting traffic?
Additional info: I've got 2 60C's and an 80. The 80 seems to have the worst case.
In addition to killing ipsengine, look for multiple instances of pyfogid and kill these as well.
To get a bit more memory out of the system:
config sys global
set av-failopen idledrop
set av-failopen-session enable
and limit the logging memory buffer
config sys log memory global-setting
set max-size 65536 (or a bit more, like 100,000)
The log memory command will take a reboot to apply.
Does someone know what cmdbsvr proccess is for?
Some background on cmdbsvr (Configuration Management DataBase SerVeR)
1) This process role in FortiOS is to provide I/O to the configuration database, meaning that all configuration activities end up with a call to cmdbsvr.
2) Configuration is held in RAM, and regularly saved on the internal flash disk using compressed files on a per-vdom and per object type basis (system settings, policies, UTM, ...). These files are used to reload configuration on system boot-up
Post a Comment