Had a strange problem today that took my entire CentOS LAMP server offline. "top" command showed httpd using almost 100% of my CPU & a massive memory consumption within moments of starting up. "free -m" command would show the httpd process spiral up to consume all the RAM in less than one minute.
Finally isolated the problem to the menu. Used devel rebuild menus to recover site. No amount of cron, cc, rr, reboots, etc would work. It was difficult to even get to this command as the server was almost completely unresponsive. Going to try to isolate the problem, could be related to either the fact that I removed "use tokens" on a number of my "main menu" items that were, in fact, not using tokens, or the addition of several menu links to views pages?
In any case, I wanted to post this here as a reminder that no matter how much time you spend on performance, sometimes the thing that is affecting your site is not apache prefork vs worker, the php.ini memory settings, or anything else that can be controlled.
I was getting a menu related error in my drupal logs that alerted me to the possibility.