Running out of tricks on how to optimize my Drupal site

FCGI is where your problem is. It can't share cache across processes. I would suggest going to mod_php and then try APC for opcode and Drupal's object cache. You'll probably see a big improvement. I run mod_php + APC on one client's site with almost the exact same specs as your server. The module load is a little lighter, but not much, plus Drupal is running 5 sites on it. Our average response time for logged in users is about 500ms. That's with over 50,000 nodes, 40,000 users and 2 million+ comments. We even had one period where we got over 60,000 page views in one hour and the server load never went above 3.

400ms sounds pretty good. Nice work.

apc.stat does give a boost, but you have to remember to clear the cache or restart php every time you update a module. The day you forget is the day some change in a module will start giving your visitors a white screen of death. Been there.

The fact that you're using apache2 worker and php through fcgi makes me question whether you're actually getting the benefit of apc at all. Unfortunately, I'm not an expert on configuration issues of apache2 worker, since I personally use nginx+php-fpm. Since you're using php 5.2.xx I'm going to assume that you're not using php-fpm.

Probably most of us assumed incorrectly that you use apache-prefork with mod_php which works beautifully with apc. But if you're using apache-worker, like it sounds that you are, you don't get the benefit of one apc cache being shared among all processes. Apache-worker has all sorts of other advantages.

I'd like other folks to chime in on this, but I can direct you to two good articles. One is by the highly respected Drupal house 2bits: http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better... where Khalid concludes: "Although we have not dug too deep into it, we have found is that APC is not shared among the processes. And because the php-cgi processes are killed and new ones spawned, the persistence of APC's user/data cache cannot be relied upon. This is not important for most users, but if you are using things like the Drupal cache router module or the performance logging module, this will impact you."

And that brings up another point. You list cacherouter among your modules, but you don't say you're using memcached. So you could possibly be using the APC user cache, but most folks prefer memcached for cache use.

So we switched from FCGI to mod_php to better exploit APC. Now my biggest problem is apache runs as 'nobody' while all my scriptsand directories are owned by the account I login with via SSH. With Apache as nobody drupal can't create directories and i had to change permissions to 777 which I really don't like.

Do you have any workaround for this 'nobody' user issue? Im afraid I'll have to ask HG to go back to FCGI, observe the performance and then decide.

Thank you for all your help so far. I learned a lot from you guys!