slow VPS site

I have a VPS site running Virtuzzo with Blacknight. Unlike Alan who also had VPS problems with Blacknight and on a similar spec-ed machine, the server isn't grinding to a halt and the server load rarely goes above 2 or 3. The site is going live this week but the performance is very poor, and no matter what I try I can't seem to improve things much.

First thing I did was to remove all the unnecessary services installed (i.e. the mailserver, spamd) which saw an immediate improvement in the performance of the site. APC is installed, as is Memcache (5 daemons) which is configured with the cache router module. And for good measure I have Boost enabled too. Using APC's reporting tools, I can see I'm getting a 90% (or higher) hit rate on its cache. I can't entirely make sense of the memcache reports however.

I ran benchmarks on the site using ab (ab -c4 -n100). Before installing memcache and boost, I was getting 10 requests per second served (up from 2 with no caching at all). With memcache enabled this went up to 16, and with boost too, this is now 18 for anonymous users. There were improvements each time, but fairly slight.

From looking at the breakdown of the percentages ab gives, the site takes on average 29 seconds to serve 2% of the requests, with the rest all being 1 second or less. I'm presuming that the major decrease is due to caching, but I can't figure out why there is such a high delay for that 2%.

This is more noticeable as a logged in user however, where I reckon at least 30% of the requests take upwards of 15 seconds. Since this site isn't just for anonymous users, it's something I can't very well ignore. Using the devel module I logged the query time. For my test page, a panel page containing about 10 blocks (created via 'add block') and no views and not much else, it has an execution time of 5463ms, with 94 queries taking 1187ms. A good 20 to 30 of the queries are taking between 11ms and 60ms to complete. This makes me think that mysql, or perhaps a rogue module, are to blame.

For example, drupal_lookup_path() took 53ms to do the query: SELECT src FROM url_alias WHERE dst = 'home' AND language IN('en', '') ORDER BY language DESC, pid DESC That's exceptionally high IMO. The same query run at the mysql prompt takes just 0.019ms to run.

I'm really at a loss at where to go from here. I'm thinking mysql might be to blame, but the tuning-primer.sh script turns up nothing of interest, and it is fine at the mysql prompt. My only other thought is that it might be a rogue module, but what? I've disabled all modules not in use, leaving 140 enabled.

Any thoughts, ideas or advice?