VPS tuning advice

alanburke's picture

Hi all,
I'm running a zen Based VPS at blacknight with some, relatively speaking, low traffic sites.

I upgraded one of those sites to Drupal 6 this morning, and now the server seems to have gone haywire.

I'd appreciate any advice as to how to recover the situation.

One of the sites is www.athenryac.com, and judging from the browser, a connection is being made to the server,
and it is is either extremely slow, or just times out.
Even ssh access is very slow.

The server is running Ubuntu 8.4.01
Php 5
with APC installed.
Mysql 5.

Phpinfo, and my.cnf pasted below.

I've noted two things which others might be able to explain for me.

  1. Running htop - it show that on the server, many of the CPUs running at 100%.
    Many Apache processes seem to be running at 100%.
    Screenshot attached.

  2. Running netstat -t shows a whole heap of connections coming from what looks like a yahoo crawler.
    But again, perhaps this is normal...
    Screenshot also attached

The server was running fine up till today, and as I learn my way with server administration,
I didn't make any huge changes.
I've been slowly changing MySQL conf based on the tuning script at
I have left Apache config as default.

[I should point out, I tried changing a lot of things in apache config (increasing max clients, decresing timeout values etc) to recover the situation, but as it did nothing useful to help I went back to square one].

Something somewhere is wrong, So any help appreciated.

Kind regards


PHP Info
from php -i

MySQL info
Information in a extra.cnf, within the /etc/mysql/conf.d directory

character-set-server = utf8
default-collation = utf8_unicode_ci
table_cache = 512
max_heap_table_size = 48M
tmp_table_size = 392M
query_cache_limit = 6M
query_cache_size = 64M

Screenshot-alanburke@vps-1016239-785.cp_.blacknight.com: ~-projects.png135.84 KB
Screenshot-alanburke@vps-1016239-785.cp_.blacknight.com: ~-projects-netstat.png240.32 KB
php.txt21.54 KB



alanburke's picture

I went off for a run to clear my head, and things have improved.

Netstat -t now shows just a few connections [10-15 total].
Htop shows the load dramatically reduced, load is way down, memory usage is down.

So any ideas what was happening?


False Positive

alanburke's picture

We're back at square one...
Loads of connections, Htop shows heavy load...

htop doesn't show some

meba's picture

htop doesn't show some details, can you post a screen from top? watch "IO wait" time also. You also seem to be trashed by yahoo crawler - did you try installing modules like Boost to send raw html pages to anonymous users like crawlers?

Also make sure that you have PHP op code cache working. I see that you have APC but "APC support => disabled" and "apc -> enabled" at once.

Building communities, performance tuning.


alanburke's picture

APC does appear to be working.
Even though the output of php -i does say its disabled,
the apc.php output suggests it is.
You can see it at
http://www.cycletraffic.com/apc.php ...
Well you could if the server wasn't dying...

Should I try to change some APC config options, and restart apache?


Notify Module..

alanburke's picture

So a closer look at Htop showed that cron was stalled on when running cron.php.
So I killed that...
Load dropped a bit, and an email from notify module hit my email account.
Now this module had not been working for a while on the site.. [ I wasn't too concerned], so the email had notified me of 4000 new nodes...
Maybe that was causing some issues...
I've disabled that module for now.

Top screenshot

alanburke's picture

Hi Jakub, thanks for the assistance.

Haven't gone as far as Boost yet.
Not sure why Yahoo is coming to visit all of a sudden, perhaps due to xmlsitemap module submitting an xmlsitemap.

Top screenshot attached.


Btw. how many apache

meba's picture

Btw. how many apache processes do you have now? (ps ax | grep apache2 | wc -l)

Also - take a look in the logs if you are still getting requests from Yahoo, try slowing it down using robots.txt (there should be a parameter for that) - tell it not to crawl faster than X pages / minute.

Given the situation and if the crawler question is right, I would go for Boost immediatelly.

It might be interesting to see what is really wrong given that the server has 8 CPUs, but I guess it's running on some host with a lot of other virtuals.

ps ax | grep apache2 | wc -l

alanburke's picture

Returns 12 at the moment for the number of Apache processes.

Drupals standard robots.txt has crawl delay at 10, so I've just doubled that to 20.

I guess there are a number of other VPSs on the server.
I understood that VPS are independent and that one VPS maxed out should have no effect on the others.

Should I ask the hosting company if they spot anything unusual at server level?

I have munin on the server, but couldn't get it to output Apache stats.
You can see the output at
if the server responds...



joshk's picture

I understood that VPS are independent and that one VPS maxed out should have no effect on the others.

That's not precisely true. Depending on the configuration, you may or may not have guaranteed memory or CPU resources.

Depending on who you host with, your physical box may or may not be oversubscribed.

From your stats, it seems apparent that the spikes in load coincided with spikes in slow queries and total mysql queries. I'll read through the rest of the thread and see what else I can add.

http://www.chapterthree.com | http://www.outlandishjosh.com

Could be other sites on your Virtuozzo based VPS

Boris Mann's picture

You say you're hosting at Blacknight (http://blacknight.com). I looked, and their VPS's are Virtuozzo, not Xen-based. Virtuozzo uses "soft" VPS -- essentially, your VPS is as bad as shared hosting - the way that Virtuozzo works, anything on the same hardware can nuke the entire server.

The only VPS I can recommend is "hard" VPS, like Xen-based ones.

Thanks Boris, My apologies -

alanburke's picture

Thanks Boris,
My apologies - I guess I assumed it was Zen based.

I did log a ticket with the hosting company, but they don't see anything unusual on the lower level.
Their analysis was that Apache was chewing up all the CPU and memory.

I'll have to find a way to tweak the Apache config.


this is not exactly true.

Alexander Ufimtsev's picture

this is not exactly true. while virtuozzo (and openvz that is is based on) can take advantage of spare cpu cycles and memory, they can be configured to have a guaranteed minimum that is always available.

some thoughts

Miguel-gdo's picture

The first thing I thought of was the Boost module also.

The next thing I thought about was restarting apache & then re-testing your findings to see if there are any differences.

Boris is totally right about Virtuozzo not being a ¨true¨ VPS & recommending ¨hard¨ VPS solutions. Did you check their status page to see if the server you´re on is having issues? I´ve never trusted ¨soft¨ VPS solutions as there´s too many ways that the host can use semantics to, in essence, give you the same services as a traditional shared host @ a higher cost.

Personally, I use www.slicehost.com. They use a Xen instancing & have tons of great features. You might be tied to Blacknight for a while, but if you´d like to switch just email me for a referral code. For my shared hosting needs, I use Dreamhost but have, at times, found it lacking for my Drupal needs.

Hope that helps.

Thanks Miguel I actually use

alanburke's picture

Thanks Miguel
I actually use the slicehost article repository as a knowledge source.
I'd prefer to use a host on this side of the pond, to reduce latency [though that seems laughable now :-) ].
The irony is that I have switched from a zen based VPS to Blacknight.

It does seem the problem is with my own VPS config, so for now, I'll have to get that sorted.



alanburke's picture

My own thoughts after a nights sleep.
At best these are educated guesses.

Actual cause of problem:
The VPS is struggling with heavy traffic, primarily from Yahoos crawler.
It is not handling the excess load gracefully

Where did this traffic suddenly come from?
As part of the upgrade to D6, I reckon the XML sitemap created a new sitemap.xml, and marked all the nodes [almost 10,000] as updated. So Yahoo decided to crawl the site again.

Apache needs to be tweaked to handle the load better. Apache is running the VPS to its limits, and slowing the whole thing down.
Netstat -t shows heaps of connections in the CLOSE_WAIT status.
Is that causing the problem?

Perhaps this analysis is way off :-)

Any more suggestions, especially on how to tweak Apache?



kbahey's picture

The Yahoo crawlers can't be that aggressive. Check how many ESTABLISHED connections you have.

To narrow down if this is an Apache bug or indeed genuine traffic install apachetop, and run it like this:

aptitude install apachetop
apachetop -T 300 -d 5 /var/log/apache2/access-whatever.log

Wait for 5 minutes and see if the number of accesses per seconds is more than a reasonable number (2-3 for a small site, 80 or higher for a high traffic one). It will also tell you if certain URLs are being hit repeatedly. We had a similar problem at a client with a strange combination of Microsoft WebDAV MiniRedir and Drupal's singlesignon = aggressive crawler.

But my hunch is that it may be something else, since you are not seeing a high MySQL load to go with the symptom of Apache pegging the CPUs at 100%.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Thanks Khalid. Its looking

alanburke's picture

Thanks Khalid.

Its looking like an Apache error alright

Apachetop is showing very little
last hit: 00:00:00 atop runtime: 0 days, 00:00:00 15:58:33
All: 0 reqs ( 0.0/sec) 0.0B ( 0.0B/sec) 0.0B/req
2xx: 0 ( 0.0%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%)
R ( 1s): 0 reqs ( 0.0/sec) 0.0B ( 0.0B/sec) 0.0B/req
2xx: 0 ( 0.0%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%)

I've pasted in some of the error log.
[Mon Apr 20 16:15:13 2009] [error] [client] File does not exist: /home/alanburke/projects/athenryac/favicon.ico
[Mon Apr 20 16:15:16 2009] [error] [client] File does not exist: /home/alanburke/projects/athenryac/favicon.ico
[Mon Apr 20 16:16:23 2009] [error] [client] File does not exist: /home/alanburke/projects/athenryac/favicon.ico
[Mon Apr 20 16:42:28 2009] [notice] child pid 11956 exit signal Segmentation fault (11)
[Mon Apr 20 16:48:43 2009] [notice] caught SIGWINCH, shutting down gracefully
[Mon Apr 20 16:51:21 2009] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch configured -- resuming normal operations
[Mon Apr 20 16:52:18 2009] [error] [client] Attempt to serve directory: /home/alanburke/projects/athenryac/files/files/, referer: http://www.athenryac.com/news/race_result?page=7
[Mon Apr 20 16:54:10 2009] [error] [client] File does not exist: /home/alanburke/projects/athenryac/favicon.ico
[Mon Apr 20 16:54:53 2009] [error] [client] File does not exist: /home/alanburke/projects/athenryac/favicon.ico
[Mon Apr 20 16:55:57 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting

I've just noticed the segfaults, so I've disabled APC for now...and added a favicon...
I'll see how that goes....


Apachetop after 5 mins

alanburke's picture

Still the same...
last hit: 00:00:00 atop runtime: 0 days, 00:14:01 16:50:12
All: 0 reqs ( 0.0/sec) 0.0B ( 0.0B/sec) 0.0B/req
2xx: 0 ( 0.0%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%)
R (300s): 0 reqs ( 0.0/sec) 0.0B ( 0.0B/sec) 0.0B/req
2xx: 0 ( 0.0%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%)

404s, segfautls and APC

joshk's picture

One thing to note: the 404 result for drupal is actually a somewhat expensive page to generate. If you see a lot of 404s coming through, this is actually more performance-intensive than repeated requests for a real page, because many caching advantages are unavailable.

If apache was segfaulting repeatedly, that could also have been a big problem. If you have to turn off APC to solve this, it's a net win, but APC is (generally) pretty stable, so it should work. Maybe check your version there, etc. You definitely want to get that working again.

http://www.chapterthree.com | http://www.outlandishjosh.com

Thanks Josh!

alanburke's picture

Thanks for all the feedback.

I haven't fully solved the problem [but I have learned a LOT!]

Yahoo isn't coming to visit as much and clogging up the connections so the site is functional again.

I do know that the APC version is a little behind, so I'll upgrade that.
APC is is switched back on again and seems to be behaving for now.

I hadn't planned on using Boost just yet, but most if the traffic to this site is anonymous, so I will probably look at it soon enough.
I'll put in a static 404 pretty soon anyway.


Access Log

alanburke's picture

This is whats showing in the access log...

Is this normal? - - [20/Apr/2009:17:00:43 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch (internal dummy connection)" - - [20/Apr/2009:17:00:43 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch (internal dummy connection)" - - [20/Apr/2009:17:00:43 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch (internal dummy connection)" - - [20/Apr/2009:17:01:36 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch (internal dummy connection)" - - [20/Apr/2009:17:02:14 +0100] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch (internal dummy connection)"

Dummy connections are

meba's picture

Dummy connections are normal.

If you have reached MaxClients and you still have enough memory, try allowing more apache processes in the config.

But still - most of the instances are computing something - either there is a bug in your website or you have just too slow VPS.

Either way - try thinking from the management point of view. You are spending A LOT of time trying to optimize this. If you just install Boost, you will probably be fine...


alanburke's picture

Hi Jakub,
Still not too sure what the root cause is.

Apache was chewing up all the CPU and memory, so increasing maxclients isn't an option I don't think.

From a management point of view, this is a learning experience!
As you might have guessed, I'm a novice server admin.

Boost is on my list of things to try out.

For now APC is disabled. It did seem to make a difference.
But Yahoo is no longer clogging up the connections.
I don't know whether thats related or a coincidence.

Thanks for the help!


Max Clients being hit very quickly

alanburke's picture

From the error log

[Mon Apr 20 17:00:55 2009] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.8-0.dotdeb.1 with Suhosin-Patch configured -- resuming normal operations
[Mon Apr 20 17:02:48 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting

One week on.

alanburke's picture

Hi all,
Still no solution in site.
The basics of it is that Apache is going mental, chewing CPU and memory.
I've done a full reinstall of Apache & PHP.
I've removed the mysql config adjustments.
I've installed Boost, and it's working [creating and serving pages].
I've reinstalled APC.

I can only imagine I'm missing something obvious and silly,
that an experienced admin wouldn't event think of.

I've tried disabling some unnecessary modules.
But eventually the Apache process just starting using lots of CPU and memory.

Any more suggestions -
My last option is moving to a new VPS, but I'm not sure that is addressing the root cause.


Focus on apache

kbahey's picture

Forget Drupal, boost et. al, forget MySQL ...etc.

The problem is that Apache is looping and not because of traffic.

Try disabling all Apache modules. Do this via a2dismod. Disable them all, then bring them in one by one and see what happens.

If that does not solve it, use strace on a looping Apache process and see if you can glean anything from the system calls it is issuing (if it is doing that).

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.


alanburke's picture

Hi Khalid
I've cut apache2 modules down to
alias authz_host dir mime php5 rewrite
I can't get apache2 to start and server Drupal without these modules.

Apache still starts out small and baloons pretty quickly.

I won't pretend to understand strace, but this is what a calls to one apache processes returned

alanburke@vps-1016239-785:~$ sudo strace -p 24210
Process 24210 attached - interrupt to quit

Doesn't tell me much I'm afraid, [something else for learning :-) ]


I give up - Paid Help ?

alanburke's picture

Hi all,
I've reached the end of my meagre abilities on this, and am admitting defeat on this.

Is there anyone willing to take on the job of examining the server and trying to sort it? I'll provide root access.

I'm willing to pay you for a few hours work, [or make a donation to the Drupal association to the same value] to see whether it can be sorted.

Failing that, I'll just move to a new VPS and hope for the best.
alanburke at fastmail.fm

It makes more sense, I

christefano's picture

It makes more sense, I think, to invest your time and money in moving to another host. Without hesitation, I recommend Slicehost and Rimuhosting. Both use Xen. Rimu costs slightly more but has fantastic support and they can help you further with these issues (if they occur again) for about US$40 an hour.


Boris Mann's picture

You've spent untold hours troubleshooting and have pretty much blown away any investment in your host. Getting a new server for even a month and seeing if it fixes it is probably a good investment. For ~$40, you can rule out or confirm that it is your host. And, really, your host should be helping with this -- I know I've gotten this kind of (free) help from hosts in the past. I still blame Virtuozzo :P

I second both recommendations. And also +1 for Rimu support, which isn't technically more expensive depending on your needs -- I usually get the 400MB or 900MB VPS size for sites, both of which are "cheaper" than the slice sizes.

VPS Latency

alanburke's picture

Hi Boris, christefano
Just to clarify, I don't feel that my time has been wasted.
Even though I didn't fix the problem [yet :-) ], the time spent troubleshooting has been a great experience.
The sites on this server are my hobby sites which I use to to try out new stuff, and see how it works on a real website.
The VPS is used to see how tuning techniques work on a live server, and not just on my laptop.

Learning on the job isn't really an option for me. I've got to use trusted solutions I can understand. That's why this VPS and sites are for.

I really appreciate everyones help - I've been reading all threads similar to this, and picking up useful advice the whole time.

Back to the issue...
I'm reluctant to move to Slicehost as they only have servers in the US, but I seems rimu have servers in London, so that's an option.


vps xen hosting in Ireland

nyciweb's picture

Hostingireland offer xen hosting packages that appear to be based in Ireland. Has anyone any reports of hosting with these?

(I'm currently with Blacknight, using plesk/virtuozo)



alanburke's picture

Hi Daniel
Digiweb do ZEN vps too, but no Ubuntu... yet .
Just Centos and Debian[close enough I guess].


I've moved my VPS with

David Henry's picture

I've moved my VPS with blacknight to their shared hosting and it's speeded up considerable, still not hyper but at least it's not 15 second loads anymore.

very new to this world


alanburke's picture

The cause of the whole problem is a rogue module [or combination of modules] - I don't know which one(s) yet.

I went back to basics, and asked myself the simple questions.

What did I last change on the server?

Well, I had a site running D6 already on the server, but upgrading athenryac.com to Drupal 6 required a few more contributed modules, than had already on the server.
So I disabled a heap of contrib modules, and hey presto, a happy server.

I don't have much time to work on the server this week, so I've just been re-enabling one module at a time, and leaving the server run to see if all is well.

Unfortunately that means my events calendar [very popular website feature] is currently unavailable :-) ]

Thanks for all the help - when I have a final answer, I'll be back.


I have had very similar problems with a OpenVZ VPS

rfay's picture

I have had enormous trouble with my OpenVz-hosted VPS since I started upgrading more and more of the sites to Drupal6.

I believe that at least one part of the picture is that other VPSs on the same physical box were competing for resources.

At this time, I've screamed loud enough that the hosting service (Spry) has moved my VPS onto a very lightly loaded box, and since then I've been fine.

The key indicator with my problems was high IOWAIT every time, which caused mysql to go tremendously slowly, thus getting all the processes behind on all their work.

I remain very interested in whatever you learn on this. But I agree that moving the VPS to a site with non-Virtuzzo hosting is the key.


Another Update

alanburke's picture

Hi all,
I'm still here :-)

So I found my rogue module - it was/is the calendar module.
From some reason, when that is enabled, apache process start balloning in memory and CPU until the whole VPS is locked.

Now, much of this could be my own doing.
I uploaded the module into a folder within the module itself [rsync error on my behalf].
So I got that sorted, and have clean module code now.

That wasn't enough, so I've now tried a complete re-install of the module [had the view backed up], so see whether that might help.

It seems to run for a while [ a few hours] before going out of control.

So next step is to learn how use use strace to debug a runway process.
Anybody know of any good tutorials/articles on this?

Thanks again for all the help,


You could try to do some xdebug profiling

stewsnooze's picture

See http://www.sitepoint.com/blogs/2007/04/23/faster-php-apps-profile-your-c...

That article covers windows cache grinding, MacCallGrind on Mac, or KCacheGrind.
You can collect the stats on your linux platform and just bring back the files locally to "grind" them.

Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.


David Henry's picture

I'm very new to Drupal but am on my second install with Blacknight VPS (Linux/Standard). The initial install was no problem but now after a couple of weeks it went to being really slow again, maybe as we installed more modules, maybe not.

The first install was on a Blacknight VPS with other sites and I figured it was because the other sites were quite large (one was 20 gigs) that the issues arose, so we signed up for another VPS and installed a single site on it, now I'm not so sure that it's not just Blacknight being new to the VPS world and as pointed out above there seems to be very little between the VPS and normal shared hosting.

by the way, when the guys in Blacknight can help they are great, but on some occassions they have found nothing to be wrong. Speeds in general drop more than usual between 3 and 4 in the afternoon.

very new to this world

Rogue Modules or Rogue Custom Code

colincalnan's picture

Recently came across a problem with a VPS load peaking at about 60 and crashing the httpd process bringing the site to it's knees.

We tried everything to fix it but couldn't figure out where the problem was coming from.

Cache was enabled, as well as JS and CSS compression but the problem persisted. We initially also suspected Cron so we made that much less frequent, still no joy.

The problem actually turned out to be a simple join database query on webform submission data from the Webform module. It was joining a table for over 200k records with a table about about 15k and running this in a block on every page. Webform, due to the ability to create arbitrary form field stores each piece of data for each field as a single row in the webform_submissions_data table. Once we disabled the block, the load dropped back to under 0.75 and everything was running smoothly. We also have a page which output the results of webform underneath it using similar date and promptly turned that off also. We're in the process of caching that data and turning the blocks back on.

Hope this helps anyone with similar issues.

are webform & calendar still a problem?

frost's picture

Just reviving this thread as I'm going through similar problems with a hosting Ireland Xen VPS with multiple Drupal sites getting slow & then dying on a regular basis. So far I haven't much new to add except that one of our sites was dead in the water until I disabled the Weather module.

We do have sites with webform & calendar so I'm wondering were these performance issues with those modules ever fixed in later releases (we would probably have latest or at least very recent releases installed) ?


BTW my take on Hosting Ireland at the moment is that they are trying their best to be helpful sorting this out, but they appear to have only 1 or 2 really experienced server people. Generally their solution is to increase the RAM & reset the VPS. This addresses the symptom for a while but then it dies again. I'm sticking with an Irish host because of SEO - my understanding is that google.ie and other Irish-specific searches consider the domain (.ie) as well as the physical location of the server when deciding what's an Irish site. Someone tell me I'm wrong and I'll move everything elsewhere in a second (we use Idologic in the USA as well, they're really good)

Troodon's picture

I'm sticking with an Irish host because of SEO - my understanding is that google.ie and other Irish-specific searches consider the domain (.ie) as well as the physical location of the server when deciding what's an Irish site.

I can only speak from limited personal experience, however the geolocation of the IP address you're on is probably not a major factor, after all this is the era of the Cloud where your actual files might be hosted anywhere. As far as Google is concerned they probably consider the top level domain, but IP location is probably far less of a factor that your specified target audience in Google Webmaster Tools. For illustration I moved a site from a shared host in Ireland to a UK based Linode with no noticeable change in ranking at all.

Not that I wont head back to Ireland as soon as I can get a decent Xen based VPS here.

Weather Module

mikeytown2's picture


We experienced this same issue