VPS tuning advice

I went off for a run to clear my head, and things have improved.

Netstat -t now shows just a few connections [10-15 total].
Htop shows the load dramatically reduced, load is way down, memory usage is down.

So any ideas what was happening?

Alan

htop doesn't show some details, can you post a screen from top? watch "IO wait" time also. You also seem to be trashed by yahoo crawler - did you try installing modules like Boost to send raw html pages to anonymous users like crawlers?

Also make sure that you have PHP op code cache working. I see that you have APC but "APC support => disabled" and "apc -> enabled" at once.

Best,
Jakub
http://technoergonomics.com
Building communities, performance tuning.

So a closer look at Htop showed that cron was stalled on when running cron.php.
So I killed that...
Load dropped a bit, and an email from notify module hit my email account.
Now this module had not been working for a while on the site.. [ I wasn't too concerned], so the email had notified me of 4000 new nodes...
Maybe that was causing some issues...
I've disabled that module for now.

Btw. how many apache processes do you have now? (ps ax | grep apache2 | wc -l)

Also - take a look in the logs if you are still getting requests from Yahoo, try slowing it down using robots.txt (there should be a parameter for that) - tell it not to crawl faster than X pages / minute.

Given the situation and if the crawler question is right, I would go for Boost immediatelly.

It might be interesting to see what is really wrong given that the server has 8 CPUs, but I guess it's running on some host with a lot of other virtuals.

The first thing I thought of was the Boost module also.

The next thing I thought about was restarting apache & then re-testing your findings to see if there are any differences.

Boris is totally right about Virtuozzo not being a ¨true¨ VPS & recommending ¨hard¨ VPS solutions. Did you check their status page to see if the server you´re on is having issues? I´ve never trusted ¨soft¨ VPS solutions as there´s too many ways that the host can use semantics to, in essence, give you the same services as a traditional shared host @ a higher cost.
http://www.blacknightstatus.com/blog/

Personally, I use www.slicehost.com. They use a Xen instancing & have tons of great features. You might be tied to Blacknight for a while, but if you´d like to switch just email me for a referral code. For my shared hosting needs, I use Dreamhost but have, at times, found it lacking for my Drupal needs.

Hope that helps.

My own thoughts after a nights sleep.
At best these are educated guesses.

Actual cause of problem:
The VPS is struggling with heavy traffic, primarily from Yahoos crawler.
It is not handling the excess load gracefully

Where did this traffic suddenly come from?
As part of the upgrade to D6, I reckon the XML sitemap created a new sitemap.xml, and marked all the nodes [almost 10,000] as updated. So Yahoo decided to crawl the site again.

Solution?
Apache needs to be tweaked to handle the load better. Apache is running the VPS to its limits, and slowing the whole thing down.
Netstat -t shows heaps of connections in the CLOSE_WAIT status.
Is that causing the problem?

Perhaps this analysis is way off :-)

Any more suggestions, especially on how to tweak Apache?

Regards
Alan

Hi all,
Still no solution in site.
The basics of it is that Apache is going mental, chewing CPU and memory.
I've done a full reinstall of Apache & PHP.
I've removed the mysql config adjustments.
I've installed Boost, and it's working [creating and serving pages].
I've reinstalled APC.

I can only imagine I'm missing something obvious and silly,
that an experienced admin wouldn't event think of.

I've tried disabling some unnecessary modules.
But eventually the Apache process just starting using lots of CPU and memory.

Any more suggestions -
My last option is moving to a new VPS, but I'm not sure that is addressing the root cause.

Regards
Alan

The cause of the whole problem is a rogue module [or combination of modules] - I don't know which one(s) yet.

I went back to basics, and asked myself the simple questions.

What did I last change on the server?

Well, I had a site running D6 already on the server, but upgrading athenryac.com to Drupal 6 required a few more contributed modules, than had already on the server.
So I disabled a heap of contrib modules, and hey presto, a happy server.

I don't have much time to work on the server this week, so I've just been re-enabling one module at a time, and leaving the server run to see if all is well.

Unfortunately that means my events calendar [very popular website feature] is currently unavailable :-) ]

Thanks for all the help - when I have a final answer, I'll be back.

Later
Alan

Hi all,
I'm still here :-)

So I found my rogue module - it was/is the calendar module.
From some reason, when that is enabled, apache process start balloning in memory and CPU until the whole VPS is locked.

Now, much of this could be my own doing.
I uploaded the module into a folder within the module itself [rsync error on my behalf].
So I got that sorted, and have clean module code now.

That wasn't enough, so I've now tried a complete re-install of the module [had the view backed up], so see whether that might help.

It seems to run for a while [ a few hours] before going out of control.

So next step is to learn how use use strace to debug a runway process.
Anybody know of any good tutorials/articles on this?

Thanks again for all the help,

Later
Alan

I'm very new to Drupal but am on my second install with Blacknight VPS (Linux/Standard). The initial install was no problem but now after a couple of weeks it went to being really slow again, maybe as we installed more modules, maybe not.

The first install was on a Blacknight VPS with other sites and I figured it was because the other sites were quite large (one was 20 gigs) that the issues arose, so we signed up for another VPS and installed a single site on it, now I'm not so sure that it's not just Blacknight being new to the VPS world and as pointed out above there seems to be very little between the VPS and normal shared hosting.

by the way, when the guys in Blacknight can help they are great, but on some occassions they have found nothing to be wrong. Speeds in general drop more than usual between 3 and 4 in the afternoon.

Just reviving this thread as I'm going through similar problems with a hosting Ireland Xen VPS with multiple Drupal sites getting slow & then dying on a regular basis. So far I haven't much new to add except that one of our sites was dead in the water until I disabled the Weather module.

We do have sites with webform & calendar so I'm wondering were these performance issues with those modules ever fixed in later releases (we would probably have latest or at least very recent releases installed) ?

Thanks!

BTW my take on Hosting Ireland at the moment is that they are trying their best to be helpful sorting this out, but they appear to have only 1 or 2 really experienced server people. Generally their solution is to increase the RAM & reset the VPS. This addresses the symptom for a while but then it dies again. I'm sticking with an Irish host because of SEO - my understanding is that google.ie and other Irish-specific searches consider the domain (.ie) as well as the physical location of the server when deciding what's an Irish site. Someone tell me I'm wrong and I'll move everything elsewhere in a second (we use Idologic in the USA as well, they're really good)