One of my sites runs behind two Varnish dedicated servers with something like 12 web servers running Pressflow 6 and four MySQL full InnoDB database servers. This more than handles the load the site runs on.....until.....I put it into maintenance mode.
At this point Varnish stops caching the pages as they aren't sending HTTP 200.
The load average on the Apache/PHP boxes shoots very quickly to above 100 and in general the boxes are looking for cliffs to jump off.
I'm wondering what others do about this.
I think these are some options.
1 Cache the maintenance mode return code (503) in Varnish
1.1 I'm worried about caching 503s during normal site operation
1.2 I guess I could cache them for something very small like 2 seconds.
2 Select a completely separate backend which is purely static
2.1 I don't really want to have to load new vcl
3 Have Varnish probe the backends and remove them should they return 503 and then serve the vcl_error page to users.
3.1 I'd be fine to do this if the site didn't have regional variations like .co.uk, .fr, .de, .com as we only take one down at a time and they have separate databases.
4 Fix maintenance mode performance?
5.....?
Any opinions on these or shared experiences on how you do it now?
Comments
What about changing the
What about changing the webserver (e.g. Apache) configuration.
That's how drupal.org managed the downtime during the redesign launch. I'm not sure if it was related to this problem or not.
knaddison blog | Morris Animal Foundation
Similarly, I have seen sites
Similarly, I have seen sites that deploy an index.html when they want to go into maintenence. They configure Apache to serve that file before index.php when available. To go out of maintenance, you remove or rename your index.html.
Obviously this technique only works for clean url sites but surely any site that uses varnish can figure out how to use clean urls.
Varnish and custom VCL
I've got a bunch of sites which - when put into maintenance mode - trigger a varnish maintenance mode: they send a new VCL to the Varnish admin port; the new VCL simply serves up a static maintenance page (without hitting any backend). Once you switch off maintenance mode, it switches back to the regular VCL.
It's worth pointing out that this can block you out of your site - this works for me because the sites in question use an authoring environment that doesn't go via varnish.
--
Marcus Deglos
Founder / Technical Architect @ Techito.
What if..
You'd not use the core Maintenance mode but a module like (my) Read Only Mode? Your site will still be accessible and Varnish gets 200's.
The only difference is that users cannot add/edit comments/nodes anymore.
Kind regards, Baris Wanschers
Maintenance mode
We never put our sites into it; you have to audit each update but for 99% of the time it's perfectly fine.
Nginx config for Drupal
Put nginx behind Varnish, and configure it to serve a static maintenance page with the error_page directive.
There is absolutely no need to fire up drupal/php to serve 403's, 404's, or 503's.
Nginx config for Drupal
In fact, you could even do this to keep varnish involved:
error_page 503 =200 /maintenance.html;Just remember, depending on your setup, you may also need:
recursive_error_pages on;
and:
fastcgi_intercept_errors on;
or:
proxy_intercept_errors on;
While I would also just
While I would also just change the VLC, and have the probe fail and let serve varnish all pages from the cache (see recent lullabot varnish config), I think I stumbled upon a possible cause of your problem:
If http://drupal.org/node/997918#comment-4301776 is correct, then the menu_router is rebuild on each page_load in maintenance mode, which would absolutely explain the high load you are seeing.
UPDATE: Yes, that can be a problem, but only for update and install.php:
From includes/menu.inc
<?phpif (defined('MAINTENANCE_MODE')) {
variable_set('menu_rebuild_needed', TRUE);
}
else {
variable_del('menu_rebuild_needed');
}
?>
That for sure could be a performance nightmare ...
But unless you are running update.php should not be the cause of the high load times.
Best Wishes,
Fabian