Varnish errors on loaded sites ... Error 503 Service unavailable... Guru Meditation
In certain instances I see the next error (which I remember vagually seeing in drupal.org.il)
Error 503 Service Unavailable
Service Unavailable
Guru Meditation:
XID: 1525473736
From googlin' i understand that this is apache not responding to varnish, Is this correct?
How do I play with the time varnish gives apache to respond?
best regards
Lior
Groups:
Login to post comments
Sky's the limit with VCL
There's a world of possibility in the Varnish Control Language:
http://varnish.projects.linpro.no/wiki/VCL
For instance:
backend www {.host = "www.example.com";
.port = "http";
.connect_timeout = 1s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
I will review these settings in the base install and look at making the defaults more forgiving for apache.
http://www.chapterthree.com | http://www.outlandishjosh.com
503 errors
I've seen 503 errors when my system is under load. We've reduced those greatly by increasing the timeouts in the vcl conf file:
backend default {.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
600 second timeouts are likely excessively high, but it's worked for us. We've also tweaked the varishd startup options (/etc/conf.d/varnishd):
VARNISHD_OPTS=”-a *:80 \-T 127.0.0.1:8181 \
-f /etc/varnish/default.vcl \
-p thread_pools=4 \
-p thread_pool_max=1500 \
-p listen_depth=2048 \
-p lru_interval=1800 \
-h classic,169313 \
-p obj_workspace=4096 \
-p connect_timeout=600 \
-p max_restarts=6 \
-s malloc,2G”
We increased the connect_timeout, thread pools and set max_restarts. This combination has virtually eliminated the 503 errors for us. We're seeing 30k hits/day on this config without any load issues (varnish and apache are on the same 12gb/4core box).
Rad
Thanks for posting this Greg. I'm going to include these settings in the beta release for sure.
http://www.chapterthree.com | http://www.outlandishjosh.com
malloc storage
Note that we chose to tell it to only use RAM for storage by setting it to malloc. Varnish defaults to file-based storage under the philosophy that the kernel will know better how to manage what needs to be buffered in RAM and what can be dropped to disk. In reading the Varnish docs, I expect this is likely not a recommended config in the eyes of the Varnish devs.
Slightly different location in mercury/debian/ubuntu...
Great Advice!
For people looking for this in a debian/mercury box the fixes go in /etc/default/varnish instead of /etc/conf.d/varnishd (I think that's rhel?)
Also it appears debian likes DAEMON_OPTS instead of VARNISHD_OPTS
So my code looks like thsi..
DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -p thread_pools=4 \ -p thread_pool_max=1500 \ -p listen_depth=2048 \ -p lru_interval=1800 \ -h classic,169313 \ -p obj_workspace=4096 \ -p connect_timeout=600 \ -s malloc,2G"Linnovate - Community Infrastructure Care
Drupal Services in Israel
http://www.linnovate.net
Gentoo, actually :-)
Sorry for the confusion. These configs are running on Gentoo, not a Debian variant.
Thanks
Another thorn in my side is figuring out if/how I can get varnish to listen on multiple ports. The way the opts are configured, adding the correct syntax seems to break the start script. I can do it directly from the command line, but that's no good.
Why multiple ports? Well, because we want to put Varnish over solr too... :)
http://www.chapterthree.com | http://www.outlandishjosh.com