Hi All,
I would appreciate some discussion/input regarding running with multiple varnish servers. We host a bunch of site on AWS, some of them are reasonably high traffic, and are integrating Varnish into our setup. We are currently considering 2 different configuration models: Option A: run with 2x Varnish frontend servers, passing off to the required backend servers; and: Option B: have each Drupal server with a dedicated Varnish machine (Mercury-type setup)...
Our problem with Option A is how to create a highly available setup. We considered using ELB, but we have not yet figured out how to get the client ip to pass unmolested through ELB, Varnish and arrive unscathed at Drupal, and we also have not yet figured out how to stop the (suspected) ELB cookies for sticky session destroying the cache hitrate for Varnish. I am trying to work these out, but would appreciate the groups opinion on this setup.
Option B seems like a waste for multiple machines, and we really like the idea of split roles, and tuning each element for its own workload.
We did write a tiny patch to the Varnish module that allows us to invalidate the cache for multiple Varnish machines, as opposed to just one - this obviously screws with the Varnish stats in Drupal, but that is less of a worry - we get that data from our management environment. We will probably investigate using the Expire module soon, and applying something similar.
If anybody has any experience with either option, (or a completely different approach) I would really appreciate your input.
Thanks!
Martijn
Comments
Using varnish and drupal
Hi Martijn,
My latest project is composed of something similar of your Option A.
* hardware loadbalancing applicance
* 2 x Varnish servers
* 2 x Drupal servers
First thing to know, you can do a lot of magic at the varnish level (i mean, you can really do crasy things) : cleaning out useless cookies, or controlling wether or not you want to cache something. Default behavior is to let it pass whenever a cookie is set, but this can be (and must be) fined tune.
For instance, you can most probably freely clean out the Load balancing cookie from ELB at the vcl_recv. My hardware loadbalancing was doing just this, i remove the cookie on sub_recv before it goes down the varnish cycle:
Another example, we wanted to activate the Poll module using a cookie based patch (so we can differentiate anonymous people voting): since it's using a cookie, we were able to teach varnish to still cache those pages, even if have this cookie. We are actually using the cookie to store multiple versions of the same anonymous pages (with the vote block activated or not).
Also on the X-FORWARED-FOR stuff, an issue has been reported depending the varnish version you are using: basically, varnish was adding an additional XFF header, while we expected it to concatenate them (Pressflow has something about this). Latests versions of varnish let you totally control this on the VCL configuration, which is very flexible. This is what you are supposed to do to let the real client_ip going all the way thru drupal (and of course configure drupal with the XFF stuff on the settings.php file).
The biggest issue i had, was to properly manage the VCL configuration depending of the version of varnish you have. For instance, debian based distribution are using a pretty old version that is using POSIX regex, while the latest versions are using PCRE regex.
You can analyse whatever http header in varnish and put some logic in there to achieve the best caching strategy.
Multiple varnish servers using each other before a backend?
Does anyone have experience setting up 2 varnish servers which use each other before resorting to sending a request to the backend?
So: A request would come in to varnish1, if a cache doesn't exist it was query varnish2, if a cache doesn't exist it would send the request to a random/round robin backend server. Varnish 2 would cache the response, as would Varnish 1...before finally responding.
This suggests it's possible, but I'm not sure how exactly to achieve this:
http://varnish-cache.org/trac/wiki/VCLExampleHashIgnoreBusy
I'm having a hard time
I'm having a hard time envisioning what the benefit of having two Varnish servers in serial (rather than parallel). Though I can't think of anything that would prevent you from doing so.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
I have to say I agree with
I have to say I agree with @Dalin, having varnish servers in serial is without a benefit. You would have 2 single points of failure in a row and the servers would have identical caches. Putting two in parallel would let you handle higher load and could give you redundancy.
HA varnish servers
so I have two varnish servers but they're not serial, they're in tandem. Having trouble with the drupal varnish module clearing them both.
My OPS guys insist on 2 varnish servers for HA (High Availability).
Any suggestions?
Should not be an issue.
Should not be an issue. There's notes on the settings form on how to add multiple IP addresses to be cleared.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
@dalin - yea, after I posted
@dalin - yea, after I posted this I read thru the code and figured it out. Sorry to bother you. Thx.
-t
Stovak - Can you expand on
Stovak - Can you expand on how you managed to do this?
Apart from the ability to add multiple Varnish Control Terminals, I'm not seeing any further guidance in the Varnish module on how to trigger a purge or cache clear on both Varnish instances when content is updated on one webserver.
Situation sounds like B above - Load balancer in front of two VMs acting as webservers on the same physical box, both VMs have an instance of Drupal and Varnish. When updating content or files on either Drupal instance, changes are instantly sync-ed on the other, and Varnish cache is cleared on content update on whichever VM the update occured. However, can't get Varnish to clear on the other VM, so out-of-date cached content persists. Would break out Varnish on to seperate VMs if I could, but out of my hands.
Any tips?
Thanks.
What you have is a pretty
What you have is a pretty common configuration. Sounds like your problem might be with firewalls. Try connecting via telnet from one webserver to another. Get that working first before you try it from within Drupal.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Thanks for the thoughts
Thanks for the thoughts Dalin.
Not sure about Telnet, but I can send a purge all request from commandline from one to the other server and Varnish clears (varnishadm -T xxx.xxx.xxx.xx:6082 -S /location/of/varnish/secret "url.purge .") - we don't have any specifc firewall in place between the two web servers.
I'm using D6 Pressflow, Varnish 2.1.5, Varnish module, Expires and Purge modules, with VCLs based heavily on the the Lullabot example (http://www.lullabot.com/blog/article/configuring-varnish-high-availabili... - but not set-up for multiple backends - each instance sitting on same VM server in front of each Drupal instance, so no multiple backends defined here). Running Varnish module Cache Clearing as 'Drupal Default' as the Expires provided 'selective' option (while highly desirable) seems to stop Varnish clearing at all on either server when content updated on one. Purge module configured with both proxy server URLs (Varnish running on port 80) and Varnish Module successfully connected to control terminals on 6082 for both Varnish instances, Drupal caching mode set to 'external' with minimum cache lifetime and page cache maximum age of 12 hours.
After updating content on one server and listening with 'varnishlog -i RxRequest' on the other I see the PURGE requests come through, but Varnish doesn't clear cache on second server.
Suspect that module development for Expires and Purge has tailed off on D6 now, unfortunately not yet in a position to upgrade to D7.
I guess my next step is to dig into the Varnish module and create a custom module to override whatever hook triggers purge request to ensure it sends to both servers when triggered on content update for either?
Forogt to mention - I've
Forogt to mention - I've included acl purge details in VCL with a sub vcl_recv clause for PURGE requests, so the PURGE requests that are getting through cross-server shoudln't be falling on deaf ears..