We have a setup like this:
4 servers running apache+php+apc with drupal6 and cacherouter using the apc
4 squid caches, each one pointing to a single machine
1 Alteon doing the balancing
We set up the settings.php to use multiple servers with $cookie_domain and $base_url (with NO trailing slash)
We are currently having around 250.000 page views a day, but we're still struggling to keep the servers running smooth.
Our main problem is that the editorial staff apparently hit different servers during the editing process and they either get logged out or just get an error about the forms...
So my question is: since it looks like our problem is synchronization between caches, is there a way of making apc aware of the existance of other machines that are also running apc? or is memcache my only other option?
Comments
First thing I would ask is -
First thing I would ask is - What algorithm do you use in squid? You should use source based IP algorithm - that people coming from one IP will be always connected to one server. Also, you should solve sharing sessions somehow...
Do you use any knd of shared storage?
Jakub
--
Jakub Suchy
Dynamite Heads http://www.dynamiteheads.com
Hi and thanks for the quick
Hi and thanks for the quick answer ;)
For squid I think we're using the default configuration with very few changes...
But... correct me if I'm wrong (and I probably am since it's my first time with squid ;)), since the alteon load balancer has session persistance active and we have a 1-to-1 connection between each squid-apache, shouldn't we have that end covered?
As for storage we have a server running nfs that shares the /sites/default/files directorybetween all of them, and we run an rsync on the rest of the tree excluding the files directory.
Tnx again
Benja
Well, obviously, you said
Well, obviously, you said that the editors are hitting different servers, that means that the session persistance is not working somewhere...
I think the problem with the
I think the problem with the sessions may be the different layers of caching we have right now:
How can I keep track of all these?
APC is constrained to hosts.
APC is constrained to hosts. Even if you are running two different VHosts on the same server, one Vhost can't access the APC cache of the other, so you need to go with something like memcached + APC. APC would be used strictly for the op-code caching (php) and memcache will be used for Drupal's caching through cache router.
The way you are running now you risk running into tons of problems. If a cached item is invalidated on server 1, then servers 2-4 will have stale items. Drupal does a lot of caching, a lot which is dependent for Drupal to run properly, so you can see how this quickly can lead to all kinds of problems - even ones you can't think of. Most likely that is what's causing your form error problems (bad form cache).
You could go with memcache a couple different routes. I would say install it on all your servers then split your cache bins up to 2 servers each. So say cache_page goes to servers 2+4 then default cache goes to servers 1+2, etc.
Another option would be to just drop one of the Apache servers and make it a memcache server. I run a site with similar traffic and here's how we do it:
1 - Quad core running Apache with Drupal + APC opcode caching
1 - Quad core database
1 - Dual core Apache running as a static server + running memcache.
There are modules and patches out there to rewrite all image, css, js files to point to a static subdomain. Since that server never runs anything SSI it has plenty of processing and memory to handle running our 4 memcache instances.
With this setup I have seen our site get over 60,000 hits in an hour (that's the nature of a political blog/news type site - you get breaking items that result in getting slammed with traffic) and our server loads never went above 3.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
I'm thinking about setting up
I'm thinking about setting up a memcache server similar to your configuration. What did you use for memcache server / bin configuration? From what I can see, a cache flush affects all of the bins on that memcache server. That seems to indicate that maybe more servers is better. What have you found for a sweet spot?
We are, in fact, running into
We are, in fact, running into these problems with the stale content.
To make things a little more difficult we have servers at 2 different locations, and we are getting problems because of line saturation: if we enable the memcache server sharing for all servers, the link that joins both locations starts spiking like crazy...
And I guess it is a matter of choice, but what do you use? cacherouter or memcache for drupal? I can't figure out which one is better, but the fact that memcache makes me patch some core files...
We're using cache router.
We're using cache router. It's been working great for over a year now.
You're basically going to be looking at the same amount of data going through the network if you use memcahedm file caching or the database to cache, so you are probably going to have to look into increasing your bandwidth. The whole goal is to keep it so that all your installs of Drupal are working off the same cache.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Ok... we're currently using
Ok... we're currently using cacherouter, but I know the other module, memcache, asks for some changes to be made on the core... But I can't see anything like that on the documentation for cacherouter.
Since you have it running... do we need to change something on the core specifically for memcache? I assume not, but since it seems you have a fairly similar setup, I'd like to know if there is something we're missing that can't be found on the documentation :)
Also, when we enable memcache, do we have to clean up all the caches to make sure it picks up only the new stuff? I have even read some instructions about literally emptying the cache tables on the database...
On D6 you don't need to do
On D6 you don't need to do any core modifications.
You don't need to worry about the cache tables in the DB, although its a good idea to go ahead an truncate them sometime soon after using memcache just in case you need to switch back to DB for some reason. Even if you don't use that, memcache will still pick up on the new stuff since it won't exist in cache.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
On D6 you don't need to do
[DOUBLE POST - darn Firefox!]
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.