Hello all,
I am running (and developing) a site that runs on multiple (app) nodes behind a load balancer, and a separate database and storage server. Each node runs Apache (MPM worker) with FCGI, APC turned on. Drupal 7 based, with the APC module enabled, with most cache tables deferred to APC.
Now, all is fine, APC helps a lot with speed improvements. I know it's best to use centralized caching (like memcached) when running multiple nodes, but for now, we do not have a spare node available yet. Plus, I think our problem is not in the fact that multiple nodes are running, but rather with threading (I think). Please do note that the load balancer will always send a user to the same node (IP address based), so cache differences between nodes is not a huge issue for now.
We cache a few objects here and there, and we have the correct hooks to clear cache when needed. The problem is that sometimes when a user refreshes the site, the user is looking at outdated cache.
For checking, I've installed the default apc.php file on the servers, and checked what's going on. There's a strange thing here when refreshing a lot of times: the 'time running' value changes from time to time, and it seems that there are different APC cache instances running (the statistics also change).
My gut feeling is that each FCGI process has its own cache, and Drupal on one process is unable to clear the cache in other processes. Could this be the case, and, if so, is there a solution? I've been looking into the FCGI_CHILDREN setting, but that seems to be FastCGI only.
Looking forward to your input! :-)
Comments
APC per PHP process
I'm rather certain that APC cache is running as part of the PHP process. So when using PHP as an Apache module, you get one APC process, but with FCGI, you get one per process.
I may be wrong, but it does not seem totally illogical to me (opcode caching is another matter, though)... Google did not bring up anything?
Yeah, Google brought up some
Yeah, Google brought up some related issues, but most of them don't seem to mention the 'cache per process' thing.
What I read somewhere is that people suggested configuring Apache to spawn a single CGI process, but with many children for that single process. That ought to share the cache, at least on a single node. I'll see if I can dig up the links tomorrow.
So because you have a
So because you have a completely separate cache per app node, if you do something while connected to node #1 that causes a cache clear it will only be cleared on #1, not on #2.
Not 'best', rather it's mandatory to prevent the issues that you describe.
You don't need a separate machine to put it on. We often put memcache on the same machine as the slave DB. It should also be fine to put it on your dedicated storage machine.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
I agree
It's a good point that you don't need a separate, dedicated, machine for memcache - as long as you have plenty of RAM for the memcache process, it can be on any machine - or a number of (although I don't personally have experience with clustered memcache)
Yeah, I know that clearing
Yeah, I know that clearing cache on node 1 will not immediately clear the cache on node 2, but as I said: these issues take place on a single node. The fact that caches are different on each node does not matter all too much, as people will always be redirected to a single node.
Thanks for the advice on running memcached on either the db node or the storage machine. I think I'll try that :-)
latency gotchas with memcached servers; repcached
I've personally had success dual-purposing web servers as memcached servers.
A gotcha I would like to highlight with memcached is that latency adds up very quickly. Unless the memcached server has, for all intensive purposes, instantaneous response when pinged, the aggregate latency can be detrimental.
Might want to check out http://repcached.lab.klab.org/ - replicating memcached for HA environments.
The lateny coming from the
The lateny coming from the network connection? They're all virtual machines, I could probably even have my hosting company make sure the VM's are all on the same host. Would that help?
Thanks for the info, I will be sure to check it out :-)
VM Latency
If they're on the same host, should be nil.
The latency problem I encountered was with a client who had a badly configured firewall and security appliance between the dedicated webserver and dedicated memcached server (overkill IMHO); it was letting the traffic through, but introduced something like .1 second lag. Not bad... until you realize that there were close to 200 hits to the cache (that's good!) which resulted in 20 seconds of delay (that's bad¡).
Anyone else have experience
Anyone else have experience with Repcached or similar solution? I'll have to check out Repcached.
On our prior host, we tried having memcached on the same server as apache, but they fought for memory too much which caused more problems than it was worth. Moving memcached to a dedicated host fixed these issues.
Since moving to hosts, a dedicated solution works, but there is much higher latency between machines (prior .1ms, curr .7ms), and it adds up. There is plenty of memory available per machine to handle memcached, now but I've have concerns. I didn't want to run memcache in a distributed setup, since there would only be minor gain and per machine would give me headaches with cache clearing. Hopefully repached will solve this.
Which FCGI Apache module are
Which FCGI Apache module are you using??
I did a LOT of testing with different configurations about a year ago..
Be careful with FCGID.. It doesn't handle multiple PHP processes well because it can only deal with a single thread and APC is near useless on it because the cache is recreated on creation of each new PHP process and cant be shared between processes..
Another option you have is to use the fastcgi Apache module but then you must use the PHP process spawning because PHP can't share cache between processes.. The only PHP process handling wasn't as good as that of fastcgi which meant you were between a rock and a hard place..
Ultimately what you want to get to is using PHP-FPM which solves all the problems and runs APC well.. I haven't set this up with Apache because I switched to nginx at the same time as implementing PHP-FPM..
As far as I know Apache 2.4 is the better option for use with PHP-FPM but it can be done with Apache 2.2 as well its just a bit more "fiddley"..
As others have said rather use memcache for your cache requirements.. Its "shared" so you can create smaller caches on each server and this will be pooled into an aggregate cache of the sum of all the caches.. So instead of having a 256MB cache on one server you could have 64MB caches on 4 servers and have the same total..
Thanks for your insights,
Thanks for your insights, they pretty much confirm what I've been finding scattered over forums. One thing I do not understand is that the APC caching does actually speed up things compared to caching things to the (rather well-tuned) database. Especially the opcode caching created a big dent in the loading times. I've seen the bootstrap time go from 200ms to about 20ms. As far as I understand, you can configure FCGI to do multiple PHP spawns per thread, right? http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxrequestsper...
I've also been reading about PHP-FPM, but I have zero experience with it, and AFAIK so does our hosting company. Would be nice to try it out, but is this stable enough to run on production environments?
Eventually I will have to set up a memcached server, or even a cluster, indeed spread out over multiple servers (maybe even the nodes themselves). Although I would really still like to tackle this issue and see if I can get it running nicely, because a lot of smaller websites I've done would benefit from good APC caching. It's relatively easy to setup.
I have been digging around
I have been digging around trying to find my notes but not sure where they are now so a lot of this is from memory..
The issue with mod_fcgid is that it has problems with concurrency.. So even when spawning multiple PHP processes or threads its not able to utilise them efficiently.. Then the APC issue is that you will see speed gains but because there are potentially multiple caches you could end up using valuable memory..
PHP-FPM was designed to address all the issues by building a reliable and efficient process manager into php.. The beauty of it is that it has effectively separated PHP from the web server and this provides lots of benefits.. As an example things like having a server dedicated to processing PHP for one of more front end web servers delivering static content would be possible.. With multiple "pools" you can isolate sites for extra security without needed modules like suPHP or suEXEC and all the drama they caused.. I could go on but I think you get what I am saying..
As far as stability is concerned I have been running PHP-PFM in production for a year now (in conjunction with nginx) and not had a single issue in all that time.. I mean zero problems with PHP!! And APC works like a dream (I only use it for opcode caching)..
My advice for what it's worth to you is if you are setting up a new system the ONLY way to run php is under PHP-FPM.. It's been a year since I did it so there is probably far more information now on how to use it with Apache.. I know Apache 2.4 has mod_proxy_fcgi which is how you would connect to PHP-FPM.. A quick search for "php-fpm with apache" returned what looked like quite a few promising results.. Alternatively there is always the option of nginx instead of apache..
Out of interest what sort of traffic volume are you catering for?
Also if you have enough traffic to warrant multiple virtual servers and a load balancer in front of it why not switch to one or two (or more) dedicated servers which would almost certainly give better IO and throughput and might even be cheaper??
different backends
The other thing worth considering in your setup is allocating to APC only those cache tables which are safe to be cached only on one node - namely form, session and user caches.
The more critical cache tables can then be served from DB until you get your memcached servers up and running.
You should see a performance boost doing this, particularly for logged in users, but avoid retrieving stale cache data for things such as variables cache.
Unless you can guarantee that
Unless you can guarantee that each user will only ever be visiting one webhead (which is possible with some load balancer configurations), you can't include the session, or user cache bins in that category.
The form bin should never be stored in a non-persistent cache since it's not actually a cache - if the contents are lost users will experience data loss. So it's only safe to have the form bin in DB or file storage (not in Memcache or APC).
One of the few cache bins that can be completely independent per webhead is the filter bin. But I doubt you will see much performance/scalability improvement by doing so.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
I don't have a cache_user nor
I don't have a cache_user nor cache_session tables at all. Should I? I've already excluded cache_form.
PHP-FPM APC cache sharing
According to Rasmus, with PHP-FPM, the APC cache is shared across threads in a single pool: http://news.php.net/php.internals/63539
What your describing certainly sounds like an issue with non-shared object cache. The solution to this is to switch to a shared cache like memcached, db or redis.
_
I have problem with installing APC and most of other caching modules like memcache. I still use Boost module to cache my pages. Now i want to know is it needed to setup APC too? Does it really speed up my site's speed?
Doubt is the father of invention ... ZendegiyeSabz
What is your problem
What is your problem installing them?
Boost is fine if you receive mostly anonymous traffic, i.e. the pages all pretty much the same for every user. In my case, users all have different pages, so I had to rely a lot on object caching (blocks, entities and query results and such)
If your website does not have (a lot of) logged in users, Boost is fine. Otherwise, APC might help speed up things, depending on your setup.
_
I installed boost and i have both anonymous and registered members traffic. Whats the best option for me now? Varnish? Boost? Memcached? Please guide me.
Doubt is the father of invention ... ZendegiyeSabz
Caching non-anonymous traffic
Caching non-anonymous traffic usually needs you to re-architect your site to identify which part of the page or http request could be partially cached. For partial caching Varnish (because the ESI) is a good choice.
Aries
_
The best option for me is caching all parts of pages except one block. But i think its not easy. So my second option is caching the whole page completely. Now what module is the best option for me in your opinion?
Doubt is the father of invention ... ZendegiyeSabz
Important to separate concepts
Op-code caching is one part of APC. Object caching is another one, where Drupal's database cache, memcached, redis or others might be alternatives.
However, op-code caching is not just nice. It is a necessary must! It speeds up PHP code execution by several factors - sometimes as much as 10 times faster, IIRC - and Boost is a module written in PHP. And regardless, at SOME point, the content must be regenerated and cached so Boost can deliver it. That content is regenerated using PHP code. So yes, you need an op-code cache and APC is the easiest and most trouble-free, I have used. Xcache is an alternative, though.
Friends don't let friends run PHP websites without op-code caching :)
Thanks all for your insights,
Thanks all for your insights, helped me a lot :-) Eventually set up a memcached server, which turns out is really easy. No idea if I need to tweak it, but preliminary tests show it's even faster than APC, so all is fine for now.
FYI: I put memcached on the database node, which is in the same pool as the app nodes (very little latency). APC it still turned on for opcode caching, because that really does seem to help, and it does not matter all too much if the data is out of sync, since the PHP code does not change frequently, and we can always restart apache after changes.
Hosting company informed me they actually do have experience with PHP-FPM, so I might give that a go later.