Hi
We had been running a site for some time with apc+memcache+cacherouter with no problems, when we suddenly we started to get blank pages (I say suddenly because we couldn't find out exactly what was going on).
Well, not blank pages, but more like blank blocks. In the end it looked like memcache was not able to return what it was supposed to and the pages were being drawn pretty badly. Flushing all the caches fixed the issue, but only for a couple of minutes.
Sadly we didn't have time to troubleshoot the issue because it was either troubleshooting or serving pages, so we had to revert all caches to DB.
We have a multiserver setup, and we have not been able to reproduce on a single-server setup the kind of problems we had on the live site, so that points to the 'multi' part of the setup, and again, memcache was the only thing that seemed to be breaking.
We tried a new setup updating all the software and compiling it from scratch, so we were testing with these versions:
apache2
apc 3.0.19
php-memcache 2.2.6
libevent 2.0.12
memcached 1.4.6
memcache API 6.x-1.9
After a while the issues showed up again, so we are back in DB mode. Anybody know how to start tackling this problem? How can I debug what is supposed to get in memcache and what is being called when we build a page?
Can this issue http://drupal.org/node/273824 have anything to do with it? We don't use zlib on php.ini but we do use deflate on the server side (apache2)
Tnx and any pointers will be greatly appreciated
Comments
Check memcache object storage
"Bad Things" can happen when memcache doesn't store objects because they're over the default 1 MB limit.
(N.b.: This is not equivalent to the "slab size" settings. There's been some confusion on these boards about these two settings in the past.)
See here for more detail: http://groups.drupal.org/node/116764
As I said there, we have two PF6 apps that have experienced different symptoms with the same cause. Apparent cache stampedes, broken views, sluggish response times, all can be attributable to overlarge objects not actually getting stored in memcache.
The solution, which is easy enough to try if you have RAM to spare, is to make a hefty bin for, e.g., cache_views and increase object size to whatever (my safety zone is 3 MB).
http://groups.drupal.org/node/116764#comment-384949
Another symptom of this, for future reference: Your site becomes sluggish after a code deployment or enabling a new module. You look in the semaphore table and notice that the variables cache semaphore is always set, and has a new value each time you reload the table. This can mean that the main "variables" object in the default cache has grown too large to store, so each httpd is regenerating during bootstrap. Try increasing your default memcache bin's object size limit and restarting.
This sounds very similar to
This sounds very similar to http://drupal.org/node/853864 - assuming those blocks are powered by views.
Out of interest, if you have a bug that you suspect is due to memcache, why not post it in the memcache issue queue instead of here?
No bugs here
I don't believe the behavior is buggy. The memcached FAQ explains the reasoning behind the limitation:
http://code.google.com/p/memcached/wiki/FAQ#Why_are_items_limited_to_1_m...?
The FAQ lists alternatives, but doesn't mention the
-Iflag -- you need to read the man pages for that.If your blocks are going
If your blocks are going empty, then it's a bug, somewhere. The OP was assuming that bug was in memcache (either the Drupal contrib project or the library) - my question was why they're posting support questions/bugs to what is supposed to be a discussion group, rather than the issue queue for the project they suspect.
Also I doubt this is the 1M size limit, it's exactly the same behaviour I observed on a client site with that Views race condition.
Sorry, thought your question
Sorry, thought your question was for me, not the OP. As it turns out, this could either be a bug or a tuning issue, and in the latter case the question is appropriate for this forum.
I've had this exact symptom on a site where the root cause was this memcache tuning issue. I've also used (and much appreciated) the patch to Views for that race condition.