I've run into what seems to be a problem with memcache. Memcache has a gig of memory and generally sits at around 85% full with over 50% cache misses. The site has been hit with some traffic spikes of around 40k+ visitors per hour (mostly anonymous visits) and this has been enough to take it down (with the additional traffic to another Drupal site running on the same servers). There are two fairly powerful web servers and I think that if memcache were working properly, things would be fine. The db server is mostly idle though it does rack up sleeping connections (mostly for sessions) and there's a load balancer with sticky sessions.
I'm running the 5.x.-1.8 version of the memcache module with version 1.2.5-2 of memcached and the 2.2.4 pecl package (with PHP 5.2.6 and eaccelerator 0.9.5.3). The current (non-peak average ) cache request rate is around 140 per second (hits + misses). Both web servers just point at their local host instance of memcache but I'm getting ready to configure them to look at both caches (and maybe run memcached on the db server too).
I wrote a small test for the caching system as follows:
function mysite_test_memcache($tries = 50, $sleep = true, $cid = 'mysite_test_memcache', $table = 'cache') {
$data = date('H:i:s l, F dS Y');
cache_set($cid, $table, $data);
drupal_set_message("SET cache to $data");
for ($i = 0; $i < $tries; $i++) {
if ($sleep) {usleep(200000);} // sleep .2 seconds
$cache = cache_get($cid, $table);
if (!empty($cache->data)) {
$hits += 1;
}
}
if ($tries) {
$hit_percent = $hits / $tries * 100;
drupal_set_message("$hits out of $tries hits ($hit_percent%) for Drupal");
}
}I regularly get less than 10% cache hits when I run this on the production server (although it occasionally gets lucky and returns 100% hits... the development server also almost always returns 100% hits). Is my test valid or did I screw something up? I ran a straight php version (skipping the Drupal memcache module altogether) and it gets similar results.
Are other people running memcache getting similarly poor cache hit rates? I would expect that, when the site gets hit with a ton of anonymous traffic going to the same few pages, hit rates would be really good. Similarly, I would expect my little test to have good hit rates as it's trying to retrieve the same cached content over and over, sequentially. Why is there a such a discrepancy between my test results on dev vs. production? Is it because of all the other traffic filling up the cache on production? All the configuration and versions are the same across the environments (afaict). Does anyone have any suggestions for what I can check or try to raise my hit rates? Larger caches (I have 8GB of RAM total)? Multiple memcache servers?
Any other tuning suggestions? I'm testing out static page caching with boost but would rather not go that route due to problems with stale content, comments, etc.. Is there a good solution to get static assets (js, css, files and images) to edge cache on a CDN (Akamai)? The CDN module is on hold until Feb. and it looks like I'd have to patch core (which I really hate doing) if I want to prepend a host name in front of the assets. Any help would be appreciated as I'm not getting a good feeling that memcache is performing as it should right now. Thanks.
Comments
update
No suggestions yet? Is anyone out there using memcache with D5 for (relatively) high traffic sites?
I now have memcache installed on the DB server and have both web servers looking at all three instances of memcache. They've got a total of 4GB of cache available and are sitting at around 15% full (after around 24h since memcache was last restarted). Unfortunately, I'm still only seeing a 50% hit rate. Since I haven't had to deal with a traffic spike since last week, I'm not sure if this new setup is any more robust than the last.
Does anyone know what advantage splitting up the Drupal cache tables into different bins has? Right now, everything is going into a default bin that is cached on all three servers. Should I set the page cache to the local web server and send the rest into default on the DB server?
Hi I am using Drupal 5 and a
Hi
I am using Drupal 5 and a Memcache Solution, but my traffic not so high. My cache hit rate is about over 70 %.
Can you provide a URL or describe me a little bit more about your site?
Is there a lot of user interaction like comments or voting? What are your user doing on your site?
My Drupal based Job Exchange Site:
Jobs Düsseldorf
You should split them
Yes, you should split the bins by tables.
This is what I use on a very high traffic web site on Drupal 5 (above 1 million page views a day).
<?php
$conf = array(
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
'memcache_servers' => array(
'server1:11211' => 'default',
'server1:11212' => 'page',
'server1:11213' => 'filter',
),
'memcache_bins' => array(
'cache' => 'default',
'cache_menu' => 'default',
'cache_page' => 'page',
'cache_filter' => 'filter',
),
);
?>
This way you get to see the hits/misses on a per table basis.
The default one will always have 99% hit ratio. The cache_page table will be the worst, with sometimes 55% ratio, but improves over time. The cache_filter will be in between
There is no point in increasing the size indefinitely. Memcache for the page table fills up and reports close to 90% and stops there. It does not actively free memory of items marked as expired, but just marks them as such and reuses the memory. Even if there are some evictions, if they are not too much, then so what?
Here are some nice pie chart after a week of running with 256MB for page cache. We set it to 1GB in the past and it did not make much of a difference, so we pared it down.
This is the default bin (cache, cache_menu).
This is the cache_page bin.
This is the cache_filter bin. Quite busy, but not too big.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
The cache_page hit is bad,
The cache_page hit is bad, because you're using boost, aren't you? And even, boost disable Drupal page cache (I don't know if this is correct for D5, I'm using D6).
Did you test this website without boost?
There's also a page cache bug...
I recently posted a patch for the D5 version of memcache that I recently contributed after discovering that the page cache hit rate was abysmally low. The problem was that the expiration time was always being set to a date in the past, and consequently pages were being regenerated almost every time they were requested.
The solution was to fix the expiration time calculation.
The same bug very likely impacts cache router.
http://drupal.org/node/685042
Blog: http://technosophos.com
QueryPath: http://querypath.org
Thanks guys, I'll give
Thanks guys, I'll give splitting up the bins a shot. If that doesn't work out, I'll post up the memcache.php images.
Similarly low hit rates on some tables
To piggyback on a similar discussion.. here's what I'm currently seeing on a site I run:
'cache' => 'bucket0', # 0, 99%, 77%, 99%
'cache_block' => 'bucket1', # empty
'cache_content' => 'bucket2', # 85%, 85%, 91%, 89%
'cache_filter' => 'bucket3', # 100% misses
'cache_menu' => 'bucket4', # 80%, 99%, 93%, 90%
'cache_node' => 'bucket5', # empty
'cache_page' => 'bucket6', # 13%, 22%, 20%, 38%
'cache_views' => 'bucket7', # 99%+
'panels_object_cache' => 'bucket8', # empty
'session' => 'bucket9', # 93%+
Notes:
* there are four servers, each bucket configured as a shared bucket across all four servers, so the numbers are from each server.
* I threw in the panels one just to see what would happen.
* The cache_page hit rate is rather low but I'm planning on tweaking some of the cache-clearing.
The one I'm particularly concerned with is cache_filter - it's configured but there are no hits on it vs hundreds of thousands of misses per server. Anyone have any idea why this would appear to not be working?
Thanks.
Damien
I'm running into the same
I'm running into the same issue with cache_filter. I have 1 hit but thousands of misses. I tried restarting the filter bin but saw no improvement. Hoping for suggestions.
Check the format
I had a situation with a client where the cache_filter bin had almost no hits.
It turns out that they had the format for nodes and blocks set to PHP, and hence none of them go to the filter bin, because the "cache" column in the filter_formats table is set to 0 for the PHP format.
You can run the following query to know how many nodes of each format you have:
SELECT f.name, f.cache, COUNT(*) AS cnt FROM node_revisions r INNER JOIN filter_formats f USING(format) GROUP BY name ORDER BY cnt DESC;So, check that, and see if it is the reason.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Ah-ha! That did the trick.
Ah-ha! That did the trick. I set the cache value to 1 for each record in the filter_formats tables and now I'm seeing more hits.
You should not do that
Using PHP in nodes is something that should be used sparingly, if at all, for many reasons, ranging from performance, to security to maintainability.
You should not do what you did.
What you should do is to make nodes be non-PHP. Use another format that is cacheable.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Perhaps there should be a
Perhaps there should be a PHP filter which is cacheable. many people use PHP that is not dependant on a context that is changing from page view to page view. Just a thought.
It is true that PHP nodes are usually a bad idea.