Here is my initial post about performance tuning our site http://www.divx.com/ This is mainly a place for me to brain dump and spark discussion and hopefully open up issues to track and get some of these fixes into Drupal core.
Servers
4 - Web servers with 8GB Memory each, Apache 2.2.x + PHP 5.2.x
2 - Memcache Servers with 8 GB Memory each. 9 bins per server
2 - Database Servers with 16GB each running MySQL 5.0.x
Traffic
We typically do about 30-40mm page views per month, about 15mm unique visitors globally. Most of our static traffic goes to Akamai (css,js,images).
Performance Tuning
What we've done so far for performance tuning.
- Cache Router (http://drupal.org/project/cacherouter) for Memcache
- Boost for caching our homepage in all 5 languages
- Convert all tables except menu_router and search_* tables to INNODB
- CDN patch FROM http://tag1consulting.com/patches/cdn for Akamai
- drupal_lookup_path patch from tag1consulting.com
- Add indexes to fix problem with i18n module
- Add hack to core for providing lock mechanism for updating menu_router
I'm sure there is more stuff, but these are the ones that I can think of off the top of my head. I will update the post with more information as I get the site running at 100% performance. Things seem to be running stable now.
Comments
Glad to hear the menu_router
Glad to hear the menu_router fix worked - was there ever any follow up on that? Any chance it's going to get fixed in core anytime soon? Nice job again on the site.
It's all ball bearings these days...
Wow!
This will make for an awesome case-study, man. Thanks a ton for being willing to share this level of info. :)
One question for starters: how are you utilizing the two DB servers? Master/Slave? Segmented database driver?
Can't wait to hear more.
cheers
-j
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Good stuff
I'm glad to see another large site come out in favor of InnoDB, it has always made sense to me.
If you don't mind, I have a few questions on the above:
* Why did you stuck with MyISAM for the search tables?
* What configuration optimizations have you done for MySQL to work best with the InnoDB tables?
* How often is the content indexed, is it via cron or automatically with each node insertion/update?
* Do you use e.g. Scheduler to schedule content publishing, and if so, how do you balance content publishing vs caching?
Thank you for the insights.
Damien
Just guessing: InnoDB in
Just guessing:
InnoDB in comparison to MyISAM uses much more RAM and disk space and because the search tables easily get huge you better stick with MyISAM. On the other hand huge search tables may slow down searches that much (see Drupal.org in 2008) you'd better go with a separate search engine integrated into your Drupal installation, but sitting on a dedicated machine - of course depending on how complex and frequent searches are in your project.
Search has always made
Search has always made trouble for us. We have around 2 million records in the search index table
Hi Dhaliwal, maybe
Hi Dhaliwal,
maybe integrating Sphinx, Xapian (that's what drupal.org is using since mid-2008) or even Apache Solr would make sense to you?
http://drupal.org/project/sphinx
http://drupal.org/project/xapian
http://drupal.org/project/apachesolr
Alex
The drupal_lookup_path patch
http://tinyurl.com/da26to
somehow the post on tag1consulting is blocked if you go through the front door.
cache router config
Hey guys,
I have been looking at cache router and memcache but I have not found any good examples of using multiple memcached servers and multiple bin's. I see from your post, slantview, that you have it setup this way, 2 memcached servers with 9 bin's each. Would it be possible for you to send or post a copy of your configuration for this setup?
The biggest concern I have is that when someone updates or deletes an item from the memcached bin, does it get updated/removed from both memcached servers? We are looking at having a cluster of memcached servers as our clients grow and we want to make sure we put it up right.
Thanks
For memcache
the example right on memcache's project page has that info:
<?php
$conf = array(
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
'memcache_servers' => array(
'10.1.1.1:11211' => 'default',
'10.1.1.1:11212' => 'default',
'10.1.1.2:11211' => 'default',
'10.1.1.3:11211' => 'cluster2',
'10.1.1.4:11211' => 'cluster2'),
'memcache_bins' => array(
'cache' => 'default',
'cache_filter' => 'cluster2',
'cache_menu' => 'cluster2'),
);
?>
See how default is spread on 3 bins, two of them on one physical machine with two memcached instances, and a third on a different machine. The same goes for cluster2, being spread on two physical machines.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Re: For memcache
Hi kbahey,
I had seen that for the memcache module, but I was mainly looking at the cacherouter since I can intermingle different caching systems. For example APC and memcache or file and memcache. I have seen the primary/secondary patch that someone posted and that looks interesting to me.
From what I understand about the cacherouter module it does not use the same variable names as memcache module does and I am wondering how the cacherouter module can utilize 2 or more memcached servers and make sure the 9 bins on each memcached server are kept updated when someone goes in and updates the web site(s).
I have not been able to find a good example of using multiple memcached servers with 9 bins each, like slantview mentioned.
Thanks for everything.
Not multi-server
Remember that not all cache backends are multi server capable. The only true multi server cache backend is memcache.
So, APC and files will not be spreadable on many servers.
Probably you already know that, but pointing it out just in case ...
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
I appreciate the info
Thanks, man, I appreciate the info and will definitely keep it in mind as we start looking to build out sites. :-) Believe me, I appreciate any and all help I get from the communities.
here you go.
Here is our config from divx.com for cacherouter. The ip addresses have been changed from our production ip addresses. There are two custom cache tables that we use for internal caching. Also, Khalid is right that memcache is usually the best backend for multiple servers, but a case could be made for non-user specific cache data that changes infrequently could be cached in something like apc. The only thing is that it could end up in an inconsistent state if you don't invalidate it correctly.
<?php
$conf['cache_inc'] = './sites/all/modules/contrib/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
'default' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11211', '192.168.0.151:11211'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
'fast_cache' => TRUE,
),
'cache_block' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11212', '192.168.0.151:11212'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_content' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11213', '192.168.0.151:11213'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_filter' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11214', '192.168.0.151:11214'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_form' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11215', '192.168.0.151:11215'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_menu' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11216', '192.168.0.151:11216'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_page' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11217', '192.168.0.151:11217'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
'fast_cache' => TRUE,
),
'cache_certified_list' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11218'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_divx_store' => array(
'engine' => 'memcache',
'server' => array('192.168.0.151:11218'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
);
?>
Slantview Media http://www.slantviewmedia.com/ | Blog http://www.slantview.com/
Very Cool Thanks
Slantview,
Thanks for the config. I see now how it needs to be setup.
The next question that I have is, how does cacherouter handle the load balancing and updating memcache flush and sets across both memcached servers when a user comes in to update content on the web site? For example, I saw where there is an addServer in the connect function of the memcache.php engine file, but I don't understand yet how it chooses which memcached server to do a get from, does it just do a round robin between the two memcached servers for each request? And if there is a flush or delete sent from the web site, does it submit the flush or delete on both memcached servers so that no matter which memcached server gets hit for the next request the new content is pulled?
Again, thanks for the config it really helps me understand how it needs to be setup in the settings file.
I don't think cacherouter
I don't think cacherouter does this at all. Memcached client implementations hash the key against all known (configured) memcached instances. The instance / server is chosen upon the hash. See http://www.linuxjournal.com/article/7451
Alex
Rad dude!
Hey so I'm using a similar setup as part of a standard EC2 AMI I'm building up.
Any interest in including this kind of info in cacherouter's README.txt or the like.
One question: are there any implications to using fast_cache on bins other than cache_page?
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Memory divisions between memcache bins
How did you divide up your memory between your various memcache bins? I'd guess that cache_page and cache_filter probably need the most, with most of the rest (except maybe your custom caches) not really needing all that much memory to hold the entire cache.
Does anybody have a rough idea of how much memory a page (for example) takes up in memcache? Or maybe a pointer to figure it out for myself?
From my understanding any
From my understanding any given data is only present in one single memcache bin at a time. Multiple bins are mainly used to add memory and spread the load. The applications use markers in the data hashes to determine which bin to use for a given data.
Alex
Correct
The client api uses a hash mechanism to spread the cache over multiple bins. However there is some level of redundancy because once a server is failed, it will store the data in a different server once a server is dead so you will still be able to serve all cached data. However the data is NOT redundant, it has to rebuild the object being stored on the second hit.
Slantview Media http://www.slantviewmedia.com/ | Blog http://www.slantview.com/