General advices for high performance mySQL ?

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
fakemark's picture

Hello,

I would like to know what are your general advices regarding to a multiserver Drupal website that has to deal with a lot of mySQL queries ? What could be the best architecture for heavy load mySQL deployement ?

I think about a mySQL master/slave configuration, with separate hardware. But I heard that Drupal isn't ready to use a real master/slave configuration, and the slave should only be used as fail-over. Could you confirm this ?

I attached my planned architecture with this post.

Thanks a lot for your time

AttachmentSize
drupal_arch_sample.png54.66 KB

Comments

You can use it for some things

joshk's picture

If you are careful and precise, you can use a slave server for any read-only task. This involves writing some custom code, and you should be certain that the mysql connection credentials Drupal will use don't have write access to the slave (don't break your binary logs), but it totally works.

If you're stuck using drupal search instead of solr, that's a good one. Anything else computationally intensive (reports, etc) are also smart.

The secret sauce is that the $db_conf variable in your settings.php can be an array. So you can have a $db_conf['default'] and $db_conf['slave'], and then in your custom code you can db_set_active():

http://api.drupal.org/api/function/db_set_active/6

This command will return whatever db was previously engaged, so a good rule of thumb is to have your code run like:

<?php
  $previous_db
= db_set_active('slave'); // we are now using the slave
  // insert slave code here; BE SURE OF WHAT YOU ARE DOING
 
db_set_active($previous_db); // back to whatever we were using before
?>

In most situations you're just going back to 'default', but inserting this little extra bit of discipline will save you should you ever get to a more complex setup like different slave servers for different purposes, or a totally unrelated external DB connection.

http://www.chapterthree.com | http://www.outlandishjosh.com

Also: InnoDB

joshk's picture

InnoDB (and the no-locks patch if you're running 5.0).

Must have for a high-performance setup, and better disaster recovery in the bargain.

http://www.chapterthree.com | http://www.outlandishjosh.com

Just upgraded to InnoDB

kyle_mathews's picture

Just upgraded to InnoDB using Maatkit. It makes upgrading entirely painfree. See this blog post:
http://www.mysqlperformanceblog.com/2008/07/30/quick-tip-how-to-convert-...

Kyle Mathews

Kyle Mathews

PHP Solution

mikeytown2's picture

PHP option for the few who don't like the command line.
http://drupal.org/node/51263#comment-1850486

mysqlnd

Voight-Kampff's picture

I'm in the process of building out a drupal 7 test server running Nginx, PHP-FPM 5.3.0 configured with mysqlnd. So far, with no APC or memcache the performance is spectacular. I don't have benchmarks but it's like night and day from my old apache setup.

There are some gotchas when setting this up (PHP-FPM, PHP 5.3.0 and mysqlnd are all definitely on the bleeding edge of a server config) but the effort is worth it.

wow! 2 unknown to me

PlayfulWolf's picture

wow! 2 unknown to me technologies in one post - something new and worth to look at ;)
So, googled a bit about mysqlnd, it is possible to just use it with Drupal 6?
From which part (exept Nginx) most performance improvement is comming?
What are server hardware specs?

---
naslenas.com. Drupal blog experiment.

drupal+me: jeweler portfolio

Interesting

kbahey's picture

Ethan

This is very interesting.

Is this the same hardware? If it is not, then we can't compare.

Also, what was the old Apache setup? If it was MPM pre-fork + mod_php, then it is not as lean as MPM Worker (threaded) + fcgid. Basically PHP-FPM is pretty much like fcgid, and manages PHP FastCGI processes better than other FastCGI setups.

The new part here is mysqlnd (Native Driver). It requires PHP 5.3 though, so not in the stable version of the distro I use.

If you can do it, we need benchmarks on the same Drupal instance, same hardware, without page caching.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

I'm actually migrating from

Voight-Kampff's picture

I'm actually migrating from MediaTemple GS to Slicehost, so not the same hardware (both cloud hosting, but there's no commonality other than buzzword there). However I am preparing to set up a test on the same hardware within Slicehost.

I'd welcome suggestions on approaches to benchmarking this. Basically, I was planning on cloning a base install of Ubuntu 8.04 LTS, shutting off all services other than web/php in both cases and running some scripts or tools to hammer on a baseline Drupal 7 installation.

Ideally, there would be a test matrix that would include caching variant such as APC, memcache, etc.

I'm researching options for hammering the server. If you have any suggestions, please let me know. I'd love to come out of this with an idea of what would be an achievable, high performance setup for new single server drupal installations (i.e. not for reverse proxy setups, etc.).

EDIT:

I want to emphasize that I'm very willing to put in the time to benchmark this accurately and would be very happy to collaborate or receive input from those with more experience on this (which is, effectively, everyone in this group ;).

ab

kbahey's picture

The simplest took that will require no added configuration is ab.

http://httpd.apache.org/docs/2.0/programs/ab.html

Use 10 or more concurrent sessions, and a couple of thousand requests, and that should give a base line.

Turn off all caching.

Post results from all scenarios, so it is comparative on the same platform.

The good scenarios would be:

  1. Apache with MPM Worker and fcgid, regular MySQL. (Details are on 2bits.com for this, and in previous threads on this group).

  2. Same as 1 but with mysqlnd.

  3. Ngnix/FPM and regular MySQL.

  4. Same as 3, but with mysqlnd.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Thanks. I'd run across

Voight-Kampff's picture

Thanks. I'd run across mention of that but had dismissed it thinking it would only be applicable for Apache testing.

I was not planning on altering the base D7 install (no caching, etc.) and was not planning on adding any sample data (just a plain jane front page load). Thoughts?

yeah!

joshk's picture

Loading /node w/no caching is a good (if basic) baseline test for page-generation times. It's vanilla, but it measures something real, and it will make for interesting comparisons.

For extra credit, running a script through jmeter or seige that loaded a number of individual nodes sequentially would be great. Hitting the frontpage without caching will tell us how good drupal is at building it up, but doesn't tell us much about how much better (or worse) the DB layer has gotten, since it'll be running the same queries every time, and mysql will cache that internally.

http://www.chapterthree.com | http://www.outlandishjosh.com

This is all good feedback. I

Voight-Kampff's picture

This is all good feedback. I think that before I execute the tests I'll draft a short test document outlining my test process and post it here, just to catch any obvious flaws prior to running it.

Use Drupal 6

kbahey's picture

If I were you, I would start with Drupal 6, with the devel generate module having some 30 nodes on the front page and as many blocks as you can put in.

If you can afterwards upgrade the site to D7 and re-run the benchmarks, it would be nice to know the current D6 vs. D7 delta. But that is a separate issue, what you are doing is Ngnix/FPM/mysqlnd vs Apache/fcgid/mysql. Don't confuse yourself with too many scenarios.

Use ab with a simple command line like:

ab -c10 -n5000 http://example.com/

You may want to increase the -c, but that is a good start.

Siege and Jmeter are more functional, but they are more complex to install and configure and use. Hence my recommendation to just using ab until you need the extra bells and whistles.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

I had planned on using

Voight-Kampff's picture

I had planned on using php5.3 (so D6 would be out of the question) as I had problems getting 5.2 php-fpm and mysqlnd up. I'll look into giving it another shot.

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week