Benchmarking Drupal core for 5.x, 6.x, and 7.x: We are getting slower

Posted by kbahey on October 27, 2008 at 12:45am

We conducted some benchmarks for Drupal 5.12, Drupal 6.6 and Drupal 7.x (Oct 24 checkout).

We used the same data set for all 3 instances:

Users 4,950
Nodes 5,000
Comments 20,000
Vocabularies 10
Terms 1,000

And same test parameters: 10 concurrent users, 2 minutes stress test on 30 individual nodes, and the home page.

      Description,   Trans,  Elap Time,  Resp Time, Trans Rate, Concurrent
Drupal 5; No APC; No cache,   1994,     120.25,       0.60,      16.58,       9.97
Drupal 6; No APC; No cache,   1787,     120.36,       0.67,      14.85,       9.96
Drupal 7; No APC; No cache,   1255,     120.47,       0.96,      10.42,       9.95
Drupal 5; No APC; w/ cache,  19481,     119.78,       0.06,     162.64,       9.98
Drupal 6; No APC; w/ cache,  15589,     120.41,       0.08,     129.47,       9.98
Drupal 7; No APC; w/ cache,  11418,     120.26,       0.11,      94.94,       9.98
Drupal 5; w/ APC; No cache,   6537,     120.42,       0.18,      54.29,       9.99
Drupal 6; w/ APC; No cache,   4706,     120.56,       0.26,      39.03,       9.97
Drupal 7; w/ APC; No cache,   2503,     120.58,       0.48,      20.76,       9.95
Drupal 5; w/ APC; w/ cache,  50698,     119.77,       0.02,     423.29,       9.96
Drupal 6; w/ APC; w/ cache,  46520,     120.49,       0.03,     386.09,       9.96
Drupal 7; w/ APC; w/ cache,  34359,     120.05,       0.03,     286.21,       9.98

* Description: Description of the test.
* Trans: Number of HTTP requests completed. 
* Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
* Resp Time: Response time per requests, in seconds across all requests.
* Trans Rate: Number of transactions per second. The higher the better.
* Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.

More here: Performance benchmarking Drupal 5.12, Drupal 6.6 and Drupal 7.x we are getting slower.

Things should have been better with the registry.

We need to a) identify the source(s) of this slowness, and b) try to remedy it before Drupal 7.x goes out.

Comments

registry optimisations

Posted by justinrandell (not verified) on October 27, 2008 at 12:53am

thanks for the benchmarks :-)

right now, the registry runs a whole bunch of queries that can be eliminated with some optimisation.

here's the first thing that needs to go in:

http://drupal.org/node/298600

after that, we can cut some more with this:

http://drupal.org/node/325665

Don't get confused as I did

Posted by FiReaNGeL on October 27, 2008 at 1:11am

Don't get confused as I did with the column titles not lining up with the values - thanks for this kbahey, pretty significant decrease there :(

Spreadsheet

Posted by kbahey on October 27, 2008 at 4:11am

Yeah, it can be confusing.

To remedy that, and for people to crunch their own numbers, I attached an Open Office spreadsheet and a CSV file to the article linked above.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

When you say cache, are you

Posted by moshe weitzman on October 27, 2008 at 2:32am

When you say cache, are you referring to page caching? What about block cache? Did you enable any blocks?
These are authenticated users or anonymous? Mysql is configured same for all? Optimized or not? If not, we might be masking important insight.

Answers

Posted by kbahey on October 27, 2008 at 4:07am

When you say cache, are you referring to page caching?

Yes, page caching, the variable in Drupal called 'cache'.

What about block cache?

No.

Did you enable any blocks?

Just one, the Login block.

These are authenticated users or anonymous?

Anonymous.

Mysql is configured same for all?

Yes. Same physical box, same MySQL instance, benchmarks ran back to back.

Optimized or not?

Not much. This is a development server, but it is all the same for all tests.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Prepared statements -vs- query cache?

Posted by markus_petrux on October 28, 2008 at 12:14am

Hi kbahey,

In your article: "They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4."

In the MySQL manual: "Before MySQL 5.1.17, prepared statements do not use the query cache."

http://dev.mysql.com/doc/refman/5.1/en/query-cache-how.html

Maybe you could try updating MySQL and see how numbers change for D7? ...or try again with MySQL 5.0, but disabling MySQL Query Cache?

I believe d7 uses emulated

Posted by mfb on October 28, 2008 at 12:44am

I believe d7 uses emulated prepared statements, not mysql native prepared statements? Although perhaps native prepared statements could be enabled on newer versions of mysql..

Same trend

Posted by kbahey on October 28, 2008 at 2:19am

I see the same trend with MySQL Query cache disabled (query_cache_type = 0 in my.cnf).

No page cache; w/ APC; MySQL query cache disabled

Description,  Trans,  Elap Time, Resp Time,  Trans Rate
   Drupal 5,   5120,     120.06,      0.23,       42.65
   Drupal 6,   3607,     120.05,      0.33,       30.05
   Drupal 7,   1847,     120.36,      0.65,       15.35

Here are the same tests with query cache re-enabled.

No page cache; w/ APC; MySQL query cache enabled

Description,  Trans,  Elap Time, Resp Time,  Trans Rate
   Drupal 5,   6459,     119.96,      0.18,       53.84
   Drupal 6,   4671,     120.28,      0.26,       38.83
   Drupal 7,   2506,     120.25,      0.48,       20.84

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

More benchmarks

Posted by kbahey on October 29, 2008 at 10:25pm

Here are some more data points to help us analyze the issue further.

This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.

    Resp Time   Trans Rate   Trans     OKAY    Failed   Elap Time   Concurrent
D5       0.26        38.94    4669     4669        0      119.89         9.98
D6       0.36        27.50    3314     3314        0      120.53         9.95
D7       0.92        10.82    1304     1304        0      120.49         9.97

What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests.

This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.

There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.

Look at the columns for average query count and average milliseconds for query time.

The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.

Sheets are here.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Performance as a priority?

Posted by juan_g on October 30, 2008 at 12:20am

Well, it seems most of the high priority performance patches proposed for Drupal 6 were delayed (excepting block cache). If they go into the Drupal 7 release, probably the situation that you are warning about will be better.

In fact, I also hope that static page caching and memory caching options will move from contributed modules to core in future Drupal versions. Until then, they are very little known by most users, even when their addition can be a great improvement over the database-only caching currently in core.

See usage -from the sample of tracked sites- for the main performance modules. On Oct. 12 (Drupal 5.x data for all excepting Cache Router, a rather 6.x module): Javascript Aggregator 3.82%, Block Cache 2.19%, Boost 1.62%, Memcache 0.78%, Advanced cache 0.36%, Cache Router 0.21%, fastpath_fscache 0.21% (the rest even less). The functionalities of the first two performance modules -almost as little known as the others- have been added to core, where surely they are going to be better known by the users.

For now, about your significant benchmarking for Drupal core versions, I think it would be very important to identify the sources of slowness. For this, have you already tested disabling different sets of core modules?

"Say no to bloated core ..."

Posted by kbahey on October 30, 2008 at 2:17am

In fact, I also hope that static page caching and memory caching options will move from contributed modules to core in future Drupal versions.

No.

Core should be about open, extensible, flexible and powerful APIs, not about monolithic features. Users should not have a bloated core that has too many features, while there is more than one way to do a certain thing (like caching).

One site can be using memcache, another can be using boost. If we put them in core, then the unused one will be just added code complexity, and possibly added memory usage.

Let us not advocate bloat and creeping featuritis.

Yes, let us concentrate our efforts on finding where the code is slow and propose ways to make it faster, not more bloated.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Looking for sources of slowness

Posted by juan_g on October 30, 2008 at 6:34pm

Khalid wrote:
> Yes, let us concentrate our efforts on finding where the code is slow

I think your testing and findings seem probably correct. Other testers have also found a progressive slowness (see some graphs at Drupal Performance Benchmarks 5.8 vs 6.3). Looking for its sources seems a necessary but difficult task:

Size of Drupal versions (total and directories)

Version         Total    root    includes  misc    modules  profiles  scripts  sites    themes
Drupal 5.12     2.26 MB  126 KB  538 KB    108 KB  1.22 MB  2.45 KB   5.40 KB  6.36 KB  282 KB
Drupal 6.6      3.31 MB  162 KB  854 KB    183 KB  1.80 MB  6.17 KB   10.2 KB  9.29 KB  320 KB
Drupal 7.x-dev  4.14 MB  165 KB  0.98 MB   185 KB  2.45 MB  11.2 KB   29.2 KB  12.6 KB  320 KB

It's quite a lot of code to review. For instance, are there a few big sources, or many little sources of slowness?

Perhaps it's that larger code often -but not necessarily- means slower code. You know, like Vista is slower than XP, and Windows slower than MS-DOS on the same machine, and more powerful computers are necessary while swiftly pass the years... BTW, my host -a good host- disagrees in many cases with this excuse and says "optimize your ways". We are talking here about one or two years, not decades.

Core modules are more than half of Drupal's size. A potentially enlightening test would be to disable all optional core modules (I think you had some enabled), to see if there is any difference or not in benchmarking across versions.

Also, as said on an above post, important performance patches have been delayed. I don't have enough Drupal experience, but it seems adequate to give higher priority to performance, if possible, as a likely solution for the current trend, which includes Drupal 7.x-dev.

Finally, given the topic covered, in my opinion it would be useful to crosspost this thread with the other performance group, the one that deals with coding standards and performance optimization.

Maybe optimizing menu_tree_page_data() ?

Posted by markus_petrux on October 30, 2008 at 2:25am

http://drupal.org/node/327918

Maybe you could test with blocks disabled, specially the navigation menu?

Comments

Posted by kbahey on October 30, 2008 at 2:41am

I commented on the issue you posted there. Basically if you or someone else has a patch, I will be happy to benchmark it.

As for blocks, it is not a factor in my tests, the reasons is that there is only one block on the D5/D6/D7 sites I tested, and that is the login block, and users are anonymous.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

_drupal_bootstrap_full

Posted by justinrandell (not verified) on October 30, 2008 at 6:38am

i had a look at some xdebug profile dumps in kcachegrind, with the http://drupal.org/node/298600 patch applied.

_drupal_bootstrap_full is still very heavy, and still calls module_load_all. getting rid of the module loading code is on my personal hitlist for the D7 release. we should be able to rely on the registry for this.

It is in now, but ...

Posted by kbahey on October 31, 2008 at 4:27pm

This patch is in core now. Good work.

However, it makes it impossible to upgrade a D6 site to a D7, since it broke another patch that was essential for this (see this http://drupal.org/node/303889).

This means I can't rerun the benchmarks with the SAME data set as I did all along.

Looking forward to all the optimizations you can do, and I will help benchmark them, once update.php works again.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

D7 just got quicker

Posted by justinrandell (not verified) on October 31, 2008 at 3:14am

yay, http://drupal.org/node/298600 is in, so D7 just got quicker, though still clearly slower than D6.

User access check for all menu items necessary always?

Posted by markus_petrux on December 28, 2008 at 9:16pm

I might be wrong, but if not, there's something in the menu system that could reduce the overhead of all page views, at least in D6.

Please, can someone take a look at this? Is it possible to optimize menu_tree_page_data() ?

another snapshot please?

Posted by catch on December 28, 2008 at 11:14pm

Since a few performance patches have gone in since this was done (multiple load and registry query caching for a start), it'd be great to get another snapshot of this. It's almost exactly two months since the last one - which'd mean we could do another just before Drupalcon DC too and would get a nice idea of where things are heading.

Benchmarking Drupal core for 5.x, 6.x, and 7.x: We are getting slower

Comments

registry optimisations

Don't get confused as I did

Spreadsheet

When you say cache, are you

Answers

Prepared statements -vs- query cache?

I believe d7 uses emulated

Same trend

More benchmarks

Performance as a priority?

"Say no to bloated core ..."

Looking for sources of slowness

Maybe optimizing menu_tree_page_data() ?

Comments

_drupal_bootstrap_full

It is in now, but ...

D7 just got quicker

User access check for all menu items necessary always?

another snapshot please?

High performance

Group organizers

New groups

Group notifications

Hot content this week