Posted by kbahey on October 27, 2008 at 12:45am
We conducted some benchmarks for Drupal 5.12, Drupal 6.6 and Drupal 7.x (Oct 24 checkout).
We used the same data set for all 3 instances:
Users 4,950
Nodes 5,000
Comments 20,000
Vocabularies 10
Terms 1,000
And same test parameters: 10 concurrent users, 2 minutes stress test on 30 individual nodes, and the home page.
Description, Trans, Elap Time, Resp Time, Trans Rate, Concurrent Drupal 5; No APC; No cache, 1994, 120.25, 0.60, 16.58, 9.97 Drupal 6; No APC; No cache, 1787, 120.36, 0.67, 14.85, 9.96 Drupal 7; No APC; No cache, 1255, 120.47, 0.96, 10.42, 9.95 Drupal 5; No APC; w/ cache, 19481, 119.78, 0.06, 162.64, 9.98 Drupal 6; No APC; w/ cache, 15589, 120.41, 0.08, 129.47, 9.98 Drupal 7; No APC; w/ cache, 11418, 120.26, 0.11, 94.94, 9.98 Drupal 5; w/ APC; No cache, 6537, 120.42, 0.18, 54.29, 9.99 Drupal 6; w/ APC; No cache, 4706, 120.56, 0.26, 39.03, 9.97 Drupal 7; w/ APC; No cache, 2503, 120.58, 0.48, 20.76, 9.95 Drupal 5; w/ APC; w/ cache, 50698, 119.77, 0.02, 423.29, 9.96 Drupal 6; w/ APC; w/ cache, 46520, 120.49, 0.03, 386.09, 9.96 Drupal 7; w/ APC; w/ cache, 34359, 120.05, 0.03, 286.21, 9.98
* Description: Description of the test.
* Trans: Number of HTTP requests completed.
* Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
* Resp Time: Response time per requests, in seconds across all requests.
* Trans Rate: Number of transactions per second. The higher the better.
* Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.
More here: Performance benchmarking Drupal 5.12, Drupal 6.6 and Drupal 7.x we are getting slower.
Things should have been better with the registry.
We need to a) identify the source(s) of this slowness, and b) try to remedy it before Drupal 7.x goes out.
Comments
registry optimisations
thanks for the benchmarks :-)
right now, the registry runs a whole bunch of queries that can be eliminated with some optimisation.
here's the first thing that needs to go in:
http://drupal.org/node/298600
after that, we can cut some more with this:
http://drupal.org/node/325665
Don't get confused as I did
Don't get confused as I did with the column titles not lining up with the values - thanks for this kbahey, pretty significant decrease there :(
Spreadsheet
Yeah, it can be confusing.
To remedy that, and for people to crunch their own numbers, I attached an Open Office spreadsheet and a CSV file to the article linked above.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
When you say cache, are you
Answers
Yes, page caching, the variable in Drupal called 'cache'.
No.
Just one, the Login block.
Anonymous.
Yes. Same physical box, same MySQL instance, benchmarks ran back to back.
Not much. This is a development server, but it is all the same for all tests.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Prepared statements -vs- query cache?
Hi kbahey,
In your article: "They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4."
In the MySQL manual: "Before MySQL 5.1.17, prepared statements do not use the query cache."
http://dev.mysql.com/doc/refman/5.1/en/query-cache-how.html
Maybe you could try updating MySQL and see how numbers change for D7? ...or try again with MySQL 5.0, but disabling MySQL Query Cache?
I believe d7 uses emulated
I believe d7 uses emulated prepared statements, not mysql native prepared statements? Although perhaps native prepared statements could be enabled on newer versions of mysql..
Same trend
I see the same trend with MySQL Query cache disabled (query_cache_type = 0 in my.cnf).
Here are the same tests with query cache re-enabled.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
More benchmarks
Here are some more data points to help us analyze the issue further.
This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.
Resp Time Trans Rate Trans OKAY Failed Elap Time Concurrent
D5 0.26 38.94 4669 4669 0 119.89 9.98
D6 0.36 27.50 3314 3314 0 120.53 9.95
D7 0.92 10.82 1304 1304 0 120.49 9.97
What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests.
This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.
There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.
Look at the columns for average query count and average milliseconds for query time.
The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.
Sheets are here.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Performance as a priority?
Well, it seems most of the high priority performance patches proposed for Drupal 6 were delayed (excepting block cache). If they go into the Drupal 7 release, probably the situation that you are warning about will be better.
In fact, I also hope that static page caching and memory caching options will move from contributed modules to core in future Drupal versions. Until then, they are very little known by most users, even when their addition can be a great improvement over the database-only caching currently in core.
See usage -from the sample of tracked sites- for the main performance modules. On Oct. 12 (Drupal 5.x data for all excepting Cache Router, a rather 6.x module): Javascript Aggregator 3.82%, Block Cache 2.19%, Boost 1.62%, Memcache 0.78%, Advanced cache 0.36%, Cache Router 0.21%, fastpath_fscache 0.21% (the rest even less). The functionalities of the first two performance modules -almost as little known as the others- have been added to core, where surely they are going to be better known by the users.
For now, about your significant benchmarking for Drupal core versions, I think it would be very important to identify the sources of slowness. For this, have you already tested disabling different sets of core modules?
"Say no to bloated core ..."
No.
Core should be about open, extensible, flexible and powerful APIs, not about monolithic features. Users should not have a bloated core that has too many features, while there is more than one way to do a certain thing (like caching).
One site can be using memcache, another can be using boost. If we put them in core, then the unused one will be just added code complexity, and possibly added memory usage.
Let us not advocate bloat and creeping featuritis.
Yes, let us concentrate our efforts on finding where the code is slow and propose ways to make it faster, not more bloated.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Looking for sources of slowness
Khalid wrote:
> Yes, let us concentrate our efforts on finding where the code is slow
I think your testing and findings seem probably correct. Other testers have also found a progressive slowness (see some graphs at Drupal Performance Benchmarks 5.8 vs 6.3). Looking for its sources seems a necessary but difficult task:
It's quite a lot of code to review. For instance, are there a few big sources, or many little sources of slowness?
Perhaps it's that larger code often -but not necessarily- means slower code. You know, like Vista is slower than XP, and Windows slower than MS-DOS on the same machine, and more powerful computers are necessary while swiftly pass the years... BTW, my host -a good host- disagrees in many cases with this excuse and says "optimize your ways". We are talking here about one or two years, not decades.
Core modules are more than half of Drupal's size. A potentially enlightening test would be to disable all optional core modules (I think you had some enabled), to see if there is any difference or not in benchmarking across versions.
Also, as said on an above post, important performance patches have been delayed. I don't have enough Drupal experience, but it seems adequate to give higher priority to performance, if possible, as a likely solution for the current trend, which includes Drupal 7.x-dev.
Finally, given the topic covered, in my opinion it would be useful to crosspost this thread with the other performance group, the one that deals with coding standards and performance optimization.
Maybe optimizing menu_tree_page_data() ?
http://drupal.org/node/327918
Maybe you could test with blocks disabled, specially the navigation menu?
Comments
I commented on the issue you posted there. Basically if you or someone else has a patch, I will be happy to benchmark it.
As for blocks, it is not a factor in my tests, the reasons is that there is only one block on the D5/D6/D7 sites I tested, and that is the login block, and users are anonymous.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
_drupal_bootstrap_full
i had a look at some xdebug profile dumps in kcachegrind, with the http://drupal.org/node/298600 patch applied.
_drupal_bootstrap_full
is still very heavy, and still callsmodule_load_all
. getting rid of the module loading code is on my personal hitlist for the D7 release. we should be able to rely on the registry for this.It is in now, but ...
This patch is in core now. Good work.
However, it makes it impossible to upgrade a D6 site to a D7, since it broke another patch that was essential for this (see this http://drupal.org/node/303889).
This means I can't rerun the benchmarks with the SAME data set as I did all along.
Looking forward to all the optimizations you can do, and I will help benchmark them, once update.php works again.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
D7 just got quicker
yay, http://drupal.org/node/298600 is in, so D7 just got quicker, though still clearly slower than D6.
User access check for all menu items necessary always?
I might be wrong, but if not, there's something in the menu system that could reduce the overhead of all page views, at least in D6.
Please, can someone take a look at this? Is it possible to optimize menu_tree_page_data() ?
another snapshot please?
Since a few performance patches have gone in since this was done (multiple load and registry query caching for a start), it'd be great to get another snapshot of this. It's almost exactly two months since the last one - which'd mean we could do another just before Drupalcon DC too and would get a nice idea of where things are heading.