benchmarking

While Mercury has previously demonstrated the raw power of Varnish to radically accelerate the delivery of cached pages with Pressflow, that's only part of the battle for high performance Drupal. Cached pages need to be generated at least once, and logged in users will be bypassing Varnish when making requests. While it's nice to know you're covered for massive traffic spikes, what about baseline load?

Well, we want to answer that question too. Keep in mind that depending on your module/theme stack, your mileage may vary. A lot. Performance tuning a complex Drupal site for logged-in traffic requires sustained diligence at the application level — code profiling, slow query log review, innovative app-specific caches, etc — as well as hardware and system optimization. However, to set some baseline expectations, we used Jacob Singh's greate starter test suite and checked out a Mercury install on a 512MB Slicehost VPS. The results are quite encouraging.

While working through some issues this weekend and preparing another blog post, I finally got around to doing some comprehensive benchmarking on Project Mercury. In the process, I discovered that the first bottleneck I hit running tests from my desktop was the local (last-mile) internet connection, so I switched to running the tests from another EC2 instance. This means that network is not a factor in my results, giving us a real sense of the raw power behind this stack.

For all these tests, I used the Mercury Alpha4 release on a small ec2 instance, loading a staging copy of Mission Bicycles, which is a good "heavy" example in that it has a lot of modules loaded, including Ubercart and Panels. My goal was to measure throughput and response times under various caching configurations, angling for the best results in terms of pages served per second, and delivery time.

I started by cutting things all the way back to nothing, and then added each layer of the caching infrastructure, running benchmarks at each point. The results are quite eye-opening. Can you say 2000+ requests per second? Read on for the full story.

In my never-ending quest to run larger and more complex drupal stacks for more and more users, I'm starting to hit the wall in what I'm able to accomplish with good old Siege, which has been my command-line tool of choice for benchmarking and performance testing for the past couple years. In particular, it breaks down too often in high-load simulations, and doesn't allow for any complex multi-threaded testing, making it very difficult to model near-reality user scenarios like "10 logged in users + 100 anons".

Lately, I've been getting into Jmeter, which has a lot more bells and whistles -- including a GUI! -- and which I think can offer a lot to Drupal developers. However, their basic web test plan barely scratches the surface of what's possible. With the right configuration, jmeter can effectively simulate complete user-behavior patterns like logging in, posting comments, etc.

I'm just getting started, but am curious if people "out there" are already way ahead of me, or if not if folks are interested in seeing the results of my testing work?

hi all,

posted this on the dev list, adding it here in case some in this group don't watch the dev list.

i've been watching the inspiring work to integrate testing patches against HEAD at testing.drupal.org, and wondering if it would be useful/possible/desirable to have benchmarking integrated into d.o. as well?

like the testing stuff, but maybe opt-in?

i'd be willing to do some dev/sysadmin work, and i'd hope there would be enough dev resources, but we'd need some hardware dedicated to it.

thoughts? suggestions? flames?

cheers
justin

We conducted some benchmarks for Drupal 5.12, Drupal 6.6 and Drupal 7.x (Oct 24 checkout).

We used the same data set for all 3 instances:

Users 4,950
Nodes 5,000
Comments 20,000
Vocabularies 10
Terms 1,000

And same test parameters: 10 concurrent users, 2 minutes stress test on 30 individual nodes, and the home page.

OpenCraft hosted a 3.5 hour seminar that I presented on Drupal performance tuning and optimization. You can find the slides from the presentation there, which can be useful to some of you.

Hi all. I've been working on a new project called Advcache (Advanced Cache) which is a package of caching patches (and a controller module) for Drupal 5. It allows you to cache many things which don't get cached normally, and is intended to increase the performance of Drupal sites, especially for authenticated users. Current caching targets include nodes, comments, taxonomy (trees, terms, vocabularies and node-term bindings), paths, search queries, and forums.

Hi there, I have some interest in building a Drupal becnhmarking suite that can be used for testing the impact of newly proposed patches on our beloved content managemanet framework.

One thing I would be really interested in seeing is a benchmark suite run on the current drupal.org installation, and then again once it's upgraded to 5.0. Is this something that could be done or is being planned? It would be an underestimate of performance gains to the extent that certain performance patches are applied directly to d.o when fixed in core.

Performance upgrades from the CHANGELOG include:

Hi all,

Here are a few sketchy to interpret, yet interesting tests results benchmarking Prototype v. jQuery, by Claudio Cicali. The blogger's conclusions: Despite rumors on the internet, jQuery is not slower than Prototype.

ajaxian.com has a few more comments on these tests.