Drupal and Jmeter
In my never-ending quest to run larger and more complex drupal stacks for more and more users, I'm starting to hit the wall in what I'm able to accomplish with good old Siege, which has been my command-line tool of choice for benchmarking and performance testing for the past couple years. In particular, it breaks down too often in high-load simulations, and doesn't allow for any complex multi-threaded testing, making it very difficult to model near-reality user scenarios like "10 logged in users + 100 anons".
Lately, I've been getting into Jmeter, which has a lot more bells and whistles -- including a GUI! -- and which I think can offer a lot to Drupal developers. However, their basic web test plan barely scratches the surface of what's possible. With the right configuration, jmeter can effectively simulate complete user-behavior patterns like logging in, posting comments, etc.
I'm just getting started, but am curious if people "out there" are already way ahead of me, or if not if folks are interested in seeing the results of my testing work?
Assuming this isn't all already done, here would be my goals:
- Develop some great "starter" test plans for drupal sites.
- Develop good template workflow components like "login" and so forth.
- Develop a Drupal module to dynamically generate site-specifc XML. (holy grail!)
Interest? Feedback? Ideas? I'm all ears.
Interesting
I found it very interesting. Hope to see updatedes soon :-)
This is something I have
This is something I have also been interested in developing. I think some standard test plans for Drupal core are really vital to be able to evaluate performance patches targeted at improving performance for sites with heavy authenticated user activity (which are really the only ones worth pursuing in depth, since anonymous traffic is pretty trivial to scale already). I was thinking an interesting end point would be an Amazon EC2 image (which I hope would be standard enough that you can compare results tested anywhere) that you can point at a patch and it would respond with performance results with and without the patch (a little like the testing service, but it would be requested for specific patches only).
regular core benchmarks
On-demand performance testing in a standard environment (more or less) is really, really needed. A lot of patches are committed without any benchmarking - especially those which aren't performance patches as such but might still have a big impact - and most benchmarks that do get done are on random core-developer's laptops.
We could do it in two stages:
First: a suite of tests we can run on HEAD, once a week, and then track progress over time to catch any serious trends.
Second: The same suite doing before/after patch comparisons.
The latter would require some integration with project module, but the first could probably be scripted independently of d.o. infrastructure and would catch the most serious regressions very quickly - we could also run some on Drupal 6 and a couple of historical snapshots of HEAD to have an immediate longer term comparison.
With EC2
As jacob replied below, getting something like this set up for on-demand testing on ec2 wouldn't be very hard, and would make for a solid test case. Basically:
1) Spin up a small instance
2) Load HEAD
3) Run devel generate (and whatever other config)
4) Trigger jmeter benchmarking suite
I'm also doing a lot of work lately w/clouds and deployments, so this sort of thing is pretty interesting overall. I'm also very interested in stronger object-level caching for Drupal 7, and plan on working on that over the summer. This all sort of fits together, so hopefully we can drive it forward!
http://www.chapterthree.com | http://www.outlandishjosh.com
regular core benchmarks
On-demand performance testing in a standard environment (more or less) is really, really needed. A lot of patches are committed without any benchmarking - especially those which aren't performance patches as such but might still have a big impact - and most benchmarks that do get done are on random core-developer's laptops.
We could do it in two stages:
First: a suite of tests we can run on HEAD, once a week, and then track progress over time to catch any serious trends.
Second: The same suite doing before/after patch comparisons.
The latter would require some integration with project module, but the first could probably be scripted independently of d.o. infrastructure and would catch the most serious regressions very quickly - we could also run some on Drupal 6 and a couple of historical snapshots of HEAD to have an immediate longer term comparison.
Economist.com
On the Economist.com Drupal build we are still deciding but we are most likely to integrate JMeter, SimpleTest, Selenium and have Cruise Control drive them. Let's share war stories.
Wow
That sounds like a "perfect storm" of test suites. I'm not as familiar with Selenium or Cruise Control, but it sounds pretty excellent.
I think a good next step is to start working out some re-usable jMeter testplan components, and possibly a way to auto-generate lists of URLS to hit directly from Drupal. I've done the same for Seige before, with good results. jMeter's XML is slightly more complex, but also offers a lot more options.
http://www.chapterthree.com | http://www.outlandishjosh.com
Also on Ec2
We are doing all of this on EC2. When we get it up I/we will provide all of our config, processes, docs e.t.c. I think we've switched from Cruise Control to Hudson though.
jmeter at Acquia Search
I implemented jmeter as an AMI + Ruby provisioning scripts using RightScale. Unfortunately, I bit hard to share at the moment, but basically, it would provision n servers in EC2, variable instances of solr on them, and configure them as I liked. Then, it runs through a series of test plans, running jmeter headless in the cloud and then aggregate the results down and chart them. Then, it would run in a loop, up some params (like loops and threads) and try again. If I ever get a chance to abstract any of this, I'd be happy to contrib it, but in the meantime, please pick my brain if required.
Best,
Jacob
Scripting jmeter
I'm still just getting started with the ability to script jmeter and use variables. Do you have any tips on how to architect your loops which increase parameters in a sane way? Any chance you can share some testplan components (e.g. XML of some sub-part) with us?
http://www.chapterthree.com | http://www.outlandishjosh.com
Hey Josh, et all... I've
Hey Josh, et all...
I've attached my testplan, although I doubt it will be very obvious what is going on...
Basically, Acquia Search hosts many "clients" or indexes. So on a given cluster, we may host 40 clients or 500 depending on how big they are. In my tests, I have some templates. For instance, to test what I call a small site, I actually use Dries's index. I duplicate it 50 or 100 or 600 times and call it drupal-small-1, -2,-3, etc...
I create a file called clients.csv that looks like this:
/solr/drupal-small-1
/solr/drupal-small-2
/solr/drupal-small-3
etc...
I also have a URL file of searches which will work it looks like this:
/select?q=foo
/select?q=bar
In my actual test, I want to create URLS that look like this and hit them:
/solr/drupal-small-1/select?q=bar
/solr/drupal-small-2/select?q=bar
So the load is distributed among all the different indexes or clients, and the queries are also distributed, this will closely resemble real-world operation.
I'll try to explain a little:
I have 2 thread groups. This is so I can test an array of small sites together with an array of larger sites. Don't worry about this, they are both the same. In each group, I have a loop controller set to loop once. This sounds stupid, it is. I can't remember exactly, but I believe that the point of this is is to get the only once controller to work. the only once controller loads the CSV file of urls to hit from the CSV data Set Config action. This file populates the $dynurl variable.
Then we go into another loop, which will, for each url defined in the CSV dataSet Config, loop through each of the clients I want to hit. This populates the $client_name variable. Finally, I have a HTTP Request Sampler which calls:
${client_name}/${dynurl} as the URL to hit.
Notice that almost everything you see is a parameter, i.e. ${__P(var_name)}
This is so I can run jmeter headless by passing in params. I then get the results out, and I run them through a script I found here:
http://blogs.atlassian.com/developer/2008/10/performance_testing_with_jm...
Also a good resource on this stuff.
Gives me some wicked looking charts showing aggregate performance. Because it is all scripted, I then run the thing in a loop, and specify more clients, or more reqs per minute or whatever.
Not the same as Drupal testing, but perhaps useful.
Best,
Jacob
Logging in users to Drupal with Jmeter
Just got done scripting out some tests using logged in users accounts.
Stumbled a bit getting users to actually log in to Drupal because each login form submission requires a unique form_build_id value that is generated each time the page is loaded.
This is specific to Drupal and took me a bit of hunting to figure it out.
Posting it here in case someone else is searching for help.
To get the unique value of the form_build_id, I used the Regular Expression Extractor
Reference Name: form_build_id
Regular Expression: id="(form-.{32})"
Template: $1$
Match No.: 1.0
Default Value: NOT FOUND
Then for the value of the form post use "${form_build_id}" as the value of form_build_id.
DLC Solutions
On demand load testing with JMeter in the Cloud
Hi.
My name is Alon. I am as well a Drupal professional. I want to introduce to you a new service, I've recently developed. This service allows uploading and running JMeter test-scripts leveraging EC2 resources. It is very simple. You just upload the test script and choose the amount of JMeterEngines to use. Then you work on all the engines as if they were running off your PC. There is no setup involved and no installation. You pay according to consumed for computing hours.
I find this community very important as all of the projects I am involved in are related to Drupal and many answers to my questions are found here.
So, if you wish to incorporate special features related to Drupal, let me know.
The site can be found at: http://www.cloud-intelligence.com/applications/jmeter/about
you can send your requests/suggestions to info@cloud-intelligence.com
Alon.