In my never-ending quest to run larger and more complex drupal stacks for more and more users, I'm starting to hit the wall in what I'm able to accomplish with good old Siege, which has been my command-line tool of choice for benchmarking and performance testing for the past couple years. In particular, it breaks down too often in high-load simulations, and doesn't allow for any complex multi-threaded testing, making it very difficult to model near-reality user scenarios like "10 logged in users + 100 anons".
Lately, I've been getting into Jmeter, which has a lot more bells and whistles -- including a GUI! -- and which I think can offer a lot to Drupal developers. However, their basic web test plan barely scratches the surface of what's possible. With the right configuration, jmeter can effectively simulate complete user-behavior patterns like logging in, posting comments, etc.
I'm just getting started, but am curious if people "out there" are already way ahead of me, or if not if folks are interested in seeing the results of my testing work?
Assuming this isn't all already done, here would be my goals:
- Develop some great "starter" test plans for drupal sites.
- Develop good template workflow components like "login" and so forth.
- Develop a Drupal module to dynamically generate site-specifc XML. (holy grail!)
Interest? Feedback? Ideas? I'm all ears.
Comments
Interesting
I found it very interesting. Hope to see updatedes soon :-)
This is something I have
This is something I have also been interested in developing. I think some standard test plans for Drupal core are really vital to be able to evaluate performance patches targeted at improving performance for sites with heavy authenticated user activity (which are really the only ones worth pursuing in depth, since anonymous traffic is pretty trivial to scale already). I was thinking an interesting end point would be an Amazon EC2 image (which I hope would be standard enough that you can compare results tested anywhere) that you can point at a patch and it would respond with performance results with and without the patch (a little like the testing service, but it would be requested for specific patches only).
regular core benchmarks
On-demand performance testing in a standard environment (more or less) is really, really needed. A lot of patches are committed without any benchmarking - especially those which aren't performance patches as such but might still have a big impact - and most benchmarks that do get done are on random core-developer's laptops.
We could do it in two stages:
First: a suite of tests we can run on HEAD, once a week, and then track progress over time to catch any serious trends.
Second: The same suite doing before/after patch comparisons.
The latter would require some integration with project module, but the first could probably be scripted independently of d.o. infrastructure and would catch the most serious regressions very quickly - we could also run some on Drupal 6 and a couple of historical snapshots of HEAD to have an immediate longer term comparison.
With EC2
As jacob replied below, getting something like this set up for on-demand testing on ec2 wouldn't be very hard, and would make for a solid test case. Basically:
1) Spin up a small instance
2) Load HEAD
3) Run devel generate (and whatever other config)
4) Trigger jmeter benchmarking suite
I'm also doing a lot of work lately w/clouds and deployments, so this sort of thing is pretty interesting overall. I'm also very interested in stronger object-level caching for Drupal 7, and plan on working on that over the summer. This all sort of fits together, so hopefully we can drive it forward!
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
regular core benchmarks
On-demand performance testing in a standard environment (more or less) is really, really needed. A lot of patches are committed without any benchmarking - especially those which aren't performance patches as such but might still have a big impact - and most benchmarks that do get done are on random core-developer's laptops.
We could do it in two stages:
First: a suite of tests we can run on HEAD, once a week, and then track progress over time to catch any serious trends.
Second: The same suite doing before/after patch comparisons.
The latter would require some integration with project module, but the first could probably be scripted independently of d.o. infrastructure and would catch the most serious regressions very quickly - we could also run some on Drupal 6 and a couple of historical snapshots of HEAD to have an immediate longer term comparison.
Economist.com
On the Economist.com Drupal build we are still deciding but we are most likely to integrate JMeter, SimpleTest, Selenium and have Cruise Control drive them. Let's share war stories.
Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.
Wow
That sounds like a "perfect storm" of test suites. I'm not as familiar with Selenium or Cruise Control, but it sounds pretty excellent.
I think a good next step is to start working out some re-usable jMeter testplan components, and possibly a way to auto-generate lists of URLS to hit directly from Drupal. I've done the same for Seige before, with good results. jMeter's XML is slightly more complex, but also offers a lot more options.
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Also on Ec2
We are doing all of this on EC2. When we get it up I/we will provide all of our config, processes, docs e.t.c. I think we've switched from Cruise Control to Hudson though.
Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.
jmeter at Acquia Search
I implemented jmeter as an AMI + Ruby provisioning scripts using RightScale. Unfortunately, I bit hard to share at the moment, but basically, it would provision n servers in EC2, variable instances of solr on them, and configure them as I liked. Then, it runs through a series of test plans, running jmeter headless in the cloud and then aggregate the results down and chart them. Then, it would run in a loop, up some params (like loops and threads) and try again. If I ever get a chance to abstract any of this, I'd be happy to contrib it, but in the meantime, please pick my brain if required.
Best,
Jacob
Scripting jmeter
I'm still just getting started with the ability to script jmeter and use variables. Do you have any tips on how to architect your loops which increase parameters in a sane way? Any chance you can share some testplan components (e.g. XML of some sub-part) with us?
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Hey Josh, et all... I've
Hey Josh, et all...
I've attached my testplan, although I doubt it will be very obvious what is going on...
Basically, Acquia Search hosts many "clients" or indexes. So on a given cluster, we may host 40 clients or 500 depending on how big they are. In my tests, I have some templates. For instance, to test what I call a small site, I actually use Dries's index. I duplicate it 50 or 100 or 600 times and call it drupal-small-1, -2,-3, etc...
I create a file called clients.csv that looks like this:
/solr/drupal-small-1
/solr/drupal-small-2
/solr/drupal-small-3
etc...
I also have a URL file of searches which will work it looks like this:
/select?q=foo
/select?q=bar
In my actual test, I want to create URLS that look like this and hit them:
/solr/drupal-small-1/select?q=bar
/solr/drupal-small-2/select?q=bar
So the load is distributed among all the different indexes or clients, and the queries are also distributed, this will closely resemble real-world operation.
I'll try to explain a little:
I have 2 thread groups. This is so I can test an array of small sites together with an array of larger sites. Don't worry about this, they are both the same. In each group, I have a loop controller set to loop once. This sounds stupid, it is. I can't remember exactly, but I believe that the point of this is is to get the only once controller to work. the only once controller loads the CSV file of urls to hit from the CSV data Set Config action. This file populates the $dynurl variable.
Then we go into another loop, which will, for each url defined in the CSV dataSet Config, loop through each of the clients I want to hit. This populates the $client_name variable. Finally, I have a HTTP Request Sampler which calls:
${client_name}/${dynurl} as the URL to hit.
Notice that almost everything you see is a parameter, i.e. ${__P(var_name)}
This is so I can run jmeter headless by passing in params. I then get the results out, and I run them through a script I found here:
http://blogs.atlassian.com/developer/2008/10/performance_testing_with_jm...
Also a good resource on this stuff.
Gives me some wicked looking charts showing aggregate performance. Because it is all scripted, I then run the thing in a loop, and specify more clients, or more reqs per minute or whatever.
Not the same as Drupal testing, but perhaps useful.
Best,
Jacob
Logging in users to Drupal with Jmeter
Just got done scripting out some tests using logged in users accounts.
Stumbled a bit getting users to actually log in to Drupal because each login form submission requires a unique form_build_id value that is generated each time the page is loaded.
This is specific to Drupal and took me a bit of hunting to figure it out.
Posting it here in case someone else is searching for help.
To get the unique value of the form_build_id, I used the Regular Expression Extractor
Reference Name: form_build_id
Regular Expression: id="(form-.{32})"
Template: $1$
Match No.: 1.0
Default Value: NOT FOUND
Then for the value of the form post use "${form_build_id}" as the value of form_build_id.
DLC Solutions
DLC Solutions
EthosCE
disable form_build_id for JMeter test
Recently I also did some functional test and stress test for my friend's new site, I encountered the same situation : the form_build_id changed each time and this caused JMeter replay failed. As each form will face to the same situation and I need to test a set of forms, instead of retrieve hidden form_build_id and put as JMeter parameters, I disabled the form_build_id validate process. The validation happens in code Drupal-x.xx/includes/form.inc, in function drupal_validate_form, in my installation, it's from line 578 to 583. I just comment off these lines while replay JMeter test cases.
A further more, I create a patch for this purpose, and patch form.inc while replaying JMeter test cases and then un-patch it while test finished.
In fact, I think there should be some place to configure this. Such as put this as a check box on admin page, enable or disable form-build-id. Then will need to perform auto test, simply disable form-build-id...
Not getting form_build_id substitution
I have followed this article: Using JMeter to test performance of Drupal with authenticated users and your post was a sanity check for my own test. But I seem to be doing what you and the article describe but when I look at the POST data in JMeter's View Results Tree for a login I see that while the name & user are populated correctly the form_build_id is not, i.e.:
POST data:
name=user1&pass=user1&form_build_id=${form_build_id}&form_id=user_login&op=Log+in
In the JMeter HTTP (POST) Request I am indeed sending the form_build_id value of ${form_build_id}
One thing I noticed is in the article is that in the writer's screencaps of JMeter there is no "Apply to:" section for the Regular Expression Extractor. In the version of JMeter that I'm using (v2.4) Apply to: defaults to "Main sample only".
I have also tried both NO DEFAULT & NOT FOUND for Default Value. What does the Template parameter reference (i.e. $1$)? The value of the Regular Expression is indeed: id=”(form-.{32})”
Screencap
Any suggestions you might have are most welcome.
Cheers
On demand load testing with JMeter in the Cloud
Hi.
My name is Alon. I am as well a Drupal professional. I want to introduce to you a new service, I've recently developed. This service allows uploading and running JMeter test-scripts leveraging EC2 resources. It is very simple. You just upload the test script and choose the amount of JMeterEngines to use. Then you work on all the engines as if they were running off your PC. There is no setup involved and no installation. You pay according to consumed for computing hours.
I find this community very important as all of the projects I am involved in are related to Drupal and many answers to my questions are found here.
So, if you wish to incorporate special features related to Drupal, let me know.
The site can be found at: http://www.cloud-intelligence.com/applications/jmeter/about
you can send your requests/suggestions to info@cloud-intelligence.com
Alon.
Figured it out
The words in the article "from the previous page" are quite important ;-)
In other words you don't see which request the Regular Expression Extractor is applied to in the article but it needs to be applied to the GET request before the POST request. Seems obvious now but I am not that familiar with Drupal's innards.
Cheers
News in Drupal 7
Hi,
My name is Cyril PIERRE de GEYER from France. I am not a Drupal professional but I'm quite good in PHP. Actually I'm playing with Drupal Web stressing.
I have read all your comments and I was really happy to find this place, I was wondering how to insert contents and comments with Jmeter on Drupal.
I have a single comment, with Drupal 7 there is little change on the regular expression you have to create in Jmeter to get the "form_build_id" and "form_token". I am using theses regular expressions :
value="(form-.{32})
name="form_token" value="(.{43})
In my Drupal 7 version I don't have the id="form-XXXXXX" in my template so you just need to catch the one which is inside the concerned value.
Cyril PIERRE de GEYER
Co auteur du livre "PHP 5 avancé"
Twitter: cyrilpdg
jmeter & drupal 7
I don't fully get what your problem is, I assume you just want to comment?
In this case you might take a look at http://github.com/jolos/drupal-jmeter/blob/master/drupal7.jmx . It's a testplan I made a few months back. It's possible that things have changed in the meantime so I can't ensure that it still works. It also contains a lot of variables to make it more automated, you probably don't need this functionality. You can however use it as an example to build your own testplan. You'll notice that I use an xpath extractor instead of regex, imo that's more suitable for html.
In short the steps to submit a comment are :
- login
- Fetch the node on which you want to comment
- use an xpath extractor to extract form_build_id ( //form[@id="comment-form"]/descendant::input[@name="form_build_id"]/@value )
- extract form_token in the same way ( just change @name to form_token )
- extract form_id
- finally submit an HTTP request with form_build_id,form_token and form_id as parameters ( and some other stuff )
The steps for submitting nodes are similar
Hopefully this helped you out
Hey Cyril, thanks for the
Hey Cyril,
thanks for the input. Just figured out, the quotes need to be escaped in your example:
value=\"(form-.{43})\"
That worked well for me.
Regards, Nils
I've used an Xpath Extractor
I've used an Xpath Extractor post-processor instead.
using the following xpath will get the form_build_id
Reference name: form_build_id
Xpath query: //input[@name='form_build_id']/@value
Default Value: NOT FOUND
works well.
Thanks Cyril and Fonsorello,
Thanks Cyril and Fonsorello, I got it sorted out for Drupal 7. In my case I am testing a Drupal Commerce site and wanted to test adding a product to the cart.
I have an HTTP Request Sampler going to the product page and then clicking on the Add To Cart button. On that Sampler I needed to a Regular Expression Extractor, one for form_build_id. (I thought I would have to add one for form_token, but I did not, Drupal accepted the post without it.
Reference Name: form_build_id
Regular Express: value="(form-.{43})"
Template: $1$
Match No. (empty)
Default Value: (empty)
To be sure you're getting the variable set, you can add a Debug Sampler after this step with a Views Results Tree as a child.