automated performance testing

This proposal is about a performance testing suite ( for drupal, obviously ).
The idea is mainly built around this issue: #638078.
In this issue an automated performance testing suite is suggested.
When I stumbled on this a few months back, I thought this was a great idea.

I think I shouldn't elaborate on the advantages of having a decent testing suite.
The integration of simpletest into drupal made clear that test driven development is a big win for drupal.
However, the current tests are mostly meant for testing if everything is functioning as it's meant to do.
But an application like drupal is more than just features,in my opinion performance is evenly important.
The current workflow is that features are getting added and only afterwards the impact on performance
is looked at. Often at this stage it's hard to reverse this.

We can see that drupal 7 is slower than drupal 6 ( #615822 ). That's nothing to be surprised about, as a lot of great features have been added.
But in the future we should certainly strive to keep drupal as fast as possible. Having a decent testing suite for perfomance
would allow to incorporate performance much earlier in the development cycle, hopefully leading to a faster drupal.

Goals/deliverables

What I think should be delivered by this project.
I've tried to be as complete as possible but additions are certainly welcome.

• developing an easily useable and extendable test suite
• the testing suite should produce easily interpretable results ( e.g. graphs )
• devising some useful/representative configurations and tests
• it should be possible to extend this, so that in the future the perfomance impact of modules can
  be measured
• raising awareness about performance
• measuring the performance of the cache ( not yet sure about this one )

How?

Obviously,this is the hard part :).
It's not easy to get representative measurements for a versatile application, like drupal.
A lot depends on the chosen configuration settings, the used data set, ... .
So quite some time should be spend on devising the default tests ( others could be added later )
This should be done in cooperation with more experienced members of the community. People
who know the characteristics of a real website.
To start with I would focus on 2 kinds of websites:

• social networks ( like d.o )
• news sites

These 2 common types of websites have obviously a different arrival pattern and thus should be handled
separately.

For the implementation, I think Jmeter is the way to go. But I might be missing some other choices.
If you're asking for a detailed architecture, I'm afraid I'll have to admit that this is something I don't have a clear view on.

Some words about myself

My name is Joris Geessels.
I'm in my 1st year master computerscience at the UGent ( Belgium )
My profile on d.o tells me my first encounter with drupal took place
about 2 and a half years ago. I've built some small websites and
developed a few modules for a project at school. In this context I also
touched jmeter but you can't really classify this as 'experience'.
I've quite some experience with java, which can come in
handy when working with jmeter.

However...

I've been hesitating to submit this proposal. Initially I didn't plan to participate in gsoc and do a 'normal' internship instead.
There's a simple reason for this, coding starts at May 23 and about a week later examinations start in Belgium.
Obviously this means that the first month I would not be able to do any work. Which I admit, is a lot.
Nonetheless there are examples of Belgian students who have sucessfuly completed gsoc. With tedious planning and a good understanding between the student and mentor, it should be possible.

I understand this greatly reduces my chances to get accepted. But still I would like to participate and thought that just trying to apply wouldn't hurt :)

Would love to hear your thoughts!