Posted by Anonymous on November 7, 2008 at 4:22am
hi all,
posted this on the dev list, adding it here in case some in this group don't watch the dev list.
i've been watching the inspiring work to integrate testing patches against HEAD at testing.drupal.org, and wondering if it would be useful/possible/desirable to have benchmarking integrated into d.o. as well?
like the testing stuff, but maybe opt-in?
i'd be willing to do some dev/sysadmin work, and i'd hope there would be enough dev resources, but we'd need some hardware dedicated to it.
thoughts? suggestions? flames?
cheers
justin
Comments
natural extension to testing
To me it seems like a natural extension to automated testing, and running benchmarks is another thing that takes a lot of reviewer time if you need to have content populated - usually doing so in D6, then upgrading to D7 - sometimes you might need a patched and unpatched upgrade if there's a schema change too.
Seems like we'd need to decide on what needs benchmarking, here's some to start off.
I reckon we need to benchmark both lightly and heavily loaded Drupal sites:
Small amount of content, few modules.
Small amount of content, lots of modules.
Large amount of content, few modules.
Large amount of content, lots of modules (i.e. all core modules enabled).
Then we'd need to be able to benchmark /node /node/1 taxonomy/term/1 - sometimes other pages like /forum which a patch might specifically affect.
We also need to test at least a few configurations:
query cache on
query cache off (some patches allow the query cache to be used)
APC (and friends) on and off.
PostgreSQL
And the capacity to add things like lighty, SQLite at a later date if resources are available.
Then AB/Siege - anonymous and authenticated users etc.
Seems like it'd be easiest to have a VM/Slave for each of these and pass it off to them. What'd be amazing, since this'd be opt-in, would be a system (and even interface) like http://browsershots.org/ - where you pick which setups you want to test with, then it sends off the queue, and each posts it's results back to the dispatch site when available. Something along those lines anyway.
Either way it's a great idea, and I have a feeling that getting hold of some hardware would be the least of our problems in setting it up.