It seems that with the growing Drupal user base, the problems with CMS (flexibility) and speed (non functional requirement) seems to be growing. We made a SAAS where one can install a small module (http://drupal.org/project/paas) and the buy a service where we scan your site. We do this on 100+ metrics and in an academic way (lots of "r" graphs, no gradients :-)
We measure your site from a selectable EC2 location and really test the site insite out and outside in. We dont just say "use parallel downloading", we measure how many host you need to get the most benefit, no yslow wisdom! :-)
More information on http://projectpaas.com/ (hint there is an easter egg, twitter it :-)
Interesting for this group, we are seeking for a small group of beta testers who are willing to install the module on the their dev site, use the testing site (free!) and give us feedback on the proces and outcome. We have only a small number of invites left and want to give to those who have been giving to the community.
Please do contact me (http://drupal.org/user/188/contact) if you want to help us test-drive the service in a month (or willing to use the commercial service :-)

Comments
Is this sort of what New
Is this sort of what New Relic offers, but more Drupal-oriented?
in a way
a bit different, new newlic measures application level (php, ruby etc) where PAAS measures the complete stack (from DNS to Drupal) and does a lot Drupal specific metrics as well (dont use module x, configure y different, use x for y etc).
The biggest difference, ProjectPAAS is a one time only printable report, not a thermometer like new relic is.
(btw: I do love newrelic, so if people compare us with them, fine with me :-)
--
bert boerland
ddotoooh
ooh, and if the infra team of d.o wants to test drive, we would be very proud! (killes?)
--
bert boerland