[GSoC Idea] A/B Testing Framework

Overview:
Develop a highly configurable, pluggable, A/B testing framework

Description:

For an upcoming project I will be starting within the next couple of months there is a requirement for some A/B testing. Specifically for this project it is just A/B testing of two versions of content displayed in a block. This requirement, however, has started the gears in my head spinning and I think this might be a great opportunity for this year's Google Summer of Code.

The idea, as it currently exists in my mind:

Testable Elements:
1. Content (Various Revisions of Content)
2. Blocks (Placement of blocks, Visibility of blocks on specific URLs)
3. Themes (Swapping out entire themes, Swapping out different templates within the same theme)

Additional testable elements could be added via a pluggable system (perhaps the ctools plugin system?).

Testing plans (determining who sees A vs. who sees B):
1. nth visitor
2. Percentage (show 30% of vistors A, and 70% B)
3. By IP Address (IP Range, Geo-Location)

Likewise, additional test plans could be added via a pluggable system.

Ideally all of this would also be schedule-able, such that you could schedule a test to begin on a particular Date/Time and end at a particular Date/Time, or end when a specified number of visitors have been tested, or not end until manually disabled, etc.

Likewise, results would be tracked. Click-rate of A vs. B, etc (I anticipate a heavy reliance on Google Analytics for this portion of the project).

This strikes me as a highly useful tool that is missing from Drupal, and one that many people wouldn't recognize as missing from their site until they have it as an available tool in their arsenal.

I can see this being of particular importance to ecommerce sites. While I don't see it being directly tied into the Drupal Commerce module (or any other drupal e-commerce solution), I can see it hugely benefiting those who are doing any kind of sales with Drupal, as they'll be able to configure ongoing tests to determine the most profitable configuration of their site layout, wording of content, and positioning of content.

As stated, this is something I will require at least a portion of in the near future already, but rather than doing a custom one-off solution for the one simple use case, I'd much rather see it become a robust framework that is usable in virtually any A/B testing use case -- or, in the event it doesn't by default, be extensible to fit any use case. Therefore, I am also happy to volunteer myself as a mentor on this project as well.

Additional ideas, feedback, etc are very welcome.