Simple A/B Testing for Drupal.org

mgifford's picture

I've been going through the Drupal.org Customizations list.

Realized with many of the changes it would to be able to evaluate with some sort of AB testing.

It would be really good to have measurable results for various changes so that we can actually test what the impact of the is.

We could use one of these modules:
https://drupal.org/project/abtest
https://drupal.org/project/multivariate

But since most of the behavior I'm interested in testing is of our user community, we could also simply do even/odd on user ID's for most elements.

Whichever we choose, we need access to data about how this affects user behavior.

We've got to have dozens of folks in this community with a bunch of experience in building these. Who could we engage?

Comments

Hi Mike, we (The DA) have an

tvn's picture

Hi Mike,

we (The DA) have an https://www.optimizely.com/ account and could use that one for testing if needed.

Users vs Anonymous

mgifford's picture

I haven't used optimizely before. It looks interesting, but what I wanted to see was what functions you could put into the PHP such that we could better track user behavior there.

I've applied a simple approach in the latest patch here:
https://drupal.org/node/2185511

Simply doing an even/odd for user id's. I'm most interested in engaging the 1 million members as a start. I know there are even more anonymous users, but I figure it's easier to start with known folks.

In changes (like the ones I've been proposing), I'd like to start using sample groups to test & evaluate the changes. We've got a large enough sample group.

This should make it easier to test changes, measure their behavior, modify & then roll out to the community (before iterating it all over again).

I'd give a -1 for doing this

mikey_p's picture

I'd give a -1 for doing this in PHP and only for anonymous users. it's not worth providing a jarring experience when an anonymous user logs in and the the interface or content of the website changes. Given the problems that d.o transitionally has with scaling and the use of things like Varnish for anonymous traffic, Optimizely would be the best bet for A/B testing thoroughly. It's also easier to install and setup vs. integrating a module on our backend (that would further impact performance for logged in users).

It certainly takes some thought

mgifford's picture

Thanks for your feedback mikey_p. And yes, certainly it takes some thought to see that it's done right. However, it does totally depend on the types of changes that are being considered. I'm not thinking that this is re-testing the entire interface, simply adding or modifying small pieces of it.

I don't know if Optimizely allows you to plug in easily into PHP. It must be able to get into the code level somehow, but I have never used it.

Using PHP might be a shortcut that has too many problems with it. If we're using Optimizely, is there an example of it's use?

Absolutely, Varnish would need to be considered. I'm totally into using Optimizely for anonymous users.

Also, we'd need to test that other caching doesn't get in the way.