Simple A/B Testing for Drupal.org

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
mgifford's picture
Posted by mgifford on January 24, 2014 at 1:04am

I've been going through the Drupal.org Customizations list.

Realized with many of the changes it would to be able to evaluate with some sort of AB testing.

It would be really good to have measurable results for various changes so that we can actually test what the impact of the is.

We could use one of these modules:
https://drupal.org/project/abtest
https://drupal.org/project/multivariate

But since most of the behavior I'm interested in testing is of our user community, we could also simply do even/odd on user ID's for most elements.

Whichever we choose, we need access to data about how this affects user behavior.

We've got to have dozens of folks in this community with a bunch of experience in building these. Who could we engage?

Categories:

Comments

Hi Mike, we (The DA) have an

Posted by tvn on January 27, 2014 at 9:14pm
tvn's picture

Hi Mike,

we (The DA) have an https://www.optimizely.com/ account and could use that one for testing if needed.

Users vs Anonymous

Posted by mgifford on January 31, 2014 at 2:01pm
mgifford's picture

I haven't used optimizely before. It looks interesting, but what I wanted to see was what functions you could put into the PHP such that we could better track user behavior there.

I've applied a simple approach in the latest patch here:
https://drupal.org/node/2185511

Simply doing an even/odd for user id's. I'm most interested in engaging the 1 million members as a start. I know there are even more anonymous users, but I figure it's easier to start with known folks.

In changes (like the ones I've been proposing), I'd like to start using sample groups to test & evaluate the changes. We've got a large enough sample group.

This should make it easier to test changes, measure their behavior, modify & then roll out to the community (before iterating it all over again).

--
OpenConcept | Twitter @mgifford | Drupal Security Guide

I'd give a -1 for doing this

Posted by mikey_p on February 26, 2014 at 10:41pm
mikey_p's picture

I'd give a -1 for doing this in PHP and only for anonymous users. it's not worth providing a jarring experience when an anonymous user logs in and the the interface or content of the website changes. Given the problems that d.o transitionally has with scaling and the use of things like Varnish for anonymous traffic, Optimizely would be the best bet for A/B testing thoroughly. It's also easier to install and setup vs. integrating a module on our backend (that would further impact performance for logged in users).

It certainly takes some thought

Posted by mgifford on March 6, 2014 at 7:29pm
mgifford's picture

Thanks for your feedback mikey_p. And yes, certainly it takes some thought to see that it's done right. However, it does totally depend on the types of changes that are being considered. I'm not thinking that this is re-testing the entire interface, simply adding or modifying small pieces of it.

I don't know if Optimizely allows you to plug in easily into PHP. It must be able to get into the code level somehow, but I have never used it.

Using PHP might be a shortcut that has too many problems with it. If we're using Optimizely, is there an example of it's use?

Absolutely, Varnish would need to be considered. I'm totally into using Optimizely for anonymous users.

Also, we'd need to test that other caching doesn't get in the way.

--
OpenConcept | Twitter @mgifford | Drupal Security Guide