Plan and discuss the first Extreme Drupal Games (aka the "Golden Neckbeard" competition) for GLADCamp 2014

Posted by christefano on January 24, 2014 at 4:57am

This year's GLADCamp conference will feature the first Extreme Drupal Games! This event will run throughout the conference, beginning on Friday, March 7th and ending when winners are announced at the closing of the conference on Sunday, March 9th.

We have a sponsor who will be donating 6 identical VMs (distro and stack TBD) for us to use during the conference. I've asked that each VM have a minimum of 2 cores and 4GB of RAM, but exact specs are still being worked out. These VMs are hosted at 1 Wilshire in Downtown Los Angeles, so there will be extremely low latency.

The plan right now is for up to 6 teams to compete over the course of the weekend, tuning both the stack and the Drupal application to get the best possible performance in the allotted time. The team with the fastest Drupal site at the end of the conference will be declared the winner.

Now that we have a date and time, the question is how we're going to load test and judge the teams. What are the rules? Do we test for both anonymous and logged in users? Will contestants be permitted to rip out pieces of the stack and install completely different ones? How about the use of premade scripts? Do we allow commercial software, or go 100% free and open source?

To be fair to all participants, I think each VM should have an identical stack and preloaded Drupal site (with a couple gigabytes of kitten photos, of course), each on their own subdomain or IP address whose location is only known to the judges and the team assigned that VM.

What do you all think? Can we do this in time? We were talking about the idea this week at the High Performance Drupal Meetup but need everyone's help and crazy ideas to make it happen. Please share your thoughts and suggestions!

Comments

Suggestions

Posted by mikeytown2 on January 24, 2014 at 5:55am

Have a clean vanilla reference install. Compare all database tables (excluding cache, etc) at the end of the competition to make sure data was not lost. Negative points can be awarded.
Allow for different databases and cache back ends. Reference should be Apache with MySQL 5.6.
Require the use of some of the most popular modules (views, etc)
Test for anonymous and logged in users. Give more weight to logged in.
Have Drush in a Cache Clear loop and time how long it takes Devel to generate 15k nodes.
Load test logged in users when the install cache back end is used (/dev/null).
Be sure to test the front end http://fourkitchens.com/blog/2013/10/30/automated-frontend-tools Compare screen shots to make sure it displays the same.
Test how long a large VBO operation takes (unpublish all nodes and then republish them).
Have a sabotage test where certain modules (non critical) are removed from the file system and opcode caches are clear. Then do a logged in user load test.
Do a massive 20k concurrent anonymous user load and see how the server handles it for 1 minutes.
Test for cache freshness. Update a block that is on every page; see how long it takes for that change to go out. Update the front page and see how long that takes. Update something in a view, test it.

While all of this is going on have 5 concurrent anonymous users hitting the front page all the time just to keep things interesting. Have it max out at 100 requests per second. Incomplete and non 200 response codes will be dinged. Long page loads will be dinged as well.

Third party caching allowed?

Posted by aiScaler Support on January 24, 2014 at 8:04pm

Do you allow the installation of third-party HTTP accelerators, like Varnish? If so, this sounds very interesting.

I would assume so

Posted by mikeytown2 on January 24, 2014 at 8:45pm

Without caching above the Drupal stack this wouldn't be a good competition.

Yes, absolutely. I'm not sure

Posted by christefano on January 25, 2014 at 9:15pm

Yes, absolutely. I'm not sure if our event rules will insist that the server stack remains 100% open source, but I think each team should be permitted to replace and install anything they want to create a high performance stack.

What do you all think? Should we require open source components or allow the use of commercial, closed source accelerators?

If commercial is allowed...

Posted by FluxSauce on January 25, 2014 at 10:11pm

I'd love to just optimize a site and host it on a regular Pantheon site environment. No external CDN/proxy like CloudFlare, just our open-source stack (nginx, php-fpm, apc, redis, varnish, mariadb).

The virtual machines and

Posted by christefano on January 25, 2014 at 11:33pm

The virtual machines and their sites for this event will be hosted by Winky Networks, one of GLADCamp's sponsors, at their datacenter in Downtown Los Angeles.

Using hosting on a "Drupal as a service" provider is fine and good, but not for this competition. We need to guarantee that everyone is using the same starting point.

My question right now is if we want to permit the competing teams to use commercial software as part of the server environments, e.g. aiCache (which I guess is now called aiScaler).

Both?

Posted by fejn on January 26, 2014 at 6:24am

a) only open source components would be an interesting benchmark for comparison
b) commercial, closed source accelerators would give vendors an extra way to show off their products, but I don't think it would be as good for the open source community, unless there was a way to encourage them to discuss what kind of changes had been made to produce their 'super-duper' version. If someone uses commercial product, they should be encouraged to discuss it in detail at one of the High Performance meetups (maybe this would be impossible ????)

Jeff

I'm excited to announce we're

Posted by christefano on January 25, 2014 at 9:12pm

I'm excited to announce we're calling this event the Extreme Drupal Games. The GLADCamp.org website now has a page at https://gladcamp.org/2014/extreme-drupal-games and we'll update that page as we work out the details.

The name "Golden Neckbeard" is hilarious (at least I think so) but for reasons I've expressed elsewhere I'm not very comfortable with that particular name.

Is the Xtreme D games

Posted by likewhoa on January 26, 2014 at 3:28am

Is the Xtreme D games exclusive to LA residents or can a user from New York City compete as well or else where?

+1 for only allowing the use of FOSS in the competition as commercial software might be beneficial to those that are willing to spend money on non FOSS application and those that leverage the true power of FOSS to bend it how they want.

@christefano

I think that forcing a specific LAMP stack should not be forced but do provide all TEAM members with a 'drush arb' archive to restore with 'drush arr' and with this competitors can tweak their VM however they want, even by replacing the existing linux distro on it with a custom one i.e arch linux.

@mikeytown2 are the load test going to be performed with httperf?

w.r.t to automated testing perhaps using 'New Relic' will add some eye candy and more detail statistics to performance.

I would go with cli tools eitherway. This competition sounds like it will be great!

bending technology to fit businesses.

my 2 cents

Posted by rcross on January 27, 2014 at 5:10am

really interesting idea.

I think it might be a good idea to only publish the details of the competition once it opens. Since performance optimisation is almost always unique to the challenges of a particular site/use case, a competition should also reflect that. That way the competition is both a reflection of drupal's capabilities and the winning team's skill level and is not just some generic benchmarking exercise. This would also allow for the competition to be run in other camps/events with different stacks, parameters, etc. and still keep the competitions interesting.

For example, a performance competition that is restricted to just pure open source options would have much different solutions from one that also allowed commercial solutions. One that favours the best performance for anonymous users (most successful page requests) would be quite different to one that favours the best performance for logged in users (highest level of concurrent users). Even highlighting different types of testing for different use cases would be important/interesting.

--Ryan
Ryan Cross
Drupal Development Services
ProjectPier project management and collaboration software

+1 for this name change

Posted by joshk on January 27, 2014 at 4:41pm

Much more inclusive, and still captures the essence! Nice idea!

https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com

Thanks, Josh! I share the

Posted by christefano on January 31, 2014 at 8:49am

Thanks, Josh! I share the credit with Rob Ristroph, who brainstormed on the name with me at DrupalCamp Austin.

I hope you and the Pantheon team can make it down for the games. Let me know if you have any questions or about anything else GLADCamp.

Thanks again, Josh, and I'm

Posted by christefano on February 17, 2014 at 7:27am

Thanks again, Josh, and I'm very excited that you and the team will be coming down for GLADCamp! I missed your keynote at SANDcamp earlier this year and I'm looking forward to your keynote at this year's GLADCamp.

In case commercial software is allowed:

Posted by aiScaler Support on January 30, 2014 at 3:47pm

I just talked with my manager, and I am allowed to offer free aiScaler licenses and support for this challenge.

In our own benchmarks aiScaler outperformed caching products from Varnish(+) and from closed source manufactures like F5 and Netscaler:
http://aiscaler.com/application-delivery-controllers-on-the-cloud
http://aiscaler.com/wp-content/uploads/2014/01/Varnish_vs_aiScaler.pdf

Any team can download a fully functional 30 day license here and use our customer support for free. Our support employees are former Linux sysadmins, or used to work for Varnish or Citrix. However, for a "fair" competition we will treat any team the same as any prospective client: support will be limited to 2 hours of phone, Skype or email support.

I'm confident that we can outperform any (commercial) caching solution on the market. A competition like this would be an excellent show-case.

Hey there, I just sent you a

Posted by christefano on February 17, 2014 at 7:25am

Hey there, I just sent you a message via your contact form.

Please get in touch with me about how we might be able to collaborate on the Extreme Drupal Games.

You can contact me directly or send a message via the general GLADCamp contact form at https://gladcamp.org/2014/contact

Thanks!

Commercial Products should purchase their own VPS to compete

Posted by joe_pixelrow on January 29, 2014 at 8:07pm

It's fine if companies want to showcase their commercial solutions, but as we only have 6 free VPS's available, they should be given to teams using entirely open source operating systems and application stacks.

The commercial solution providers can purchase their own identical VPS for the competition and design a stack using whatever combination of open-source and commercial products they wish.

This will provide the widest variety of solutions for the competition.

Documentation Requirement

Posted by fullerja on January 31, 2014 at 2:27pm

This is a great idea! I think it should also be a requirement to fully and properly document all the things that are added, tweaks that are made etc. It would be great if the specs and test results could be posted somewhere (here, GLADcamp site) for the rest of us to learn from.

Hi everybody, we're about to

Posted by christefano on February 17, 2014 at 7:31am

Hi everybody, we're about to put the finishing touches on the rules for the Extreme Drupal Games.

We will not be announcing all the rules until the start of the games in the morning hours of March 8th, but we will have a few strict guidelines to share when we post the event announcement in the next few days.

We're playing around with the idea of allowing one remote team to participate (think of it as the "play from home" edition of the game). So, if anyone out there wants to put a team together to participate and compete with our teams at GLADCamp, start your engines.