Plan and discuss the first Extreme Drupal Games (aka the "Golden Neckbeard" competition) for GLADCamp 2014

christefano's picture

This year's GLADCamp conference will feature the first Extreme Drupal Games! This event will run throughout the conference, beginning on Friday, March 7th and ending when winners are announced at the closing of the conference on Sunday, March 9th.

We have a sponsor who will be donating 6 identical VMs (distro and stack TBD) for us to use during the conference. I've asked that each VM have a minimum of 2 cores and 4GB of RAM, but exact specs are still being worked out. These VMs are hosted at 1 Wilshire in Downtown Los Angeles, so there will be extremely low latency.

The plan right now is for up to 6 teams to compete over the course of the weekend, tuning both the stack and the Drupal application to get the best possible performance in the allotted time. The team with the fastest Drupal site at the end of the conference will be declared the winner.

Now that we have a date and time, the question is how we're going to load test and judge the teams. What are the rules? Do we test for both anonymous and logged in users? Will contestants be permitted to rip out pieces of the stack and install completely different ones? How about the use of premade scripts? Do we allow commercial software, or go 100% free and open source?

To be fair to all participants, I think each VM should have an identical stack and preloaded Drupal site (with a couple gigabytes of kitten photos, of course), each on their own subdomain or IP address whose location is only known to the judges and the team assigned that VM.

What do you all think? Can we do this in time? We were talking about the idea this week at the High Performance Drupal Meetup but need everyone's help and crazy ideas to make it happen. Please share your thoughts and suggestions!

Comments

Suggestions

mikeytown2's picture
  • Have a clean vanilla reference install. Compare all database tables (excluding cache, etc) at the end of the competition to make sure data was not lost. Negative points can be awarded.
  • Allow for different databases and cache back ends. Reference should be Apache with MySQL 5.6.
  • Require the use of some of the most popular modules (views, etc)
  • Test for anonymous and logged in users. Give more weight to logged in.
  • Have Drush in a Cache Clear loop and time how long it takes Devel to generate 15k nodes.
  • Load test logged in users when the install cache back end is used (/dev/null).
  • Be sure to test the front end http://fourkitchens.com/blog/2013/10/30/automated-frontend-tools Compare screen shots to make sure it displays the same.
  • Test how long a large VBO operation takes (unpublish all nodes and then republish them).
  • Have a sabotage test where certain modules (non critical) are removed from the file system and opcode caches are clear. Then do a logged in user load test.
  • Do a massive 20k concurrent anonymous user load and see how the server handles it for 1 minutes.
  • Test for cache freshness. Update a block that is on every page; see how long it takes for that change to go out. Update the front page and see how long that takes. Update something in a view, test it.

While all of this is going on have 5 concurrent anonymous users hitting the front page all the time just to keep things interesting. Have it max out at 100 requests per second. Incomplete and non 200 response codes will be dinged. Long page loads will be dinged as well.

Third party caching allowed?

aiScaler Support's picture

Do you allow the installation of third-party HTTP accelerators, like Varnish? If so, this sounds very interesting.

I would assume so

mikeytown2's picture

Without caching above the Drupal stack this wouldn't be a good competition.

Yes, absolutely. I'm not sure

christefano's picture

Yes, absolutely. I'm not sure if our event rules will insist that the server stack remains 100% open source, but I think each team should be permitted to replace and install anything they want to create a high performance stack.

What do you all think? Should we require open source components or allow the use of commercial, closed source accelerators?


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

If commercial is allowed...

FluxSauce's picture

I'd love to just optimize a site and host it on a regular Pantheon site environment. No external CDN/proxy like CloudFlare, just our open-source stack (nginx, php-fpm, apc, redis, varnish, mariadb).

The virtual machines and

christefano's picture

The virtual machines and their sites for this event will be hosted by Winky Networks, one of GLADCamp's sponsors, at their datacenter in Downtown Los Angeles.

Using hosting on a "Drupal as a service" provider is fine and good, but not for this competition. We need to guarantee that everyone is using the same starting point.

My question right now is if we want to permit the competing teams to use commercial software as part of the server environments, e.g. aiCache (which I guess is now called aiScaler).


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

Both?

fejn's picture

a) only open source components would be an interesting benchmark for comparison
b) commercial, closed source accelerators would give vendors an extra way to show off their products, but I don't think it would be as good for the open source community, unless there was a way to encourage them to discuss what kind of changes had been made to produce their 'super-duper' version. If someone uses commercial product, they should be encouraged to discuss it in detail at one of the High Performance meetups (maybe this would be impossible ????)

Jeff

I'm excited to announce we're

christefano's picture

I'm excited to announce we're calling this event the Extreme Drupal Games. The GLADCamp.org website now has a page at https://gladcamp.org/2014/extreme-drupal-games and we'll update that page as we work out the details.

The name "Golden Neckbeard" is hilarious (at least I think so) but for reasons I've expressed elsewhere I'm not very comfortable with that particular name.


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

Is the Xtreme D games

likewhoa's picture

Is the Xtreme D games exclusive to LA residents or can a user from New York City compete as well or else where?

+1 for only allowing the use of FOSS in the competition as commercial software might be beneficial to those that are willing to spend money on non FOSS application and those that leverage the true power of FOSS to bend it how they want.

@christefano

I think that forcing a specific LAMP stack should not be forced but do provide all TEAM members with a 'drush arb' archive to restore with 'drush arr' and with this competitors can tweak their VM however they want, even by replacing the existing linux distro on it with a custom one i.e arch linux.

@mikeytown2 are the load test going to be performed with httperf?

w.r.t to automated testing perhaps using 'New Relic' will add some eye candy and more detail statistics to performance.

I would go with cli tools eitherway. This competition sounds like it will be great!

bending technology to fit businesses.

my 2 cents

rcross's picture

really interesting idea.

I think it might be a good idea to only publish the details of the competition once it opens. Since performance optimisation is almost always unique to the challenges of a particular site/use case, a competition should also reflect that. That way the competition is both a reflection of drupal's capabilities and the winning team's skill level and is not just some generic benchmarking exercise. This would also allow for the competition to be run in other camps/events with different stacks, parameters, etc. and still keep the competitions interesting.

For example, a performance competition that is restricted to just pure open source options would have much different solutions from one that also allowed commercial solutions. One that favours the best performance for anonymous users (most successful page requests) would be quite different to one that favours the best performance for logged in users (highest level of concurrent users). Even highlighting different types of testing for different use cases would be important/interesting.

+1 for this name change

joshk's picture

Much more inclusive, and still captures the essence! Nice idea!

Thanks, Josh! I share the

christefano's picture

Thanks, Josh! I share the credit with Rob Ristroph, who brainstormed on the name with me at DrupalCamp Austin.

I hope you and the Pantheon team can make it down for the games. Let me know if you have any questions or about anything else GLADCamp.


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

Thanks again, Josh, and I'm

christefano's picture

Thanks again, Josh, and I'm very excited that you and the team will be coming down for GLADCamp! I missed your keynote at SANDcamp earlier this year and I'm looking forward to your keynote at this year's GLADCamp.


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

In case commercial software is allowed:

aiScaler Support's picture

I just talked with my manager, and I am allowed to offer free aiScaler licenses and support for this challenge.

In our own benchmarks aiScaler outperformed caching products from Varnish(+) and from closed source manufactures like F5 and Netscaler:
http://aiscaler.com/application-delivery-controllers-on-the-cloud
http://aiscaler.com/wp-content/uploads/2014/01/Varnish_vs_aiScaler.pdf

Any team can download a fully functional 30 day license here and use our customer support for free. Our support employees are former Linux sysadmins, or used to work for Varnish or Citrix. However, for a "fair" competition we will treat any team the same as any prospective client: support will be limited to 2 hours of phone, Skype or email support.

I'm confident that we can outperform any (commercial) caching solution on the market. A competition like this would be an excellent show-case.

Hey there, I just sent you a

christefano's picture

Hey there, I just sent you a message via your contact form.

Please get in touch with me about how we might be able to collaborate on the Extreme Drupal Games.

You can contact me directly or send a message via the general GLADCamp contact form at https://gladcamp.org/2014/contact

Thanks!


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  
joe_pixelrow's picture

It's fine if companies want to showcase their commercial solutions, but as we only have 6 free VPS's available, they should be given to teams using entirely open source operating systems and application stacks.

The commercial solution providers can purchase their own identical VPS for the competition and design a stack using whatever combination of open-source and commercial products they wish.

This will provide the widest variety of solutions for the competition.

Documentation Requirement

fullerja's picture

This is a great idea! I think it should also be a requirement to fully and properly document all the things that are added, tweaks that are made etc. It would be great if the specs and test results could be posted somewhere (here, GLADcamp site) for the rest of us to learn from.

Hi everybody, we're about to

christefano's picture

Hi everybody, we're about to put the finishing touches on the rules for the Extreme Drupal Games.

We will not be announcing all the rules until the start of the games in the morning hours of March 8th, but we will have a few strict guidelines to share when we post the event announcement in the next few days.

We're playing around with the idea of allowing one remote team to participate (think of it as the "play from home" edition of the game). So, if anyone out there wants to put a team together to participate and compete with our teams at GLADCamp, start your engines.


Exaltation of Larks
Founder, CEO
http://www.larks.la  
Droplabs
Founder, Lead Burrito Analyst
http://droplabs.net  
Greater Los Angeles Drupal
Organizer, Drupal Adventure Guide
http://drupal.la  

High performance

Group events

  • 2015-11-16 08:00 - 2015-11-20 13:00 America/New_York
Add to calendar

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week