Design and development of a new generation of testing infrastructure

What's your idea?

On October 18th, 2007, Chad (hummonk) Phillips made an initial commit to the ‘Project Issue File Test’ module, with the description “initial commit of an integrated file testing platform for project issue module”. This kicked off the pursuit of automated testing integration between drupal.org and qa.d.o (then referred to as test.drupal.org). In 2009, Jimmy (boombatower) Berry performed a significant overhaul and redesign of the system, introducing a number of structural and architectural changes.

By the time Drupal 8 is launched, we’ll have been running for 5 years on the current architecture, and almost 6½ years on the project. In that time it has served us well; but has not kept up with technology and the changing needs of the Drupal community. As such, we need to start planning for the next iteration of Drupal’s Continuous Integration tools.

Currently, we have a single-purposed 'Simpletest' testing architecture stack. I envision the next generation of testing infrastructure breaking away from this model, and instead serving as more of a generic job dispatcher for a large assortment of automation tasks ... with the flexibility to support a variety of different job runners, so that it can evolve to support the changing requirements of the community in the future. (For example, a PIFR plugin for simpletest, a PHPUnit-only plugin, a Travis.CI integration plugin, etc.)

With a generic job dispatcher, worker functionality need not be restricted to testing purposes. We could have plugins for patch conflict detection, automatic rerolls, Drupal CodeSniffer assessments, etc. Worker plugin selection and/or environment specific arguments could be passed to the dispatcher in the job description (referential array or .travis.yml like file).

The dispatcher itself may be fully custom, or just a skin interfacing with something like jenkins on the back-end.

I also envision an upgrade to the simpletest worker environment, supporting on-demand turn-up/down of workers to deal with dynamic capacity requirements, "fast-fail" logic, preliminary (during testing) results, etc.

What are the benefits?

• Opportunity to break today's environment/plugin coupling, and test all supported environments
• New testing feature functionality
• Support changing testing requirements as the community/project evolves

What are the risks?

• Scope of work
• Maintaining stability of existing environment while implementing the new

How can we measure the impact of this idea? (metrics)

• Core testing suite run time
• Number of nasty testbot cursing tweets per month. ;)

Who directly benefits from / will use this improvement? (target audiences)

• Core and contrib developers / maintainers

Are additional resources available for discovery/implementation? (volunteer effort, financial backing, etc.)

I'll definitely make myself available as much as possible, and willing to contribute as much as able on both the overall architecture and project coordination angles ... though I can't quantify how much 'as able' is (from a weekend warrior perspective).