Back in April, a handful of us from the Git team set out to create functional tests to tell us if Drupal.org is working as expected when new features are developed and before new code is deployed.
We had the great good fortune to start this project when CapGemini offered the Drupal community three Drupal developers full time for six months. It was my privilege to work closely with these talented and motivated individuals to implement a comprehensive set of tests. We could not have come nearly so far as we have without them.
During our six months we implemented, 655 scenarios that describe 134 features - in all 3,070 unique tests. We've supported these tests with code for the Drupal 6 branch of Drupal.org and have updated them all as much as possible for the Drupal 7 version as well.
December 21 Demo highlights
Test Output on December 21, 2012
Tests and Projects
There were myriad learning opportunities throughout the project - learning the tools, Behat and Mink, discovering just how Drupal.org is supposed to behave, and striving to support the development process, but we’re proud of what we were able to accomplish.
The biggest challenge is that the maintenance and integration of the tests into the Drupal.org process is in an uncertain position as we wrap up this first phase.
The upgrade of Drupal.org from D6 to D7 is on hold pending the outcome of the Governance initiative, leaving the suite straddling two sites, neither of which can be reliably tested:
- We don’t have a complete staging environment for Drupal 6 tests so we can’t use them in a meaningful way with continuous integration. It made sense to put our finite resource toward the 7 site at the time.
- We don’t know whether or when significant new functionality will be introduced to the Drupal 6 version of Drupal.org.
- We do have a more complete testing environment for the Drupal 7 version, but since it’s neither in production use nor under active development, it’s less useful.
Ultimately, the plan is to divide tests into manageable groups to allow volunteers to maintain them by interpreting failures and opening issues in the appropriate places, but nothing is complete or clear enough to allow that to happen.
Despite these challenges, the tests continue to serve one of their primary purposes. They’re an accessible record of how the Drupal.org interface is expected to work. When development of the Drupal 7 version re-commences, they’ll provide a valuable roadmap of what the site needs to do and will contribute to a clear definition of done.
Huge thanks to everyone on the team, but especially Pradeep, Sundar, and Suresh. It was an honor to benefit from their gracious determination to work around limitations, their dedication to the best possible work, and their general good company.
Phase 1 Testing Team:
Thanks to the many people who supported the effort in one way or another, halstead, nnewton, michaelemeyers, jhodgdon, Senpai, tizzo, KentBye, lotyrin, skjalf, dww, weaverryan, bleedev, nagwani, the Drupal Association, others.