DOCUMENT PURPOSE
The purpose of this document is to outline a proposed roadmap for Drupal testbot evolution, as discussed during the testbot evolution sprint at DrupalCon Denver on Friday, March 23rd, 2012. Participating in the discussion were Jeremy Thorson, Jimmy Berry, and Randy Fay. This proposal is also influenced by additional consideration and input provided by participants in the TestBot BoF held one day earlier.
There are two main drivers behind the creation of this proposal. The first is the desire to port Drupal.org to Drupal 7 (and the dependency on a PIFT port as part of that exercise), and the second is the desire to provide new automated testing features and functionality which cannot be implemented today given the testing infrastructure’s current architecture and code.
GOALS
From a high level perspective, we have outlined the following strategy in an attempt to
- minimize the amount of work required to facilitate the migration of drupal.org to D7,
- maintain a maximum level of stability for the testing infrastructure throughout the evolutionary transition, and
- facilitate and simplify the development and introduction of new testing-related features.
HIGH LEVEL SUMMARY
The following list outlines the proposed approach. Motivation and reasoning behind each of these steps is provided later within this document.
- Leave qa.drupal.org and the testbots on D6 for the time being.
- Port PIFT to Drupal 7 ‘as-is’, with no new feature additions.
- Leverage the ported PIFT module on drupal.org after the migration, along with the D6 versions of qa.drupal.org and testbot, to provide a ‘stable’ testing layer.
- Develop and implement a second test trigger module for deployment on drupal.org (referred to as the ‘d.o automated testing module’ in this proposal), which would host additional features, drupal.org 'testing' user interfaces, and other ‘new’ functionality over and above what is currently provided by the existing PIFT/PIFR combination.
- Develop and implement updated ‘qa.drupal.org’ and ‘testbot’ components, separate from the existing D6 infrastructure, which would be coupled with this ‘d.o automated testing’ module and host any new feature development.
- Once development of the ‘d.o automated testing’ module and related components have matured and stabilized, migrate the existing ‘branch’ and ‘patch’ testing functionality from the original qa.d.o/testbot sites onto the new architecture.
- Once the migration is complete, and functionality of the new architecture confirmed, decommission the existing testbots and retire the existing qa.d.o site (leaving it intact to provide historical testing results). At some point in the future, once the value of the historical results on qa.d.o has diminished (2-3 years?), the existing qa.d.o site would also be decommissioned.
MOTIVATIONS
1. Leave qa.drupal.org and testbots on D6 for now
Because qa.drupal.org and the testbots are logically decoupled from the PIFT module which is running on drupal.org, there is no need to port them at the same time as the drupal.org components. We initially considered porting these components in advance of the PIFT porting work … but felt that porting for the sake of porting could introduce additional instability into the testing infrastructure, and would only be justified if we were to also introduce new functionality into the system during the process (which also carries the risk of additional instability).
Note that this does not necessarily mean that we are abandoning the existing qa.d.o and testbot codebase … one of the potential paths forward would see us leverage the existing PIFR code as a base, and re-architecting the logic under a new project name. The second available path would see us leverage a different set of modules as the base for the new functionality (Jimmy Berry’s conduit/worker architecture), in which case the existing PIFR codebase would indeed be retired over the long term.
2. Port PIFT to Drupal 7 'as-is', with no new feature additions
In order to simplify the migration of drupal.org to D7, and to prevent the testing infrastructure from becoming the next roadblock preventing that migration activity, we concluded that a straight ‘as-is’ port of PIFT to Drupal 7 would carry the least risk. Much of this work can be done ahead of the “D7 drupal.org” sprint occurring in Portland in April; but due to a number of Project* linkages and dependencies within the existing PIFT module, we will not be able to fully complete this port until the Project* work is completed and we know what the new project-related data structures look like.
3. Deploy PIFT port & existing qa.d.o/testbots along with the drupal.org D7 migration
The intent would be to deploy the D7 port of PIFT, coupled with the existing qa.drupal.org and testbot implementations, during the migration of the drupal.org site to a Drupal 7 codebase. This should help minimize the amount of change and potential for disruption being introduced during the port, and help to provide a ‘stable’ testing infrastructure layer throughout the d.o migration process and push to D8 launch. This testing layer would remain operational alongside the development and deployment of any ‘evolved’ testing infrastructure.
4. Develop and introduce a 'd.o automated testing' module with/after the Drupal.org D7 migration
This ‘d.o automated testing’ module would be deployed in addition to the D7 port of the existing PIFT module, but would reside under a different project name (actual name yet to be determined). This would become the home of all “new” testing user interface components and features on drupal.org, the trigger point for new ‘on-demand’ testing capabilities, and the gateway responsible for communication between drupal.org and a new test dispatcher component (the evolution of qa.d.o).
This module would initially be more dynamic, and subject to more frequent updates, changes, and outages; but running the original PIFT alongside it will allow for evolution of this component to continue while leaving the original in a stable and steady state during the push to Drupal 8; thus avoiding a repeat of the scenario encountered during the final Drupal 7 push (where testbot evolution was forced to take a backseat to the need for testing stability).
5. Develop and introduce new equivalent 'qa.d.o' and 'testbot' components, coupled with the 'd.o automated testing' module
Similar to the scenario with PIFT, these projects would reside under different module names than the existing PIFR project, and would represent the next generation of the qa.drupal.org and testbot functionality. These new modules would contain the architectural and data structure changes required to implement the new ‘on-demand’ testing capabilities and other features triggered by the new drupal.org PIFT interfaces; and operate in parallel with the existing testing infrastructure (i.e. Today’s qa.d.o and bots).
New features and development would occur primarily on these new projects, leaving the existing infrastructure alone to provide the desired stability intact for simpletest branch and patch testing. If the decision is made to move to a new testing framework for Drupal 8, development of the associated changes would likely occur only on the ‘new’ infrastructure.
There are currently two approaches being considered for this portion of the evolutionary roadmap:
- Refactor the existing PIFR, correcting the underlying architectural issues inherent in the module, or
- Evaluate Jimmy Berry’s ‘conduit’ and ‘worker’ modules, which were developed to help address many of these same inherent limitations of the existing codebase.
While the first approach sustains the benefit of sustained familiarity with the existing code base for those who have been involved in maintaining the testing infrastructure to date, the second approach comes with the benefit of existing and finished code (which in turn should require less overall development in order to implement). The current intent is to perform a deep-dive into the conduit/worker architecture and code, and compare it to the existing codebase on the metrics of simplicity, learnability, functionality, flexibility, and sustainability; and then make a final choice on which of these two approaches to pursue once the conduit/worker evaluation is complete.
6. Once proven, migrate qa.d.o and testbot tests to the 'new' architecture
Once development and deployment of the ‘d.o automated testing’ module and related components have matured and stabilized, and new functionality is introduced over and above what is available with the current infrastructure, we would then begin to migrate the existing ‘branch’ and ‘patch’ testing functionality away from the original qa.d.o/testbot sites and onto the new architecture.
Because both systems would be running in parallel, we can take a staged approach to the migration, allowing us to switch projects back and forth to help identify any bugs and edge cases before performing a full cutover.
7. Decommission existing testbots and 'semi-retire' qa.d.o
Once the above cutover is complete, we would then decommission the existing testbots and transition qa.d.o to a ‘retired’ state, leaving it intact in an archive mode for some predetermined length of time (2-3 years?) to provide historical test results and support the ‘View Details’ link on existing tests within the issue queues. (It’s expected that some sort of test result pruning mechanism will eventually be implemented, which would help determine when the final decommissioning of qa.d.o would take place.
SO WHAT'S NEXT?
It is expected that things will need to move into full gear shortly after the Portland sprint (April 23rd - 27th), at which point we will have a better understanding of the new D7 Project* architecture; and what implications it will introduce on the D7 port of the PIFT module.
In order to be ready for the next phase once the porting of the PIFT module is complete, we will need to i) define the final requirements for the next generation of testing infrastructure, ii) complete the evaluation of the Conduit/Worker code (and verify its ability to meet these requirements), and iii) develop UI/UX mockups which help define the triggers and scope the functionality to be included in the new 'd.o automated testing' module.
We're certainly interested in the communities input and assistance as we begin to execute the above plan ... if any of the above sounds appealing to you, the team would be happy to have you on board - look for me on IRC, watch the posts in this group, or hop on over to my d.o contact form to get involved!