Background
Back in April, a handful of us from the Git team set out to create functional tests to tell us if Drupal.org is working as expected when new features are developed and before new code is deployed.
We had the great good fortune to start this project when CapGemini offered the Drupal community three Drupal developers full time for six months. It was my privilege to work closely with these talented and motivated individuals to implement a comprehensive set of tests. We could not have come nearly so far as we have without them.
Accomplishments
During our six months we implemented, 655 scenarios that describe 134 features - in all 3,070 unique tests. We've supported these tests with code for the Drupal 6 branch of Drupal.org and have updated them all as much as possible for the Drupal 7 version as well.
December 21 Demo highlights
Test Output on December 21, 2012
Tests and Projects
Challenges
There were myriad learning opportunities throughout the project - learning the tools, Behat and Mink, discovering just how Drupal.org is supposed to behave, and striving to support the development process, but we’re proud of what we were able to accomplish.
The biggest challenge is that the maintenance and integration of the tests into the Drupal.org process is in an uncertain position as we wrap up this first phase.
The upgrade of Drupal.org from D6 to D7 is on hold pending the outcome of the Governance initiative, leaving the suite straddling two sites, neither of which can be reliably tested:
- We don’t have a complete staging environment for Drupal 6 tests so we can’t use them in a meaningful way with continuous integration. It made sense to put our finite resource toward the 7 site at the time.
- We don’t know whether or when significant new functionality will be introduced to the Drupal 6 version of Drupal.org.
- We do have a more complete testing environment for the Drupal 7 version, but since it’s neither in production use nor under active development, it’s less useful.
Next Steps
Ultimately, the plan is to divide tests into manageable groups to allow volunteers to maintain them by interpreting failures and opening issues in the appropriate places, but nothing is complete or clear enough to allow that to happen.
Despite these challenges, the tests continue to serve one of their primary purposes. They’re an accessible record of how the Drupal.org interface is expected to work. When development of the Drupal 7 version re-commences, they’ll provide a valuable roadmap of what the site needs to do and will contribute to a clear definition of done.
Thank you
Huge thanks to everyone on the team, but especially Pradeep, Sundar, and Suresh. It was an honor to benefit from their gracious determination to work around limitations, their dedication to the best possible work, and their general good company.
Phase 1 Testing Team:
Thanks to the many people who supported the effort in one way or another, halstead, nnewton, michaelemeyers, jhodgdon, Senpai, tizzo, KentBye, lotyrin, skjalf, dww, weaverryan, bleedev, nagwani, the Drupal Association, others.
Comments
A few words from the team
One of the challenging works I have ever been part of, I would say.
Initially when we started, we didn't even have heard of Behat. Apart from that, Behat recommends Linux for the development but Capgemini basically had a Windows platform. So getting everything set up on Windows was another tough job. The initial two weeks were all about self learning and environment set up. Within these limitations we were able to create a few test scripts and have it demonstrated to the senior management from Acquia as well as Capgemini. Following the demo, our work was greatly appreciated and this really boosted the confidence of the team. Later we got 2 week training on Behat/Mink and it was more than enough to structure and shape up the already gained knowledge through self learning in Behat/Mink.
We used to have daily scrum calls with Melissa to update the current status of the work and decide on what needs to be done when we get stuck at anything. We also used to have a weekly call with Michael Mayers from Acquia and Sujit from Capgemini in the early months and then Jacob Redding also joined. We used to demonstrate them the work we did in the past week and discuss the progress we make and the challenges we face.
The Capgemini team had an internal review process that helped us understand all the features each of us were working on. It also helped us to know mistakes and correct them before submitting the code for external review. The review process had really helped us in saving time and effort reducing the extra time required in the normal to and fro communication to correct the possible errors in each task.
Some of the test scripts created and documented by the team had been presented in DrupalCon 2012 – Munich by the officials from Acquia and we consider this as one of the achievements we made. Also most of the common steps we developed have been taken to DrupalExtension - an extension that can be used in the automation testing of Drupal websites. This also has considerably increased the confidence level of the team.
During the development, we faced a lot of challenges. The main issue we faced is the intermittent network failures that happen while the scripts are running. There were another couple of issues too like loosing the user data/posts which we maintain for testing the scripts. Later on, this issue got solved by automating it with the build process.
We also faced user permission issues, while creating scripts that include GIT related operations.
The talent, dedication, teamwork and professionalism shown by my team mates Sundar and Suresh is invaluable and I personally appreciate them for the good work they were keeping.
We got a lot of support from Melissa Anderson who guided and led the team on every step we were stuck and she was very kind enough to solve them quickly so that the team were never left idle and Without her, we would have not got as far as we are now. She was present in all the demos the team had and gave us support each time we were blinking! The daily stand up calls( that too very early in the morning in PST) with Melissa cleared our doubts and made way for our work on the next day. On behalf of the team, I thank Melissa for the support she gave to us, her commitment and dedication and the high level of professionalism she has shown.
Special thanks to Michael Mayers and Jacob Redding for attending our weekly demonstrations and giving us valuable feedback.
We sincerely thank our management for giving us a chance to work in this project and guiding us in the right direction whenever required.
We are very thankful to all the community members who have directly or indirectly associated with this project and provided help whenever we needed.