TestBot sprint in Portland

Day 1 of the DrupalCI sprint has been a whirlwind!

Truth be told, we actually had a Day 0 where some of the community team joined those of us in the DA office and had a few conversations getting up to speed and getting everyone on the same page.

I'm going to organize these daily notes roughly around the DrupalCI component areas.

Containers

One of the major issues to resolve going into the sprint has been finalizing our container architecture and consolidating the work of several contributors

Wins:

• We've finalized the container architecture.
• The base container is complete
• The db base container is complete
• The db containers are complete
• The php version containers are almost complete

Blockers:

• Optimization is critical, we want to minimze duplication and minimize the size of all containers.

API

Another major issue going into the sprint is that the architecture of the API had been a bit fuzzy and aspirational. With everyone working in the same room we've had a much better opportunity to finalize the design and requirements for the API.

Wins:

• Silex has tests, and is working
• We've finalized an MVP API design

Blockers:

• We need to complete a mock api so we can integrate with the other DrupalCI systems

Drupal.org Integration

Integration with Drupal.org is obviously a critical component of the DrupalCI sprint. The goal for MVP is for Drupal.org to be able to poll the API on a pull basis for test data - with an ultimate goal of moving to a push model in future iterations.

Wins:

• A UI has been built for configuring what tests should be run on what branches.
• Work has been done to clean up back end data structures, to prepare a table for results, and to test for edge cases in integration.

Blockers:

• There's a lot left to do to integrate all the cases that should trigger tests, to pass the appropriate parameters, and to interact with the results server.

Test Runner

The test runner does the bulk of the work- taking job definitions and interacting with the other components of CI to actually get containers and perform tests. An important part of the sprint is code refactor and clean up of the test runner, and integrating it with the rest of the work (such as on the API)

Wins:

• Refactoring for better dev experience and consistency.
• Significant restructuring to make more sense.

Blockers:

• The Test Runner is an integration point for lots of the other work, so getting it running again properly as other elements change is going to be an ongoing challenge throughout the sprint.
• Bootstrapping PHPUnit so we can start writing tests.

Results Site

The results site is the clearing host for creating results objects, updating status of tests, and storing artifacts.

Wins:

• Created a base image for all components of the results site. Includes S3 bucket info, back end db config, just about everything it needs.
• Working on deploying two instances of the results site behind a load balancer.

Blockers:

• The rest of the work on the results site is largely on hold until the D.O implementation and how it will poll for Data comes through.

Infrastructure

The DA infrastructure staff have been present to help integrate the DrupalCI system onto production hardware, and ensure that images are configured securely and with good availability.

Wins:

• Cleared some configuration/security blockers to building out images.
• Got the hosts and such in place on AWS.

Blockers:

• N/A

Documentation

A goal for this sprint is to create some decent skeleton documentation for the MVP that we will produce.

Wins:

• Started some initial documentation at: https://www.drupal.org/node/2463169

Blockers:

• Need to convert some skeletal readme files from the repo into true documentation.

There's no sign of slowing this evening as I post this update - so the work will continue! Looking forward to an equally productive day 2!

Day 2

Just like yesterday- as I write this update the sprint pace continues around me at a high level. The great forward momentum is continuing.

We have a few soft blockers and a couple hard blockers/gaps to fill this evening and into yesterday, but tomorrow we strike up the band to see if this baby can dance!

We also tagged and branched DrupalCI legacy for the pre-sprint work that some community members have been running experimentally as we bring all this work together and take it online.

Containers

Work to finalize the containers is wrapping up - and we're getting the test runner working with the new container stack

Wins:

• Big win: Merged with DrupalCI Console
• Backported new containers to old test runner to find missing packages
• Cut container size down (alhtough it's growing again, as we bring in a few packages - see above)
• PHP versions working: 5.3, 5.4, 5.5, AND 7
• DB service containers working: MySQL, MariaDB 5.5, 10 Postgres 9.1.2 (to prevent an update step), 9.4, Mongo DB, SQLlite

Blockers:

• Finding a few missing packages and cleaning those up. Keeping the sizes down.

API

The API has really solidified from an aspirational structure into something structured (and that works!)

Wins:

• "It's awesome right now! It actually does stuff. :)"
• The API is running in production
• 2 running behind load balancer
• Authentication
• You can send the repo the branch the patch - kicks off all the builds, results - returns a whole bunch of links and artifacts
• We're at the stage now where we can 'fulfill the json contract' that's been outlined in the readme/documnetation (parallelizing a serial build command)
• But we can start pushing "hey build this" to the API

Blockers:

• Mile23 not here tomorrow :( (such a tremendous help!)
• Being smarter about the permutations of the builds -- parallelizing what is currently one track for MVP.

Drupal.org Integration

Speccing out the integration and structuring the data continues

Wins:

• Draft of what the results will look like on Drupal.org
• Refining the data storage of results
• Progress on implementing the API

Blockers:

• Need to keep building db data structure

Test Runner

The test runner continues to be critical and has undergone a great deal of refactoring. We have a few gaps to fill to hook it into the rest of the CI system, this evening and on our last day.

Wins:

• Wrote documentation
• Refactored the output (singleton), centralized
• Got unblocked on PHP unit for tests. (Tested our first case of actual test coverage)
• Wrote a couple tests
• In future we will be able to function test using DrupalCI to test itself. (Inception achieved! BWOOOM!)
• Merged in containers (see above)
• moved building of containers to docker_php library
• Almost unblocked on container volumes not having the data they should - we believe we have the code to unblock that. (mid implementation of fix)
• Build process was adapted to use best mirror for ubuntu packages
• We now have base class for testing off of which thing will grow

Blockers:

• (Soft blocker, just a gap) Refactoring to use a pull from a docker hub, as opposed to build by default
• (Harder blocker) Update config step to take the default definition for a job type, merge environment variables. (whereas right now it only runs with a local complete definition file) --- so building out the complete file based on job definition + env variables where available, etc.
• (soft block, annoyance, unexpected behavior) DrupalCI build command sometimes times out and dumps to command line while docker build continues in background.
• Local build of web-5.4 fails - not installed and no such directory errors on pear repository? (Can we mirror or proxy) Possibly our ip is getting blocked/rate limited by pear?
• Future state to use packages from hub instead of building

Results Site

The results site is the clearing host for creating results objects, updating status of tests, and storing artifacts.

Wins:

• Little bit more documentation.
• Truthfully though, the results site is pretty mature, just waiting on integration with the other components.

Blockers:

• The rest of the work on the results site is largely on hold until the D.O implementation and how it will poll for Data comes through.

Infrastructure

More work to bring CI into a production envorinment.

Wins:

• All of the packer builds are automated in the Drupal Infrastructure projects on Drupal.org and jenkins jobs to build out deploy
• Job pruning works
• Got the hosts and such in place on AWS.

Blockers:

• N/A

Documentation

A goal for this sprint is to create some decent skeleton documentation for the MVP that we will produce.

Wins:

• Updated our TLDR one page to rule them all documentation page more: https://www.drupal.org/node/2463169
• Pulled in a bunch of readme files (to the extent that they weren't totally legacy after recent sprint work)

Blockers:

• Need to get more raw docs to clean up after the recent changes.
• Will need to split documentation by component eventually.

We continue to barrel forward at an epic pace! Day 3 will be exciting as we try to light this candle.

Day 4

It's alive! It's ALIVE!!

The DrupalCI Bonus sprint day hit some awesome milestones! Chief among them - we ran the system end to end!

In this case, end-to-end means curling the API the same way the D.O integration does, triggering the legacy runner, querying the sqllite db of results, pushing those results to the results server.

We also have a sneak preview of the UI for the Drupal.org integration with DrupalCI:

Our MVP

We laid out a pretty ambitious goal for the sprint, but also one that gave us a very pragmatic place to reach for:

Deliver a MVP product on production hardware, with integration to Drupal.org, which can be used to demonstrate a full end-to-end D8 core ‘simpletest’ test run request … from Drupal.org through to results appearing on the results server.

So how did we do?

• We're on production hardware.
• We have integration with Drupal.org - admittedly, still on a dev site - we'll need a bit more testing and then deployments.
• We've demonstrated a full end-to-end D8 core simpletest run (see nick_schuch's screenshot above).
• The results appear on the results server.

Where does this lead next?

Following on the succesful MVP work above, we want to demo a few testing scenarios all the way from Drupal.org. There are definitely still some moving parts to work on here - but these demos will become possible as the team finishes polishing things up and we're able to deploy the work we've done on the Drupal.org integration side.

• Demo per-commit test running integration with Drupal.org
• Demo per-patch test running integration with Drupal.org
• Demo contrib test running integration with Drupal.org. (ensuring we have all the moving parts in place for contrib testing will certainly more work behind it.)

We have a hit list of additional items below - and will be recapping further with the DrupalCI sprint team, and internally at the Drupal Association.

Our other Stretch Goals

Before running through the hitlist of today's successful items, and the roadmap of where we're going with this next, let's talk about the stretch goals we hit for the sprint.

• Working containers for: PHP versions 5.3, 5.4, 5.5, 5.6 and PHP 7.
• Working containers for: MySQL, Postgres, SQLlite, and MongoDB
• Containers hosted on Docker Hub so they can be easily pulled down
• A high availability architecture with simple deployment for all of Drupal.org's hosted components of DrupalCI.

Final Day Successes and Roadmap

Containers

Wins:

• The containers are DONE, committed, and available on Docker Hub.

Roadmap

• Small optimizations for size/performance.
• Any tweaks that we may discover we need to ensure that run_tests.sh functions (such as the timezone fix we had to make)
• Documentation
  • Moving some of the architectural documentation available on the project docs into the readme.md
  • Updating documentation in both places to talk about what components are included in each test runner (basically, the docker files)

API

Wins:

• Refined to a single test per-query model. (Allows D.O to simply send separate queries per type of test wanted)
• The data structure for the API IO is final

Roadmap:

• Not much! Docs can always be improved. :)

Results Site

Wins:

• Shifted from working with XML objects to test artifacts in a SQLlite DB for now.

Roadmap:

• Switch back to an XML parsing model, when we are able to slot in the JUnit XML object publisher. (A community member not at the sprint is lined up to assist us with this)
• Need to solve a (D8) issue with running the results site in High Availability due to the implications of compiled php in a freshly spun up web instance running against an existing db of the previous web instance.

Drupal.org Integration

Wins:

• Data structure is good.
• API is largely implemented.
• We are queueing tests.
• We have a testing UI

Roadmap:

• We need a UI for retesting.
• We need to finalize the integration point with the API (on a dev site at the moment)
• We need to make several small deployments to bring this work into the wild.

Test Runner

We have two test runner elements that exist in parallel. A legacy test_runner in bash, and the symfony console based new test runner that is essentially complete, but being refind and slotted into place.

Wins:

• We fixed the timezone issue causing a FATAL at the point of run_tests (really this was a container issue)
• All of the goal architecture for MVP was finalized and implemented.
• We have a generic job type that the test runner completes.

Roadmap:

• After some of the latest merges the containers are not starting in the new test runner - likely just needs a bit of debugging.
• We want to complete end-to-end simpletest job type with the new test runner.
• We want to extend the capability of the new test runner using the (simple and powerful) plugin system that has been developed.
• We need to define more job types and run more tests.

run_tests.sh

At this point, as we just iteratively try and complete tests using the full chain, we'll need core dev support to help troubleshoot the root causes of any failures in run_tests.sh.

These will likely be small environment issues (such as the timezone issue alluded to above) but may result in small fixes to the project.

Documentation

Wins:

• The doc page to rule them all is still getting updated here: https://www.drupal.org/node/2463169
• There are also some updates needed to the readme.md files throughout the project.

Roadmap:

• Better documentation of the test runner is needed
• More documentation of job definition files is needed.
• More 'how to', rather than architectural documentation is needed.
• Documentation of examples of API implementation are needed.

Thanks for all the fish!

Most of our sprinters are on their way back home, and getting some much needed rest after long nights, and hard work. They did an outstanding job. We'll be setting up a time to recap with the DrupalCI sprint team (and the remote folks who've helped us out along the way) in the next week or so.

And that's it for the DrupalCI sprint. Massive thanks to everyone who came and visited with us in the Drupal Association offices here in Portland. Tremendous thanks to all the donors to D8Accelerate who made this sprint possible!

Thanks to the DA for hosting and feeding us, and to everyone for being a bunch of smart people in a room. :-)