Recap: Modernizing Testbot Initiative meeting September 1st

The Drupal 8 RC blocker issue has been marked fixed:
https://www.drupal.org/node/2467925

This is great news - but of the ~10ish remaining Drupal 8 RC criticals a number of them relate to Postgres support. There is still some concern that postgres failures could be due to postgres bot configuration issues rather than code level issues. DA staff need close coordination with the core devs working on the postgres issues to nail this down.

The primary focus for DA staff is now on:
https://www.drupal.org/node/2534132

DrupalCI and the old PIFT/PIFR testbots are running in parallel. This is great for validating that the new bots work at least as well as the old bots, and to ensure that we are feature complete before disabling the old bots, but it does represent a doubling in testing costs for the DA.

The six issues identified at the top of the issue summary will allow us to disable the PIFT/PIFR D8 testing bots only - IF and only if we can complete those issues fast enough allow D8 core and contrib devs to validate that the new bots obviate the old ones. That means a deadline of around September 9th which is quite ambitious. If we can't reach that point we'll likely be testing on dual systems through BCN and absorbing the costs.

• #2538160: Make e-mail notifications much more betterer
  • Drumm is working on these now.
  • Any user for any test job will be able to subscribe to notifications about that test.
• #2518056: Need the ability to cancel a test job
  • Drumm is also working on this functionality
  • The test runner code for this is in place, it is just the D.O ui for this that needs to be added.
  • Permission to cancel a test will be given to the person who created it and to project maintainers.
• #2538462: Contrib Testing Test discovery not working
  • PIFT/PIFR contains a work around for poor test discover in core run_tests.sh
  • Project dependency handles the side of making sure that all necessary submodule code is checked out to run tests, but the test discovery piece itself does not work.
  • jthorson has started a patch for a core solution in #2551981 - Add --directory option to run-tests.sh test discovery
  • mixologic is working on a workaround until we can get a core patch in place.
• #2534144: Patches that fail to apply should show as fails on drupal.org
  • jthorson has done a code refactor in issue #2558929 - Testrunner code cleanup and refactoring
  • This should allow for generic build step failures to be caught as failures.
• #2538478: PHP Fatal Errors during unit tests should show as a failure
  • This is another case where ideally the fix would be in core - see issue #2560643
  • We can work around this with log parsing in the same way that pift/pifr does temporarily.
• #2560555: Issues should be set to "needs work" when patches fail DrupalCI tests
  • This simply needs to be swapped over once the DrupalCI D8 tests become the canonical tests.

We spent a bit of time discussing some general architectural issues: 1) That there are a number of 'work-around' solutions that ideally we would not be replicating - but because the run_tests.sh is embedded in core it can be hard to push through changes - under less time pressure that might not be such an issue 2) Because we want to be able to push new features a model where run_tests.sh did not have to be embedded in core would allow for more rapid feature adds in future. Needs much more thought.