Automated Coder Reviews for New Project Applications - Planning/Development

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

This wiki page has been set up to continue the 'Automated Coder' discussion initially started at http://groups.drupal.org/node/155084#comment-517764.

Development Tasks

  1. Update PIFR_Client to support manually running an independent 'Coder' review, seperate from SimpleTest. Needs to support both full and sandbox projects. Implementation will likely be through a new form/URL on testbot client implementations, and take project nid and branch as inputs.

    • In Progress: Can run 'Coder' independent of SimpleTest for Drupal Core versions, full projects, and sandboxes. Coder form built, which takes full Git URL and branch name as inputs. Had to hardcode removal of some simpletest-specific code in pifr_drupal.client.inc, so task will not be marked complete until after changes added to re-enable SimpleTest support.
  2. Enhance form in #1 to provide various configuration options for manual 'Coder' runs. Set Severity Level, which Coder Review tests to include, enable Coder_Tough_Love, etc.

    • In Progress: Can set which Coder Review tests to include; Severity level is still hard-coded in pifr_coder.client.inc.
  3. Enhance form in #2 to add support for other PIFR plugin types (default settings).

  4. Enhance form in #3 to add configuration options for other PIFR plugin types.

  5. Development Checkpoint: Discuss with rfay/boombatower regarding implementation (i.e. whether to integrate with existing 'canned' tests interface). Initial implementation in #1-4 above will be attempted with minimal impact on existing test client code; this item represents the integration step.

  6. Investigate extension of sectionalized test capability to PIFR_Server (may be some minor tweaks required to PIFR_Server and PIFR_Client classes to make some of the test array components optional ... for example, no need to calculate dependencies or build a $files[] array if only running a Coder test on a complete repository). Some of this may occur in earlier steps, as workarounds may be required to meet the goals of earlier phases.

  7. Development Checkpoint: Determine go-forward testbot strategy for Coder-only or 'custom' tests. (Mix in with existing tests or run on dedicated testbot?)

  8. Define d.o workflow/access/governance/business rules for extending 'sectionalized' test capability and triggering workflow for d.o.

  9. Duplicate manual test form built for testbot clients in #1-4 within PIFT, and provide d.o implementation described in #8.

  10. Develop drupalorg_projectapps module (functionality as outlined below), and trigger automated Coder Reviews for new Project Applications.

Questions

  • Do we want automated reviews for every issue opened in the queue?
  • Would we prefer a semi-automated approach which requires an applicant or reviewer to specifically trigger the automated code review process?
  • How would we want to trigger re-testing of automated code review runs? Would we leverage Issue statuses, similar to the Automated testing process? How would this work into our existing workflows?
  • We need to determine some workflow and governance around how and under what conditions we would trigger a Coder review request

Application Pre-Requisites for Leverage Existing qa.drupal.org Functionality

There are a couple of pre-requisites which would have to be satisfied in order to work with the current qa.drupal.org infrastructure:

  • The application needs to have a -dev release created (TODO: Check pift_cron_get_release() code, to see if having a -dev branch is enough to satisfy PIFT)
  • Contrib modules/applications with dependencies on other contrib modules can not be tested (Global limitation of the current infrastructure)
  • Currently, the re-review would be triggered on patches posted to issues in the project issue queue, or on the next/subsequent git commits. We'd need to build our own custom function for the re-triggering of tests based on updates within the 'Project Application Queue' (which isn't actually associated with the project we want tested).

Proposed Approach - Drupalorg_ProjectApps Module

With the above pre-requisites in mind, I cut a first draft of a potential approach, based around the creation of a dedicated drupalorg_projectapps module, to host code related specifically to the Project Application process.

  1. Adds the 'Age' column to the '/project/issues/projectapplications' and 'project/issues/search/projectapplications' views.

  2. Creates a new 'drupalorg_projectapplications' table to track project application status.

    • Fields: Application Nid, Project Nid, Project Title, Repo Id, Repo Link, Release Exists, CoderCheckPassed, LicenseCheckPassed, SecurityCheckPassed, Closed Date
  3. Acts on newly created applications in the 'projectapplications' issue queue:

    • Create a new {drupalorg_projectapplications} entry to represent the application
    • Parse the application for a 'sandbox' link.

      • If a sandbox link can not be found:

        • Auto-posts a "Please provide your sandbox link" comment in the issue
        • Sets issue status to 'needs work'
      • If a sandbox link can be found:

        • Parse it to determine the NID for the sandbox project
        • Determine the project title name (node title from {node})
        • Determine the corresponding repo ID from {versioncontrol_project_projects} (20520)
        • Updates {drupalorg_projectapplications} with appropriate Nid/Title/Repo information
        • Check for a '-dev' release branch (or better) by querying {versioncontrol_labels} with the repo_id (and type = 2)

          • If no dev release found:

            • Adds a 'Please create a -dev release in your repository to facilitate automated testing' comment to the issue queue
            • Sets the issue status to 'needs work'
          • If dev release (or better) found:

            • Updates {drupalorg_projectapplications} 'Release Exists' value
            • Adds a 'Queuing for qa.drupal.org testing' comment to the issue queue
            • Adds tag "PAR:RequestQATestRun"
  4. Acts on newly created comments in the 'projectapplications' issue queue

    • Checks for issue state changes

      • Issue change to 'Needs Review':

        • If {drupalorg_projectapplications}.ProjectNid not set:

          • Parse application sandbox comments (most recent comment first), looking for a sandbox link. If found, follow the steps outlined in #3 above
        • ElseIf {drupalorg_projectapplications}.ReleaseExists not set:

          • Check for a '-dev' release branch (or better) by querying {versioncontrol_labels} with the repo_id (and type = 2), and react as per the steps in #3 above.
    • Check for specific PAR (Project Application Review) tags:

      • "PAR:RequestQATestRun":

        • Ensure state is set to 'needs review' (i.e. don't queue test for a 'needs work' application)
        • Trigger one-time, on-demand test request to qa.drupal.org
        • Upon test return, clear flag
      • "PAR:CoderCheckPassed" (ie. Reviewer manually ran the coder check):

        • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
        • Update {drupalorg_projectapplications}.CoderReviewPassed for the application
        • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)
      • "PAR:CoderCheckFailed" (ie. Reviewer manually reversing Coder Pass Flag):

        • Clear {drupalorg_projectapplications}.CoderCheckPassed if TRUE.
      • "PAR:LicenseCheckPassed" (ie. No 3rd party licensing issues found/remaining)

        • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
        • Update {drupalorg_projectapplications}.LicenceCheckPassed for the application
        • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)
      • "PAR:LicenseIssueFound"

        • Clear {drupalorg_projectapplications}.LicenseCheckPassed if TRUE.
      • "PAR:SecurityCheckPassed" (ie. No Security Issues found)

        • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
        • Update {drupalorg_projectapplications}.SecurityCheckPassed for the application
        • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)
      • "PAR:SecurityIssueFound"

        • Clear {drupalorg_projectapplications}.SecurityCheckPassed if TRUE.
  5. Adds a table with project application status (ie. based on {drupalorg_projectapplications} columns) to the project application issue view, providing a single-table summary of Coder, Licence, and Security check status.

PIFT/PIFR Testing Enhancements/Integration

Options:

  • Add capability for PIFT to trigger on-demand tests for sandbox projects which aren't enabled via the pift_projects table. (How? Button on sandbox page? Specific Tag in Project Issue Queue? Menu callback w/ confirm form?)

    • Dependent on being able to pull a release from a sandbox page (ie. is having a 'branch' enough?)
    • Trigger would need to validate a branch is present, and that there are no contrib dependencies.
  • Send test requests directly to PIFR, bypassing PIFT

  • Build a manual interface directly on testbots (as per http://drupal.org/node/1176982), and send sandbox tests directly to a testbot (bypassing both PIFT/PIFR)
  • Implement a specific testbot environment that only performs coder reviews, and a new PIFT_TEST_* type; as well as a mechanism to tell PIFT/PIFR to send this particular test type only to that testbot

    • Benefit: Coder can run on un-enabled modules, so this gets around any issues with calculating and downloading dependencies.