Testing Infra Evolution - Requirements

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!
Posted by jthorson on April 22, 2012 at 4:50am
Last updated by jthorson on Sun, 2012-05-06 01:48

Purpose

This page is intended as a collection point for requirements for the next generation of Drupal.org's automated testing environment. Please feel free to update this page with your own thoughts and ideas, and help contribute to the requirements which will define the next (r)evolution of Drupal's testing infrastructure.

Components

Currently, we do not anticipate a significant diversion from the current model, which includes three main sites:

  • Drupal.org:
    • Used to initiate 'automatic' and 'on-demand' testing
    • Communicates test requests and retrieves test results from the 'test dispatcher' site
    • Provides feedback on results via the 'automated testing' tab, in issue threads, and possibly on project pages
    • Stores 'summary' results for each test
  • QA.Drupal.org
    • Acts as the central 'registration agent' for individual testbots, and maintains information regarding individual testbot capabilities and environment
    • Periodically validates individual testbot operation, proactively identifying issues with testbot sanity, disabling suspect testbots, and flagging potential issues for investigation by the testing infrastructure team
    • Receives test request information from Drupal.org
    • Validates incoming test information for syntax and sanity
    • Stores valid test requests, generating and managing a 'test queue' for the testbots
    • Adds additional test information and additional test parameters as may required by the testbots
    • Acts as the 'test dispatcher' for testbots, delivering test information and parameters to individual testbots in response to incoming test queries
    • Receives test status and results (both summary and detailed) from bots during and upon completion of test execution
    • Feeds test status and summary results back to Drupal.org
    • Stores test status, summary, and detailed results for each test, providing users a means of retrieving historical test result information as needed
  • Testbot:
    • Registers with the qa.d.o component, and provides status on availability to receive tests
    • Polls the qa.d.o component for new test requests. (The poll method is used to ensure that testbots can operate without being directly accessible via the internet.) (beejeebus: we can use a queue to push test requests to testbots without them being accessible from the internet.)
    • Receives testing parameters from the test dispatcher, prepares the testing environment, and executes the test
    • Provides test results (status and assertions) back to the test dispatcher

Testing Use Cases

In order to help identify the various requirements of the testing infrastructure, this section is intended as a repository for individual testing use cases. (This may eventually be split out into a seperate wiki page as the list grows, but feel free to add your use case ideas here for now.)

TODO: Elaborate on the following use cases, and define new ones:

  • Simpletest 'Branch test' use cases
    • Re-test branch in response to each git commit
    • On-demand tests, manually triggered on Drupal.org
  • Simpletest 'Patch test' use cases
    • In response to a patch being uploaded to an issue queue
    • Manually testing of a patch before uploading it to an issue queue
    • Testing a patch against a specific feature branch within a given project (i.e. non-release branches, which could by extension include sandbox projects)
    • Automatic periodic re-testing of RTBC patches (auto-flagging for re-roll if conflicts or failures encountered)
  • 'Project Quality' indicator use cases (both automatic and on-demand)
    • Git repository 'health check', validating code structure and branch/tag naming patterns
    • Licensing check, searching for 3rd party code and libraries within a repository
    • Coding Standards tests (eg. Coder, Drupal CodeSniffer, PAReview)
    • Security Scans (screening for XSS, SQL injection problems, etc.)
    • Performance Testing (xhprof, testing before & after patch, etc.)
    • Documentation validation (validate PHP docblocks, README files, code commenting, drupal.org project page, etc)
  • 'Utility' testing capabilities
    • Patch conflict determination (identify existing RTBC patches which conflict with a given patch, and would require re-roll if current patch is committed

Required Test Triggers

This section is intended as a collection of 'triggers' which would be required by the above use cases in order to initiate a given test flow.

TODO: Define additional triggers

  • Git post-commit hooks
  • Cron (for periodic retesting)
  • User initiated triggers (drupal.org forms/links)
  • Issue Queue triggers (patch upload to issue/comment)
  • Process initiated triggers (for example, triggered by the 'promote to full project' link)

Testing 'Building Blocks'

The testing process can be broken down into a number of individual 'building blocks', which may or may not be required for any given testing scenario. This section is intended as a collection point for defining the individual building blocks and testing functionality which may be required by the testing infrastructure.

TODO: Define additional 'Building Blocks'

  • Setup
    • Validate test parameters
    • Validate system requirements (e.g. internet access?)
    • Prepare a sanitized test environment
    • Create directories / directory structure
    • Assemble required sub-test components
    • Set Site / Environment Variables
    • Determine if any pre-test conditions need to be re-run (e.g. Re-scan project files?)
    • Define repository parameters
  • Fetch Resources
    • HTTP download of a given file
    • Retrieval of a local file resource
    • Extraction of one or more files from a local file archive
    • Repository checkout of Drupal Core
    • Repository checkout of a Contrib release
    • Repository checkout of a Core/Contrib 'feature branch'
    • Repository checkout of 'dependent' projects
  • Scan Resources
    • Validate patch file format
    • Compile file listing
    • Compile test class listing
    • Syntax checking of PHP files
    • Set resource attribute flags (e.g. file type, core vs. 3rd party, etc)
  • Pre-test Conditions
    • Apply patch
    • Remove patch
    • Validate whether patch applies
    • Copy/Delete/Move files
    • Ensure file exists/does not exist
    • Execute command
    • Apply resource mask (identify which file/test resources to include in test)
    • Install Drupal
  • Execute Test
    • Run test
    • Run simulated test
    • Prepare results
    • Store results
  • Report Results
    • Retrieve results
    • Screen results (are we only looking for a specific result type)
    • Determine Pass/Fail status
    • Determine Summary Result
    • Retrieve Detailed Result
    • Format results
    • Deliver results

User Interface Mockups

This section is intended as a collection point for links to testing-related user interface mockups, which illustrate potential implementations for the drupal.org test triggers, qa.d.o results displays, and testbot user interfaces.

TODO: Define additional mockup links

Administration Capabilities and Instrumentation

This section is intended as a collection of capability requirements surrounding the instrumentation and administration capabilities needed on the testbots.

  • Enable/Disable a testbot:
    • From the testbot gui
    • From the qa.d.o gui
    • When idle
    • While testing
  • Blacklist any given test and prevent it from running
  • Support 'Private' tests (i.e. test/results not public) for things like security fixes