Defining the Drupal 8 testing gate

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Hi there, testing team! :)

Back at DrupalCon Chicago, Dries outlined a strategy for Drupal 8 involving a series of "gates" that would help ensure core code quality in a number of different categories: Documentation, Performance, Accessibility, Usability, and Testing. The purpose of gates is to define essentially a set of "checkboxes" which outline the most important aspects to each category, and does so in a way that is both not overwhelming to core developers (list of hoops to jump through is kept as small as possible) and also creates very little additional burden on the team in question (contains sample links to documentation/resources so developers/reviewers can help themselves).

Since we have already traditionally had requirements around documentation, it made sense to start there. Jennifer Hodgdon and some other folks from the documentation team have put together the Documentation gate, which is available at http://drupal.org/node/1203498. What we need is a similar table for each of the other gate areas. And that's where you come in! :D

So I'd love to kick off a brain-storm about what the top testing gates might be (please, no more than 5 if possible), with the goal being to fill out the following table. (This is a wiki page, so be bold!)

Description When is this needed? Details Resources
Check test coverage and ensure all tests pass When changing/refactoring existing code When making changes to existing code, ensure the code being worked on has test coverage. This will be the case if some tests fail during the course of working on the patch, so may not need to be checked manually. No patch can be marked RTBC if there are outstanding test failures, since core has a 100% pass rate policy. http://qa.drupal.org
Add new tests When adding new features When new features are added, they should be accompanied by automated tests. If the feature does not have an implementation, either unit tests, or a test module that demonstrates the functionality can be provided. http://drupal.org/simpletest
Upload a test case that fails When fixing bugs in PHP code Bug fixes should ideally be accompanied by changes to simpletest (either modifying an existing test case or adding a new one) that demonstrate the bug. This can be shown by uploading a patch with only the test changes, which Drupal.org's automated testing system should show as a fail, alongside a patch containing both the tests and the bug fix itself. If a bug fix is not accompanied by tests, there should be an explanation in the issue for why no tests are needed (usually only for trivial bugs where testing would be convoluted) before it is marked RTBC. Patches that need tests written can have the 'needs tests' tag added. http://drupal.org/simpletest, http://qa.drupal.org, http://drupal.org/project/issues/search/drupal?status[]=Open&version[]=8.x&issue_tags=Needs+tests
Manually test in browsers and provide screenshots or screencasts When making markup, CSS or JavaScript browsers Drupal core does not have automated acceptance or unit testing for non-PHP code. In these cases changes to CSS, markup or JavaScript should normally be accompanied by confirmation that this has been tested in the browsers Drupal core currently supports. For large changes or bug fixes, screenshots or screencasts are ideal for demonstrating the impact of the changes. ...
Provide an example module When a new feature is not implemented by Drupal core When adding major new features that are not implemented by core (i.e. API-only features), example code should be provided that demonstrates the feature. This may take the form of a test module, but could also be a contrib sandbox or project, or follow-up core patches that introduce the functionality (and can be tested as proof of concept prior to committing the feature).
Title

Comments

Let's get this one

catch's picture

Let's get this one started:

When fixing bugs, write a test that exposes the bug. If the bug is not worth adding a test for (generally only the case if the test would be very convoluted compared to the severity of the bug) explain why in the issue.

It is common practice to attach a patch to the issue that only contains the test, so that there is a clear record in the issue that the test will fail without the patch applied. It's easy to write tests that don't actually test the exact code being changed even if visually they look fine, and this practice protects against false negatives.

Issues that only refactor existing code should either add tests if there is no code coverage, or point to existing tests where they exist.

Issues that add new features should add tests for those features.

Demonstrate the feature with a module or other code

rfay's picture

In the heat of Drupal 7, lots of things went into core that had no practical demonstration, and as a result we didn't find out how totally broken they were until contrib modules finally started to use them. An example of this is #states... It went in with many, many bugs, nobody really understanding how to use it (or perhaps even what it was).

I'm all in favor of demonstrating things with tests, where that works, but often a test is aimed at breaking a piece of code, not expressing to others what it can do.

I'd propose some sort of gate for nontrivial new features where a demonstration contrib module is prepared in the process of getting the feature committed. Now that we have sandboxes and git that should be easy enough. And I don't object to a mock module (for testing) being the demonstration, as long as it actually shows what the new feature can do in a way that people can understand it (before the feature goes in!)

Demonstrating features with

boombatower's picture

Demonstrating features with tests sounds great so long as there is a note in the related API's documentation that say go read X test. The table looks thorough.

Demonstrate with a mock module

rfay's picture

I was actually suggesting that we demonstrate with a mock module. I probably wasn't clear about that. Demonstrating with a test alone is not very useful. We want code that uses a feature and makes it obvious that it's been tried.

Makes sense, but from my

boombatower's picture

Makes sense, but from my browsing of Drupal 7 APIs even when their is a good test or even mock module there is rarely a note in the documentation. Having the combination of mock module and reference in docs would be awesome.

Great start!

webchick's picture

Some comments...

  • "Check test coverage and ensure all tests pass" - do we have a handbook page somewhere that talks about the testbot and how it works? http://qa.drupal.org is a little bit too vague of a resource
  • "If the feature does not have an implementation, either unit tests, or a test module that demonstrates the functionality can be provided." I don't understand that. Do I need new tests with a new feature or not?
  • "Manually test in browsers" needs more work. What browsers do we support? What UI features are likely to break when things like jQuery/AJAX system are updated? I think we need a list like http://groups.drupal.org/node/5974#javascript to outline what these things are.
  • "Example module" could use an example. :) Does Examples module provide a template that would be useful for people trying to write demo modules? Might as well kill two birds with one stone...
  • http://drupal.org/simpletest is a pretty overwhelming resource. Do we have something in that section that would be a simple one-pager or similar to discuss the fundamentals, or is there a section of that handbook we could link to more specifically?

Testing Markup

Shyamala's picture

Testing Markup, js and css, js and ajax:

  • Use of appropriate validators to meet W3C standards
  • Review mark up for accessibility using tools like Wave

Where a know change in the Markup is envisaged, a visual check in terms of alignment must be carried out for IE 8 and above, Firefox, Chrome, Safari and in Mac OS.

A good resource that talks about theming standards: http://drupal.org/node/37156

Documentation Resource for Simple tests: http://drupal.org/simpletest

type and quality of tests

beejeebus's picture

not sure if this is out of place, but: can we start to address the type and quality of tests as well here?

a couple of things that IMHO would give us greater value from our tests in D8:

prefer unit tests over web tests

this means asking questions like "is it possible to mock up the necessary state to test this functionality without installing a child drupal?". we have to be sensible here, because part of the issue here is that so much of drupal is just too randomly coupled with other parts of drupal, but we have to start somewhere.

to be clear what this is about - we want this question to be asked not primarily because we want more unit tests, but because we want to change the way we code drupal so its easier to test.

prefer the testing profile to the standard profile

apart from the fact that this makes things faster, this follows from the first point - "what is the smallest possible state on top of a child drupal needed to test this functionality?".

hmm

catch's picture

There's two ways of looking at the gates:

  • a list of things that need to happen during the lifecycle of an issue so it can be marked fixed.
  • a quality bar for the actual code in the patches.

Currently between the various gate discussions, there are combinations of the two. I think it would probably better to only have the gates for the first, but try to work on documentation for things like test quality, performant/scalable code etc. to the extent they can be qualified as a separate project same as we have for code style (which isn't a gate).

Part of the problem with the quality issue is that core doesn't meet its own standards (or nascent standards) in so many places (especially test quality and performance).

But I don't know what everyone else thinks about this, it really needs a collective discussion between all the people who are involved in drafting the gates, but that hasn't happened yet, but IMO it's going to be really tough for people trying to meet all these if the gates themselves are inconsistently worded and have very different emphasis.

new "Testing system" tag

beejeebus's picture

just note that sun and i decided to make a new tag "Testing system" to track improvements to the underlying testing system through D8.

http://drupal.org/project/issues/search/drupal?issue_tags=Testing%20system

Possible to wrap up by Friday?

webchick's picture

Since Dries wants to announce these gates at DrupalCon London, which is in 3 weeks, we basically need to wrap this up by the end of the week in order to give ample time for review and revision. Does this sound doable? How can I help?

Moved to http://drupal.org/core-gates

webchick's picture

Thanks so much for your hard work on this, everyone. This is a really solid first stab, so I went ahead and moved it to http://drupal.org/core-gates, where we can always refine it later if necessary.

Testing and Quality Assurance

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: