Hi there, testing team! :)
Back at DrupalCon Chicago, Dries outlined a strategy for Drupal 8 involving a series of "gates" that would help ensure core code quality in a number of different categories: Documentation, Performance, Accessibility, Usability, and Testing. The purpose of gates is to define essentially a set of "checkboxes" which outline the most important aspects to each category, and does so in a way that is both not overwhelming to core developers (list of hoops to jump through is kept as small as possible) and also creates very little additional burden on the team in question (contains sample links to documentation/resources so developers/reviewers can help themselves).
Since we have already traditionally had requirements around documentation, it made sense to start there. Jennifer Hodgdon and some other folks from the documentation team have put together the Documentation gate, which is available at http://drupal.org/node/1203498. What we need is a similar table for each of the other gate areas. And that's where you come in! :D
So I'd love to kick off a brain-storm about what the top testing gates might be (please, no more than 5 if possible), with the goal being to fill out the following table. (This is a wiki page, so be bold!)
| Description | When is this needed? | Details | Resources |
|---|---|---|---|
| Check test coverage and ensure all tests pass | When changing/refactoring existing code | When making changes to existing code, ensure the code being worked on has test coverage. This will be the case if some tests fail during the course of working on the patch, so may not need to be checked manually. No patch can be marked RTBC if there are outstanding test failures, since core has a 100% pass rate policy. | http://qa.drupal.org |
| Add new tests | When adding new features | When new features are added, they should be accompanied by automated tests. If the feature does not have an implementation, either unit tests, or a test module that demonstrates the functionality can be provided. | http://drupal.org/simpletest |
| Upload a test case that fails | When fixing bugs in PHP code | Bug fixes should ideally be accompanied by changes to simpletest (either modifying an existing test case or adding a new one) that demonstrate the bug. This can be shown by uploading a patch with only the test changes, which Drupal.org's automated testing system should show as a fail, alongside a patch containing both the tests and the bug fix itself. If a bug fix is not accompanied by tests, there should be an explanation in the issue for why no tests are needed (usually only for trivial bugs where testing would be convoluted) before it is marked RTBC. Patches that need tests written can have the 'needs tests' tag added. | http://drupal.org/simpletest, http://qa.drupal.org, http://drupal.org/project/issues/search/drupal?status[]=Open&version[]=8.x&issue_tags=Needs+tests |
| Manually test in browsers and provide screenshots or screencasts | When making markup, CSS or JavaScript browsers | Drupal core does not have automated acceptance or unit testing for non-PHP code. In these cases changes to CSS, markup or JavaScript should normally be accompanied by confirmation that this has been tested in the browsers Drupal core currently supports. For large changes or bug fixes, screenshots or screencasts are ideal for demonstrating the impact of the changes. | ... |
| Provide an example module | When a new feature is not implemented by Drupal core | When adding major new features that are not implemented by core (i.e. API-only features), example code should be provided that demonstrates the feature. This may take the form of a test module, but could also be a contrib sandbox or project, or follow-up core patches that introduce the functionality (and can be tested as proof of concept prior to committing the feature). | |
| Title |
Comments
Let's get this one
Let's get this one started:
When fixing bugs, write a test that exposes the bug. If the bug is not worth adding a test for (generally only the case if the test would be very convoluted compared to the severity of the bug) explain why in the issue.
It is common practice to attach a patch to the issue that only contains the test, so that there is a clear record in the issue that the test will fail without the patch applied. It's easy to write tests that don't actually test the exact code being changed even if visually they look fine, and this practice protects against false negatives.
Issues that only refactor existing code should either add tests if there is no code coverage, or point to existing tests where they exist.
Issues that add new features should add tests for those features.
Demonstrate the feature with a module or other code
In the heat of Drupal 7, lots of things went into core that had no practical demonstration, and as a result we didn't find out how totally broken they were until contrib modules finally started to use them. An example of this is #states... It went in with many, many bugs, nobody really understanding how to use it (or perhaps even what it was).
I'm all in favor of demonstrating things with tests, where that works, but often a test is aimed at breaking a piece of code, not expressing to others what it can do.
I'd propose some sort of gate for nontrivial new features where a demonstration contrib module is prepared in the process of getting the feature committed. Now that we have sandboxes and git that should be easy enough. And I don't object to a mock module (for testing) being the demonstration, as long as it actually shows what the new feature can do in a way that people can understand it (before the feature goes in!)
Demonstrating features with
Demonstrating features with tests sounds great so long as there is a note in the related API's documentation that say go read X test. The table looks thorough.
Demonstrate with a mock module
I was actually suggesting that we demonstrate with a mock module. I probably wasn't clear about that. Demonstrating with a test alone is not very useful. We want code that uses a feature and makes it obvious that it's been tried.
Makes sense, but from my
Makes sense, but from my browsing of Drupal 7 APIs even when their is a good test or even mock module there is rarely a note in the documentation. Having the combination of mock module and reference in docs would be awesome.
Great start!
Some comments...
Testing Markup
Testing Markup, js and css, js and ajax:
Where a know change in the Markup is envisaged, a visual check in terms of alignment must be carried out for IE 8 and above, Firefox, Chrome, Safari and in Mac OS.
A good resource that talks about theming standards: http://drupal.org/node/37156
Documentation Resource for Simple tests: http://drupal.org/simpletest
Shyamala
Unimity Solutions
type and quality of tests
not sure if this is out of place, but: can we start to address the type and quality of tests as well here?
a couple of things that IMHO would give us greater value from our tests in D8:
prefer unit tests over web tests
this means asking questions like "is it possible to mock up the necessary state to test this functionality without installing a child drupal?". we have to be sensible here, because part of the issue here is that so much of drupal is just too randomly coupled with other parts of drupal, but we have to start somewhere.
to be clear what this is about - we want this question to be asked not primarily because we want more unit tests, but because we want to change the way we code drupal so its easier to test.
prefer the testing profile to the standard profile
apart from the fact that this makes things faster, this follows from the first point - "what is the smallest possible state on top of a child drupal needed to test this functionality?".
hmm
There's two ways of looking at the gates:
Currently between the various gate discussions, there are combinations of the two. I think it would probably better to only have the gates for the first, but try to work on documentation for things like test quality, performant/scalable code etc. to the extent they can be qualified as a separate project same as we have for code style (which isn't a gate).
Part of the problem with the quality issue is that core doesn't meet its own standards (or nascent standards) in so many places (especially test quality and performance).
But I don't know what everyone else thinks about this, it really needs a collective discussion between all the people who are involved in drafting the gates, but that hasn't happened yet, but IMO it's going to be really tough for people trying to meet all these if the gates themselves are inconsistently worded and have very different emphasis.
new "Testing system" tag
just note that sun and i decided to make a new tag "Testing system" to track improvements to the underlying testing system through D8.
http://drupal.org/project/issues/search/drupal?issue_tags=Testing%20system
Possible to wrap up by Friday?
Since Dries wants to announce these gates at DrupalCon London, which is in 3 weeks, we basically need to wrap this up by the end of the week in order to give ample time for review and revision. Does this sound doable? How can I help?
Moved to http://drupal.org/core-gates
Thanks so much for your hard work on this, everyone. This is a really solid first stab, so I went ahead and moved it to http://drupal.org/core-gates, where we can always refine it later if necessary.