HOWTO: Submit tests with your patch

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Note: This document is under development and targeted towards Drupal 7

What is Automated Testing?

Automated testing encompasses UI, API and unit tests. It's a technique for testing small sections of code to ensure it works as expected. For example, a UI test might check a form submission process to ensure form display, validation of input and submission works as expected. A unit test might test a function for checking street addresses by passing a variety of well-formed and mal-formed addresses to ensure it handles them properly.

Automated tests are used to aid debugging, ensure compatibility with other elements of the application, and assist in maintaining backward compatibility as code develops.

Why Test?

  • Modules and patches can be quickly reviewed and committed.
  • Tests provide insight into how code is expected to perform.
  • Tests help avoid introducing bugs when developing new features or modifying existing code.
  • Code that has been developed with testing in mind tends to be easier to maintain and understand.
  • Bug patches submitted with a test ensure the bug remains fixed.

SimpleTest: A Tool for Unit Testing

Drupal comes with the SimpleTest module, which is based on the SimpleTest framework. The SimpleTest module is off by default and can be enabled through the module administration screen, although it's recommended that the SimpleTest module only be enabled on development systems, as unit tests may affect data in unexpected ways.

Types of Tests

There are three types of tests:

  • UI tests: These are "black box" tests that simulate the user clicking on various links and buttons in order to test module functionality.
  • API tests: For API functions that aren't easily represented in the UI, functions that are called by other modules, and so on, API-level tests may also be written. These are "white box" tests which will check database state and so on in order to ensure that APIs are working as expected. They can also serve as a form of documentation for how the API is supposed to behave, as they serve as examples of how functions are supposed to work.
  • Unit tests: These types of tests test a part of the system in isolation from other parts so that system inter-dependencies don't come into play. In most cases, this requires mocking functions (via runkit or other methods).

The Drupal community is initially focusing on UI and API testing in order to get maximum coverage in those areas, with as low a barrier to writing and running tests as possible.

This tutorial will focus on writing both API and UI level tests.

Where do test files go and what do I call them?

Most of the time, you want to test a single module, in which case you simply create a file called module.test in the module's directory.

One exception to this rule is for various core library files, such as those files found in the /includes directory. For these you would create a file called filename.test in the /includes directory.

Modules that wish to have multiple test files (for example, both a .test file for the module itself and a .test file to ensure the module conforms to external RFCs or specifications) may also create a "tests" subdirectory and place multiple .test files within it.

Taking a tour: What's in a test file?

TestCase classes (test groupings)

Each .test file must contain one or more TestCase classes which hold groups of tests. TestCase classes appear in the testing module user interface, checkboxes for each class allow different combinations to be run for each test run.

An example TestCase class declaration is:

<?php
class ModuleNameTestCase extends DrupalWebTestCase {
  ...
}
?>

A typical simple module can get away with having only one TestCase class to hold all of its tests. However, there are circumstances where you may want to add additional TestCase classes:

  • If it's conceivable that someone would want to test different parts of your module independently of one another; for example, it makes sense to have different test groupings for system module's module-related tests vs. its theme-related tests.
  • If there would be performance impacts of running all of the tests at once (for example, all node module or all database system tests), it makes sense to split them up into separate groupings.
  • If the module just plain does a lot of stuff and you want to break things up a bit to make them more manageable, additional test groupings also can make sense.

When creating multiple testing groups in a single .test file, use a ModuleNameDescriptionOfTestTestCase naming convention instead (for example, DatabaseSelectTestCase, DatabaseFetchTestCase, DatabaseInsertTestCase, etc.).

getInfo() method (metadata about test group)

Each TestCase class requires a getInfo() method, which returns meta-data about the test: its name, description, and what "group" it belongs to (such as the module it's written for).

An example getInfo() method is:

<?php
 
// Inside TestCase class...

  /**
   * Implementation of getInfo().
   */
 
function getInfo() {
    return array(
     
// 'name' should start with what is being tested (menu item) followed by what about it
      // is being tested (creation/deletion).
     
'name' => t('Noun verbion'),
     
// 'description' should be one or more complete sentences that provide more details on what
      // exactly is being tested.
     
'description' => t('Create a custom whoozit, add thingies to the stuff, and clean up afterwards.'),
     
// 'group' should be a logical grouping of test cases, like a category.  In most cases, that
      // is the module the test case is for.
     
'group' => t('Module'),
    );
  }
?>

Test methods (the tests themselves)

The vast bulk of .test files are the test methods, which contain the tests themselves. At the heart of all test methods are assertions.

A list of all available assertions can be found at http://drupal.org/node/265828

Here are example test methods in a typical TestCase:

<?php
 
// Inside TestCase class...

 
/<strong>
   *
One-sentence description of test.
   */
  function
testModuleNameTaskVariation() {
   
// Code that does something to be tested.

    // Test that the code did what it was supposed to do.
   
$this->assertEquals($some_value, $another_value, t('Testing equality of some value and another value'));
  }

  /</
strong>
   *
One-sentence description of test.
   */
  function
testModuleNameTaskVariation() {
   
// ...
   
$this->assertNotEquals($some_value, $another_value, t('Testing inequality of some value and another value'));
  }
?>

Some guidelines about test methods:

  • Aim for each test method to test only one thing. This makes it much easier to hone in on what was being tested when failures appear. For example, rather than an enormous testUserLogin() function with several variations, instead do testUserLoginDuplicateUser() and testUserLoginRequiredFieldsMissing().
  • For UI tests, name the methods testModuleVerbVariation(), such as testUserLoginDuplicateUser().
  • For API tests, name the methods testFunctionNameVariation, such as testVariableGetMissingValue().

setUp()/tearDown() methods (setup and/or cleanup after tests)

setUp() and tearDown() are two optional methods that perform setup tasks and cleanup tasks, respectively. These methods run before and after each test method.

An example:

<?php
 
// Inside TestCase class...
 
/<strong>
   *
Implementation of setUp().
   */
  function
setUp() {
   
// The first thing a setUp() method should always do is call its parent setUp() method.
    // If you need to enable any modules (including the one being tested),
    // add them as function parameters.
   
parent::setUp('statistics');

   
// Next, perform any required steps for the test methods within this test grouping.
   
variable_set('some_variable');
  }

  /</
strong>
   *
Implementation of tearDown().
   */
  function
tearDown() {
   
// Perform any clean-up tasks.
   
variable_del('some_variable');

   
// The last thing a tearDown() method should always do is call its parent tearDown() method.
   
parent::tearDown();
  }
?>

If you only call parent: setUp(); and nothing else in your setUp() function, then you do not need to call tearDown() since SimpleTest module handles this basic cleanup automatically.

References on unit testing and Drupal:

Comments

"Documentation-driven development" ;)

webchick's picture

After writing the above, I came up with a list of things we need to decide on:
- Naming conventions for test files. module.test for module tests, system.filename.extension.test for non-module files, such as common.inc?
- Directory structure for tests. Looks like modules/X/tests/X.test, the only exception being system.module which will have things like modules/system/tests/system.common.inc.test?
- What classes do tests extend from? I put DrupalWebTestCase for UI tests, and DrupalTestCase for non-UI tests. Correct?
- Naming conventions for test cases. I put ModuleNameUITestCase and ModuleNameAPITestCase. We might need more though. For example, node.module is /extensive/ and testing all of its UI in one class might be insane.
- Naming conventions for test functions. I put testModuleNameVerb() for UI tests, and test_function_name() for API tests. Does this distinction make sense?
- A bunch of other things. See @todos.

We also desperately need API documentation to cover both what SimpleTest assertion functions are available, as well as what extra stuff DrupalTestCase and friends give us. These files aren't being parsed by API module. :(

Smoke and Integration tests?

IncrediblyKenzi's picture

This seems fairly comprehensive.

I have a question about smoke tests and integration tests.. How does one categorize tests that validate integration between two different elements? An example would be validating that a stock install of Drupal renders blocks in their appropriate places.

Thoughts? My thinking is a root /tests dir that contains auxiliary tests.

Good question...

webchick's picture

I'm loathe to make a top-level /tests directory because I might conceivably want to test privatemsg and buddylist module's integration, and I'd need to ask my users to move .test files up to the root /tests directory rather than leaving them put. Also, that intermixes "my" stuff (contrib modules I added) which should always be under the /sites directory with "Drupal's" stuff which is everything else.

chx and I kicked it around tonight, and think the best way to handle this (at least for now) is just stick integration tests into an existing module's test file (in your case, block; in my case, privatemsg). If things get too crazy, we can always revisit this.

Good call..

IncrediblyKenzi's picture

I agree 100%.. Could get a little muddled down the line, but good enough for now.

But it raises an interesting dillemma.. When I have an integration test (we'll use the buddylist and privatemsg example), how do I differentiate that from the other automated tests? I'm thinking of the case where I make a change to privatemsg and want to run all tests to validate my change.. Does that mean I need to have buddylist installed in order to kick off the test suite?

Not sure I have an answer for that one.

Separate class

Crell's picture

Put your buddylist-centric integration tests in a separate test class, BuddylistPrivateMsgIntegrationCase(), and have a setUp() task of module_enable('privatemsg'). Then if you know it's not enabled you can skip over that class and not run it. Or if it does try to run, you will get a bunch of errors on those tests, because that's exactly what should happen if you try to run that code when that module is not available. You can also then text what happens with the code in question when it tries to run without privatemsg enabled.

Does anyone have concerns

mfb's picture

Does anyone have concerns about namespace issues? For example would it make sense for all drupal core classes to have a "Drupal" prefix? Not that I'm a fan of long class names. (Too bad we're not requiring PHP 5.3 which adds namespaces.)

Not too severe..

IncrediblyKenzi's picture

I wouldn't be overly concerned about namespacing.. In this case we have some semblance of control over the environment, though I do see a case where there could be some overlap (say you're implementing against PHPMailer, for example). I think this is somewhat of an edge case, however.

well I don't think it's an

mfb's picture

well I don't think it's an edge case for a site to integrate drupal with other php apps and libraries. many php projects do use prefixes of some sort, but maybe that means drupal doesn't have to..

Integrate with other libraries, maybe...

webchick's picture

But other libraries whose tests are going to run via Testing module, I doubt. ;)

If it becomes a problem later, we can always do a quick rename without much hassle.

Crap-load of edits made

webchick's picture

Larry and Karoly spent a lot of time with me tonight hammering out some stuff. Particularly, the file naming convention part and everything to do with "What's in a test file" have been totally re-worked, and naming conventions on test methods have changed. Please look it over and provide feedback as to the sense-making of it.

I'm not sure when the next time is that I'll have 3+ hours to work on this; might be a couple weeks. :( But please continue to refine/improve and then shove it in the handbook when it's ready. I'll check in when I can.

Basic Review

boombatower's picture

I read of the documentation (haven't read through comments), and came up with the following notes. Overall the documentation looks great is should provide a nice overview for people.

  • Once testing.drupal.org is complete tests will be required with patches and if existing tests are broken by a patch intentionally the test will be required to be updated with the patch. This comes from discussion at the testing sprint.
  • Agree with UI tests instead of Functional tests.
  • Should the class name convention be changed to ModuleNameTestCase, as is the current convention in core?
  • The DrupalUnitTestCase was a starter for the unit testing plan that was abondon and was thus removed from core.
  • Theme related testing is a bad example since there are several things we should add to SimpleTest to make that easier to test themes. I have played with a few, but they need flushing out.
  • The name attribute in getInfo should be Module functionality or something standard if only one TestCase is being used. Other than that I think the standards are arbitrary.
  • The test method names for API functions were discussed and the consensus was test_function_name.
    Examples:
    db_query -> test_db_query
    _db_query -> test__db_query

    From the comment above

    Crell brought up the good point about test_function_name() being both a) inconsistent with naming conventions in every other unit testing framework/OOP package, and b) undesirable because it violates the "one method per thing you want to test" rule. So I've updated the standards. We still need to fill in the ??? though. -webchick

    I see that the convention will no longer be used, although I still think it makes the most sense. The next bullet may solve the issue raised.

  • The point was made that private, prefixed with _, functions shouldn't be tested since they are intended to be called by public functions. If we agree on that then the problem is alleviated.
  • A good reason for the need of a setup function is creating a test user with the same privileges for each test method and enabling required modules since the db is flushed each time.
  • The function module_enable can be used instead of drupalModuleEnable, but drupalModuleEnable acts like an assertion and is thus useful to assert that the module was in fact enabled or disabled if using the disable method.
  • I would recommend the block or path (although the path test is correct, but the code was broken) tests for an example since they are fairly simple.

lots of edits

catch's picture

I've made a bunch of edits, tidied up some paragraphs, got rid of all the inline discussion, removed stuff that looked deprecated, and done as many TODOs as I could etc. At least cosmetically, it's looking a lot tidier, hopefully I didn't introduce any errors.

If there's not major problems left with this, I think we should move this over to the handbook, then make edits directly there and/or issues in the documentation queue. IMO the general structure is good, but we should have the heavily commented example on a separate page otherwise this is going to get very long.

Any more feedback on this?

catch's picture

Any more feedback on this? I'm tempted to move what we've got over to the handbook if there's no serious objections - then we can still edit it once it there.

Also - after the commit of common.test there's a few patches coming in testing individual functions - format_size() valid_url() - since they're testing the one function, I think we should name them "valid_url()" instead of "Test url validation" or similar. "Test" is redundant, and using the function name makes it easier to find in the code, on api.drupal.org etc.

moved to handbook

catch's picture

We should keep discussing changes here, but I've posted this at http://drupal.org/node/273612

I'm a little confused about

clemens.tolboom's picture

I'm a little confused about test for the includes directory. Enabling simpletest gives me a XML-RPC test. But there is no xmlrpc.test file (EDIT: in the includes directory). Simpletest module has a xmlrpc_test.module (EDIT: and a xmlrpc.test). So how is that part working and how can I submit a patch for ie http://drupal.org/node/265973

Furthermore at DrupalCon I attend Testing Party - part 2 and tried to write a test not very successfully yet. To run this small test I had to run ALL menu.test tests. Which was a drag. I would like to suggest to use a base class with all helper functions and then extend this class with all the tests.

<?php
class MenuBaseTestCase extends DrupalWebTestCase {
 
// add all helper functions here
 
function addCustomMenu() { ...
  }
}
class
MenuTestCase extends MenuBaseTestCase {
  function
testMenu() {
  }
}
?>

This way I can write my own test and run (way faster) first and after finished merge it into the real test.

<?php
class Menu_293525_TestCase extends MenuBaseTestCase {
  function
testMainMenuAsSecondaryMenu() { ...
  }
}
?>

Does this make sense?

A lot of the tests need to

catch's picture

A lot of the tests need to be split up into helper classes + more smaller tests. I agree it's really annoying trying to add to something with a couple of hundred assertions (or for that matter using the test for debugging).

The modules inside the simpletest folder use the hidden property in .info - which allows you to implement hooks in a real module and test whether it works, without it showing up anywhere other than the simpletest interface.

How does one sort tests

sangbk's picture

I have an inquiry regarding smoke tests and joining tests.. How does one sort tests that accept combination between two distinct components? A sample would be approving that a stock introduce of Drupal renders hinders in their proper spots.