Tests in core: What's next?

We now have a testing framework in Drupal core. What we need to move forward is some structure that makes writing tests easy and efficient so that we can go towards our 100% code coverage goal without wasting development resources.

After today's IRC conversation on the topic of unit testing, the decision was taken to implement unit tests the same way as functional tests are currently implemented, without mock objects or functional overrides at runtime (see this issue for an explanation: http://groups.drupal.org/node/10957). However, the discussion also touched on many key points that are relevant no matter what approach we take, so here's a summary for those who weren't there.

One of the main outcomes was the need for a better organization of the various kinds of tests that we have. With a clear structure in place, we could put test files in place with minimal code, thus making it easier for contributors to create patches without having to figure out where to add files or how to trick cvs into creating a patch for new files. These basic files would also make it easier to point people to simple tasks that can be processed in a short amount of time without the need for an extensive understanding of the code.

In order for tests to be easy to write, they must be easy to read and easy to understand. The existing tests are the best documentation that we have for contributors who are new to automated testing. We currently have many tests that try to cover a whole module's functionalities with a long list of assertions. The problem with such tests is that they make it hard to track down what is happening when a test fails (even if we know what assertion failed, we might need to debug the code that sets the fixture – the environment in which the test is run). It is also a lot easier to create a small test than it is to make a small addition to an existing mastodon. In other words, tests need to be refactored, just like code.

Besides reorganizing the structure of the tests, it is also important to make a distinction between the different kinds of tests. It's not because we are using the same mechanism to process “unit” tests as we do for functional tests that they are the same thing. We currently have tests that simulate specific sequences of user actions (like user.test), tests that simulate the execution of code with specific parameters (like search.test), and some that are somewhere in between (path.test). In order to better structure our tests, we need to be able to indicate what kind of simulation they perform. Should unit tests be in separate files, separate classes, or even simply separate methods? Wherever we put that separation, we also need to make sure that the naming conventions reflect it.

Finally, making Drupal testable requires refactoring code, and refactoring code breaks existing tests, but this is what is supposed to happen. The whole reason to use automated tests is to be agile and make this kind of overhaul possible by making us aware of how we break things. Converting Drupal to a fully tested framework is a big task, but the work we do now will make future work easier and our testing system (both the testing framework and the tests themselves) will get more stable, to the point where we can make big improvements without breaking everything.

Note: Credits for many of these points go to the people involved in the IRC discussion (mainly webchick and chx), but I've mixed in my own opinion so criticisms go to me.