Drupal core test performance improvements

A first big milestone landed today:

#1373142: Use the Testing profile. Speed up testbot by 50%

All core tests are using the Testing installation profile now.

It improves the performance of Drupal core tests. Running all tests took 40+ minutes previously. Now it takes ~19 minutes.

These numbers only apply to Drupal 8, and only for testbots of drupal.org's testing infrastructure.

How is that possible?

For each test to be run, Simpletest installed Drupal with the Standard profile. That profile installs 27 modules in total, including their database tables and configuration. However, each test only needs the modules that it actually wants to test — which most often means only ~6 modules out of the 27.

By changing the default installation profile for running tests to the Testing profile, Drupal is installed with the bare essential modules only. Plus the module(s) that need to be tested in a certain test case.

Usage of Drupal's default installation profile was an overlooked mistake when Simpletest was introduced for Drupal core — it wasn't apparent until Drupal core had reached a critical mass of tests that had to be run for every single patch in the queue.

The additional modules and configuration not only decreased performance; they also made test authors write tests against add-on functionality, instead of the main functionality in question. So this is not only about performance, but also about proper separation of concerns — do not implicitly test irrelevant functionality that you shouldn't test.

To that extent, we already changed the default theme to Stark for Drupal 8, which means that all tests are running against the "un-themed" default output of Drupal core modules, instead of implicitly testing the Bartik theme.

Impact

All patches against Drupal 8 are tested twice as fast now.

Developers get testing results earlier, and thus, can fix bugs and change things twice as fast. It potentially means less context-switching, previously caused by having to wait for test results.

Drupal's Quality Assurance infrastructure needs less testing client servers, because the same amount of resources can handle twice of the load.

We are one (first) step closer to being able to run the full core test suite on local developer machines again. This was possible 2+ years ago, but required more than half a day to complete on otherwise highly performant machines recently. (Testbots are tweaked for performance in every possible way, which explains the huge difference.)

What about Drupal 7 and contrib?

Contributed modules (and custom projects) are able to leverage the Testing profile since Drupal 7.0. All you need to do is to override the installation profile to use:

Do yourself a favor and use it! :)

Best practice: Use a base MymoduleWebTestCase for all of your tests, which not only sets the $profile, but also unifies the setUp() and possibly existing test helper methods.

For Drupal 7 core, there's a good chance for being able to backport the testing profile patch.

For D6, there is no other installation profile, so make sure you do not commit the $profile property to D6 tests.

Are we done yet?

We're not done yet. The action plan:

1. Convert remaining tests

   Various test cases in Drupal core were simply too hard to convert for above mentioned monster patch, so they override the new default profile and still use the Standard profile instead. Fixing/converting those will likely boost up the bot by another ~3 minutes (16%).

2. Remove Node module from Testing profile

   In order to make the Testing profile the default, we had to add the Node module as dependency to the Testing profile. Too many tests assumed that Node was there. Removing that dependency and fixing those tests will likely mean another boost by ~2 minutes.

3. Replace required modules with proper dependencies

   A couple of Drupal core modules are marked as required, even though they're not. Tests for Toolbar module and others have absolutely no interest in Entity, Field, Text, Filter, and Field SQL storage modules, but since those are doomed to be bare essential modules, they have to be installed, and there's no way around that currently. Fixing those required but not really required modules will mean another nice boost for all tests that simply don't need 'em.

4. Only install Drupal once per test case

   The difference between test cases and test methods is nonsensical currently, because there is none. For each test method, Drupal is installed from scratch and destructed afterwards.

   #1411074: Question: How many tests will actually fail if setUp() is only executed once for a test case?

   investigates to change exactly that. The non-deterministic performance benchmark result of that patch revealed another 50% performance improvement compared to the current state.

   This will happen last, as the next big milestone.

/me still needs to create issues for most of these tasks...

See all Testing system issues for Drupal core.