Performance of Behat tests

The point of testing is to cover you code you have written, as that is what you have the most control over.

Drupal is bad for this due to the way it works with a database, but also browser testing is slow as well. So try to keep you BDD tests just testing your code, not the Drupal system.

Konstantin did a talk which was quite a good explanation on why he dropped parallel runners:
http://www.meetup.com/BDDLondon/events/181454232/
They did film it, so not sure where that has gone.

For us, system tests and smoke tests are slow but we only spend 5mins doing it. Rest is all in code so very quick. Our full build (inc deploy to staging) is about 35mins.

This is something that Panopoly has been struggling with too, especially recently. We run our tests on Travis-CI, which has a 50 minute limit on test runs. So, we tend to go through cycles like: (a) tests take 20-30 minutes to run, (b) we add more tests, so they're now hitting the 50 minute limit, and can't add more tests, (c) we optimize something and return to (a). :-)

Since we're on Travis-CI, we've got some unique problems and limitations. For example, we can't throw hardware at the problem! We're stuck with what Travis-CI gives us. Also, Travis-CI rebuilds the environment in which the tests are run everytime, so we've got to deal with both setup time and it's a lot harder to optimize the environment the tests run in, since we can't just SSH in and tweak settings.

Here's a meta-issue that we've been using for the most recent round of optimizations:

https://www.drupal.org/node/2437927

To sort of summarize the types of things we've been trying so far:

• Optimizing the performance of the Drupal site in the normal ways. So, enabling APC, tweeking MySQL settings, enabling CSS/JS aggregation, etc.
• Profiling/optimizing our Behat steps and @afterStep, @afterScenario, etc. The last round of this reduced the test run time by about 10 minutes: https://www.drupal.org/node/2447839. We also recently had a Javascript error in WYSIWYG (unrelated to what we were testing) cause one of our tests to take an extra 15 minutes: https://www.drupal.org/node/2449495. These sort of inefficiencies sneak in over time, and unfortunately, we don't take the time to really dig into them until we hit that 50 minute limit.
• "Focusing" the tests on just what needs testing, and making "Given" steps for the rest. It's clicking through forms and stuff in Selenium that takes the most time for us. If we are testing using something or editing it, we don't need the test to step through the creation of the thing - we can bake that into a "Given" step and only test the actual creation one time.

In response to something you wrote above:

wherever we had 2 or more steps with "when I run drush..." right after each other we could save a fair bit of time combining those into just 1 step.

... and then:

changing our api_driver from "drush" to "drupal" (or vice versa) - in spite of docs saying one is faster, this didn't make a noticeable change for us

So, the reason two drush's takes so long, is because it's bootstrapping Drupal twice. If you change the api_drive to "drupal" and then it'll bootstrap Drupal once per Scenario, directly inside of Behat. So, if you do what you would do with drush directly in a "Given" step in your FeatureContext (because you now have access to the full Drupal API, in a fully bootstrapped Drupal site), it should definitely make your tests run faster!

Of course, this isn't just switching the api_driver - you also have to stop calling drush directly in your tests and write some custom "Given" steps. But it really sounds like you could make some performance gains there.

Things I'd like to experiment with in Panopoly in the future:

• Finding ways to test what we need to test without using Selenium. Unfortunately, Panopoly's whole experience depends on lots of Javascript magic (via Panels and CTools). So, even though what we're testing is just a Drupal form, you need to do a bunch of Javascript stuff for that form to even appear (ie. in a CTools modal). Maybe we could find some way to expose the form (just for the tests) without doing the Javascript stuff, and then run those tests with Goutte?
• Breaking up the tests in smaller units that can be tested independently. Panopoly is made up a bunch of Feature modules. We're currently testing them all together as the whole distribution. But if we can decouple the modules and tests enough to run them independently, then we can run just the tests that exercise "panopoly_admin", for example, and get much shorter test runs with the same feedback to the developer.

I'm interested to hear what other ideas other folks have to increase their test performance!

I've just finished a 9 month Laravel project where we used Behat extensively for functional tests. After mocking all APIs our test suite was still taking 30 mins to run. I switched the DB to SQLite and the execution time dropped to less than 4 minutes.
The application was much less DB intensive than Drupal, so I'm not sure how it will pan out using SQLIte as the Drupal DB in testing.

I've just started a Drupal 7 project which needs BDD tests so I'll try SQLite in the next few days and report back.

It was discussed during a BOF in Barcelona, that the current method of checking for a logged in user results in several unnecessary bootstraps since it is just checking for markup in a rendered page (eg, the logout link).

The proposed resolution here is to a) log the user in directly via session manipulation, and then b) use that session to check if a user is logged in or not. There is a somewhat abandoned PR here: https://github.com/jhedstrom/drupalextension/pull/131 and it might make sense to move much of that into the DrupalDrivers.