Testing and Quality Assurance

Here is an effort at documenting Drupal testing patterns from what's currently used in core tests. The objective is to promote reusability of standardized patterns throughout core and contributed tests.

// Catch - this looks good, but I think we should completely ignore stuff like getContent() which is used once per year and just focus one what 70% of basic tests will use - the API docs etc. are there, and linkable at the end, for everything else.

It stands to reason that the tests in simpletest/tests will be the ones that people will use to base their own tests. These tests were written by at least 5-6 different people, and it shows. As a result, it's really confusing for a new developer to get a sense of what your own module's SimpleTest integration ought to look like.

http://acquia.com/latest-drupal-test-results

The results are updated around 3 AM Eastern Daylight Timezone every night.

Acquia is a new software company looking for a QA AUTOMATION ROCKSTAR to work in our Andover, MA office as a part of our amazing development team providing value added software products and services for Drupal's open source social publishing software!

Here are the results of running our whole test suite on PostgreSQL 8.3.

• Actions configuration 30 passes, 8 fails, 0 exceptions
  See http://drupal.org/node/261859
• Block functionality 57 passes, 1 fail, 0 exceptions
  No known issue (yet!)
• Blog API functionality 45 passes, 2 fails, 1 exception
  No known issue (yet!)
• Search engine ranking 21 passes, 5 fails, 20 exceptions
  See http://drupal.org/node/296624
• Module list functionality 53 passes, 4 fails, 0 exceptions
  No known issue (yet!)

In porting the Pathauto tests from the Simpletest 6.x-1.x to 6.x-2.x style I found a few things I thought I would document here in the hopes that they benefit others:

1. Put the .test file into the root directory of your module. If you had it in modules/tests/module.test it now belongs in modules/module.test
2. DrupalTestCase is now DrupalWebTestCase
3. get_info() is now getInfo()
4. in getInfo, desc is now description
5. $this->drupalCreateUserRolePerm is now $this->drupalCreateUser(
6. drupalLoginUser is now drupalLogin

Splitting the discussion from here:

http://lists.drupal.org/pipermail/development/2008-August/030879.html

afaik, the steps to get this running are:

1. Abstract comment creation functionality in project_issue.module - http://drupal.org/node/271216
2. Allow testing.drupal.org to post back to issues and mark as code needs work when patches don't apply.
3. Get all known failures cleared
4. Allow testing.drupal.org to post back to issues and mark as code needs work when core tests fail
5. Extend this to Drupal 6

Submitted for perusal by the group, as passed to me by a fellow developer.

"[Google ratproxy is a] semi-automated, largely passive web application security audit tool, optimized for an accurate and sensitive detection, and automatic annotation, of potential problems and security-relevant design patterns based on the observation of existing, user-initiated traffic in complex web 2.0 environments.

Great news! The Awesome Testing Party session was accepted at Drupalcon Szeged! Woohoo!! It'll be at 9am on August 28th, but people should probably be there a bit earlier, like 8:30am.

There's lots to prepare, and very little time to do it in! We had a discussion today in #drupal to talk over logistics. Here's where we stand. Please strike things as you get them done. Thanks!

Stuff for lead-up to the event

• Write a testing hand-out that summarizes stuff from the intro to testing session (flobruit)
• Do a PR blitz to Drupal Planet about the party, talk about its benefits and goals, how it's going to work, and ask Szeged attendees to bring a box of chocolate with them (webchick) - http://webchick.net/awesome-testing-party
• Re-run test coverage scripts to see where we stand (catch) - http://coverage.ca.tchpole.net/coverage/html/1
• Do a screencast about how to write tests (sdboyer)
• Create a .test template that people can copy/paste from (webchick)
• Create issues for each candidate test and title them TestingParty08: Description of test" (boombatower and catch) - list here
• Write up a hand-out describing how testing party will work for participants (webchick)

Stuff for the day of the event

• Look into getting Hungarian pancakes for breakfast ;) (chx)
• Make lots of photocopies of the testing handout to bring with (webchick)
• Go buy some index cards and write all the TestingParty08 issue Node IDs on them (webchick) - they're even multicoloured!
• Bring chocolate for the big "chocolate stash" at the front of the room (everyone!)

Agenda/workflow

• Pancake bar as people come in
• 5-10 demonstration on how to do testing
• Pair up testing veterans with testing newbies
• Do test cases for the rest of the time:
  • Get pancakes
  • Get a testing issue
  • Solve and eat testing issue and pancakes, respectively.
  • Submit patch for review.
  • Claim a new task/pancake if you have no tasks at cnw, otherwise tidy up those issues.
  • Repeat.
  • Chocolate fits in here somewhere too.

"Volunteers"

• cwgordon7
• webchick
• sdboyer
• chx
• boombatower
• catch
• flobruit
• damz

If you see anything missing from this list, or would like to volunteer to help with any of these items, feel free to edit away. :)

Post-event

To make it easier to Dries to commit the tests and to ensure the quality of the tests the patches should be reviewed by those involved in testing development.

1. Aggregate testing party patches into bulk patches related to each file/module they test instead of small tasks. (Without "TestingParty08" prefix)
2. Review the tests while aggregating and ensure that they all at least pass.
3. Create issues for the bulk patches or fill in existing ones.

While Dries is away, and at least before the DrupalCon (and its awesome Testing Party!), we really should clean-up the Simpletest module and core tests.

I'm working on a project that does some fairly crazy extensions to drupal core functionality (biggest single item right now: $user goes from a stdClass object into an actual classed object, and I use OO fun to make a seven-way distinction between different user 'types'), and have resolved to basically take a TDD approach to driving the site forward, given the complexity and breadth of these changes.

Today the Drupal Testing framework is many an UI ("functional") testing. Should we rename this group to simply "Testing"?

Damien Tournoud
http://drupalfr.org

1. Too heavy to write simple test

if I changed code in blog_link to remove author's blog link, how can I write a small code to ensure it? logic as

testnode = new node;
node->type = blog;
testlink = blog_link('node', testnode);
assertNotContains('tom's blog', testlink);

2. Not IDE friendly, even for simpletest series.

Using simpletest eclipse plugin, you can't run test file because of the .test extention, have to rename to .php. After renaming, it still can be run because of complex file including.

Any thoughts?

We're down to under 60 core test failures, and there's at least four RTBC patches in the queue which will reduce that further, meaning we'll be down to 5-6 unresolved issues dealing with core test patch failures pretty soon.

At the moment, there's only one test that fails by design - for the url filter. However that core bug is likely to get fixed before all tests are working again, so it's unlikely to impact on much.

If you're looking for current status, look at the most recent comment

Since we've still got core test failures, and no automated patch testing, I thought it'd be worth doing a snapshot of tests that pass week by week (or more frequently if there's lots of commits). This allows us to keep a historical overview of where bugs have been introduced into either core code, simpletest.module or tests themselves. It should also hopefully help to reduce duplicate issues about broken tests (since a test failure can be down to any one of those conditions).

I have created a feature request at http://drupal.org/node/266220

Feedback please.

Now that SimpleTest is apart of Drupal core issues related to the 7.x development have been moved to the core issue queue. If you would like to be involved I have included two links which will filter the issues to only display SimpleTest related issue. You can obviously used the advanced search feature to narrow your results to the SimpleTest related components instead of using the links.

Hello everyone!

I keep getting 'Undefined index' exceptions in my tests:

As you all know, they occur in this kind of situations:

One option to avoid this is to check if the variable is set:

Hello everyone!

I'm having problems when trying to add tags to the story/page creation form using simpletest.

I have a vocabulary which uses FREE TAGGING, and I want create a new page with the tag "cat". The vid of the vocabulary is stored in the class attribute 'free_tagging_voc'. What I've been trying to do is the following:

So in the process of writing my first test I found a bunch of documentation discrepancies at http://drupal.org/simpletest

I'm not yet familiar enough with everything to know exactly what's deprecated, what's best practices/optional etc. so starting this to bring them up - then we can just edit stuff directly on the page once it's confirmed it needs changing. Or if they're not salvageable, just archive them quick so we're not presenting misleading information and replace with http://groups.drupal.org/node/11020 when it's ready.

First, for those that don't want to read the full post, here's the "30-second elevator speech":
This post to to discuss using Seleinum with Drupal. Specifically using Maven to run the selenium tests and writing the Selenium tests in Java.
The example code be downloaded from the Workhabit Inc. public repository here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk
One must have the following installed to run:
1. Firefox
2. Java 1.5+
3. Maven
All 3 are easy to install and aquire. Once you download the code from the repository, you can just navigate to the directory and run:

Having not written a single test yet I figured it'd be a good idea to document the process, and I happen to have an issue in the queue which is perfect for doing so.

Background:
* Access rules were removed from core
* Statistics module had a direct call to the {access} table which I knew nothing bout, so is now broken
* Dries said "We should probably write a test for it too so we don't reintroduce this problem."

see: http://drupal.org/node/252920

This profile module test suite also has a lot of redundant code.

Are other tests broken? Is it possible that I'm the first one noticing this?

Note: This document is under development and targeted towards Drupal 7

What is Automated Testing?

Automated testing encompasses UI, API and unit tests. It's a technique for testing small sections of code to ensure it works as expected. For example, a UI test might check a form submission process to ensure form display, validation of input and submission works as expected. A unit test might test a function for checking street addresses by passing a variety of well-formed and mal-formed addresses to ensure it handles them properly.

We now have a testing framework in Drupal core. What we need to move forward is some structure that makes writing tests easy and efficient so that we can go towards our 100% code coverage goal without wasting development resources.

We are pretty ahead with functional tests but unit tests are a completely different problem. The function we want to test calls other functions and so on. While theoretically one can unit test the leaves of the call tree and work from there, saying "drupal_validate_utf8 is already tested and we know it works.

I ran all the unit tests and I get
Test cases run: 57/58, Passes: 3123, Failures: 282, Exceptions: 34
Is this what other folks get?

Question 1: Do we need to coordinate the fixing of them now?
Question 2: Are we striving for 0 failures at the end of all this?

Summary: 20 000 patches submitted by 1200 contributors for Drupal 7 will require:
* testing tools development with patch testing automation
* 1200 core contributors to be trained on how to write and maintain core tests

Please correct me if I'm wrong, but it seems that this group's purpose is to concentrate on unit testing and code coverage. Excellent. This is a definite sign of maturity for Drupal.

But I also see some posts in here that refer more to UI centric testing. Is there yet an effort for organizing this testing as well?

There are many things that need to be done to SimpleTest is to provide full test coverage, as requested by Dries, and be placed in the core. A lot of work has already been completed and we are making great strives towards making this a reality, but there is still much to do.

I have provided a general overview of what needs to be accomplished.

1. Complete functional tests. (list)
2. Review functional tests and ensure that they pass against HEAD and test necessary functionality.
3. Plan unit testing
4. Generate stub unit tests using SimpleTest Unit Testing
5. Write unit tests.

What I wanna develop for SOC 2008 is a module called security (or add security function to simpletest existing module) to enable users checking their drupal installation against xss and sql injection vulnerabilities.
It will be also good for module developers, in fact they can check their module before submitting them to drupal website. Users could be more protected against vulnerabilities that became from third part modules.
The objective of this work is to realize automated penetration test on drupal installation.

It will be based upon SimpleTest, already used by Rasmus (php core developer) to develop his own closed source xss scanner. SimpleTest is a jUnit similar library written for php.

My module could easily been extended to add more functionalities about security, but basically I think that this two are the most important.
If someone has functionality ideas to improve my project and make it better I'm here, listening for more proposal.

<p>By: Robin Monks</p> 
<h1>Introduction</h1> 
<p>Drupal is community. It’s a community of visionaries that make Drupal what it is, we all want to see Drupal be more powerful, more flexible, and just plain better than the competition!</p>

Hello, I've written a perl script (fastens fire-proof suit) that generates module files as well as a test file complete with a test class and stubbed methods. I've posted the code as an issue to the simpletest module as webchick suggested: http://drupal.org/node/233261

From the documentation:

Features:
- creates a module folder and associated file skeletons
- generates a test class and test methods and places them in a test file
recognized by SimpleTest. This currently only covers unit tests,
and not higher level tests like functional or integration tests.

Some SimpleTests would make great DROP tasks. So I propose we create a list of the most-needed SimpleTests, and at any given time, the top 3-5 are active DROP tasks. Edit the table below; I'll be updating the DROP issue queue(s) as necessary.

I've been working away at a Drupal 6 module that will automate the process of performing a Drupal cvs checkout, installation and testing against core. The intent of this module is to be used on http://testing.drupal.org/ ... I was able to get the checkout and simpletest based installation working up to the point where the JS generated progress bar starts. There is a redirect in place that I believe is generated from the Batch API that does not play well with simpletest. As such, using simpletest to perform an installation currently does not seem to be possible. The details of the problem are outlined in two tickets that I have found.

http://drupal.org/node/204374 has been marked as a duplicate of http://drupal.org/node/229905 ... reading both issues helps understand the problem as it relates to writing a simpletest based installation.

I am in search of feedback/ideas on how to progress with this issue.

Hold on, hear me out :o) We're a really focused and strong (and almost too commercial) community, and we really have put our focus on features and output, and we've lacked a lot when it's come to making it all work correctly in every possible case.

That's where quality testing comes in. I've been thinking about Drupal's quality assurance process for a couple years now, but in the last few months it's really come to a head for me. My personal "dam-breaking" moment came when the Batch API in Drupal 6 prevented install, and other tasks depending on the API from running automatically, or in text based browsers, or in browsers were people have been privacy conscious and disabled JS and Meta refresh tags.

Without proper quality trails this will continue to happen. And here's another point I'd like to make:

Continue reading and comment at DrupalFiles >

Kieran's trying to put together a testing sprint this spring. We should come up with an agenda of what we're going to need to cover:

Goals and Deadlines

The Drupal 7 development life cycle may be extended for as long at 4 months from June, to October, 2008. This is possible if Drupal 7 has solid test coverage including functional, unit, and JavaScript testing.

Added to official ideas list at http://drupal.org/node/234712

(Draft SoC 2008 project proposal)

Drupal will be using simpletest for php functional and unit testing, but currently we have no equivalent framework for javascript.

This means that every time there's a minor jQuery update (or any other javascript patch to core), about six different people have to spend hours clicking around to see if anything broke.

Background and Purpose
After the SimpleTest session Wednesday morning I talked with sethcohn about some possible solutions to some of the issues that are being faced with full scale implementation of the unit test coverage. Some of the ideas are specific to the SimpleTest framework, but much could be applied to other testing frameworks. According to chx it sounds like the SimpleTest framework is going to be used which makes this information especially relevent.

To start off the biggest unknown currently is how to go about implementing "function unit testing" (which I beleive is the term). In other words calling an individual function with different arguements and checking the results against the know correct output. Until now this has been virtually untested and, to my knowledge, not really thought about at all. Instead "functional" testing has been worked on (as I was apart of).

We received, by show of hands, a considerable number of people who claim to be interested in contributing to the testing challenge, as introduced by Dries in his keynote. If anywhere near the number of people that raised their hands contribute, it could create a very chaotic situation and put a lot of pressure on the SimpleTest maintainer(s). In an attempt to address this I have compiled some ideas and thoughts.

A question which a bunch of people asked at Drupalcon (including me) was how to know which modules and functions have tests (and which don't). This is useful both for computing test coverage and for knowing where to focus efforts as we try to build initial coverage for core.

There are tools which do % of test coverage, but according to the BOF today, they tell you line numbers rather than functions - a useful metric but not perfect.

In Dries' keynote on Monday, he announced that he was willing to push back the code freeze to mid-October if Drupal has full test coverage. This is going to require a cultural shift towards test-driven development that is supported by a community who writes both functional and unit tests. This will ensure that improvements in one area of Drupal don't cascade and break other areas, which means that more time is spent in the development cycle fixing bugs rather than adding new features.

The unit testing working group met with Dries on Tuesday morning, and they came up with a tentatively-approved plan to start including SimpleTests in core. This will emphasize the importance of writing testable code, but requires active support from the Drupal community to help write functional and unit tests.

If there isn't enough progress by May, then Dries will be forced to shorten the code freeze and focus on fixing bugs. On Monday morning, four members of the testing working group presented the following information on the SimpleTest framework and the future of test-driven development in Drupal.

I created a GHOP proposal to lay grounds for a simpletest suite of CCK field storage in http://drupal.org/node/209391.

This Wiki page aims at defining the actual list of tests that would constitute an exhaustive suite when we have an overall structure. The actual scope of the proposed GHOP task is not to implement all these tests, but to provide the overall structure and reusable code (note the 'Repeat the above' items), and actually code, say, the first two 'dark bullets' items below.

So I am writing a new CCK field and because I practice test driven development I find myself testing that I have implemented the CCK api hooks correctly. This basically means testing against the interface. Usually in a pure OO class orientated world for unit testing we would use a mock library (e.g.