Testing and Quality Assurance

Here is an effort at documenting Drupal testing patterns from what's currently used in core tests. The objective is to promote reusability of standardized patterns throughout core and contributed tests.

// Catch - this looks good, but I think we should completely ignore stuff like getContent() which is used once per year and just focus one what 70% of basic tests will use - the API docs etc. are there, and linkable at the end, for everything else.

It stands to reason that the tests in simpletest/tests will be the ones that people will use to base their own tests. These tests were written by at least 5-6 different people, and it shows. As a result, it's really confusing for a new developer to get a sense of what your own module's SimpleTest integration ought to look like.

http://acquia.com/latest-drupal-test-results

The results are updated around 3 AM Eastern Daylight Timezone every night.

Acquia is a new software company looking for a QA AUTOMATION ROCKSTAR to work in our Andover, MA office as a part of our amazing development team providing value added software products and services for Drupal's open source social publishing software!

Here are the results of running our whole test suite on PostgreSQL 8.3.

• Actions configuration 30 passes, 8 fails, 0 exceptions
  See http://drupal.org/node/261859
• Block functionality 57 passes, 1 fail, 0 exceptions
  No known issue (yet!)
• Blog API functionality 45 passes, 2 fails, 1 exception
  No known issue (yet!)
• Search engine ranking 21 passes, 5 fails, 20 exceptions
  See http://drupal.org/node/296624
• Module list functionality 53 passes, 4 fails, 0 exceptions
  No known issue (yet!)

In porting the Pathauto tests from the Simpletest 6.x-1.x to 6.x-2.x style I found a few things I thought I would document here in the hopes that they benefit others:

1. Put the .test file into the root directory of your module. If you had it in modules/tests/module.test it now belongs in modules/module.test
2. DrupalTestCase is now DrupalWebTestCase
3. get_info() is now getInfo()
4. in getInfo, desc is now description
5. $this->drupalCreateUserRolePerm is now $this->drupalCreateUser(
6. drupalLoginUser is now drupalLogin

Splitting the discussion from here:

http://lists.drupal.org/pipermail/development/2008-August/030879.html

afaik, the steps to get this running are:

1. Abstract comment creation functionality in project_issue.module - http://drupal.org/node/271216
2. Allow testing.drupal.org to post back to issues and mark as code needs work when patches don't apply.
3. Get all known failures cleared
4. Allow testing.drupal.org to post back to issues and mark as code needs work when core tests fail
5. Extend this to Drupal 6

Submitted for perusal by the group, as passed to me by a fellow developer.

"[Google ratproxy is a] semi-automated, largely passive web application security audit tool, optimized for an accurate and sensitive detection, and automatic annotation, of potential problems and security-relevant design patterns based on the observation of existing, user-initiated traffic in complex web 2.0 environments.

Great news! The Awesome Testing Party session was accepted at Drupalcon Szeged! Woohoo!! It'll be at 9am on August 28th, but people should probably be there a bit earlier, like 8:30am.

There's lots to prepare, and very little time to do it in! We had a discussion today in #drupal to talk over logistics. Here's where we stand. Please strike things as you get them done. Thanks!

Stuff for lead-up to the event

• Write a testing hand-out that summarizes stuff from the intro to testing session (flobruit)
• Do a PR blitz to Drupal Planet about the party, talk about its benefits and goals, how it's going to work, and ask Szeged attendees to bring a box of chocolate with them (webchick) - http://webchick.net/awesome-testing-party
• Re-run test coverage scripts to see where we stand (catch) - http://coverage.ca.tchpole.net/coverage/html/1
• Do a screencast about how to write tests (sdboyer)
• Create a .test template that people can copy/paste from (webchick)
• Create issues for each candidate test and title them TestingParty08: Description of test" (boombatower and catch) - list here
• Write up a hand-out describing how testing party will work for participants (webchick)

Stuff for the day of the event

• Look into getting Hungarian pancakes for breakfast ;) (chx)
• Make lots of photocopies of the testing handout to bring with (webchick)
• Go buy some index cards and write all the TestingParty08 issue Node IDs on them (webchick) - they're even multicoloured!
• Bring chocolate for the big "chocolate stash" at the front of the room (everyone!)

Agenda/workflow

• Pancake bar as people come in
• 5-10 demonstration on how to do testing
• Pair up testing veterans with testing newbies
• Do test cases for the rest of the time:
  • Get pancakes
  • Get a testing issue
  • Solve and eat testing issue and pancakes, respectively.
  • Submit patch for review.
  • Claim a new task/pancake if you have no tasks at cnw, otherwise tidy up those issues.
  • Repeat.
  • Chocolate fits in here somewhere too.

"Volunteers"

• cwgordon7
• webchick
• sdboyer
• chx
• boombatower
• catch
• flobruit
• damz

If you see anything missing from this list, or would like to volunteer to help with any of these items, feel free to edit away. :)

Post-event

To make it easier to Dries to commit the tests and to ensure the quality of the tests the patches should be reviewed by those involved in testing development.

1. Aggregate testing party patches into bulk patches related to each file/module they test instead of small tasks. (Without "TestingParty08" prefix)
2. Review the tests while aggregating and ensure that they all at least pass.
3. Create issues for the bulk patches or fill in existing ones.

While Dries is away, and at least before the DrupalCon (and its awesome Testing Party!), we really should clean-up the Simpletest module and core tests.

I'm working on a project that does some fairly crazy extensions to drupal core functionality (biggest single item right now: $user goes from a stdClass object into an actual classed object, and I use OO fun to make a seven-way distinction between different user 'types'), and have resolved to basically take a TDD approach to driving the site forward, given the complexity and breadth of these changes.

Today the Drupal Testing framework is many an UI ("functional") testing. Should we rename this group to simply "Testing"?

Damien Tournoud
http://drupalfr.org

1. Too heavy to write simple test

if I changed code in blog_link to remove author's blog link, how can I write a small code to ensure it? logic as

testnode = new node;
node->type = blog;
testlink = blog_link('node', testnode);
assertNotContains('tom's blog', testlink);

2. Not IDE friendly, even for simpletest series.

Using simpletest eclipse plugin, you can't run test file because of the .test extention, have to rename to .php. After renaming, it still can be run because of complex file including.

Any thoughts?

We're down to under 60 core test failures, and there's at least four RTBC patches in the queue which will reduce that further, meaning we'll be down to 5-6 unresolved issues dealing with core test patch failures pretty soon.

At the moment, there's only one test that fails by design - for the url filter. However that core bug is likely to get fixed before all tests are working again, so it's unlikely to impact on much.

If you're looking for current status, look at the most recent comment

Since we've still got core test failures, and no automated patch testing, I thought it'd be worth doing a snapshot of tests that pass week by week (or more frequently if there's lots of commits). This allows us to keep a historical overview of where bugs have been introduced into either core code, simpletest.module or tests themselves. It should also hopefully help to reduce duplicate issues about broken tests (since a test failure can be down to any one of those conditions).

I have created a feature request at http://drupal.org/node/266220

Feedback please.

Now that SimpleTest is apart of Drupal core issues related to the 7.x development have been moved to the core issue queue. If you would like to be involved I have included two links which will filter the issues to only display SimpleTest related issue. You can obviously used the advanced search feature to narrow your results to the SimpleTest related components instead of using the links.

Hello everyone!

I keep getting 'Undefined index' exceptions in my tests:

As you all know, they occur in this kind of situations:

One option to avoid this is to check if the variable is set:

Hello everyone!

I'm having problems when trying to add tags to the story/page creation form using simpletest.

I have a vocabulary which uses FREE TAGGING, and I want create a new page with the tag "cat". The vid of the vocabulary is stored in the class attribute 'free_tagging_voc'. What I've been trying to do is the following:

So in the process of writing my first test I found a bunch of documentation discrepancies at http://drupal.org/simpletest

I'm not yet familiar enough with everything to know exactly what's deprecated, what's best practices/optional etc. so starting this to bring them up - then we can just edit stuff directly on the page once it's confirmed it needs changing. Or if they're not salvageable, just archive them quick so we're not presenting misleading information and replace with http://groups.drupal.org/node/11020 when it's ready.

First, for those that don't want to read the full post, here's the "30-second elevator speech":
This post to to discuss using Seleinum with Drupal. Specifically using Maven to run the selenium tests and writing the Selenium tests in Java.
The example code be downloaded from the Workhabit Inc. public repository here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk
One must have the following installed to run:
1. Firefox
2. Java 1.5+
3. Maven
All 3 are easy to install and aquire. Once you download the code from the repository, you can just navigate to the directory and run:

Having not written a single test yet I figured it'd be a good idea to document the process, and I happen to have an issue in the queue which is perfect for doing so.

Background:
* Access rules were removed from core
* Statistics module had a direct call to the {access} table which I knew nothing bout, so is now broken
* Dries said "We should probably write a test for it too so we don't reintroduce this problem."

see: http://drupal.org/node/252920

This profile module test suite also has a lot of redundant code.

Are other tests broken? Is it possible that I'm the first one noticing this?

Note: This document is under development and targeted towards Drupal 7

What is Automated Testing?

Automated testing encompasses UI, API and unit tests. It's a technique for testing small sections of code to ensure it works as expected. For example, a UI test might check a form submission process to ensure form display, validation of input and submission works as expected. A unit test might test a function for checking street addresses by passing a variety of well-formed and mal-formed addresses to ensure it handles them properly.

We now have a testing framework in Drupal core. What we need to move forward is some structure that makes writing tests easy and efficient so that we can go towards our 100% code coverage goal without wasting development resources.

We are pretty ahead with functional tests but unit tests are a completely different problem. The function we want to test calls other functions and so on. While theoretically one can unit test the leaves of the call tree and work from there, saying "drupal_validate_utf8 is already tested and we know it works.

I ran all the unit tests and I get
Test cases run: 57/58, Passes: 3123, Failures: 282, Exceptions: 34
Is this what other folks get?

Question 1: Do we need to coordinate the fixing of them now?
Question 2: Are we striving for 0 failures at the end of all this?

I ran all the unit tests and I get
Test cases run: 57/58, Passes: 3123, Failures: 282, Exceptions: 34
Is this what other folks get?

Question 1: Do we need to coordinate the fixing of them now?
Question 2: Are we striving for 0 failures at the end of all this?

Summary: 20 000 patches submitted by 1200 contributors for Drupal 7 will require:
* testing tools development with patch testing automation
* 1200 core contributors to be trained on how to write and maintain core tests

Please correct me if I'm wrong, but it seems that this group's purpose is to concentrate on unit testing and code coverage. Excellent. This is a definite sign of maturity for Drupal.

But I also see some posts in here that refer more to UI centric testing. Is there yet an effort for organizing this testing as well?

Please correct me if I'm wrong, but it seems that this group's purpose is to concentrate on unit testing and code coverage. Excellent. This is a definite sign of maturity for Drupal.

But I also see some posts in here that refer more to UI centric testing. Is there yet an effort for organizing this testing as well?

There are many things that need to be done to SimpleTest is to provide full test coverage, as requested by Dries, and be placed in the core. A lot of work has already been completed and we are making great strives towards making this a reality, but there is still much to do.

I have provided a general overview of what needs to be accomplished.

1. Complete functional tests. (list)
2. Review functional tests and ensure that they pass against HEAD and test necessary functionality.
3. Plan unit testing
4. Generate stub unit tests using SimpleTest Unit Testing
5. Write unit tests.

What I wanna develop for SOC 2008 is a module called security (or add security function to simpletest existing module) to enable users checking their drupal installation against xss and sql injection vulnerabilities.
It will be also good for module developers, in fact they can check their module before submitting them to drupal website. Users could be more protected against vulnerabilities that became from third part modules.
The objective of this work is to realize automated penetration test on drupal installation.

It will be based upon SimpleTest, already used by Rasmus (php core developer) to develop his own closed source xss scanner. SimpleTest is a jUnit similar library written for php.

My module could easily been extended to add more functionalities about security, but basically I think that this two are the most important.
If someone has functionality ideas to improve my project and make it better I'm here, listening for more proposal.

What I wanna develop for SOC 2008 is a module called security (or add security function to simpletest existing module) to enable users checking their drupal installation against xss and sql injection vulnerabilities.
It will be also good for module developers, in fact they can check their module before submitting them to drupal website. Users could be more protected against vulnerabilities that became from third part modules.
The objective of this work is to realize automated penetration test on drupal installation.

It will be based upon SimpleTest, already used by Rasmus (php core developer) to develop his own closed source xss scanner. SimpleTest is a jUnit similar library written for php.

My module could easily been extended to add more functionalities about security, but basically I think that this two are the most important.
If someone has functionality ideas to improve my project and make it better I'm here, listening for more proposal.

<p>By: Robin Monks</p> 
<h1>Introduction</h1> 
<p>Drupal is community. It’s a community of visionaries that make Drupal what it is, we all want to see Drupal be more powerful, more flexible, and just plain better than the competition!</p>