Unit Testing

Hello everyone!

I keep getting 'Undefined index' exceptions in my tests:

As you all know, they occur in this kind of situations:

One option to avoid this is to check if the variable is set:

Hello everyone!

I'm having problems when trying to add tags to the story/page creation form using simpletest.

I have a vocabulary which uses FREE TAGGING, and I want create a new page with the tag "cat". The vid of the vocabulary is stored in the class attribute 'free_tagging_voc'. What I've been trying to do is the following:

So in the process of writing my first test I found a bunch of documentation discrepancies at http://drupal.org/simpletest

I'm not yet familiar enough with everything to know exactly what's deprecated, what's best practices/optional etc. so starting this to bring them up - then we can just edit stuff directly on the page once it's confirmed it needs changing. Or if they're not salvageable, just archive them quick so we're not presenting misleading information and replace with http://groups.drupal.org/node/11020 when it's ready.

First, for those that don't want to read the full post, here's the "30-second elevator speech":
This post to to discuss using Seleinum with Drupal. Specifically using Maven to run the selenium tests and writing the Selenium tests in Java.
The example code be downloaded from the Workhabit Inc. public repository here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk
One must have the following installed to run:
1. Firefox
2. Java 1.5+
3. Maven
All 3 are easy to install and aquire. Once you download the code from the repository, you can just navigate to the directory and run:

Having not written a single test yet I figured it'd be a good idea to document the process, and I happen to have an issue in the queue which is perfect for doing so.

Background:
* Access rules were removed from core
* Statistics module had a direct call to the {access} table which I knew nothing bout, so is now broken
* Dries said "We should probably write a test for it too so we don't reintroduce this problem."

see: http://drupal.org/node/252920

This profile module test suite also has a lot of redundant code.

Are other tests broken? Is it possible that I'm the first one noticing this?

Tonight, dmitri asked me how to write a test for drupal_add_js to go along with a patch he was writing for it. I did my best to help him out, and wrote out our journey here. This wiki page contains at least some of the stuff we'll need in actual handbook documentation. Please jump in to help flesh this out.

Incidentally, it's not actually possible to write a test for a non-module file yet, so that rather killed our journey very quickly. :P See http://drupal.org/node/251473.

Introduction

We now have a testing framework in Drupal core. What we need to move forward is some structure that makes writing tests easy and efficient so that we can go towards our 100% code coverage goal without wasting development resources.

We are pretty ahead with functional tests but unit tests are a completely different problem. The function we want to test calls other functions and so on. While theoretically one can unit test the leaves of the call tree and work from there, saying "drupal_validate_utf8 is already tested and we know it works.

I ran all the unit tests and I get
Test cases run: 57/58, Passes: 3123, Failures: 282, Exceptions: 34
Is this what other folks get?

Question 1: Do we need to coordinate the fixing of them now?
Question 2: Are we striving for 0 failures at the end of all this?

Summary: 20 000 patches submitted by 1200 contributors for Drupal 7 will require:
* testing tools development with patch testing automation
* 1200 core contributors to be trained on how to write and maintain core tests

Please correct me if I'm wrong, but it seems that this group's purpose is to concentrate on unit testing and code coverage. Excellent. This is a definite sign of maturity for Drupal.

But I also see some posts in here that refer more to UI centric testing. Is there yet an effort for organizing this testing as well?

There are many things that need to be done to SimpleTest is to provide full test coverage, as requested by Dries, and be placed in the core. A lot of work has already been completed and we are making great strives towards making this a reality, but there is still much to do.

I have provided a general overview of what needs to be accomplished.

1. Complete functional tests. (list)
2. Review functional tests and ensure that they pass against HEAD and test necessary functionality.
3. Plan unit testing
4. Generate stub unit tests using SimpleTest Unit Testing
5. Write unit tests.

What I wanna develop for SOC 2008 is a module called security (or add security function to simpletest existing module) to enable users checking their drupal installation against xss and sql injection vulnerabilities.
It will be also good for module developers, in fact they can check their module before submitting them to drupal website. Users could be more protected against vulnerabilities that became from third part modules.
The objective of this work is to realize automated penetration test on drupal installation.

It will be based upon SimpleTest, already used by Rasmus (php core developer) to develop his own closed source xss scanner. SimpleTest is a jUnit similar library written for php.

My module could easily been extended to add more functionalities about security, but basically I think that this two are the most important.
If someone has functionality ideas to improve my project and make it better I'm here, listening for more proposal.

Hello, I've written a perl script (fastens fire-proof suit) that generates module files as well as a test file complete with a test class and stubbed methods. I've posted the code as an issue to the simpletest module as webchick suggested: http://drupal.org/node/233261

From the documentation:

Features:
- creates a module folder and associated file skeletons
- generates a test class and test methods and places them in a test file
recognized by SimpleTest. This currently only covers unit tests,
and not higher level tests like functional or integration tests.

Some SimpleTests would make great DROP tasks. So I propose we create a list of the most-needed SimpleTests, and at any given time, the top 3-5 are active DROP tasks. Edit the table below; I'll be updating the DROP issue queue(s) as necessary.

I've been working away at a Drupal 6 module that will automate the process of performing a Drupal cvs checkout, installation and testing against core. The intent of this module is to be used on http://testing.drupal.org/ ... I was able to get the checkout and simpletest based installation working up to the point where the JS generated progress bar starts. There is a redirect in place that I believe is generated from the Batch API that does not play well with simpletest. As such, using simpletest to perform an installation currently does not seem to be possible. The details of the problem are outlined in two tickets that I have found.

http://drupal.org/node/204374 has been marked as a duplicate of http://drupal.org/node/229905 ... reading both issues helps understand the problem as it relates to writing a simpletest based installation.

I am in search of feedback/ideas on how to progress with this issue.

Hold on, hear me out :o) We're a really focused and strong (and almost too commercial) community, and we really have put our focus on features and output, and we've lacked a lot when it's come to making it all work correctly in every possible case.

That's where quality testing comes in. I've been thinking about Drupal's quality assurance process for a couple years now, but in the last few months it's really come to a head for me. My personal "dam-breaking" moment came when the Batch API in Drupal 6 prevented install, and other tasks depending on the API from running automatically, or in text based browsers, or in browsers were people have been privacy conscious and disabled JS and Meta refresh tags.

Without proper quality trails this will continue to happen. And here's another point I'd like to make:

Continue reading and comment at DrupalFiles >

Kieran's trying to put together a testing sprint this spring. We should come up with an agenda of what we're going to need to cover:

Goals and Deadlines

The Drupal 7 development life cycle may be extended for as long at 4 months from June, to October, 2008. This is possible if Drupal 7 has solid test coverage including functional, unit, and JavaScript testing.

Added to official ideas list at http://drupal.org/node/234712

(Draft SoC 2008 project proposal)

Drupal will be using simpletest for php functional and unit testing, but currently we have no equivalent framework for javascript.

This means that every time there's a minor jQuery update (or any other javascript patch to core), about six different people have to spend hours clicking around to see if anything broke.

Background and Purpose
After the SimpleTest session Wednesday morning I talked with sethcohn about some possible solutions to some of the issues that are being faced with full scale implementation of the unit test coverage. Some of the ideas are specific to the SimpleTest framework, but much could be applied to other testing frameworks. According to chx it sounds like the SimpleTest framework is going to be used which makes this information especially relevent.

To start off the biggest unknown currently is how to go about implementing "function unit testing" (which I beleive is the term). In other words calling an individual function with different arguements and checking the results against the know correct output. Until now this has been virtually untested and, to my knowledge, not really thought about at all. Instead "functional" testing has been worked on (as I was apart of).

We received, by show of hands, a considerable number of people who claim to be interested in contributing to the testing challenge, as introduced by Dries in his keynote. If anywhere near the number of people that raised their hands contribute, it could create a very chaotic situation and put a lot of pressure on the SimpleTest maintainer(s). In an attempt to address this I have compiled some ideas and thoughts.

A question which a bunch of people asked at Drupalcon (including me) was how to know which modules and functions have tests (and which don't). This is useful both for computing test coverage and for knowing where to focus efforts as we try to build initial coverage for core.

There are tools which do % of test coverage, but according to the BOF today, they tell you line numbers rather than functions - a useful metric but not perfect.

In Dries' keynote on Monday, he announced that he was willing to push back the code freeze to mid-October if Drupal has full test coverage. This is going to require a cultural shift towards test-driven development that is supported by a community who writes both functional and unit tests. This will ensure that improvements in one area of Drupal don't cascade and break other areas, which means that more time is spent in the development cycle fixing bugs rather than adding new features.

The unit testing working group met with Dries on Tuesday morning, and they came up with a tentatively-approved plan to start including SimpleTests in core. This will emphasize the importance of writing testable code, but requires active support from the Drupal community to help write functional and unit tests.

If there isn't enough progress by May, then Dries will be forced to shorten the code freeze and focus on fixing bugs. On Monday morning, four members of the testing working group presented the following information on the SimpleTest framework and the future of test-driven development in Drupal.

I created a GHOP proposal to lay grounds for a simpletest suite of CCK field storage in http://drupal.org/node/209391.

This Wiki page aims at defining the actual list of tests that would constitute an exhaustive suite when we have an overall structure. The actual scope of the proposed GHOP task is not to implement all these tests, but to provide the overall structure and reusable code (note the 'Repeat the above' items), and actually code, say, the first two 'dark bullets' items below.

So I am writing a new CCK field and because I practice test driven development I find myself testing that I have implemented the CCK api hooks correctly. This basically means testing against the interface. Usually in a pure OO class orientated world for unit testing we would use a mock library (e.g.

It stands to reason that the tests in simpletest/tests will be the ones that people will use to base their own tests. These tests were written by at least 5-6 different people, and it shows. As a result, it's really confusing for a new developer to get a sense of what your own module's SimpleTest integration ought to look like.

So let's create a template and some guidelines for what simpletests ought to look like, and then make a GHOP task for formatting all of the existing tests in this way. This is a wiki page, so please edit and improve.

<?php
// $Id$

/**
 * Brief description of test case here.
 */
class FileNameTest extends DrupalTestCase {

  /**
   * Drupal SimpleTest method: return metadata about the test.
   */
  function get_info() {
    return array(
      'name' => t('A few words describing tests.'),
      'desc' => t('A complete sentence describing an overview of the tests included in the file.'),
      'group' => t('Unique grouping name'),
    );
  }

  /**
   * SimpleTest core method: code run before each and every test method.
   *
   * Optional. You only need this if you have setup tasks.
   */
  function setUp() {
    parent::setUp();

    // Setup tasks go here.
  }

  /**
   * SimpleTest core method: code run after each and every test method.
   *
   * Optional. You only need this if you have teardown tasks.
   */
  function tearDown() {
    // Teardown tasks go here.

    parent::tearDown();
  }

  /**
   * One-sentence description of what test entails.
   */
  function testExampleThing() {
    // Assertions, etc. go here.
  }

  /**
   * One-sentence description of what test entails.
   */
  function testNextExampleThing() {
    // Assertions, etc. go here.
  }

  // Can have many more testFoo functions.
}

Questions:

1. What should be the file naming convention? module_name_verb describing task.test seems to be common; for example, user_access.test, and page_view.test. Is that ok?
2. When does it make sense to split a module up into multiple .test files, vs. one big one that tests everything? What's the guideline for this?

Hi everyone. I've written an article introducing unit testing and the Simpletest module on Lullabot.com. Hope you enjoy it!

There will be a discussion about Drupal testing and how to make it happen at Drupalcon Barcelona BOF room 1.
Sorry for such a short notice.

Hi Folks,

I just created a group as a "meta testing group" at http://groups.drupal.org/quality-assurance

Webchick pointed out this Unit Testing group which has a fair number of "general testing" notes in it beyond just unit testing.

So, either

1) A lot of you are probably interested in joining that other group as well
2) We can just merge them and rename this one to be a general testing group - perhaps "Testing and QA" could be the title so that it's as general as possible, though I'm not big on the importance of titles...

This is a list of manual tests I perform every time I test Drupal core. I figure this is maybe a good start for things that we should have unit tests for. If nothing else, it provides a checklist for people who make changes to other modules of things to try to ensure they still work. Please feel free to add to this list!

Common things to check for:
- Visible errors (notices, warnings, etc.)
- Broken functionality
- "White screens of death (WSOD)" - Basically a completely blank screen when you go to a page.
- Regressions (when something used to work in a previous version of Drupal and no longer does)
- Broken links
- Incomplete or unclear help text/descriptions
- Typos in text, comments or code.

This is the best bit from my blog post: http://starbowconsulting.com/blog/tao/simpletest-tips
(Also, if anyone want to read me rant about the state of unit testing in drupal: http://starbowconsulting.com/blog/tao/joys-unit-testing)
Validating HTML Fragments
I am currently working on code that generates html fragments. Simple Test out-of-the-box does a great job helping you test entire http pages, but only it if goes and gets them itself. I could not find any easy way to load in a local page/fragment. So I did it the hard way:

I am trying to functional testing on a custom node creation form, and cannot figure out the correct syntax for selecting the taxonomy terms.

A couple of modules exist to aid the Drupal developer in creating functionally correct and robust code. Specifically for testing two types exist : there are the white box testing modules such as simple-test and the black box testing modules such as selenium.

Does anyone out there have a server which they control and is willing to let us install/maintain a simpletest-auto installation there? this will continously check developer patches to see if they break test suite. will also check core drupal itself.

Here you will find all the information you need to setup an environment where you will be able to automatically test patches.

One of the biggest concerns about automatic testing is how to prevent potentially dangerous code to cause harm to the server and other tests.

Here are short instrucions on how to use this module.

After some discussions with my mentor Thomas and other people on devel mailling list I have created sort of a first prototype.

How it works

One part of this module is used to catch the patches and submites them to other part of the module via xml-rpc call. The second part receives this call and tests the patch. Results are stored in a new node on test server node address is returend by xml-rpc call response.

Congrats to student Rok Zlender and mentor Thomas Ilsche! Here's the list of Drupal SOC approved projects, the Drupal Google Summer of Code Page and link to the Drupal Unit Testing Automation page. Please let us know how we can help!

After the jump, here's what I know is happening as of late May 2006. Add a comment and or post your activities related to Unit Testing because i am sure this is not comprehensive:

Ok - I'll get yea olde ball rolling with a question: why are the tests for the image and profile modules part of the main "simpletest" distribution?

Are these modules being moved into core?

These tests (duh) fail when these modules aren't present..which they are not in a default drupal + simpletest distro.

One of the big psychological goals of testing is to "prove" that everything is working as it should...so having the default simpletest suite fail because it expects two modules which are not core to be present is counter to that goal...

Thoughts?

s