Selenium and Drupal

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Souvent22's picture

First, for those that don't want to read the full post, here's the "30-second elevator speech":
This post to to discuss using Seleinum with Drupal. Specifically using Maven to run the selenium tests and writing the Selenium tests in Java.
The example code be downloaded from the Workhabit Inc. public repository here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk
One must have the following installed to run:
1. Firefox
2. Java 1.5+
3. Maven
All 3 are easy to install and aquire. Once you download the code from the repository, you can just navigate to the directory and run:
mvn -P drupal integration-test
Maven and writing tests are beyond the scope of this blog post.
I'm writing this up for comments, thoughts, and hopefully for others to use and/or contribute.

Ok, now the long winded version:

I've been in the Java world for a bit and I'm now bringing my self back into the PHP/Drupal realm.
With that said on my last project we had to validate ALL code with unit tests, and then we did fully functional integration tests using Selenium.

The Build management tool we used was Maven ( way past the scope of discussion here ). However, what I have done is GREATLY simplified down what we used into a very short, simple, and small file which allows one to run Selenium tests out of the box. It uses Maven/Java to do the tests...but, that's neither here nor there.
It is available here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk

I know that there is a Selenium module for Drupal...however when I thought about this, I also felt that this was an inherit problem. This dawned on me when I read the link that there is a patch to port it to 5.x. There in lies the problem. As we move versions/etc., we're updating the testing frame-work instep with the item we're testing. E.g. PHPUnit may be at 2.3, and we can still use PHPUnit 2.3 for Drupal 4.7, 5.1, 6.x, etc. However, by tightly integrating the testing frame work into the item we are testing, I feel one is creating a cyclic-redundancy in which the end result is chaos.

Goals

My only goals of this project is 2 fold:

  1. Give a solid base so that anyone can download the drupal-seleinum project and write tests for their drupal site.
  2. The community should have an entire suite of tests to test the Drupal installation as a base install ( I have currently started these in the com.workhabit.drupal.selenium.core package. Inside the package, it think they should be split into their subsystems, e.g. Authentication, Taxonomy, CCK, etc.).

Why Java/Maven?

Well, there are a number of reasons; I will not list/discuss all of them here as they are not really part of this post. But on the short list; Java allows one to set break points where Selenium is running to see why a test may be failing ( it should be noted you can also do this with PHP; if you setup your IDE correct and get xdebug installed/setup correctly also).
Also, by using Maven, one doesn't have to worry about dependencies with the unit testing suite (versions, etc.) as Maven takes care of all of that. Java/Maven will also allow us to do more in terms of continuous-build integration. Further, writing tests in Java is not that much harder/different that writing them in PHP. The Selenium IDE gives code in many languages (HTML, PHP, Java, Ruby, etc.) so anyone should be able to create tests easily. I'm hoping to write a post of how to write/add tests in Java. Most of the hard work has already been setup in the code, so all one has to do for a new test pretty much is create a new class and create an annotated java method, such as:

@Test
public void myDrupalTest() {
  BROWSER.open("/path/to/my/site");
  // ....other browser test code from Selenium IDE
  // ...Assertions
  return;
}

Also, in the coming days, I shall be adding to this suite a "MockSoapServer". This will allow you to hit a webservice in point (in this case the MockSoapServer), and it will return "canned" results. Maven will fire up both the Selenium server AND the MockSoapServer BEFORE running the test suites. It will then create a report and take down both after the tests are run.

Environment Setup

First things first, you will need to have Maven installed on your local machine in order to use the Drupal-Seleinum tests. Installing maven is very simple and straight forward. See the Apache Maven Site for more information on installing Maven.

Setup your Tests

So, lets be real; Usually when we are developing, we are changing some of the default workings of Drupal; So we will need to have a some-what custom test suite for each site we develop; or at least we will be extending the tests with more/new tests.

  • *EXPORT* the base Selenium maven build from the Workhabit repository.
    svn export https://svn.workhabit.com/svn/public/sandbox/earnest/drupal/drupal-selenium
  • Update the POM (If you are unfamiliar with Maven, ignore this step)
    You can update the POM if you want to better describe you project.
  • Modify the "setup.xml"

    You will notice when you download the code that there is a file called "setup-example.xml" in the src/test/resources folder. This file is read by the test suite and contains some general settings regarding your site. The settings are pretty self explanatory. The main ones you will need to change out of the box are:

    • siteRootPath
    • siteDomain
    • testUsername
    • testPassword

    I will hopefully be documenting the settings further, but for now, I apologize that the documentation is a bit sparse.

  • Test to see that you have everything setup correctly by running a maven integration test (Note: All tests are done with the drupal plugin)
    mvn -P drupal integration-test
    This should run the DrupalLoginTest which will bring up your Drupal site, and login a user.
  • Add the package to your own repository.
    svn add drupal-selenium && svn commit -m "Adding the drupal-selenium package to my site"
    You now have a customized maven/Selenium test suite for the project that you are working on.

Comments

Thanks souvent22. While you

moshe weitzman's picture

Thanks souvent22. While you were cavorting around with Java, we committed simpletest framework to core, along with a boatload of tests. See /modules/simpletest in HEAD.

Bridging the Gap

Souvent22's picture

Moshe,

Yes, i remember from DrupalCon Boston, and much to my delight that SimpleTest ( at that time and "undecided" frame work, but SimpleTest made it) was going to be committed into core. I have seen the tests sky rocket! However, I am, and have been still a bit un-impressed with SimpleTests scritable browser. I think Seleinum is a bit more robust, and thus why I wanted to push out this maven Selenium project for feed back to see what people thought. I do think that SimpleTest and Selenium are mutually exclusive, but can be collectively exhaustive to give a VERY high degree of confidence to a site/module(s)/custom-feature(s).

Indeed

calebgilbert's picture

...there is little in simpletest that is a substitute for what is available in something like selenium.

Practically speaking (if not literally) simpletest = unit testing. It falls far short of what is needed for acceptance/UI testing duties. (which documentation on the simpletest project site itself seems to acknowledge - see bottom of page)

Wonderful!

IncrediblyKenzi's picture

Moshe, that's excellent.. I've been keeping an eye on that effort as well, and it seems as though things are both well under way and extremely well thought out (props to everyone involved.. it's quite the herculean effort).

I do think there's a call for both here.. In the first case you have unit tests and functional/integration tests. Selenium closes the loop by providing automated acceptance testing.

We've done a bit of work in this area since we've been experimenting with bridging the gap between Java and Drupal (namely extensive webservices integration) for which we've needed a solid test harness. Selenium provided that, and we've successfully managed to build a test harness for Drupal using maven to run user acceptance tests against a drupal install in a quercus container. Stay tuned next week, I should have a working maven archetype that facilitates selenium tests against contrib modules and custom themes.

Roland Tanglao's picture

i don't see the big picture here: are you using java and maven because it allows you to remotely kick off the tests on another server?

does this handle cross browser (i.e. IE as well as firefox testing on both mac and windows?)

anyways a high level block diagram of your black box testing vision with maven, java etc showing the browsers would help a lot here IMHO

...Roland

In a word, yes..

IncrediblyKenzi's picture

In a word, yes.. Using Selenium-RC allows you to remote control different browsers on different platforms.

please renew your ssl

Roland Tanglao's picture

please renew your ssl certificate so i don't have to accept those annoying certificate prompts :-)

Roland Tanglao's picture
  1. record tests using Selenium-IDE
  2. save the test as ruby script (or python or your favourite scripting language)
  3. edit the test script and integrate it into a black box ruby or whatever language drupal testing framework (to handle multiple browsers and reporting, i haven't written a black box ruby testing framework, i am hoping somebody else does!)
  4. add the test script to the your automatic black box testing overnight test suite (again this overnight automatic test suite is vapourware)
  5. sleep easy knowing you have automated testing yet another part of your drupal site!

and my black box testing vision is oriented to custom tests for custom sites not a generic black box test for generic drupal functionality like login which is what earnest appears to have built (or is building)

We're fairly matched on this point..

IncrediblyKenzi's picture

Login is ostensibly just a reference case.. I think in that particular case it overlaps with what the community as a whole is achieving through simpletest integration.

Selenium is more useful when implementing tests against custom functionality.

Maven with selenium and dbunit provides the blackbox you're talking about. It allows one to bootstrap an environment (think: php code, database snapshot, and server configuration), and it ultimately solves the pain surrounding the creation of that black box.

In our case we used Java simply because maven plays nice with it. :) Building an antrun script or something similar to get ruby to run is possible, though for simplicity we opted to forego it.

As for your overnight test suite, Earnest and I talked about that as well earlier today.. We were considering the notion that something like Hudson, CruiseControl, or TeamCity would be ideal for this.

ah i see

Roland Tanglao's picture

You have an "integrated unit test (using simpletest) and blackbox test (using selenium-rc)" vision

my vision was to run the overnight black box tests separate from the simpletest unit testing stuff but upon reflection, that's probably unrealistic

i would prefer that the entire testing stack was open sourced which TeamCity doesn't appear to be but Hudson and Cruise Control appear to be

anyways my handwaving is a lot less useful than the real code you are writing, looking forward to seeing how this progresses!

Reference Case

Souvent22's picture

Roland,

Hey, yeah, the DrupalLoginTest is purley just a reference test case to show that it's working, and as an overly simplistic example...it's my little "Hello World!" :).

Here's a quick and dirty

IncrediblyKenzi's picture

Here's a quick and dirty representation of what I'm thinking:

This is just an example use case.. The idea is to allow the build to be extensible, and to be container agnostic (e.g. run it from the command line or a continuous integration server).

Only local images are allowed.

your replies are very helpul, aaron

Roland Tanglao's picture

thanks! and the diagram is lovely and exactly what i was looking for

trouble running (on ubuntu 8.04?)

dlhubler's picture

I had a little trouble running this, I'm on ubuntu which installs firefox 3 beta, which may be my problem, but I get

23:53:11,496 INFO [org.openqa.selenium.server.browserlaunchers.FirefoxCustomProfileLauncher] Preparing Firefox profile...

in setup.xml
selBrowserType is "*firefox /usr/lib/firefox-3.0b5/firefox"

but I'll look into it more in the AM.

tip to folks, I was instructed to download selenium-RC separately, I got it from here
http://release.openqa.org/selenium-remote-control/0.9.2/selenium-remote-...

I use buildbot for continuous integration. I'm not sure if I'd recommend it, but I've climbed the learning curve and somewhat glad I did. Written in Python so not a lot of java integration like cruisecontrol has.

Check out TeamCity..

IncrediblyKenzi's picture

We're using it internally, and it's wicked easy to set up. They have a free version as well:

http://www.jetbrains.com/teamcity/index.html

confirmed firefox 3 issue

dlhubler's picture

firefox 2 on ubunutu 8.04 launches fine.

But now DrupalLoginTest fails on line 67

public void testDrupalLogin() {
    // Login                                                                                                                                                               
    BROWSER.open(siteRootPath);
    BROWSER.type("edit-name", DrupalTest.getTestUsername());
    BROWSER.type("edit-pass", DrupalTest.getTestPassword());
    BROWSER.click("//input[@type='submit' and @value='Log in']");
    BROWSER.waitForPageToLoad("30000");                                                    <= Fails here
    assertTrue(BROWSER.isElementPresent("link=Log out"));
    return;
}

I haven't used selenium before, could you recommend any resources on working w/selenium.

Thanks for Using

Souvent22's picture

Dlhubler,

Hey, thanks for using. I really appreciate the feed back. Hm....could you post the exact "fail" message here? I'm going to try and setup an issue/ticketing system so anyone else who tries can post bugs/feed back.
I'm going to try and get together a highly detailed step-by-step of setting up Maven to use this "drupal Selenium maven build" today.
In the mean time, if you'll post the exact fail message, i'll look into it today. I'm assuming you're getting a Timeout message.

As far as Selenium resources; I usually use the Selenium site. This page has a link to many other good posts about using Selenium:
http://wiki.openqa.org/display/SEL/Getting+Started+with+Selenium+Core
Remember that all the ports of selenium have an object with the same methods. E.g. The following are equivalent:
PHP

$SeleniumBrowserReference->open("http://google.com");

Java
SeleniumBrowserReference.open("http://google.com");

I hope this helps.

SVN UP

Souvent22's picture

Hubler,

Try and SVN update, i just committed some changes. You'll need to update your setup.xml file though to include the protocol in the domain.
e.g. instead of just localhost, you'll just make it http://localhost.
Lemme know if you're still having issues.

As noted, there is a

Owen Barton's picture

As noted, there is a selenium module I wrote for Drupal 4.7 at http://drupal.org/project/selenium

It never got any attention from other folk, and I didn't have the time to support it just by myself, however it does a pretty good job of managing your tests (you can have them in the database or in SVN) and running them as a configurable suite (and recording/e-mailing reports). In retrospect I would just go for tests in SVN, and perhaps use folder structure for grouping suites - better to focus on using Drupal to simply set up and run the tests. The disadvantage of this approach is that it is not easy (cross-platform) to rebuild the Drupal site itself and test against an identical database snapshot each time (cross-platform). Using an external tool makes this easier, but has a much MUCH bigger overhead for users without lots of time and root-access.
Anyway, if anyone is interested in resurrecting the module please drop me a line.

I kind of see the point about having a testing framework separate from the thing you are testing (i.e. Drupal), however the actual testing engine (i.e. the selenium script) is independent of the module, so the amount of code you need to update is pretty minimal. Also, with a Drupal upgrade you will likely need to rewrite a very large proportion of the tests anyway so the test-runner module would be a minor piece of work.

Finally, some thoughts (pros/cons) on Selenium as a whole:
+ Selenium allows you to test things that Simpletest et al just cannot reach - the big one here is testing cross-browser Javascript and AJAX functionality. A Javascript unit testing library can also test these, but add the potential problem that the tests themselves need to work cross-browser (i.e. a test could look like it passes in IE, but actually you are just seeing 2 bugs cancel out) - a chicken/egg issue. With selenium it is stable enough on all browsers that this is much less on an issue.
+ The recording tool is great as a debugging helper (e.g. when you need to run a user registration again and again and again to catch some hard to find bug)
+ It's also great to write prototype tests (but see next line)
- If you want to repeat the tests created by the recording suite you either need (a) an identical starting database, (b) you need to manually add intelligence to the tests to account for incremented IDs etc or (c) you need to be clever when recording the tests to make sure there are no IDs involved, and the database is left in a consistent enough state to allow the test to run without conflicts.
- The tests can take a long time to run - they are fine for testing a few dozen very specific things (e.g. with a client site), but a test suite that attempted to test all functionality (and especially one that attempted 100% Drupal code coverage!) would run forever (making continuous testing quite non-continuous).
- The OS and browser requirement also adds overhead in terms of setting it up for automated tests (in terms setting up a VM, scripts to fire up the browser(s), deal with memory leaks etc).

Thoughts...

Souvent22's picture

Gurgnog,

Excellent comment post. I agree about testing against an identical database snapshot for each run..this can be a problem. However, we are addressing this with dbunit (http://www.dbunit.org/). It "...puts your database into a known state between test runs".

You are definatly correct in that the true amount of code (e.g. selenium testing code) to update is minimal, and b/t Drupal upgrades, the tests will have to be updated also; I just feel that, for example, using an unstable Drupal HEAD to run Selenium tests against itself is an issue. However, this can/could be addressed by say using a Drupal 6 install with the seleinum tests to run against the unstable HEAD.
But, one of the nice features of using Maven to run the Selenium tests is that, from a developers stand point, 1 command will fire up my Seleinum server locally, and then run my tests. Also, hopefully by next week, I'll be adding the MockSoap server to.
So the stack starts to look like this:

  1. Fire up local Selenium server
  2. Fire up the MockSoap server
  3. Open the Drupal site
  4. Test webservice X, which hits the MockSoap server and gets a canned XML/SOAP response
  5. Test weather Drupal does the correct thing after recieving the response.
  6. Shutdown the Selenium server
  7. Shutdown the MockSoap server

That's a quick and dirty.

Regarding tests taking a long time; Yeah, our Seleinum tests usually take about 10-12 min. to run our full suite. However, running the full suite of SimpleTests also can take a long time. One of the nice features though of Selenium is it's distributed grid testing architecture which we could leverage to speed up the tests.

>using an unstable Drupal

dlhubler's picture

using an unstable Drupal HEAD to run Selenium tests against itself is an issue.
why? I'm running on trunk...err I mean HEAD I guess... is it because the db schema keeps changing?

I think what would show why this is so complementary to simpletest, is to write a test for something simpletest could not test, some of the more dynamic stuff like views2. Then screencast it selenium doing it's thing.

Thanks for putting this together!

Selenium

calebgilbert's picture

I'm very interested in getting selenium rc to work, but not so interested in Maven+selenium (also, fyi the link to the workhabit svn repo is dead). I've never read about that combination, the selenium project itself and everything I can find online about it says that selenium rc requires phpunit, which after looking at maven sounds more appealing to deal with. (e.g., maven looks like it's own whole world - one that requires java experience/knowledge)

I've posted an article about several things related to Drupal, simpletest, selenium, and acceptance testing that I'm hoping will gin up some progress/feedback. Also, included a link to this post...

Update: An even newer article with screencast regarding a simpler selenium testing framework

Selenium Test Cases for Drupal.org

Shyamala's picture

Interested in supporting Selenium Test cases for Drupal.org, signup at http://groups.drupal.org/node/129939.

Usability

Group organizers

Group categories

UX topics

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: