Develop an automated javascript testing framework

Posted by catch on March 7, 2008 at 6:28pm

Added to official ideas list at http://drupal.org/node/234712

(Draft SoC 2008 project proposal)

Drupal will be using simpletest for php functional and unit testing, but currently we have no equivalent framework for javascript.

This means that every time there's a minor jQuery update (or any other javascript patch to core), about six different people have to spend hours clicking around to see if anything broke.

The project should have integration with the Drupal.org issue queue and also be possible for a developer to set up locally (i.e. using open source tools). This will involve not only Drupal/php, but also javascript, jQuery and other fun stuff.

The framework must be pluggable for different OS and browsers (i.e. IE6/7, Firefox, Safari, Opera, Konqueror).

References:

http://testing.drupal.org
http://wtr.rubyforge.org/ (WATIR browser automation library, written in Ruby)
Simpletest automation was the product of a 2006 SoC project: http://drupal.org/node/58881
http://browsershots.org does a similar thing for css/markup on multiple platforms
http://groups.drupal.org/javascript
http://groups.drupal.org/unit-testing

Comments

Selenium? Bevan/

Posted by Bevan on March 14, 2008 at 9:05pm

Selenium?

Bevan/

A few people at the unit

Posted by catch on March 15, 2008 at 10:25am

A few people at the unit testing BoF in Boston said they'd tried Selenium and didn't have good experiences with it at all. I personally don't have an opinion on this (although watir looked pretty good) - as long as one day we can get rid of some of the clicky clicky on jQuery updates etc. :)

I love this idea, but we need more "meat"...

Posted by webchick on March 15, 2008 at 7:08pm

At the moment, this looks like a research task rather than a coding task. I don't mind there being a research component in a SoC project, but we need a lot more details on what would actually be involved in the coding part, which it seems like we don't have because we don't know what JS testing library we're using, which is why we have a SoC project to figure it out. :\ Chicken/egg. :\

Anyone else have experience with this who could help? I've also cross-posted this to the Unit Testing group.

Just to make it clear to

Posted by catch on March 15, 2008 at 10:35pm

Just to make it clear to anyone reading, I have no idea how to actually implement this. I just don't want to get RSI every time there's a jQuery update :)

chx: on irc suggested (jokingly) rewrite watir in php/drupal. It'd been in the back of my mind but I have no idea if it'd be possible.

However "reimplement watir in php/drupal" is very, very specific.

OK via irc this is what

Posted by catch on March 15, 2008 at 11:18pm

OK via irc this is what we've got so far:

Three steps to the task:

research the various javascript unit testing options to find one that will allow us to automate functional and unit testing of Drupal's javascript.

This means a framework that developers can run on their own system and write tests for (so either in php, or really, really easy if it's using something else)

Write a module that integrates this framework with Drupal - so that tests can be submitted as patches, and tests can be triggered from the browser or command line.
Write some example tests based on webchick's handy big ass (but not exhaustive) list of javascript tests for core http://groups.drupal.org/node/5974

Just found this 4.7 module

Posted by catch on March 16, 2008 at 11:26am

Just found this 4.7 module by grugnog which does Drupal/Selenium integration: http://drupal.org/project/selenium

A good start would probably

Posted by Nick Lewis on March 17, 2008 at 5:41pm

A good start would probably be to ask jQuery how they've handled testing.

"I'm not concerned about all hell breaking loose, but that a PART of hell will break loose... it'll be much harder to detect." - George Carlin
work: http://www.onnetworks.com
blog: http://www.nicklewis.org

"We are all worms. But I believe that I am a glow-worm." - Winston Churchill
work: http://www.chapterthree.com
blog: http://www.nicklewis.org

I've given some more thought

Posted by catch on March 17, 2008 at 8:10pm

I've given some more thought on how I'd like to see this working.

At a minimum we'd end up with this:

A farm of OS/browser combinations. At a minimum - IE6/IE7, Opera 9, Firefox 2/3, Safari 3 (and 2? and Konqueror?) on windows/linux/mac.

Tests implemented for a browser automation framework (watir/selenium - ideally in php/drupal*) which allows us to automate various core operations involving javascript. Drag and drop, scrolling down for floating tableheaders, teaser splitter, expanding a collapsed fieldset, clicking an add more AHAH button, stuff like that. And maybe a click through 'all of core' one.

You submit a patch, the farm goes off and runs the test - you can then view a video of what happened for the various combos.

This would be on almost exactly the same model as browsershots.org except it gives you video back instead of images.

*Do we absolutely need to use php/drupal to run the tests from?

For stage two, we'd have some automated error reporting in there as well. But this kind of stuff is subjective (how do you tell if a fieldset expanding is flickery or not?) so I think that's less of a concern than a way to centralise browser compatibility tests - especially since often people working on js bugs (or features) don't have access to three operating systems and all browser configurations.

Recieved this via e-mail

Posted by catch on March 18, 2008 at 12:00am

Recieved this via e-mail from rolfkleef because he's having trouble posting directly:

I'm trying to publicly respond to http://groups.drupal.org/node/9511 but
keep getting a "Validation error, please try again."... So here it is
directly, hopefully :-)

I cannot help a lot with the "meat", but I have a very little bit of
experience with Selenium (and working with ThoughtWorks, the company that
developed Selenium). A quick search leads to an article about using
Selenium to test AJAX apps
(http://www.infoq.com/articles/testing-ajax-selenium). Perhaps there's also
interest at ThoughtWorks to help with this SoC project? (The issue of
javascript testing has come up on the Selenium list as well, and I've also
seen discussions or tools about translating Selenium tests into SimpleTest
scripts)

And BrowserCam might be a useful resource in developing the framework: it
not only offers screenshots in various browsers (like browsershots), but
also 2-hour VNC sessions to do more interactive testing. I think it's
possible to let Selenium post the results back to a specific URL, and tell
Browsercam to wait for a period of time before making a screen capture
(pointing to the Selenium test suite and allowing some time to run the
tests and post the results?). (see http://www.browsercam.com)

I can't guarantee suitability :-) and don't have time to test it myself
:-( but I'm happy to provide browsercam access under our account for people
to check it out!