Please correct me if I'm wrong, but it seems that this group's purpose is to concentrate on unit testing and code coverage. Excellent. This is a definite sign of maturity for Drupal.
But I also see some posts in here that refer more to UI centric testing. Is there yet an effort for organizing this testing as well?
Unit testing is a great way to find problems that may not always be obvious in the UI, but they are not always the best way to catch problems which may be obvious in the UI. For example, the unit tests for the forms API may confirm that the form tokens are being generated properly but not pick up on the fact that another module alters the token (making this up here :) and so the form submission ultimately fails. At the end of the day, you really need to run through use cases to make sure it all works.
Typically, folks will run functional/UI tests by hand (create, edit, delete a node by hand, etc...). This is OK, but not ideal. Manual testing is time consuming and difficult to reproduce.
I've done testing of applications where automation was not an option, and as standardized as I tried to make the list of tests I ran, they were never run the same way twice, leaving little crevices for regression to creep through.
I believe we should consider trying to automate UI testing as well as unit testing. This would give us a gorgeous set of regression tests.
I guess my main question is whether this is the place/time to bring that up. I don't think it would take any attention/resources away from the unit testing efforts here.
Comments
Related software
Browser automation tools allow for scripted testing of web applications through an actual browser. There's no better way to automate use case tests as far as I know.
I have experience using Silktest and Evalid. However, both of these are commercial packages and so probably not suitable. Watir is an FOSS alternative. I've never used it.
Anyone else think this type of testing would be worth looking into in addition to the current unit testing efforts?
--
Drupal tips, tricks and services
http://devbee.com/ - Effective Drupal
--
Drupal tips, tricks and services
http://devbee.net/ - Effective Drupal
Webload
Webload, from RadView, was made open source, and includes an authoring tool that records your actions in a browser and allows for playback. They're recorded in Javascript, and so can be edited, such as if you wanted to make your recorded script access a file to test 500 different codes to make sure they all work in a form field.
It definitely works in Windows. They have been working on making it work in Linux, but the results have been varied and I can't even find the link to download the app in Linux now.
http://www.webload.org
Josh McCormack
Owner, InteractiveQA
Social Network Development & QA testing
http://www.interactiveqa.com
917.620.4902
aim: jmccormciti
skype: joshmccormack
Josh McCormack
Owner, InteractiveQA
Social Network Development & QA testing
http://www.interactiveqa.com
917.620.4902
AIM:
jmccormciti
Skype:
joshmccormack
Yahoo:
joshmccormack
Google Talk:
joshmccormack@gmail.com
Windows Live
SimpleTest includes user agent testing
Was just reading this: http://www.lullabot.com/blog/drupalcon-simpletest-and-future-test-driven...
Didn't realize simpletest handles functional testing too.
So maybe all forms of testing can be handled using simpletest. This would simplify things :)
I'd never heard of simpletest before it came up here, so I guess I need to do some reading up.
--
Drupal tips, tricks and services
http://devbee.com/ - Effective Drupal
--
Drupal tips, tricks and services
http://devbee.net/ - Effective Drupal
There's currently something
There's currently something like 60% SimpleTest coverage for functional tests in core, and < 1% coverage for unit tests.
What we don't have yet is javascript testing, which I've written up as a GSoC project idea - http://groups.drupal.org/node/9511 - Dmitri's also got a patch for JavaScript unit testing here - http://drupal.org/node/237566
Conflation of terms...
Harry, I agree, I think there's some confusion as to exactly what the focus of the group is. What I learned is that "Unit Testing" is ensuring that your functions/methods meet their contracts by passing various arguments and comparing the expected results. Functional testing (perhaps the cause of the confusion) is for higher-level "functionality", e.g. logging in. Then there's regression testing any time a change is made (basically re-running all unit and functional tests. Finally, user acceptance testing, which is pretty much functional testing, but is usually performed outside the development organization.
For all intents and purposes, lumping them together here isn't the worst thing that could happen. As the group, and Drupal testing, matures, I think we'll see some splintering of this group into groups that are focused on each different type of testing. Perhaps we can set forth a definition of the various testing domains and create associated groups?
That being said, Simpletest does seem to blur the lines between unit and functional testing, which doesn't help clarify the situation, but does bring more features to the party.
re: scripted browser testing, there are several tools out there. I've even gone so far as to write WSH scripts to turn IE into an automated use-case tester. Not the most elegant, but certainly doable and free.
Have we considered using
Have we considered using something like Selenium and its IDE to provide acceptance tests for core functionality? It provides browser-based testing, has a fairly comprehensive IDE, and if we get really clever we could start integrating it with remote control to handle the continuous integration case down the line.