Posted by dlhubler on April 14, 2008 at 3:41am
I ran all the unit tests and I get
Test cases run: 57/58, Passes: 3123, Failures: 282, Exceptions: 34
Is this what other folks get?
Question 1: Do we need to coordinate the fixing of them now?
Question 2: Are we striving for 0 failures at the end of all this?
For me question 2 comes first. Unless we decide we're striving for 0 failures (and I think we should) then if someone puts the time in now to fix tests, tomorrow they may break again and no one will notice. If no one notices, we cannot correlate commits to unit test failures which is an important part of testing, if not the most important part.
| Attachment | Size |
|---|---|
| unit-test-pass-fail.txt | 3.37 KB |
Comments
side note: my 2 hour end result trying to fix a failure
test file: functional/comment.test/testAnonymous
line:146
// Post comment with contact info (required).$anonymous_comment3 = $this->postComment($this->node, $this->randomName(), $this->randomName(), TRUE,
array('mail' => 'tester@simpletest.org'));
apparently there is a "Save" button on first comment post that is unexpected, a "Preview" was expected. I cannot tell if this is a bad test or bad code, so I try to use drupal and setup a new node but I'm not sure what an admin would need to do to achieve the equivalent of this:
$this->setCommentAnonymous('2');which is called in the test case before the failed assertion.
SimpleTest Issue Queue
Please post the issue with comment in the queue.
Yes, we are striving for zero failures and I have fixed the majority of the tests with help from a few others. The tests have come along way. At the beginning of the process we had a much smaller percentage of tests written and only a hand full passed. If you look at the status list you will see that we are getting close.
I have been aware of the issue with comment, but I'm not sure why it occurs. I can't confirm it is a bug in the interface or the test. The test passed at one point. I will look into it further, but have been busy fixing other tests that completely fail. Feel free to do your own investigation. As a note the equivalent to
setCommentAnonymousis located on the content type edit pageadmin/content/node-type/[type].Another issue is changes to the core that break the tests which happens fairly frequently. I discuss that issue in my blog post.
changes to core
With changes to core I think probably the best thing to be done now is name and shame. i.e. post up on the issue that broke the test and say "this broke X core tests, here's the issue for fixing them". Once things are in core, if patches are committed without tests, then they can be marked to needs work, the same way as currently happens for documentation and upgrade instructions.
I've been slowly looking
I've been slowly looking through some of the tests that fail and did a before/after comparison on this registry followup issue. One thing I think would help, a lot, is if you had a list of all failing tests/assertions right at the top of the testing report. This'd be great for simple counts of what fails in which test, and when we have daily runs of tests against HEAD, we could maybe log a list of failing tests on Drupal.org somewhere - even with revisions to see what's new and what's old.
edit: feature request - http://drupal.org/node/257923