I have just re-discovered the http://drupal.org/project/issues/statistics in d.o's user menu. Below of the overall project statistics you get a nice overview of total issues by project and status. After clicking through the pages and changing the table sorting to Total, and while still looking at rather meaningless statistics, this idea crossed my mind:
- The amount of issues in a project is my primary quality factor.
I don't care how popular a module is, or who likes or uses which modules, because I contribute (patches) to modules I use. And IMHO, this should be the primary purpose of project metrics: help projects to become better. - The amount of issues, compared to the overall lifetime of a project (creation date)
is an indicator for a module's reliability and popularity (not necessarily; see next bullet).
You can see relatively new projects on the first pages of the current project statistics, ordered by total issues! - The ratio (percentage) between open and overall issues
gives a pretty good idea about how actively a module is maintained.To further break this down,
- the amount of bug reports, compared to overall issues
tells me how well a module is coded, and how much a module has been tested. - the amount of fixed bug reports, compared to overall bug reports
tells me how well a module is supported. - the amount of tasks, compared to overall issues
tells me something about the quality and cautiousness of the maintainer(s), however, this one didn't work out in my examples (see below). - the amount of feature requests, compared to overall issues
tells me how many users actually use a module (and are interested in it). - the amount of fixed/closed feature requests, compared to overall feature requests
tells me how actively and dynamically a module is evolving. - the amount of open support requests, compared to open issues
tells me something about usability and how well the documentation of a module is. - the amount of open support requests, compared to overall support requests,
as well as the amount of participants (different users) in follow-ups of all issues
tells me how many users are helping other users of a module.
- the amount of bug reports, compared to overall issues
These simple statements could be translated into actual statistics easily. We don't need additional modules, nor extra features to implement them as a (cached) block on project pages.
Lessons learned from drupalmodules.com: Voting in multiple axis on modules doesn't tell me something. Counting the downloads neither. A module can't be poor or worse than another just by relying on a download count or votes of other users. Such metrics only make already popular modules more popular and do not tell anything about real quality.
One could argue that such statistics should be calculated separately for each major version of a module. But I don't think that would buy us something. Instead, I'd rather start with overall statistics (like above) and try to find an automated way to determine whether a module has been re-written in great parts. For example, if a patch is committed that changes -1000 +1000 lines of a 1200 lines module, that might be an indicator for a whole new revision of a module (which would not work out on a second thought). Anyway, I'm sure someone will step up with an enlightening idea.
Example of project statistics with real world data:
Project active act(nm) CNR CNW RTBC ptbp fixed duple postp won'tf bydes closed Total
D Admin Menu 7 4 3 2 3 14 11 15 75 134
Views 412 73 54 17 25 1 35 180 8 209 123 1122 2259
TinyMCE 278 8 30 2 7 1 3 80 5 17 42 364 837Please note that some of the issue counts below are only approximate.
DAM
88% activity -1(16 open / 134 all)+1
43% stable -1(77 bugs / 134 all)+1
91% supported -1(7 open bugs / 77 bugs)+1
81% interest -1(25 features / 134 all)+1
72% evolution -1(7 open features / 25 features)+1
88% usability -1(2 open support / 16 open)+1
90% help -1*(2 open support / 20 support)+1Views
74% activity 582/2259
53% stable 1060/2259
77% supported 240/1060
80% interest 440/2259
68% evolution 140/240
62% usability 220/582
66% help 220/660TinyMCE
61% activity 326/837
52% stable 400/837
60% supported 160/400
81% interest 160/837
59% evolution 65/160
67% usability 108/326
58% help 108/260The example of TinyMCE actually represents my personal experience with the TinyMCE module/issue queue in hard numbers. Furthermore, I wasn't really aware of the high bug ratio in Drupal Administration Menu yet - I mean, sure, there have been "some", but now I learned that new features in DAM need more reviewers and tests in the future ;)
Comments
This looks great. One
This looks great. One comment:
At the very least, it should only be calculated for actually supported versions (i.e. Views 1 in 4.7 shouldn't be influencing metrics of Views 2 in D6). Additionally Drupal major versions sometimes (not always) coincide with new maintainership as well. For 5.x-1.x to 5.x-2.x kind of changes I'm not sure it'd be worth it though (and those are often arbitrary between different modules anyway).
Code + Maintainers
As mentioned in the post, most modules do not change much between major Drupal core compatibility (5.x -> 6.x), and often not between major module versions (5.x-2.x -> 5.x-3.x). However, I agree with you that the quality of a module might totally change if a module is re-written, or maybe also, if there is a new maintainer. We would need a really intelligent way to figure this out.
On the other hand, since above statistics actually depend on the issues in a project's queue only, it could turn out to be completely unnecessary to "reset" these project quality statistics/metrics on a certain date. In the end, the proposed solution would solely depend on the love spent to a project's issue queue. And IMHO that's the way to go - since we removed module-specific forum containers from d.o recently, the issue queue is the primary quality factor of a project.
Daniel F. Kudwien
unleashed mind
Daniel F. Kudwien
netzstrategen
True, I guess if it's issue
True, I guess if it's issue queue metrics, then old mouldy 4.6 issues at the back of the queue are fair game too.
Ideally a new maintainer would go through and mark old issues as dups/closed/fixed/won't fix as part of the takeover process. Obviously that's not always the case but it'd be an easy way to fix inherited bad rankings.
Attention: calculation
Based on the proposed calculation method above, "closing" outdated issues would not have that much positive effect on project metrics, since they are based on overall (all-time) issues. And I think that's correct. Just because a module has been ported to a new major version of Drupal core, that doesn't mean its quality has increased. I often find myself closing outdated issues (f.e. for 4.6.x) without even knowing if those might still be valid. I think they are not. But often I simply can't tell, because I'm just a new co-maintainer that started to work with a module compatible to D5 or D6.
Daniel F. Kudwien
unleashed mind
Daniel F. Kudwien
netzstrategen
Merge existing ideas
FYI: Drupal Project Metrics contains some valuable ideas about additional project statistics that could be incorporated into the proposed metrics here.
Daniel F. Kudwien
unleashed mind
Daniel F. Kudwien
netzstrategen
These statistics don't reflect quality.
I don't believe these statistics will really tell you much about quality.
Partly because different maintainers deal with issues in different ways. Some might be in the habit of closing everything, some might be more lax about leaving them open. One may feel more comfortable dishing out lots of "won't fix" or "by design", while another may leave the door open with a "postponed" or "needs more info". That doesn't really reflect on the quality of code in the module.
If anything, these stats will more often reflect the mindset of the average user of that module. If a module appeals to noobs, it will attract more support requests, and worse, more false bug reports.
In other words, a module's target audience will significantly skew the stats in ways much heavier than the actual quality of the module will. Remember, most issues are created by users, not by coders. This is really important to consider when evaluating what the numbers mean.
Other stats will also be affected by what kind of module it is. If it's something people feel a need to customize, it will get lots of feature requests. You will see this number go up in certain module categories, and down in others. But it won't tell you much of anything about module quality.
There are lies, damned lies, and statistics :)
PS: I agree that the ranking system on Drupal Modules does have a "rich get richer" effect, but this is really hard to do anything about. People love Top 10 lists, and those kind of lists have an unfortunate tendency to be self-reinforcing. It's my hope that the strong search facilities present on the site help to make lesser known modules more visible.
--
John Forsythe
Interesting view
That's an interesting view... Here comes mine, paragraph by paragraph:
If a module maintainer does not close issues, he proves poor maintainer quality. If issues are not (re-)classified correctly, that's poor quality. If a queue is full of issues marked postponed or "needs more info", that's poor quality.
If an issue queue is full of support requests, the usability and/or documentation of a module has poor quality. Even the code might be of poor quality.
If an issue queue is full of feature requests but contains almost no bug reports (example), that speaks of high quality, high interest, but not much activity.
PS: I was not my intention to debase your efforts on drupalmodules.com. You know, I like it, since at least one of my modules is listed in both top lists. But anyway, I do not think that voting on projects/modules helps any Drupal user.
Daniel F. Kudwien
unleashed mind
Daniel F. Kudwien
netzstrategen
Interest isn't quality.
It depends on what you're looking for. Someone who's skilled at coding, or someone who's a dedicated issue queue manager. One isn't a guarantee of the other (on the other hand, they're not mutually exclusive, either).
Issue queues full of support requests could simply indicate a complex module that appeals to inexperienced (or lazy) users.
An issue queue full of feature requests but no bugs doesn't necessarily indicate quality. It could indicate a module that doesn't do what most people need, so no one actually uses it. Hence lots of feature requests, but no bug reports.
I definitely agree you could use these statistics to find interest. Obviously more issues = more interest. But interest doesn't mean quality.
Some modules attract demanding, inexperienced users who post tons of duplicate issues, useless feature requests, or inaccurate bug reports. Some modules are owned by godlike coders with no time to moderate an issue queue full of those kinds of useless issues, and so they only deal with the ones that really matter. Some modules only attract programmers who post detailed bug reports. Some modules often get issues created for things unrelated to the actual module. Some modules work perfectly and have an empty issue queue. Some modules have no users and an empty issue queue.
The issue queue doesn't reflect module quality, it reflects user base, and issue queue moderation style.
Module quality cannot be reliably determined by how users act. There is no relation between code quality and user type. You can't rely on users to generate quality issues, thus you can't rely on issues to indicate quality modules.
At best, this is like trying to guess the color of a dog by its shadow.
PS: I'm not trying to debase your efforts, either. These stats are very good, but not, in my opinion, for determining module quality. Time spent pruning the issue queue, sure. User interest, definitely. Quality... I don't see it.
Also, I agree that numeric ratings, even multi-axis, are not that useful. Not on their own, anyway. Ratings coupled with (well written) reviews, on the other hand, do have value.
--
John Forsythe
easily gamed
While Derek and others in other issues and discussions (liked http://groups.drupal.org/node/7191) mentioned that machine stats are the true thing, and human voting and rating is gamed, these machine stats you have are also easily gameable. I can have a few users and submit issues against my module just to quickly fix them with another one. Some people are cautious about adding rating to drupal.org because of the same stuff, that you could rate your stuff with multiple users.
I think we just need to accept that stats will be gameable, and not treat "machine stats" as any more reliable as "human input stats". The above stats are just as working with human input as download stats or ratings would do, it is just not as obvious as a click on a voting widget.