Module metrics and ranking

Parsing .info files for dependencies

catch — Wed, 04 Jun 2008 14:12:01 +0000

Idea came up on this issue (twice) http://drupal.org/node/265450

This information would be useful for a couple of reasons:

Showing dependencies automatically on project pages - some maintainers are kind enough to list them in the description, but what's in the .info file is the best bet.

Showing related modules - modules like token, views, voting API could show dependent modules, modules with dependencies could show other modules with the same dependencies (cck, jQuery UI).

I guess we've already got code which determines what the dependencies are (although I've never looked at it) so showing dependencies on a project page might not be the hardest thing in the world, using it for recommendations would of course be a bit trickier. Does this sound like something that'd be workable? Should it be a feature request against project* if so?

Module metrics and ranking

Project taxonomy revamp proposal

catch — Wed, 04 Jun 2008 13:54:07 +0000

Every week it seems like there's a new issue in the webmasters/issue queue about new taxonomy terms for projects, this is a sure sign that the existing categories don't fit current needs. An obvious issue is that some of the categories are huuuge, which means in some cases you're just as well off browsing the full modules list. Who wants to look through 414 'utility' modules? What does 'content display' even mean.

So... let's look at reworking it. I see this having two elements to it. Overall, the categories should be task based, I want to look at modules which fulfil a particular need, I'm unlikely to want to view a list of modules that happen to include third party integration (although I might want to exclude those, which afaik I can't at the moment). For that, it'd be better to have a longer list of more specific categories, and less modules in them.

Additionally, more technical categories can be useful categories for some kinds of browsing especially for developers - say I want to find a module that supports Views2, or want to see examples of third party integration. However, I don't think they belong on the main modules listing - and strongly think we should move them to a separate free tagging vocabulary.

current list
* 3rd party integration (291)
* Administration (236)
* CCK (158)
* Commerce / advertising (78)
* Community (166)
* Content (392)
* Content display (422)
* Developer (157)
* e-Commerce (64)
* Evaluation/rating (65)
* Event (41)
* File management (54)
* Filters/editors (141)
* Import/export (63)
* Javascript Utilities (96)
* Location (38)
* Mail (100)
* Media (143)
* Multilingual (25)
* Organic Groups (53)
* Paging (19)
* RDF (15)
* Search (63)
* Security (55)
* Syndication (60)
* Taxonomy (119)
* Theme related (115)
* User access/authentication (138)
* User management (127)
* Utility (414)
* Views (92)

Rough draft of a new list:
* Advertising
* Amusements
* Audio
* Content administration
* Content rating and recommendation
* Content organisation
* Development
* E-commerce
* E-mail and messaging
* Events
* File management
* Form building and enhancement
* Forums
* Images
* Import/export
* Javascript
* Mapping and addresses
* Menus
* Multilingual
* Performance
* Permissions and access control
* Search
* Security
* Site building tools
* Syndication and aggregation
* Text editors
* Text filters and processors
* User authentication
* User administration
* User interface enhancements
* Video
* Web services and semantic web

That's 30 categories, I reckon it could get to 40-45 pretty quick (currently there's 38).

The next thing though, would be adding a separate free-tagging vocabulary, this would handle stuff like module groupings (views, cck, panels, ecommerce, ubercart, RDF API, Organic Groups), and stuff that's either not useful to 90% of Drupal users trying to find a module, or not task-oriented enough to go in the above list - things like Third party integration, taxonomy, node_access, third party migration, blogging, community etc. - which either have little to do with what the module actually does, or are too broad to be useful.

These tags could be displayed on project pages, in a list linked off project/modules somewhere, maybe a tagadelic thing at some point, and we could then use modules like "similar by terms" to show related modules based on this (or use it alongside pivots and other recommendation algorithms). We could also very slowly move tags from the freetagging vocabulary into the fixed one, if they turned out to be very useful, or vice versa.

I'd volunteer to do some of the work of reclassifying modules if something like this was implemented, obviously it'd be quite a big job though, but since only project maintainers and drupal.org site maintainers would be able to tag, it'd probably have some limitation on noise and bad tags.

Module metrics and ranking

Project module's hidden project metrics

sun — Fri, 11 Apr 2008 02:56:36 +0000

I have just re-discovered the http://drupal.org/project/issues/statistics in d.o's user menu. Below of the overall project statistics you get a nice overview of total issues by project and status. After clicking through the pages and changing the table sorting to Total, and while still looking at rather meaningless statistics, this idea crossed my mind:

The amount of issues in a project is my primary quality factor.
I don't care how popular a module is, or who likes or uses which modules, because I contribute (patches) to modules I use. And IMHO, this should be the primary purpose of project metrics: help projects to become better.
The amount of issues, compared to the overall lifetime of a project (creation date)
is an indicator for a module's reliability and popularity (not necessarily; see next bullet).
You can see relatively new projects on the first pages of the current project statistics, ordered by total issues!
The ratio (percentage) between open and overall issues
gives a pretty good idea about how actively a module is maintained.

To further break this down,
- the amount of bug reports, compared to overall issues
  tells me how well a module is coded, and how much a module has been tested.
- the amount of fixed bug reports, compared to overall bug reports
  tells me how well a module is supported.
- the amount of tasks, compared to overall issues
  tells me something about the quality and cautiousness of the maintainer(s), however, this one didn't work out in my examples (see below).
- the amount of feature requests, compared to overall issues
  tells me how many users actually use a module (and are interested in it).
- the amount of fixed/closed feature requests, compared to overall feature requests
  tells me how actively and dynamically a module is evolving.
- the amount of open support requests, compared to open issues
  tells me something about usability and how well the documentation of a module is.
- the amount of open support requests, compared to overall support requests,
  as well as the amount of participants (different users) in follow-ups of all issues
  tells me how many users are helping other users of a module.

These simple statements could be translated into actual statistics easily. We don't need additional modules, nor extra features to implement them as a (cached) block on project pages.

Lessons learned from drupalmodules.com: Voting in multiple axis on modules doesn't tell me something. Counting the downloads neither. A module can't be poor or worse than another just by relying on a download count or votes of other users. Such metrics only make already popular modules more popular and do not tell anything about real quality.

One could argue that such statistics should be calculated separately for each major version of a module. But I don't think that would buy us something. Instead, I'd rather start with overall statistics (like above) and try to find an automated way to determine whether a module has been re-written in great parts. For example, if a patch is committed that changes -1000 +1000 lines of a 1200 lines module, that might be an indicator for a whole new revision of a module (which would not work out on a second thought). Anyway, I'm sure someone will step up with an enlightening idea.

Example of project statistics with real world data:

Project         active  act(nm) CNR     CNW     RTBC    ptbp    fixed   duple   postp   won'tf  bydes   closed  Total
D Admin Menu    7       4       3       2                       3       14              11      15      75      134
Views           412     73      54      17      25      1       35      180     8       209     123     1122    2259
TinyMCE         278     8       30      2       7       1       3       80      5       17      42      364     837

Please note that some of the issue counts below are only approximate.

DAM
88% activity    -1*(16 open / 134 all)+1
43% stable      -1*(77 bugs / 134 all)+1
91% supported   -1*(7 open bugs / 77 bugs)+1
81% interest    -1*(25 features / 134 all)+1
72% evolution   -1*(7 open features / 25 features)+1
88% usability   -1*(2 open support / 16 open)+1
90% help        -1*(2 open support / 20 support)+1

Views
74% activity    582/2259
53% stable      1060/2259
77% supported   240/1060
80% interest    440/2259
68% evolution   140/240
62% usability   220/582
66% help        220/660

TinyMCE
61% activity    326/837
52% stable      400/837
60% supported   160/400
81% interest    160/837
59% evolution   65/160
67% usability   108/326
58% help        108/260

The example of TinyMCE actually represents my personal experience with the TinyMCE module/issue queue in hard numbers. Furthermore, I wasn't really aware of the high bug ratio in Drupal Administration Menu yet - I mean, sure, there have been "some", but now I learned that new features in DAM need more reviewers and tests in the future ;)

Module metrics and ranking

Building Drupal Modules

ccharlton — Mon, 10 Mar 2008 22:52:19 +0000

This session will introduce Drupal users to the beauty that is creating custom modules. We will start out by doing an overview of what a module is, what it is composed of, and how modules interact with the Drupal core. The lesson quickly turns from lecture to hands-on when the basics are over and students code their modules and learn all about Drupal's hooks and API offerings.

This session will go over how to build and architecture a new custom Drupal module.

Who is this course for?
This course is designed for people who have a basic understanding of Drupal and some coding or web development experience with technologies like HTML, CSS, and at least a little PHP. If you're interested in getting up and running with Drupal module principles then this is the course for you. All source will be handed out during or after the session.

Course outline and info available at http://www.richmediainstitute.com/training/drupalmodules/

Southern California

New Community-Powered Drupal Module Review Site

johnforsythe@drupal.org — Sun, 02 Mar 2008 22:28:41 +0000

Drupal Modules is a community-powered Drupal module review system I've been working on for about a year now. I'm happy to announce the site is finally open to the public!

Here's a brief list of what you can do on the site:

Rate and review any module
Compare modules to see which has the best features, documentation, etc
Check how many downloads a module is getting
Easily search modules and reviews for keywords
Get automatic recommendations for similar modules
And more!

This site is the result of approximately 1,000 hours of design and programming work over the last year, and I'm glad to finally launch it! Now I'm looking for your thoughts and opinions on how to make this the best resource it can be. I'm open to ideas about integrating with Drupal.org, but I also believe this site can flourish outside of the official infrastructure, much like the Theme Garden does.

I'd also like to extend a personal invitation to every Drupal user to sign in and review some modules! If you ever wanted to say thanks to a module developer, now's your chance to give them a glowing review. Maybe you found a hidden gem, and you want to tell the world! Perhaps you tried a module that didn't quite work, and you want to suggest a good alternative. The more reviews we compile, the more useful this site becomes for everyone. :)

If you'd like to learn more about what's powering the site, check out the about page.

One more thing: Thanks to Walt for creating the excellent Contributed Module Status list, it's a wonderful resource, and I've added a link to it on the site.

--
John Forsythe
http://DrupalModules.com/

Social Networking Sites

Duplicate efforts?

Chris Johnson — Wed, 12 Dec 2007 09:46:49 +0000

See this node http://groups.drupal.org/node/7191 titled "Project Quality Metrics" in the Drupal.org redesign plan for the Drupal Association group. It looks like all that activity should either be here, or all the activity intended for this group should be there, i.e. duplicate efforts.

Module metrics and ranking

Design document

MattKelly@drupal.org — Tue, 04 Dec 2007 05:25:20 +0000

Drupal Module "Directory"

Greg Knaddison (greggles), Matt Kelly (MattKelly), Michelle Cox (Michelle), Chris Johnson (chrisxj)

Keep in mind this is just a rough draft comprised of dialog between the above people, so far.

I suggest we try to plan the features in several phases. The completion of features in the first phase would be the point at which we first release it for public use. I think it would be a good idea to have some plans for future phases before we start so that we don't make decisions in building the first phase that cut off paths to future ideas. Greg had some good ideas for searching/filtering/rating criteria that he mentioned in IRC. I hope he will add them here. (added below - greggles) Feel free to edit this document. Once everybody has gotten started, we can delete a bunch of this introductory narrative stuff I've put here to get it going. [chris]

Motivation and Initial Thoughts

This is just narrative at the moment. Eventually it would be nice to parse it up into lists with details for each point, or some other structured way.

I was prompted to build this project by the repeated frustration I felt each time I tried to find modules to do various things. Often, it was simply trying to find a module I knew existed and what it did, but just couldn't remember the exact name. Other times it was trying to discover if a module existed which supplied functionality "X". And yet other times, it was trying to find the best module to provide functionality that I knew was supplied by several modules (e.g. have you tried figuring out which module or modules are best for providing in-line image insertion in D5 using an editor other than TinyMCE? A solution does exist. It's just hard to find!).

It's obviously needed in the community, as can be seen from the forums, irc channel, and this blog post from Dries: http://buytaert.net/drupal-org-wishlist [matt]

The solution seems to be to create a better module searching application and add to it the ability to rate and criticize the modules. The current drupal.org listing and taxonomy system is too cumbersome. And with hundreds of modules and a growing number, it's not getting any better.

Lastly, I think we can provide the Drupal community a valuable service by solving the above problems. And we might even produce two additional positive effects: (1) encourage better modules via the rating/critiques, and (2) boost Drupal's reputation around the world by making it easier for people to see the wide diversity and great depth in the contributed modules available. [chris]

Google-style Weighting [matt]

To deter authors from artificially boosting their rating--as seen on other sites, like Joomla--we should promote a more complex weighting system. Taking many of the factors mentioned below, under "some random features", into consideration, we could properly weight every modules importance.

Inspiration and Ideas [matt]

http://groups.drupal.org/node/3314 (Project metrics, GSoC 2007 project)
o We should contact drewish to see if he wants to help
http://drupal.org/node/139291 (Peer review facility for modules)
http://drupal.org/node/77976 (Project quality indicator)
http://www.ohloh.net/projects/3189?p=Drupal+%28core%29 (Ohloh project tracking)

Some random features

Fast search of modules by key word or partial word. Great for people who know a module exists but can't quite remember the name.
Search filters for various important criteria, such as install file, official release, documentation, recent commits.
o Does it have 1) documentation page 2) hook_install/hook_uninstall 3) pass a coder "standards" check 4) pass a coder security check 5) have simpletests 6) have complete simpletests
Rating by popularity.
o Views
o Downloads
"Health/Activity", based on [matt]
o Developers working on the project
o Number of commits
Rating by "completeness" in meeting a set of criteria (similar to search filters above).
We have a choice: we can either generate a comprehensive list from CVS or some other method, or we can choose to only list modules worthy of looking at. Listing only worthy modules has some advantages. One disadvantage might be we may anger some module developers whose modules got excluded. This point is probably worth further thought and discussion.
Allow user voting / rating of modules -- this means any user who visits the site (authenticated?).
Rating of modules by an "editorial" staff -- i.e. a hopefully large list of known Drupal community members, rather than just anybody who comes along.
Make use of tagging keywords. This would be useful to make an interactive experience for newbies. [matt]
o For example, a newbie could go to a form that asks them questions like "What type of website are you looking to build?". If they select something like "blog", the module would pull up useful/recommended modules based on tags.

Project Quality Metrics on Drupal.org (meta document)

Caleb G — Sun, 18 Nov 2007 00:19:16 +0000

According to a survey of over 1,000 Drupal users, the most requested feature for Drupal.org is a recommendation system for the modules section. With the exponential proliferation of Drupal modules it's certainly easy to imagine why.

Because the discussion about this subject has been splintered into so many places, it was suggested (see here and here) to create a thread within the D.O. Redesign group in order to centralize the brainstorming, mockups, proposals, etc for this initiative.

What this thread is intended for (comments from dww):

Meta-discussion on the proposals: which ones are clearly a good idea now, which ones need more thought, which ones aren't going to work at all, etc.
Volunteers to work on the "low hanging fruit" -- the proposals that we can right now agree would be worth doing, but which won't take a huge amount of code to accomplish.
Further discussion on the specific proposals that need to be fine-tuned or otherwise better thought-out. I consider this particular proposal for end-user ratings and reviews (issue #50605) in this category. There are a lot of potential problems this will create, so we need to have good answers for those if we want to continue exploring this.

Here's an initial list of all known efforts in this space:

#32124: Enable download statistics
#187019: Checkout statistics from CVS
#50605: Project.module: Add user ratings for projects
#52475: browse projects by # of downloads
#66013: Create project sorting method using data from update.module
#77976: Project quality indicator: rating system [duplicate with #50605 -- but still some potentially useful ideas]
#79550: Automate gathering of quality metrics
#99466: add 'orphaned' module category?
#63491: Drupal Version-Module Support Matrix
#165380: Make usage statistics (from update_status) visible
#192410: Deploy module recommendations based on content in page, and recommend modules on module project pages (aka "Pivots")
#203313: Add a way for maintainers to indicate multiple supported branches
http://groups.drupal.org/node/3314 drewish's original SoC proposal about project quality metrics
http://groups.drupal.org/node/5022 Drupal Project Metrics (wiki during Drewish's SoC work)
http://groups.drupal.org/node/6186 Project node UI redesign

There are probably others scattered around, but this is at least a start... ;)

Before posting please consider reviewing some of the links included above, if you haven't already. There has already been lots of discussion and some activity for this topic, the goal now is to reach a point of broader consensus and action.

Module metrics and ranking