Project quality metrics system

Andrew Morton
http://drupal.org/user/34869

Synopsis

I propose work on enhancing Drupal's Project module to add a
system that computes a number of quality metrics to assist in rating projects.
These metrics will help users determine which of the many contributed modules
and themes are being actively developed and maintained.

The Problem

Perhaps one of the most frustrating parts of building a Drupal site is
evaluating the many contributed modules to determine which meet the needs of
the site. For many non-technical users who are unable to fix their own bugs,
finding a module or theme that does what they need is not enough;
non-technical users need a module that is actively maintained and supported.

Recent changes to the Project module have made it far easier to find modules
and theme by category and by Drupal version compatibility but there is little
to help determine which modules are well supported. By looking at a theme or
module's project page, it is not readily apparent if there are developers and
users who can help answer questions, fix bugs, and ensure the modules are kept
up-to-date and operable. 

It's been suggested that count of downloads would provide a solution. I think
a simple ranking-by-number of downloads would only encourage developers to
"game the system", starting downloads to artificially increase the popularity
of their modules. A more comprehensive approach is needed, one that helps to
highlight the well maintained but un-popular (perhaps because of
a badly chosen name) module, and deemphasize the popular but
ill-maintained modules.

My Solution

The first, and most important, goal of my project will be to build a flexible,
efficient structure for computing metrics quality. Once that is in place, the
task shifts to determining what data will best indicate the activity around a
theme or module. Making sure that data is captured and available for analysis
is the natural next step. The final piece, and this is mostly specific to
Drupal.org, is to set weights for the metrics so they match with users
subjective feelings about a module's activity.

I envision building a sub-module for the Project module named project_metrics.
The module would:

• Compute and cache the metrics periodically from within a
  hook_cron() implementation.
• Use a plug-in architecture that groups metrics into .inc files. Each .inc
  file will have a dependency function to avoid loading the plug-in when its
  prerequisites are not in place. For instance, a user quality rating metric
  would depend on having a voting module enabled.
• Display a block on each project page with a textual summary of the metrics.
• Add "browse by" pages for users to view modules and themes sorted by their
  quality metrics.

Each metric will:

• Present itself as a value from -1 to 1 representing "very bad" to "very
  good"
• Present itself as a string understandable by an end user, e.g. "This is one
  of the most used modules on Drupal sites."
• Provide an additional "help" string explaining what it means, e.g. "This
  module is used by more than 80% of the sites that have enabled the Drupal
  module and allow it to report usages statistics."
• Provide configuration form to allow the administrator control over its
  "magic numbers", e.g. between X and Y commits in N months is good but more
  than indicates extremely active development and likely an unstable module.

The metrics will be grouped into categories, each reflecting a different usage
axis:

• Developer activity - how often are new releases offered, how recent are
  commits, the number of feature requests opened vs. the number of closed, the
  number of uncommitted patches, etc.
• User activity - the number of downloads, the number of sites using it based
  on Drupal.module callback data.
• Support activity - the number of support requests opened vs. the number
  closed, the number of unique people commenting on issues.

The metrics will be averaged to determine a rating for each category. The
administrator can alter the weight of each metric within the category to make
one more important or to set the weight to zero, disabling the metric. For
example, the number of sites using it is twice as important as the number of
download, or don't factor the number of downloads in at all.

Time allowing, an additional aspect to considered is the role releases should
play in computing the metrics. Specifically, how should differences in version
upgrades be  represented (or weighted) if the current version works
perfectly but the newest release is buggy? If the 4.7 version of a module
worked perfectly and was widely used but the 5.0 version is a buggy,
incomplete upgrade with many open support requests, how should that be
reflected? Issues and bug reports are now linked to specific release nodes so
it should be possible to generate metrics on a per release basis. Research
would need to be done to determine whether and how this information will be
more useful. 

Profit for Drupal

This project will provide an immediate benefit for every Drupal user who is
trying to determine which modules and themes to use on a site. They will be
able to skip over the unsupported and buggy modules and themes and spend time
evaluating ones of higher quality. By highlighting the best Drupal has to
offer, new Drupal users will be given a much better first impression.

Roadmap

The following is a guide to what I hope to have finished at points through the
summer.

June 11th

• Setup a test site with a database dump of realistic project information that
  can be used during development to test the performance and scalability of
  the metrics.
• Chose two or three simple metrics from data the project module currently
  collects, e.g. number of downloads, number of open support requests, number
  of maintainers, time since new release was built.

July 1st

• Create a project_metrics module skeleton and implement the basics of the
  metric plug-in infrastructure.

July 16th

• Complete several metrics for data already collected in the project module.
• Start developing the user interface for displaying the metrics and browsing
  for modules by quality.

August 1st

• Create additional, more complicated metrics.
• Enable the module on scratch.drupal.org.
• Begin to determine appropriate weights for Drupal.org.

August 20th:

• Upload my final code to Google's site.
• Enable the module on Drupal.org.

All my work will be done in coordination with Derek Wright, the Project module
maintainer, to ensure my contributions meet with his approval and can be
committed to the module at the end of the summer and put into use on
Drupal.org.

Biography

I am in my senior year studying Computer Science at Portland State University
in Portland, Oregon. I have been working with open source PHP projects for
over four years. My first effort was Phlickr, a
Flickr API kit written for PHP 5. With Rob Kunkle, I co-authored
Building Flickr Applications with PHP published by Apress. This book covers using Phlickr as a tool for building websites and scripts to manage photos on Flickr.

I became involved with the Drupal project a year and a half ago while working
as the web administrator at KPSU (Portand State University's college radio
station). I have had several patches accepted for core (#34031, #33808,
#36029, #40847, #41703, #47853, #50234, #80079) and a few that are still
pending (#110981, #113385) but the majority of my work has been developing and
maintaining contrib modules. I wrote the Station module which is
used by many college, community and commercial radio stations to display their
programs, play lists and weekly schedule. I re-wrote and now maintain the
Audio module, which is used at KPSU to provide an archive of all shows
broadcast over our web stream. I wrote the Flickr module and recently have
become a maintainer of the Image module.

I am an excellent candidate to undertake this project. My prior work with
Drupal demonstrates my familiarity with both the code base and developer
community, and my ability to complete complex modules.