Issue tracking and software releases

Here are some notes that I took while comparing these system, based on a couple hours of poking around and reading code. Anyone feel free to jump in here and correct me on any of this stuff, especially if you've actually /used/ either of these modules before. :P

Idea came up on this issue (twice) http://drupal.org/node/265450

This information would be useful for a couple of reasons:

Showing dependencies automatically on project pages - some maintainers are kind enough to list them in the description, but what's in the .info file is the best bet.

Showing related modules - modules like token, views, voting API could show dependent modules, modules with dependencies could show other modules with the same dependencies (cck, jQuery UI).

The following issues are all (at the time I'm writing this) either RTBC or CNR
* Add possibility to retrieve a list of projects from the server (http://drupal.org/node/157514)
* Prevent/limit project_usage abuse (http://drupal.org/node/168009)
* Make usage statistics visible (http://drupal.org/node/165380)
* project_page_overview() should use {project_release_supported_versions} (http://drupal.org/node/235037)
* Add search block for just project nodes (http://drupal.org/node/168819)

Note: Project issue 5.x-2.2 has been released. See http://drupal.org/node/249127.

Here's a list of issues to finish up for the upcoming 5.x-2.2 release of project issue:

• Russian Translation: http://drupal.org/node/221640 -- Fixed 2008-04-13
• Unaccessible project titles and uris visible: http://drupal.org/node/233785 -- Postponed from this release pending further discussion of the implementation
• Advanced search only matches an exact phrase: http://drupal.org/node/235097 -- Fixed 2008-04-13

I like Case Tracker because it is Drupal. It has some weirdities, but otherwise it's a good baseline and I've extended it pretty easily with cck and views and it's really starting to hum. So I thought it was time to share where I'm at with Case Tracker.

I tried Organic Groups integration with Case Tracker in an earlier trial (my current support site is the 4th and final attempt). I have a client node-type and I use the CCK node reference module to link projects to clients. I then have half a module worth of glue code to improve navigation and usability.

Hi everyone!

My name is Markus Schanta, I'm a 21 year old Computer Science student from Vienna, Austria and I would love to participate in the 2008 SoC and do some work on the project* modules.

The project* modules provide project management for Drupal sites. Generally projects are assumed to represent software that has source code, releases, and so on. The project* collection of modules (Project, Project issue tracking, and CVS integration) is the largest set of code running on drupal.org besides Drupal core.

Added to official ideas list at http://drupal.org/node/234735

(draft project outline).

Contributing a theme to Drupal.org currently requires a CVS account. This isn't good for getting new designers and themers involved. So this project would involve creating a UI so that themes can be uploaded via a browser, yet still have a project page and versions. Some developers are likely to want to continue managing their themes directly from CVS of course.

Here's the more detailed plan for views-i-fying the project and project issue tracking modules.

NOTE: As of April 10, 2008, the current timeline for adding views integration is to quickly finish the basic Views 1 functionality for the project issue module in Refactor project issue to use views, commit that code, and then begin the port of project* to Drupal 6. During that port, Views 2 integration will be added and Views 2 will be required for the Drupal 6 version of project*.

dww, hunmonk, and I will be meeting at MIT for the DrupalCon 2008 Boston code sprint around 10:00am today. If any of you are interested in helping out and are in town, feel free to join us-we'd love to have additional help. If you aren't in Boston right now, drop in the #drupal-project channel on irc.freenode.net and talk to us there.

I wanted to write down our current plan (as I understand it) for the day to help us organize assistance better. The items below are those that we want to tackle, roughly in this order.

Yeah, those were the times. Weekly status updates during the Google Summer of Code... well, I'm too lazy to do that when I'm not forced to :-P
Let's see... the version control api term says that the last status update was in September '07. Way too long ago. So what's up going on in the Version Control API realm at the moment? Lots of good stuff. (Read on! It's just the <!--break-->.)

A while back catch created a feature request for the Project issue tracking module to allow free tagging on issues (see http://drupal.org/node/187480). The use case he indicated was to be able to attach certain labels to issues, for example benchmark, needs re-roll, needs doxygen, etc. Other use cases would include tagging issues that need testing in certain ways (eg. MySQL, PGSQL, Javascript, IE, etc.) and tagging issues that are good for certain audiences (eg. newbie, GHOP, DROP, etc.).

Some of you may have seen Kieran Lal's pitch to the Drupal community to help get Derek Wright (dww) and Chad Phillips (hunmonk) to DrupalCon Boston to work on the crucial Project* modules, and many of you chipped in to help Derek and Chad reach their goals. But the project* team can still use your help!

Adam Light (aclight on Drupal.org) is also trying to get to DrupalCon to work on Project*, and trust me, it will be worth ten times whatever you can give. I have only had limited interactions with Adam, who I don't know personally, but even those limited interactions have blown my socks off. Adam was a tireless volunteer for the GHOP program, and his work with these students was truly out of this world (take a look at any of the GHOP issues that Adam helped out with and you'll see what I'm referring to). As Webchick noted on her blog post asking for help for Adam and Jimmy "boombatower" Berry: "Adam was the one primarily carrying the torch during the latter half of the GHOP program, and was critical to ensuring its success."

I happened upon a blog post Adam wrote in which he requested assistance on the DrupalCon Boston feed, and I think that Adam is a bit too modest to post his request here, so that's why I'm writing this. Here's how Adam describes his contributions to Drupal:

How to grant access to repositories at all is an important issue, and it potentially comes with a slight regression compared to the current work flow for managing CVS accounts. One advantage of CVS is that it's easy to administer - in terms of user accounts, that would be a simple "passwd" file that contains all usernames that are allowed to commit. Dead easy to generate, and at least possible to keep in sync even in an automated way. However, more recent version control systems are less nice to handle - also caused by a better eye on security concerns. An overview.

I just spent some time in #git to further investigate how our CVS access control scripts would translate to distributed version control systems, in order to help determine the right direction for our new GHOP-powered Git and Mercurial backends (currently being worked on, more on that to come later). Short answer: keep out of that altogether - it's not DVCS style to restrict project maintainers like that. Read on for a more detailed analysis.

For the past two months, I have been acting as one of the administrators of the Drupal side of the Google Highly Open Participation program (GHOP for short). Briefly, this is a contest that is sponsored by Google in which secondary students (ages 13-18) can claim and complete short one week tasks created by the Drupal community for cash prizes. One of the requirements of the program is that everyone has to use the Google Code task/issue tracker for tracking the "official" progress of the students throughout their tasks. As I have been pretty involved with development of our own issue tracker (the Project issue tracking module used on drupal.org), I thought it would be useful to provide a comparison of the features of these two different systems and make some suggestions of how we can improve the Project issue tracking module to make it even better than it already is.

I'll start by giving an introduction to the main issue tracking features of both the Project issue tracking module and the Google code tracker. I'll also give a description of the administrative user interface from an individual project owner/maintainer's perspective. Next, I'll provide a feature comparison and point out the pros and cons of both systems. Finally, I'll provide some recommendations on specific areas where we can add or improve the Project issue tracking module to make it better than it already is. I want to point out that I am not mentioning any of the features of either tracker that allow it to interface with code, releases, or repositories since we did not use any such features for the GHOP program and thus I would not be able to make a fair comparison.

Following is a worksheet of necessary items to complete in order to get Project issue 5.x-2.0 out the door:

Update: Now released: project_issue 5.x-2.0

I've been evaluating solutions for Project management (for the duration of this post, that includes what i describe as project, issue, ticketing, case tracking, and pretty much anything that falls in that category) solutions with Drupal, over a year actually. I keep being enticed by the features of each individual solution, and new promises that are announced for each module(s) and trying them out and coming to the same conclusion with each of them. And yes, all of them seem to have the same problems that I'm hitting repeatedly.

There are a few limitations and problems with the current interaction of project* (project and project_release) and update_status. One of them is a fairly serious bug, something that was part of the existing design but that was omitted in the current implementation of update_status. Since we have to fix this problem anyway, we might as well Do It Right^TM and fix the other limitations while we're at it. Moreover, time is very short to fix this in D6. Read on for an explanation of the problems and my proposed solution to all of them.

checklist complete -- just need to cut the releases. releases cut. ;)

Part of the Project* roadmap for D6 involves getting another round of official releases out for project and project_issue. We've fixed a bunch of bugs, and it'd be nice to get new releases out at this point, before we start committing changes to views-ify project*. However, there are some lingering issues that either need to be solved, or in most cases, backported, before we're ready for the releases. If anyone can work on any of these, that'd be great.

This was posted to the devel list, and I'm cross-posting here because I think there might be some interest...

Issue followups as comments is almost a reality on drupal.org! This is a major leap in issue queue functionality, including:

1. multiple file attachments per followup
2. tracker integration
3. ability to edit followups (unpub/delete for admins as well)

Following a request on the dev list, an issue and first-version code has been created to return a list of current issues as a single set of data, to be downloaded and used offline.

It includes, in a single JSON object:

• the issues, selected by project, priorities, categories, and states
• the followups/comments
• the attached files

See the issue for more detail and to submit revisions.

Seems I haven't given up on Version Control API & friends by now, which one could say is a good thing. Due to my rather silent nature, there haven't been as many g.d.o posts as during the Summer of Code (namely, none until now). Nevertheless a good share of remaining issues have been resolved, and missing features have been added. Here's a short rundown of what has been achieved since part 2 of my wrap-up, which was written a month ago.

Prompted by some recent confusion regarding the project node UI (e.g. what a project node looks like on d.o), webchick, hunmonk and myself were just discussing the need for a more coherent plan on redesigning the UI.

Some background issues of interest:
* http://drupal.org/node/165380 -- Make usage statistics visible (see especially comment #75
* http://drupal.org/node/96971 -- Make better use of tabs and subtabs on project nodes

A comprehensive list of other issues related to the project node UI:

Now that Drupal 6 is out, getting the project* family of modules ported to D6 is going to become an urgent task. We want to ensure that by the time the final release candidates are out, we're ready to upgrade drupal.org to D6 (which must happen before the official 6.0 release is possible). That's going to take a lot of effort, and hunmonk and I have some very specific plans for how it should all happen. This page will be the place to keep track of what has to get done, in what order, so that anyone who wants to help knows where to put their efforts. It's obviously a work-in-progress, so feel free to help keep it updated, add issue links where appropriate, cross off things that are completed, or add other steps that need to happen.

I'm cross-posting this to a lot of groups since: a) converting project* to use views is a big part of this effort (so we have significantly less code to support in project* and port to newer versions of core), b) since we need lots of volunteers/help, and c) in case anyone is able to help sponsor some of this work to ensure that everything is completed well before the core maintainers would like to ship 6.0. Thanks!

If you already had a go at part 1, you have been waiting half a week for this one. Lack of internet, party intermezzos and my first company outing caused a minor delay, but here you are!

It's over! Google Summer of Code 2007 had its official "pencils down" on Monday, 19:00 UTC - which was 21:00 at my place, perfect for a last sprint... but I digress. According to Google, mentors are supposed to evaluate projects based on the state that they were in at that time. Which means it's time for me to wrap it up and explain what I achieved in those two or three months. In addition to these writings, I set up a test site at http://www.petsovits.at/versioncontrol/ where you can try out most of these things in action.

The wrap-up is becoming too long to conveniently fit into one single blog entry, so I'll split it up into two halves. This one basically covers the hard facts: which modules I wrote during the Summer of Code, what they do, and how they work. (Part 2 is now also available.)

The end of the Google Summer of Code is nearing, and stuff is coming together in my version control modules. It seems to me like I'll not be achieving everything that I would have liked, but most of the important stuff is going to be ready next Monday. And now for a short report of the last one-and-a-half weeks.

I've been spending most of the last week working on the project_usage.module. I got a copy of the usage data that's been reported to Drupal.org and spent some time fixing bugs (#167074) and making some small improvements (#167079, #167048). My big goal has been to get some code together to make the usage data visible (#165380). I've attached a few screen shots to show how it works.

As of about 2pm PDT Drupal.org has been tracking the module usage. #128827 has been committed to the project module and installed on d.o. There's a follow up issue to make this usage data visible to users (#165380). I've added support to the metrics module to factor that into the score.
I'm not sure if the metrics module will be installed on d.o until the current database troubles resolved.

Instead of keeping potentially stale copies of a TODO list in the README.txt files in CVS, I'm moving these list to a wiki page here so there's just a single canonical copy. This is general stuff about the modules. Other to-do lists of interest include:

• Future work for the release system
• Suggestions for improving the d.o issue queue (especially for core)

At http://drupal.org/node/150278 we've started serious work on an official 5.x-1.0 release of project* (project, project_issue, and cvslog).

Here's my current list of issues I'd like to see us resolve for the 5.x-1.0 releases, if possible. Mostly, these are either a) critical bugs, or b) patches I've already written for relatively easy-to-solve problems. Feel free to add comments or update this list. And, of course, please work on any of the issues below -- reviewing, testing, rolling patches, etc. Thanks! -Derek

This week, my Version Control SoC Project saw further completions of the API functions' implementations (the API itself is completely implemented now, if I'm not mistaken) and the creation of extendable admin screens, mainly for repositories for now. There's now a list of repositories separately for each version control system, and each backend can easily customize creation, editing and display of repository information by means of hook_form_alter() and a small number of more specific hooks. Repository management is now at the same level of functionality as the one in cvs.module, only that it's generic at the basics and other backends can without much effort provide their own custom stuff, just like CVS provides CVS modules and different methods of commit information retrieval. Sure, you want screenshots:

Er, right, this update post has taken longer than it should have. I've been active on the code though, and this time I even have a screenshot for you. Without further ado, I present Commit Log:

I'd totally fallen off of writing these updates so I'll start by discussing what I've been up to.

I've got a framework in place for computing, ranking and displaying the metrics. You can view the code that's in my sandbox. I've been able to test it out on a sanitized version of the Drupal.org database.

Well, I promised you an update on the API module, so that's what you'll get. Just a short notice beforehand: the CVS backend's database structure is now online and available for everyone's reviewing pleasure. Module functions will be added in a week or two, but before that, I'll get the xcvs-* scripts to fill in the entries for the new database structure. I've also made a (slightly different, but largely similar) SVN version of the .install file, you can find that one attached to this post. (As a Subversion backend module is only an optional deliverable, I'd rather not start a versioncontrol_svn project and its directory in CVS just yet.) So it seems to me that deliverable number three, a finished database schema for the CVS backend, can be marked as done.

Get to the point, dude!

Ok, ok. My remaining problem with the Version Control API was that it lacked a way to consistently deal with tags and branches, and to uniquely identify files and directories even if they are not versioned themselves (which is the case for directories in CVS and other version control systems). So I thought a bit more about how to deal with this, and the outcome is a new building block object for items, in addition to the previously existing ones for repositories and commits.

Actually, I wanted to write a piece on how the Version Control API is maturing, but I've got to leave here in a few minutes and won't have internet available until tomorrow, so this must wait for now. Instead, a short coverage of the Version Control / Project Node Integration module (versioncontrol_project.module).

[disclaimer] I'm writing this update while drupal.org (including g.d.o) is attacked by database-eating gremlins, so I can't currently check back on previous entries. Apologies in advance if I mention something that has already been said before. Yes, I could use the Google cache, but that's kinda tedious. [end of disclaimer]

So, I've been pretty absent for quite some time. After getting my university stuff finished, I found a bit less time for my Summer of Code project than I had expected, moving out from the dorms, packing bags, wrapping up undone stuff, whatever. Last week (June 29th to July 7th) I've been attending aKademy, the KDE conference, as previously announced. Great event, great people, great location (this year it took place in Glasgow, Scotland). You can read lots of coverage on KDE Dot News - start here (you can find me in the group photo) and go left in the navigation at the buttom, the marketing team has been active as never before. I wonder in which ways DrupalCon will be different from aKademy, as in my view the projects and their attitudes already differ fundamentally. Ok, so that's that.

Not having found the time before, I reallocated a few hours from aKademy's hacking sessions to my Summer of Code project. Since the last real update, the RCS API module has changed its name to Version Control API (versioncontrol.module), and the former project_rcs.module is now Version Control / Project Node Integration (versioncontrol_project.module). Both now contain a good amount of API functions for accessing commit data. Even if the implementation is still missing out (which, according to the project proposal, has been planned this way), they are the core of what other modules will use from the API module. (Read more...)

So, Mercurial, or "hg" in short. One of the two revision control systems originally written for being the Linux kernel's BitKeeper replacement - in the end, git was chosen for that job, but Mercurial lived on nevertheless. The requirement of being suitable for kernel development implied two features: performance (although git is still a tiny bit faster) and distributed development methodologies. Both are also present in git, and in fact, the two systems are very similar in scope, paradigms and usage patterns.

While git is written in pure C, Mercurial mostly consists of Python code, with only small, critical code paths implemented in C. This has benefits for Mercurial's portability, as can even be observed with other SoC students. git is pretty much focused on just Linux, whereas Mercurial can also be installed more easily on Windows and other Unixes. Not that straightforward graphical tools are yet as mature as the ones that exist for CVS or SVN.

CVS, or Concurrent Versions System, was the de-facto standard open source version control system for a long time. It came up in the late 80s and grew popular enough to manage the source repositories of virtually all open source projects until only a few years ago (when Subversion started to take over). It's still widely used, including on drupal.org, but hardly deployed on newly created repositories anymore. Development is strictly fixed to a centralized client-server methology, and while distributed version control systems (or Subversion with svk) strive to make a checkout independent from the server, with CVS you're completely dependent on a fast and reliable connection to upstream.

In summary, there's little surprises in this CVS coverage, especially for the resident drupal.org admins. But necessary nevertheless, if only for later reference. Obviously, no new requirements for the API module were found, as the CVS setup on drupal.org together with the Project module is the status quo at the moment.

My project definition says that I still got to document the specifics of SVN and CVS (git has been taken care of). Coming from the KDE camp, I'm already acquainted to Subversion to a certain degree, as the whole huge KDE repository switched from CVS in May 2005. (No one, including the SVN admin, did regret the switch).

Subversion (SVN) came up as a replacement for CVS - motto: "CVS done right" - and incorporates its centralized development methology and its straight-forward workflow, while improving on its weak parts. Compared to CVS, Subversion features good stuff like atomic commits, renaming, serverless diffs, symlink support and version-controlled directories, to name the most popular ones. It fulfills its role as a drop-in replacement brilliantly and has kinda deprecated CVS, at least for newly installed repositories. (That, of course, doesn't stop Linus Torvalds from claiming that centralized development methologies are outright wrong.)

There was a quote of someone saying that inexperienced programmers worry about code, while advanced programmers worry about data structures. Don't remember who said that, but personally I find this an excellent analysis.

Let's try to formulate the important questions that need to be answered in order to get the Revision Control API right:

1. What kinds of functionality do users need from the API?
2. What kinds of functionality do different revision control systems provide?
3. How does the data that is delivered to the caller look like?
4. Which functions are required to deliver the data to the caller?

There's a great introduction on git for SVN people like me, which made it twice as easy for me to look into how this thing works. Git only recently released their 1.5 version which is the first one that's supposed to be usable to the masses. (It might not yet be available pre-packaged for your Linux distribution, or available at all if you're running Windows, which could be a small hurdle at the beginning.) After reading the introductory couse and trying it out by myself, I must say I'm hooked.

For those who didn't know, git is the distributed RCS that was created by Linus and the other kernel folks because they needed to get rid of BitKeeper, and as the Linux kernel is a very demanding project both in code size and in patch management, git is quite capable indeed from an efficiency point of view. Currently in use by the Linux kernel itself, X.org, Wine, and One Laptop Per Child, to name a few popular projects.

As promised in my SoC application, here's a short rundown of features that are important to this abstraction layer.

So that's that. After having used the past two months for improving filefield and imagefield, it's about time that I get started with my RCS abstraction project. aclight has been as kind as to publish his work and findings in time with the Summer of Code, so that's a good starting point already. His work on the xcvs scripts (...bootstrapping Drupal, yay) benefits my project a lot, and the issues with password storage are good to know of.

In order to come up with a set of good questions that I'll need to research, I had a first shot on the database schema and the overall idea of how this thing should work. I thought it would be a good idea to see what we want to achieve and what we can rely on, before figuring out how to abstract all the stuff that's supposed to be different among the various revision control systems. Bullet points galore.

Hi Everyone

For the past few months, I've been working on a fork of the cvslog and project* modules to be able to use them with subversion instead of cvs. After consulting with dww, maintainer of these modules, I learned that someone (jpetso) had applied, and by now has received, a Google SoC position for creating an RCS abstraction layer. Eventually, such a layer would presumably make it a little easier to use a version control system other than CVS.

The project* collection of modules (Project, Project issue tracking, and CVS integration) is the largest set of code running on drupal.org besides Drupal core. They are the key tools that power all Drupal development, including the Drupal issue queue and Drupal release system. Because of the huge user base, high visibility, complex requirements and feature requests, and size and scope of the existing code base (and issue queues) there is a ton of work to be done. This wiki page is how I (dww, the primary maintainer of project*) will try to communicate the best ways for other people to get involved in helping. One of the ways to help is to improve this list, so please add your own ideas here.

this is an idea i've been thinking about. posting it here so other folks can add to it, provide feedback, etc.

Summary

we could really use an install profile that setup a test site similarly to how drupal.org is configured (especially the project* modules and CVS integration). obviously, it wouldn't have the bluebeach theme, but otherwise, it would look fairly similar to d.o in terms of structure, content, permissions, etc.

this would greatly help people who wanted to reproduce bugs and test patches for the project* family of modules, which is something we desperately need more of.

I just uploaded my application to Google's Summer of Code web app, and I can say I'm quite proud of its current state. The application has seen a couple of iterations based on valuable feedback by Adam Light and Derek Wright, both of which are working on RCS stuff for the Project module. Being fed into Google, this means that the Drupal mentors can now place private comments and supporting votes there.

Given that there are a good number of comments on some of the other proposals in this group, I'm still wondering if there is something I can do to spread the word and get more input from the greater public. After all, work on the Project module severely affects the great majority of drupal.org users, and getting rid of CVS as a hard dependency should really be discussed in the wider community. So do communicate your opinions about my application, its benefits for Drupal, its chances to get elected, its shortcomings, challenges, etc. ...I'm listening.

Myself, Webchick and Killes have come up with two proposals for this year's SoC which involve the drupal.org infrastructure and the project module. Both of which could solve some of the worst drupal.org usability problems, and yet are interesting, flexible projects where someone with a lot of creativity and talent could have a huge, highly visible (and valuable) impact on the entire Drupal community.