Drupal 'ports' collection

What is it?

My recent work on Aegir has been very deeply oriented with Drupal package management and dependency checking.

From the perspective I have been working from (unix command line scripts), the currently existing functionality of the update module is simply not useful.
I am in need of an easily mirrorable meta-info repository of all the drupal projects, and all the drupal modules. I need to be able to figure out dependencies
before downloading the packages. This is further troubled by the fact that Drupal cares about the modules / themes, but Drupal.org and the xml feeds care about the drupal.org project nodes.

Please keep in mind, that this has very very little to do with Drupal's front end needs. This will never be the right tool for the job to update a Drupal site from within itself. I personally am not even sure that's all that wise, I liken it to apache trying to apt-get update itself.

Inspired by the FreeBSD ports collection, i started banging around some code.

Back-end

What we will have is a backend component, that generates a directory tree of meta-info files.
I've started writing a script which gets the xml files and releases from drupal.org, and re-formats them
into this filesystem structure.

Here's an example file :

Each of the API versions (i only start from v 5.x onwards), then has it's own xml file containing releases, for which i create a yaml file.

I have chosen YAML because it is a easy to read and understand serialization format. I do not want to depend on Drupal version specific
.info file formats, and XML is out of the question.

I am working on a second pass after this, which downloads the tarballs for each of these releases, and parses out which
modules, themes, install profiles etc we are dealing with, generates checksums for all the files etc. etc.

I already have the vast majority of this code written for the package management in Provision,
so I already have a Drupal release agnostic algorithm for mapping the actual contents of each of the packages, and extract all the relevant meta-information.

Once this is done, we have a very portable meta-data repository, that can be distributed using rsync, or any other
method really. I think it might be useful to have a 'bootstrap' process where a tarball of the current repository is
automatically downloaded, and then rsync'ed from there.

The beauty of this method is, that we only need to have the script running in one place, and it will index Drupal.org for us, and we can
then easily mirror the code to multiple locations. There's even a possibility of the main repository being hosted on updates.drupal.org.

Additionally, at the moment, this will only be targetting Drupal.org as a package source, but it's being designed with the idea that you should be
able to index your own SVN, GIT, etc. repository, and add them as additional sources. This is the beauty of having a consistent intermediate format,
instead of relying on the d.o xml feeds. I'd also like to add acquia carbon as a source.

Client side

For the next step I will need to write a script (probably for drush) that parses all this information from the filesystem, and creates a sqlite
based data cache, so we can easily search and match for dependencies. I will also be porting the Drupal 7 acyclic graph api to Drush, as it is general enough to allow us to extend it to do all this stuff.

Implementation wise, i am also toying around with the idea of turning all the version numbers into a numeric index (kind of how the comment nesting
works in Drupal). This will allow us to determine higher versions through simple integer comparison sql queries , instead of the complex string manipulation and parsing we need to do.

Our dependency handling needs will far outstrip Drupal's own needs in fairly short order, as we will need to handle things such as the
'provides' keyword. IE: a drupal project contains 5 modules, by installing the package it provides: module-module_1, module-module_2.

We will also be breaking ground on versioned dependencies (ie: to install v 6.10 of somecoredrupalmodule, it needs to get v6.10 of Drupal itself). We will also be delving into cross package type dependencies. Currently you can't have a module that depends on a theme and vice versa. Install profiles depending on modules and themes for instance.

Once that is figured out, we should be able to 'apt-get update' and 'apt-get install' any drupal packages to our heart's content.
With the update.php and install.php code being ported to drush, we'll be able to simply call the update commands in the backend when we need them.

Major version upgrades, i'm not certain about. I am loath to upgrade sites in-line, as it's very difficult to recover from that,
and i'm also not sure i want this stuff tied to the provision / aegir codebase. Aegir handles the problem by creating a re-deployable backup
of the site, and migrating the site across to a newer Drupal release.

In some way, our requirements are orders of magnitudes more complex than what apt-get et. al. have to deal with. We need to be able
to run multiple versions of modules across multiple drupal installs across multiple sites in each of those drupal installs, simultaneously,
where as the other tools only need to deal with the currently installed, and currently available packages.

Front end

As this is being developed as part of my work on Aegir, this will need to be tied into the Hosting front end.
This will be trickier than it sounds, especially when we are working with multiple servers (Aegir is fully distributed), and the amount of meta-info we'll be transferring is quite large.

This part needs a lot more thought, to avoid unnecessary code duplication.

I hope these notes are useful.