Aegir 0.2 and Drush 2.x ... sitting in a tree

As our 0.1 release is nearing, I have been making some steps toward our 0.2 release. The onus for this progress has been some really exciting changes happening upstream in the Drush project.

Our 0.1 release was originally only going to manage a Drupal platform, with no support for hosting sites on Drupal 6. As the project wore on, we realized we needed to become invested in Drupal 6 very quickly, so we ported only the back end to Drupal 6, and have been maintaining the releases in parallel.

Living in sin with Drush 1.x

When we implemented the functionality (as of ALPHA 2 or so), we started running into some serious roadblocks with Drush:

Where is drush.php?
Firstly, to be able to run Drush at all it needed to be available to the site, and unless we required the platform location to be specified for every platform, we had no idea of finding out WHERE it had been installed. We ended up working past this by making the setup command create a symlink in the root of the platform. This was the simplest solution we could find and has served us pretty well.

Many things need to be present for it to work at all, even when we use none of that functionality.
We found that to even use the Drush command, the site had to be both installed and have the right module enabled. Because the only way to install a Drupal site is through a web based install wizard, we were left with no other option but to create an additional install profile,
the HostSlave project. This is the mirror of the Hostmaster profile, but removes all of the front end modules. The requirements for adding a new platform or release of Drupal also now involved creating a new virtual host file (by hand), creating a new database (by hand) and then installing Drupal, running the setup command and adding the node to the hosting system. Because of the inelegance of this solution, and one other crucial missing piece (migrating sites between platforms, which is the basis of our upgrade process), the multiple platform support in Aegir 0.1 is only marked as experimental.

We were duplicating an AWFUL lot of code , which would make for very difficult upgrade of the hosting system in the future.
Each platform needed it's own copy of provision and drush. We had to create an additional branch of provision to manage this, and spent some time refactoring the code to make sure that all the code that was version specific was stored
in separate files. We are now at a point where all patches to Provision that do not specifically modify the version specific files (which were split off into non drush scripts) will apply cleanly. But we still need separate checkouts on each platform on code that is 95% the same on all platforms.

Drush needing to bootstrap fully, with it's own database.
This meant that it was impossible at the time to write an install command for Drupal capable of loading up in the namespace context of the site, yet still have access to the provisioning backend modules (which should not be available to the sites actually being hosted, ever). Prior to the second alpha (i think), we had partial views integration, but when creating install profile support, we started running into namespace collisions when trying to install a site on the master platform with the views module included. Specifically, if the views module was in profiles/hostmaster/modules/views, and the site tried to install profiles/bryghtbasic/modules/views, they would cause namespace errors.

We were also using the db_set_active and real time manipulation of the global $db_url, but Drupal's multi-db-connection support was very very weak, and all sorts of errors could result from this. I even managed to install drupal into my 'mysql' database at one point in development. The fact that we had to pretend to be multiple Drupal sites in a single session was a hack, and yet we needed to be able to use Drush to do things before the site could be initialized.

To bypass all of this kludging , we have moved all code that actually work on the site, into individual command line php files, which do not use Drush. This solved both of the issues we found above, and allowed us to spend time perfecting our command invocation, to such a point that we are now using bidirectional unix pipes for communication and can actually pass data structures into the scripts we call, and have completed a lot of the legwork for distributed (multi-server) environments. Because of this refactoring, upgrading provision to the latest Drupal release means we are only modifying a handful of files, and yet we still need to maintain full trees of practically identical code. (see above).

Drush was using the system table to figure out which modules you needed to load

In simple situations, this is not much of a problem, but because of the dichotomy of aegir development (front-end and back-end are separate projects), and the addition of 'Features' that can enable modules that provide certain features, it became impossible to ensure
that all platforms have the same functionality available. To that end I wrote code that manipulates the module_listing to automatically load
modules that are specified by --load-mymodule-name on the command line. We're just lucky that this code is the same in D5 and D6, because it smacks of the messing with Drupal internals that we had to work through for the database and namespace collissions.

Enter Drush 2.x

The latest changes in Drush's 2.x branch do the following :

• Drush does not need to be stored in the path of the Drupal site. You can place your copy in /usr/local/bin/drush if you wish.
• Drush does not need a valid database with a Drupal installation to be able to bootstrap. If you can bootstrap, you will have additional command available over the base commands.
• Drush is Drupal version agnostic. You do not have to have the right version of Drush in each instance of Drupal (platform in aegir-speak).
• Drush core is only the version agnostic parts, the parts that rely on specific parts of the Drupal api have been moved into the Drush extras project.
• Drush is now able to download the Drupal specific parts that are required, and place them where you specify.
• Drush can also now download and initiate a drupal instance by itself.
• Drush scans all of the directories in it's include path (which include the platform and site specific paths when available), to find files ending with '.drush.inc'.

Before these changes landed, we were in the untenable position of possibly replacing Drush, but these changes are literally the answers to all our problems.

What does this all mean for Aegir though?
We can drastically REDUCE the amount of code we maintain too, by managing all of provision core (the parts that are the same for each version, essentially EVERYTHING apart form provision_platform),
in a single branch in it's own project, so we will not be applying each patch multiple times, cutting down our commits
and testing down to a third of what we currently have.

Each server we manage will in the future only have a single copy of Drush core and a single copy of provision core (probably maintained in ~/.drush). When you add a new platform node in the front end, the provision backend for
that server will be able to check the directory to see if there is a Drupal installation, if there is, make sure that the
correct Drupal release' provision_drupal is available (downloading it from d.o if necessary). Once that's out of the way,
it will verify the platform, and import any existing sites there may be.

If there is no platform installed there, it will also now be able to check out that Drupal release to that path (if it can), and initialize it to allow you to create and manage sites on it as soon as it is verified. In the future, upgrading to a new release could be as simple as creating a platform node (you could specify to schedule migration of sites from an existing platform once the new one is verified).

A number of changes that occurred during the time between BETA2 and RC1 releases we specifically related to either Drush at present (cleaner workarounds as discussed earlier) and changes being made with Drush 2.x in mind.

While I was experimenting with the initial patch, i spent 3 or 4 hours hacking provision and hosting locally to get it playing nicely with the new sensibilities. I was able to get a running Aegir for Drupal 6 in that period of time. Satisfied that this was moving in the right direction, I started slowly reworking things that could still be made in our 0.1 release and still be in our best interest (I clarified API's, removed Drupal API calls, and just generally made the code move in the right direction.
Another amazing bonus feature is that it simplifies a lot of the work that goes into 0.3 as well, because of our script invocation layer, we are very close to be able to manage multiple servers.

How is Drush going to benefit?

Provision as it exists is essentially a set of loosely coupled components. You have the code specifically related to a Drupal version (provision_drupal and friends), you have the code that is Drupal agnostic (stuff like provisioning databases and virtual host files, which have no interaction with the Drupal API) and then there is the Provision API.

It is the Provision API that Drush most stands to benefit from Aegir, because it essentially acts as a superset of the Drush API. Much of this functionality is incredibly general in nature and are prime candidates for being moved upstream.

It is my hope to integrate as much of the API into Drush, which will allow us to focus on problems that are more specifically
related to provisioning. It will also solve another burning problem that Aegir has, namely that we can not rely on existing Drush
commands because they have no generic interface to communicate with, which will result in us implementing a lot of additional glue code to communicate with other Drush commands.

The are some things that Provision has also worked around, because Drush was not capable of at the time. Specifically the install and update commands. I would prefer to have these be part of drush_extras, instead of maintaining our own copy of it.

Some important part of the Provision API are

Remote script invocation
All provision scripts expect output in a certain format (a serialized array), which allows you to write scripts which call other scripts (ie: if you run update on the platform, it can run the update command for each of the sites, without any weird hacks).

Some of the basic features :

• Logging, instead of just printing to the screen all logging is passed in a way that can be collected and passed to calling scripts. The logging also integrates with drupal_set_message when available and merges all the output.
• Error handling, Provision uses an exit code bitmask so the calling script has more information about what went wrong during the remote call. This is useful in almost any situation you want to use drush in a script.
• Unix pipes. Because of how Provision does this, it allows you to pass sensitive information via the stdin invoked script.
• Call scripts on remote servers, calling a script on a local server is exactly the same as communicating over an ssh pipe, which means you have a secure scripting mechanism for distributed tasks.

Powerful AOP API
Provision, especially in HEAD currently, has a very flexible mechanism for process flow, which integrates directly with
the logging and the error handling. One of the Drush developers remarked that is essentially Forms API for Drush (without all the arrays of course =) ).

The interface allows you to :

• Generalized context, which allows you to communicate between separate parts of the system, without using globals or variable_get.
• Re-use existing commands, without having to call external scripts. This is useful if you want to do things like back up a site automatically when you do a site upgrade. This can be accomplished with one function call, through a standard interface instead of figuring out how each command's internals work.
• Extend existing commands, by simply creating a hook implementation in your own drush module. You can add code before, during or after any command is invoked.
• Rollback changes that were made. When an error occurs, it goes through all the functions it executed in reverse, and checks if they have a $func_rollback variant.
• Play nicely with both internally invoked and remote scripts. All information is rolled back into the calling scripts context, so if you run an external script and an error occurs, the script itself will rollback to it's starting position and then your script will roll back from where it called the script.

You can find out more information at the drush_invoke issue on drupal.org

There's definitely more to come on this subject, and others relating to the 0.2 release (I haven't even touched the front end yet, for instance).