Last updated by DavidMorris on Sun, 2015-10-11 02:54
This wiki page is dedicated to the GSoC 2007 project 'Drupal Automated Staging Toolkit'; hereafter I dub thee DAST. Research and work on DAST touches on a lot of different areas, so my mentors (Grugnog and Zlender) and I thought that I should share all the materials I've accumulated here.
A formatted copy of my original SoC proposal is here: http://docs.google.com/Doc?id=dcgv33zb_9hr7tcg
Introduction
Right now developing for Drupal requires a lot of manual tasks. If you're a developer or tester looking to run or test code on a particular Drupal site configuration i.e. specific versions of core or module or patches, specific server configurations, and a specific server location where you want to host the site - creating this site requires a lot of manual steps. Drupal developers have several scripts to automate part of these manual steps. DAST's primary objective is to standardize automating the Drupal build process, much like Ant does for the Java build process, or make for C/C++ projects.
DAST is a build tool. It's like Ant for Drupal. Actually it's a build process which will use an automated build framework like Ant to enable creation of a fully functional Drupal site, given the site's description and configuration specified in an XML build file. DAST is for developers and testers who need the ability to quickly build a Drupal site with specific versions of code and enabled modules, a specific app/database/web server configuration, staged to a specific server location. Right now, building a Drupal site is like building a C++ project, source file by source file, tool-by-tool from the command line. With DAST, developers can build and stage a site to a target physical or virtual server the same way they use make or Ant to build C++ or Java software.
Build Frameworks
There are three build frameworks I've done initial reading on: Ant[1], Ruby Rake[2] and Phing[3]. Ruby Rake is make done Ruby style, and is pretty interesting in its own right (much like Ruby itself) Martin Fowler wrote a great article on Rake[4] comparing it to other build tools and expounding on DSL - Domain Specific Languages[5], under which the whole family of build tool languages fall.
Ant
Ant is standard in the Java community. Ant is very powerful and extensible but it was designed for Java applications and requires full-blown Java code and an available Java runtime for defining and executing tasks and steps (Fowler describes Ant as an 'external' DSL[6] because the language used to code the build process in Ant's XML build file is separate from and is translated to the language which actually runs during the build process.
Rake
Rake is what Fowler calls an 'internal' DSL[7]. The language used to describe the build inputs, tasks, targets, outputs, dependencies,...are all defined in pure Ruby code. Rake is certainly capable of hosting DAST, but in order to use it you need to know Ruby and have Ruby available on the machine you're staging the site from.
Phing
Phing (http://phing.info) is the automated build tool I want to use from DAST. From the Phing website home page:
PHing Is Not GNU make; it's a project build system based on Apache Ant. You can do anything with it that you could do with a traditional build system like GNU make, and its use of simple XML build files and extensible PHP "task" classes make it an easy-to-use and highly flexible build framework. Features include file transformations (e.g. token replacement, XSLT transformation, Smarty template transformations), file system operations, interactive build support, SQL execution, CVS operations, tools for creating PEAR packages, and much more.
And from the Phing features page:
If you find yourself writing custom scripts to handle the packaging, deploying, or testing of your applications, then we suggest looking at the Phing framework. Phing can comes with numerous out-of-the-box operation modules (tasks), and an easy-to-use OO model for adding your own custom tasks.
Phing provides the following features:
* Simple XML buildfiles
* Rich set of provided tasks
* Easily extendable via PHP classes
* Platform-independent: works on UNIX, Windows, MacOSX
* No required external dependencies
* Built & optimized for ZendEngine2/PHP5
So Phing sounds like the ideal framework to host DAST.
Generating virtual images
A lot of interest has been expressed in building virtual images for Amazon's EC2 service. Here also I think DAST will shine because DAST doesn't distinguish between physical and virtual servers, and also, you can specify any additional build steps you want as the inputs go through the build pipeline. If you want to stage the Drupal site to a virtual machine image, you can specify the tasks needed to bootstrap the VM as the first set of tasks to run in the DAST pipeline, and then after successful completion of all stages in the pipeline, specify tasks for additional configuring and deploying the virtual image.
Scripting Drupal
Some command-line scripting of Drupal will be be necessary during the build process. There are two projects currently providing a command-line scripting interface for Drupal using PHP-CLI: Sympal Scripts[8] and drush[9]. There's also a third Drupal scripting project - Dript[10], a Lisp-based internal scripting language intended to be an alternative to PHP code inside Drupal modules.
The Sympal Scripts and drush projects have similar objectives but are implemented differently. drush has its own module which plugs into Drupal and exposes an API that itself and other Drupal modules can use to implement drush actions for drush services. These actions can then be called externally on the command line by the drush command interpreter. After reading some more on how drush works I really like the design because each modules knows the best way to script itself. drush can be used to provide true Drupal automation from the command-line.
Sympal Scripts seem to be more oriented around administrative tasks related to setting up Drupal. The author also also talks about a Sympal distro[11] which seems to be a unique Drupal distro capable of being totally controlled through scripting
Since DAST is all about coding Drupal installation and configuration tasks, it can both use and contribute code to the these scripting projects.
Although I originally envisioned DAST as just a staging toolkit - clearly the build pipeline can be extended (not in the scope of this SoC project but definitely afterwards) for more tasks. A post-build task, for example, can run unit-tests on the site to ensure that every major area of functionality is working. There can also be tasks for provisioning the site with a set of named users, or groups, or even preliminary content. I know that this is what installation profiles are for and I don't want to duplicate functionality which already exists. But there should be a distinction between tasks external to the Drupal site (performed by DAST,) which run before the site is considered full operational, and tasks internal to the fully operational site (which can be handled by profiles.) Also the reason I like Drupal scripting automation is that they provide a reusable, human-readable repository for Drupal actions which can be run any-time.
Two questions (from Ben of Agaric Design Collective) –
Great project! First, do you know anything about http://www.workhabit.org/projects/autopilot and if there's a possibility for overlapping or complimentary functionality?
Jonathan of workhabit and I have exchanged emails about possibly collaborating on Autopilot - the proposed build management functionality has a lot in common with the goals of my project. The Autopilot team has been pretty close-lipped about their product so far, so when the first version of AP is released we'll know better how to move the collaboration forward. I really like what the workhabit guys and gals are doing around the area of change management in Drupal - trying to formalize a set of patterns processes and tools for managing a Drupal site - which will raise its profile greatly in corporate IT shops. So hopefully as both our projects matures we'll be able to work together to achieve some of those things.
Second, I think that DAST's relationship with install profiles should be in physically installing the modules which can be enabled and configured with install profiles. Perhaps an install profile could be packaged inside DAST, or an install profile could call for DAST?
That's the number one question I get asked - what's the relationship between my project and installation profiles and where to draw the line between the functionality of profiles and the functionality of a build script?
Well, installation profiles are designed to run in a working Drupal site after it has been deployed. DAST's purpose is to deploy the Drupal site. So once your site has been bootstrapped, everything related to configuration and customization should be handled by installation profiles - that's their purpose. DAST build files handle the setup steps of grabbing the core source and modules, and configuring the database and web server. So, let's say you created a distro called Milkdrop you wanted to distribute - you could bundle a DAST build script with your distro which the user could run to download or check for updates of core and modules required by Milkdrop and then setup the server - which would make the whole process of creating the Drupal site much more user-friendly and less error-prone. After the site is built the install profile will run to continue the installation inside Drupal.
However, I intend to make it possible for DAST scripts to be able to manipulate a Drupal site externally, for the purpose of automatically running unit tests and collecting the results for example. I'd like to use drush for manipulating Drupal from the command-line.
Oh, and can Wiki pages be tagged to put this in the DAST tag?
Dunno, doesn't seem like it
[1] http://ant.apache.org
[2] http://rake.rubyforge.org
[3] http://phing.info
[4] http://www.martinfowler.com/articles/rake.html
[5] http://martinfowler.com/bliki/DomainSpecificLanguage.html
[6] http://martinfowler.com/articles/languageWorkbench.html#ExternalDsl
[7] http://martinfowler.com/articles/languageWorkbench.html#InternalDsl
[8] http://drupal.org/project/sympal_scripts
[9] http://drupal.org/project/drush
[10] http://drupal.org/project/dript
[11] http://webschuur.com/publications/blogs/2006-02-13-early_announcement_of...