Project Mercury: PressFlow Drupal+Varnish AMI Alpha4 Release

Today I'm glad to announce the latest release in this line of AMI development. This update solves a number of issues and moves us one step closer to a sable beta release. The current AMI ID is ami-c353b2aa, and you can find this AMI by searching for "chapter3" or "mercury" in your AWS console.

For more background information about this project, see my initial g.d.o post and my blog post announcing the initial release.

Below you will find the notes for this release. Also in this post I will include a development roadmap, as well as some more explicit explanation of the techniques I'm using for making the AMI work out of the box.

Alpha4 Release Notes:

This release improves stability and adds some critical missing components from previous images.

• Switched from experimental memecached back to APC for simplicity/stability
• Increased the APC SHM size to 256M to accommodate Drupal caches as well as opcode
• Installed mod_rpaf for apache2
• Configured a launchpad ppa for BZR so it is up to date
• Installed postfix mta and included an internal metadata lookup to set hostname, etc.

Future Release Roadmap:

Future releases will be focused on increasing stability, improving configuration, and delivering an image that can be used in small to mid-scale production environments, and serve as a building-block for more advanced architecture.

If you have any interest in helping with the below, or have other suggestions, feel free to drop me a comment or reach out to josh@chapterthree.com.

Goals:

• Improve boot script process to accept user-data as part of launch command, enabling the auto-attachment of stable EBS volumes for storage. This is absolutely necessary for data persistence in production use.
• Develop a 64-bit version for those who want to run this on more powerful instances.
• Test with a more robust/real-world stack of modules, especially around functionality that could be glitchy with Varnish (e.g. AJAX, etc).
• Investigate server-initiated Varnish cache invalidation. This is possible using the control channel, and would be super-hot to get working.
• Figure out some ways to package pre-configuration without hard-coding user #1. Is there anything less labor intensive than an install profile?

Anatomy Of This Release

Previous posts have given a general list of features used or explained their advantageous features, but I wanted to give a sysadmin-level view of how the stack actually functions on this server so that people can start collaborating more meaningfully on the development of this project.

Varnish

Varnish is one of the main keys to the high-performance value of this stack. When you request a page from Mercury, if you are not logged in and Drupal has previously served the page before, the resulting page is delivered by Varnish, not Apache or Drupal. This is orders of magnitude faster and more scalable.

Varnish is started at boot time, and is controlled as a service in the typical Debian/Ubuntu way. You can start/stop or restart it with /etc/init.d/varnish <command>

Configuration for Varnish is in to places:

• /etc/default/varnish contains the configuration which is used when Varnish is launched on boot, or as a result of a service (re)start. Among other things, this tells the system to look for...
• /etc/varnish/default.vcl contains the basic Varnish Control Language declarations that work with Pressflow Drupal. Basically, if there's no cookies and no post or get, try and serve from cache. Otherwise query Drupal.

Beyond testing for edge-cases with Varnish, one thing I'm very interested in exploring is the Control Channel, which allows for more active manipulation of the service from the back-end. In particular, it's possible to create an active-cache invalidation system, which would allow us to set a very high TTL within Drupal, and actively invalidate urls/pages when they are updated.

The Boot Script

By default Varnish will set up a file-system storage space in /var, but that's not a great place on an EC2 image because /var is on the root file-system partition, which is only 10GB in size, so one of the things we do at boot time is move this over into /mnt, which is the main storage area for an EC2 image.

We need to do the same with mysql's files as they could also take up a lot of space (though really these should really be on an EBS).

In order to move these things around, and to be sure Pressflow and our Ubuntu distro are up to date, I've included a simple boot script that runs at startup. This is in /etc/mercury/init.sh and are invoked via /etc/rc.local. The boot script will run some updates, move some files, dump all output to /etc/mercury/bootlog and then write the current date/time to /etc/mercury/incep. The first thing the script does is look for this file, which is how we can be sure it will only run on the initial spin-up, and not on every reboot.

Development of this script is an important area for making Mercury better. We should be looking to include user data parameters from the ec2 launch command to detect and mounting an EBS volume for mysql (and sites/all/files) rather than just using /mnt. There are many other improvements that could be made here as well.

Other System Config of Note

There's still quite a lot of tuning that can be done to the more familiar areas of the system, and I am all-ears for suggestions on how best to set up Apache, MySQL, PHP, APC or any other component to run most effectively in the cloud. Below are the interesting bits that seem worth calling out to me.

• Apache on port 8080: since Varnish is the gatekeeper for all web requests, that means we need Apache to run on an alternate port which Varnish can query behind the scenes when it actually needs a page. This is configured in /etc/apache2/ports.conf and picked up again in /etc/apache2/sites-available/default which is the virtualhost that serves requests.
• APC: this is installed via standard pecl install apc, and configured in /etc/php5/conf.d/apc.ini. I have given it a large SHM memory footprint because we are using it as the main cache backend for Drupal. While pages won't be cached there (they're in Varnish), everything else will be, and we don't want to run out of room.
• Pressflow: the main document root for the site is in /var/www, and Pressflow is checked out there (and freshened by the boot script) from Four Kitchen's BZR repository.
• Cacherouter: finally, we're utilizing the cacherouter module as our Drupal caching system. This is installed in /var/www/pressflow/sites/all/modules/cacherouter and a very basic/vanilly configuration file lives in ...sites/default/settings.php. This is the mechanism by which we utilize the local APC memory cache for Drupal's object caching system.

In conclusion

This process is getting more and more like second nature to me. I'd like to figure out how to engage more people in development and innovation, and to build a shared repository of tools and practices for this kind of development.

Big ideas going forward would be configuring other kinds of useful Drupal AMIs (e.g. performance benchmarking, automated testing, etc) as well as mappoing out ways to build high-performance/redundant Drupal Clusters in the cloud.