project mercury

An FYI. I will be investigating creation of the Project Mercury environment on an OpenSolaris 2009.06 AMI, with the goal of publishing that AMI in the next month or two. Starting with joshk's excellent step-by-step on Jaunty, I hope to reproduce all these pieces in OSOL over the next few weeks. Comments? Things to watch out for? Look for a writeup here as I go...

While Mercury has previously demonstrated the raw power of Varnish to radically accelerate the delivery of cached pages with Pressflow, that's only part of the battle for high performance Drupal. Cached pages need to be generated at least once, and logged in users will be bypassing Varnish when making requests. While it's nice to know you're covered for massive traffic spikes, what about baseline load?

Well, we want to answer that question too. Keep in mind that depending on your module/theme stack, your mileage may vary. A lot. Performance tuning a complex Drupal site for logged-in traffic requires sustained diligence at the application level — code profiling, slow query log review, innovative app-specific caches, etc — as well as hardware and system optimization. However, to set some baseline expectations, we used Jacob Singh's greate starter test suite and checked out a Mercury install on a 512MB Slicehost VPS. The results are quite encouraging.

Mercury 0.71-Beta has been released with small bugfixes (see https://bugs.launchpad.net/projectmercury for details).

AMI IDs are:

US 32-bit: ami-bd7c9fd4 - chapter3-storage/PANTHEON-pressflow-mercury-0.71-Beta.manifest.xml
US 64-bit: ami-b17c9fd8 - chapter3-storage/PANTHEON-pressflow-mercury64-0.71-Beta.manifest.xml
EU 32-bit: ami-8f2c07fb - chapter3-storage-europe/PANTHEON-pressflow-mercury-0.71-Beta.manifest.xml
EU 64-bit: ami-73230807 - chapter3-storage-europe/PANTHEON-pressflow-mercury64-0.71-Beta.manifest.xml

With great pride, and after six alpha-level releases, I'm announcing of our seventh iteration on the Project Mercury stack, finally baked enough to call "beta".

At this point, we know that many people are using the Mercury EC2 image in production environments, and we've tuned this release conservatively to prevent it from breaking down under heavy load. We've also verified that the stack will work under a resource-constrained VPS (e.g. one with 1/4th the RAM of a small EC2 image), which gives us more confident that this configuration is stable. We also have a kickass logo:

Just a heads up; as we move towards more and more stable builds of the Mercury stack, we are starting to look at deploying it on other infrastructure besides EC2. This week, we set it up and tuned for a modest (512MB of ram) VPS. These tests were successful. We were able to simulate a mix of non-cached traffic along side the simple ApacheBench battering, and the system held up well, even without gigabytes of ram to support it.

I'm happy to announce the 0.6 Alpha release of the Mercury AMI, now including ApacheSolr as the search backend! This is the last piece of major infrastructure we want to integrate into the stack for scalability purposes. You can now move from a single-server install based on Mercury to a best-practice vertically scaled architecture with separate hardware to run front-end cache, application, back-end cache, search and database!

The quickest way to find it is by searching Amazon EC for "Pantheon" or "Mercury". The manifest path for the latest release (in 32bit and 64 bit flavors) is:

• chapter3-storage/PANTHEON-pressflow-mercury-alpha-6.manifest.xml
• chapter3-storage/PANTHEON-pressflow-mercury64-alpha-6.2.manifest.xml (back!)

If you'd like to "roll your own" we've updated the wiki instructions page with a new set of instructions for getting Solr up and running as part of the process. Feel free to improve that documentation, as it's definitely a community process.

This will likely be one of the last releases before we move the project into the Beta phase, at which point we'll be focusing on fine tuning and stability as well as portabilty onto non EC2 systems moreso than new features. If you have ideas for additional things you'd like to see integrated in the stack, please chime in. We're also going to be documenting real-world "how to" use-cases — e.g. "how do I put my existing site on Mercury" in user-friendly detail — so stay tuned for that.

As always, let us know what you think of the release, what you'd like to see in future iterations, and how your experience is in using the stack. There's plenty more to come.

Just wanted to let everyone know we're hard at work on a Beta release of Project Mercury. Currently we are tuning Varnish further, working to package Apache Solr on board as the search backend, and testing the whole system for bugs. We will likely issue one more alpha release (0.6) before we get to a Beta, but I want to have that launch by the end of the month if possible.

The high level road map:

• Package Solr
• Improve default Varnish config
• Thoroughly document "putting your site on mercury"
• Resolve lingering bugs (e.g. cron.php issues)

Once we hit Beta we'll be pushing for wider adoption, which means pursuing alternate non-Amazon hosting options. Currently we're looking at alternate cloud providers, VPS partnerships, and other packaging options like VM images, etc. So stay tuned, and let us know what else you'd like to see us look at including early in the Beta cycle!

I just wanted to post an announcement that we've finally gotten out a point-update of the Project Mercury AMI. Just in case you were wondering if this project would continue, it will! I've been really excited and encouraged by all the positive feedback so far, so keep your ideas and questions coming. The 0.51-Alpha release includes a number of bugfixes and improvements, most notably it:

• Is based on the latest Pressflow including Drupal Core 6.14 and Simpletest 2.9
• Fixes the self-update process to merge correctly and pull from Pressflow's lovely new VCS home on Launchpad
• Includes the rc1 version of cacherouter
• Fixes postfix and s3 metadata issues so there's now a working MTA out of the box

Most importantly for people considering this stack for production deployments, we're now bundling 64-bit images with every release. The quickest way to find the AMIs is to keyword search for "mercury" in your favorite EC2 console. More information and AMI ids are below the fold. Let me know what you think, and what you'd like to see next!

While working through some issues this weekend and preparing another blog post, I finally got around to doing some comprehensive benchmarking on Project Mercury. In the process, I discovered that the first bottleneck I hit running tests from my desktop was the local (last-mile) internet connection, so I switched to running the tests from another EC2 instance. This means that network is not a factor in my results, giving us a real sense of the raw power behind this stack.

For all these tests, I used the Mercury Alpha4 release on a small ec2 instance, loading a staging copy of Mission Bicycles, which is a good "heavy" example in that it has a lot of modules loaded, including Ubercart and Panels. My goal was to measure throughput and response times under various caching configurations, angling for the best results in terms of pages served per second, and delivery time.

I started by cutting things all the way back to nothing, and then added each layer of the caching infrastructure, running benchmarks at each point. The results are quite eye-opening. Can you say 2000+ requests per second? Read on for the full story.

Today I'm glad to announce the latest release in this line of AMI development. This update solves a number of issues and moves us one step closer to a sable beta release. The current AMI ID is ami-c353b2aa, and you can find this AMI by searching for "chapter3" or "mercury" in your AWS console.

For more background information about this project, see my initial g.d.o post and my blog post announcing the initial release.

Below you will find the notes for this release. Also in this post I will include a development roadmap, as well as some more explicit explanation of the techniques I'm using for making the AMI work out of the box.