Amazon Web Services (S3, EC2)

This group should probably have more organizers. See documentation on this recommendation.
joshk's picture

PANTHEON/Mercury BoF Friday at DrupalCon Paris

2009-09-04 14:50 Europe/Paris
Event type: 
User group meeting

Just a quick note: we're going to hold a BoF for people interested in Project Mercury (and other packages from PANTHEON) in Paris! It will be after lunch on Friday at 2:50.

Looking forward to talking to those of you who are here at DrupalCon.

Read more
joshk's picture

Project Mercury Benchmarks: 2000+ Requests Per Second!

While working through some issues this weekend and preparing another blog post, I finally got around to doing some comprehensive benchmarking on Project Mercury. In the process, I discovered that the first bottleneck I hit running tests from my desktop was the local (last-mile) internet connection, so I switched to running the tests from another EC2 instance. This means that network is not a factor in my results, giving us a real sense of the raw power behind this stack.

For all these tests, I used the Mercury Alpha4 release on a small ec2 instance, loading a staging copy of Mission Bicycles, which is a good "heavy" example in that it has a lot of modules loaded, including Ubercart and Panels. My goal was to measure throughput and response times under various caching configurations, angling for the best results in terms of pages served per second, and delivery time.

I started by cutting things all the way back to nothing, and then added each layer of the caching infrastructure, running benchmarks at each point. The results are quite eye-opening. Can you say 2000+ requests per second? Read on for the full story.

Read more

Step-by-step: Setting up Project Mercury (Varnish, Apache, APC, Memcached and Solr)

Refer to the new wiki for better docs. This page is here for posterity.

Here are step-by-step instructions for building Project Mercury on a fresh server using the configuration manager BCFG2. Project Mercury has been developed on AWS ec2 but should work on Rackspace and many other VPSs. We have public AMIs available of Ubuntu Jaunty versions (in both 32 and 64 bit flavors) - see

This is a wiki page which I'll will try to keep up to date as the project evolves. Please feel free to comment, add notes, and correct any mistakes you see. For the BCFG2 version of these instructions (and pre-made, ready-to-go AMIs), see

3/2/10: 1.0 updates
1/11/10 Explain how to configure the external caching backend on Drupal.
12/2/09 Mercury 0.81
11/19/09 Mercury 0.8-Beta
11/14/09 Restructured instructions
10/22/09 Configuration file download instructions
10/8/09 Added Apache Solr to the install
9/20/09 Numerous fixes and the new pressflow BZR location

Read more
joshk's picture

Project Mercury: PressFlow Drupal+Varnish AMI Alpha4 Release

Today I'm glad to announce the latest release in this line of AMI development. This update solves a number of issues and moves us one step closer to a sable beta release. The current AMI ID is ami-c353b2aa, and you can find this AMI by searching for "chapter3" or "mercury" in your AWS console.

For more background information about this project, see my initial g.d.o post and my blog post announcing the initial release.

Below you will find the notes for this release. Also in this post I will include a development roadmap, as well as some more explicit explanation of the techniques I'm using for making the AMI work out of the box.

Read more
joshk's picture

Project Mercury: Pre-configured Drupal+Varnish AMI

Inspired by my own work over the past year with Amazon EC2 and this great post from Eric Hammond at Alestic on how to bundle public AMIs, today I released my first public machine image. I call the project "mercury," and the goal is to combine the power of Varnish and Pressflow Drupal in one easy-to-run package.

Why is this important? Because Varnish fills the same role as the Boost module, except it can handle 1000s of requests per second. Your constrained resources are going to be network and bandwidth. Getting it working well takes a bit of doing, but thanks to PressFlow and support from davidstrauss and DamZ, I've gotten a vanilla system working.

Read more
Jonas E's picture

EC2/S3 for DB-intensive drupal system

Hi everybody,

First of all, I am very new to Drupal and actually not actively coding myself but more involved as a project manager. However, as I've done some programming before and had some computer science modules at university I think I do have a basic grasp on the bigger issues around cloud computing.

We're just finishing a new system which we of course hope will become a huge hit but right now I am more concerned about general hosting issues.

Read more
muldos's picture

AWS Integration for drupal modules with Creeper

Hi all, I have posted a module for using easily the AWS services within drupal modules.
It is based on the tarzan api library
And I have provided as example / documentation the source code of a module that plugs the drupal
authentication system on an Amazon SimpleDB domain.

This is done here :

Feedback comments and suggestions welcome :)



Read more
Amazon's picture

Multi-master synchronous replication system for MySQL on AWS with Drupal as POC

At MySQLcon I ran into the guys from Codership. They did a 25 minute presentation on how to do multi-master synchronous replication across many instances of MySQL.

In layman's terms.

If you are running 5 Master MySQL databases behind your giant Drupal site, and one of those servers crashes, you won't lose a transaction, and you won't corrupt your logs and destroy your database.

Read more
Toktik's picture

Amazon S3 and Drupal

Hi all!

There is available any working module for Drupal 6 + S3? If yes, what features they have? If not, what we have at this moment, which is more or less working?

Thank you :)

Read more
bradlarson's picture

Tutorial for setting up a Drupal 6 site on Amazon EC2

I just completed a tutorial for running Drupal 6 on Amazon's EC2 at . With the advent of the persistent storage that Amazon's Elastic Block Store provides, I figured that it was finally practical to operate a website out of EC2.

Read more
greggles's picture

Amazon Web Services Discussion in Boulder

Per an Amazon Web Services evangelist will be in Colorado this coming week to give a talk and evangelize.

From the post:

Mike will be in town visiting TechStars on Tuesday, June 3rd. While he’s here, and since he’s such a nice guy, he’s also holding a get-together at The Cup in Boulder for anyone interested in Amazon Web Services. This will be at 10am on Tuesday. This is the guy to talk to if you want to learn more

Read more
bfelix's picture

Amazon FPS link

Just curious if anyone has integrated Drupal with Amazon's FPS. We are evaluating Drupal from a number of perspectives but we are looking at some microtransaction (<$10) business model which makes Amazon FPS appealing. Seems like there are links to but not yet to FPS?

Thanks in advance,

Read more

Resources for Drupal with Amazon Web Services

Thought it would be a good idea to have a resources wiki page where we could all share stuff relating to Drupal on Amazon


Amazon EC2 Console -

AMI's (selectable version of drupal + preconfigure xdebug/vim and output for kcachegrind/wincachegrind) Read more
Amazon's picture

Amazon releases elastic IPs - so you can resolve your domain name to your AWS instance

In order to host most Drupal sites you've needed a reliable IP which you can have a DNS mapping for. This is now available with AWS.

What other barriers need to be resolved for AWS to become the Drupal platform of choice?

Read more
fudge-gdo's picture

SysAdmin Amazon EC2/S3 & Drupal setup | SM

Employment type: 
Not allowed


Require a EC2/S3 expert to tweak and configure a drupal site running on AWS that we notice not performing as it should.
We are preparing for launching a project in the next week and we need to make sure all is configured correctly.
We have provisioned the Centos+Virtualmim AMI and installed drupal afterwards.

Appreciate your assistance.

George {at} gmailcom

Read more
toemaz's picture

FOSDEM 2008 presentation: Combining AWS with Drupal

If you are interested in a case on Drupal + AWS, you may want to watch my presentation captured at FOSDEM 2008. You can the slideshow as well. I used the Wikifonia project as the case study.

Want to start developing? Check out the aws module.

Read more
dragonwize's picture

SQS Changes its API and pricing

Amazon has just announced a new API and much lower pricing structure for their SQS service. Check out the migration guide.

Read more
dragonwize's picture

Amazon Web Services API Module

I have just uploaded the beginning of my Amazon Web Services API module.

This module is for other module developers that want to consume Amazon's web services. It abstracts the services to functions allowing developers to easily use them without dealing with the lower level details of creating, sending, and receiving the requests. Currently, only REST style requests are supported but SOAP is planned for a later release.

Read more
greggles's picture

ec2 image for Drupal

I keep seen these images for Django or MovableType or whatever - does anyone see value in an AMI for Drupal? I'm not sure if it would be valuable enough or not. I just setup a site based on the Gutsy Public Base Image and on Victor's guide to taking a base Ubuntu server to Drupal host. It wasn't impossible, but it was harder than I'd like someone using ec2 to try out a system to have to endure.

Read more
budda's picture

S3 integration module

I was expecting a Drupal module for S3 to already be in existence by now, as there isn't I have a proposal for a module I'm considering developing. Any feedback or suggestions on the implementation/features are greatly welcome.

This is not intended to be a full FileAPI-like module, but more a helper for general node related usage.

  • Support for node attachments (Core upload.module), CCK filefield uploads etc that are attached to a node.
  • Upon publishing of a node, any associated files are streamed over to Amazon S3 bucket(s).
Read more
Subscribe with RSS Syndicate content