Amazon Web Services (S3, EC2)

Events happening in the community are now at Drupal community events on www.drupal.org.
This group should probably have more organizers. See documentation on this recommendation.
R-H's picture

SSH Authentication with Private Keys

Hello,

I've managed to authenticate. My first missteps were not understanding that ports 80 and 22 had to be open. I get the concept but for a non-sysadmin I assumed the EC2 defaults for Security Groups would be set to work automatically. I did finally manage to get to root using Terminal though once these ports were open.

Since I prefer a GUI I'm now trying to get past having to use Terminal on the Mac, but I can't really find a Mac SSH client that works with EC2's private keys. They all want me to authenticate with a username/pw.

Read more
R-H's picture

Just wanted to say hello and thanks

Hey Ho!

Read more
Greg Coit's picture

Mercury 0.71-Beta Released

Mercury 0.71-Beta has been released with small bugfixes (see https://bugs.launchpad.net/projectmercury for details).

AMI IDs are:

US 32-bit: ami-bd7c9fd4 - chapter3-storage/PANTHEON-pressflow-mercury-0.71-Beta.manifest.xml
US 64-bit: ami-b17c9fd8 - chapter3-storage/PANTHEON-pressflow-mercury64-0.71-Beta.manifest.xml
EU 32-bit: ami-8f2c07fb - chapter3-storage-europe/PANTHEON-pressflow-mercury-0.71-Beta.manifest.xml
EU 64-bit: ami-73230807 - chapter3-storage-europe/PANTHEON-pressflow-mercury64-0.71-Beta.manifest.xml

Read more
SeanBannister's picture

User-Data integration with Project Mercury

I've been working on better integration with EC2 user-data for Project Mercury. The goal is to allow users to specify configuration options when they launch a Mercury instance to simplify the process of manual configuration.

For example if you'd like the EC2 instance to receive an Elastic IP address you'd specify:
IP="new"
Or an existing elastic IP address:
IP="123.123.123.123"

Read more

Setting up EBS for Mercury [Beta]

Once you've followed the steps outlined for installing your site on Mercury you'll then want to take your site to the next level using Elastic Block Store. This is going to keep your MySQL database(s) persistent should your server instance crash. Also make it easy to replicate data for testing etc.

Read more
joshk's picture

Project Mercury Beta!

With great pride, and after six alpha-level releases, I'm announcing of our seventh iteration on the Project Mercury stack, finally baked enough to call "beta".

At this point, we know that many people are using the Mercury EC2 image in production environments, and we've tuned this release conservatively to prevent it from breaking down under heavy load. We've also verified that the stack will work under a resource-constrained VPS (e.g. one with 1/4th the RAM of a small EC2 image), which gives us more confident that this configuration is stable. We also have a kickass logo:

Only local images are allowed.

Read more
Greg Coit's picture

Additional testing of Mercury with 2GB and 512MB RAM

My name is Greg Coit, sysadmin for Chapter 3 and I've been helping with Mercury development and testing.

We wanted a get a quick idea of how hard we could push mercury under more "real world" circumstances, so I combined siege and ab to generate a broad spectrum of hits. ab (short for apache benchmark and part of the apache2-utils package) allows you to generate a very large number of hits on one url, while siege (a perl script which comes in a self-titled debian/ubuntu package) lets you spread the hits across many urls, most of which won't be cached. This mixed-load is a much more nuanced and accurate way of looking at performance than peak throughput on a single url.

Read more
joshk's picture

EC2 Offering "MySQL Cloud"

Today Amazon announced a provided service for hosting MySQL databases in the cloud: Amazon Relational Database Service. It starts at $0.11/hr, plus $0.10 per GB/month of storage and $0.10 per million I/O requests. The real interesting question is how these will perform. The instances come in many flavors:

  • Small: 1.7 GB memory, 1 virtual core with 1 ECU. $0.11/hr
  • Large: 7.5 GB memory, 2 virtual cores with 2 ECUs each. $0.44/hr
  • Extra Large: 15 GB of memory, 4 virtual cores with 2 ECUs each. $0.88/hr
  • Double Extra Large: 34 GB of memory, 4 virtual cores with 3,25 ECUs each. $1.55/hr
  • Quadruple Extra Large: 68 GB of memory, 8 virtual cores with 3.25 ECUs each. $3.10/hr

That "Quadrupal XL" is pretty big! I'll be investigating this for Pantheon, but anyone else with experience/info, let's figure out how good these things really are!

Read more
joshk's picture

Initial Mercury Results From 512MB VPS deploy

Just a heads up; as we move towards more and more stable builds of the Mercury stack, we are starting to look at deploying it on other infrastructure besides EC2. This week, we set it up and tuned for a modest (512MB of ram) VPS. These tests were successful. We were able to simulate a mix of non-cached traffic along side the simple ApacheBench battering, and the system held up well, even without gigabytes of ram to support it.

Read more
joshk's picture

Project Mercury Alpha 6: Now With Solr!

I'm happy to announce the 0.6 Alpha release of the Mercury AMI, now including ApacheSolr as the search backend! This is the last piece of major infrastructure we want to integrate into the stack for scalability purposes. You can now move from a single-server install based on Mercury to a best-practice vertically scaled architecture with separate hardware to run front-end cache, application, back-end cache, search and database!

The quickest way to find it is by searching Amazon EC for "Pantheon" or "Mercury". The manifest path for the latest release (in 32bit and 64 bit flavors) is:

  • chapter3-storage/PANTHEON-pressflow-mercury-alpha-6.manifest.xml
  • chapter3-storage/PANTHEON-pressflow-mercury64-alpha-6.2.manifest.xml (back!)

If you'd like to "roll your own" we've updated the wiki instructions page with a new set of instructions for getting Solr up and running as part of the process. Feel free to improve that documentation, as it's definitely a community process.

This will likely be one of the last releases before we move the project into the Beta phase, at which point we'll be focusing on fine tuning and stability as well as portabilty onto non EC2 systems moreso than new features. If you have ideas for additional things you'd like to see integrated in the stack, please chime in. We're also going to be documenting real-world "how to" use-cases — e.g. "how do I put my existing site on Mercury" in user-friendly detail — so stay tuned for that.

As always, let us know what you think of the release, what you'd like to see in future iterations, and how your experience is in using the stack. There's plenty more to come.

Read more
joshk's picture

Mercury Update: Beta Coming Soon!

Just wanted to let everyone know we're hard at work on a Beta release of Project Mercury. Currently we are tuning Varnish further, working to package Apache Solr on board as the search backend, and testing the whole system for bugs. We will likely issue one more alpha release (0.6) before we get to a Beta, but I want to have that launch by the end of the month if possible.

The high level road map:

  • Package Solr
  • Improve default Varnish config
  • Thoroughly document "putting your site on mercury"
  • Resolve lingering bugs (e.g. cron.php issues)

Once we hit Beta we'll be pushing for wider adoption, which means pursuing alternate non-Amazon hosting options. Currently we're looking at alternate cloud providers, VPS partnerships, and other packaging options like VM images, etc. So stay tuned, and let us know what else you'd like to see us look at including early in the Beta cycle!

Read more
steven jones's picture

How do you install Drupal?

We're looking into beefing up our automated testing framework, and I'm wondering about how others are doing it. The main sticking point for us is how we actually install Drupal. I know that the economist.com guys have a database import, and then run updates against it, which is how you'd have to test an existing site, and I guess has simplified the problem, but does mean that they never visit install.php and actually do an install. This works for them because they only have one site.

Read more
iaminawe's picture

Pantheon Aegir Database settings

Hi,

In trying to take the Pantheon Aegir AMI for a spin, I hit my first obstacle with the database settings during the install.

This may be very obvious beginner mysql mistake but do I need to create the database, user and password before attempting the install?

I tried logging into mysql to create a database and set it up but realised that nowhere in the basic documentation does it mention the mysql password.
I tried the usual suspects , leaving it blank, drupal, mysql, root but none seem to work. I should mention I am trying to login as "root".

Read more
joshk's picture

New 0.51 Mercury AMIs: Many Fixes and 64bit!

I just wanted to post an announcement that we've finally gotten out a point-update of the Project Mercury AMI. Just in case you were wondering if this project would continue, it will! I've been really excited and encouraged by all the positive feedback so far, so keep your ideas and questions coming. The 0.51-Alpha release includes a number of bugfixes and improvements, most notably it:

  • Is based on the latest Pressflow including Drupal Core 6.14 and Simpletest 2.9
  • Fixes the self-update process to merge correctly and pull from Pressflow's lovely new VCS home on Launchpad
  • Includes the rc1 version of cacherouter
  • Fixes postfix and s3 metadata issues so there's now a working MTA out of the box

Most importantly for people considering this stack for production deployments, we're now bundling 64-bit images with every release. The quickest way to find the AMIs is to keyword search for "mercury" in your favorite EC2 console. More information and AMI ids are below the fold. Let me know what you think, and what you'd like to see next!

Read more
iaminawe's picture

Using Project Mercury and your own site

Hi,

I was wondering if there is one consolidated set of instructions anywhere that deals with how to set your own drupal based site over the base pressflow site launched over your EC2 mercury instance?

Basically I would like to know more about the following steps.These seem to be covered in the www.getpantheon.com intro video but the section is sped up and very difficult to try and dissect to understand what is going on.

I have successfully launched a small instance running the Mercury AMI. I am able to login to pressflow and have root access via SSH to the EC2 box.

Read more
joshk's picture

PANTHEON/Mercury BoF Friday at DrupalCon Paris

Start: 
2009-09-04 14:50 Europe/Paris
Organizers: 
Event type: 
User group meeting

Just a quick note: we're going to hold a BoF for people interested in Project Mercury (and other packages from PANTHEON) in Paris! It will be after lunch on Friday at 2:50.

Looking forward to talking to those of you who are here at DrupalCon.

Read more
joshk's picture

Project Mercury Benchmarks: 2000+ Requests Per Second!

While working through some issues this weekend and preparing another blog post, I finally got around to doing some comprehensive benchmarking on Project Mercury. In the process, I discovered that the first bottleneck I hit running tests from my desktop was the local (last-mile) internet connection, so I switched to running the tests from another EC2 instance. This means that network is not a factor in my results, giving us a real sense of the raw power behind this stack.

For all these tests, I used the Mercury Alpha4 release on a small ec2 instance, loading a staging copy of Mission Bicycles, which is a good "heavy" example in that it has a lot of modules loaded, including Ubercart and Panels. My goal was to measure throughput and response times under various caching configurations, angling for the best results in terms of pages served per second, and delivery time.

I started by cutting things all the way back to nothing, and then added each layer of the caching infrastructure, running benchmarks at each point. The results are quite eye-opening. Can you say 2000+ requests per second? Read on for the full story.

Read more

Step-by-step: Setting up Project Mercury (Varnish, Apache, APC, Memcached and Solr)

THIS DOCUMENTATION IS DEPRECATED!
Refer to the new wiki for better docs. This page is here for posterity.

Here are step-by-step instructions for building Project Mercury on a fresh server using the configuration manager BCFG2. Project Mercury has been developed on AWS ec2 but should work on Rackspace and many other VPSs. We have public AMIs available of Ubuntu Jaunty versions (in both 32 and 64 bit flavors) - see http://getpantheon.com.

This is a wiki page which I'll will try to keep up to date as the project evolves. Please feel free to comment, add notes, and correct any mistakes you see. For the BCFG2 version of these instructions (and pre-made, ready-to-go AMIs), see http://groups.drupal.org/node/50408

3/2/10: 1.0 updates
1/11/10 Explain how to configure the external caching backend on Drupal.
12/2/09 Mercury 0.81
11/19/09 Mercury 0.8-Beta
11/14/09 Restructured instructions
10/22/09 Configuration file download instructions
10/8/09 Added Apache Solr to the install
9/20/09 Numerous fixes and the new pressflow BZR location

Read more
joshk's picture

Project Mercury: PressFlow Drupal+Varnish AMI Alpha4 Release

Today I'm glad to announce the latest release in this line of AMI development. This update solves a number of issues and moves us one step closer to a sable beta release. The current AMI ID is ami-c353b2aa, and you can find this AMI by searching for "chapter3" or "mercury" in your AWS console.

For more background information about this project, see my initial g.d.o post and my blog post announcing the initial release.

Below you will find the notes for this release. Also in this post I will include a development roadmap, as well as some more explicit explanation of the techniques I'm using for making the AMI work out of the box.

Read more
joshk's picture

Project Mercury: Pre-configured Drupal+Varnish AMI

Inspired by my own work over the past year with Amazon EC2 and this great post from Eric Hammond at Alestic on how to bundle public AMIs, today I released my first public machine image. I call the project "mercury," and the goal is to combine the power of Varnish and Pressflow Drupal in one easy-to-run package.

Why is this important? Because Varnish fills the same role as the Boost module, except it can handle 1000s of requests per second. Your constrained resources are going to be network and bandwidth. Getting it working well takes a bit of doing, but thanks to PressFlow and support from davidstrauss and DamZ, I've gotten a vanilla system working.

Read more
Subscribe with RSS Syndicate content

Amazon Web Services (S3, EC2)

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: