User-Data integration with Project Mercury

Events happening in the community are now at Drupal community events on www.drupal.org.
SeanBannister's picture

I've been working on better integration with EC2 user-data for Project Mercury. The goal is to allow users to specify configuration options when they launch a Mercury instance to simplify the process of manual configuration.

For example if you'd like the EC2 instance to receive an Elastic IP address you'd specify:
IP="new"
Or an existing elastic IP address:
IP="123.123.123.123"

Features that are currently working :

  • Change the MySQL Password and also change it in Drupals settings.php.
  • Set the username, password and email of Drupals Admin user (User 1).
  • Create a new Elastic IP address or attach an existing Elastic IP to the EC2 instance.
  • Set the servers hostnames (to be placed in /etc/hostname or /etc/hosts).
  • Create a new EBS Volume (Create, Attach, Format and Mount the Volume).
  • Attach and Mount an existing EBS Volume.
  • Move existing data from the server to an EBS volume (or any other location such as /mnt).
  • Use existing data on an EBS volume in place of data on the server.

Future features that'd like to work on include :

  • Creating Raids from EBS volumes to increase I/O performance
  • Automated backups of EBS volumes (based on Eric Hammonds script)

Currently this is a private image as I'm still fixing a number of bugs but I plan to release a beta or get these features into the next release of Project Mercury, if possible.

Comments

EBS RAID

nullvariable's picture

I found this article helpful for setting up EBS RAIDs: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/

This blog post has a great cron scripts for EBS snapshots: http://orensol.com/2009/08/10/cron-script-to-snapshot-any-attached-ebs-v... it doesn't do RAID since all the drives need to be frozen and 'snapshotted' at the same time but it does use AWS command line tools to enumerate all the drives mounted to the instance. It should be a short leap to freezing all the volumes at the same time using pretty much the same idea.

Doug
http://www.nullvariable.com | http://dougcone.com

Thanks for that. I really

SeanBannister's picture

Thanks for that. I really like the concept of snapshotting all volumes attached to an EC2 instance. But I'm a fan of Eric Hammond's script http://alestic.com/2009/09/ec2-consistent-snapshot which already does snapshots of Raided EBS volumes, and it's pretty advanced.

What I might do is create a wrapper around ec2-consistent-snapshots with a loop to pass all EBS volumes to it. Or I could submit a patch and see if they'll accept it into ec2-consistent-snapshots.

Just an update on features

SeanBannister's picture

Just an update on features that I'm planning to work on:

  • Automated backups of EBS volumes (based on Eric Hammonds script) Almost complete
  • Creating Raids from EBS volumes to increase I/O performance
  • Automatically set memory limits such as APCs, PHPs, Varnishes and Servers shared memory based on the amount of memory the server has.
  • Support for launching an Amazon RDS server, moving the database to it and changing the database server in settings.php
  • The ability to permanently disable a service, for example if you've launched an RDS server you no longer want MySQL running on the EC2 server. This could also apply to permanently disabling varnish on larger sites where you run your varnish server separately from Apache.
  • Integration with Amazon Cloud Front

Amazon Web Services (S3, EC2)

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: