Posted by SeanBannister on December 4, 2009 at 2:04am
Amazon just announced a new feature that allows EC2 instances to boot directly from an EBS Volume. This means the root partition of the instance is actually stored on an EBS volume, making the instance completely persistent. After reading the documentation it appears you can also automatically attach multiple EBS volumes to the instance when it's launched.
So this opens some interesting concepts for Pantheon and Project Mercury. And brings some interesting questions..... thinking out loud....:
- Should Pantheon use the this new EBS method or stick with the S3 method.
- Because we're no longer limited by a 10gb root partition do we really need to move mysql and tomcat from /tmp to /mnt/tmp? Should we just increase the size of the root partitions EBS volume.
- Should we move the mysql database and web root to a secondary EBS volume to promote a best practice of keeping that data separate. And if so how do we attach an existing EBS volume when a new version of Mercury is released?
Comments
It all sounds good I suppose.
It all sounds good I suppose. I would think it would be best to offer the existing method and consider offering the new EBS method as another option. I don't think it would be too much of a challenge to offer both.
I currently am running a 4 disk raid with EBS volumes to optimize performance on my instances so I don't really benefit from this improvement.
I love the idea of running EBS for my test instances with them persisting between being on/off.
Still wrapping my head around the rest of the potential directions that this opens up but it's cool stuff that's for sure.
Doug
@nullvariable | www.nullvariable.com
I wish I had enough time to
I wish I had enough time to try it now.. I found a handy script that should do the job well. http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html converts the old style to the new from a running instance.
I was thinking I could use the ebs instance store as well as Doug's great idea for the raid0 setup, should be able to do both? I was thinking a regular main root drive and then other raid0 rigs for the db files and other bits. Doug what is it with the raid setup that negates the instances as EBS? ...or were you speaking to something else?
Im testing the standard mercury ami now but im thinking I'll try this next week sometime and see how it runs.
now that I think through it I
now that I think through it I suppose it would work just fine. Just have to be able to mount the other four instances into your main instance. Since you're just converting an instance into EBS don't see why you can't connect other EBS volumes to it.
Doug
@nullvariable | www.nullvariable.com
Sweet thats what I am hoping,
Sweet thats what I am hoping, I will give it a whirl soon. Seems like it may be useful for a lot of things especially the full snapshots, thats very cool. I still have some more study to do but im thinking its going to work out quite nice. Im also wondering if it will speed up load balancing since the instances boot so much faster, im a ways off from testing that out though, and still may take things into Scalr.
The thing that puzzles me is updates to the ami, my guess is that I need to document my changes well so I can redo them when a new AMI comes out, that or the other way around take the new changes I want made to the AMI and then do them manually to my build.
Im just about ready to dive into the Solr stuff now .. Im thinking it will set our real estate search well above our local competition, with all that speed and cool features! wOOt!
Boot from EBS
I created a AMI from the 32 bit version of pantheon. I followed the steps outlined here,
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
But after the reboot, ssh doesnt seem to be enabled. Any ideas?
vssaokar ... im not really
vssaokar ... im not really sure what could have caused that ... did you try doing it over fresh again? Ive not tried this out yet so Im not the best judge. Maybe try adding a comment on their page about it if you cant figure it.
Has anyone gotten Mercury to
Has anyone gotten Mercury to boot from EBS yet? Very curious.
Thanks
I havnt tried it yet .. been
I havnt tried it yet .. been too busy, but there is no reason it wouldn't work. You could even go the other way around .. start your own (or use existing) ebs bootable ami. Then use the bgfg install scripts mentioned in the latest release. http://groups.drupal.org/node/46832 and here http://getpantheon.com/developer-zone
I wish I had time to try it out but I will sooner or later. I think the ebs store method may be more flexible.
Anyone have a working base AMI?
Well, here I am trying to do the same thing; hit the XFS problem
I'm going to try .9 from the existing 9.04 server 32-bit AMI (Eric Hammond's, used for Pantheon). If anyone else already has this working, let me know :)
~kwt
booting
Got it booting from EBS, that's fine, I just gotta get the /mnt mount over from /dev/sda2 (and restrict it's size, I don't need the full instance store 160gb for mysql just yet...)
So that means it's running, ssh works great, but because mysql & varnish are symlinked into the /mnt mount I don't have anything else running yet.
When I figure this little thing out I'll post...
working
So I got it running with a few caveats since XFS wouldn't play nicely:
I tried to move the ephemeral /mnt to a separate EBS volume with an XFS file system as told here - http://groups.drupal.org/node/33092 and everything was fine until a reboot, at which point it does a soft lockup and chokes after mounting xfs apparently - there is some possibly related writeups regarding this http://developer.amazonwebservices.com/connect/thread.jspa?threadID=2896...
So since the ext3 root EBS volume was just fine I instead copied everything back onto it and linked it all back up to itself...and voila after a few permissions changes to some folders it works
Next step is to move data back off to a XFS filesystem that works
Yeh ... traveled down that
Yeh ... traveled down that path several times in my tests, I've also just tested this on a 64-bit ami with the same results that I had on th 32-bit. XFS no likey the kernel being used in the Mercury AMI's (Ubuntu 9.04) {when combined with ec2 + ebs} ... Im going to try an ext3 raid now and test, XFS in this mix is too much of a PITA at this point.
So your saying you have mercury on an EBS bootable AMI?
Im tempted to try it again
Im tempted to try it again because XFS is overall better.
I just noticed that you can select a kernel when launching an instance. I wonder which one holds the fix...?
http://groups.google.com/group/ec2ubuntu/browse_thread/thread/7a2a7c1138...
I selected the default last time and got the reboot bug.
kernel
Yep got a self contained instance (ext3) of mercury running off of an EBS boot volume, works fine...
Would love it if you found a kernel that worked but that sounds scary since everything else is tested with the current kernel...this should be submitted as a bug on launchpad me thinks
Yeah .. there was one started
Yeah .. there was one started a while back. But not solution yet other than not to use XFS :(
https://bugs.launchpad.net/projectmercury/+bug/485563
Reading all the various postings around the net on this It seems its supposed to be fixed in the latest ubuntu AMI releases but I think the latest Mercury AMI still defaults to the wrong kernel. The Mercury AMI I just tested (64-bit) ami-e346ab8a the default kernel ID for it is Kernel ID: aki-a3d737ca ... Ill ask on IRC and dig around to see if anyone has a guess at a different kernel.
Thats very cool that you have the EBS instance working, did you do it starting from an ebs mounted & run config scripts; or convert the instance-store to an ebs bootable?
Im going to start with the
Im going to start with the Ubuntu 9.10 Karmic -Alestic, ubuntu EBS boot - the kernel is based on the latest Canonical AMI release. Eric Hammond said on IRC that that certainly has the XFS fix in it along with several other fixes and best yet he's made it an EBS store instance. I'm going to manually install the mercury stack on top of it. Its probably best that I do it this way anyhow as I will need this to be a snapshot with my desired spec including Aegir and multi-site memcached, multi-core solr etc. Being an ebs store I think is going to streamline things a lot!
The latest alestic & canonical releases can be viewed here http://alestic.com/ click your region under the page title and you can see them.
Be great if you could keep us
Be great if you could keep us up to date with how you go because I'll be needing to achieve a similar thing.
I'm wondering if BCFG2 support will make the install easier? Maybe you just run the BCFG2 configuration and it sets things up??
I installed bcfg2 on a fresh
I installed bcfg2 on a fresh instance and it seemed to be pretty straight forward .. however I didnt complete the install.... im on some time constraints to get my dev server back up and dev'n ... I had a bit of a detour with the whole XFS fun stuff.
Once I get a look at the process a bit more and figure how to get all my customizations combined with the stock mercury and/or mercury tweaks I may desire; then I think I can move on that, but its all a tad bit unclear how all that fits together happily. Thats why I figure the EBS store is good I can keep mucking around with things and always have many clean points to return to.
I followed the instructions from the mercury install wiki but referenced the config files from the latest release on launchpad.
Well Ive got everything
Well Ive got everything working well on the new Ubuntu 64-bit 9.10 Karmic -Alestic, ubuntu EBS boot.... aegir, mercury stack and all. With no XFS issues and my raid array that now also happily shows up for duty on reboots! YAY!
Lots more to do but things seem to be running great so far. Im loving being able to snapshot the main instance!
Was just thinking, that'll
Was just thinking, that'll send extra I/O to the EBS volume, an of course EBS volumes charge for that whereas EC2 doesn't. Be interesting to know how much I/O the operating system creates compared to mysql or apache.
True .. but its probably not
True .. but its probably not much of a difference. But truly its all relative isnt it? The way I look at it is that the EBS bootables are simply giving us an easier way to manage instances; its still the same hardware underneath it all. I dont think there will be much of a difference in the charge at least not in my case, and Amazon's track record has been to reduce prices as popularity grows.
EBS vs EC2
Does that mean that we don't need to bother with EC2 any more? We can just put everything on EBS and point a domain name at it? If so, is there any disadvantage to doing this. Or if not, what do I have wrong about your setup?
Christopher Pelham
Director
CRS (Center for Remembering & Sharing)
http://www.crsny.org
CRS (Center for Remembering & Sharing) is an arts & healing center located just south of Union Square in Manhattan.
Any chance on getting a wiki started for this?
Would love to know how to accomplish this.
Thanks!
I'd personally like to see
I'd personally like to see the 1.0 release officially support an EBS AMI but I guess that'll be up to Chapter Three. If not someone could always release a public EBS AMI.
I second that!
Would love to be able to spin up an instance, tweak it to my needs, and keep it around. Are there any downsides to this? It seems like a no brainer.
The only downside I can see
The only downside I can see is because the root partition is hosted on an EBS volume your charged for I/O where as your not charged for I/O on a normal EC2 instance. As I previously mentioned I was interested in the cost so I worked out a way to calculate it, on 3 small e-commerce sites the cost was $5.70 a year :) Can't complain about that.
For anyone wanting to calculate their their I/O usage I just worked out an easy way, type:
cat /proc/diskstats
In the list find the location of your root partition, usually sda1. The output will look something like:
8 1 sda1 129871 38178 5743690 767396 6473673 10558585 136258456 14476752 0 5425472 15244144
Add the 1st and the 5th number together to get the total I/O since last boot. And calculate it based on amazons $0.10 per 1 million I/O requests. For example :
129871 + 6473673 / 1000000 * 0.10 = $0.66
Keep in mind that will be the cost since you last boot/reboot the instance.
Thanks for that
EDIT: I just did this calculation for my site, and it came in around $70 for the year (relatively busy site). Another thing to consider is that Varnish will probably lower disk activity quite a bit, so I think this is noise.
Any sense of I/O performance differences?
BCFG2 and Ubuntu 9.10 Karmic -Alestic, ubuntu EBS boot
Anyone know if the Mercury BCFG2 supports Ubuntu Karmic? It seems like it is geared for Jaunty.
currently tailored for jaunty
I was looking this over yesterday, and I think you're right... For example only jaunty is mentioned at http://bazaar.launchpad.net/~pantheon-developers/pantheon/trunk/annotate... and http://bazaar.launchpad.net/~pantheon-developers/pantheon/trunk/files/he...
That said, it doesn't seem like there's too much work to create karmic versions of the various configuration files.
Ubuntu Karmic
We don't have bcfg2 config files for Ubuntu Karmic out yet, but noahb is correct in that it shouldn't be a huge change from jaunty.
The change is that the Karmic AMIs from Canonical and (hence) Alestic use the "ubuntu" account - not root. This means that most of the commands in our Mercury instructions (http://groups.drupal.org/node/50408) need to be run using sudo.
Greg
--
Greg Coit
Systems Administrator
http://www.chapterthree.com
Hi Gregg, Thanks for all the
Hi Gregg,
Thanks for all the fantastic work you (and Chapter 3) have been been doing...
I was able to get mercury installed on my rackspace cloud last night, and was wondering if where I can submit bcfg2 patches that allow for this... I'm not sure they're perfect yet, mainly because of the difference btn rackspace and EC2 versions of karmic (?), but they're a good start.
RE: Hi Gregg
Noahb,
We have a group on launchpad called pantheon-developers (https://launchpad.net/~pantheon-developers) which folks can submit patches/code/config_files for the project pantheon (https://launchpad.net/pantheon).
Thanks!
Greg
--
Greg Coit
Systems Administrator
http://www.chapterthree.com