Posted by kcoop on January 30, 2010 at 12:08am
I've been keenly following Mercury's progress, and I've now had a chance to play with an instance or two. Great stuff!
I'm wondering if anyone is using it in production. If so, what are you doing about persistence? It appears a mounted EBS volume is the recommended path, but I'm wondering what happens if the server crashes/needs to be rebooted? The wiki doesn't mention anything about startup scripts. Is it expected that you'll run through the volume attachment and mounting procedure at that point? And if I install and configure other code (like munin) I'm going to have to run through that again? Seems rather awkward.
Comments
Running in production with EBS
http://musescore.org is running Mercury in production since October 2009. It uses EBS to store the httpdocs, mysql index and solr index on. Initially, with the 0.7 release, no problems where encountered. With the 0.8 release, a problem popped up with the xfs formatted EBS (See issue https://bugs.launchpad.net/projectmercury/+bug/485563). This caused the instance to crash: it was not possible anymore to recover the instance (no ssh access, reboot failed as well). In this case, we killed the instance, restarted a new image and connected it with the existing EBS which apparently did not corrupt. This happened 4 times until we figured that the problem came from the xfs block. Now, we run successfully since more than 1 month with a ext-3 formatted block.
For more info, you can find me at DrupalCon SF. Don't hesitate to vote for my session once the voting is open: http://sf2010.drupal.org/conference/sessions/drupal-made-us-grow-1000
EBS as a Raid
That's too bad. I just followed the guide "Setting up EBS as a RAID" (http://groups.drupal.org/node/36750) and XFS is used as the file system for EBS.
Any idea if Mercury will be updated to address the XFS bugs?
If not any guidance for how to use ext-3 instead of XFS is appreciate.
Thanks
AFAICT even the new ami is
AFAICT even the new ami is 9.04 and the fix needed is in 9.10 .. that said .. most have said that there have been no complaints from 64-bit ami's, for some reason. I am running a 64-bit instance now so I will test for sure, id much rather use XFS for the better consistent snapshots... supposedly they can be accomplished with the ext3 too but its less than the level XFS can provide .. as far as freezing mysql whilst your task is completed; plus XFS is supposed to be faster too.
Managed high availability hosting
Hi, if you are looking for persistence and high availability for mercury you might want to consider Acquia hosting. We use many of the same optimizations on AWS that project mercury uses but it's configured in a high availability cluster which uses configuration automation so that when an instance fails it both fails over to another load balancer, webservers, master database server where appropriate, and then restarts and re-configures the new server back into the cluster automatically.
This is all monitored and maintained by an experienced systems operations team 24/7 with full support. There's also a lot of optimizations that you can do across a cluster of servers that you can't do on a single AWS EC2 instance.
For more details see:
http://acquia.com/products-services/acquia-hosting
http://acquia.com/products-services/benefits-acquia-hosting
http://acquia.com/products-services/acquia-hosting-features
http://acquia.com/resources/acquia-tv/look-under-hood-acquia-hosting-dru...
Cheers,
Kieran
Thanks Kieran
Sounds like Acquia would remove many headaches for me, but my budget is a bit more restricted than a minimum of $500/month. :-(
You could use amazons Cloud
You could use amazons Cloud Watch, Load Balancer, Amazon Relational Database Service (or ur own db ami), and go about splitting your instance up a bit into individually scalable pieces; but this will require you learn and maintain another level of control over your system in-house. It might cost less than the aquia in your monthly fee but you'd be spending/investing more time/money to make it work without a doubt. Aquia service comes with an ease of use that certainly makes it worth the price IMO. Ive not used aquia personally but I think its fairly priced for what your getting.
There is also http://code.google.com/p/scalr/ its opensourced now...
and is also SaaS too https://scalr.net/ I have had the chance to try the scalr SaaS and its pretty neat no doubt!
Be interesting to see if we could use the GNU scalr goodies with Mercury .. my brief glance over it doesnt scream out any potential conflicts with the Mercury stack AFAICT.