Posted by joshk on October 7, 2009 at 8:51pm
Just wanted to let everyone know we're hard at work on a Beta release of Project Mercury. Currently we are tuning Varnish further, working to package Apache Solr on board as the search backend, and testing the whole system for bugs. We will likely issue one more alpha release (0.6) before we get to a Beta, but I want to have that launch by the end of the month if possible.
The high level road map:
- Package Solr
- Improve default Varnish config
- Thoroughly document "putting your site on mercury"
- Resolve lingering bugs (e.g. cron.php issues)
Once we hit Beta we'll be pushing for wider adoption, which means pursuing alternate non-Amazon hosting options. Currently we're looking at alternate cloud providers, VPS partnerships, and other packaging options like VM images, etc. So stay tuned, and let us know what else you'd like to see us look at including early in the Beta cycle!
Comments
Suggestions from IRC
I got some suggestions in IRC that I want to keep track of here:
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Hi Josh, Are backup scripts
Hi Josh,
Are backup scripts for Amazon S3 included as well, perhaps even preconfigured for cron runs?
For many I believe this is one of the most important things :)
****Me and Drupal :)****
Clickbank IPN - Sell online or create a membership site with the largest affiliate network!
Review Critical - One of my sites
Not S3 but EBS
Hi giorgio79,
This AMI will use EBS, an elastic block in order to store everything safely. You can always create a snapshot from an EBS and dump it to S3, but EBS is redundantly stored as well. So don't worry about backups. Great, isn't it?
The EBS should contain:
- /[EBS-mount]/www folder (change the apache directive to the EBS mount)
- /[EBS-mount]/mysql (change the directory in the mysql conf file)
- /[EBS-mount]/solr
and that's about it.
Not so fast...
I'd recommend writing log files to EBS. Same does for any other custom configuration. Finally, redundancy does not = backup. You'll want to save snapshots in case a file gets deleted or you somehow screw up your EBS. You can never be too safe.
I'm currently working on an
I'm currently working on an automated EBS backup solution that'll integrate with the user-data scripts I'm writing. So eventually you'll be able to launch a Mercury instance with an EBS volume ready to go and backing itself up.
Check this out as
Check this out as well:
http://rsnapshot.org/
I have had great success with it :)
****Me and Drupal :)****
Clickbank IPN - Sell online or create a membership site with the largest affiliate network!
Review Critical - One of my sites
And here is the one I use for
And here is the one I use for mysql :
http://sourceforge.net/projects/automysqlbackup/
:)
Probably one of the best out there IMHO
****Me and Drupal :)****
Clickbank IPN - Sell online or create a membership site with the largest affiliate network!
Review Critical - One of my sites
Thanks for those, I'll check
Thanks for those, I'll check them out. My goal at the moment is using EBS snapshots as it'll make it easy to restore or launch more Pressflow instances. I'm sure we'll need more in-depth solutions in the future but this is a good start and something everyone will need for convenience.
MultiSites
I've not noticed anything about running multisite install on Mercury. Are multisite install being considered in the design?
--
http://abrah.am
Haven't had any problems with multisite + Mercury
It was as easy to set up as multisite with any normal installation, but for one difference: edit settings.php to include the cacherouter config for each multisite and make sure it's using a prefix for each site, to prevent collisions.
Awesome
Awesome. Good to know.
--
http://abrah.am
EBS, snapshots, ec2-api-tools
EBS is not fully redundant according to Amazaon (http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/). Could it be an idea to include the excellent Alestic / Eric Hammond script for creating consistent snapshots. http://alestic.com/2009/09/ec2-consistent-snapshot.
Also, what about the EC2-api-tools. They are needed for a lot of operations (i.e making a rebundle of the image, deleting snapshots ..) - or is that to much bloat.
Finally - an intelligent set of mounting scripts for different situations; A new instance & and a new volume, A new instance with an old volumen .?
I'm looking forward to
I'm looking forward to authenticated users benchmarks.
Yup.
I agree. Authenticated user benchmarks would be very helpful in evaluating Mercury. They wouldn't tell the real story though. Even if authenticated users see little benefit from Varnish, using the reverse proxy will free up resources in typical mixed workloads. For example, if 50% of site traffic is anonymous and handled by Varnish with ease, you've just freed up resources to better handle logged in users.
It'll be best-practice
You're correct that Varnish helps out mostly by taking simple work off the back of Apache/PHP, so that it can focus on the more important task of serving logged-in users. Even if your site is only 50% anonymous traffic, that's still 50% more horsepower for the logged-in users.
Aside from that, Mercury doesn't (yet) introduce any breakthroughs in logged-in user performance. That will have to wait until we get the advcache module up to snuff and possibly start working ESI support into Pressflow/Varnish. For now, you're just going to be getting the Best That Drupal Can Currently Do, which is still pretty good. ;)
The other important point is that with Varnish in play, we can tune Apache/PHP as an application server, meaning we can reduce the number of MaxClients down to a relatively low number since they'll only be used for unique pages. This allows you to have more safety within your environment: you can caluculate your average memory usage and be fairly certain that you won't run your server into swap, which is a death-spiral, without sacrificing performance under normal loads.
In plain Wnglish that means that while your site may still get slow, you can weather spikes in logged-in traffic without literally crashing (being forced to reboot) the server. Anyone who's ever seen boxes dip into swap under load will know what I mean. It's generally a non-recoverable situation.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Sounds good to me
Following best practices and some tuning should go fairly far for logged in users compared with a vanilla LAMP install (no APC, default PHP/Apache/MySQL. From experience, I completely agree about needing to avoiding swapping. Edge Side Includes support would be amazing... I'd assume an order of magnitude improvement.
Thank you, and I understand
Thank you, and I understand completely, the usefulness of varnish, etc. Still the difference it makes for me is probably going to be a bit less than that it makes for you. I use a module to convert all anon users into registered users, and though obviously the performance is much worst (thus far, w/o authcache, though that module is supposed to rock my world), the usability benefits are enormous.
It would be nice to see auth benchmarks for the purpose of comparing Best That Drupal Can Currently Do to my own setup, see how much I'm lacking, and whether or not I urgently need to convert to Mercury and so on. But I'm definitely going over to Mercury, this is too smooth of a bandwagon to not ride.
Thanks for the great project again.
Varnish memory consumption
Hi Josh,
I launched mercury (ami-a855b5c1) on a small EC2 instance and I'm testing it with a live site. Currently $top returns this list ordered on memory usage. It takes a lot of memory. I have no experience with Varnish so I wondered whether this is expected memory consumption (~42% of 1.7GB)?
Varnish is an in-memory cache
Therefore it naturally reserves a lot of memory.
That's a good thing. Ideally in production you always want 100% memory consumption, because otherwise you're letting capacity go to waste when it could be used for caching (and therefore accelerating) data.
The important thing is just that the memory is being put to good use.
D
====
Dan Walmsley
Managing Director, Gravity Rail Pty Ltd
Connecting Artists to Audiences
Yep
For safety, the varnish cache is capped at 1gb, so it won't eat all the space. If you have more memory, you can increase its size. We are also working on some auto-tuning scripts so we can launch on smaller VPS systesms with a 100MB varnish cache. Not as robust, but certainly helpful for most sites.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
and other packaging options
I woud love to see some VM images, so we can do local testing with things like VirtualBox before we go and put it onto a hosting service.