Aegir in Mercury & other questions

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Macronomicus's picture

Ok so I noticed your Aegir build but also noticed joshk's comment here http://groups.drupal.org/node/27134#comment-92680 So if its really different am I to assume it doesn't include all the goodies in Mercury? Im guessing so based on the description over at getpantheon.com ... so I currently have a dedicated server in which i have been running aegir for some time now quite happily ... what I wanted to know is if any of you are running Aegir inside your Mercury goodness? Are there any reasons not to?

I've been toiling for a while now trying to decide on our next server move as in 2010 we are going to get hit hard and fast because of a lot of tv/press etc slated for several of our companies. The EC2 goodness has more than sparked my interest for obvious reasons I'm sure I need not explain here. So I am curious as to how happy you all are with the setup, and how you would compare it to other cloud solutions? Any thoughts will be much appreciated!

Also I just started looking at Aquia's cloud hosting offering and would love to see any comparative bits about it next to EC2 if you've found or know of any, I just contacted them so i should have more info soon; I'm curious as to other folks thoughts, especially if they use it.

Cheers!
Joshua

Comments

I ran the Aegir install on a

SeanBannister's picture

I ran the Aegir install on a Mercury instance a few days ago, unfortunately I didn't have much time so I didn't get everything working as I would of liked. There is a slight bug in the install that doesn't appear to be Mercury related, it's most likely Ubuntu related or more specifically Dash. It causes drush.php to not run from the absolute path that the script calls so you have to run the commands manually. Hopefully this will be fixed in the near future.

I spoke with Josh about the Pantheon Ageir image and it doesn't include all the Mercury goodness, I believe it's goal is to manage Mercury instances in large scale deployments. My personal use case is for my clients who only have one web server but they want the benefits on Ageir on it. So I would also eventually like to see a Mercury Instance with Aegir pre-installed, I'll have to role one for my own use so maybe I'll release it and maintain it if need be. Just need to hear what the plans are for Pantheon first as we don't want to fragment development.

Acquia Hosting is actually built on EC2 theres a reference to it at http://acquia.com/blog/acquia-2009-roadmap "We built acquia.com in the cloud (using Amazon EC2)". I'd be interested to hear the prices your quoted.

Yeah I think I remember

Macronomicus's picture

Yeah I think I remember seeing some discussions on irc about the Dash thing .. I use Bash so ive not had the troubles thankfully .. I think I would go into shock without Aegir/Drush .. im addicted!

I see your point about the aegir inside the image but its probably best left as an add on in case someone doesn't want it .. might be a lot to maintain two separate images just for aegir.. its a sinch to install anyhow so long as your sys is setup for it.

Ahh I did not see that post, thanks for the link. Ok cool so they are in AWS but they have done all the work for setting up and maintain the high availability stuff, ive still not seen to many tech details on what they have yet. May be worth it in the end, especially since it supports Drupal. I suppose it depends on the entry cost and specifics .. but im guessing it should be setup quite well, I should hear back from them soon. I could let ya know the price but I dont want to put their prices up on here as they dont list them on their site yet and I dont want to jump ahead of them, might be kinda rude since they are just starting out and rolling out things as they see fit.

Cheers for sharing your thoughts. ^_^

Im getting ready to test

Macronomicus's picture

Im getting ready to test Aegir in Murcury on aws and see how it goes, I cant think of any reasons why they would be incompatible.

All thing appear to be up and

Macronomicus's picture

All thing appear to be up and running well enough .. I was a bit confused by the 8080 port at first, and had to manually edit the vhost.d file for the main aegir site, guess I forgot to set 8080 in the install screen

I went ahead and left the Aegir as is and then setup my other platforms for sites as pressflow drupal. Ill have to find some methods for testing in more detail but so far my basic tests are fantastically fast. I dont want to get into the larger tests until ive setup my load balancer.

So yeah I was a bit worried things wouldn't be compatible but it seems that they are and Aegir is perfectly happy married to Mercury, lol.

In case someone else wants to

Macronomicus's picture

In case someone else wants to try this .. there was one snafu that I missed until now, in order to make cacherouter work (direct drupal cache's to apc) you need to customize your settings.php which is not outta the box possible withe Aegir; the reason being is that every time you run a Verify task on a site or platform the settings.php files are all regenerated and you loose your customizations. ...though thankfully there is a workaround.

After installing Aegir you will need to...
First unset the forced normal cache bit in...
/.drush/provision/platform/provision_drupal_settings.tpl.php
just comment out the conf bit like such
#$conf['cache'] = 1;

Next create a global.inc file in aegir's main includes dir /vol/aegir/config/includes/global.inc
Then copy over your bits from the main pressflow site included in the AMI.. I also set the offline theme, anything u'd place in settings.php can go here

<?php # global settings.php

$thishost = $_SERVER[HTTP_HOST];
if(
preg_match("/(dev\.mesite\.net)/", $thishost)) {
   
# CUSTOM STUFF HERE
   
$conf['maintenance_theme'] = 'me_theme';
   
   
# below is stuff from the mercury ami's default pressflow site settings.php
   
$GLOBALS['simpletest_installed'] = TRUE;
    if (
preg_match("/^simpletest\d+$/", $_SERVER['HTTP_USER_AGENT'])) {
     
$db_prefix = $_SERVER['HTTP_USER_AGENT'];
    }   
   
# Cacherouter: use APC for all local caching
   
$conf['cache_inc'] = './sites/all/modules/aa_pressflow/cacherouter/cacherouter.inc';
   
$conf['cacherouter'] = array(
     
'default' => array(
       
'engine' => 'apc',
       
'shared' => FALSE,
       
'prefix' => '',
       
'static' => FALSE,
       
'fast_cache' => TRUE,
      ),
    );
   
# Varnish reverse proxy on localhost
   
$conf['reverse_proxy'] = TRUE;          
   
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
   
# END CUSTOM STUFF
}
?>

Then set your cache to external on the performance page and your done.
Also back up your global.inc file as you will need to add it again should you wipe and rebuild aegir for whatever reason.

Do this for each site you have in pressflow platforms and your good to go with custom bits for each site.

We are using something like

omega8cc's picture

We are using something like that for one-for-all global configuration with http://drupal.org/project/cache, since it has cache handlers chaining (for example redis or memcache with auto-failover to standard db cache), and $thishost works great as 'prefix' => $thishost,.

~Grace

Is there some way I can check

qbnflaco's picture

Is there some way I can check and apply these to only the sites that are running on the mercury install profile?

This would make it much easier than having to hand code each url. I tried just putting it outside the if statements and seemed to mess up my non-mercury sites. :-( Some kind of if statement to affect only mercury sites would be great.

This if in your global.inc

omega8cc's picture

This if in your global.inc should work:

if(preg_match("/(mercury)/", $conf['install_profile'])) {

/* custom stuff here */

}

Here's what I ended up

qbnflaco's picture

Here's what I ended up using...

<?php # global settings.php

if(preg_match("/(mercury)/", $conf['install_profile'])) {

# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

$thishost = $_SERVER[HTTP_HOST];

# Memcached configuration - uncomment to use Memcached instead of Cacherouter:
$conf = array(
    'memcache_key_prefix' => $thishost,
    'cache_inc' => './sites/all/modules/memcache/memcache.db.inc',
   'memcache_servers' => array(
         '127.0.0.1:11211' => 'default',
         '127.0.0.1:11212' => 'menu',
         '127.0.0.1:11213' => 'filter',
         '127.0.0.1:11214' => 'form',
         '127.0.0.1:11215' => 'block',
         '127.0.0.1:11216' => 'update',
         '127.0.0.1:11217' => 'views',
         '127.0.0.1:11218' => 'content',
         '127.0.0.1:11219' => 'apachesolr',
       ),
   'memcache_bins' => array(
          'cache'        => 'default',
          'cache_menu'   => 'menu',
          'cache_filter' => 'filter',
          'cache_form'   => 'form',
          'cache_block'  => 'block',
          'cache_update' => 'update',
          'cache_views'  => 'views',
          'cache_views_data'  => 'views',
          'cache_content'  => 'content',
          'cache_apachesolr'  => 'apachesolr',
       ),
);
}

Difference between memcached and cacherouter

secoif's picture

Hm, I'm comparing macrocosm's .inc and yours, what's the difference here between the two configurations... memcached vs cacherouter?

The previous instructions

qbnflaco's picture

The previous instructions that are depreciated say to use the cacherouter module and memcached module. The current instructions no longer have cacherouter included.

nevermind

Macronomicus's picture

I fixed it above .. the filter was messing up the code...

global.inc shouldn't end with

omega8cc's picture

global.inc shouldn't end with ?>, of course.
In above example it is only to get an easy to read highlighted PHP code.
Just in case someone would try to copy it as-is.

Best,
~Grace

A few questions

iaminawe's picture

Hi,

I am really interested in doing what you mention about running aegir on a mercury stack.

If Aegir is running behind the Varnish proxy, does that mean that only sites provisioned on pressflow based drupal platforms will see the benefits of the mercury setup? Will normal core drupal based sites still work properly? Will they also experience any performance/capacity improvements?

Thanks for any answers
Gregg

I found that to be the case

Macronomicus's picture

I found that to be the case however the modules still seem to be picked up by APC .. so you will get the benefit of caching those. Aside from that ive found that amazon is quite fast on its own, especially with having Aegir & webroot on a raid0 and mysql on a raid0, I may even move mysql to its own instance if our real estate site becomes too intensive, that way the database can scale independently of the main machines scaling. Might even be nice to move varnish and cache off into its own higher ram instance too. But thats a bit too much money for me to get into now, im trying to keep things cheap and cheerful and prove the platform then expand later this year.

I dont see many reasons not to use pressflow platforms, for me at least; the only non-pressflow platform I have is the actual Aegir platform. I suppose there will be other platforms as soon as I find the time to move them into my new stuff .. like Open Atrium etc.

As far as varnish, its one aspect ive not explored deeply... yet. Im wondering if the Varnish bit I have added with my code above couldn't be more generalized on the global.inc to work for all sites, so placing it outside of the if domain preg_match goody (which I got from the rocking omega8cc!). Much study to go in the varnish arena, but what ive read so far it seems only Pressflow, D7 and a patched D6 will work with Varnish. http://drupal.org/project/varnish

I can tell you that things are blazing fast, though ive not run tests yet, just using it to build our real estate site. There are some things that ive found to still be slightly slow (large batch operations) but I think that is running up on some limiting aspects of php and how large the datasets ive been kicking around are ... I hear that pressflow may soon include ways to abstract out parts of core into java, which would it seems speed up some things considerably! ....exciting stuff to come!

I say go on and give it a shot .. it will probably only cost ya a few bucks to tinker around with it for a bit; I got things working in pretty short order even being an AWS virgin.

Cheers!

As I wrote above:

omega8cc's picture

As I wrote above: http://groups.drupal.org/node/35984#comment-117150 you can use this global cache setup for all sites, so without wrapping it inside if(), and to make it still cache-per-domain, you should use unique prefix - 'prefix' =>$thishost,.

~Grace

Aegir itself doesn't benefit

omega8cc's picture

Aegir itself doesn't benefit from Pressflow as core, and even can lead to some problems when you have more than one Aegir instance on the same VPS, so it is better to use just vanilla Drupal for Aegir platform (BTW: I recommend to not use Aegir platform for any non-Aegir site, just to keep things easier to manage and more secure).

But you can use Pressflow as a core for any non-Aegir platform. For example: almost all our hosted palettes http://omega8.cc/palettes are build on Pressflow core since September 2009 - Open Atrium, Managing News, Acquia, ProsePoint, OpenPublish and all Custom Color options.

So, unless you need cookies for anonymous visitors, you can use Pressflow as a core for any kind of Drupal distro/palette.

HTH,
~Grace

Yeah when I was setting this

Macronomicus's picture

Yeah when I was setting this up I figured that Aegir would be better as intended with the base drupal core, it runs perfectly and flys anyways .. I totally agree with your sentiments and I dont use the aegir platform for any site other than aegir .. havnt tested out multiple aegir's yet though that sounds very cool!

So Open Atrium can run on Pressflow and all those others too, sweet! I also didnt know about the anon cookies bit with pressflow but that makes sense when I think about it.

So you don't recommend Acquia on Aegir?

luiginica's picture

Hi

So you don't recommend Acquia on Aegir?
That's because I want use Aegir to manage hosting and Acquia as Drupal platform. I like the multilanguage feature and the fact that all the modules work together. I mention that I'm new to Drupal.

Regards,
Luigi Cristian Nica

Yeah of course you can do

Macronomicus's picture

Yeah of course you can do that .. what I meant is that I use a simple basic Drupal as the basis to build the Aeigr app itself. At this point though you dont even have to think about it as the Aegir install scripts pulls the drupal it needs and sets up everything for you.

You can have whatever platforms you like, above I was referencing the fact that I dont use Pressflow Drupal as the basis for the Aegir app. Pressflow is a version of Drupal .. unlike like Acquia which is a basic Drupal with a mash-up of various modules, Pressflow is a rewrite of Drupal that pulls out all the unneeded cruft created by Drupal attempting to support multiple versions (old) of PHP it uses real PHP5 and also ends support for multiple types of Databases, plus a lot of other performance tweaks, Pressflow makes Drupal fast & Mercury makes Drupal super fast!

Cheers!
Joshua

Very clear picture

luiginica's picture

Thanks.

Now it is very clear to me.

Useful info

iaminawe's picture

Thanks for the reply. I am actually running mercury on a linode at the moment (installed via the step by step) as its quite a bit cheaper than AWS for a production server. So far everything seems to be working great , other than the fact that I got my first Guru Meditation 503 service unavailable message

I will setup Aegir once I have had some time to test this Mercury setup properly

Thanks for the advice
Gregg

Another piece of the puzzle

Macronomicus's picture

I found another nearly required step in the process, at least it's required if you want more than one site to use Solr.
Apache Solr Multi-core Setup - Allot the different sites their own solr core.

EDIT:
Also look at the Mercury Install Wiki http://groups.drupal.org/pantheon/mercurywiki especially (Step 9) and see how the existing solr is setup in the AMI so you dont have overlap.

There is a wiki started now

Macronomicus's picture

There is a wiki started now and some basic install instructions too, maybe we can work out the kinks some more together and whatnot...

Main Mercury + Aegir Wiki page
http://groups.drupal.org/node/55033

Fresh Mercury + Aegir install instructions
http://groups.drupal.org/node/55393