Mercury 1.0 Released

Events happening in the community are now at Drupal community events on www.drupal.org.
joshk's picture

It's been nine months and over a thousand developer hours, but it's finally here: Project Mercury has gone 1.0. Thanks to everyone for making this possible, and to my colleague Greg for doing so much leg-work over the past month to get things ready.

Help us spread the word on the twitter and if you're so inclined, take a short user survey. Install instructions below the fold.

For our legions on Amazon, the AMIs are everywhere:

pantheon-amis-us-east/mercury-pressflow-drupal-32-1.0.manifest.xml      ami-9f9f70f6
pantheon-amis-us-east/mercury-pressflow-drupal-64-1.0.manifest.xml      ami-879f70ee

pantheon-amis-us-west/mercury-pressflow-drupal-64-1.0.manifest.xml      ami-e54c1da0
pantheon-amis-us-west/mercury-pressflow-drupal-32-1.0.manifest.xml      ami-e14c1da4

pantheon-amis-eu-west/mercury-pressflow-drupal-32-1.0.manifest.xml      ami-11b79c65
pantheon-amis-eu-west/mercury-pressflow-drupal-64-1.0.manifest.xml      ami-1db79c69

And for the bootstrappers: the wiki is up to date. Have at it!

We'll have much more to talk about and document in the coming weeks and months, but for now it's just a really great feeling to get this out the door. Thanks again to everyone for their enthusiasm and support.

Comments

Wooohooo!

kcoop's picture

This is great. I've been holding off on a major release pending this. Thanks for all the hard work.

I'm almost ready to put this into production, but keep wondering about the issue of persistence. Is there any more thought towards creating a boot EBS? I could create an EBS volume, and redirect www, solr, and mysql to it, but there are still logs, and any other installs/tweaks I might do.

Thanks again. You guys are da bomb!

ESB Yes

joshk's picture

Now that we've hit 1.0, we'll be doing the following:

  • Bootable EBS
  • VirtualBox
  • VMware image community
  • Linode stackscript
  • ???? (open to suggestions)

However, it will take a few days/weeks to line all those up. Stay tuned though!

Great!

kcoop's picture

I hesitated to even ask, but it seemed like something that would be a general issue for Amazon users. Is there a use case for a varnish-class drupal app that doesn't want persistent state?

But again, to be clear, this is a great effort you've mustered here!

re: Wooohooo!

Greg Coit's picture

kcoop,

You have several options:

One is to start with an EBS boot AMI - Alestic.com has a few listed (Karmic and Hardy only). Karmic should work with very little changes using the instructions we provide here: http://groups.drupal.org/node/50408.

The other option is to modify an AMI we provide - there are some very good tutorials out there on how to do so.

The third option is just push the important, non-replaceable stuff to an EBS - you seem to have a good grasp of the steps involved there.

I would like for us provide EBS AMIs (and we may someday), but there is so many other projects and so little time..... Woot - looks like it's on the docket! :)

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Can you elaborate on option 3

chules's picture

Greg, I already moved my database to an EBS Raid Volume using the how to at http://groups.drupal.rg/node/36750 which by the way is a very well written wiki page for Mercury users. What is the "important, non-replaceable stuff" that we should additional push to EBS whether it's the one created for the database or we need to create additional EBS. It would be a real benefit to the community if we could add to the node/36750 document the other important stuff. There was a README.txt included in the previous Mercury install where at the end it encouraged you to take the step in moving MySQL to EBS. Can we do something similar.

Thank you

chules

Wiki Install vs. AMI

R-H's picture

Just wondering what the diff is between installing via the Wiki instructions (very thorough and helpful guys!) vs. using the AMI. AMI is obviously much easier. Does the AMI offer same functionality as the the wiki install: BCFG2, etc.?

Cheers and thanks kindly! Looking forward to saying hi in person at SXSW if anyone is game.

Ryan

re: Wiki Install vs. AMI

Greg Coit's picture

NyRy,

We make the AMIs from the BCFG2 version of the Mercury install wiki: http://groups.drupal.org/node/50408 so they are almost exact. The only change we make with the AMIs we release is we tell BCFG2 that the hostname is localhost so that the hostname can change (as it does when someone boots an AMI) and it will still work without be re-configured.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Very Cool

R-H's picture

After booting up from your AMI, is there any BCFG2 setup that we need to do? I haven't fully research BCFG2 yet, but my idea is that it allows my instance to grab Mercury configuration from your config profile? Is that correct? I take it that BCFG2 allows for my instance to auto-update in a sense without having to boot up a new instance when you release a new Mercury AMI, v 1.1 or something. Correct?

Thanks

re: Very Cool

Greg Coit's picture

NyRy,

There's no setup needed for BCFG2 using our AMIs - it's already configured. If you want to make changes to your system and want BCFG2 to manage those changes, then you can edit the BCFG2 settings (how to do this is out of scope here but there are some great BCFG2 docs at http://trac.mcs.anl.gov/projects/bcfg2 and some very helpful folks on #bcfg2.

You can update your Mercury server by running /usr/local/bin/update_mercury.sh - it will look for any updates on our launchpad project (launchpad.com/pantheon) and apply them. You can also update pressflow by running /usr/local/bin/update_pressflow.sh.

A version of both scripts runs at first boot to make sure that your system gets any bugfixes, after that they have to be run manually.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

bcfg2 vs others

bejames's picture

Hi Greg,

Wondering if you would mind sharing your reasons for selecting bcfg2 vs cfengine or puppet etc. I'm looking at these tools now and I'm interested in understanding some of the pro/cons of each.

cheers
James

The honest answer

joshk's picture

The honest answer is we got bcfg2 working quickly thanks to some help from David Strauss, and once we had it working we liked it and didn't see a reason to change. There's also a great community in IRC for the project. So the answer is really the people around the project.

From a technical standpoint it has a good diagnostic mode, and doesn't try to stomp all over everything, both of which fit our use-case of people tuning their configurations further, and possibly needing support. I like it's notion of services. I like that it's XML-based rather than language-object based. I also have access to more and better resources that can speak python vs. ruby or java.

As for the competition, I don't see anything vastly superior vs Puppet or Chef, and Cfengine is too dictatorial and complex.

Basically, all these seem to be maturing and useful tools. Pick the poison that works for you!

Awesome!

OpenChimp's picture

This project is moving along really fast. Congratulations on the 1.0 release. And a big thanks! I'm sure there is going to be some very exciting talk about this at DrupalCon.

I can see why it may not be appropriate for Mercury, but I'm curious if there has been any thought of including VirtualMin in another flavor of Pantheon. ??

VirtualMin

joshk's picture

We've gotten lots of requests for some kind of cPanel, so that's definitely in the mix of things we'll need to consider. Do you have good experience with VirtualMin?

Ive used webmin which is nice

Macronomicus's picture

Ive used webmin which is nice .. but virtualmin looks like the latest edition of webmin and they seem to hold back some features for the paid version; Im not entirely sure what the diff is between them. I used to use it a lot but found just installing phpmyadmin and getting used to the command line was a lot better for me at least. Still Im going to take the new viertualmin for a spin and see how much they've changed it. I used the demo but im not sure how many of those features are in the gpl version.

IspCP

marcus_w's picture

Hey,

I just stumbled upon IspCp (http://isp-control.net/), but if I want to use it, it messes with the apache vhosts. It seems like a nice system for the Mercury servers though, with e-mail en data traffic and user control panel funtionalities.

I haven't tried it myself, but did anyone get a Mercury (multisite) working with IspCp? If so, how?

Would be nice to see some cPanel integrated into mercury.

Thanks for the great work on mercury by the way! Really love it.

Mark

Nice work, I've fired up an

atomicjeep's picture

Nice work, I've fired up an AMI instance but it cant seem to find the solr instance? I've restarted tomcat but still no luck, any ideas? Displays the following message:

Apache Solr: Your site was unable to contact the Apache Solr server.

thanks
paul

re: Nice work, I've fired up an

Greg Coit's picture

Paul,

Is it possible you didn't choose the Mercury profile when you installed Pressflow? The default Pressflow profile looks for solr on 8080, but we run it on port 8180 since apache is using 8080.

If that's not it, what AMI ID did you use?

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Thanks Greg, You were right

atomicjeep's picture

Thanks Greg,
You were right on the mark, all up and running now :)

Great stuff, thanks again
paul

Paul, Is it possible you

AntiNSA's picture

Paul,

Is it possible you didn't choose the Mercury profile when you installed Pressflow? The default Pressflow profile looks for solr on 8080, but we run it on port 8180 since apache is using 8080.

Im using mercury 1.1 beta, I am having the same error. Are there other possible implications if I selected the wrong profile when installing that I now need to fix? Are there any other ways to fis this and select the corret profile?

Thanks!

re: Paul, Is it possible you

Greg Coit's picture

Yes, there are several implications. The Mercury profile configures solr, varnish, memcached and caching options for best performance.

You can check mercury.profile to see what's set.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

This is awesome! What is the

j0k3z's picture

This is awesome!

What is the best way to move a non-pressflow drupal website to Mercury & Pressflow?

Also, is it worth waiting for a bootable ebs or are we still a few weeks away from those?

kcoop's picture
  1. Get the latest version of pressflow
    wget http://launchpad.net/pressflow/6.x/6.16.77/+download/pressflow-6.16.77.t...
    tar xvfz pressflow-6.16.77.tar.gz
    mv /var/www /var/www.mercury
    mv pressflow-6.16.77 /var/www
  2. If you've made any changes to core, you'll need to patch the pressflow version.
  3. Copy any modules you've added (sites/all/modules/*) to pressflow. Alternatively, if you're under source control, create a symbolic link.
  4. Install the following modules, if you don't have them already: varnish, memcache, and apachesolr.
  5. Add the following to the bottom of sites/default/settings.php (stripping out the php delimiters):
    <?php
    # Varnish reverse proxy on localhost
    $conf['reverse_proxy'] = TRUE;
    $conf['reverse_proxy_addresses'] = array('127.0.0.1');

    # Memcached configuration
    $conf = array(
      
    'cache_inc' => './sites/all/modules/memcache/memcache.inc',
      
    'memcache_servers' => array(
            
    '127.0.0.1:11211' => 'default',
            
    '127.0.0.1:11212' => 'menu',
            
    '127.0.0.1:11213' => 'filter',
            
    '127.0.0.1:11214' => 'form',
            
    '127.0.0.1:11215' => 'block',
            
    '127.0.0.1:11216' => 'update',
            
    '127.0.0.1:11217' => 'views',
            
    '127.0.0.1:11218' => 'content',
            
    '127.0.0.1:11219' => 'apachesolr',
           ),
      
    'memcache_bins' => array(
             
    'cache'        => 'default',
             
    'cache_menu'   => 'menu',
             
    'cache_filter' => 'filter',
             
    'cache_form'   => 'form',
             
    'cache_block'  => 'block',
             
    'cache_update' => 'update',
             
    'cache_views'  => 'views',
             
    'cache_views_data'  => 'views',
             
    'cache_content'  => 'content',
             
    'cache_apachesolr'  => 'apachesolr',
           ),
    );
    ?>
  6. Create a user, a database, and password protect root
    mysql -u root
    mysql> set password for root@localhost=PASSWORD('YOURROOTPASSWORD');
    mysql> create database YOURDBNAME;
    mysql> grant all on YOURDBNAME.* to YOURUSERNAME@localhost identified by 'YOURUSERPASSWORD';
    mysql> flush privileges;
    mysql> \q
  7. Dump and copy your existing database wherever it is (you'll need to get your .pem file onto your other host)
    mysqldump -u YOURUSERNAME -p YOURDBNAME > sqldump.sql
    tar cfvz sqldump.tar.tgz sqldump.tar
    scp -i YOURKEY.pem sqldump.tar.tgz root@YOURPUBLICDNS:/root/sqldump.tar.tgz
    ## on the new server...
    cd /root
    tar xvfz sqldump.tar.tgz
  8. Optionally, convert your tables to INNODB
    echo 'SET AUTOCOMMIT=0;' > sqldump.inno.sql
    sed 's/MyISAM/INNODB/g' sqldump.sql sqldump.inno.sql
    echo 'COMMIT;' >> sqldump.inno.sql
  9. Restore your database on the new server
    mysql YOURDBNAME -u YOURUSERNAME -p < ~/sqldump.inno.sql
  10. Finally, go to the site, and do the following:

  11. Run update.php
  12. Enable modules solr, memcache, varnish
  13. Configure SOLR. Host name: localhost, Solr Port: 8180

That got my site running.

Awesome! Thanks

8ballsaysiwin's picture

Awesome! Thanks

Couple more little things

kcoop's picture

I neglected to mention copying sites/all/themes, they should be there too of course.

Also, I found that at least for my purposes, 128M was not enough java heap space for SOLR to manage my (fairly large) index. I upped it to 256M in /etc/init.d/tomcat6.

There may be more, as I'm working the kinks out now...

Moving additional items

chules's picture

Kcoop,

I am using a large Mercury instance and moved the database over to an EBS RAID volume. What else have you moved? You mentioned copying sites/all/themes. For example, did you create another EBS Volume for the document root? I am looking for additional instructions for AMI installations. The next steps after you move your database?

Thank you -chules

I haven't dealt with EBS yet

kcoop's picture

I'm hoping to use an EBS boot AMI, as I'd like to add other components to the server and not have to consider what is persisted and what isn't in the event of failure. If you want to go that route, this thread may be useful: http://groups.drupal.org/node/38188

Greg Coit's picture

A quick note. If using Mercury 1.0, one can update pressflow using the script /usr/local/bin/update_pressflow.sh which does the following:

cd /var/www/; bzr merge --force;

It also commits local changes so the bzr merge doesn't fail.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Copied an error

kcoop's picture

The code in step #5 above should read like so:

<?php

# Memcached configuration
$conf = array(
  
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
  
'memcache_servers' => array(
        
'127.0.0.1:11211' => 'default',
        
'127.0.0.1:11212' => 'menu',
        
'127.0.0.1:11213' => 'filter',
        
'127.0.0.1:11214' => 'form',
        
'127.0.0.1:11215' => 'block',
        
'127.0.0.1:11216' => 'update',
        
'127.0.0.1:11217' => 'views',
        
'127.0.0.1:11218' => 'content',
        
'127.0.0.1:11219' => 'apachesolr',
       ),
  
'memcache_bins' => array(
         
'cache'        => 'default',
         
'cache_menu'   => 'menu',
         
'cache_filter' => 'filter',
         
'cache_form'   => 'form',
         
'cache_block'  => 'block',
         
'cache_update' => 'update',
         
'cache_views'  => 'views',
         
'cache_views_data'  => 'views',
         
'cache_content'  => 'content',
         
'cache_apachesolr'  => 'apachesolr',
       ),
);

# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
?>

Thanks, toemaz, for catching that!

I've been trying to find the

choloepus's picture

I've been trying to find the answer to my question w/o any luck.

What is the reasoning behind setting up 1 memcached server for each bin?
I know that there is a 2gb limit on the size of the memcached instance, but there is no way that most of the cache bin ever reach that.

Totally awesome!

Macronomicus's picture

I was able to move into the new AMI quite nicely, and its great to see the bcfg2 config all together and working! I will have to study that though to be sure I can make/keep my modifications in addition to being able to update too. More than that though the Memcached is totally awesome! ...I like that there is a solr cache now too very smart.

Only big thing left for me to figure is to fully make Mercury capable of handling drupal Multi-sites. All pieces seem to do just that but Solr, as it stands If you try you will get cross-pollution from all sites that enable solr and none will have a functional search of just their own content. Only one of my sites uses solr but im guessing there are a lot of folks that use multi-sites so we should flesh this out sometime.

Cheers!
Joshua

Is there a tutorial for

Flying Drupalist's picture

Is there a tutorial for adding settings to BCFG2? For example PECL uploadprogress. Is the idea that, you set this once and every time the AMI is launched the PECL uploadprogress would auto install without too much work?

Thanks.

re: Is there a tutorial for

Greg Coit's picture

BCFG2 has a good quick start at http://trac.mcs.anl.gov/projects/bcfg2/wiki/QuickStart and very detailed descriptions of all it's modules and functions at http://trac.mcs.anl.gov/projects/bcfg2/wiki/TitleIndex.

I'm not aware of a plugin for BCFG2 that handles PECL installs. What BCFG2 does do is manage packages using the OS and distro specific package manager, plus config files. For instance, one you get a server setup exactly the way you want (using BCFG2), and it comes time to make another just like it, simply copy over the BCFG2 files, install BCFG2, and it will install all the needed packages and config files.

For our purposes, it simplifies the install instructions a great deal, and allows us to make a new, updated AMI very easily. It should also ease the process of porting Mercury to other OSs since BCFG2 handles the package-manager details.

If you were setting up a lot of servers, you could script the PECL install and add it to the BCFG2s config so that it runs the first time a new server boots (see how we setup /etc/mercury/init.sh for an example - this is stored in /var/lib/bcfg2/CFG/etc/mercury/init.sh/).

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Upgrading

pajoh's picture

I have an older beta instance running. Is it possible to upgrade without starting over with a whole new instance? Are there applicable instructions anywhere?

Thanks,
-Paul

re: Upgrading

Greg Coit's picture

Paul,

There are no upgrade instructions for any of the Beta releases, but I'm certain there will be from 1.0 forward. Unfortunately, there were some significant changes/updates right up to the release of 1.0 which makes upgrading difficult and probably more difficult than migrating to a new server.

Migration from/to mercury instances would be no different than with any other drupal server migration and there are lots of good instructions online that are better than what I could write.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Thanks Greg. I've got a new

pajoh's picture

Thanks Greg. I've got a new AMI up and running... drupal is reporting apachesolr and varnish are already in need of updates... should I manually update like any other module?

re: Thanks Greg. I've got a new

Greg Coit's picture

pajoh,

Yes, you can update the apachesolr and varnish modules using either pressflow or drush.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Karmic Install

iaminawe's picture

Hi,

I understand that the wiki instructions are for Jaunty and should work for Karmic (with a few changes)
I have tried to follow the wiki on Karmic and so far have run into the following

When I run step 2
apt-get -y install bzr bcfg2-server gamin python-gamin

I get the following error

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
bcfg2-server: Depends: bcfg2 (= 1.0.1-1~ppa1~jaunty1) but it is not going to be installed
E: Broken packages

If I try this:

apt-get -y install bcfg2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
bcfg2: Depends: debsums but it is not installable
E: Broken packages

Does this mean that BCFG cannot yet be installed on Ubuntu 9.10?

Any help would be appreciated

Thanks
Gregg

re: Karmic Install

Greg Coit's picture

Gregg,

The apt-sources we provided for bcfg2 and bzr are jaunty-specific. In the instructions (http://groups.drupal.org/node/50408) where we say:

echo "deb http://ppa.launchpad.net/bzr/ppa/ubuntu jaunty main" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bzr/ppa/ubuntu jaunty main" >> /etc/apt/sources.list
echo "deb http://ppa.launchpad.net/bcfg2/ppa/ubuntu jaunty main" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bcfg2/ppa/ubuntu jaunty main" >> /etc/apt/sources.list

try this instead:

echo "deb http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main" >> /etc/apt/sources.list
echo "deb http://ppa.launchpad.net/bcfg2/ppa/ubuntu karmic main" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bcfg2/ppa/ubuntu karmic main" >> /etc/apt/sources.list

You can also edit /etc/apt/sources.list and replace all instances of "jaunty" with "karmic" if you've already added the Jaunty sources.

Hope this helps and please tell us how the install goes,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Oops my mistake

iaminawe's picture

I needed to change each of the "jaunty" references in step 1 to "karmic"

BCFG Config and Karmic

iaminawe's picture

Wow that was quick thanks. I just worked it out and posted a reply and you had already given the answer.

The install has gone fine up to the point where I need to set the config files up for BCFG. I think it may be because the Karmic config does not exist in Launchpad yet so maybe I am trying to reference something that is not there but step by step this is where I am stuck

Step 6: I edit groups.xml to look like so

Try again

iaminawe's picture

Damn I somehow lost half of my post ...trying again
Step 6: I edit groups.xml to look like so

Step 7 clients.xml to look like this
Clients version="3.0">

Then when I try start the server the first time I get
Starting Configuration Management Server: * bcfg2-server
Mar 9 18:06:59 localhost bcfg2-server[2788]: Failed to load /var/lib/bcfg2/Pkgmgr indices
Mar 9 18:06:59 localhost bcfg2-server[2788]: Failed to instantiate plugin Pkgmgr
Mar 9 18:06:59 localhost bcfg2-server[2788]: Failed to load /var/lib/bcfg2/Rules indices
Mar 9 18:06:59 localhost bcfg2-server[2788]: Failed to instantiate plugin Rules
Mar 9 18:06:59 localhost bcfg2-server[2788]: Failed to load SSHbase repository from /var/lib/bcfg2/SSHbase
Mar 9 18:15:16 localhost bcfg2-server[2817]: Packages: Updating http://us.archive.ubuntu.com/ubuntu/dists/jaunty/restricted/binary-i386/...
Mar 9 18:15:16 localhost bcfg2-server[2817]: Packages: Updating http://us.archive.ubuntu.com/ubuntu/dists/jaunty/universe/binary-amd64/P...
Mar 9 18:15:16 localhost bcfg2-server[2817]: Packages: Updating http://us.archive.ubuntu.com/ubuntu/dists/jaunty/universe/binary-i386/Pa...
Mar 9 18:15:32 localhost bcfg2-server[2817]: Failed to read file probed.xml
Mar 9 18:15:32 localhost bcfg2-server[2817]: Loading experimental plugin(s): Packages
Mar 9 18:15:32 localhost bcfg2-server[2817]: NOTE: Interfaces subject to change
Mar 9 18:15:32 localhost bcfg2-server[2817]: service available at https://localhost:6789
Mar 9 18:15:32 localhost bcfg2-server[2817]: serving bcfg2-server at https://localhost:6789
Mar 9 18:15:32 localhost bcfg2-server[2817]: serve_forever() [start]
Mar 9 18:15:32 localhost bcfg2-server[2817]: Handled 33 events in 0.015s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 32 events in 0.005s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 86 events in 0.023s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 67 events in 0.020s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 38 events in 0.014s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 7 events in 0.003s

The next time I try start it it fails and I get this
Starting Configuration Management Server: * bcfg2-server
Mar 9 18:15:32 localhost bcfg2-server[2817]: service available at https://localhost:6789
Mar 9 18:15:32 localhost bcfg2-server[2817]: serving bcfg2-server at https://localhost:6789
Mar 9 18:15:32 localhost bcfg2-server[2817]: serve_forever() [start]
Mar 9 18:15:32 localhost bcfg2-server[2817]: Handled 33 events in 0.015s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 32 events in 0.005s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 86 events in 0.023s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 67 events in 0.020s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 38 events in 0.014s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 7 events in 0.003s
Mar 9 18:17:02 localhost CRON[2827]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 9 18:17:05 localhost bcfg2-server[2835]: Failed to read file probed.xml
Mar 9 18:17:05 localhost bcfg2-server[2835]: Loading experimental plugin(s): Packages
Mar 9 18:17:05 localhost bcfg2-server[2835]: NOTE: Interfaces subject to change
Mar 9 18:17:05 localhost bcfg2-server[2835]: Failed to bind to socket
Mar 9 18:17:05 localhost bcfg2-server[2835]: Server startup failed

I am totally new to BCFG and pretty new to Linux so would appreciate any ideas or guidance

Thanks
Gregg

re: Try again

Greg Coit's picture

the output from your first attempt may look like errors, but they're not really (except for maybe the Rules one - see below). When you see:

Mar 9 18:15:32 localhost bcfg2-server[2817]: Handled 33 events in 0.015s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 32 events in 0.005s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 86 events in 0.023s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 67 events in 0.020s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 38 events in 0.014s
Mar 9 18:15:33 localhost bcfg2-server[2817]: Handled 7 events in 0.003s

Then you know the server is up and ready for you to run the client (bcfg2 -vq).

The second attempt is failing due to a socket error - most likely because the server is already running? Remember that the instructions group the /etc/init.d/bcfg2-server start and tail -f /var/log/syslog commands together - when you control-c - you are canceling the tail command, not the bcfg2-server command.

If you want to check the validity of your config files, run:

bcfg2-repo-validate

Note that you will get (and can ignore) the following errors because we are using attributes of the element BoundPath that are available in bcfg2 but haven't made it to bcfg2-repo-validate yet:

/var/lib/bcfg2/Bundler/amazon-web-services.xml FAILS to verify            <----
/var/lib/bcfg2/Bundler/amazon-web-services.xml:15: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'type': The attribute 'type' is not allowed.
/var/lib/bcfg2/Bundler/amazon-web-services.xml:15: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'owner': The attribute 'owner' is not allowed.
/var/lib/bcfg2/Bundler/amazon-web-services.xml:15: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'group': The attribute 'group' is not allowed.
/var/lib/bcfg2/Bundler/amazon-web-services.xml:15: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'perms': The attribute 'perms' is not allowed.
/var/lib/bcfg2/Bundler/amazon-web-services.xml fails to validate
/var/lib/bcfg2/Bundler/memcached.xml FAILS to verify              <----
/var/lib/bcfg2/Bundler/memcached.xml:16: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'type': The attribute 'type' is not allowed.
/var/lib/bcfg2/Bundler/memcached.xml fails to validate
/var/lib/bcfg2/Bundler/varnish.xml FAILS to verify                <----
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'type': The attribute 'type' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'owner': The attribute 'owner' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'group': The attribute 'group' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'perms': The attribute 'perms' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml fails to validate

Finally, to post an xml files to g.d.o, you might try placing them within <code></code> tags.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Client seems to run fine

iaminawe's picture

OK so I couldn't help myself and needed to try it before going to sleep and as far as I can see it worked !!! Thanks for your help.

I did notice the following "Failed" messages that may or may not be of concern.

The Following Bundles have been modified:
cacti-client memcached mercury postfix tomcat6 varnish

UID normalization failed for /var/lib/varnish/pressflow
UID normalization failed for /var/lib/varnish/pressflow
GID normalization failed for /var/lib/varnish/pressflow
Installing Service apache2
Installing Service mysql
Installing Service mysql-ndb
Installing Service mysql-ndb-mgm
Installing Service varnish
Installing Service varnishlog
Installing Service varnishncsa
Installing Service postfix
Installing Service tomcat6
Installing Service memcached
Installing Service snmpd
Installing Symlink /etc/apache2/mods-enabled/expires.load
Installing Symlink /etc/apache2/mods-enabled/rewrite.load
Installing Symlink /etc/apache2/sites-enabled/000-default
Installing Directory /var/lib/varnish/pressflow
UID normalization failed for /var/lib/varnish/pressflow
GID normalization failed for /var/lib/varnish/pressflow
Failed to manipulate service apache2
Failed to manipulate service mysql
Failed to manipulate service mysql-ndb
Failed to manipulate service mysql-ndb-mgm
Failed to manipulate service varnish
Failed to manipulate service varnishlog
Failed to manipulate service varnishncsa
Failed to manipulate service postfix
Failed to manipulate service tomcat6
Failed to manipulate service snmpd

Will pick it up in the morning and test the server properly but I just wanted to check if there are any red flags that I should look out for because of the fails or if they are part of the process and the end system stability would not be affected.

Thanks again, really stoked to see this working... the possibilities are awesome.

re: Client seems to run fine

Greg Coit's picture

iaminawe,

I've not the UID/GID errors before - sounds like BCFG2 is unable to set the UID/GID of /var/lib/varnish/pressflow as defined in /var/lib/bcfg2/Bundle/varnish.xml:

<BoundPath name='/var/lib/varnish/pressflow' type='directory' owner='varnish' group='varnish' perms='0755'/>

Are you running bcfg2 as root? If not, that would explain the UID/GID and "Failed to manipulate service" errrors.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Root

iaminawe's picture

I am running bcfg2 as root. By this point I have not installed Pressflow yet. Could this be the issue?
Or could it be related to the earlier errors I received when starting up BCFG server?

/var/lib/bcfg2/Bundler/varnish.xml FAILS to verify <----
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'type': The attribute 'type' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'owner': The attribute 'owner' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'group': The attribute 'group' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'perms': The attribute 'perms' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml fails to validate

If I need to manually go in and fix things, will this affect the auto updates using the shell script?

re: Root

Greg Coit's picture

We use the same instructions to create our images as you are using: http://groups.drupal.org/node/50408 so the order should be correct. In fact, BCFG2 should be installing varnish for you.

The following errors are more of bcfg2-repo-validate not being quite as up to date as the BCFG2 client And are not a concern:

/var/lib/bcfg2/Bundler/varnish.xml FAILS to verify <----
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'type': The attribute 'type' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'owner': The attribute 'owner' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'group': The attribute 'group' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml:8: element BoundPath: Schemas validity error : Element 'BoundPath', attribute 'perms': The attribute 'perms' is not allowed.
/var/lib/bcfg2/Bundler/varnish.xml fails to validate

But you shouldn't be getting all the "failed to manipulate" errors (with the exception of apache which bcfg2 is unable to restart for some reason).

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

I also received a ton of

pcave's picture

I also received a ton of "failed to manipulate" errors while following the wiki instructions. I was trying to install on Ubuntu 9.04 on a rackspace cloud server instance. One thing that is confusing in the wiki are the changes to the groups.xml file. I don't know if I didn't configure that correctly or what, but I ended up giving up after step 9 (bcfg2 -vq) because no packages (mysql, apache, etc) were installed.

I'm trying to install on

pcave's picture

I'm trying to install on jaunty as well, what did your groups.xml and clients.xml end up looking like?

I cant seem to post my xml data here

iaminawe's picture

I would like some guidance on how to config groups.xml and clients.xml for Ubuntu 9.10 (Karmic)

My current config is causing the errors posted above and I am not sure why. I am trying to set this up on a Linode server.

Cant wait for the stackscript :)

Thanks Greg

iaminawe's picture

I will continue with this in the morning but I appreciate your help and am stoked to know that it actually is working.

I did try placing the xml in the tags as well but they still did not render in the message.

Took a stab at Karmic with BCFG2

kcoop's picture

After a soft reset this morning, I've lost faith in Jaunty. I also want to get EBS boot on a 32 bit instance.

I followed the wiki's instructions, substituting karmic where I saw jaunty, then did the following massaging before I fired up the bcfg2 server and client:

  1. Made a copy of /var/lib/bcfg2/Bundler/ubuntu-jaunty.xml, named it ubuntu-karmic, did /jaunty/karmic/ on it
  2. Did /jaunty/karmic/ on /var/lib/bcfg2/etc/apt/sources.list/sources.list
  3. Did /jaunty/karmic/ on /var/lib/bcfg2/Packages/config.xml

This got me part of the way there, but the big issue appears to be missing libraries. After adding alestic's ppa, I'm still missing some. I looked into it a little, and it appears some of the libraries are no longer valid for karmic (like libisccc40, upstart-compat-sysv, and upstart-logd). I'm in the dark on what is critical and what can be substituted/jettisoned. Here's the output from bcfg2 -vq:

sudo bcfg2 -vq
Running probe groups
Probe groups has result:
i386
Loaded tool drivers:
APT      Action   DebInit  POSIX  
Package libltdl7-dev not installed
Package ec2-ami-tools not installed
Package libdns46 not installed
Package librpm4.4 not installed
Package libbeecrypt6 not installed
Package libbind9-40 not installed
Package libdatrie0 not installed
Package libdirectfb-1.0-0 not installed
Package libisc45 not installed
Package libisccc40 not installed
Package libisccfg40 not installed
Package liblwres40 not installed
Package libntfs-3g49 not installed
Package libvolume-id1 not installed
Package startup-tasks not installed
Package system-services not installed
Package upstart-compat-sysv not installed
Package upstart-logd not installed
Package libevent1 not installed

Phase: initial
Correct entries: 584
Incorrect entries:  23
Total managed entries:   607
Unmanaged entries:  123

Installing Service ec2-run-user-data
Installing Service bootlogs.sh
Installing Service ec2-get-credentials
Installing Service rmnologin
APT has no information about package liblwres40
APT has no information about package upstart-compat-sysv
APT has no information about package upstart-logd
APT has no information about package libbeecrypt6
APT has no information about package libdatrie0
APT has no information about package libdns46
APT has no information about package startup-tasks
APT has no information about package libisccfg40
APT has no information about package libbind9-40
APT has no information about package librpm4.4
APT has no information about package libltdl7-dev
APT has no information about package libisccc40
APT has no information about package libntfs-3g49
APT has no information about package libevent1
APT has no information about package libdirectfb-1.0-0
APT has no information about package libisc45
APT has no information about package ec2-ami-tools
APT has no information about package system-services
APT has no information about package libvolume-id1
The Following Bundles have been modified:
amazon-web-services  ubuntu-karmic      

Package ec2-ami-tools not installed
Package libdns46 not installed
Package librpm4.4 not installed
Package libbeecrypt6 not installed
Package libbind9-40 not installed
Package libdatrie0 not installed
Package libdirectfb-1.0-0 not installed
Package libisc45 not installed
Package libisccc40 not installed
Package libisccfg40 not installed
Package liblwres40 not installed
Package libntfs-3g49 not installed
Package libvolume-id1 not installed
Package startup-tasks not installed
Package system-services not installed
Package upstart-compat-sysv not installed
Package upstart-logd not installed
Installing Service ec2-get-credentials
Installing Service ec2-run-user-data
Installing Service bootlogs.sh
Installing Service rmnologin
APT has no information about package ec2-ami-tools
APT has no information about package libdns46
APT has no information about package librpm4.4
APT has no information about package libbeecrypt6
APT has no information about package libbind9-40
APT has no information about package libdatrie0
APT has no information about package libdirectfb-1.0-0
APT has no information about package libisc45
APT has no information about package libisccc40
APT has no information about package libisccfg40
APT has no information about package liblwres40
APT has no information about package libntfs-3g49
APT has no information about package libvolume-id1
APT has no information about package startup-tasks
APT has no information about package system-services
APT has no information about package upstart-compat-sysv
APT has no information about package upstart-logd
Failed to manipulate service ec2-get-credentials
Failed to manipulate service ec2-run-user-data
Failed to manipulate service bootlogs.sh
Failed to manipulate service killprocs
Failed to manipulate service ondemand
Failed to manipulate service rc.local
Failed to manipulate service rmnologin
Failed to manipulate service single

Phase: final
Correct entries:   584
Incorrect entries:  23
Package:ec2-ami-tools        Package:libisccc40           Package:system-services    
Package:libbeecrypt6         Package:libisccfg40          Package:upstart-compat-sysv
Package:libbind9-40          Package:libltdl7-dev         Package:upstart-logd       
Package:libdatrie0           Package:liblwres40           Service:bootlogs.sh        
Package:libdirectfb-1.0-0    Package:libntfs-3g49         Service:ec2-get-credentials
Package:libdns46             Package:librpm4.4            Service:ec2-run-user-data  
Package:libevent1            Package:libvolume-id1        Service:rmnologin          
Package:libisc45             Package:startup-tasks      
Total managed entries:    607
Unmanaged entries:  123

Anyone else been down this road?

Karmic versus Juanty Packages

jamesJonas's picture

I tried the same install with Karmic, stopping (like you) when I saw the large number of packages that where not installed.

Several of the packages are misnamed. For instance.
libbind9-40 is for Juanty
libbind9-50 is for Karmic

others: libbeecrypt7, libdatrie1...

While others have the correct (generic) name, but still failed to install. For example "ec2-ami-tools".

See: http://www.ubuntuupdates.org/package_metas

The issue seems to be the launchpad bcfg2 files.

The ubuntu-jaunty.xml file contains the packages specific names. Example

See: http://bazaar.launchpad.net/%7Epantheon-developers/pantheon/trunk/files/...

As I'm not a bcfg2 expert, so this may be a bit of a reach, but it seems that we need to create a Karmic xml file [ubuntu-karmic.xml], with the updated packages names, which we then reference during our install.

Hope this helps.

Thanks for that ubuntuupdates.org link

kcoop's picture

With it I was able to update all the bcfg2 package references in /var/lib/bcfg2/Bundler (there were multiple files affected), and I now have a working (varnish and solr confirmed) karmic server, with EBS boot and a second EBS drive with my database on it for faster snapshots.

I would love to see this integrated into the mercury config, so if there's anything I can do to help...

Sweet

Macronomicus's picture

I did notice a lot of depreciated packages when I tried to run bcfg2 on karmic, I also had trouble with solr but that was likely user error lol.

You could always post up your tweaked bcfg2 configs in here somewhere.. or maybe the pantheon folks will let you add another branch to the repository for karmic bcfg2. Im so tempted to do what you just did, I would much rather have the ebs bootable ami for all is benefits.

Cheers!

re: Sweet

Greg Coit's picture

I don't think we need a new branch. I would just create a /var/bcfg2/Bundle/ubuntu-karmic.xml file within the current branch (the, if someone wanted to use it, just replace ubuntu-jaunty with ubuntu-karmic in /var/bcfg2/Metadata/groups.xml). If someone has a one they would like me to add, please do send it to me. Officially, it looks like we're going to support Lucid when it's released (5 weeks now), but I don't see a reason why karmic can't be unofficially supported.

Thanks!

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Great!

kcoop's picture

I'll email you, Greg.

Karmic files uploaded to launchpad

Greg Coit's picture

Kcoop's files have been uploaded to launchpad. I still need to clean things up a bit (329 unmanaged files) but this shouldn't effect functionality. I tried to do this in a way that wouldn't impact jaunty installs but please let me know if you see anything strange with either the karmic or jaunty versions.

I'll update the instructions (http://groups.drupal.org/node/50408) now.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Thanks Greg!

kcoop's picture

I'll give it a whirl.

I don't see the karmic versions of some files in Bundler

kcoop's picture

amazon.web-services.xml, mercury.xml, and memcached.xml, to be specific. bcfg client fails for me because of this.

EDIT: I see you did make some changes to these. Maybe this is pilot error. I'll investigate some more.

BTW, here's the output from bcfg2 I'm wondering about:

APT has no information about package ec2-ami-tools
APT has no information about package libvolume-id1
APT has no information about package libevent1
APT has no information about package libltdl7-dev

Thanks for the Karmic xml file in Bundle

jamesJonas's picture

Here is my untested map from Juanty to Karmic for these items.

libevent1 > libevent-1.4-2
libdns46 > libdns53 (may have used another dns package)
libltdl7-dev > libltdl7

libvolume-id1 - this package looks like it should work with Karmic. May need more recent repository.

ec2-ami-tools - Unclear what this issue is with this package. This shows as being Karmic compatible. May need to reference a more recent repository.

For anyone wanting to try this

kcoop's picture

Here are the other files I replaced in /var/lib/bcfg2/Bundler (and don't forget the multiverse on the repos in the original /etc/apt/sources).

amazon.web-services.xml

<Bundle name='amazon-web-services'>
  <Package name='alien'/>
  <Package name='dpkg-dev'/>
  <Package name='debhelper'/>
  <Package name='ec2-ami-tools'/>
  <Package name='ec2-consistent-snapshot'/>
  <Package name='libdns53'/>
  <Package name='librpm0'/>
  <Package name='make'/>
  <Package name='rpm'/>
  <Package name='runurl'/>
  <Path name="/usr/local/bin/ec2-metadata"/>
  <Service name='ec2-get-credentials'/>
  <Service name='ec2-run-user-data'/>
  <BoundPath name='/mnt/mysql/tmp' type='directory' owner='mysql' group='mysql' perms='0755'/>
</Bundle>

memcached.xml
<Bundle name='memcached'>
  <Package name='memcached'/>
  <Package name='libevent-1.4-2'/>
  <Package name='php5-memcache'/>
  <Path name="/etc/memcached_1.conf"/>
  <Path name="/etc/memcached_2.conf"/>
  <Path name="/etc/memcached_3.conf"/>
  <Path name="/etc/memcached_4.conf"/>
  <Path name="/etc/memcached_5.conf"/>
  <Path name="/etc/memcached_6.conf"/>
  <Path name="/etc/memcached_7.conf"/>
  <Path name="/etc/memcached_8.conf"/>
  <Path name="/etc/memcached_9.conf"/>
  <Path name="/etc/init.d/memcached"/>
  <Path name="/usr/share/memcached/scripts/start-memcached"/>
  <BoundPath name='/etc/memcached.conf' type='nonexistent'/>
  <Service name='memcached'/>
</Bundle>

mercury.xml
<Bundle name='mercury'>
  <Package name='apache2'/>
  <Package name='autoconf'/>
  <Package name='automake'/>
  <Package name='automake1.4'/>
  <Package name='bzr'/>
  <Package name='defoma'/>
  <Package name='fontconfig-config'/>
  <Package name='libapache2-mod-rpaf'/>
  <Package name='libfontconfig1'/>
  <Package name='libltdl7'/>
  <Package name='libltdl-dev'/>
  <Package name='libpng12-0'/>
  <Package name='libssl-dev'/>
  <Package name='libt1-5'/>
  <Package name='libtool'/>
  <Package name='libxpm4'/>
  <Package name='mysql-client-5.0'/>
  <Package name='mysql-server-5.0'/>
  <Package name='php5'/>
  <Package name='php5-cli'/>
  <Package name='php5-curl'/>
  <Package name='php5-dev'/>
  <Package name='php5-gd'/>
  <Package name='php5-imagick'/>
  <Package name='php5-mysql'/>
  <Package name='php5-xmlrpc'/>
  <Package name='php-apc'/>
  <Package name='php-pear'/>
  <Package name='shtool'/>
  <Package name='subversion'/>
  <Package name='ttf-dejavu'/>
  <Package name='ttf-dejavu-core'/>
  <Package name='ttf-dejavu-extra'/>
  <Package name='zlib1g-dev'/>

  <!-- symlinks -->
  <Path name="/etc/apache2/mods-enabled/expires.load"/>
  <Path name="/etc/apache2/mods-enabled/rewrite.load"/>
  <Path name="/etc/apache2/sites-enabled/000-default"/>
  <!-- mercury config files -->
  <Path name="/etc/apache2/ports.conf"/>
  <Path name="/etc/apache2/sites-available/default"/>
  <Path name="/etc/apache2/conf.d/mercury"/>
  <Path name="/etc/apt/sources.list"/>
  <Path name="/etc/cron.d/drupal"/>
  <Path name="/etc/mercury/boot.d/S050ec2.sh"/>
  <Path name="/etc/mercury/boot.d/S100main.sh"/>
  <Path name="/etc/mercury/config_mem.sh"/>
  <Path name="/etc/mercury/init.sh"/>
  <Path name="/etc/motd.tail"/>
  <Path name="/etc/mysql/my.cnf"/>
  <Path name="/etc/php5/apache2/php.ini"/>
  <Path name="/etc/php5/conf.d/apc.ini"/>
  <Path name="/etc/rc.local"/>
  <Path name="/root/CHANGELOG.txt"/>
  <Path name="/root/README.txt"/>
  <Path name="/usr/local/bin/update_mercury.sh"/>
  <Path name="/usr/local/bin/update_pressflow.sh"/>

  <!-- mercury services -->
  <Service name='apache2'/>
  <Service name='mysql'/>
  <Service name='mysql-ndb'/>
  <Service name='mysql-ndb-mgm'/>
</Bundle>

ubuntu-karmic.xml

Greg Coit's picture

Kcoop,

BCFG2 is not suppsoed to need packages listed in Bundle that are depended on (or reqired or pulled in) by other packages that are listed So, instead of making Operating System specific versions of amazon-web-services.xml. mercury.xml, memcached.xml, etc.., I was able to determine that the differences you gave me were these mind of packages - hence BCFG2 shouldn't need them to be specified. However, there seems to a bug in BCFG2 where it's calling packages that are required by others as "unmanaged". I'll submit a bug report soon on this to BCFG2. Fortunately, this won't affect functionality.

As for ec2-ami-tools, this should become available using the new instructions here http://groups.drupal.org/node/50408:

echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

If not, then I need to make some changes.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

That certainly would be better

kcoop's picture

Are you able to build mercury on Karmic? Stepping through the directions, running bcfg2 client just hangs for me after complaining of having no information for several packages. And when I kill it, apache is in a bad state (not running, can't start).

Offending packages

Greg Coit's picture

I've removed the offending packages from the BCFG2 config on launchad - Mercury should install on Karmic now.

This thread seems a little long so I created a Karmic-specific one:

http://groups.drupal.org/node/57093

Thanks!

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

One issue I see...

kcoop's picture

Some of these packages required multiverse on the repo. See the /etc/apt/sources I sent you, Greg.

Still having some issues even after updating that. Haven't tried using the versions of amazon.web-services.xml et.al I sent yet though. Gotta run for now.

Added Multiverse

Greg Coit's picture

Kcoop,

I added multiverse to the instructions - thanks!

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Glad to help

jamesJonas's picture

Like you, I wish to deploy a karmic/EBS boot platform. I like your idea of a second EBS drive for the database. In my current (old) Amazon configuration, I moved all my customized configuration, storage files and Drupal customizations into EBS. This allowed me to not only snapshot, but to prep my server for future upgrades.

yea ebs is pretty awesome ..

Macronomicus's picture

yea ebs is pretty awesome .. I like them separate so I can snapshot the db more frequently than the other drives. I also noticed that when you create another ami off an ebs bootable it detects any attached drives and bundles them with the ami; this can be run directly inside the aws console, I thought that was pretty sweet!

Cant wait to test that karmic bcfg2 script... sounds awesome!

Cheers!

Snapshots against a live server

kcoop's picture

I'm just starting to look at frequent snapshots, which are an important part of my overall plan. I've just discovered that another step is required to enable this, since alestic's karmic ami is not xfs (!). Did you do something like this?

Clearly, Lucid is going to be the game changer.

oh I hadnt noticed that ..

Macronomicus's picture

oh I hadnt noticed that .. didnt setup auto snapshots on the ebs boot drive .. just did a few manual ones.

It should be noted with a big red WARNING

kcoop's picture

Since ec2-consistent-snapshot depends on xfs to freeze the file system. Can you say "corrupted file system"?

I scratched my head for a minute trying to understand why Schlomo had to do this. I mean, Eric Hammond, the guy who wrote ec2-consistent-snapshot, didn't use xfs when he built the ami, even though it was preloaded?!? Then I saw him talking in the comments here about sticking as close as possible to Canonical's releases, and that they're waiting for Lucid to start doing EBS boot. Hopefully Mercury will get a whole lot more robust when it arrives.

yeah I used the karmic ebs

Macronomicus's picture

yeah I used the karmic ebs boot ami from aelstic.com as per Erics suggestion on IRC .. he made that from the canonical output with a few small tweaks ... and more interestingly he made an ebs bootable version. I haven't tried Lucid yet but im hearing good things. Regardless as long as you keep your constantly changing bits out of the ebs bootable drive and on separate xfs drives you can auto snapshot them .. and with the main boot drive you can just do manual snapshots when you make changes to the box, which shouldn't be too often really. IMO a manual snapshot is probably the best frequency for the boot drive anyhow.

$conf question for settings.php

toemaz's picture

When starting a fresh 1.0 Mercury, I find the snippet below at the end of the settings.php. It looks to me as if the Varnish $conf array is overwritten by the memcached one. Or do I see this wrong?

<?php
# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

# Memcached configuration
$conf = array(
  
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
  
'memcache_servers' => array(
        
'127.0.0.1:11211' => 'default',
        
'127.0.0.1:11212' => 'menu',
        
'127.0.0.1:11213' => 'filter',
        
'127.0.0.1:11214' => 'form',
        
'127.0.0.1:11215' => 'block',
        
'127.0.0.1:11216' => 'update',
        
'127.0.0.1:11217' => 'views',
        
'127.0.0.1:11218' => 'content',
        
'127.0.0.1:11219' => 'apachesolr',
       ),
  
'memcache_bins' => array(
         
'cache'        => 'default',
         
'cache_menu'   => 'menu',
         
'cache_filter' => 'filter',
         
'cache_form'   => 'form',
         
'cache_block'  => 'block',
         
'cache_update' => 'update',
         
'cache_views'  => 'views',
         
'cache_views_data'  => 'views',
         
'cache_content'  => 'content',
         
'cache_apachesolr'  => 'apachesolr',
       ),
);
?>

better fix

toemaz's picture

As mentioned kcoop, the fix is to switch the Varnish $conf params after the one from Memcache. I would like to see a better fix and change Memcache instead:

<?php
# Memcached configuration
$conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
$conf['memcache_servers'] = array(
        
'127.0.0.1:11211' => 'default',
        
'127.0.0.1:11212' => 'menu',
        
'127.0.0.1:11213' => 'filter',
        
'127.0.0.1:11214' => 'form',
        
'127.0.0.1:11215' => 'block',
        
'127.0.0.1:11216' => 'update',
        
'127.0.0.1:11217' => 'views',
        
'127.0.0.1:11218' => 'content',
        
'127.0.0.1:11219' => 'apachesolr',
       );
$conf['memcache_bins'] = array(
         
'cache'        => 'default',
         
'cache_menu'   => 'menu',
         
'cache_filter' => 'filter',
         
'cache_form'   => 'form',
         
'cache_block'  => 'block',
         
'cache_update' => 'update',
         
'cache_views'  => 'views',
         
'cache_views_data'  => 'views',
         
'cache_content'  => 'content',
         
'cache_apachesolr'  => 'apachesolr',
       );
?>

Good catch

re: Good catch

Greg Coit's picture

This has been fixed in the mercury docs and in the launchpad files.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

I'm using Rackspace. What

pcave's picture

I'm using Rackspace. What does the 'replace with .' statement mean in step 6.

6) Edit /var/lib/bcfg2/Metadata/groups.xml:

Add/remove i386/amd64/ec2/rackspace groups, etc (default is Ubuntu Jaunty 32bit on EC2 - configure group profile to match your needs (e.g. if host is Rackspace, replace with .

Sample groups.xml file for a rackspace instance

pcave's picture

Can someone post a sample groups.xml file for a rackspace instance running Ubuntu Karmic. I've followed all other instructions but I still don't know what the groups.xml file should look like.

I'm really struggling here. bcfg2 -vq doesn't install anything.

Running probe groups
Probe groups has result:
amd64
Loaded tool drivers:
APT Action DebInit POSIX
Package apache2 not installed
Package autoconf not installed
Package automake not installed
Package automake1.4 not installed
Package libapache2-mod-rpaf not installed
Package libltdl7-dev not installed
Package libssl-dev not installed
Package libt1-5 not installed
Package libtool not installed
Package mysql-client-5.0 not installed
Package mysql-server-5.0 not installed
Package php5 not installed
Package php5-cli not installed
Package php5-curl not installed
Package php5-dev not installed
Package php5-gd not installed
Package php5-imagick not installed
Package php5-mysql not installed
Package php5-xmlrpc not installed
Package php-apc not installed
Package php-pear not installed
Package shtool not installed
Package subversion not installed
Package zlib1g-dev not installed
Package varnish not installed
Package ssh not installed
Package postfix not installed
Package ssl-cert not installed
Package apache2-mpm-prefork not installed
Package apt-transport-https not installed
Package bash-completion not installed
Package bind9-host not installed
Package binutils not installed
Package build-essential not installed
Package command-not-found not installed
Package command-not-found-data not installed
Package deborphan not installed
Package dialog not installed
Package emacs22-nox not installed
Package friendly-recovery not installed
Package fuse-utils not installed
Package g++ not installed
Package g++-4.3 not installed
Package gcc not installed
Package gcc-4.3 not installed
Package gettext not installed
Package gettext-base not installed
Package html2text not installed
Package intltool-debian not installed
Package iputils-arping not installed
Package iputils-tracepath not installed
Package klogd not installed
Package libalgorithm-c3-perl not installed
Package libapache2-mod-php5 not installed
Package libbeecrypt7 not installed
Package libbind9-50 not installed
Package libc6-dev not installed
Package libclass-c3-perl not installed
Package libclass-c3-xs-perl not installed
Package libclass-mop-perl not installed
Package libcompress-raw-zlib-perl not installed
Package libcompress-zlib-perl not installed
Package libcroco3 not installed
Package libdata-optlist-perl not installed
Package libdevel-globaldestruction-perl not installed
Package libdigest-hmac-perl not installed
Package libdigest-sha1-perl not installed
Package libfile-slurp-perl not installed
Package libfont-afm-perl not installed
Package libfuse2 not installed
Package libgc1c2 not installed
Package libgd2-xpm not installed
Package libgomp1 not installed
Package libhtml-format-perl not installed
Package libhtml-parser-perl not installed
Package libhtml-tagset-perl not installed
Package libhtml-template-perl not installed
Package libhtml-tree-perl not installed
Package libio-compress-base-perl not installed
Package libio-compress-zlib-perl not installed
Package libisc50 not installed
Package libisccc50 not installed
Package libisccfg50 not installed
Package liblist-moreutils-perl not installed
Package liblwres50 not installed
Package libmail-sendmail-perl not installed
Package libmailtools-perl not installed
Package libmoose-perl not installed
Package libmro-compat-perl not installed
Package libneon27 not installed
Package libntfs-3g54 not installed
Package libparams-util-perl not installed
Package libparams-validate-perl not installed
Package libpcap0.8 not installed
Package libscope-guard-perl not installed
Package libstdc++6-4.3-dev not installed
Package libsub-exporter-perl not installed
Package libsub-identify-perl not installed
Package libsub-install-perl not installed
Package libsub-uplevel-perl not installed
Package libsys-hostname-long-perl not installed
Package libterm-readkey-perl not installed
Package libtest-exception-perl not installed
Package libtest-simple-perl not installed
Package liburi-perl not installed
Package libvolume-id1 not installed
Package libwww-perl not installed
Package libxml-namespacesupport-perl not installed
Package libxml-parser-perl not installed
Package libxml-sax-expat-perl not installed
Package libxml-sax-perl not installed
Package libxml-simple-perl not installed
Package linux-libc-dev not installed
Package lynx not installed
Package lynx-cur not installed
Package mktemp not installed
Package mlocate not installed
Package mtr-tiny not installed
Package netcat-tradition not installed
Package nmap not installed
Package ntfs-3g not installed
Package po-debconf not installed
Package ppp not installed
Package pppconfig not installed
Package pppoeconf not installed
Package python-gdbm not installed
Package python-gnupginterface not installed
Package reiserfsprogs not installed
Package sysklogd not installed
Package tcpdump not installed
Package telnet not installed
Package ubuntu-standard not installed
Package ufw not installed
Package update-manager-core not installed
Package update-motd not installed
Package uuid-runtime not installed
Package w3m not installed
Package xfsprogs not installed
Package ca-certificates-java not installed
Package default-jre-headless not installed
Package gcj-4.3-base not installed
Package icedtea-6-jre-cacao not installed
Package java-common not installed
Package libaccess-bridge-java not installed
Package libcommons-collections-java not installed
Package libcommons-dbcp-java not installed
Package libcommons-pool-java not installed
Package libcups2 not installed
Package libecj-java not installed
Package libecj-java-gcj not installed
Package libgcj9-0 not installed
Package libgcj9-jar not installed
Package libgcj-bc not installed
Package libgcj-common not installed
Package liblcms1 not installed
Package libservlet2.5-java not installed
Package rhino not installed
Package tomcat6 not installed
Package tzdata-java not installed
Package libc6-xen not installed
Package memcached not installed
Package libevent1 not installed
Package php5-memcache not installed
Package snmp not installed
Package snmpd not installed
Package apache2-utils not installed
Package apache2.2-common not installed
Package at not installed
Package autotools-dev not installed
Package cpp-4.3 not installed
Package dnsutils not installed
Package dosfstools not installed
Package dpkg-dev not installed
Package ed not installed
Package emacs22-bin-common not installed
Package emacs22-common not installed
Package emacsen-common not installed
Package ftp not installed
Package gcc-4.3-base not installed
Package gsfonts not installed
Package hdparm not installed
Package info not installed
Package jsvc not installed
Package libapr1 not installed
Package libaprutil1 not installed
Package libasound2 not installed
Package libbind9-40 not installed
Package libc6-i686 not installed
Package libcommons-daemon-java not installed
Package libdatrie0 not installed
Package libdb4.6 not installed
Package libdbd-mysql-perl not installed
Package libdbi-perl not installed
Package libdirectfb-1.0-0 not installed
Package libdjvulibre-text not installed
Package libdjvulibre21 not installed
Package libdns45 not installed
Package libelf1 not installed
Package libilmbase6 not installed
Package libisc45 not installed
Package libisccc40 not installed
Package libisccfg40 not installed
Package libjasper1 not installed
Package libkrb53 not installed
Package liblwres40 not installed
Package libmagickcore1 not installed
Package libmagickwand1 not installed
Package libmysqlclient15off not installed
Package libneon27-gnutls not installed
Package libnet-daemon-perl not installed
Package libntfs-3g49 not installed
Package libopenexr6 not installed
Package libparted1.8-10 not installed
Package libpci3 not installed
Package libperl5.10 not installed
Package libplrpc-perl not installed
Package libpq5 not installed
Package libreadline5 not installed
Package libsensors3 not installed
Package libsnmp-base not installed
Package libsnmp15 not installed
Package libsvn1 not installed
Package libtiff4 not installed
Package libtomcat6-java not installed
Package libvarnish1 not installed
Package libwmf0.2-7 not installed
Package lshw not installed
Package lsof not installed
Package ltrace not installed
Package m4 not installed
Package memtest86+ not installed
Package mysql-common not installed
Package mysql-server-core-5.0 not installed
Package openjdk-6-jre-headless not installed
Package openjdk-6-jre-lib not installed
Package parted not installed
Package pciutils not installed
Package php5-common not installed
Package popularity-contest not installed
Package psmisc not installed
Package python-cheetah not installed
Package startup-tasks not installed
Package strace not installed
Package system-services not installed
Package time not installed
Package tomcat6-common not installed
Package upstart-compat-sysv not installed
Package upstart-logd not installed
Package usbutils not installed

Phase: initial
Correct entries: 254
Incorrect entries: 306
Total managed entries: 560
Unmanaged entries: 26

Installing Service mysql-ndb
Installing Service varnishlog
Installing Service rmnologin
Installing Service snmpd
Installing Service postfix
Installing Service apache2
Installing Service dns-clean
Installing Service bootlogs.sh
Installing Service atd
Installing Service varnish
Installing Service pppd-dns
Installing Service memcached
Installing Service sysklogd
Installing Service mysql
Installing Service varnishncsa
Installing Service mysql-ndb-mgm
Installing Service klogd
Installing Service tomcat6
APT has no information about package upstart-logd
APT has no information about package libparted1.8-10
APT has no information about package libkrb53
APT has no information about package libdirectfb-1.0-0
APT has no information about package libvolume-id1
APT has no information about package libltdl7-dev
APT has no information about package libbind9-40
APT has no information about package libisc45
APT has no information about package libdatrie0
APT has no information about package libmagickcore1
APT has no information about package system-services
APT has no information about package libc6-xen
APT has no information about package libisccc40
APT has no information about package libevent1
APT has no information about package netcat-tradition
APT has no information about package upstart-compat-sysv
APT has no information about package libmagickwand1
APT has no information about package libntfs-3g49
APT has no information about package libc6-i686
APT has no information about package libisccfg40
APT has no information about package liblwres40
APT has no information about package startup-tasks
APT has no information about package libdns45

groups.xml

Greg Coit's picture

Let's say you want to install Mercury on Ubuntu-Karmic on Rackspace:
Note 1) ubuntu-karmic.xml just got uploaded to launchpad
Note 2) unneeded entries removed for clarification

<Groups version='3.0'>
  <Group profile='true' public='true' default='true' name='mercury-ubuntu-karmic-32'>
    <Group name='ubuntu-karmic'/>
    <Group name='i386'/>
    <Group name='rackspace'/>
    <Group name='mercury'/>
    <Group name='bcfg2'/>
  </Group>

  <!-- Operating systems -->
  <Group name='ubuntu-karmic'>
    <Group name='ubuntu'/>
    <Bundle name='ubuntu-karmic'/>
  </Group>
  <Group name='ubuntu'/>

  <!-- pantheon options -->
  <Group name='mercury'>
    <Bundle name='mercury'/>
    <Bundle name='memcached'/>
    <Bundle name='postfix'/>
    <Bundle name='tomcat6'/>
    <Bundle name='varnish'/>
    <Bundle name='cacti-client'/>
  </Group>

  <!-- hosting providers -->
  <Group name='rackspace'>
    <Bundle name='rackspace'/>
  </Group>

  <!-- other options -->
  <Group name='i386'>
    <Bundle name='i386'/>
  </Group>

  <Group name='bcfg2'>
    <Bundle name='bcfg2'/>
  </Group>
</Groups>

And then your clients.xml file should say:

profile="mercury-ubuntu-karmic-32"

You could just as easily call it mercury-ubuntu-karmic-rackspace-32 in both the groups.xmls and clients.xml (which we will eventually do when we start supporting VPSs other than Amazon Web Services.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Any update on Linode?

bcjenkins's picture

Hey all,

I am looking to set this up on Linode and wonder if there will be a stackscript soon? Also, given that I can purchase several Linodes for the same price as one EC2 small, is there any benefit to breaking out the roles - or should it all be dumped into bigger units?

Thanks!

b

re: Any update on Linode?

Greg Coit's picture

Thanks to help from a great Volunteer, linode support will be added in the new few days if not sooner.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Small thing in the AMI README.txt

RobertWise's picture

I just started up this AMI:
pantheon-amis-us-east/mercury-pressflow-drupal-32-1.0.manifest.xml ami-9f9f70f6

I noticed a tiny thing in the /root/README.txt:

Step 2 states that the settings.php file lives here:
/var/www/pressflow/sites/default/settings.php

and I found the settings.php living here instead:
/var/www/sites/default/settings.php

Other than that everything seems to have worked like a charm.

-Robert

re: Small thing in the AMI README.txt

Greg Coit's picture

Robert,

Thanks - the type has been fixed and will be uploaded to launchpad today.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

What do you do to the Groups,

WhenInRome's picture

What do you do to the Groups, Clients, and Configs.xml files if you're just putting this on a dedicated server?

I edited most of it but kept the amazon-web-services part to see if it worked. I'm sure the erros below are because i'm not in the amazon cloud....

First time i started i got these errors:

Starting Configuration Management Server: * bcfg2-server
bcfg2-server[67467]: Failed to load /var/lib/bcfg2/Pkgmgr indices
bcfg2-server[67467]: Failed to instantiate plugin Pkgmgr
bcfg2-server[67467]: Failed to load /var/lib/bcfg2/Rules indices
bcfg2-server[67467]: Failed to instantiate plugin Rules
bcfg2-server[67467]: Failed to load SSHbase repository from /var/lib/bcfg2/SSHbase

bcfg2-server[67467]: Failed to read file probed.xml
bcfg2-server[67467]: Loading experimental plugin(s): Packages
bcfg2-server[67467]: NOTE: Interfaces subject to change
bcfg2-server[67467]: service available at https://localhost:6789
bcfg2-server[67467]: serving bcfg2-server at https://localhost:6789
bcfg2-server[67467]: serve_forever() [start]
bcfg2-server[67467]: Handled 33 events in 0.015s
bcfg2-server[67467]: Handled 32 events in 0.005s
bcfg2-server[67467]: Handled 86 events in 0.023s
bcfg2-server[67467]: Handled 67 events in 0.020s
bcfg2-server[67467]: Handled 38 events in 0.014s
bcfg2-server[67467]: Handled 7 events in 0.003s

I killed the process, then started it again. Why don't i get the above errors any more? I get these new ones:

root@pulsar:/# sudo /etc/init.d/bcfg2-server start; tail -f /var/log/syslog
Starting Configuration Management Server: * bcfg2-server
Mar 30 01:10:50 pulsar bcfg2-server[18740]: Server startup failed
Mar 30 01:17:01 pulsar /USR/SBIN/CRON[18766]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Failed to read file probed.xml
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Loading experimental plugin(s): Packages
Mar 30 01:17:14 pulsar bcfg2-server[18779]: NOTE: Interfaces subject to change
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Failed to bind to socket
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Server startup failed
Mar 30 01:20:01 pulsar /USR/SBIN/CRON[18791]: (root) CMD ([ -x /usr/sbin/update-motd ] && /usr/sbin/update-motd 2>
Mar 30 01:20:44 pulsar bcfg2-server[18626]: serve_forever() [stop]
Mar 30 01:20:44 pulsar bcfg2-server[18626]: server_close()
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Failed to read file probed.xml
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Loading experimental plugin(s): Packages
Mar 30 01:21:04 pulsar bcfg2-server[18873]: NOTE: Interfaces subject to change
Mar 30 01:21:04 pulsar bcfg2-server[18873]: service available at https://pulsar:6789
Mar 30 01:21:04 pulsar bcfg2-server[18873]: serving bcfg2-server at https://pulsar:6789
Mar 30 01:21:04 pulsar bcfg2-server[18873]: serve_forever() [start]
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 35 events in 0.007s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 31 events in 0.003s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 83 events in 0.026s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 64 events in 0.022s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 38 events in 0.017s
Mar 30 01:21:05 pulsar bcfg2-server[18873]: Handled 7 events in 0.004s

Thanks,

re: errors

Greg Coit's picture

I forgot to address your question about errors.

Starting Configuration Management Server: * bcfg2-server
bcfg2-server[67467]: Failed to load /var/lib/bcfg2/Pkgmgr indices
bcfg2-server[67467]: Failed to instantiate plugin Pkgmgr
bcfg2-server[67467]: Failed to load /var/lib/bcfg2/Rules indices
bcfg2-server[67467]: Failed to instantiate plugin Rules
bcfg2-server[67467]: Failed to load SSHbase repository from /var/lib/bcfg2/SSHbase

bcfg2-server[67467]: Failed to read file probed.xml

These look like errors, but aren't - we don't use those bcfg2 plugins and bcfg2 is just nothing that fact.

bcfg2-server[67467]: Loading experimental plugin(s): Packages
bcfg2-server[67467]: NOTE: Interfaces subject to change
bcfg2-server[67467]: service available at https://localhost:6789
bcfg2-server[67467]: serving bcfg2-server at https://localhost:6789
bcfg2-server[67467]: serve_forever() [start]
bcfg2-server[67467]: Handled 33 events in 0.015s
bcfg2-server[67467]: Handled 32 events in 0.005s
bcfg2-server[67467]: Handled 86 events in 0.023s
bcfg2-server[67467]: Handled 67 events in 0.020s
bcfg2-server[67467]: Handled 38 events in 0.014s
bcfg2-server[67467]: Handled 7 events in 0.003s
<code>

This looks like normal bcfg2-server startup.

<code>
Starting Configuration Management Server: * bcfg2-server
Mar 30 01:10:50 pulsar bcfg2-server[18740]: Server startup failed
Mar 30 01:17:01 pulsar /USR/SBIN/CRON[18766]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Failed to read file probed.xml
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Loading experimental plugin(s): Packages
Mar 30 01:17:14 pulsar bcfg2-server[18779]: NOTE: Interfaces subject to change
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Failed to bind to socket
Mar 30 01:17:14 pulsar bcfg2-server[18779]: Server startup failed
Mar 30 01:20:01 pulsar /USR/SBIN/CRON[18791]: (root) CMD ([ -x /usr/sbin/update-motd ] && /usr/sbin/update-motd 2>
Mar 30 01:20:44 pulsar bcfg2-server[18626]: serve_forever() [stop]
Mar 30 01:20:44 pulsar bcfg2-server[18626]: server_close()
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Failed to read file probed.xml
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Loading experimental plugin(s): Packages
Mar 30 01:21:04 pulsar bcfg2-server[18873]: NOTE: Interfaces subject to change
Mar 30 01:21:04 pulsar bcfg2-server[18873]: service available at https://pulsar:6789
Mar 30 01:21:04 pulsar bcfg2-server[18873]: serving bcfg2-server at https://pulsar:6789
Mar 30 01:21:04 pulsar bcfg2-server[18873]: serve_forever() [start]
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 35 events in 0.007s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 31 events in 0.003s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 83 events in 0.026s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 64 events in 0.022s
Mar 30 01:21:04 pulsar bcfg2-server[18873]: Handled 38 events in 0.017s
Mar 30 01:21:05 pulsar bcfg2-server[18873]: Handled 7 events in 0.004s

Everything here looks good too except the two "Server startup failed" messages which I don't remember seeing. Was there anything differernt between those 2 times and the times where it did start?

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

re: What do you do to the Groups,

Greg Coit's picture

Mustang302,

For a non-VPS, I would just remove amazon-web-services from your profile in /var/lib/bcfg2/Metadata/groups.xml:

<Groups version='3.0'>
  <Group profile='true' public='true' default='true' name='mercury-ubuntu-jaunty-32'>
    <Group name='mercury'/>
    <Group name='ubuntu-jaunty'/>
    <Group name='i386'/>
    <Group name='bcfg2'/>
  </Group>

And then leave the rest of the file as is - there's no harm in having an amazon-web-services group on a non-vps server as long as it's not part of your profile.

I would then edit your /var/lib/bcfg2/Metadata/clients.xml so that the name matches the name used in /var/lib/bcfg2/Metadata/groups.xml.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Just tried this all over

WhenInRome's picture

Just tried this all over again, everything seems fine except now since i've deleted the aws group in groups.xml, I get this error. Not sure where this F'ed up.

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Any ideas?

Thanks,

re: Just tried this all over

Greg Coit's picture

Mustang302,

My best guess is that /etc/mercury/boot.d/S05ec2.sh ran when it wasn;t supposed to and it created some symlinks for mysql to places that only exist on AWS. For some reason, out test to not run Just tried this all over on non-aws servers went missing - it was restored a few days ago.

An /etc/init.d/mysql restart should confirm this for you (the error will mention an unaccessible directory).

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

I'm trying the same thing in

qbnflaco's picture

I'm trying the same thing in vmware fusion at home and skipped out the aws config and am getting the same error 2002. When running /etc/init.d/mysql restart I get

* Stopping MySQL database server mysqld
   ...done.
* Starting MySQL database server mysqld
   ...fail!

Trying mysqld gives me

100504 20:40:52 [Warning] Can't create test file /mnt/mysql/ubuntu.lower-test
100504 20:40:52 [Warning] Can't create test file /mnt/mysql/ubuntu.lower-test
100504 20:40:52 [Warning] One can only use the --user switch if running as root

100504 20:40:52  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

mysql user does have rights to this folder. I had to manually point the data location in the config file to /mnt/mysql where the files are located. However, it still seems to be giving me this error.

Apparmor?

doka's picture

Same problem for me on Jaunty, but switch off Apparmor helps:

sudo /etc/init.d/apparmor stop
sudo /etc/init.d/mysql restart

The permanent solution could be to change the Apparmor config file (/etc/apparmor.d/usr.sbin.mysqld), but I could not figure out what should be changed.

And I guess the change in my.conf is not necessary, since there are symlinks in /var/lib and /var/log to /mnt/mysql/...

I added in these lines to the

qbnflaco's picture

I added in these lines to the file...

  /mnt/mysql/tmp/ r,
  /mnt/mysql/tmp/** rwk,
  /mnt/mysql/ r,
  /mnt/mysql/** rwk,

and it seemed to work.

luismiguel2001us's picture

Hi folks,

I am absolutely thrilled with the Mercury profile. Now that I have this running, I wanted to do a multi-site setup, but one of the additional sites that I am trying to setup needs to run in Spanish. Unfortunately it looks like your profile installation cannot be changed to install in a different language other than English. This is problematic since changing the language setting after the initial install is not 100%. Many entries in Drupal stay in English if the language is switched after the install has been completed. Is there a way to enable language selection during the profile installation process? Thanks for your help.

Cheers,
Luis

Greg Coit's picture

Yes, the instructions you see by clicking "Learn how to install Pantheon in other languages" in the install.php should still work with Pantheon/Mercury.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: