Step-by-step: Setting up Project Mercury 1.0 (Varnish, Apache, APC, Memcached and Solr) using BCFG2

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Here are step-by-step instructions for building Project Mercury on a fresh server using the configuration manager BCFG2. Project Mercury has been developed on AWS ec2 but should work on Rackspace and many other VPSs, or on your home machine. We have public AMIs available of Ubuntu Jaunty versions (in both 32 and 64 bit flavors) - see http://getpantheon.com.

If you'd like to try a more current version, refer to the new wiki page on setting up Mercury 1.1 for details. You may well find the recipe for 1.1 to be simpler and quicker than this sequence. Better yet, Mercury 1.1 supports Ubuntu Lucid, the most recent LTS release.

As of 2010-05-21: A NEWER VERSION OF MERCURY IS IN BETA!
3/19/10: Ubuntu Karmic added (BETA!)
3/1/10: 1.0 updates
2/25/10: Some more pre-1.0 updates including fixed launchpad paths for bzr and drush 3.0-beta1.
2/13/10: Pre-1.0 instructions updated and migrated from http://groups.drupal.org/node/50153 to take advantage of the wiki format.

This is a wiki page which I'll will try to keep up to date as the project evolves. Please feel free to comment, add notes, and correct any mistakes you see. For the non-BCFG2 version of these instructions, see http://groups.drupal.org/node/25425. If you wish to use pre-configured servers (ie, no install necessary), here are the Project-Mercury 1.0 AMIs:

• US East, 32 Bit: ami-9f9f70f6
• US East, 64 Bit: ami-879f70ee

• US West, 32 Bit: ami-e14c1da4
• US West, 64 Bit: ami-e54c1da0

• EU West, 32 Bit: ami-11b79c65
• EU West, 64 Bit: ami-1db79c69

Project Mercury Step By Step (using the BCFG2 configuration manager)

1) Start Ubuntu Jaunty (9.04) or Ubuntu Karmic (9.10):


Our work is currently based on Jaunty distribution of Ubuntu. If you want to use Amazon for your infrastructure there are many AMIs available, we prefer the ones provided by Alestic.com and/or Canonical. View a list of AMIs provided by both (including EBS-boot AMIs) at http://alestic.com/.

We currently have BCFG2 config files for Ubuntu Jaunty and Ubuntu Karmic (Beta!) running on AWS and rackspace. We hope to have config files for CentOS 5.2 plus more VPSs soon. We are also planning to update everything to the forthcoming 10.4 (Lucid) release, which has been designated by Ubuntu as a Long Term Support (LTS) release.

However, any clean installation of Ubuntu 9.04/Jaunty should be a fine starting place.

Note: The Karmic files in launchpad are BETA - please know they have not been tested as thoroughly as the Jaunty-specific files have. Also note that the AWS versions of Karmic (and Lucid) require one to login as user "ubuntu", not root. To account for this, we've added sudo where required. This should not cause issues if you are already root (ie, Ubuntu Jaunty).

2) Configure apt sources and get updates:


Regardless of the version of Ubuntu or the VPS you are using, make sure that main, restricted, universe and multiverse are enabled in apt. The easiest way to do this is open /etc/sources/apt.sources with your favorite text editor and make sure that each deb line has "main restricted universe multiverse" (without the quotes) at the end of it or running the following commands (replacing your existing /etc/apt/sources.list:

#UBUNTU JAUNTY ONLY:
echo "deb http://archive.ubuntu.com/ubuntu/ jaunty main restricted universe multiverse" | sudo tee /etc/apt/sources.list
echo "deb-src http://archive.ubuntu.com/ubuntu/ jaunty main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list

#UBUNTU KARMIC ONLY:
echo "deb http://archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse" | sudo tee /etc/apt/sources.list
echo "deb-src http://archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list

Now lets add the apt-sources for BCFG2 and bzr:

#UBUNTU JAUNTY ONLY:
echo "deb http://ppa.launchpad.net/bzr/ppa/ubuntu jaunty main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bzr/ppa/ubuntu jaunty main" | sudo tee -a /etc/apt/sources.list
echo "deb http://ppa.launchpad.net/bcfg2/ppa/ubuntu jaunty main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bcfg2/ppa/ubuntu jaunty main" | sudo tee -a /etc/apt/sources.list

#UBUNTU KARMIC ONLY:
echo "deb http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main" | sudo tee -a  /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list
echo "deb http://ppa.launchpad.net/bcfg2/ppa/ubuntu karmic main" | sudo tee -a  /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/bcfg2/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list

#Ubuntu Karmic on AWS only:
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" | sudo tee -a /etc/apt/sources.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

#ALL VERSIONS/VPSs:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8C6C1EFD
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 98932BEC
sudo apt-get update

#RACKSPACE ONLY:
sudo apt-get -y install language-pack-en-base
sudo dpkg-reconfigure locales

#ALL VERSIONS/VPSs:
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade

3) Install and configure BCFG2:


sudo apt-get -y install bzr bcfg2-server gamin python-gamin
sudo rm -rf /etc/bcfg2* /var/lib/bcfg2
# accept defaults:
sudo bcfg2-admin init
# Press enter to accept default for "Store bcfg2 configuration in [/etc/bcfg2.conf]:"
# Press enter to accept default for "Location of bcfg2 repository [/var/lib/bcfg2]:"
# Press enter to accept default for "Input password used for communication verification (without echoing; leave blank for a random):"
# Press enter to accept default for "What is the server's hostname: [your AWS server default]:"
# Press enter to accept default for "Input the server location [your AWS server default]:"
# Enter number corresponding to your base OS
  # Input base Operating System for clients:
  # 1: Redhat/Fedora/RHEL/RHAS/Centos
  # 2: SUSE/SLES 
  # 3: Mandrake
  # 4: Debian
  # 5: Ubuntu
  # 6: Gentoo
  # 7: FreeBSD
# System generates a private key
  # Generating a 1024 bit RSA private key
  # ....++++++
  # ...........++++++
  # writing new private key to '/etc/bcfg2.key'
  # -----
  # Signature ok
  # subject
  # Getting Private key
  # Repository created successfuly in /var/lib/bcfg2

4) Get and install BCFG2 config files:


We want to keep the clients.xml file that bcfg2-admin init created so we're going to copy it to ~/ before we delete the old bcfg2 config files. An account on launchpad is not required for any of the bzr commands we give - only for uploading code back to launchpad It's ok to ignore the error message "You have not informed bzr of your Launchpad ID...".

sudo cp /var/lib/bcfg2/Metadata/clients.xml ~/
sudo rm -rf /var/lib/bcfg2/
sudo bzr branch lp:pantheon/bcfg2 /var/lib/bcfg2
sudo mv ~/clients.xml /var/lib/bcfg2/Metadata/

5) Edit /etc/bcfg2.conf:


Replace the "plugins" line in /etc/bcfg2.conf by launching nano (or your favorite editor):

sudo nano -w /etc/bcfg2.conf

and replace:

plugins = Base,Bundler,Cfg,Metadata,Pkgmgr,Rules,SSHbase

with:

plugins = Base,Bundler,Cfg,Metadata,Packages,Probes,Rules,SSHbase
filemonitor = gamin

6) Edit /var/lib/bcfg2/Metadata/groups.xml:


Add/remove i386/amd64/ec2/rackspace groups, etc (default is Ubuntu Jaunty 32bit on EC2 - configure group profile to match your needs (e.g. if host is Rackspace, replace Group name='amazon-web-services' with Group name='rackspace').

sudo nano -w /var/lib/bcfg2/Metadata/groups.xml

7) Edit /var/lib/bcfg2/Metadata/clients.xml:


Change profile='basic' to profile='mercury-ubuntu-jaunty-32' or whatever you used in /var/lib/bcfg2/Metadata/groups.xml (i.e. match the value of 'profile' in "clients.xml" to the value of 'name' in "groups.xml".

sudo nano -w /var/lib/bcfg2/Metadata/clients.xml

8) Edit /var/lib/bcfg2/Packages/config.xml:


make sure your operating system is set correctly (ie, replace Jaunty with Karmic if using Karmic).

sudo nano -w /var/lib/bcfg2/Packages/config.xml

9) Start the BCFG2 server:


Wait until the bcfg2-server spins up - you know the server is ready when see "serve_forever() [start]".

sudo /etc/init.d/bcfg2-server start; tail -f /var/log/syslog

10) Start the BCFG2 client:


This will auto-configure your server - it is finished when the command prompt is returned or you see "Failed to manipulate service apache2". If you get the "failed to manipulate" message it is safe to exit with a control-c. To be sure, you can re-run the command to get the status. We're investigating why this is happening. In the meantime we start apache by hand below.

sudo bcfg2 -vq

11) Install pressflow:


sudo rm -rf /var/www
sudo bzr branch lp:pressflow /var/www
echo "create database pressflow;" | mysql -u root
sudo mkdir /var/www/sites/default/files

12) Install Mercury profile:


sudo mv /var/www/profiles/default ./
sudo rmdir /var/www/profiles
sudo bzr branch lp:pantheon/profiles /var/www/profiles/
sudo mv default /var/www/profiles

13) Install drush:


wget http://ftp.drupal.org/files/projects/drush-6.x-3.0-rc3.tar.gz
tar xvzf drush-6.x-3.0-rc3.tar.gz
sudo chmod 555 drush/drush
sudo chown -R root:root drush
sudo mv drush /usr/local/
sudo ln -s /usr/local/drush/drush /usr/local/bin/drush

14) Install apachesolr:


The location of apache solr nightly builds has moved to http://hudson.zones.apache.org/hudson/job/Solr-trunk/lastSuccessfulBuild... but they appear to be broken ATM. We now recommend downloading apache-solr-1.4.0.tgz - a list of mirrors is available at http://www.apache.org/dyn/closer.cgi/lucene/solr/
wget http://apache.osuosl.org/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
tar xvzf apache-solr-1.4.0.tgz
sudo mv apache-solr-1.4.0/example/solr /var/
sudo mv apache-solr-1.4.0/dist/apache-solr-1.4.0.war /var/solr/solr.war
sudo chown -R tomcat6:root /var/solr/
sudo drush dl --destination=/var/www/sites/all/modules apachesolr
sudo svn checkout -r22 http://solr-php-client.googlecode.com/svn/trunk/ /var/www/sites/all/modules/apachesolr/SolrPhpClient
sudo mv /var/www/sites/all/modules/apachesolr/schema.xml /var/solr/conf/
sudo mv /var/www/sites/all/modules/apachesolr/solrconfig.xml /var/solr/conf/

15) Install memcache and varnish:


sudo drush dl --destination=/var/www/sites/all/modules memcache-6.x-1.x-dev
sudo drush dl --destination=/var/www/sites/all/modules varnish

16) Prepare pressflow files and dirs:


sudo cp /var/www/sites/default/default.settings.php /var/www/sites/default/settings.php
sudo chown -R root:www-data /var/www/*
sudo chmod -R 775 /var/www/sites
sudo chmod 755 /var/www/sites/all/modules/

17) Run /etc/mercury/init.sh:

This is the point where we make our AMIs. /etc/mercury/init.sh is designed to run at first boot and:

a) setup mysql, tomcat and varnish to use /mnt to save space on / (which is only 10GB on a EC2 small instance).

b) configure postfix with a usable hostname and check for (and download) updates to mercury and pressflow (and apply the mercury updates via BCFG2).

c) configure APC, varnish, tomcat and PHP memory usage based on the system memory using the /etc/mercury/config_mem.sh script

d) send a unique but non-identifying hash to us so we can get an idea of how many Mercury users there are.

If you choose to not reboot your system, simply run:

sudo /etc/mercury/init.sh

A logfile is placed in /etc/mercury/bootlog

18) Configure Pressflow:


a) Go to your site in your web browser
b) Choose the Mercury profile (This will setup the appropriate modules and settings.)
c) Database name = "pressflow"
d) Datebase username ="root" (no password yet - we set it below)

After pressflow is configured:

e) set the mysql root password and create a non-root account (changing new_user and new_password to appropriate values):

# Login to MySQL as root
mysql -u root
# Set root user password
mysql> set password for root@localhost=PASSWORD('new_password');
# Create new user and give a password
mysql> create user 'new_user'@'localhost' identified by 'new_password';
# Set privileges for new user
mysql> grant all on pressflow.* to new_user@localhost identified by 'new_password';
mysql> flush privileges;
mysql> \q

f) update your Pressflow install with the new mysql account information by editing
/var/www/pressflow/sites/default/settings.php (again using the appropriate new_user and new_password values):

nano -w /var/www/sites/default/settings.php

and change:

$db_url = 'mysqli://username:password@localhost/databasename';

to what you set up for the new user above. Make sure to edit the 'mysqli:' and not the 'mysql:' portion.
$db_url = 'mysqli://new_user:new_password@localhost/pressflow';

g) change permissions on /var/www/pressflow/sites/default/settings.php to not be writable
chmod 440 /var/www/sites/default/settings.php. If you're running the site under a regular user home directory you can set it at 640. With the ownership being <user>.www-data. Otherwise the settings.php file should be read-only. If you need to edit it. Do a chmod to 640 before. I'm assuming that the file is owned by www-data.www-data.

h) change permissions on sites/default to not be writable
chmod 755 /var/www/sites/default/

Time to populate your new Mercury server!

Mercury

Group organizers

Group categories

Post Type

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week