Step-by-step: Setting up Project Mercury (Varnish, Apache, APC, Memcached and Solr)

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Refer to the new wiki for better docs. This page is here for posterity.

Here are step-by-step instructions for building Project Mercury on a fresh server using the configuration manager BCFG2. Project Mercury has been developed on AWS ec2 but should work on Rackspace and many other VPSs. We have public AMIs available of Ubuntu Jaunty versions (in both 32 and 64 bit flavors) - see

This is a wiki page which I'll will try to keep up to date as the project evolves. Please feel free to comment, add notes, and correct any mistakes you see. For the BCFG2 version of these instructions (and pre-made, ready-to-go AMIs), see

3/2/10: 1.0 updates
1/11/10 Explain how to configure the external caching backend on Drupal.
12/2/09 Mercury 0.81
11/19/09 Mercury 0.8-Beta
11/14/09 Restructured instructions
10/22/09 Configuration file download instructions
10/8/09 Added Apache Solr to the install
9/20/09 Numerous fixes and the new pressflow BZR location

Project Mercury Step By Step

1) Basic startup

a) Launch a new Ubuntu image. I use alestic ubuntu-9.04-jaunty-base.
b) Add the bzr/bzrtools ppa as per
c) Enable the Universe repositories by uncommenting them at /etc/apt/sources.list (Necessary for 9.10)
d) Get updates of installed packages and install needed packages:

apt-get update; apt-get upgrade; apt-get install apache2-mpm-prefork apache2-utils apache2.2-common autoconf automake automake1.4 autotools-dev bzr defoma fontconfig-config libapache2-mod-php5 libapache2-mod-rpaf libapr1 libaprutil1 libdbd-mysql-perl libdbi-perl libfontconfig1 libfreetype6 libgd2-xpm libhtml-template-perl libjpeg62 libltdl7 libltdl7-dev libmysqlclient15off libnet-daemon-perl libplrpc-perl libpng12-0 libpq5 libssl-dev libt1-5 libtool libvarnish1 libxpm4 m4 mysql-common mysql-server-core-5.0 mysql-server-5.0 mysql-client-5.0 php-apc php5 php5-cli php5-curl php5-common php5-dev php5-gd php5-mysql php5-xmlrpc php-pear postfix shtool ssl-cert subversion ttf-dejavu ttf-dejavu-core ttf-dejavu-extra varnish zlib1g-dev

Note: For our Mercury AMIs we leave the mysql root password blank (so the owner can choose it), but you should pick one during the install. Also we include postfix in the above instructions – replace this if you have a preference for another Mail Transport Agent (e.g. exim, sendmail).

If you are installing on Red Hat Enterprise Linux/CentOS 5, you may install Varnish from the EPEL repository as follows:

2) Optional: Download config files:

Note: we have removed the instructions for using our configuration files since they have been modified to be used with BCFG2. To create a Mercury server using our pre-modified config files, we recommend using the instructions here

If you wish to use drush to help with the install:

tar xvzf drush-All-versions-3.0-beta1.tar.gz
chmod 555 drush/drush
mv drush /usr/local/
ln -s /usr/local/drush/drush /usr/local/bin/drush

3) Configure apache and varnish to work together:

a) Change port 80 to port 8080 in /etc/apache2/ports.conf
b) Change port 80 to port 8080 in /etc/apache2/sites-available/default
c) Make a pressflow directory for varnish ( mkdir -p /var/lib/varnish/pressflow; chown varnish.varnish /var/lib/varnish/pressflow)
d) Edit /etc/default/varnish and:

1) Change INSTANCE=pressflow
2) Change port 6081 to port 80. Change the port from "-a: 6081" to "-a: 80".
3) Make sure "Alternative 2, Configuration with VCL" is set
4) Change the 1G at the end of the DAEMON_OPTS line to the amount of RAM you want Varnish to take.  Too much and your server crashes under a heavy load, too little and Varnish isn't really helping.  Start with 25% of the server's RAM and go from there (an Amazon Web Services small instance has 2G of Ram, so we set ours to 512MB).  Never set this number higher than the amount of RAM you have.

(Note: the file is called /etc/sysconfig/varnish under Red Hat.)

e) To allow Drupal's .htaccess file to function, edit /etc/apache2/sites-available/default and change "AllowOverride None" to "AllowOverride All" in the Directory /var/www/ block.
f) Enable mod-rewrite, restart apache and restart varnish (a2enmod rewrite; /etc/init.d/apache2 restart; /etc/init.d/varnish restart)
g) Load the public DNS of your instance. You should get the basic "It works!" page. This is coming via Varnish on port 80, pulled from Apache on the backend!
h) (Extra credit) Install the Live HTTP headers Firefox plugin or the Net tab on Firebug and look for headers like

Via: 1.1 varnish
X-Varnish: 1429427693 1429427691
Age: 320

Then try loading it with the port 8080 attached (e.g. http://public_AWS_dns:8080). Note the differences. The X-Varnish header should show 2 numbers (one for the current request, one for the request that was initially cached) or else the page has missed the Varnish cache, and has been served from the Apache backend.

4) Set up APC:

a) Edit /etc/php5/conf.d/apc.ini and make sure it has the following:
apc.include_once_override = 1

b) If you're setting up a stable (rather than a development) server, consider adding the apc.apc.stat = 0 option as well. This will prevent APC from re-checking cached files, and improves performance. However, if you're changing files frequently (e.g. the site is still under development), it's not a good idea. For more information, see this 2bits article.

c) Increase the max memory available to php by changing memory_limit = 16M to memory_limit = 64M in /etc/php5/apache2/php.ini.

5) Configure Mysql:

a) Create a pressflow database
b) Grant access to a non-root user
c) Optional: Change the default storage engine from MyISAM to InnoDB and restart apache (add default-storage-engine=innodb to /etv/my/my.conf)

mysql -u root -p

mysql> create database pressflow;
mysql> grant all on pressflow.* to user@localhost identified by 'password';
mysql> flush privileges;
mysql> \q

#if you wish to change the default-storage-engine:
mv -i trunk/distro/jaunty/conf/my.cnf /etc/mysql/my.cnf
/etc/init.d/mysql restart

6) Download and configure Pressflow:

We're going to replace the "It works!" /var/www directory with a BZR checkout of pressflow (from its new home on lauchpad):

a) Delete the /var/www directory (rm -r /var/www)
b) Download pressflow into /var/www (bzr branch lp:pressflow /var/www). If you do not have bzr on your machine, you may also download from
c) Make files dir for Pressflow and set ownership and permissions (mkdir /var/www/sites/default/files; chown -R root:www-data /var/www/sites/default/; chmod -R 775 /var/www/sites/default/)

To recap step 6:

rm -r www
bzr branch lp:pressflow /var/www
mkdir /var/www/sites/default/files
chown -R root:www-data /var/www/sites/default/
chmod -R 775 /var/www/sites/default/

7) Configure Varnish to work with Pressflow:

Edit /etc/varnish/default.vcl and add this, replacing the included "backend default" declaration:

backend default {
.host = "";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;

sub vcl_recv {
  // Remove has_js and Google Analytics cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");

  // To users: if you have additional cookies being set by your system (e.g.
  // from a javascript analytics file or similar) you will need to add VCL
  // at this point to strip these cookies from the req object, otherwise
  // Varnish will not cache the response. This is safe for cookies that your
  // backed (Drupal) doesn't process.
  // Again, the common example is an analytics or other Javascript add-on.
  // You should do this here, before the other cookie stuff, or by adding
  // to the regular-expression above.

  // Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
  // Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;

  // No varnish for install, update, or cron
  if (req.url ~ "install\.php|update\.php|cron\.php") {
    return (pass);

  // Normalize the Accept-Encoding header
  // as per:
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
   } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;

  // Let's have a little grace
  set req.grace = 30s;


// Strip any cookies before an image/js/css is inserted into cache.
// Also: future-support for ESI.
sub vcl_fetch {
  if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {
    unset obj.http.set-cookie;

sub vcl_hash {
  if (req.http.Cookie) {
    set req.hash += req.http.Cookie;

sub vcl_error {
  // Let's deliver a slightly more friedly error page.
  // You can customize this as you wish.
  set obj.http.Content-Type = "text/html; charset=utf-8";
  synthetic {"
  <?xml version="1.0" encoding="utf-8"?>
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
      <title>"} obj.status " " obj.response {"</title>
      <style type="text/css">
      #page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black; background-color: #FFF;}
      p {margin-left:20px;}
      body {background-color: #DDD; margin: auto;}
    <div id="page">
    <h1>Page Could Not Be Loaded</h1>
    <p>We're very sorry, but the page could not be loaded properly. This should be fixed very soon, and we apologize for any inconvenience.</p>
    <hr />
    <h4>Debug Info:</h4>
Status: "} obj.status {"
Response: "} obj.response {"
XID: "} req.xid {"
      <address><a href="">Varnish</a></address>

Then /etc/init.d/varnish restart

Once Varnish is up and running, then you must log in to the Pressflow site and visit admin/settings/performance. Change the Caching Mode to External and the Page Cache Maximum Age to an appropriate value (anything over 0). (See step 11 for more details.)

This will work to basically configure Varnish and Drupal to work together in terms of serving cached pages.

8) Install and configure Memcached:

a) Download memcache-6.x-1.x-dev from, untar and move to Pressflow's modules directory. If this version generates warnings then it is probably better to use latest stable version of memcache module.
b) Download cacherouter-6.x-1.0-rc1 from, untar and move to Pressflow's modules directory.
c) Install PECL Memcache extension.
d) Setup PECL Memcache extension. Edit /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini and add this code to extensions section in both files (this will enable memcache extension for both apache2 and cli php configs, we need cli because of drush, if you do not use drush then forget about the second file).

To recap step 8:

drush dl memcache-6.x-1.x-dev
drush dl cacherouter
pecl install memcache


tar xvzf memcache-6.x-1.x-dev.tar.gz
mkdir -p /var/www/sites/all/modules/
mv -i memcache /var/www/sites/all/modules/

9) Install and configure Apache Solr

a) Install Tomcat6, the JAVA servlet container for Solr (apt-get install tomcat6)
b) Change #TOMCAT6_SECURITY=yes to TOMCAT6_SECURITY=no and JVM_TMP=/tmp/tomcat6-temp to JVM_TMP=/mnt/tomcat6-temp in /etc/default/tomcat6 (note that we also uncommented both lines)
c) Download and install Apache Solr. We recommend using the nightly builds as Solr development is still pretty rapid. if the code below do not work for you it means that there are no nightly builds for today date and you need to go to in browser and download latest nightly release by yourself.

wget`date +%Y-%m-%d`.tgz
tar xvzf solr-`date +%Y-%m-%d`.tgz
mv apache-solr-1.5-dev/example/solr /var/
mv apache-solr-1.5-dev/dist/apache-solr-1.5-dev.war /var/solr/solr.war

d) Insert the following text into /etc/tomcat6/Catalina/localhost/solr.xml (this tells Tomcat where to find Solr):

<Context docBase="/var/solr/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
        <Environment name="solr/home" type="java.lang.String" value="/var/solr" override="true" />

e) Replace all instances of 8080 with 8180 in /etc/tomcat6/server.xml (since 8080 is being used by apache)
f) Download the Apache Solr module from, untar and move to Pressflow's module directory

drush dl apachesolr

Or if you don't have drush:

tar xvzf apachesolr-6.x-1.0-rc3.tar.gz
mv -i apachesolr /var/www/sites/all/modules/

g) Download php files for the Apache Solr module:

drush solr phpclient

Or if you don't have drush:

svn checkout -r22 /var/www/sites/all/modules/apachesolr/SolrPhpClient

h) Move Drupal-specific config and schema files that are packaged with Apache Solr Module into /var/solr/conf

mv -i /var/www/sites/all/modules/apachesolr/schema.xml /var/solr/conf/
mv -i /var/www/sites/all/modules/apachesolr/solrconfig.xml /var/solr/conf/

i) Set permissions for Apache Solr dir (chown -R tomcat6:root /var/solr/)
j) Restart tomcat (/etc/init.d/tomcat6 restart)

10) Optional: Download ec2-metadata and configure Postfix:

a) We recommend installing the very handy ec2-metadata shellscript. It makes finding out info about your instance a breeze.

wget -q
mv -i ec2-metadata /usr/local/bin/
chmod +x /usr/local/bin/ec2-metadata
/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //' > /etc/mailname

b) If you insalled Postfix, configure using the following:

postconf -e "myhostname = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`"
postconf -e "mydomain = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`"
postconf -e "mydestination = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`, localhost"

c) Restart Postfix (/etc/init.d/postfix restart)

To recap step 10:

wget -q
mv -i ec2-metadata /usr/local/bin/
chmod +x /usr/local/bin/ec2-metadata
/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //' > /etc/mailname
postconf -e "myhostname = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`"
postconf -e "mydomain = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`"
postconf -e "mydestination = `/usr/local/bin/ec2-metadata -p | sed 's/public-hostname: //'`, localhost"
/etc/init.d/postfix restart

11) Install Pressflow:

a) copy the /var/www/pressflow/sites/default/default.settings.php file to /var/www/pressflow/sites/default/settings.php

b) Connect to your AMI using a web browser and install Pressflow

c) Enable Apache Solr framework, Apache Solr search, Search and CacheRouter

Using drush:

drush enable apachesolr apachesolr_search search cacherouter

Otherwise navigate to admin/build/modules and enable them via the web interface.

d) Go to Site configuration > Apache Solr and set:
Solr Port: 8180
Number of items to index per cron run: 50
Make Apache Solr Search the default: Enabled
Enable spellchecker and suggestions: checked

Or if you are using drush:

drush vset --yes apachesolr_port 8180;
drush vset --yes apachesolr_cron_limit 50;
drush vset --yes apachesolr_search_make_default 1;
drush vset --yes apachesolr_search_spellcheck 1;

e) Go to Site configuration > Search Settings and set:
Number of items to index per cron run: 50

Or if you are using drush:

drush vset --yes search_cron_limit 50

f) Go to Site configuration > Performance and set:
Caching mode: External
Minimum cache lifetime: None
Page cache maximum age: 10 minutes
Page compression: Disabled
Block cache: Enabled
Optimize CSS files: Enabled
Optimize JavaScript files: Enabled

Or if you are using drush:

# Set cache to External
drush vset --yes cache 3;

# Minimum cache lifetime
drush vset --yes cache_lifetime 0;

# 10 minutes * 60 seconds = 600;
drush vset --yes page_cache_max_age 600;

# Disable page compression
drush vset --yes page_compression 0;

# Enable block cache
drush vset --yes block_cache 1;

# Compile CSS & JS
drush vset --yes preprocess_css 1;
drush vset --yes preprocess_js 1;

g) Go to User Management -> Permissions and set:
search content: anonymous user and authenticated user
use advanced search: anonymous user and authenticated user

h) Add the following to the bottom of /var/www/sites/default/settings.php

# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('');

# Memcached configuration
$conf['cache_inc'] = './sites/all/modules/memcache/';
$conf['memcache_servers'] = array(
         '' => 'default',
         '' => 'menu',
         '' => 'filter',
         '' => 'form',
         '' => 'block',
         '' => 'update',
         '' => 'views',
         '' => 'content',
         '' => 'apachesolr',
$conf['memcache_bins'] = array(
          'cache'        => 'default',
          'cache_menu'   => 'menu',
          'cache_filter' => 'filter',
          'cache_form'   => 'form',
          'cache_block'  => 'block',
          'cache_update' => 'update',
          'cache_views'  => 'views',
          'cache_views_data'  => 'views',
          'cache_content'  => 'content',
          'cache_apachesolr'  => 'apachesolr',

i) Change the permission of settings.php to protect it (chmod 755 /var/www/sites/default)

12) Setup the /mnt directory for storage, enable cron and create a boot script

Since root storage on AMIs are limited, we recommend using the /mnt directory for storage of Mysql files and the Varnish cache. We will soon have optional instructions for using Amazon's Elastic Block Storage (EBS) for storage.

a) Change the Mysql datadir from /var/log/mysql to /mnt/mysql in /etc/mysql/my.cnf.

b) Change the Mysql tmpdir from /tmp /mnt/tmp in the same file.

c) Change /var/lib/varnish/$INSTANCE/varnish_storage.bin to /mnt/varnish/$INSTANCE/varnish_storage.bin in /etc/default/varnish

d) Make the temp dir for Mysql (mkdir /mnt/tmp && chmod 666 /mnt/tmp)

e) Create a file (/etc/cron.d/drupal) with the following lines (this automates updating the Apache Solr database and other Drupal maintenance:

m h  dom mon dow   command
0 * * * * root /usr/bin/wget -O - -q -t 1 http://localhost/cron.php

f) Set up a one-time boot script by adding the following to /etc/rc.local (keeping "exit 0" at the end of the file):

# Mercury init script; only runs once at first boot.
# Produces /etc/mercury/incep with start time and a log of all bootstrap actions in /etc/mercury/bootlog

g) Create the directory for mercury (mkdir /etc/mercury), add the following to /etc/mercury/ and make it executable (chmod +x /etc/mercury/


if [ -e /etc/mercury/incep ]; then
    exit 0

exec &> /etc/mercury/bootlog

cd /var/www/; bzr merge --force

# Move mysql and varnish to /mnt
# TODO support for EBS and RDS
/etc/init.d/mysql stop
/etc/init.d/varnish stop
mkdir -p /mnt/mysql/tmp
chown mysql:mysql /mnt/mysql/
chmod 777 /mnt/mysql/tmp
mkdir /mnt/varnish
mv /var/log/mysql /mnt/mysql/log
mv /var/lib/mysql /mnt/mysql/lib
mv /var/lib/varnish /mnt/varnish/lib
sed --in-place=.bak s*/tmp*/mnt/mysql/tmp* /etc/mysql/my.cnf
ln -s /mnt/mysql/log /var/log/mysql
ln -s /mnt/mysql/lib /var/lib/mysql
ln -s /mnt/varnish/lib /var/lib/varnish
/etc/init.d/mysql start
/etc/init.d/varnish start

# Update packages
apt-get update
apt-get -y upgrade

# Config Memory
#uncomment the following if you would like to use our script to automatically
#configure apc, php, tomcat and barnish based on the RAM available on your system

# Unset ssh key gen:
chmod -x /etc/init.d/ec2-ssh-host-key-gen

# Mark incep date
echo `date` > /etc/mercury/incep

13) Repackage Your AMI

Here's my basic AMI packaging script based on Eric Hammond's work. In order to make this run, you need to put your certs in /tmp. See Eric's great blog post for more details:

#!/usr/bin/php -q
//This is largeley based on
//Helpful script for bundling AMIs

//Accepts command line args for version-number and "public". If public, it will remove potentially sensitive data.

//Readline input function.

function read() {
$fp=fopen("/dev/stdin", "r");
$input=fgets($fp, 255);
"Make sure to chmod +x /etc/init.d/ec2-ssh-host-key-gen  Continue? (y/n)\n";
$confirm = read();
if (
strtolower($confirm) != 'y') {
"User cancelled script\n");
if (
$argv[1] == '') {
"Specify a version number! (e.g. 0.1)";
$bucket = 'chapter3-storage'; // update w/your bucket
$prefix = 'drupal-pressflow-mercury-jaunty32-'. $argv['1']; // change to start a new line
$exclude = '/mnt,/tmp,/root/.ssh';
$AWS_USER_ID = ''; // your user id (e.g. 5555-5555-5555)
$AWS_ACCESS_KEY_ID = ''; // your access key
$AWS_SECRET_ACCESS_KEY = ''; // secret key

if ($argv[2] == 'public') {
"Marking this as a public release!\n\n";
$exclude = '/mnt,/tmp,/root/.ssh'; // excluding my local user account
`sudo rm -f /root/.*hist*`;
sudo rm -f /var/log/*.gz`;
$logs = explode("\n", trim(shell_exec('sudo find /var/log -name mysql -prune -o -type f -print')));
$logs as $log) {
sudo cp /dev/null $log`;
echo `
rm /etc/mercury/incep`; // clear incep date
echo `rm -rf /var/lib/varnish/*`; // clear old varnish configs
echo "Bundling $prefix\n\n";
if (
trim(`uname -m`) == 'x86_64') {
$arch = 'x86_64';
else {
$arch = 'i386';
sudo -E ec2-bundle-vol           \
$arch                       \
  -d /mnt                        \
$prefix                     \
$AWS_USER_ID                \
  -k /tmp/ec2-keys/pk-*.pem               \
  -c /tmp/ec2-keys/cert-*.pem             \
  -s 10240                       \
ec2-upload-bundle                \
$bucket                   \
    -m /mnt/
$prefix.manifest.xml \
"You can now register this AMI at\n";
"Now chmod -x /etc/init.d/ec2-ssh-host-key-gen.\n";

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week