scalability

I just wanted to post an announcement that we've finally gotten out a point-update of the Project Mercury AMI. Just in case you were wondering if this project would continue, it will! I've been really excited and encouraged by all the positive feedback so far, so keep your ideas and questions coming. The 0.51-Alpha release includes a number of bugfixes and improvements, most notably it:

• Is based on the latest Pressflow including Drupal Core 6.14 and Simpletest 2.9
• Fixes the self-update process to merge correctly and pull from Pressflow's lovely new VCS home on Launchpad
• Includes the rc1 version of cacherouter
• Fixes postfix and s3 metadata issues so there's now a working MTA out of the box

Most importantly for people considering this stack for production deployments, we're now bundling 64-bit images with every release. The quickest way to find the AMIs is to keyword search for "mercury" in your favorite EC2 console. More information and AMI ids are below the fold. Let me know what you think, and what you'd like to see next!

These days I've added a feature table to Caching: Modules that make Drupal scale, this group's wiki page for comparison of performance and scalability modules, started by Mike Carper some time ago.

This follows other Drupal module comparison pages, trying to make it simple and clear for most normal cases, so that a majority of users can easily understand the basics and use them. Anyone can naturally share additional info. Let's hope the wiki will be useful for Drupal users.

Today I'm glad to announce the latest release in this line of AMI development. This update solves a number of issues and moves us one step closer to a sable beta release. The current AMI ID is ami-c353b2aa, and you can find this AMI by searching for "chapter3" or "mercury" in your AWS console.

For more background information about this project, see my initial g.d.o post and my blog post announcing the initial release.

Below you will find the notes for this release. Also in this post I will include a development roadmap, as well as some more explicit explanation of the techniques I'm using for making the AMI work out of the box.

The BoF notes from DrupalCon DC here: http://groups.drupal.org/node/19737 mention that "When you have more than 100,000 contacts, it makes sense to look into other systems. Example: issues with Oregon Democratic Party site". I'm interested in knowing more about the question of scalability with CiviCRM.

In my never-ending quest to run larger and more complex drupal stacks for more and more users, I'm starting to hit the wall in what I'm able to accomplish with good old Siege, which has been my command-line tool of choice for benchmarking and performance testing for the past couple years. In particular, it breaks down too often in high-load simulations, and doesn't allow for any complex multi-threaded testing, making it very difficult to model near-reality user scenarios like "10 logged in users + 100 anons".

Lately, I've been getting into Jmeter, which has a lot more bells and whistles -- including a GUI! -- and which I think can offer a lot to Drupal developers. However, their basic web test plan barely scratches the surface of what's possible. With the right configuration, jmeter can effectively simulate complete user-behavior patterns like logging in, posting comments, etc.

I'm just getting started, but am curious if people "out there" are already way ahead of me, or if not if folks are interested in seeing the results of my testing work?

Overview: Drupal Authenticated User Scalability

I'm new to the High Performance group and I'm interested to find out if people here have run a site with the following stats:

• 4,000 simultaneous visitors, up to 500 who are logged in at once
• approaching 2 million comments
• 200,000 forum posts
• 30,000 registered users

In my never-ending quest for greater Drupal Glory, I've been spending the past year boning up on the various ways to improve site performance and address issues of scalability. Today, doing some noodling with Amazon EC2 instances (and remaining unconvinced about their raw performance as potential master database servers) I had a thought:

What would it take to extend the static node cache in node_load() beyond the individual drupal bootstrap?

Like just about everyone else, I've been loving how much memcached helps speed site performance. It simply rocks, and everyone looking to reduce server load and speed page responses should be looking into it. One of the better things about it is that it can store and return data objects natively, meaning not only are you letting PHP pull something out of a lightning-fast memory cloud, you also don't incur the CPU overhead of having to unserialize() a string into an object or array.

This let me to my thought. If you want a massively scalable interactive drupal site, you need ready access to tons of nodes. Inevitably you will hit the wall with logged-in requests for these from your database. But what if we were able to take the performance boost we get from node_load()'s static cache, and make it work persistently across an entire site, rather than just for one pageload?

Hi All,

I'm using memcache module for cache tables.

Now I am interested in doing some testing of using memcache for sessions.

In production we are using RHEL and CentOS in testing so for setup of memcached I'm using the start-memcached and memcached init scripts offered here:

http://www.dev411.com/wiki/Memcached_startup_files_for_Red_Hat_(RHEL)

This uses a /etc/memcached.conf which looks something like this:

-u apache
-d
-m 32
-l 127.0.0.1
-p 11211

Based on info contained within this thread:

Hi All,

What (if any) best practices are there for achieving a configuration where the cache tables aren't writing to the default DB?

My specific concern is related to http://groups.drupal.org/node/12890 (MySQL Binary Logs of Death). Would like to write the cache mutator statements to a different DB so they aren't included in the binary log.

We're running 6.8. Using APC for opcode cache.

It seems like the options are:

• set different DB/tables in place of cache.inc. Specifics?
• use Cache Router. Specifics?
• use Memcache.