GlusterFS and Drupal 7 horizontal scaling

what are you trying to accomplish. To be honest I see very little upside on using something like GlusterFS for your site. The module files are static in the sense that unless you upgrade modules or install new ones they stay the same.

For the files directory, using a distributed fs is an option. Other is using NFS and/or something like lsyncd which is scriptable in Lua.

No need also to use Varnish, if you're using something like Nginx you can stay in the wonderland of evented loops and use the Nginx cache for whatever type of assets you want. No need to introduce more moving parts and placing a threaded engine in front of an evented one.

Elaborate on what you pretend do achieve. What's your requisites and we'll be able to help you better.

In the topic of horizontal scaling. Using a load balancer is an option with several cheap backends. You can use, e.g, Nginx proxy cache to have a central cache and distribute the load on several backends.

That's correct - there is no state stored with the module or theme files. Database handles it - and typically sites/all/files is only static assets.

Apache should not be able to write to any folder except the sites/all/files, which is what prevents anyone from doing that.

Any NFS tutorial should work - nothing special, just create a share and mount it at the correct mount point. Just make sure to follow guides on optimising it.

is for using when you want to have replication. NFS doesn't replicate. It justs allows you to mount remote filesystems in a given machine. It's the easiest way to setup a central asset server and a bunch of frontends that use the same drupal filesystem. But bear in mind: no replication.

GlusterFS is nice if you have a NAS or a any type of storage cluster. If you're going to host a lot of sites or if your data has very demanding requisites, then it's a good option. For most people out there IMO it's an overkill.

In my sites I never use the /var/www directory. I run all sites under a regular user account and just give permissions to the server user for writing in the files directory. I want to have the minimal interaction possible with the root account. Setting an Expires or Cache-Control header doesn't provide any server side caching. You're totally dependent on the client side cache policy.

Most of the configs are on the db, although if using features or CTools you could have a lot of stuff in code, as long as you don't alter the features being provided, which in features parlance is referred to as overriding a feature .

We've got a similar set up, and are using NFS to share the entire drupal directory (code + files directory) between instances in our cluster.

I was expecting that with APC, NFS shouldn't really result in any performance hit for dynamic pages, since all the code should be cached in memory. However, running synthetic benchmarks with AB shows a major performance hit when loading a page (just the dynamic HTML). When accessing PHP off of NFS vs "local" disk (EBS volume on AWS), performance is 25-75% slower (depending on the number of simultaneous connections varying from 1-10).

Any thoughts as to what might be causing this performance hit? APC has plenty of cache, and is set to stat=0. What else might be at work here, and how do we go about identifying (and fixing) the bottleneck?

We were also looking into GlusterFS, since it seems that beyond the redundancy advantage that it provides (single point of failure at the moment with the NFS server), it seems that it might also help with performance if we chose to replicate files locally rather than server them over the wire. Thoughts on this?

Thanks!

We were originally planning to just host the files directory on NFS and setting up a separate system to push code updates to the machines in the cluster.

However, we're using Aegir to host a number of sites and it's actually quite a pain to carve out the files directory for each hosted site and host only that directory on NFS.

So hosting the whole install on NFS was the "quick win" solution to kill both issues with one stone. No need to carve out the files directory, and no need to write something new to manage code updates.

But like many "quick wins", the win isn't as big as we had hoped in and we pay the price in terms of performance. (Not very clear as to what APC / PHP / Apache is actually doing in terms of accessing code files, but benchmarking would seem to confirm that even if the bytecode is cached in memory, there's still file access going on for each load)

So we're now back to square one. Either we come up with a better performing alternative to NFS (does Gluster qualify for what you call a real distributed system?), or we go back to rolling a full, if complicated system to host the individual file directories via NFS and push code updates to each machine in the cluster.

Thoughts?

Thanks!

Thanks for the suggestions!

CacheFS looked promising, but doesn't seem particularly well supported, and I couldn't get it working in Centos (kept giving me Kernel permissions errors).

I looked at lsyncd, but am not sure how easily it handles two-way synchro. If I have a cluster with 3 web servers, each of which can update the Files directory, will lsyncd be able to determine, on each machine, whether there is a need to sync our own edits to the other machines (or a central server), and vice-versa, whether there our edits on other machines which need to be synced from the server (or other machines) to the local instance? Rsync seems really best suited to one-way changes, such as updating the code base, and I've known it to run into issues if changes have been made to both the local and remote targets.

Looked briefly at DRDB, but this seems more like a solution to provide networked redundancy to a "locally" accessed array, but is not specifically designed to providing "common" access to files to 3-4 different machines in a cluster. Perhaps I'm missing something though.

Looking again at GlusterFS, not sure if the performance improvement is really that much greater than NFS, as it seems to also come with a lot of overhead.

CacheFS would probably have been ideal, had I been able to get it working, but right now I'm leaning back towards carving out the Files directory and hosting that on NFS, and putting together an lsyncd or rsync option to keep the codebase up to date.

Have you taken a look at something like Capistrano? I use it in my production environment to push code to multiple load balanced apache servers. It will pull from something like a git repository, setup any symlinks you need for file directories, and then update the 'current' symlink across all the servers at the same time. It works flawlessly for me. The file directories still need to be on a shared FS like NFS, but at least all the code and regular static content is on a local fast FS on each server.