I am setting up 2 load balanced servers under 1 ip to run my Drupal sites (multisite installation).
The sites will be using a separate dedicated server for the MySQL databases, on the same rack and 100Mbps network for faster access.
All 3 servers (2 app servers load balanced under one ip and the dedicated mysql server) are on the same physical rack.
Several questions come to mind, in particular...
APPROACH 1: rsync /home directories accross both servers.
I was planning to rsync /home directories on both servers, although it's mostly one site under /home and all Drupal sites defined under /home/public_html/sites in a standard multisite installation.
- What happens when a user uploads a file, but rsync has not yet mirrored the directories? Can this be a problem if Drupal is checking for the existence of the uploaded file?
- Should I worry about the cache css files? I'm guessing that if the server does not see the cached file, it will create another, however since load balancing would distribute traffice say 50%-50%, wouldn't this end up generating a new cached file every time it's searched for and therefore hitting my performance?
- Can constant rsyncing be a performance issue? How often should I rsync? every 2 minutes? every 5 minutes?
- Is this the right way to go?
APPROACH 2: nfs+symlink sites directory from one server to the other
After wondering about the issues above, I think I might be better off mounting a virtual drive with nfs to share the /sites directory in one server and have it accessed (as a symlink I guess) by the other server.
This approach however:
- might put an uneven burden of drive read/writes on one server (reading settings.php, css cached, temp and uploaded files).
- eliminates the full redundancy I was planning for as a plus to the load balancing scheme...
(If the server with the physical /sites directory dies, the other server becomes useless... although I could rsync a copy every 10min. to the server with the symlink as an emergency backup which could manually be copied over or have the symlink point to in case of such an event). - Any other issues?
Thanks in advance for any comments and suggestions,
Carlos
Comments
NFS FTW
While the drawbacks you note are real, option #2 is the only one I've seen work reliably, and it's the only one that scales as you add more web-heads. Having a shared NFS where drupal can manage files is the only way to prevent all kinds of weirdness (e.g. trying to post a new node w/an attachment regularly breaking on preview, broken images on the front page of your site, inability to use imagecache, etc).
Generally rsync is useful for deploying code changes from a staging server to the web-heads, but much like you only want one database for content, since /files is also techincally content you only want one of those as well.
http://www.chapterthree.com | http://www.outlandishjosh.com
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Thanks for your comments
Yep, that's what I was thinking.
I'm still wondering if there be any issue I should concern about or check for on option #2?
I will deploy a pilot tonight and see how it works...
con paciencia y calma,
sube un burro a una palma.
con paciencia y calma,
sube un burro a una palma.
Option 2 is the way to go
The best way to do this is with real shared storage, which generally speaking means fibre channel or iSCSI. The first is expensive and the free versions of the second don't work well.
Option 1 is not a good idea. You can't realistically rsync fast enough, and if you have any module that vaguely attempts to work with files, you're out of luck, as most of them will want to work with the file as soon as its uploaded. You *can* work around these problems with the right kind of load balancer, which sets up 'sessions' by IP address. Both pound and varnish will do these IP-based sessions, meaning that a given client will always hit the same backend. This means that a user, while uploading a file, will immediately have access to it, even though another random user might not. Combined with fast enough rsync--say, 1 minute--you could *maybe* get away with this.
However, before you go down this path, remember that you simply can't mathematically get meaningful redundancy out of 3 servers. With only one database, you can't say that a database is any less likely to fail than a front end. The only thing you're gaining is the performance of two front ends (which can be significant). With this in mind, the NFS server from one web server really does have a meaningful impact and is significantly more manageable. It's also the way to go with future upgrades, as you can perhaps one day end up with a server with more drives to act as the NFS server.
When you're doing multisite with NFS, remember that symlinking/mounting /sites can be tricky. If you symlink /sites to an NFS mount, all code versions you're using will all have the same /sites/all/modules directory, which can be a problem if you are running both 5 and 6 (or even different versions of these). You can also symlink /sites to a different mount point for each version, but that can get tricky, making software updates now involve system administration tasks. Our installation mounts /sites to NFS for all versions and uses /profiles/version/modules to do the same sort of thing as /sites/all/modules.
While option 2 is better for
While option 2 is better for the reasons stated above, option 1 is quite easy to implement
All you need is a simple rewrite on webserver redirecting request to the other on file not found (with a protection for infinite loops - tip: use symlink to /files as target).
And now it all works immediately - you run rsync just as often as you can afford without reducing performance.
This solution is usable for 2 web servers and might just work for you up to a point you will be able to afford a proper shared storage.
Thank you all for your comments...
All have been most helpful and now I feel more confident on my choices and options.
I'm almost pretty sure I will go with option 2 for now.
Then in January will move the dedicated MySQL server to a load balanced config of its own, on the same rack.
And then in February will consider a redundant file storage solution for /sites in order to achieve full redundancy...
Since I have to do it all on my own (and own pocket), I will do it one step - and one payment ;-) - at a time.
Will let you all know how it goes...
con paciencia y calma,
sube un burro a una palma.
con paciencia y calma,
sube un burro a una palma.
Mix and match
I'm currently using a combination of option 1 and 2. I'm using rsync to deploy tested code to my webheads and using NFS for all uploaded content.
Using rsync for all code results in a much more stable config, if the NFS server dies the site won't go down, it will just lack some avatars, some photo's, etc. rsync for the code is also quite a bit faster (locking over NFS sucks :) and using it is a breeze once you've set it up properly (ssh keys etc) and created some trivial scripts to rsync and kill -HUP apache to tell the APC cache (in my case) that some .php files have changed.
Using rsync to sync all uploaded content is just way too wasteful, it takes up way too much storage and rsyncing a non-trivial amount of data just takes too many resources, especially if you want to do this every couple of minutes.
Interesting
That's an interesting option. We also have code replicated on each server, but it doesn't help availability, as settings.php is still nowhere to be found if the NFS server fails.
We considered it for our installation, but as it requires a separate NFS mount point for each site, but in previous experience I've run in to trouble with more than 50 NFS mount points (more locking issues under load, for example). How many sites are you running on your setup?
Also consider using rsync +
Also consider using rsync + inotify (available in Linux from 2.6.17 i think). use inotify to watch for changed/deleted/created files in a directory, rsync them using a shell script. As soon as a file is created, inotify runs your rsync script. Works OK if you don't have a LOTS of changed files at once.
As others mentioned, with option 2 and even with option 1 mysql, you are not getting any redundancy, in fact, the reliability of the cluster will be a bit lower than a single server (adding complexity, not adding redundancy).
Also consider using DRBD, which is a RAID1 over network. It works well, but is limited to 2 servers (3 servers for commercial version). I would consider it a best option for you in this case.
When you add shared storage (you don't have to use FC or iSCSI, try SAS too, it's much cheaper and entry-level SAS storages from IBM/Dell/SUN can be bought for approx. $8000 (very rough calculation from Czech Crowns)). When you add shared storage, you will have to solve read/write concurrence. The BEST solution for this is Red Hat Global File System, available commercially or as opensource in most distributions (I really recommend using commercial solution for this as you get proper support, the price for 2x RHEL Advanced Platform is not that high...)
GFS allows you to mount a filesystem from a shared storage on both nodes - read write, so you can add files on both of them and GFS ensures they are visible on both sides. Note: You DON'T get storage redundancy with shared storage.
DRBD and GFS
Although my personal experience with DRBD was not great, it's worth noting that DRBD is capable of running GFS in read/write mode for both nodes, which is only way I know of to do it without buying hardware (last checked ~1 year ago). It's quite cool for that. If you want to use it for real-time data integrity as opposed to hot-spare redundancy (as I did, which is why I stopped using it), though, I would suggest just rsyncing/mysql replicating off of the primary constantly.
GFS is really cool tech, but it's difficult to get running and you really won't get a ton of benefit over NFS without hardware (iSCSI or FC). I would also agree with buying the support, you get 9-5 for a year for like $200 and it's not a bad deal at all, but just use CentOS and call them when you have real problems--almost all of the problems I had with 'real' RHEL were due to complications getting their servers to let me download updates. The support guys won't know the difference.
Lastly, a quick note on Entry-level SAS: If you are really on a budget, you can do the same things you can with those with a Dell 2950 for half as much and decently reliable hardware (but do use two hot spares on the RAID arrray). The difference is that you get to configure a Linux box from the hardware up to do it all, and, ah, there are still bugs there :)
My comment: Yes, DRBD + GFS
My comment: Yes, DRBD + GFS works but you get a lose a LOT of performance. See linux-cluster mailing list for some reports, performance goes down tons of percents.
I need to correct the price of RHEL (as i am working for Red Hat Premium Business Partner :) For GFS, you need at least RHEL Advanced Platform Standard - $1499 / year (save 10 % for 3 years) and you need at least 2 subscriptions, for 2 servers (Yes, they won't know you are switching the license between servers, but we are playing fair, right? :)
Actually, GFS is pretty easy, but the key is that you need fencing device for it and need to configure cluster suite at least a bit. Then it's just a matter of gfs_mkfs and mounting on both servers.
DB does NFS Serving?
I guess the reason you want to do this is because your Drupal sites have lots of heavy PHP processing? You still have single point of failure on the MySQL server so no real advantage from a fault tolerant point of view in your plan.
Given that - did you consider letting the MySQL server also act as a NFS server? Serving up files is trivial in comparison to DB queries and if you have enough RAM in the two web machines then disk access should be minimal anyhow as most things are cached.
If you wanted to do things on the cheap you could even have the two web machines run diskless and boot from the DB/NFS server too.
I have also heard DRDB with GFS will work, but think the other guys have covered the disadvantages there.
Varnish 2.0
I'm just about to start playing with Drupal 6 and Varnish 2.0, which looks like it has some nice improvements over Varnish 1.x. I was wondering if anyone has a recipe for setting up this combo, especially the big issue of cookies, which seemed to prevent Varnish 1.x from actually hitting the cache. According to http://drupalcamp.dk/node/92 maybe someone has it figured out.
This?
Check this configuration, and the patch for it. http://drupal.org/node/147310#comment-996321
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Also please support this
Also please support this issue which eliminates session cookies for anonymous users.
Server Rookie
No one can seem to give me the same answer when it comes to what I need to run e-mail campaigns. I am running my cmpaigns thru a dedicated that I pay for each month. I am using a Windows 2003 based e-mail software that I use from my office. I need to know what I need to do to send ou 50K e-mails an hour. Will a server help me do this? If it will, then what do I need to purchase. Do I need a Linux platform? Do I need to purchase other IP addresses? Do I just need a couple more hard drive and seats for the software and not a server?
Any advice is welcome. Everyone I talked to has a different answer.
P.S. The dedicated server is only used as outgoing e-mails only. THere will be 0 incomig mail.
Just to rule out the not
Just to rule out the not very likely: Do you mean that you are sending the emails from your office via email account on your dedicated? If so the almost certain problem is the connection between your pc and the server.
The more likely: the emails are created on the server and sent - the problem will be the MTA performance (or lack of it) - the most likely causes are: spam/virus filters checking every single outgoing message (do not remove this checks no matter what, or you will soon become a major spam source against your will - just add a very speciffic exception allowing your mailing to bypass it) or the MTA itself (possible causes: no proper message queue, improperly set up concurrent connections, etc). As for accepting emails - do it if only for the proper bounce handling and not annoying spam filters.
50k emails / h is less than 15 per second - an average VPS should handle it nicely.
If you need some more help fell free to contact me (note: I don't do Windows, Linux only)
cluster and MySQL slowness
Hi,
Sorry for hi-jacking the thread, but maybe you guys with clusters have an idea of how to resolve my problem.
The production server is a cluster of 10 machines. 1 takes the incoming connections (and is backed-up by another machine) and distributes them across the other machines. MySQL server is on another machine.
MySQL queries are very slow compared to my local dev server (which is a LAMP stack), details here : http://drupal.org/node/311626#comment-1814828
I find it interesting people
I find it interesting people are using NFS to host the entire Drupal directory, why would you not just symlink sites/default/files to an NFS share on all nodes? I use Capistrano to deploy to my production env and have it symlink the files directory to the shared directory. I then have the shared/files directory as a symlink to my nfs share. It looks like this:
/var/www/sites/mysite.com/current -> /var/www/sites/mysite.com/releases/345090435839
/var/www/sites/mysite.com/releases/345090435839/public_html/sites/default/files -> /var/www/sites/mysite.com/shared/files
/var/www/sites/mysite.com/shared/files -> /mnt/shared-nfs-mount/mysite.com
Where apache uses /var/www/sites/mysite.com/current as the web root.
I think what you describe is
I think what you describe is in fact the most common way to setup a Drupal cluster, though Capistrano may be replaced with custom rsync scripts.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Because NFS is a monster
I really hate NFS... I'm currently using it to have a central upload directory but this is only used for uploads. I.e. in a Drupal environment my current /files/ directory would be an NFS share on all webnodes but when serving content everything comes from a proxy server that doesn't have to go through NFS. Everything besides
NFS has given me too many headaches, crashing, locking problems, concurrency problem, performance issues, security, permissions, etc, etc, etc. More importantly it introduces a new single point of failure and running a redundant NFS cluster is something that I really don't enjoy doing :+
Yea I agree NFS has issues,
Yea I agree NFS has issues, there are other solutions like http://wiki.lustre.org/index.php/Main_Page
The reason people use NFS is because its very easy to configure and use, and really it works for 90% of drupal sites (like this one) out of the box almost. Using squid infront of your cluster really reduces the amount of traffic that NFS has to handle to an insignificant amount anyway, so I don't really see huge problems using it.
NFS problems
I've also had a lot of NFS problems. Sometimes just copying a live site to devel is enough to crash or lock the service, which takes down the whole site (ssh-ing into the nfs box and rsyncing from there crashes less often). I'd recommend investigating similar options before trying NFS.
Also, I'd caution against sharing code this way. It's not hard to sync code between servers via source control. Only share the files folder(s), or otherwise the minimal number of files. Your multisite install might make it a little less convenient if you're always adding new sites, but still only share what you must. You can change the files folder to not be inside the sites/[domain] path to cut down on the number of shares you need (ex: files/[domain])
Once you get to the point where you're only sharing the files folder, you should really consider a CDN.
Ken Winters
APC + O/S Cache?
Does this really make a difference? I mean once APC has loaded your scripts, and even then, probably via the O/S cache any underlying file system accesses should be insignificant.
Since NFS isn't 100% stable,
Since NFS isn't 100% stable, limiting the number of files shared decreases the opportunities to crash it.
Ken Winters
My vote: csync2 + lsyncd
I see this thread is a couple years old now, but an option has opened up for serving Drupal files across multiple servers that we've had success with. Details:
http://bshensky.livejournal.com/#entry_24815
lsyncd is a daemon that leverages the inotify capabilities in the kernel to run programs of your choice on changes to the filesystem.
csync2 is a daemon that guarantees fast and reliable file synchronization among participating hosts in a LAN.
The two, when fused together as described in my post, allow our "node 1" to behave as a master file "source", and nodes 2-N automagically get updated from node 1.
This works best (only?) for file deployment. For storage and distribution of content uploaded to the site by the Drupal site itself? Well, that has been a challenge of a different sort - getting files that have been uploaded to slave nodes to propagate to their bretheren and master nodes remains a challenge. Looking into GlusterFS serving up sites/default/files at the moment for that.