Posted by gateway69 on January 19, 2012 at 10:21pm
Hi, Im finally thinking of taking the plunge and getting rid of my LAMP stack in favor for nginx.. I was wondering if anyone had a decent install script for ubuntu 10.04 64 bit, I have a linode instance up and running.. If not is their a decent walk though to get nginx, pressflow, varnish, memcache, apc and all the other bells up and running ?
Sorry if this was asked here before might be worth a sticky if their is such a walk though or script.
Comments
First
you need to ask yourself if you really need all that.
The installation is straightforward. There are packages for Debian and they should be in Ubuntu also.
There's no script just install the packages and raise here any issue you might have.
from everything I hear with
from everything I hear with Nginx vs apache is the fact that when you get a lot of concurrent users logged in that well nginx blows apache away and uses less memory.
On a personal project im working on a huge social craftbeer site where we will have quite a bit of logged in users and anon users.. so I just want to make sure I squeeze every bit of performance from memory, connections, etc.. I know its a broad thing to ask for but im hearing good things about nginx..
Yes
(empty)
Yes
It's just that you've thrown a lot of stuff. Do you really need Varnish? What about memcache? Couldn't just APC be enough. The general attitude should be to have it as simple as possible.
Varnish for anon users and
Varnish for anon users and puts the load off of apache , not sure if its needed with Nginx, hence why im saying im the newbie here and asking :)
memcached for any db queries..
You can get
drupal caching to work with APC you just need to install cache backport on Pressflow. It will save you memory and be faster than memcache. It will be local. If you want to have a distributed cache then use memcache over TCP sockets.
Varnish may or may not be necessary. Nginx has a cache. And there's the concept of microcaching. I find the all expiration logic thing very complicated and a throwback to the days when caching was expensive.
BOA and be done with it
In the spirit of keeping it simple, why not install BOA and then learn from that. It is a LEMP stack completely optimized for every popular VPS environment.
On Linode:
1 - Build a fresh Ubuntu 10.04 install (no extra features or packages)
2 - Change the host name to match your DNS records or create a reverse record for the current hostname on Linode
3 - Go to: https://github.com/omega8cc/nginx-for-drupal/ and read about all the awesome features in BOA and install the stable version
4 - Spend time with your family!
Peace,
Michael Clendening
Thanks but isnt BOA overkill
Thanks but isnt BOA overkill for a single site, I hear and I could be completely wrong that BOA is used for a drupal setup that runs multi sites?
Aegir (and BOA) is good even
Aegir (and BOA) is good even if you have just one site to manage. Of course unless you prefer to work too hard when installing/configuring the server and later when you need to test new stuff and quickly run tests and migration/upgrades in minutes/seconds. I mean, if you prefer hours of tedious tasks over automation, then sure, forget about Aegir/BOA ;)
I don't think
that's the question to ask. The question is what's the price that your're willing to pay for having full control of your setup. Any type of automation or wrapper provides benefits. But there's no free lunches. If you're ok with having it automated then go ahead.
Also I wouldn't use Aegir for a single site. It's more code and more complexity: hence more fragile. But it's up to you. Aegir is nice for multisite. But regarding that we already had a discussion here about that. And I agree with the Pantheon people on that: multisite is too much trouble for little gain. And a throwback to the heydays of shared hosting. As usual Caveat emptor. It's up to you and only you.
I disagree completely.
I disagree completely. Automation is not an opposite to having full control. We have literally hundreds of hosted Aegir instances used to manage just one live site (plus one or more dev sites), and all users are saying the same - they would never go back to manage their site without Aegir.
And it is not a matter of choice either.
It is a matter of who you (the site owner/admin) are. If you are a geek type and you simply like to do the things your way, and you enjoy keeping the mythical 'full control', then sure, you don't need anything like Aegir or Pantheon - but not because those systems are fragile or add complexity, but because you simply prefer to manage the stuff your way, probably on command line, without any extra layers/UI etc.
And that's fine.
Just remember that there are many people who have no time and no interest in learning things interesting for geeks, they just need tools to make their work done and this doesn't make the tools better or worse, or complex or fragile - rather simply useful - but for them, not for you, if you are a geek type user.
Epiphenomenon vs. the phenomenom
Grace I think that you're mistaking the epiphenomenom by the underlying phenomenon. It's logic. It could be a preference or a psychological thing but that's an just a lateral aspect . The facts are:
On a complex system introducing more subsystems and moving parts makes it more prone to failure. The more fundamental
a certain part is the more the failure mode will become an exponential curve with rare extreme events.
The more simple and more control you have the more you can sustain the brunt of those rare events.
It boils down to a question of economics. How vital is for your business that the site runs smoothly and that you're prepared for
malfunctions? An example I gave elsewhere on g.d.o was the fact that Amazon business logic doesn't run on EC2. Why? Because it's
critical for their business to have full control of the setup without any cookie cutter like layer that makes it more prone to extreme rare
events that are economically disastrous.
Either you're unaware of this or you can just live happily with rare events happening because your site is not mission critical.
The geek vs. normal person interpretation is just skimming over all this IMO. There is no this is the Way. There are many paths. You should choose whichever option makes more sense to you. I'm not suggesting this or that. Rather Im suggesting that you choose but bear mind that there's no free lunches and to quote an old economist there's the seen and the unseen. Just because you're not aware doesn't mean it isn't there.
I know that were veering of course and this is not the best venue for this type of argument. Anyway I believe Nginx is an example of something that makes your setup more robust, hence, less fragile. And this is the reason why we are here :)
I think the Aegir system is
I think the Aegir system is made for intelligent people and people who need help learning.
For instance, its like someone took thousands of hours of their experience and made something that worked very well. All I have to do to install it is change my email address on a script. Wow.
Then, I can reverse-engineer it, to fully understand it. It takes time, but at least I have something in the first place that makes sense and is well-tested.
I have also sampled various nginx recipes and how-to's, the BOA scripts are beyond amazing - i.e. among the coolest things I have ever downloaded for free. Sure, there are problems here and there - but then you get to learn about filing bug reports, which help the creators and everyone else!
Anyhow - the world is big enough for all of our ways!
Aegir is no panacea.. I did
Aegir is no panacea.. I did quite a lot of testing and it broke a number of times which illustrates perusio's points about more complexity making it more fragile..
Some issues I had in particular was the way it handled static IP's and SSL hosts.. Another issue was the fact that its only really been created to run under Apache/Mod_php, when I tested it under php-cgi and fcgid it exposed the database credentials in phpinfo output.. They have designed it to scrub these but it only works under mod_php.. Maybe not a critical issue if your site is locked down correctly but still something to be aware of.. (not tested with the Barracuda scripts because by the time I found out about them I had already decided not to use Aegir)..
Also keep in mind that these complex setup's aren't a "set and forget" scenario.. You have to make sure they are always updated and an update to Aegir or whetever system could break everything so the less there is to go wrong the better..
However I do agree with "the world is big enough for all of our ways!".. :)
You evaluate the risks vs the benefits and make your own choice..
Personally I have always come back to the command line and a few bash scripts to automate tasks (site creation/deletion, checking permissions, setup sftp jails, etc..)..
Please read my comment again
Please read my comment again as you seem to read things I didn't wrote ;)
I just tried to explain that you are wrong by using your opinion and personal preference about Aegir as a basis for recommendation you are addressing to someone (gateway69) clearly looking for a tool and not for a geek experience.
I didn't wrote that tools like Aegir or Pantheon don't add complexity or that they don't make entire setup more "fragile". They do, of course. But it doesn't matter in this context and for any person looking for a tool and not for a geek experience. That is the point. Just read the subject again: "Nginx Newbie here.. any good script installer for linode?"
Also, I didn't suggest that it is a strictly personal/psychological kind of choice. No, I only tried to explain that your personal preferences, experiences (or lack of them) are something what really determines your choices, but these preferences and choices don't make the tool better or worse or not good for particular job for someone else.
When you say "Aegir is nice (only) for multisite" or "multisite is too much trouble for little gain" or "a throwback to the heydays of shared hosting" you are simply wrong, sorry. It is your opinion only, determined by your preferences, but you are trying to show it as fact - also by including "Pantheon people" opinion.
Opinion never create facts, and we should always try to keep it clear, that what we are describing is our personal preference based on our own experience, needs etc.
It always helps when we share our experiences, our passion, so let's do more awesome stuff in the Nginx context.
I'm personally very grateful for your passion, inspiration and your great Nginx related work. Thank you Perusio!
@omega8cc - Fair point.. As
@omega8cc - Fair point.. As the saying goes.. "each to their own"..
I will say though that because of the fact that there is almost never one single way to do things people's opinions and personal experiences are indeed valid for helping others, newbies included, to make decisions on how they solve their problems..
Hello
Grace. It's just that you placed so much focus on geek experience that all other things looked small by comparison. Like I wrote above there's no this is the Way. Rather what you can do is always to say:
I gave my sincere opinion. It could be wrong and not going to set off in tangent about nihilism, pessimism and will to power ;) Anything that is not trivial has much more than facts. Opinions matter. They encapsulate your experience in a intuitivel way. Most people ask opinions and in fact being opinionated is tantamount for going further and learn, with your mistakes and those of others. I have no problem in expressing my opinion, I never tried to pass for facts. Because facts are nice but they're also very elusive and it's only when something is dead that we can speak of facts. Until that there's mostly opinions, and to each is own. Is precisely that diversity that makes it rich and stimulating. There can only be learning when there's tension and two people disagree. If we all thought the same then it would be boring and a tell tale sign that this topic has ceased to stir passions and is the home to boredom. Gladly we don't have this here. This is a diverse enough group that we can provide divergent advice that in the end still conduces to a good solution, no matter what.
Update: Today I've been enlightened by a friend on Aegir. In fact I stand corrected and I now see that most shots at it come not from any inherent flaw but rather from mere commercial motivations. Pantheon and DevCloud see it as competition.
António, I agree that it is
António, I agree that it is (very) good to disagree and still discuss to explain why we disagree, as it is probably the only good way to better understand each other.
I hope I explained my point of view and now I also better understand your point.
Final digression now: seriously, I never expected that anyone could see Aegir as a competition for Pantheon and DevCloud, but who knows? They are partially closed systems, backed by big money, while Aegir is 100% open source effort, and we (Omega8.cc) actively encourage people to clone us and start their own drupal hosting, leveraging Aegir and Nginx power with the help of our also 100% open source BOA stack. So maybe we should feel a bit like Android rebels, if they perceive themselves as an Apple or Microsoft? Just kidding! :)
Hmm
This has given me some ideas. We should strive for a DrupalFreedomCloud in the manner of the Freedom Box where people could spin their own drupal cloud. This would be a modern bleeding edge thing. Not some legacy setup marketed as bleeding edge. For example I've been told that AcquiaDevCloud still operates Apache 2 prefork with mod_php as the PHP processor. It seems that they don't trust fpm or cannot make it work reliably. Which is really strange since wordpress.com runs on Nginx + php-fpm.
There's market for something truly cutting edge, modern, light, highly-performant that you could spin to your liking without having to commit yourself to one vendor and this introduce a fragile single point of failure. This would include failover and CDN like capabilities out of the box. It would have an option to use Aegir or just leave it in the CLI.
This would be free as in freedom and could foster some enterprisign folks to start a business. Why do we have to accept that the drupal cloud market be dominated by two players? It's bad for drupal and for innovation in the hosted web app space.
Just an idea. I really think that this is an idea that has potential.
Oh they do trust in FPM.
Oh they do trust in FPM. Acquia uses nginx + fpm for their projects (not just the big ones). I know that.
Maybe they still think apache is more maintainable for the dev cloud, though... Which brings us to what you said: "cannot make it work reliably."
Sounds awesome. It's not that easy to find people with your level of knowledge on nginx/drupal, though... Think about that for a minute.
This is not something that can be achieved with mere good intentions/money, different than most other businesses.
Oh, this and this sounds like
Oh, this and this sounds like a subject for another group post... no?
This sounds like a good idea
This sounds like a good idea to explore.
After all, we never accepted, and even never cared about this kind of domination by one or more big players. There is a room for all of us and what we do is based on really simple recipe - make it awesome, so people will love to use it, make it free and 100% open source, without any proprietary extras/specials built-in and locked/closed. And it works great, it doesn't require big money (because it doesn't approach high enterprise market), and it gives people a choice. There is no domination possible even, and that's good and healthy to everyone.
Fork us!
Solly
for the thread hijacking @gateway69. Anyway. Definitely. It would be inspired by BOA and Barracuda. Here's some loose ideas:
Use the debian package system instead of a shell script.
Use PHP 5.3.x - always the latest package. No support for D5 or previous, hélas.
Create a repo with properly signed packages of those packages that Debian/Ubuntu/Mint doesn't provide.
Use something like gdsnd for failover support and CDN implementation.
Have security considerations in mind: use Samhain for example.
Use the Nginx cache as the basis for caching setups with microcaching.
Lua as an embedded extension language for server side scripting.
Proper firewalling: run sshd on non standard ports, block all ports except 80, 443, ssh and mail ports. ssh for root disabled.
Access only with public key.
Ability to capture the config as code so that it can easily be replicated/restored.
Starting with Nginx and the microcaching setup is the way forward. Is also the thing that is more advanced. @Fidelix I think we should create a wiki page and detail the first steps regarding Nginx + caching. Perhaps we can have something to show for the drupal dev days.
For search
Use ElasticSearch vs. Solr.
Update: Today I've been
I'm certain that there is nothing but admiration and appreciation for Aegir in the offices of Acquia and Pantheon. I work at Acquia, so I do have first-hand information about this. I also work closely with anarcat as we maintain the Drush project together. Furthermore, Josh and David and the rest of the guys at Pantheon don't take shots at other open source projects either. I've worked closely with them on numerous projects like Field API for Drupal 7 and managing this very web site (groups.drupal.org). They don't operate like that. Acquia and Pantheon are even quite friendly despite competing for the same customers. Really, we are enjoying one big open source hippee party and posts like yours spoil the buzz. Please refrain from stating speculation as a fact. It is quite easy for a reader to trust your words as fact when the opposite is true.
Here is my very basic install
Here is my very basic install instructions for getting nginx and php-fpm onto ubuntu 10.04..
Install PHP-FPM
aptitude install python-software-properties
add-apt-repository ppa:nginx/php5
aptitude update
aptitude install php5-fpm
Install nginx
add-apt-repository ppa:nginx/stable
aptitude update
aptitude install nginx
You will probably want to add php5-cli and other bits but that should get you started..
As the others have said start with the basic setup and add other components (varnish, memcache etc..) as the need arises.. Also when building your site test the performance of different ways of doing things and the modules you cloose to use.. Remember nginx is very different to apache and takes a bit of learning.. It has caching mechanisms built in which might mean you don't need varnish which frees up that memory for other things..
I've thought about doing
I've thought about doing something like this, and now that a new long term support version of Ubuntu is going to come out it's even more tempting.
I already have all the notes that I use to configure a new server, and I could convert them to a series of instructions.
But then I think about taking it to the next step and making it automated. One way to do this is in a bash script, but there are also devops tools designed specifically for this task. Unfortunately I have no experience with Puppet or Chef or bcfg2 or any devops tools. I know Pantheon went with bcfg2 for their original setup, but it seems like the devops landscape has been evolving quickly.
Does anyone have any experience with them?
On Debian
and Ubuntu you can use tasksel to create an installation task and have it setup all the needed packages.
No need to use a Puppet, Chef or any other thing like that. They do come handy but only when you have a certain number of machines to manage.
I've been thinking on starting a small tool that uses Nginx and Lua for doing that leveraging ZeroMQ. Needs thought and work.
I know its the most
I know its the most simplistic but for me a bash script would be my recommendation.. Easy to use and run, no additional software requirements and easily customisable if the user so desires..
Of course omega8's barracuda script might be a good starting point for lots of handy tools.. A stripped down version that just installed mysql, niginx and php-fpm and a basic Drupal config would be useful for anyone starting out.. :)
Yeah, that is exactly like
Yeah, that is exactly like BOA was born, but then it has grown into a big monster :)
See the original, pre-BOA stuff in someone else early fork: https://github.com/tegatai/nginx-for-drupal
And the original thread: http://groups.drupal.org/node/26363
Take a look at cdist:
Take a look at cdist: http://www.nico.schottelius.org/software/cdist/#index1h2
It looks like promising alternative to bcfg2, chef and puppet.
However, for minimal install, a bash script would be probably a good idea, as it is easily portable also to other, non-Debian systems.
I love how these threads are
I love how these threads are so timely. This is actually on my list to start this coming week. My plans are to build a Puppet managed system that would span various distros. My particular business need is for a managed hosting platform focusing on stability and security. Between mig5, omega8cc, Aegir crew, and numerous other sources, the hard work is already done. I was just going to stitch it up with puppet, share it, and hopefully improve it to the point that I can roll it out in a production environment.
Moshe is right. There is plenty of room in the market, and ALL of the players have a tremendous amount of integrity as well as a really good understanding of how to work on and with open source projects effectively.
Im giving this a shot
Im giving this a shot https://github.com/perusio/drupal-with-nginx scroll down to instructions.. I personally like to usually build my installs by hand a few times to know what im installing and how things act, so while this pain a few times sucks it does help me to get informed and be able to test things instead of just voodoo going on in the background.
btw i appreciate everyone's comments ..
Ok peeps, I didnt mean to
Ok peeps, I didnt mean to start a flame war between whats better than what, I was just wanting to get introduced to looking into Nginx and what is offers over a pure LAMP stack.
question: would most people agree for a high performance site with lots of anon traffic and actually logged in users is nginx php-fm a better solution than apache.. while apache can handle a lot it seems nginx can get you more performance on the same hardware. While a bit more of a pain to set up. I'm going to port over a site we are about to launch to nginx and run some tests to see what works better for the vps im using..
anyhow I do apperciate people who take the time to write scripts and setups, but I always like to play first, yes spend hours doing so, getting frustrated, Googleing around but I also like to learn by doing .. I did try BOA and had it up and running wasn't sure where it installed all its stuff and sorta felt like a black box to me...
Ill report back on my findings if anyone cares, but now its time to brew some beer!
I honestly think nginx is
I honestly think nginx is MUCH easier than Apache.
But yes, at first, very different than apache. If you think that just by being different is hard, then yeah, nginx is EXTREMELY difficult.
For anonymous traffic you
For anonymous traffic you want either a reverse proxy like Varnish or nginx or you want the Boost module.
If you go with the Boost module you want nginx for serving the static Boost files.
If you absolutely want Apache, either for comfort or for a certain feature or if you absolutely want php 5.2 then do either nginx or Varnish in front of Apache.
you can run 5.2 behind nginx,
you can run 5.2 behind nginx, you just need to pin it to an earlier package set, tho some things like xcache become really hard due to dependancies
http://dgtlmoon.com
You can, but it's a fair bit
You can, but it's a fair bit of extra work.
There is no "official" php-fpm for 5.2. There is a patch created years ago by a dev who abandoned the project. It has no developer, no security updates, no bug fixes, no evolutionary development. You have to compile it manually into 5.2, which itself is barely supported, and if you upgrade to a new 5.2 with security updates, you can only hope that the old patch still applies and compiles correctly.
Now php-fpm on 5.3 is an official php.net project with assigned devs, a bug tracker, a mailing list, and new features with each release.
I know php-fpm on 5.2 has a good track record so far, but I still recommend running apache2-prefork behind nginx if you need 5.2. It's a stable, supported solution that addresses most of the shortcomings with using apache2-prefork alone.
I just checked your site
I just checked your site (which has nice info, btw) and I see you're talking about php-cgi with spawn-fcgi, so no process management. Then you're talking about configuring some additional daemon like Monit to monitor processes? Still not a great option.
Sure, it is not in the 5.2.x
Sure, it is not in the 5.2.x core, but it exists and was updated for every 5.2.x release, up to last 5.2.17 on Jan 11, 2011, and still works without issues - http://php-fpm.org - for those, who still have to support Drupal distros/contrib modules not updated to support PHP 5.3. Believe it or not, but many people still use D5 and they are happy that there is a system like Aegir which supports automated upgrades D5->D6->D7 out of the box, so PHP-FPM 5.2.17 is still useful for them.
That's all true, and if
That's all true, and if you're in a hosted environment where someone knowledgeable like yourself is going to maintain the php-fpm 5.2 backend with security patches and make sure the php-fpm patch works and compiles, well...then you know the risks and benefits.
For folks setting up their own VPS who must use php 5.2, they can install Ubuntu 10.04 LTS and use the official Ubuntu apache2 as a backend while Ubuntu guarantees to maintain security patches until 2015 straight from your package manager without any compiling. Then use the latest nginx from a Launchpad repo as your front end and you've solved 95% of our gripes about apache2. You get fast static file serving, predictable memory use, guaranteed compatibility, etc, and you never have to compile anything or track any security updates.
That's my recommendation, anyways, for security and stability and convenience and compatibility.
I glean from your comment that Aegir would need new code to handle nginx+apache2? Well that's unfortunate and certainly something to take into account.
I count 26 security patches
I count 26 security patches applied to php5.2 since January 12, 2011 that won't come from php.net.
http://changelogs.ubuntu.com/changelogs/pool/main/p/php5/php5_5.3.2-1ubu...
I agree that it would be best
I agree that it would be best to just drop support for PHP 5.2 and thus be able to install everything from supported packages only.
We didn't have this option in the BOA stack previously, because we started with Apache-free approach and some older distros supported, and the list of supported distros still includes full range: Karmic, Lucid, Maverick, Natty, Oneiric, Lenny and Squeeze. We will have to drop support for old stuff at some point, of course, to stop maintain services compiled from sources.
While Aegir supports both Apache and Nginx (but not in parallel of course), it assumes that it is enough to restart/reload Apache or Nginx when it performs any changes to sites vhosts, and doesn't really care how the PHP backend thing is done and managed, so I think it could work also with Nginx+Apache as a PHP backend, probably with some minor tweaks only - it is not a problem to even add Apache backend reload command when Nginx is used as a frontend.
My tests showed that on the
My tests showed that on the same virtual server nginx delivered a single static file request about 50% faster than Apache running MPM-Worker (threaded model).. Scaled up for multiple requests saw this improvement continue..
I found that there wasn't much difference in the php performance when running php-fpm under nginx and using mod_fcgid to access php on Apache.. Again single request.. Multiple requests would depend on how you are managing and limiting php processes..
I found php-fpm far more configurable to your needs.. You can have a single shared pool for multiple sites (low memory requirement) or you can isolate a site in it own pool (with the corresponding memory allocation for php).. You can tune the spare threads and the maximum threads meaning you have full control over whats going on.. APC apparently works with php-fpm as well but I have't tested it yet.. APC doesn't work with php-cgi and fcgid.. Php-fpm stays running as its own service where php under apache is spawned and terminated by apache (depending on the configuration)..
In terms of memory footprint nginx is smaller than apache but php processes must be considered in the calculation also.. There are so many variables to this that you would have to calculate this for yourself.. As load increases the footprint of nginx stays fairly static compared to apache that rises..
Since most pages are php with a number of static files the nginx setup will see performance advantage over apache..
If a significant number of anonymous visitors is expected then as mentioned the boost module seems to be the ultimate weapon.. I have seen reports of 1500+ requests per second from a boost enabled site when running under apache..
Its not so easy to speed up authenticated users.. From what I have seen the order of drupal optimisation is as follows..
There may be other things and you may wan't to use a different order but that's up to you.. Wish I could find the screencast where these options were all examined and they showed the relative performance improvements..
Remember that nginx has caching abilities so might be used instead of varnish for all but the highest traffic sites.. Also nginx can be configured as a load balancing front end (its what wordpress.com are using) to multiple backend web servers..
Hope that helps..
Try
using the
open file cachefor faster static file delivery. Next step would be to proxy the delivery to another host and tune the proxy buffers.you need nginx on linode
you need nginx on linode because the RAM allocations are small and dont really suite apache2, works really great under nginx tho.
theres many fastcgi/nginx tutes out there, make that your first port-of-call
http://dgtlmoon.com
thanks, yea about to get it
thanks, yea about to get it installed :) recommend any other hosts in the same price range?
wow what a thread everyone..
wow what a thread everyone.. maybe this should split off from my what I wanted in the original post.. btw I hear that this strife between paid services and open source is quite big in the drupal community, its a shame that stuff like this has to happen but oh well.. im still on the nginx test bandwagon I have added percona mysql to the mix as well...