Barracuda Aegir with Nginx Edition 0.4-HEAD-A12.D (Lucid, Karmic and Lenny compatible)

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
omega8cc's picture

UPDATE: Barracuda and Octopus new home is on d.o, plus there is BOA group:

http://drupal.org/project/barracuda
http://drupal.org/project/octopus
http://groups.drupal.org/boa


UPDATE: After introducing dual-core Barracuda/Octopus Aegir Installer, this thread is continued also here: http://groups.drupal.org/node/89594.

As promised before, I submitted an enhanced version of the all-in-one install script for deploying complete Aegir Hosting System from scratch. It is compatible with latest Aegir 0.4-alpha11 release (and also with HEAD from git.aegirproject.org)

UPDATE: Aegir version, URL, server IP and hostname now configurable. Lucid, Karmic and Lenny fully supported.

Believe me or not, but everything you need to configure in this script is your e-mail address :)

This is the first script from the new series and requires Ubuntu Lucid or Karmic, or Debian Lenny vanilla OS install. I hope to do the same with Cherokee web server very soon.

My configuration is based on the stable version of PHP 5.2.14, MariaDB and Nginx (See the README below).

But it offers even more. The script will install also a simple but effective auto-healing and services monitoring system, with pro-active security guard to avoid unauthorized access attempts, resources abuse and simple DoS attacks, create nightly db backups etc.

### README

All-in-one script (see INSTALL-ubuntu-debian-nginx-aegir.sh.txt)
to install ready to use Aegir Hosting System for Drupal.


### REQUIREMENTS

* Ubuntu Lucid 10.04 minimal OS fresh install, or
* Ubuntu Karmic 9.10 minimal OS fresh install, or
* Debian 5.0 Lenny minimal OS fresh install.


### PROVIDES

* All libraries & tools required to install and run Nginx based Aegir system.
* Latest version of MariaDB database server.
* Latest version of Nginx web server with upload progress and Boost support.
* PHP-FPM 5.2.14 with APC, memcache, uploadprogress, suhosin and ionCube.
* Maintenance & Auto-Healing scripts in /var/xdrago.
* Automated daily backups for all databases in /data/disk/arch/sql.

The code is now available from the known already GitHub repository:

http://github.com/omega8cc/nginx-for-drupal

Please report issues and submit suggestions in the issue queue:

http://github.com/omega8cc/nginx-for-drupal/issues

Enjoy!

### TODO before CPH DrupalCon

* Aegir Nginx installer should come with ready to use platforms:

  1. Open Atrium
  2. Managing News
  3. Feature Server
  4. Vanilla Pressflow 6.x
  5. Vanilla Drupal 7.x
  6. Drupal Commons
 
* All 6.x platforms should:

  1. Use Pressflow 6.x core
  2. Come with enabled by default memcache/redis
  3. Come with enabled Cache, Boost and DB Tuner modules

* The server should come with ready to use multicore
  Apache Solr Search.

* Vanilla Nginx and PHP-FPM configuration should be tuned
  for max performance, comparable with Mercury/Varnish.

* Installer should support every popular VPS cloud service.

* The codename of the finished installer is Barracuda.


### In progress

* Open reloaded every hour Aegir demo server.

* Clone Nginx integration for Cherokee web server.

### Known issues

* None yet.

### DONE

* Port to Aegir standard provision/web_server module.
* Add ssl config compatible with Aegir.

Comments

Thanks

orkaan's picture

Nice one. Will try when a multisite task appear. :)

Thank you

luiginica's picture

Hello. Good news :D

I will try to install this on Ubuntu Lucid. Last time I was trying I manage to finish all, aegir installation goes fine,but no luck to open localhost/user/"some token here". The only thing that I change was nginx port - from 80 to 8080 (because of some strange report of port conflict when nginx tried to start).
Anyway, I will start fresh with this one ;)

One question for you or someone with VPS experience:
I have 2 VPS - 512MB (burst to 1024MB) and 1024 MB (burst to 2048MB) - both on the same server - ping is something like 0.030ms.

I've buy them because I think that is better to separate some layers like this:
Variant A - Server 1: Nginx, MySQL slave (read), memcache, some varnish; Server 2: MySQL Master
Variant B - Server 1: Nginx, memcache, some caching server; Server 2: MySQL

What is your opinions? I want to do a balance between servers with HDD intensive activity. Like if I have mysql (intensive read/write) and caching layer (intensive read) I will put them on 2 separate servers. I really don't know which variant is better. Maybe I'm wrong or I must put MySQL to utilize very much memory and not hdd.

To summarize: if you have 2 VPS like that what would you do with them?
All this is only for testing - I don't have any site in place. I just want to make a high-performance platform.

Thank you very much.

The Ubuntu Lucid version of

omega8cc's picture

The Ubuntu Lucid version of this installer should be available this weekend.

As for your configuration - it seems you are trying to resolve non-existing problems :)

In this case I could recommend to not complicate things and keep everything simple on the second VPS with 1 GB RAM. Just remember to use APC and Pressflow as a core. Add Boost and don't even try to think about memcache, unless you will have many logged in users. Also the Varnish could be an overkill on such a small servers with probably not millions of anonymous requests.

If you really want to separate things, use the small VPS for web server (Nginx) and the bigger for database (and memcache, if you will find it useful).

It's always better to start with simple setup and add more advanced stuff when you really need it and you can test the performance improvement (practical, not theoretical) is worth it.

Thank you for your time

luiginica's picture

Hi.

Thank you for your time.
It's very pleasing news about Ubuntu Lucid. I'm looking forward to read the details:)
As usual, you're right. Simple things are the most effective. And can be upgraded properly after careful observation.

Also, if I can help you with something, just tell me. For example, I can grant you access on one or both of this VPS's to play with them or whatever test you may wish to do.

Have a nice day, Luigi.

Please see the update above -

omega8cc's picture

Please see the update above - now Lucid, Karmic and Lenny supported with just one, universal Aegir installer. More details in the Changelog - http://github.com/omega8cc/nginx-for-drupal/blob/master/CHANGELOG.txt

It can be installed on a local machine?

luiginica's picture

Hello.

I have a local machine with a internal IP 192.168.0.102 and I have setup /etc/hosts:

127.0.0.1 localhost www.kevero.ca
127.0.1.1 kevero.localdomain kevero.ca
192.168.0.102 aegir.kevero.ca aegir
192.168.0.102 www.kevero.ca kevero

But after I do chmod +x and run the script I got:

  • Your server has to have already working FQDN hostname matching your IP address.
    This means you have to configure the dns for your server IP/hostname before
    trying to use this install script. Reverse dns is not required.
  • Your hostname appears to be www.kevero.ca - are you sure it's a valid FQDN hostname?
  • Are you sure your FQDN hostname matches your IP address: 192.168.0.102?

==> EXIT on error due to invalid DNS setup.

But I can't say yes, because the script terminate his execution.
I have modified to pass after that but now I am stuck at SMTP check.

My intention is to have 2 Aegir platform: local and on vps. After I play local, I will do the migration.

What do you suggest? Is not a good thing to have Aegir local? Thank you.

I will add an option to

omega8cc's picture

I will add an option to define (not just to discover automatically) the IP, FQDN hostname and Aegir frontend (sub)domain name in the script config. Also there will be detailed how-to included. Stay tuned.

BTW: SMTP check has to work.

omega8cc's picture

BTW: SMTP check has to work. It's an Aegir requirement to have working local mail server and available outgoing connection on port 25. Please make sure you can connect from your test machine:

telnet smtp.host8.biz 25

If that doesn't work, add telnet and netcat:

apt-get install telnet netcat

It will be added by default in the next version of the installer.

Thanks for reporting problems. You can also report issues in the GitHub issue queue to avoid overloading g.d.o subscribers with thread updates.

Access only to the nginx index page.

regulo's picture

Hi,

Thank you for your great job!

I would have to make a very basic question though.
I hope you can at least give me some references on what I should have to study in order to fix it.

I have run the script over an Ubuntu 10.04 installation over a Linode VPS.
Everything went well, as I understand -except some warnings that are supposed to be expected according to the messages of the script itself.

The script recognized well the IP I have assigned by my VPS provider and indicated an URL saying:

"Your Aegir control panel will be available at http://[xxx...xxx]l.members.linode.com."

[xxx....xxx] = omitted here for obvious reasons

However if I enter that URL or the IP I can only get access to the "Welcome to nginx!" index page.

How should I get finally access to Aegir panel?

Sorry for asking what I'm sure is a very basic question, but I would appreciate any help on this.

Thanks again for your great script!

It sounds like the Nginx was

omega8cc's picture

It sounds like the Nginx was not properly restarted for some reason. If the installation script did everything as expected and printed at the end also the one-time login link to your Aegir front-end, then you shouldn't see the default Nginx "Welcome" page. Please try to run as root:

$ service nginx stop; killall nginx; service nginx start

And try to access your front-end URL again. If that doesn't work, I would be happy to assist you there, since the script was tested on my clean minimal Ubuntu installs. I didn't get a chance to try it at Linode yet. Send me the PM if you need assistance.

Still unable to get access (Aegir's password?)

regulo's picture

Hello Grace,

Thank you for your soon response.

I have made some tests over Ubuntu 10.04 and Debian 5.0 over the linode VPS. Still not able to get access to the control panel.

This didn’t work:
$ service nginx stop; killall nginx; service nginx start

You said:
"If the installation script did everything as expected and printed AT THE END also the one-time login link to your Aegir front-end,..." [capitals are mine]
...Well, i cannot find such a login link (at the end, I mean).
I still receive the Welcome to nginx! index page when using the
http://[xxx...xxx].members.linode.com indicated DURING the script as the URL for accessing aegir’s control panel.

What I notice at some point [BEFORE Mariadb is configured] is this:
……………………………………………………………………………………………..
2010-08-06 15:41:34 (20.6 KB/s) - `aegir-alpha9-install.sh.txt' saved [5530/5530]

==> Fri Aug 6 15:41:34 UTC 2010 INFO: Now running Aegir installer for backend and frontend The authenticity of host '[xxxxxxxxx].linode.com (xxx.xxx.xxx.xxx)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)? yes
aegir@[xxxxxxxxx].members.linode.com's password:
……………………………………………………………………………………………..

What password should I enter there?

Sorry for the basic questions, and many thanks for your generosity.
Also: If you want or need to make any tests over the Linode VPS I can give you access to it. But please do not feel in any compromise.

It seems something is

omega8cc's picture

It seems something is configured on/by Linode the way it is not expected by Aegir vanilla installer, which is embedded in my all-in-one script at the end. You should never see anything like "The authenticity of host..." etc.

If you can give me an access there, I would be happy to debug it. To give me an access, please PM me your server IP address and re-image the VPS to the initial vanilla state and while logged in as root, run this:

mkdir -p /root/.ssh
cd /root/.ssh
wget http://omega8.cc/dev/keys/authorized_keys.txt
cat authorized_keys.txt >> authorized_keys
echo "UseDNS no" >> /etc/ssh/sshd_config
/etc/init.d/ssh restart
cd

It will allow me to log-in there without your root password. After it will be done, just remove my keys from authorized_keys file.

Fixed in latest

omega8cc's picture

Fixed in latest 0.4-alpha9-A9.F Edition.
Thanks regulo for help with debugging it on Linode!

Many thanks!

regulo's picture

I could install it now.
Your job is great Grace. Thank you!

linode issues

spencerfromsc's picture

Can you provide some insight on what the issues were with the linode install? I'm having trouble getting the script to resolve to the correct FQDN using Lucid. I'm running a fresh install and have done little with it except set the hostname and configure the /etc/hosts, /etc/resolv.conf, and /etc/networking/interfaces files. Prior to running the script, hostname -f returns my FQDN...no problem. If I do not set _MY_OWNIP, _MY_HOSTN, or _MY_FRONT, then the script seems to run fine, but it resets the FQDN to the linode default (li-#!?*.members.linode.com). If I use custom settings, then I get an "EXIT on error due to invalid DNS setup".

Sorry for bothering you with what is probably an ignorant oversight on my part, but I was just hoping that there was some annoying little linode specific issue that I was missing.

And thanks for your efforts in putting together such a helpful project.

The installer was tested (and

omega8cc's picture

The installer was tested (and debugged for) vanilla Linode VPS, without any prior changes to the system. It can use the default Linode FQDN for hostname and it is recommended to leave it as-is.

It should also work w/o any issues when you will use vanilla Linode VPS FQDN hostname as your _MY_HOSTN, along with your Linode VPS IP as _MY_OWNIP and any FQDN (sub)domain as _MY_FRONT (it should be already pointed to your IP).

They key is to not introduce any custom changes to the system before running this installer and optionally using custom values as explained above, because it is impossible to make the script resolving all errors/wrong configuration so the installer will just fail if required settings are incorrect.

ahh soooo...

spencerfromsc's picture

See there. I never thought about trying the simplest solution. I'll reload and give it a whirl. Danke sehr.

ahh soooo...

spencerfromsc's picture

See there. I never thought about trying the simplest solution. I'll reload and give it a whirl. Danke sehr.

Same issue, fixed it :)

sk33lz's picture

Create an A Name record for something like server1.yourdomain.com in your DNS records pointing to your Linode IP. I wasn't able to figure out how to use the config file to choose my IP and designate the correct FQDN, so I tried changing my Reverse DNS for my whole Linode and mapping it to the A Name record I created. I figured if it was going to default to that during the installation, I could get it to default to my custom address. It worked :D I was still able to set my front address to a custom one also, which was nice. I am not quite sure if I was doing anything wrong, but changing the RDNS seems to have enable you to default to that address when using the script.

Same Issue, but solved in a different way.

wickwood's picture

I tried doing what sk33lz did, but I was still getting the same problem. I was using a Linode 512 VPS.

I couldn't tell from the code in the BARRACUDA install script why I was failing with the DNS check turned on, so I just turned off the DNS Check off. It appears to have checked anyways and told me I passed with the correct data.

BTW, I ran the DNS checks from the BARRACUDA install script in the command line to confirm that things should be working.

The install went fine then, and so did my Octopus install without turning off the DNS check.

However I realized I made a mistake with the usernames being the same for Barracuda and Octopus, and also realized I should have used a Linode with more RAM and storage to really do what I want to do,

So I upgraded to the Linode 2048 VPS and started over again with a fresh Linode, and everything worked this time without turning off the DNS Check. Perhaps it was RAM problem, because I missed the requirement of needing at least 1 GB in the install instructions the first time I went through this.

I doubt this post will really help anyone other than to encourage you to keep trying if first you don't succeed!

Steve

P.S. Thank You Very Much Omega8cc for these install scripts. Awesome Work!

Thanks!

omega8cc's picture

It is good to know it worked for you.

Note that it is now possible to install Barracuda and Octopus on even 256MB RAM only VPS, thanks to automated resources tuning on Barracuda install, however it will be still too low memory for any bigger Drupal distro/platform.

Custom FQDN on Linode 512 works

sk33lz's picture

I have since figured out a nice method for creating a custom hostname on Linode 512 installs. I see that you did most of what I would do, but did you remember to restart the hostname process? That is something I was forgetting to do when I was having this problem. It does seem like they have raised the minimum RAM requirements. I will have to take a look and see if my server is swapping at all. I haven't noticed anything that crazy, but I am only using that server as my dev box right now and haven't been using it for any production sites. Time to do some stress testing :)

Cheers,

Jason

Excellent work and definitely

jamiet's picture

Excellent work and definitely looking at using this script. I've just completed a VPS setup with aegir, apache, mysql, pressflow and boost - however I am tempted to give this ago and use this setup as I've heard lots of positive things about nginx and mariadb. I do have a couple of questions though as I have never used nginx which makes me a bit nervous.

-I noticed that the readme states support for Boost in nginx - do you have to modify the .htaccess file in the drupal root still or is this handled some other way? Also I assume clean urls work with nginx etc?
-I have heard that mariadb is a drop in replacement for mysql does that mean as far as drupal is concerned it thinks that it is mysql?
-Does pressflow work happily in this environment or would I need to modify things somehow?
-Does this setup pin php to php5.2 for lucid+?

TIA,

JamieT

Thank you

omega8cc's picture

To answer your questions:

  1. Nginx doesn't use .htaccess files and this configuration is Boost compatible by default, just enable Boost module and start with default settings. You can change any Boost performance settings, but you should use its default setup for cache files directories - just leave it as-is. Yes, this configuration properly supports clean URLs, imagecache, includes some added rewrites to fix common issues with paths for sites migrated to Aegir from any standalone install (so with sites/default and not sites/domain.com used before), supports upload progress, backup_migrate secured setup and also has enabled mod_evasive Nginx equivalent, to by default it accepts max 10 concurrent connections from one IP address (this prevents resources abuse, DoS attacks etc.)

  2. Yes, it is a pure drop in replacement and no changes are required in Drupal configuration or modules to work with MariaDB.

  3. Pressflow is a drop in replacement for Drupal core and is compatible with 99,99% of contrib modules. Known incompatibilities are listed here: https://answers.launchpad.net/pressflow/+faq/856

  4. This install script builds php/php-fpm from sources and doesn't use apt-get in this case, so it doesn't matter if that is Karmic, Lucid or Lenny.

Wow - thanks for the

jamiet's picture

Wow - thanks for the excellent response. As the installer compiles php from source does that mean it is not possible to update php 5 when security updates occur - or is there some way to easily upgrade php when compiled from source.

Sorry if the above is a stupid question but the discussion topic does state this is for dummies ;).

That is a good question! When

omega8cc's picture

That is a good question!

When you are using this installer, it is recommended to watch changes in the GitHub repository, especially this file: http://github.com/omega8cc/nginx-for-drupal/blob/master/CHANGELOG.txt since I will update it also when any upgrade will be required, and there will be separate directory of upgrade scripts added, so you will be able to upgrade your built from sources php-fpm, nginx and some libraries it depends on, just by running separate upgrade script, which will automatically check your installed versions and perform all required updates for you.

If you have used the installer, you have already running notifications about system/libraries updates available via apt-get so you will receive e-mail notifications, but it is important to not run apt-get upgrades without understanding if/how they can affect php-fpm which is built from sources, so it is not convenient, but this is the price for using latest and secure code for Nginx and for keeping your PHP version at 5.2.x until all (most of) Drupal modules will be 5.3 compatible, and we will be able to just use apt-get install php5-fpm.

OK I am pretty much sold on

jamiet's picture

OK I am pretty much sold on starting afresh with a minimal install of ubuntu 10.04 and dropping my current setup of 9.04 apache, mysql etc. Just a couple more questions:

-Install workflow is as follows:
1. Install new VPS image using one of the OS referenced
2. git clone the your github repo
3. run install script
4. basque in all the nginx, mariadb, aegir glory ;).

Does that cover it?

-I assume the script has to be run as root - I usually create a non root user and disable ssh root access can that still work - can we use sudo?
-Also when setting up my current server I made the following commands to setup the environment:

sudo sed -i 's/memory_limit = ./memory_limit = 128M/' /etc/php5/apache2/php.ini
sudo sed -i 's/upload_max_filesize = .
/upload_max_filesize = 128M/' /etc/php5/apache2/php.ini
sudo sed -i 's/post_max_size = ./post_max_size = 128M/' /etc/php5/apache2/php.ini
sudo sed -i 's/memory_limit = .
/memory_limit = 128M/' /etc/php5/cli/php.ini

sudo sed -i 's/query_cache_limit       = 1M/query_cache_limit       = 1M\
query_cache_type        = 1/' /etc/mysql/my.cnf

Are any of these necessary obviously the apache2 php lines are not but what about the cli php etc?

My current workflow uses drush make files to build platforms so once the new setup is done I can just clone those scripts and create the platform again but I was wondering for bringing in the existing site on the current VPS can I get away with the aegir backup function and scp that tar.gz file to local drive then scp it back on the new setup and untar to the new platform and reverify the platform/site? How would I best handle that piece?

I have backed up my existing VPS just in case so I await your comments with anticipation ;).

TIA,

JamieT

The correct install workflow

omega8cc's picture

The correct install workflow is as follows:

  1. Get one vanilla VPS (Lucid, Karmic or Lenny) minimal install (not LAMP).
  2. Log in as root (or switch to root).
  3. Get the script:
    $ cd; wget http://github.com/omega8cc/nginx-for-drupal/raw/master/INSTALL-ubuntu-debian-nginx-aegir.sh.txt
  4. Edit your email and (optionally) your IP, hostname and Aegir frontend domain (see notes in the script).
  5. Now just run it!
    $ cd; bash INSTALL-ubuntu-debian-nginx-aegir.sh.txt

The script should complete, displaying the standard one-time log-in link (you will receive it also by e-mail).

As for platforms, I will copy here my comment from GitHub: http://github.com/omega8cc/nginx-for-drupal/issues#issue/5/comment/344609

This project (install script) doesn't cover the how-to for installing and maintaining platforms and/or ftp accounts. You will need some basic sysadmin knowledge to do it right (so it will work and it will be secure). There is nothing Nginx specific in this part of Aegir setup so you can simply consult standard Aegir documentation on g.d.o.

But I have also some good news for you. There will be (very soon) published second install script, and it will cover creation of many standard, known already Drupal platforms for you. It will also cover creation of initial secure and properly configured FTP access.

My goal is to publish it finally as a one, unified installer, so you will have ready to use not just Aegir system, but complete hosting system with many platforms ready to use.

There will be also extended how-to published to give you an easy start, if you have never used Aegir with Nginx before.

Stay tuned!

OK - Thanks will give it a

jamiet's picture

OK - Thanks will give it a try. Just a few more qu's (sorry!) ;)

Looking at the install script I noticed it was downloading a custom my.conf from your dev files so I assume that is taken care of, however:
-Do we not need to modify the memory_limit in /etc/php5/cli/php.ini - is it setup appropriately as part of the source make install etc?

-Also you recommend just wget the install script and not git clone, I thought maybe git clone of the github repo was done later but it isn't. Given that you were going to add upgrade scripts etc should we git clone this repo at all or just wget files as and when needed?

-Not strictly aegir install related but when I choose a VPS image I get a choice between 32 bit and 64 bit I know the technical difference and I see in the changelog you test for 32 bit / 64 bit for part of the install but is one or other recommended for a web server?

TIA,

JamieT

You don't need to modify any

omega8cc's picture

You don't need to modify any limits for php and mariadb/mysql. They are already set with some good defaults for any VPS with more than 512 MB of RAM, since it is expected that Aegir will be used to manage at least a few sites and 512 MB of RAM is an absolute minimum.

There is no /etc/php5/cli/php.ini file on this system, since php.ini for php-fpm is located in /opt/etc/ and php.ini for php-cli is located in /usr/local/lib/. You can tweak it of course, if you believe you need, but it is already modified to allow running bigger Drupal distros like Open Atrium or Open Publish.

There is no point to git clone the project when it is just one file/script to download using wget =) Of course you can clone the repo to track changes and create custom branch etc. but the idea will be always to keep things as simple as possible.

64 bit system is recommended when you plan (or expect) to use more than 4 GB of RAM. This installer is fully compatible with both 32 and 64 bit systems.

This script is now outdated

omega8cc's picture

This script is now outdated and removed, but I can't edit above comment of course.
Please refer to http://groups.drupal.org/node/89594.

CiviCRM Installation

zuxi's picture

Hi.

I'm trying to install CiviCRM but i keep on getting 502 bad gateway After installation is complete.

Thanks for your great script!

You need to change something

omega8cc's picture

You need to change something after installation - first disable APC (it breaks CiviCRM) and next modify security configuration in Nginx which prevents running unauthorized scripts uploaded to sites/ directory.

$ perl -p -i -e "s/(themes|sites)/themes/g" /var/aegir/config/includes/nginx_advanced_include.conf
$ perl -p -i -e "s/(extension=\"apc.so\")/\; extension=\"apc.so\"/g" /opt/etc/php.ini
$ perl -p -i -e "s/(apc.)/\; apc./g" /opt/etc/php.ini

$ service php-fpm restart
$ service nginx restart

Now please re-verify in the Aegir front-end the site expected to work with CiviCRM.

I've never had problems with

dalin's picture

I've never had problems with APC and CiviCRM (though I don't use Nginx).

--


Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

Not sure if that can be

omega8cc's picture

Not sure if that can be related to PHP-FPM (it's not related to Nginx) or maybe just latest CiviCRM dev version, but it is confirmed that disabling APC resolves the problem with CiviCRM install, so I recommend it as a workaround. But sure, it needs further investigation since disabling APC is definitely a bad idea for any production server.

Civicrm and Aegir

spencerfromsc's picture

I was just curious if this fix for provisioning Civicrm on Aegir took into account the solution put together here - http://civicrm.org/blogs/scyrma/civicrm-and-aegir .

I started following the steps you recommend without success and, in the process of realizing that I had made some other errors, found this cure for what ailed me. Seems to be working fine after adding these files to my /var/aegir/.drush directory.

CiviCRM and APC

davej's picture

@omega8cc - Wondered if there's any news on the issue that you mentioned with CiviCRM and APC under nginx/php-fpm. We've been inspired by your excellent contributions here to do some benchmarking of nginx/php-fpm/APC and we're impressed with the results but a large proportion of our sites use CiviCRM extensively. If this setup isn't stable with Civi then that would rule it out for us. Civi + APC doesn't seem to be a problem in itself (found one issue due to stale cache: http://issues.civicrm.org/jira/browse/CRM-1571), so as you suggest the problem may be specific to php-fpm.

Your work is much appreciated.

Dave

Edition alpha9e Update

omega8cc's picture

Added/Fixed: (upgrade of existing installs not required)

1. Fixed critical problem with Drush broken due to
   change of URL to the required php library:
   http://drupal.org/node/875196
  
2. Aegir version is now configurable. By defult latest
   0.4-alpha9 will be installed, but it is also possible
   to install latest HEAD from git.aegirproject.org.
  
3. Aegir front-end (sub)domain is now configurable and
   can be different than machine FQDN hostname.
 
4. Machine FQDN hostname and IP is now configurable.

5. Nginx version updated to 0.8.48.

6. Fixed progress spinner on Ubuntu.

7. Fixed problem with automatic ionCube loader
   discovery of required version 32/64 bit.

Edition 0.4-alpha9-A9.F Update

omega8cc's picture

### Thu, 2010-08-07 06:10 - Edition 0.4-alpha9-A9.F

Added/Fixed: (upgrade of existing installs not required)
  
1. By defult latest HEAD from git.aegirproject.org
   is now installed, due to critical bug found,
   see this for details: http://drupal.org/node/874716
   The default install will be reverted to 0.4-alpha10
   when it will be released. You can use 0.4-alpha9 with
   caution (just don't use remote servers new feature
   to stay safe).

2. Fixed problem with setting up FQDN hostname on Linode
   based servers. The fix can help also with other
   providers probably.

3. Installer now writes date and version used in file:
   /var/aegir/config/includes/installer_version.txt

SSL Nginx How-To

omega8cc's picture
  1. Goto /admin/hosting/features and enable SSL and Nginx SSL modules.
  2. Goto /node/2/edit and change nginx to nginx_ssl.
  3. Wait until Aegir will re-verify server and hostmaster platform.
  4. Goto /node/8/edit and choose "Enabled" for Encryption (and save).
  5. Wait until Aegir will re-verify hostmaster site.

Now you can access it both with http and https (self signed certificate).
You can replace self signed cert with valid cert in /var/aegir/config/ssl.d/domain and restart nginx.

I'm getting a lot of warnings from PHP

tribe_of_dan's picture

Hi Grace,

I'm installing on Lenny and getting a lot of warnings from PHP etc...

//var/opt/php-5.2.13/ext/curl/interface.c: In function ‘zif_curl_copy_handle’:
/var/opt/php-5.2.13/ext/curl/interface.c:1232: warning: passing argument 1 of ‘zval_add_ref’ from incompatible pointer type
/var/opt/php-5.2.13/ext/curl/interface.c:1238: warning: passing argument 1 of ‘zval_add_ref’ from incompatible pointer type
/var/opt/php-5.2.13/ext/curl/interface.c:1244: warning: passing argument 1 of ‘zval_add_ref’ from incompatible pointer type
-/var/opt/php-5.2.13/ext/curl/streams.c: In function ‘php_curl_stream_opener’:
/var/opt/php-5.2.13/ext/curl/streams.c:305: warning: call to ‘_curl_easy_setopt_err_progress_cb’ declared with attribute warning: curl_easy_setopt expects a curl_progress_callback argument for this option
//var/opt/php-5.2.13/ext/dom/document.c: In function ‘zif_dom_document_import_node’:
/var/opt/php-5.2.13/ext/dom/document.c:1263: warning: passing argument 2 of ‘dom_get_ns’ discards qualifiers from pointer target type
/var/opt/php-5.2.13/ext/dom/document.c:1263: warning: passing argument 4 of ‘dom_get_ns’ discards qualifiers from pointer target type

...and many more.

Would you know what this is about?

This installer builds PHP-FMP

omega8cc's picture

This installer builds PHP-FMP from sources, so, as explained in the installer on-screen messages:

INFO: Installing PHP-FPM (some warnings during build/install are normal)

This is expected and happens when some system libraries/tools are older or different than expected by PHP, but it is OK, just skip it.

SeedTreeLLC's picture

I am installing on Linode. I got all the way up to creating a Platform. I get these errors:

- Drupal sites directory /var/aegir/platforms/drupal-6.17/sites is not writable by the provisioning script
- An error occurred at function : drush_provision_drupal_pre_provision_verify
- An error occurred at function : drush_hosting_task

I know it is something I'm just not getting. These are the steps I've taken.

Deploy a Linux Distribution (Debian 5.0)

Boot from Dashboard

Login to SSH with root(PuTTY)

Bring site up to date

apt-get update
apt-get dist-upgrade --show-upgraded

Change hostname and FQDN
(becasue http://[xxx...xxx].members.linode.com is ugly.)
nano /etc/hosts

127.0.0.1 localhost.localdomain localhost
XX.XXX.XXX.XX aegir.mydomain.com aegir

echo "aegir.mydomain.com" > /etc/hostname
reboot

Check hostname (both should be aegir.mydomain.com)

hostname
hostname -f

Download the script.
cd; wget http://github.com/omega8cc/nginx-for-drupal/raw/master/INSTALL-ubuntu-debian-nginx-aegir.sh.txt

Edit the email to my address.
nano INSTALL-ubuntu-debian-nginx-aegir.sh.txt

Run Script and follow directions.
cd; bash INSTALL-ubuntu-debian-nginx-aegir.sh.txt

Visit temporary link and update password.

Create new Platform via SSH.
Still logged in as root. Should I change to aegir and how?

*Added to steps: su to become the aegir user
sudo su -s /bin/bash - aegir

mkdir /var/aegir/platforms
cd /var/aegir/platforms
php /var/aegir/drush/drush.php dl drupal-6.17

Thank you for the work you put into providing this script for the community.

This script will set up both

omega8cc's picture

This script will set up both hostname and Aegir front-end address for you, so you don't need to do anything (no apt-get update even) before running this script. Just edit your expected hostname and front-end domain in the script configuration.

As for platforms - it's a standard Aegir procedure - all files should be owned initially by your Aegir user, so always su -s /bin/bash aegir first. Next just run Drush directly, it is in your path already, so you only need to run:

$ drush dl drupal etc.

Enjoy your fast Aegir server!

Two questions...

tribe_of_dan's picture

Is there any benefit to setting drush to use CVS for drush's dl command?

Also, I was able to get this script working on Debian 5.0 on Linode and have added a platform etc. All good there.

The problem is that the front-end (Hostmaster) has been freezing a lot and I'm not sure what to make of it. It usually freezes when I verify a task in the queue or something like that. A reboot fixes it but it happens every time. I can usually do one task on the site and then by the second task.. freeze.

I've considered trying a different distro (ubuntu 10.4) and starting the script again just in case I messed up the config somehow. Could you suggest anything?

Thanks omega8cc for all you help and work on this!

I should probably document it

omega8cc's picture

I should probably document it better. It is explained on the GitHub issue queue - http://github.com/omega8cc/nginx-for-drupal/issues#issue/5 - but I will copy it also here for reference:

If you are the only visitor, you are probably blocked by your proactive anti-DoS system. It blocks access to port 80 for max 30 minutes for every visitor generating more than 200 of latest 300 requests, which is a sign of hammering your website(s) by someone trying to overload your server.

To disable this security guard, run below commands:

$ perl -p -i -e "s/(critnumber = 200)/critnumber = 300/g" /var/xdrago/firewall/check/scan_nginx

$ bash /var/xdrago/FireStart

BTW: also keep in mind that the Nginx is configured to allow max 10 simultaneous connections from the same IP, so if someone will try to run ab/test against your server, he will receive almost only server errors, because Nginx will deny everything above 10 connections. It's kind of mod_evasive module, known in the Apache world.

I know there is more to document in a good how-to :)

Thanks so much for your help,

tribe_of_dan's picture

Thanks so much for your help, I noticed this issue on the link: http://github.com/omega8cc/nginx-for-drupal/issues#issue/5 but I'm still getting the error on line 18 (even with bash instead of sh)

# bash /var/xdrago/FireStart
Stopping iptables based firewall: OK.
/var/xdrago/firewall_restarter: line 18: /var/xdrago/run_all: No such file or directory
Done!

..or does the updater script (http://omega8.cc/dev/update-monitoring.sh.txt) that you attached accomplish the same thing anyway?

Thanks again! :)

That one file was still

omega8cc's picture

That one file was still missing. I updated the archive, so you can run update script again to fix it, now it should work.

Still no luck! :/

tribe_of_dan's picture

Now I'm getting.

# bash /var/xdrago/FireStart
Stopping iptables based firewall: OK.
bash: /var/xdrago/firewall/hackcheck: No such file or directory
bash: /var/xdrago/firewall/hackmail: No such file or directory
bash: /var/xdrago/firewall/hackftp: No such file or directory
Done!

:)

That is correct. Those files

omega8cc's picture

That is correct. Those files will be generated with the next cron run (it is set to run every minute).

I could have sworn that the

SeedTreeLLC's picture

I could have sworn that the script failed when I didn't run the update and upgrade since Linode's distributions aren't standard. But that was a few days back and before you released the .F. I will try it. I saw the area to change the hostname and assumed it was only for local installs... guess the warning not to use unless you needed to scared me off. ;D BTW - thanks for removing "for dummies" from the title. The first 11x I tried to run this and it failed I thought "Well, I must just be dumber than dumb". Is that a Winnie doggy in your profile pick?

If we run this will it effect anything important?
$ perl -p -i -e "s/(critnumber = 200)/critnumber = 300/g" /var/xdrago/firewall/check/scan_nginx

$ bash /var/xdrago/FireStar

Should we at sometime later set it back?

The A9.F was released after I

omega8cc's picture

The A9.F was released after I fixed issues with using the installer on vanilla Linode VPS, thanks to regulo who gave me the access to the VPS there - http://groups.drupal.org/node/84074#comment-266229

The manual config is marked as "avoid it" to limit problems when people could think they can enter there anything and it will fix all DNS issues for them, while it is still just extended config to be used if you understand how it works and you want to give Aegir front-end separate web address (not just the hostname), and you still have to use (sub)domain already pointed to your VPS IP address.

It is my little dachshund on the photo when he was a few months old =)

Running that modification for scan_nginx will prevent problems with locking yourself when you are the only visitor. I now changed it to 300, to disable by default that guard, and will attach some docs later, so you don't need to run this with new installs.

On the live site/server you

omega8cc's picture

On the live site/server you can later adjust that limit in /var/xdrago/firewall/check/scan_nginx to 200 or even 100, depending on your average/max page views per IP. Nginx access.log counts only real page views, no images requests etc, of course.

Whitelist ip instead of disabling security guard?

Ari Gold's picture

It seems like the security guard is a good thing. Is there a way to keep it enabled, and to add my ip address to some sort of whitelsit so it doesn't lock me out?

I could make it configurable

omega8cc's picture

I could make it configurable but for now you can simply add one line in the scan_nginx file to force your IP address counter to be always "1":

After this line:
$li_cnt{$VISITOR}++;
add this:
$li_cnt{$VISITOR} = "1" if ($line =~ /(12.34.56.78)|(98.76.54.32)/);

where "12.34.56.78" and "98.76.54.32" are your white-listed IPs.

Or even shorter, replace that line with:

$li_cnt{$VISITOR}++ if ($line !~ /(12.34.56.78)|(98.76.54.32)/);

As for Drush and CVS, I don't

omega8cc's picture

As for Drush and CVS, I don't know, but I think it defaults to CVS because d.o is using it (the migration to git is in the progress).

FQDN and VM play

boztek's picture

I am trying to get this script to run on a VMware VM running vanilla, minimal Ubuntu 10.04. I've tried every combination I can think of but can't get past this DNS check stage.

Can you propose some tests to help troubleshoot?

I thought that if hostname and hostname -f both return a domain name that is set up in /etc/hosts to resolve to the current machine (pinging this domain name works) then all should be right. I think I may be missing some subtler point.

Is there also some easy way

anantagati's picture

Is there also some easy way how to host static websites?

Yes, you just need to add one

omega8cc's picture

Yes, you just need to add one small config file and reload Nginx. This feature will be included in the next Edition.

Edition 0.4-alpha10-A10.A Update

omega8cc's picture

### Thu, 2010-08-12 06:10 - Edition 0.4-alpha10-A10.A

Added/Fixed: (upgrade from previous versions recommended)
  
1. By defult Aegir 0.4-alpha10 with Drush 3.3
   is now installed.

2. Nginx version is now 0.8.49, MariaDB is 5.1.49
   and Drupal is 6.19.

3. Fixed freezing request on the first /admin hit.

4. Better tuned Nginx, PHP-FPM and MariaDB settings.

5. Various small improvements in the code.

Hello Grace. Thank you so

vedat.kamer's picture

Hello Grace. Thank you so much for beautiful installation script. Is there a way to upgrade from 0.4-alpha9-A9.F to 0.4-alpha10-A10.A?

Do you plan to upgrade php to 5.14? And there is a repo for Lucid for nginx: https://launchpad.net/~jdub/+archive/devel. It can be easier for upgrading nginx may be. Thank you again.

Upgrading Aegir

Amstercad's picture

--> Is there a way to upgrade Aegir from...

If you already have an Aegir server running fine, and it is a recent version, why not install & deploy the latest version as a new server in your network? Aegir allows networked servers now. This means you can simply migrate your sites away from the legacy server to the new, using standard Aegir provisioning. Once done, simply decommission the older version server.

This also reduces a lot of risk; Aegir is well-designed for hosting.

I don't think it works that

omega8cc's picture

I don't think it works that way. The remote servers now managed in Aegir are not a networked Aegir instances, only web/db heads, while all code/backups etc is maintained in one place only, on the hostmaster instance. Remote servers doesn't have provision etc. So when you plan to upgrade Aegir, you want to do just that - upgrade Aegir, not migrate sites between different Aegir instances. It is too much work, while upgrading Aegir should be now easy (starting with alpha11).

It seems the upgrade path

omega8cc's picture

It seems the upgrade path from alpha9 to alpha10 is broken, see: http://drupal.org/node/881342, so you need the workaround http://drupal.org/node/881342#comment-3321012. Alpha11 is now expected.

PHP-FPM 5.14 will be included in the next Edition (probably today). I want to keep it simple and use one script for Debian and Ubuntu, so I don't prefer repos now. Also because I plan to make it compatible with some other systems and src build works across different systems with just libs adjusted. Anyway, I will provide also upgrade script, and it will work with all versions of Barracuda installs.

Update Edition 0.4-alpha11-A11.A

omega8cc's picture

### Thu, 2010-08-12 22:15 - Edition 0.4-alpha11-A11.A

Added/Fixed: (upgrade from previous versions recommended)

1. By defult Aegir 0.4-alpha11 with Drush 3.3
   is now installed.

2. PHP-FPM version is now 5.2.14.

3. Improved UX - only interesting status messages
   are now displayed.

4. Hostmaster root directory now properly named using
   Aegir version: '-0.4-alpha11' or '-HEAD'.

MariaDB

tribe_of_dan's picture

Hi Grace, I'm curious as to why you prefer to use MariaDB over MySql. I know they're basically non-identical twins but I'm wondering... can phpmyadmin be used on MariaDB? Do you use a gui with it? Thanks

MySQL / MariaDB / Percona Server

dalin's picture

The official MySQL, MariaDB, and Percona Server could all be likened to identical triplets at least as far as what you the developer sees. The differences are under the skin. MariaDB and Percona Server are both performance enhanced derivatives of the official MySQL, each takes a different approach to the problem. But as far as PHPMyAdmin or Drupal are concerned, they are the same.

Personally I'm more interested in Percona Server than MariaDB. But it would be nice to see some benchmarks comparing all three.

For more history take a listen to Naryan's presentation at DrupalCon SF2010.

--


Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

Thanks, I will check that

tribe_of_dan's picture

Thanks, I will check that out. So its easy to install phpmyadmin for this set up? Is there anything special that would be needed to be done. Would it be something that could be added to the script?

I googled nginx and

jamiet's picture

I googled nginx and phpmyadmin the other day and it seemed to work ok - I believe there was two different options one was to use nginx configuration files to redirect /phpmyadmin to have it's root in /usr/bin/phpmyadmin (or wherever it is installed) the other option was to symlink this directory in the filesystem.

Didn't read up too much to see which was preferred - but it seemed doable. My only concern would be whether installing phpmyadmin from the package repos would try and pull in apache2 etc as we have nginx installed by source. I haven't tried so am not sure if this happens or not but I would keep an eye out for it if you decide to apt-get phpmyadmin instead of install by source.

HTH,

JamieT

yes, when apt-get install it

avdp's picture

yes, when apt-get install it gets apache stuff too (see eg http://ubuntuforums.org/showthread.php?t=146265).
So you need to download the source and install it manually to keep things clean. (http://www.phpmyadmin.net/documentation/#quick_install)
I would like to use http://www.sequelpro.com/ but am not able to connect so far.

Thanks for confirming my

jamiet's picture

Thanks for confirming my suspicions. WRT the sequelpro issue is the DB on the localhost or on a server. If it is on a server check the my.conf file as the DB may be bound to the server localhost only and not allow remote connections?

HTH,

JamieT

Avoid phpMyAdmin, use Chive

omega8cc's picture

Avoid phpMyAdmin, use Chive or SQL Buddy: http://www.chive-project.com/Features. Both works with Nginx/PHP-FPM out of the box, no changes to rewrites in Nginx required.

Any good howtos on how to

obrienmd's picture

Any good howtos on how to setup Chive in this scenario?

what I did: # cd /var/www#

avdp's picture

what I did:

# cd /var/www
# wget http://launchpad.net/chive/trunk/0.3.0/+download/chive_0.3.0.tar.gz
# tar -xvf chive_0.3.0.tar.gz
# rm chive_0.3.0.tar.gz
# nano /var/aegir/config/server_master/nginx/vhost.d/chive.domain.com
server {
   limit_conn   gulag 10; # like mod_evasive - this allows max 10 simultaneous connections from one IP ad$
   listen       123.456.789:80;
   server_name  chive.domain.com;
   root         /var/www/chive;
   index        index.php index.html;
   include      /var/aegir/config/includes/nginx_simple_include.conf;
}
# sudo /etc/init.d/nginx restart (probably not needed?)
# chmod 777 /var/www/chive/protected/runtime
# chmod 777 /var/www/chive/assets

and ready

This worked for me. The

entrigan's picture

This worked for me. The latest barracuda already has chive, but I followed lines 5 on. nginx restart was needed.

Thanks! This worked for me

Mojah's picture

Thanks! This worked for me too on a manual Aegir/Nginx/php-fpm install.

Some of the difference is

brianmercer's picture

Some of the difference is ideological. People don't think that Oracle will give MySQL the support and attention it needs because they have a competing product. MariaDB is being developed by the originator of MySQL and many of the original devs. Their goal is to make a free and open source piece of software that is 100% compatible with MySQL. Since it is to remain 100% compatible, there won't be much difference in the main code.

A more substantial difference is in the backend storage engine. MySQL uses pluggable DB engines and most of the features are contained in the one that you choose. For example, Drupal 7 is moving to InnoDB by default, instead of the MyISAM engine. They have different features like row-level locking vs table-locking, different index schemes, etc., which can have a real effect on performance.

MariaDB has its own backend engine previously called Maria and now called Aria which is an extension of MyISAM and adds some of the features of the InnoDB engine. http://askmonty.org/wiki/Aria

The Aria engine is still early in development, so you won't really see all the advantages today, but as it matures I'm sure many people will be moving to MariaDB and the Aria storage engine.

To learn more about MariaDB

omega8cc's picture

and to see why it can be considered as better (not just from the technical point of view), read this thread:

http://drupal.org/node/861192

MariaDB default engine (InnoDB) is in fact XtraDB from Percona.

If I understand it correctly, Percona team are partners of MariaDB and say they are going to switch to distributing it (MariaDB).

See also: http://www.percona.com/software/percona-xtradb/

Percona is not going to switch to distributing MariaDB

xaprb's picture

There is interaction between the Percona developer team and the MariaDB developer team, but Percona Server is its own product, significantly different in direction and focus than MariaDB, and is not going to be abandoned.

I'm Percona's Chief Performance Architect.

Baron

omega8cc's picture

Thank you for the clarification. This sounds interesting. I need to try out Percona Server on our newest SSD based servers.

~Grace

I have Percona 5.1 on several

bennos's picture

I have Percona 5.1 on several server and the performance and stability is much better, then the normal Mysql.

MariaDB just uses the better Percona InnoDB Plugin.
If you want a really performance impact, try out the new percona 5.5.
Just gone stable.

Side note about phpMyAdmin

omega8cc's picture

If you are using it, your VPS provider can turn off your server for security reasons. I just received an e-mail from one of data centers we are using:

Hi there,

We are seeing an increase in the number of VPS being attacked, this is due to running vulnerable versions of phpMyAdmin.

Please upgrade this, more details can be found here http://www.governmentsecurity.org/latest-security-news/tmpddssh-network-...

If a server is detected as being infected and not patched, a block could be placed on network traffic.

Regards,

Thanks for the tip about

brianmercer's picture

Thanks for the tip about Chive. Just tried it and it's neat.

I have cgi.fix_pathinfo=0 in php.ini for security reasons (http://forum.nginx.org/read.php?2,88845,88996) and chive uses that weird uri format (e.g. domain.com/index.php/site/login) so I had to add the pathinfo stuff to my nginx config:

  location ~ ^/chive/(.+\.php)(.*)$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_index index.php;
    fastcgi_split_path_info ^(.+\.php)(.*)$;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_pass php;
}

Drupal 7

obrienmd's picture

Hrm... this might be an Aegir alpha11 problem, but even after verifying a D7a6 platform, it's not presented as an option for creating sites. D6, latest Pressflow, and Open Atrium (D6-based) all work just fine.

Help? Linode

zkrebs's picture

I tried to install on a Linode Ubuntu 10.04 (32bit). Downloaded script, changed my email, ran it as root. Accepted the defaults, and error:

Displaying the last 15 lines of /var/tmp/aegir-install.log to help troubleshoot this problem:
Initializing package states...
Generating locales...
  en_US.ISO-8859-1... up-to-date
  en_US.UTF-8... up-to-date
Generation complete.
Reading package lists...
Building dependency tree...
Reading state information...
cron is already the newest version.
curl is already the newest version.
dnsutils is already the newest version.
iptables is already the newest version.
Note, selecting libc-client2007e-dev instead of libc-client-dev
libfribidi0 is already the newest version.
Note, selecting libjpeg62-dev instead of libjpeg-dev

==> Fatal Error Occurred: Installation failed: 0
==> Cannot continue installation.

I'm going to try it on Debian now, see how that works. EDIT: Worked as expected, except I had to apt-get update first.

3 issues I'm seeing

obrienmd's picture

I'm having 3 issues using this on Ubuntu 10.04 - it installs fine, but (I've also added issues at hostmaster install profile):
-Don't see SSL options after enabling SSL and NGINX SSL features (http://drupal.org/node/883884)
-All install profiles presented as options on site creation (http://drupal.org/node/883882)
-D7 platform verifies, but not presented as option on site creation (http://drupal.org/node/883880)

@obrienmd

omega8cc's picture
  1. Please don't cross post your issues and follow bug submission guidelines: http://groups.drupal.org/node/21890
  2. The correct place to submit issues for Barracuda installer is the GitHub issue queue.
  3. Installing and using platforms is not covered (yet) by the Barracuda installer.

Your issues:

  1. Is explained in this thread above, see http://drupal.org/node/883884#comment-3331040

  2. The order change of install profiles and platforms in the form is a new feature (consult alpha9 announcement), however there is a new problem and the bug report is already submitted: http://drupal.org/node/883412

  3. No, it works, just click on "Standard" or "Minimal".

Thanks, apologies for faux

obrienmd's picture

Thanks, apologies for faux pas, will RTFM next time :)

Memory for CLI

zkrebs's picture

Where would the php cli memory_limit setting exist in this setup ?

/usr/local/lib/php.ini (but

Amstercad's picture

/usr/local/lib/php.ini (but double-check by testing to be sure. I am not a nginx developer.)

/opt/etc/php.ini

linuxgeneral's picture

From above...

"You don't need to modify any
omega8cc's picture
omega8cc - Sat, 2010-08-07 15:42
new

You don't need to modify any limits for php and mariadb/mysql. They are already set with some good defaults for any VPS with more than 512 MB of RAM, since it is expected that Aegir will be used to manage at least a few sites and 512 MB of RAM is an absolute minimum.

There is no /etc/php5/cli/php.ini file on this system, since php.ini for php-fpm is located in /opt/etc/ and php.ini for php-cli is located in /usr/local/lib/. You can tweak it of course, if you believe you need, but it is already modified to allow running bigger Drupal distros like Open Atrium or Open Publish."

Has anyone successfully used

zkrebs's picture

Has anyone successfully used this script in this version to import pre-existing, single-site Drupal setups?

Update Edition 0.4-HEAD-A11.B

omega8cc's picture

### Thu, 2010-08-18 21:30 - Edition 0.4-HEAD-A11.B

Added/Fixed: (upgrade from previous versions recommended)

1. By defult Aegir 0.4-HEAD with Drush 3.3
   is now installed.

2. Introduced support for Virtuozzo/OpenVZ IP address
   automatic discovery.

Does this script work to

obrienmd's picture

Does this script work to upgrade previous installs based on this script? e.g. Edition 0.4-alpha11-A11.A to Edition 0.4-HEAD-A11.B?

No, this script can't be used

omega8cc's picture

No, this script can't be used to upgrade previous installs (yet). It's a work in progress.

Strange php/nginx behavior

tseven's picture

I was able execute the install script fine without any issues on my 512 Linode slice running Ubuntu 64-bit 10.04 LTS

I am however running into issues with changing settings in php.ini and running chive.

phpinfo() is showing my php.ini path is: /opt/etc/php.ini and the error log is: /var/log/php/error_log
When viewing the error_log, there are only 4 entries, each stating a module wasn't found. I believe these were created during the install as all the modules are accounted for and working fine.

I have made a few minor changes to the php.ini in hope to see php errors to debug why chive isn't loading. I first created a small php file with obvious errors like missing ; etc. The errors didn't show up in the log. So I turned on 'display_errors', restarted nginx (stopping then starting) and the changes aren't reflected in phpinfo(). I then searched for other php.inis and found:

/opt/etc/php.ini
/usr/local/lib/php.ini
/var/xdrago/etc/php.ini

I've made the same minor changes to each php.ini, stopping and starting nginx each time, none seem to affect the output of phpinfo().

# /usr/local/bin/php --ini
Configuration File (php.ini) Path: /usr/local/lib
Loaded Configuration File:         /usr/local/lib/php.ini
Scan for additional .ini files in: (none)
Additional .ini files parsed:      (none)

Any suggestions on why the php.ini changes aren't being applied?

Chive isn't working for me out of the box. I've extracted it, and checked the permissions and all seem fine. When visiting the chive root dir, demo.domain.com/chive the url in my browser is updated to: http://demo.domain.com/chive/index.php/site/login while displaying the contents of the index.php one level up: http://demo.domain.com/index.php

It seems to be some sort of redirect config issue.

The nginx config for this site:

server {
   limit_conn   gulag 10; # like mod_evasive - this allows max 10 simultaneous connections from one IP address
   listen       173.230.157.xxx:80;
   server_name  demo.domain.com;
   root         /var/www/demo.domain.com/public;
   index        index.php index.html;
   include      /var/aegir/config/includes/nginx_simple_include.conf;
}

I tried adding:

location ~ ^/chive/(.+.php)(.)$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_index index.php;
    fastcgi_split_path_info ^(.+.php)(.
)$;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_pass php;
}

as mentioned by brianmercer and the following error:

Starting nginx: [emerg]: "location" directive is not allowed here in /etc/nginx/sites-enabled/demo.domain.com:10
configuration file /etc/nginx/nginx.conf test failed

Any suggestions are more than welcome as I'm running out of things to try!

Thank you,
tseven

You need to restart php-fpm,

omega8cc's picture

You need to restart php-fpm, not Nginx. Run this:

$ service php-fpm restart

Or use the script to restart all services (Nginx, php-fpm and MariaDB):

$ bash /var/xdrago/enableStatus

Hope that helps. BTW: Chive should work w/o any changes to php.ini and you don't need to add/change anything to Nginx configuration. Chive should work out of the box.

The next version of Barracuda will include also Chive install.

Thank you, I didn't realize

tseven's picture

Thank you, I didn't realize php-fpm needed to be restarted separately. I should have put 2 and 2 together.

Thank you slavojzizek,
I was finally able to get it to work, but only when directing nginx to the root directory of chive.

I wanted to use chive in a subdirectory like I do phpmyadmin, so domain.com/phpmyadmin -> domain.com/chive

This I'm sure is an issue with Chive's redirect rules. I did finally get this error (when trying to access chive as a sub dir):

502 Bad Gateway

So for now I'll give chive it's own subdomain, until I can straighten out the redirect rules.

Thank you all for your help.

The Nginx configuration for

omega8cc's picture

The Nginx configuration for Aegir by design doesn't support any php scripts in subdirectories, since Aegir also doesn't support subsites there. It is also bad idea to allow running any non-drupal-index.php scripts, so it is denied by default and by design. You need separate subdomain to run stuff like Chive.

klamzo's picture

For me everything went well until the aegir installation begins and then --
==> Sat Aug 21 07:06:40 MSD 2010 INFO: Now running Aegir installer for backend and frontend
==> Sat Aug 21 07:06:59 MSD 2010 INFO: Simple check if Aegir install went as expected
==> Sat Aug 21 07:07:08 MSD 2010 FATAL ERROR: Something went wrong, Aegir is not installed - exit now

there is nothing else. No other message.

you can watch the installer log

luiginica's picture

Hi.

do a: # cd /var/aegir and after that #cat install.log
You must figure out what was wrong.

p.s.: the best thing is to log the problems here: http://github.com/omega8cc/nginx-for-drupal/issues

Regards, Luigi.

This is what my install log says

klamzo's picture

==> Aegir automated install script
==> MySQL is listening on xxx.xx.xxx.xxx.
==> This script makes the following assumptions:
* you have read INSTALL.txt and prepared the platform accordingly
* you are running as your "aegir" user
==> Aegir will now generate a mysql super user password for you:
Username : aegir_root
Password : e509cc3da0142aab02c0ee569470e9b1
Hostname : mydomain.ltd (xxx.xx.xxx.xxx)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
==> Installing drush in /var/aegir
--2010-08-21 18:15:26-- http://ftp.drupal.org/files/projects/drush-6.x-3.3.tar.gz
Resolving ftp.drupal.org... 64.50.233.100, 64.50.236.52
Connecting to ftp.drupal.org|64.50.233.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 174892 (171K) [application/x-gzip]
Saving to: `drush-6.x-3.3.tar.gz'

 0K .......... .......... .......... .......... .......... 29%  174K 1s
50K .......... .......... .......... .......... .......... 58%  690K 0s

100K .......... .......... .......... .......... .......... 87% 693K 0s
150K .......... .......... 100% 81.5M=0.4s

2010-08-21 18:15:26 (396 KB/s) - `drush-6.x-3.3.tar.gz' saved [174892/174892]

aegir-alpha-install.sh.txt: line 168: php: command not found
==> Drush is broken (php /var/aegir/drush/drush.php help failed)

The error was probably

omega8cc's picture

The error was probably because the git.aegirproject.org was down due to dns problems. It is now back online and working, so you can use the installer with default HEAD (from git) install. Don't try to install alpha11, because: 1. it is broken and requires two patches, 2. files.aegirproject.org dns is still broken and no one can install alpha11 now. I have send an alert to info@developmentseed.org about it.

BTW. Aegir has a new homepage now: http://aegirproject.org

Yes you're right -- Sorry about that

klamzo's picture

I'll try that and post my findings..on github. thanks for your response.

We can fetch all Barracuda files offline?

luiginica's picture

Hello Grace.

For situation like this I was thinking at a backup installation solution. I log the idea here: http://github.com/omega8cc/nginx-for-drupal/issues#issue/19

@klamzo - I can't find your issue at http://github.com/omega8cc/nginx-for-drupal/issues
Also, probably Grace is right, but you also have this error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Can you tell me how much free memory do you have? Like do a #top from command line.

Have a nice day,
Luigi

Yeah I will do that

klamzo's picture

I have this:

top - 22:06:19 up 3:43, 1 user, load average: 0.09, 0.08, 0.01
Tasks: 22 total, 1 running, 21 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 102776k used, 945800k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

Thanks

Unable to find a strip option in your version of tar

alexxo's picture

Using a vanilla VPS Debian 5.0
After runing the INSTALL-ubuntu-debian-nginx-aegir.sh.txt script I get no errors but instead of getting the link for the first login I only see:
$ ==> ACCESS ->

A look at the /var/aegir/install.log shows that Aegir was not installed.
All the errors seem to be related to unpacking the downloaded .tar files

'http://ftp.drupal.org/files/projects/drupal-6.19.tar.gz.'
Unable to find a strip option in your version of tar

'http://ftp.drupal.org/files/projects/admin_menu-6.x-1.5.tar.gz.'
Unable to find a strip option in your version of tar
...

Could somebody point me in the right direction to fix the issue? tks!

Hmm, that's strange

perusio's picture

I'm using a Debian Lenny like you, and my tar version has the --strip--components, -strip option.

Here's my version:

Package: tar
Essential: yes
Status: install ok installed
Priority: required
Section: utils
Installed-Size: 2248
Maintainer: Bdale Garbee <bdale@gag.com>
Architecture: i386
Version: 1.20-1+lenny1
Replaces: cpio
Pre-Depends: libc6 (>= 2.7-1)
Suggests: bzip2, ncompress
Conflicts: cpio (<= 2.4.2-38)
Conffiles:
/etc/rmt 3c58b7cd13da1085eff0acc6a00f43c7
Description: GNU version of the tar archiving utility
Tar is a program for packaging a set of files as a single archive in tar
format.  The function it performs is conceptually similar to cpio, and to
things like PKZIP in the DOS world.  It is heavily used by the Debian package
management system, and is useful for performing system backups and exchanging
sets of files with others.

Suggested user permissions for adding modules, themes etc.

tseven's picture

My server is a basic linode setup modified by Omega8cc's setup script.
I have disabled root login, and created a user account for administration.

I have a pressflow-6 platfrom setup. I'd like to add modules/themes etc, but am getting permission denied since all the folders/files are owned by aegir.

What is the recommended user permission setup for managing the site files?

Should I add my admin account to the aegir group?
'su' to aegir? (can't because I don't know the aegir's account password).

What is the default aegir account's password? Shouldn't this be changed for security reasons?

I don't think aegir has a

tribe_of_dan's picture

I don't think aegir has a password. I normally log in as root and then switch to aegir user using su -s /bin/sh aegir

If theres a best practice I'd like to know also.

Aegir is setup as a system

jamiet's picture

Aegir is setup as a system account as a result it does not have a default shell or password - this is setup this way precisely for security reasons.

You can either switch to root and then switch to the aegir system user using the command

su -s /bin/bash aegir

Alternatively if you have created a non-root user for admin purposes you should add them to the /etc/sudoers file with ALL and then you can switch directly to the aegir user with the following command:

sudo su -s /bin/bash aegir

You will be prompted for your user password (the admin user) and then will switch straight to the aegir user.

HTH,

JamieT

Perfect

tseven's picture

This was exactly what I needed.
Thank you both for replying.

Difficulty with private files

linuxgeneral's picture

I was having difficulty with setting the files to private with Open Atrium.

I can across this post:
Changes to file download method (public/private downloads) not supported
http://drupal.org/node/610912

Starting with a plain Drupal install, I created local.settings.php and /var/aegir/private_files as described in the postings.

I appears that the URL is not getting rewritten properly and nginx gives a 404 error.

Questions:

  1. Is there any edits required in nginx_advanced_include.conf to allow for a private file system?

  2. Is XSend an option?

Private downloads were not

omega8cc's picture

Private downloads were not supported in Aegir by design, before this commit: http://git.aegirproject.org/?p=provision.git;a=commitdiff;h=6a59aac56c77...

This is why my Nginx config didn't include support for private downloads before, but we need to add it now.

Still not able to import any sites with HEAD...

zkrebs's picture

Could someone take a looksy over here? http://drupal.org/node/882970#comment-3367344

This has been fixed today in

omega8cc's picture

This has been fixed today in HEAD, see the patch: http://drupal.org/node/882970#comment-3382542

omega8cc This is nothing short of awesome!

backwardgraphics's picture

Thanks for the Contributing this back to the community.
I am having an issue which appears to be related to how NGINX handles xml files. Basically I am trying to add Cooliris to my site, and when I add the required cross-domain.xml it is not recognized and asks me to add it to the root of my web folder. The the location of the file is http://healthzite.com/crossdomain.xml. The reason I believe it is related to the way that nginx handles XML is because when I goto the file above it looks as if it renders the file in in RTF format. Any help would be appreciated, However I understand that you are extremely busy. Thanks.

In this Nginx configuration

omega8cc's picture

In this Nginx configuration .xml is forced by default with mime for RSS feeds, but I have added already exceptions to support (f)ckeditor. It looks like we need to add also exception for this file. Thanks for reporting it.

Private files working for anyone?

tseven's picture

Using the default settings on Linode Ubuntu 10.04LTS 64-bit

I get a 404 File not found when trying to access files using drupal's private feature.

I have a feeling it's how the file redirects are configured in: nginx_simple_include.conf

It appears nginx_simple_include.conf ignores the possibility the files may be served by php/drupal and not accessed directly.

Does anybody have modified/new rules to allow private downloads to work again?

Thanks

Re: Difficulty with private files

linuxgeneral's picture

I am using a Linode default install on Debian Lenny

See above.

I think you are correct because if you edit the URL to remove /system or the group name in the case of a default install of open atrium the files in /private/files/ upload and display just fine.

Is anyone having problems with apache?

Got it working, sort of

tseven's picture

After upgrading to Open Atrium beta 8, and commenting out the following code in nginx_simple_include.conf and I can now download the files. But unfortunately so can everyone else; it's not private.

    ###
    ### serve & no-log static files & images directly, without all standard drupal rewrites, php-fpm etc.
    ###
    location ~* ^.+.(jpg|jpeg|gif|png|ico|swf|pdf|doc|xls|tiff|tif|txt|shtml|cgi|bat|pl|dll|asp|exe|class)$ {
        access_log      off;
        expires         30d;
        # allow files to be accessed without /sites/fqdn/
        rewrite  ^/files/(.*)$  /sites/$host/files/$1 last;
        try_files $uri =404;
    }

I've tried adding "SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006;" to my nginx config, since that's what is in the .htaccess for apache. I figure this is what tells drupal how to handle the files, but this breaks things.

Also adding "deny all;" to the config does exactly that, and nobody can access the files. It doesn't seem like drupal/php is handling the files, but is simply being redirected by nginx.

Ultimately I'd like to get the modified version of the x-send module to work: http://groups.drupal.org/node/36892.

I figured I'd should get drupal's standard private files to work first.

Without testing, probably

brianmercer's picture

Without testing, probably something like this

location ^~ ^/system/files/.+$ {
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

If private files are accessible directly then either they need to be moved out of the document root or blocked with another location like:

location ^~ ^/private/.+$
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

Answered above:

omega8cc's picture

Answered above: http://groups.drupal.org/node/84074#comment-281999

Please test the recipe from Brian and feel free to submit a patch!

Mmm...those probably won't

brianmercer's picture

Mmm...those probably won't work, instead try these. I will install a test setup this weekend and confirm.

location ^~ /system/files/ {
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

location ^~ /private/ {
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

Brian

omega8cc's picture

How about using your previous recipe? http://groups.drupal.org/node/36892

I'll test the x-send stuff

brianmercer's picture

I'll test the x-send stuff also when I get the test install up.

I didn't mention it in that post but any nginx config for private files will need something like

location ^~ /system/files/ {
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

so that private files (which are served with the following format: http://example.com/system/files/image.jpg) get served by Drupal and not directly by nginx which leads to the 404.

I've gotten things working on

brianmercer's picture

I've gotten things working on my test setup with private files and xsend.

To get private files working:

1. Change to private files at http://domain.com/admin/settings/file-system

2. Change the location of files. in aegir this is hard set in the settings.php file, so cannot be changed in the web interface. the location of the settings.php file will vary based on platform and domain name but will be something like this:

  /data/disk/host/distro/001/[platform]/sites/[domain_name]/settings.php

You want to go in there and change
  $conf['file_directory_path'] = 'sites/[domain_name]/files';

to
  $conf['file_directory_path'] = 'sites/[domain_name]/private/files';

I prefer placing private files outside the domain root, but that'd be up to the aegir devs if they want to create and manage another directory for private data. For now it seems they want to keep them inside the main tree so they've already created the /private/files/ directory used above.

3. Add these lines anywhere to /data/disk/host/config/includes/nginx_advanced_include.conf. The bottom is fine:

  ## This location is required to serve private files
  location ^~ /system/files/ {
    rewrite ^/(.*)$  /index.php?q=$1 last;
  }

4. Add these lines towards the top of /data/disk/host/config/includes/nginx_advanced_include.conf

  ## This location protects the private file directories from ordinary access
  location ~* /sites/.+/private.* {
    deny all;
  }

That should make private files work and also protect against direct access.

To make xsend work you need to

5. Add these anywhere to /data/disk/host/config/includes/nginx_advanced_include.conf. Bottom is fine.

  ## This location is for serving files from the xsend module
  location ^~ /private/files/ {
    internal;
    try_files /sites/$host$uri =404;
  }

5. Then you need to install the xsend module and hack it up as described in http://groups.drupal.org/node/36892. The module has changed a little since then. You also need to use a different directory than I used in that article. Instead of

    drupal_set_header('X-Accel-Redirect: /protected/'.$filepath);

you should do
    drupal_set_header('X-Accel-Redirect: /private/files/'.$filepath);

And that will get them to work. It's still not as fast as serving static files directly because you have to bootstrap Drupal to check that the client has access to the file. But especially for large files, it will reduce the load on the server because the entire file will not be served by PHP.

I'm not sure if this is something that aegir or omega8cc will want to support officially until it's a bit less hackish.

Great work on this project, omega8cc.

Brian, Thank you for working

tseven's picture

Brian,
Thank you for working this out. I'm trying to get this working on my site, but I'm still running into 404s. I'd like to debug this myself, but for some reason nginx isn't spitting anything out in the error.log. Logging is on, and in the older log I see some config errors, but I'm not seeing any 404, or permission denied errors.

If I can see the paths being called, I can probably work things out.

Do you know how I can turn on 404 error logging?

I just realized the error.log might be only for nginx related errors. Then I'd assume the 404 errors would show up in the access.log, but they aren't. Perhaps there is a flag I must set?

Edit:
Ok looks like I got the 404 showing up, it was being turned off in the nginx_advanced_include.conf.

"68.190.213.xxx" domain.net [29/Aug/2010:17:19:29 +0000] "GET /group_name/system/files/Agave_3.jpg HTTP/1.1" 404 198 1027 407 "http://domain.net/yuco/node/29" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.127 Safari/533.4" 0.000 "3.05"

Using OpenAtrium, the group name is being put before the file call.

Regardless, it seems that the

    ### serve & no-log static files & images directly, without all standard drupal rewrites, php-fpm etc.
    ###
    location ~* ^.+.(jpg|jpeg|gif|png|ico|swf|pdf|doc|xls|tiff|tif|txt|shtml|cgi|bat|pl|dll|asp|exe|class)$ {
#        access_log      off;
        expires         30d;
        # allow files to be accessed without /sites/fqdn/
        rewrite  ^/files/(.*)$  /sites/$host/files/$1 last;
        try_files $uri =404;
    }

is still catching the redirect before the rules for /system/files and /private/files near the bottom of the config.

Changing:

## This location is required to serve private files
  location ^~ /system/files/ {
    rewrite ^/(.*)$  /index.php?q=$1 last;
  }

to:

## This location is required to serve private files
  location ~* /system/files/ {
    rewrite ^/(.*)$  /index.php?q=$1 last;
  }

and moving it above the afore mentioned location rule seems to have done the trick.

Even though the x-send module is enabled and modifed, it doesn't seem to be working properly. Going to do some more digging.

Thanks

Debugging output is not

brianmercer's picture

Debugging output is not compiled into the Barracuda version of nginx. nginx would have to be recompiled.

At what point does it not work? Does private file serving work at all, before trying to get xsend to work?

You can edit the locations like this:

  location ^~ /system/files/ {
#    rewrite ^/(.*)$  /index.php?q=$1 last;
    return 503:
}

and then see if your 404 turns to a 503, and that will tell you if that particular location is catching the request.

Also, the version I installed yesterday "0.4-alpha12" may not be identical to the ones people installed last week. This file /data/disk/host/config/includes/nginx_advanced_include.conf on my version is symlinked to nginx_simple_include.conf so they're identical, but that might not have been the case with an earlier version.

Take a look to see if the simple version is symlinked to the advanced version. If not, you might have to make the changes to the nginx_simple_include.conf file instead.

Yes, the following redirect

tseven's picture

Yes, the following redirect is working.

  location ^~ /system/files/ {
#    rewrite ^/(.*)$  /index.php?q=$1 last;
    return 503:
}

But in the x-send module I have:

header('X-Accel-Redirect: /private/files/'.$filepath);

So shouldn't the following redirect be the one handling the files?

## This location is for serving files from the xsend module
  location ~* /private/files/ {
    internal;
    try_files /sites/domain.net/$uri =404;
  }

Yes, just trying to diagnose

brianmercer's picture

Yes, just trying to diagnose the point of failure.

The $uri variable includes the leading slash, so that would have to be

## This location is for serving files from the xsend module
  location ~* /private/files/ {
    internal;
    try_files /sites/domain.net$uri =404;
  }

and then you can temporarily change that to 503 also to see if the location is catching the request. If it is, then it's just a matter of getting the directory right.

Using the $host variable from my example didn't work?

try_files /sites/$host$uri =404;

It's working :)

tseven's picture

changing the try_files rule to:

try_files /sites/$host$uri =404;

Is working. I had to restart nginx a few times and enable/disable the xsend module. But it does seem to be redirecting correctly.

Thank you for your helpful feedback.

And of course thank you omega8cc for all your hard work :)

Sounds good. I've never

brianmercer's picture

Sounds good. I've never worked with organic groups before.

Thanks Brian!

omega8cc's picture

It is definitely worth to be included as an optional setup for those looking for those features. Maybe we could even open a wiki on g.d.o to keep this in easier to find "nginx workroom"? What you think?

linuxgeneral's picture

My results

The settings.php file will get overwritten and there is a new local.settings.php

http://drupal.org/node/610912#comment-3340240
"if you want to shoot yourself in the foot you can use the new 'local.settings.php' in the site directory to override the file dir."

But thats still a real pain to manually edit every site. I set the AEgir host to be private files with edits to the /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php.

$conf['file_directory_path'] =
Changed /files to /private/files

$conf['file_directory_temp'] =
Changed /files/tmp to /private/temp

$conf['file_downloads'] = 2;
Added to enforce private files

Now all new sites are private by default. Clone and migrate work but the cache files wont build due to permission errors

private/files/ctools/css could not be created due to a misconfigured files directory. Please ensure that the files directory is correctly configured and that the webserver has permission to create directories.

private/files/imagecache is not writeable by the webserver.

Have not tried xsend yet and I am puzzled as to why cache is even enforced.

Has anyone else had this problem?

Thanks for all the great work here!

Cluster Configuration

jerodfritz's picture

Perhaps I'm going about the cluster management in a way that was not intended. I was hoping for feedback to help debug where I am currently at.

My Setup consists of 6 virtual machines:

1 server that holds aegir master and includes a db and web server (web-master)
1 large db server (db-master)
2 additional web servers with no db (web-slave1 & web-slave2)
2 mirrored servers running HaProxy and Heartbeat with Failover IP to load balance requests to the 3 web servers

After initial server commissioning on (web-master) using Barracuda I created 3 additional servers and create aegir server nodes that verify properly. I then create a cluster server selecting the 3 webservers. Next I add platform and then create a site on the cluster. The site files are sycned to all servers but the webserver only starts on the server hostmaster lives. nginx throws the following error on the other web servers in the cluster:

Cannot assign requested address and : zero size shared memory zone "uploads"

Any suggestions?

upload_progress uploads 1m

jerodfritz's picture

could it be that provision needs to output the nginx advanced include file like this one on the remote server:

/var/aegir/config/server_master/nginx/conf.d/custom_nginx.conf

which includes the following declarations:

gzip_static on;
upload_progress uploads 1m;

It needs more testing and

omega8cc's picture

It needs more testing and some changes in the nginx configuration will be also submitted to the Aegir head. Thanks for the note.

404 on github

travischristopher's picture

I'm getting a 404 for some reason

cd; wget http://github.com/omega8cc/nginx-for-drupal/raw/master/INSTALL-ubuntu-de...

/INSTALL-ubuntu-debian-nginx-aegir.sh.txtom/omega8cc/nginx-for-drupal/raw/master
--2010-09-04 03:08:32-- http://github.com/omega8cc/nginx-for-drupal/raw/master/INSTALL-ubuntu-de...
Resolving github.com... 207.97.227.239
Connecting to github.com|207.97.227.239|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2010-09-04 03:08:32 ERROR 404: Not Found.

EDIT: never mind i went to the install.txt tried this again and it worked...

TravisC

You just kept trying the

bmx269's picture

You just kept trying the http://github.com/omega8cc/nginx-for-drupal/raw/master/INSTALL-ubuntu-de... address and it worked?

I too am getting 404

I need to update this link,

omega8cc's picture

I need to update this link, since this old script is now replaced by dual-core Barracuda + Octopus.

More information:

http://groups.drupal.org/node/89594
http://github.com/omega8cc/nginx-for-drupal

Aegir user pass?

obrienmd's picture

We use SFTP mounted drives to remotely work on theme files / etc... But would like to access these files as aegir, not another system user. However, I'm not sure how to find the aegir user password, or if changing / adding a password would screw something up... Thoughts on this?

Aegir user doesn't have

omega8cc's picture

Aegir user doesn't have enabled password. However latest Octopus version creates limited shell account with enough privileges for every instance.

static website configuration

mindtrades's picture

Hi,
Thanks for the great script.I have installed octopus and barracuda on a blank lucid vps and everything is working fine.
I have it on aegir..com put i need to have a website on the /.
Where should i configure the virtual host and how. I did try the normal standard method by creating - site-available and site-enabled in /etc/nginx and creating a v.host. I did put my static folder here : /home/admin.ftp/static/public_html/.com . i did also try to put in /var/www/nginx-default
But none of these things work. the root page on my site shows the standard under construction page.
Please help me explaining how can i make this thing happen.
I want to keep a drupal site itself on /
Thanks
Vin

CiviCRM working

Ari Gold's picture

I did get CiviCRM installed without disabling APC by following the comments in
/data/disk/octopus-instance/config/includes/nginx_simple_include.conf and nginx_advanced_include.conf
and /etc/opt/php.ini

one problem i had was that i was editing the wrong nginx conf files in /var/aegir instead of /data/disk

Low RAM tuning

dev286's picture

First, great job on Barracuda and Octopus... Super fast! Everything works great so far.

I experience some performance issues... My setup is on Linode 512, Ubuntu 10.04 LTS, basic Drupal platforms. After boot, the whole thing runs really fast, but after it begins to hit virtual memory it slows down... just a little, maybe 5-10 seconds. I guess this is due to the virtual memory access time on the server. Can you suggest any performance tweaks for low RAM setups.

Thanx
dev286

Have you deployed any sites

Anonymous's picture

Have you deployed any sites yet, or is it hitting VM with just the hosting site running? There are min memory requirements of 1Gig for some of the more complex (Openpublish - Atrium) platforms due to the complexity and number of modules. Check the latest barracuda build, it has some php.ini changes in the memory from previous builds (went from 512M to 256M, do not tweak any lower than 256M). Also, the current build has the MySql database tweaked pretty good for a low memory environment.

Peace,
Michael Clendening

With low memory VPS you also

omega8cc's picture

With low memory VPS you also shouldn't install Solr/Tomcat etc. The defaults are now tuned to run on 512 MB of RAM, and you can use the TUNER script to adjust some other limits if you want. But anyway, recommended minimum is 1 GB of RAM.

One site deployed in Drupal 6.19

dev286's picture

I have one site deployed in Drupal 6.19 I am working on and most of the platforms installed (not used). I did update the install to AEGIR_VERSION=0.4-beta2.

The website has about a dozen extra modules installed and all caches turned off for development. I do have Solr/Tomcat installed coz I thought it is a requirement. What other packages should I get rid of to tune it further?

Thanx
dev286

Also, keep in mind that

omega8cc's picture

Also, keep in mind that Linode has rather bad I/O performance in general so you should increase the RAM to avoid swapping which slows everything there drastically. Or change memory limits in /opt/etc/php.ini to something like 128MB etc.

Running Cron from localhost

bailsbails's picture

I've come across a problem with BARRACUDA + cron.

We use wget on localhost to fire off cron jobs e.g

/usr/bin/wget -O - -q -t 1 http://fqdn.com/cron.php

With the default nginx_simple_include.conf. We were getting a 444, we fixed this by removing wget from:

if ($http_user_agent ~* (HTTrack|HTMLParser|wget|libwww|AutomaticSiteMap).

Which resulted in a 403, we fixed this by:

location ~* /(cron|install|drushrc).php$ {
allow 127.0.0.1;
deny all;

Now we get a 200, however cron still never runs. Can anyone shed any light on this?

Cheers

Bails

I am no expert, but have been

Anonymous's picture

I am no expert, but have been following the latest Barracuda updates on a daily basis, see this commit over on Git concerning memcache:

https://github.com/omega8cc/nginx-for-drupal/commit/f5dcfb80053bd0418319...

Peace,
Michael Clendening

Instead of wget, you may want

brianmercer's picture

Instead of wget, you may want to use drush for cron. drush @[sitealias] cron -q

Hmm, that's not documented

perusio's picture

the -q switch is not documented anywhere. Not as a global option, not as cron command option. I've peeked the code and it invokes drupal_cron_run and that's it.

I'm using the version from HEAD. The way I silence cron is by using the rather ugly redirection:

drush @my-site cron 2> /dev/null

Otherwise I always get a mail from cron, when it runs.

I just filed a bug about this

brianmercer's picture

I just filed a bug about this on 4.xbeta last week. http://drupal.org/node/1010630 The -q switch was working fine in the latest 3.x version and then it started spamming me when I went to 4.xbeta5. It's been fixed now and if you're using 4.1 then -q works fine for cron.

BTW, I upgraded my ppa:brianmercer/drush to version 4.1.

Barracuda by default and by

omega8cc's picture

Barracuda by default and by design denies access to cron.php because it is insecure and you don't need it. Aegir already runs cron for all your sites (using drush) and there is no need to add anything to the system cron.

Your regex

perusio's picture

has a bug.
It should be:

location ~* ^/(?:cron|install|drushrc)\.php$ {
  allow 127.0.0.1;
  deny all;
  ## Oops no FCGI invocation here. Nothing will happen since there's no way Nginx can handle PHP per se.
}

I fail to see the logic in adding the drushrc.php to this location. Shouldn't it be located in ~/.drush? Furthermore it's drush run control file and has absolutely nothing to do with Nginx, which doesn't care about drush at all.

If you want to run cron through the web using wget or curl, then use something like this:

location /cron.php {
     allow 127.0.0.1;
      fastcgi_pass unix:/tmp/php-cgi/php-cgi.socket; # change for TCP socket instead of UNIX
      error_page 403 =404;
      deny all;
}

This way you're adding a little security by obscurity kind of thing. Since if someone tries to run cron it will get a 404. Thus not disclosing if cron.phpis in place or not.

I introduced drushrc.php in

omega8cc's picture

I introduced drushrc.php in this location because every site created in Aegir does have this file both in the drupal root (for platform) and in the sites/domain (for site). In fact, this is an extra paranoid mode, because that file gets secure permissions anyway, so it is not available for the web server, but still, it is always possible someone changed the chmod while working on the platform/site and didn't run platform verify, so I prefer to secure that file also on the web server level.

Oh, I see

perusio's picture

My ignorance on Aegir is beyond obvious :) Makes all the sense denying access to drushrc.php to anyone not on the same machine. It also solves the issue of being acessible from the web on external addresses.

I might add also

perusio's picture

That relaying the responsability of running cron to the CLI via drush, leaves your CGI/whatever process that is running the web serving part of your site less strained. And depending on the setup of your site, cron can be a quite heavy thing.

==============>omega you

AntiNSA's picture

==============>omega you said that octopus supports imagecache from teh go http://groups.drupal.org/node/84074 But I can tell you on a clean install of a fresh drupal 6 pressflow sites with no modules enabled but the transliteration, imageapi and imagecache required, imagecache is not working. When trying to create a preset image size, the sample image is created in the root of the /files folder, and no preset directories are created within the /files/imagecache folder. I have not touched or altered any settigns but to install the image cache mosule and attempt to use it.

It was not working on my imported site so I decided to try it on a fresh drupal 6 site in order to verify I was not making an error.

http://community.aegirproject.org/node/308#comment-292 is the link to were I describe the same issue.

I'm experiencing the exact

tseven's picture

I'm experiencing the exact same behavior in Cocomore.
I'm not 100% sure the bug is drupal. Two of my other sites running pressflow on a standard LAMP stack are not having issues with imagecache.

What about the barracuda/octopus setup could be affecting the creation of the thumbnails?
Is anyone else experiencing this issue?

My setup:
Debian 5 Lenny
Barracuda/Octopus
Cocomore
All latest versions.

It works for me....

dev286's picture

I works for me fine... structure and files.... 30+ modules enabled....

I have a vanilla

Anonymous's picture

I have a vanilla Barracuda/Octopus with latest revision having the same problem on a bare metal (non VPS) Ubuntu Lucid.

Peace,
Michael Clendening

Got this figured out...at

Anonymous's picture

Got this figured out...at least on my small planet;

Image cache is working in this way for me on a Barracuda/Octopus (latest version) Lucid bare metal install.
I did not need to make any changes to the directory or permissions.

Modules:
imagecache
imageapi
imageapi imageMagick
CCK
FileField for CCK
ImageField for CCK

Steps to test:

  1. Enable the modules CCK, FileFiled, ImageField, Imagecache, Imagecache_UI, Imageapi, Imageapi ImageMagick
  2. Go to Home → Administer → Site configuration → ImageAPI and choose ImageMagick for the image ToolKit
  3. Go to the configure tab and turn on debugging (radio button at bottom, will turn this off later)
  4. Go to Home → Administer → Site building → ImageCache and create an Imagecache preset
  5. Go to Home → Administer → Site configuration → Input formats and select configure Full HTML and add the "img" tag to the allowed HTML tags
  6. Go to Home → Administer → Content management → Page and select Manage Fields and add a new file field to it and choose image under the widget
  7. Once the file field has been added, select configure and go to the insert section and check "Enable insert button" and check the box with the name of the preset you created in step 4
  8. Go to Create Content → Page and expand the Input Format section and select Full HTML. Give it title and upload a pict. Place your mouse in the body area. Go to the upload Style dropdown box and select the Imageache preset and click insert.
  9. Click the preview button and notice the debugging information at the top of the webpage...you should see all green text with the file path info.

The bottom line for me is that there is no preset directory created in the imagecache directory until I actually use the preset.

(Great Job once again Grace!!!)

Peace,
Michael Clendening

I can verify on Debian 5 its

AntiNSA's picture

I can verify on Debian 5 its not working, I have tried imagemagic and G2, and it is not working on both imported site and fresh install. I am updating the newest head from gitorius now. will try the above method... Althought I am not using lucid, and I prefer to use G2/// Will update stus within the next hour.

After updatin to the newest

Disable Caching on per-site basis

dev286's picture

A newbie question...

How do I disable caching on per-site basis for development... When I do it through the Performance page no changes are saved...

Thanx!

p.s. Is there a chart of the Omega8 server setup with explanations?

You can find the cache

tseven's picture

You can find the cache setting in your site's settings.php.

You can change it here, or comment out, but its not recommended since it could be overwritten by aegir.

Aegir allows you to override these and any other settings by creating local-settings.php and placing your config variables in there.

You can see this file is included at the bottom of settings.php.

You can get more info on how Aegir works here.

LEMP + Aegir

dev286's picture

I'm looking for LEMP + Aegir description...
I don't have Server Admin experience and trying to figure out the whole thing...

not spamming

zkrebs's picture

but i had a hard time figuring it out, and wrote about barracuda aegir here: http://www.mindfulintegrations.com/home/blog/2011/01/11/what-are-barracu... simple explanations...

I would just use dev in the

stevenwood's picture

I would just use dev in the subdomain title this will disable your settings.

devsub.mydom.com

Doh....

dev286's picture

Doh.... forgot about that......
Thanx!

A question about Cache module on Octopus

Strange domain error.

bmx269's picture

I am trying to clone, and migrate a site that using a number as the first character in the domain name. I am getting this error: "You have not specified a valid domain name for this site. It should start with a letter." I am using the stable 1.0 of Barracuda / Octopus from about a week ago. What is the setting to allow this to go through, as I need to migrate the site today?

Please don't cross-post

omega8cc's picture

Please don't cross-post issues, we don't use g.d.o as an issue queue, and your question has been already answered by Robert, 9 hours ago: https://github.com/omega8cc/nginx-for-drupal/issues/254

New to admin

yazzou's picture

Hello,
I am fairly new in system administration. It took me one month to set up a server with Nginx, Apc, php5-fpm. I got my first drupal site running on it and it is running great untill know (i have migrated this site from joomla!) . I have other sites running on the same server. Joomla ones and other based on framework smarty. Now i have been watching some videos about Barracuda and the hosting solution Aegir..i want to have this on my server. What would you advice ? Do i have to back up everything install your system and import my sites again ? Can I run other sites if i do so ?

Is it possible to simply run the barracuda over my current instalation ?

Thanks in advance for your help
Best Regards
Yaz

@yazzou

omega8cc's picture

You need fresh, minimal OS install on the server or local VM to install Barracuda and Octopus. You should never run it on any system with already installed Drupal (live) sites.

Where to start from:

http://drupal.org/project/barracuda
http://drupal.org/project/octopus
http://groups.drupal.org/boa

Good luck!

P.S. I'm sorry for the late reply in this old thread.

Thank you very much for your

yazzou's picture

Thank you very much for your reply. I have followed your advice and acquired another server on wich i installed octopus. Everything went well and i got aegir running. So now i will have to import one by one all the sites to my newly instlled server. As i understood, dns settup has to be done by me...i mean creating a new site on Aegir and (maybe) unabling the aegir dns feature is not enough...
Also, i want to be sure about security and logs. Do i have to add some other firewall (fail2ban) and logwatch, etc....or your barracuda install is just enough.
Thanks agin for your help

Grammar Police

markjbrown's picture

I'd like to report them to the grammar police as well

I guess everything we can do

omega8cc's picture

I guess everything we can do is to rate spam comments with -1 points so it grabs attention of g.d.o admins hopefully, but we really need the "report spam" button..

Not quite

perusio's picture

We can open an issue on the infrastructure issue queue and report it.

Done!