We are proud to announce the 0.4 Alpha 9 release of the Aegir Hosting System. This release introduces our 'multi-server' functionality, which mostly resolves the primary goal for the 0.4 release cycle.
UPDATED: critical errors found in this release:
#875192 PHP.net switched to subversion from CVS, breaking Drush. We need to roll a new release of drush.
#874716 Files directories on remote web servers are being deleted on rsync. DO NOT USE THIS FEATURE ON PRODUCTION SITES YET!
Expect us to roll new releases of Aegir and Drush with these issues resolved by early next week.
Major updates :
- Simplified installation by removing the hosting wizard. The entire process is automated now.
- Entirely rewritten backend to allow for multi-server support.
- New backend makes heavy use of drush site-aliases, allowing provision to be used without the hosting front end.
- Introduced the ability to host sites across external databases or web servers.
- Merged and simplified migrate/clone/restore functionality, to allow a site to be renamed during migration, or to have it migrated between database servers in the process.
- Entirely rewritten site form to allow for usability and API improvements.
- Changed the functionality of the site form so that a user first chooses the install profile, and then chooses the platform to create the site on. This simplifies the form and makes it easier for users to get their expected results.
- Entirely rewritten 'ports' configuration to handle most common situations.
- Introduced support for SSL encrypted sites.
- Introduced NGINX web server support. (Contributed by Omega8cc)
- Introduced support for web server clusters.
- Various usability and performance improvements.
The last outstanding feature before we can consider the 0.4 release feature complete, and move onto beta testing, is the ability to provision and manage DNS for your hosted sites. This is even more important now that we can migrate sites between web servers.
DNS support is slated for Alpha 10, but for now enjoy what we feel is by far the best release we have put together.
This release was made possible partly through the sponsorship of Chapter Three, who contributed significant resources to the project allowing us to focus on getting the details right.
--
The Aegir Project
Installing and upgrading :
The canonical source of installation documentation is as usual the excellent INSTALL.txt. We have an automated and scripted upgrade process for the front end, which you can read more about in UPGRADE.txt.
For those updating from before 0.4-alpha8, please note that there are instructions at the bottom of the UPGRADE.txt for you. This obviously also applies to people upgrading from 0.3.
NOTE REGARDING SSL: If you had the SSL module/hosting feature installed from the previous alphas, you need to disable it first before attempting an upgrade.
This release also requires fixes that are only present in the 3.1 release of Drush. The install script will automatically fetch the correct release of Drush for you, but you need to upgrade Drush manually when upgrading Aegir.
Tasks completed :
#836162 : Introduce Apache w/ SSL service type
#712958 : Assign IP to virtual host based on SSL certificate chosen.
#596408 : Origin field for DNS zones should be copied into node title
#596412 : Add a message telling that the zone file is updated after verify task
#836166 : IP Pool management for SSL
#836136 : Simplify ports support.
#533760 : 0.4 - D7 support: set $conf['cache'] : 0.4 - D7 support: set $conf['cache'] properly
#319257 : Configure what gets created upon signup
#770658 : Add server verify task
#586000 : make a "server verify" task
#699698 : correct documentation of required MySQL grants
#723288 : Introduce the concept of 'shortnames' for all major entities.
#736742 : Extendable API to manage Drush aliases.
#564824 : Upgrade documentation can be confusing
#736726 : Refactor provision to match the new Services API.
#770482 : Try webservd group for OpenSolaris
#269520 : Move to PDO for all database access
#711760 : separate backend and frontend install process
#772824 : rip most of the wizard out of hostmaster.profile
#782436 : hosting_task module function hosting_task_fetch_tasks can be made more efficient in the following two ways
#736706 : Separate out the http service, file service and backend service.
Feature requests :
#838044 : Allow naming backups.
#537016 : simple certificate management
#537020 : create SSL certificate content type
#366816 : dns zones breadcrumbs
#475756 : Add permissions to view packages
#533330 : Quota API
#783808 : OS X Aegir install HINTS file
#267592 : Support symlinks for import
#588424 : Default port for web servers
#570980 : allow vhosts of the same name on a different port
#666636 : provide "drush help" documentation
#366418 : 0.4: multiple web server support
#823978 : Dont create an additional verify task when migrating
#605414 : multiple database server support
#369722 : Check vhost.d is directory
#808828 : Goto Site Link enhanced - patch and icon
#500362 : dispatcher locking
#535098 : domain redirection
#697954 : user 1 name is hardcoded. allow provisioners to specify name for admin user.
#374182 : site rename frontend interface
#275737 : Rename sites (provision/backend)
Bug fixes:
#500342 : Aegir ignores the 'name' of install profiles in frontend
#597750 : 16 mysql limit on username exceeded by automatic creation of username if more than 1 digit (more than 9)
#475776 : view own tasks doesn't show own tasks
#761498 : warning: Parameter 1 to hosting_nodeapi_platform_load() expected to be a reference
#831172 : Primary links issue when message box is visible
#839968 : Hostmaster install profiles with implode warnings
#849976 : hosting_platform.module doesn't support database prefix
#732836 : Client node sql error
#827258 : php 5.3: parameter 2 to hosting_task_confirm_form exposed to be a ref
#585814 : client/user relationship UI present even when client feature is disabled
#767030 : Bunk user_access call in hosting client page
#730040 : Compatible themes marked as "incompatible with this release of Drupal"
#683926 : hostmaster-migrate doesn't fail if drush make fails
#831304 : allow using a language that is not installed in the platform for existing nodes
#475764 : changing the client of a site doesn't really work
#830866 : strtolower site titles when site nodes are created
#724182 : Re-save site node: "not specified a valid port for this site"
#823360 : Static table names in hosting migrate
#829418 : upgrade to head fails
#726984 : all sites go to aegir
#485646 : Site migration fails on multiple SQL servers
#739830 : OOM failures go undetected in some circumstances
#722286 : provision-verify fails on hostmaster install with remote database
#695370 : deploy fails with remote SQL server
#722512 : can't create db servers even if admin
#821612 : Hostmaster platform verify throws php errors
#818568 : Platform lock throws hook errors
#784860 : Drupal 7 Alpha 4 Provisioning Broken?
#733860 : Calls to provision_path() should not use dt() for success/failure messages
#772616 : call to provision_drupal_provision_apache_dir_config() creating in-effective rewriterule in platform config file
#769800 : Hosting signup form: site form was altered twice
#475782 : viewing tasks Disable and Enable in double
#592730 : restore destroys the database after restoring it
#641506 : How I got 0.4 alpha3 working on debian lenny
#595294 : Migrate package comparison assumes 6.x-1.x-dev is older than 6.x-1.0
#600508 : aegir suggests the wrong hosting setup uri for drush when installed on a non-default port
#814368 : PHP E_ALL fixes
#814356 : PHP E_ALL fixes
#599790 : eldir hates views2 with a passion
#811574 : One-time login link I'd been sent didn't expire when it said it would
#710830 : "compare platforms" only displays if openened in a new tab
#486934 : broken IPC communication on Drupal 6 (AKA "welcome email not being sent")
#783374 : Verifications of platforms containing sites throw package-related PHP errors
#799152 : Broken links on Ageir group documentation page
#559006 : dispatching aegir leads to high load (slow query)
#336937 : Retry button css issue in opera
#712016 : Provision-intsall fails due to db_host defaulting to localhost, FQDN expected
#594940 : Added new platform, all .conf files missing.
#592350 : Migrating a site using provision deploy fails
#665950 : provision backup fails if files have been uploaded
Comments
Wow guys this is a huge
Wow guys this is a huge release!
Will take me a day to check out the new stuff!
Thanks for all your hard work - aegir is awesome stuff.
Totally mind-blowing
Totally mind-blowing progress!
Congratulations!
Great work guys. I'll get on upgrading asap.
Just Upgraded From 0.4-Alpha8
Hi - excellent work thanks for all the hard work.
I have just upgraded from alpha 8 to 9 and had a few minor hiccups.
1 - when issuing the hosting-migrate command I received warnings about /var/aegir/config/server_master and it subdirectories not existing. It looks like all of the directories were created at somepoint though as they existed after hosting-migrate.
2 - once the hosting-migrate command has completed a number of verify tasks were kicked of for all the platforms - however this included trying to run a verify task for platforms I had deleted these obviously failed.
3 - apache config was broken for my sites and I believe this related to item 1 above. This was because the apache.conf file in /var/aegir/config/apache.conf became a symlink to /var/aegir/config/server_master/apache.conf within the new directory and the include lines changed to use vhost.d platform.d and conf.d directories within /var/aegir/config/server_master/apache/ directory. Re-Verifying each site fixed this configuration issue and sites were back online again. Also I now have /var/aegir/config/apache.d vhost.d and platform.d directories - are these now redundant and should I delete them?
Nothing major and maybe related more to my interpretation of the upgrade text than a problem with the upgrade but thought I'd let you know how it went.
Thanks again,
JamieT
1 - when issuing the
This is expected. It is saying it didnt find the dirs, so it needs to create them.
cosmetic issue, doesnt really do anything.
We now generate the files in a different place (as you can tell). You can delete those files if you want, or keep them around just in case.
Issue with upgrade - SSL related
Hello - I attempted an upgrade and now my system's offline. During the
I receive warnings:
Then aegir is adjusting the database, and has the following errors:
and after the error:
The result is that I have an unfinished install, and my original hostmaster-0.4-alpha8 no longer works (am now looking into restoring it).
I am hoping one of you can help me resolve this issue faster than I can by myself! Is there an easy fix?
Thanks!
Okay, fixed it.
GOT MY SITES WORKING AGAIN
I had to add "NameVirtualHost *:80" back into my apache http.conf file. Somewhere along the way I had that added into the /var/aegir/config/apache.conf file. When I upgraded to alpha9, the file was overwritten, and not backed up, so tracking it down took me a bit of effort. (I am not an apache expert.)
My suggestion would be to back up any files that would be overwritten for future restore or triage. Now that we're on with the /var/aegir/config directory - that whole directory can be backed up in case of failure.
TURN FEATURE - SSL SUPPORT - OFF BEFORE UPGRADE
I turned the experimental SSL support OFF and then performed the hostmaster-migrate command. Everything worked smoothly after doing this.
MORE BULK TASKS NEEDED
I had to manually verify all of my platforms and sites since they did not do so automatically.
Is it simple to add a task in provision that will re-verify all sites on a given platform?
Heck, why not have platform-level tasks - back up all sites on a platform. Why should migrate get all the glory?
And why not server level? Why stop at platform?
My thought here is that while re-verifying 20 sites one at a time is not a ton of work, what happens when someone reaches 1000 sites on several platforms and servers?
I had to add "NameVirtualHost
Never edit any file generated by Aegir. It will overwrite it. It's how we can make sure things keep working.
We also do generate a NameVirtualHost line, but we do it for all the ip addresses. So ours is more specific.
This is new in alpha 9, so if you re-verify all your sites all the config files will be rewritten by the more specific $ip:$port version.
The way aegir works is we store the information we need to generate the config files whenever we need to. The data is important, not the config files themselves. This is why you can switch from nginx to apache and vice versa.
This was an oversight, we usually add in an update function to re-verify everything.
<
blockquote>
Is it simple to add a task in provision that will re-verify all sites on a given platform?
<
blockquote>
Verifying a platform should trigger verify tasks for all the sites on it.
Backups should be handled in a regular scheduled manner. IE: all sites backed up in a certain timeframe.
The big issue we have at the moment is that we dont delete the older backups. This means disk
space would run out.
Yes, it will take some time. Generally when aegir upgrades happen the sites will still be accessible until they have been re-verified. Because we changed the config file layout in this release we had to take a different approach.
It will still take a long time to verify thousands of sites. The reason there arent more bulk tasks is because it's easy to bring your server to your knees with them.
I had to manually verify all
How did you manually verify the hostmaster platform?
EDIT: Fixed. Working great. Thank you
there should be a fix
try the following :
<?php
drush @hostmaster sqlc
mysql> delete from system where name='hosting_ssl';
?>
In alpha8 we shipped with an experimental SSL module, that got removed completely in development.
The new module therefor doesnt have it's schema installed if you had ssl enabled before.
Just in time :-)
This came out just in time for my new Linode VPS(after first shot on my mac) where I was planning on testing Aegir and then hopefully Mercury/Pantheon when they come out with their release with Aegir integrated with it.
I had also just watched Mig5's old (a5?) video of the install process which showed how simple the install can be. One question (since I'm a bad typist). In the video he copy & pasted many of the commands. What was he using to run his test server, Parallels or VMware? I haven't used my copy for a while but I'm doing my test under (Parallels v4) and I can't seem to get copy&paste to work. I could have sworn I used opy&paste before, and Mig5's video implies that it can work with Ubuntu. Ant hints appreciated.
the install process was changed
so those videos arent applicable anymore. it still involves a lot of copying and pasting, but it's much shorter and simpler now.
I have aegir installed directly on my mac using MAMP, and on some vmware ubuntu instances. fwiw.
I know the video is out of date(things have gotten better since
I know the video is out of date(things have gotten better since alpha 5), but it was still good to see as a motivator and confirmation that the process may be technical but well documented.
The copy&paste itself really wouldn't be a problem, if I could get parallelsV4 (on my Mac) to actually do it. But your config looks like another vote for VMware Fusion.
I just upgraded my aegir
I just upgraded my aegir installation to the latest 0.9 release. The upgrade was successful. However, I cannot move all sites from my 0.8 platform to 0.9 platform. The error that I get is
warning: Parameter 3 to hosting_migrate_platform_batch() expected to be a reference, value given in /home/aegir/hostmaster-0.4-alpha9/includes/batch.inc on line 189.
I also got an error message while updating the frontend. It says
6011 - Clients and packages were imported as belonging to the anonymous user.
Reassign them to the admin user.
Do you wish to run all pending updates? (y/n): y
WD php: Parameter 1 to hosting_site_nodeapi_site_presave() expected [error]
to be a reference, value given in
/home/aegir/hostmaster-0.4-alpha9/includes/module.inc on line 483.
Dispatch command was run successfully
I get "warning: Parameter 1
I get "warning: Parameter 1 to hosting_site_nodeapi_site_presave() expected to be a reference, value given in /var/aegir/hostmaster-HEAD/includes/module.inc on line 483." When I add a new site. I started from scratch with alpha9. I was hoping to add a few sites before I upgraded to alpha10 for testing purposes. I know the upgrade process was messed up. I have been following aegir of and on since 0.2.
--
Christian
Adrian, 10. Introduced NGINX
Adrian,
I have around 10 sites in aegir (apache). I just upgraded aegir to 0.9 version. How do I now move my aegir installation to nginx? is there any documentation available on how to move aegir from apache to nginx?
there is some documentation
afaik it's only been tested on debian.
Additional server configuration for nginx will also be necessary, off the top of my head :
In the aegir hostmaster
Keep in mind there might be a weird clash here because apache is still running and trying to listen on port 80.
It might be a good idea to edit the server node and have apache host on port 81 first. This will then re-verify
all your sites and put them on $url:81.
When you then change to nginx, put it on port 80 again, and all your sites will move back but on nginx this time.
Correct me if i'm wrong omega8cc?
Using one of these newer
Using one of these newer cloud hosting options, would it be possible to migrate to NGINX by setting up a new VPS and installing Aegir w/NGINX fresh, then adding it as a server on the old Aegir/Apache install, then just sending the sites over that way?
it doesnt work that way anymore
You dont need to install aegir on each of the web servers anymore.
But yes, you can migrate your sites to the new server. Keep in mind that their IP addresses will change, so DNS will need to be modified.
Also, platforms are associated to servers, not sites. You should be able to just edit your platform nodes and select a different web server.
FYI
It was tested many times not only on Debian Lenny. It works/tested also on Ubuntu Lucid and Karmic. Other system not confirmed yet, however every system where you can install php-fpm and Nginx should work. The lowest supported Nginx version is 0.6.36.
When switching from Apache to
When switching from Apache to Nginx there is more to do than just installing Nginx, since it needs also working php-fpm daemon. Therefore it's not just
apt-get install nginx
. The php-fpm is not yet available out of the box with anything likeapt-get install php5-fpm
for php 5.2.x only php 5.3 is available for this kind of install, so we need to build php-fpm from sources to get it working on Lenny, Karmic, Lucid etc. This is why I recommend to start with the all-in-one install script (or use it as a reference if you want to start with already used system). It is explained in the Nginx group on g.d.o at: http://groups.drupal.org/node/84074Correct. The missing Nginx
Correct. The missing Nginx hints in the docs should say something like I wrote in comments and in the issue queue before:
http://git.aegirproject.org/?p=provision.git;a=blob;f=http/nginx/nginx.c...
http://drupal.org/node/864214
So on the OS level you need first:
$ echo "aegir ALL=NOPASSWD: /etc/init.d/nginx" >> /etc/sudoers
$ apt-get install nginx
$ ln -s /var/aegir/config/nginx.conf /etc/nginx/conf.d/aegir.conf
$ sudo /etc/init.d/nginx restart
And when you want to start fresh, use the Nginx variant of install:
$ su -s /bin/sh aegir -c "sh install.sh.txt aegir.example.com --http_service_type='nginx'"
Awesome work Ades!
That's a long list of issues! Well done:)
Nice job
Nice update guys, I updated without to many problems, but I just found out that cron wasn't working anymore
aegir: crontab -e has the following lines
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
*/1 * * * * /var/aegir/drush/drush.php '@hostmaster' hosting-dispatch
This seems to work, cos /admin/reports/dblog shows me cron run completed every 2 hours
But all my sites show that cron was last run more than a day ago, so I checked the hosting_site table and this shows that crons was run like 30 minutes ago
Any ideas?
Have you tried reverifying
Have you tried reverifying the problematic sites. Alpha9 uses the new aliasing feature of drush and maybe a verify fixes this for the sites? It may help not sure?
Regards,
JamieT
Trying right now, I'll let
Trying right now, I'll let you now
update: it's working, i had aliases for all my platforms, but not for the sites. Verifying seems to help.
question: how to verify all sites fast and easy?
Following upgrade.txt to go from alpha7 to alpha9 issues
Having issues going with the hostmaster-migrate command:
drush_variable_post_hostmaster_pause
drush_watchdog_post_hostmaster_pause
Dummy connection failed to fail: ERROR 2003 (HY000): Can't connect to MySQL server on 'web1' (111) [error]
Dummy connection failed to fail: ERROR 2003 (HY000): Can't connect to MySQL server on 'web1' (111) [error]
Created web1praececom database [success]
An error occurred at function : drush_db_pre_provision_deploy [error]
Dummy connection failed to fail: ERROR 2003 (HY000): Can't connect to MySQL server on 'web1' (111) [error]
Dummy connection failed to fail: ERROR 2003 (HY000): Can't connect to MySQL server on 'web1' (111) [error]
An error occurred at function : drush_provision_drupal_provision_migrate [error]
An error occurred at function : drush_provision_hostmaster_migrate [error]
Have You Figured This Out?
Obrienmd. I have encountered the same problem. Any idea what the solution is? Thanks.
HAJ
Same errors here.
Same errors here.
I'm wondering if it's related
I'm wondering if it's related to this at all: http://drupal.org/node/875192
I thought I'd seen something about missing "Console_Table/Table.php" amidst the flurry of other errors. Sounds like a pretty serious issue: "This isnt killing aegir installs, it is killing ALL drush installs."
Have You Figured This Out?
The problem here is with the /etc/hosts file and the URI you enter during the hostmaster-install for your aegir server (e.g., aegir.example.com). Whatever URI you enter here must be in /etc/hosts and must resolve to an IP address whether public or loopback (e.g., 127.0.0.1).
HAJ
You guys rock my effing
You guys rock my effing socks!
Site down
Haha, right when I'm ready to upgrade, aegirproject.org is down.
Nice job guys! \o/
Nice job guys! \o/
I just tried to upgrade from
I just tried to upgrade from alpha7 to alpha9 following UPGRADE.TXT. I got this string of errors:
Dummy connection failed to fail: mysql: Error reading file [error]
'UNOPENED' (Errcode: 11)
Dummy connection failed to fail: mysql: Error reading file [error]
'UNOPENED' (Errcode: 11)
Created aegirsilverdeerl database [success]
An error occurred at function : drush_db_pre_provision_deploy [error]
Dummy connection failed to fail: mysql: Error reading file [error]
'UNOPENED' (Errcode: 11)
Dummy connection failed to fail: mysql: Error reading file [error]
'UNOPENED' (Errcode: 11)
An error occurred at function : [error]
drush_provision_drupal_provision_migrate
An error occurred at function : drush_provision_hostmaster_migrate [error]
When I visit my Aegir site, of course, it reports that all of the tables for Aegir cannot be found.
What has gone wrong here?
Thanks.
HAJ
Recommended Linux Distro?
I'm in the process of selecting a host for my new VPS and I'm wondering what distribution to choose to run aegir. I have experience with Ubuntu 10.4 LTS and I've tried installing aegir on ubuntu 8.10 (which gave me errors). Is there a distro of choice?
I'd like to go with 10.4 but I'm worried php5.3 will give me hassles. It seems most modules are now on top of the 5.3 issues but as per the INSTALL.txt, its still recommended I stay with php5.2.
So I thought I'd throw it out there to the community. What is the collective wisdom on this issue? Thanks.
a hack's opinion
As someone who had no idea 7 years ago. I spent a lot of time on CentOS (which is fundamentally equivalent to Redhat) because it was "recommended" for cPanel (a server administration thing written in perl I think). I did a few things on the command line.
When I was introduced to a Ubuntu Server it was literally like a breath of fresh air. All the configuration made so much sense to me. Since then I've moved to Debian because I learned that Ubuntu was a derivative of Debian. I prefer Debian now, but you google "Ubuntu guide to anything" and the articles you find are easy to read. Even though they only work perfectly on Debian 90% of the time, I still sometimes google ubuntu for debian issues.
So skip CentOS/Redhat. Debian/Ubuntu you'll find a lot of Drupallers use Debian, including Aegir devs (is this right?). Other distros I couldn't comment on of course.
Go for PHP 5.2. Apparently the latest Ubuntu ships with 5.3, so look out for that becuase there are still crinkles with Drupal and PHP 5.3
You can use my "Aegir with
You can use my "Aegir with Nginx for Dummies Edition alpha9e (Lucid, Karmic and Lenny compatible)" installer: http://groups.drupal.org/node/84074
It should work on any VPS with Ubuntu 10.04 LTS (Lucid) minimal install. Readme and Changelog includes full description/details.
What a good job!!
What a good job!!