Progress update - To D6, or not to D6

Events happening in the community are now at Drupal community events on www.drupal.org.
adrian's picture

We're currently experiencing a bit of a roadmap issue. I spent a few hours running hosting through the absolutely amazing Deadwood module. I've updated the ticket for this issue

One of the major goals of this project is to keep it easy to maintain, which in many cases means not biting off more than we can chew,
but it also means avoiding hacks and non-standard code as much as possible.

During development of 0.1, we ended up having to manage SEVEN distinct branches. One each for hosting and hostmaster (5.x-0.1), two for hostslave (5.x-0.1 and 6.x-0.1) and 3 for provision (5.x-0.1, 6.x-0.1 and HEAD).

Even though we spent a not-insignificant amount of time minimizing differences between the different provision branches, it still meant applying every patch multiple times. To be able to use our time most efficiently, we aim to maintain only one current release, and ease upgrade to the latest version as much as we can (this is kind of the goal of the whole project after all).

So this leaves us with a problem. We have a stable Drupal 5 front end, but we are heavily in development of both the 0.2 backend and the 2.x branch of Drush on which it depends. Because of the change in architecture that Drush 2.x allows us to implement, it's inevitable that we will need to make fairly large changes to the front end to support them (although our excellent API does mitigate this somewhat, we were able to implement basic site upgrades through a simple contrib module : http://drupal.org/node/319269)

The easiest way forward would be to keep the stable Drupal 5 front end, and only implement the new 0.2 features on it. This would provide a stable release to migrate to D6 from for 0.3. This is our current course of action. The Drupal 5 front end would still have full support for provisioning Drupal 5, Drupal 6 and Drupal 7 sites.

But there are some desirable reasons to attempt a D6 migration too. Some of the kludges in the D5 release were specifically implemented to mimic D6 features we didn't have yet. The installation wizard could get implemented properly in D6. The hiding of node types based on features could get implemented in hook_menu_alter, instead of being a hack of how menus worked in d5. The features include files could very easily be replaced with entries in the .info files.

Not going D6 now will make the D6 port more costly too, as we will have more contributed modules that need to be ported. It will also mean a shorter time between the D6 and D7 ports.

It could also complicate the upgrade process, so that users would need to upgrade to 0.2 before going to d6 0.3.

Having a D6 port also opens us up to being able to implement views 2.x, which is a very useful feature that could stand to remove a bunch of the listing code we are using.

This issue will probably be decided soon, but I would appreciate any input that people might have on the subject.

Comments

i support the d5 status quo

anarcat's picture

I am a strong supporter of staying with Drupal 5 at this point. We are going to have some refactoring to do, mostly in the backend, for 0.2. A lot of instability have been introduced on HEAD because of that refactoring, and having Drush as a moving target doesn't help either. So adding into the mix a port to D6 would probably be explosive.

The way I see it, Hostmaster should be able to bootstrap, deploy and upgrade itself. The major new feature in 0.2 is to have multiple platform support, which includes upgrades, so I expect 0.2 to be able to upgrade itself. That's an excellent test of the API of the coherence of a system. I compare this to writing compilers: a good and complete compiler should be able to compile its own source code.

I am sensitive to the arguments brought in for the D6 upgrade, but I feel there's a counterpoint for every one presented so far:

  • 0.2 as a required upgrade path: i think this is necessary anyways. 0.1 platforms can't be migrated to 0.2 platforms, so people should be expected to stop by 0.2 to go ahead with the other milestones. Besides, I don't think 0.1 has such wide adoption yet, so that's not a big issue (although it could be if we delay 0.2 because, for example, of a d6 port :P).
  • kludges in Hosting and Hosting wizard: those kludges are there now and they work. Porting them to D6 is going to require extra work that should be better spent in improving the backend.
  • new features in hosting will need to be ported: of all (13 right now) the critical issues targeted for the 0.2 release, only two really relate to new features in hosting: package dependency checking and platform creation automation, both of which probably will not require much extra work to port to D6. So I don't think the potential instability is worth the risk there either.

I understand that work already has started on this, so maybe all those points are moot and we should just bite the bullet already, but at the very minimum, we should not maintain two branches. I would be very strongly against splitting the tree again, unless absolutely necessary, and I don't feel it is now. If we jump to D6, we jump forever.

I am a strong supporter of frequent releases and I think we should concentrate on getting 0.2 out as soon as possible now, without extra gizmos and bells and whistles. I feel that we have much more important rearchitecture changes to do than a d6 port, the server abstraction comes to mind immediatly. Those are the real roadmap blockers in my opinion, not the D6 port, unless we really expect the hosting code to grow more and more in directions that will add more and more time to the D6 port.

I agree with all of anarcat's points too

adrian's picture

But I felt it was important to enumerate the pros/cons of either approach for the developer community. ("teach the controversy" =P )

As far as being able to bootstrap / upgrade hostmaster, I am thinking that we might provide a command line 'hosting bootstrap' command,
that is an interactive shell script that manages the installation steps necessary to get a working hostmaster install. (ie: it will ask you for all the things necessary to create your initial hostmaster site, and generate a default virtual host using the template from : http://drupal.org/node/262005). We might even use it to download the latest drupal release and hostmaster profile itself.

In the same way, upgrading from 0.1 to 0.2 involves :

  1. Remove the crontab entry.
  2. Remove the old hostmaster profile. (this is why it is recommended that all aegir contrib modules are in the install profile, and not in sites/all)
  3. Replace it with the 0.2 profile (which has far fewer modules).
  4. Run the new provision update command.
  5. Add a new crontab entry.

That's something that is pretty scriptable.

We can also use the bootstrap to do the validation of the server. http://drupal.org/node/344967

This could serve to simplify the installation wizard too.

--
The future is so Bryght, I have to wear shades.

If it works to deploy all the way to D7 today ...

boris mann's picture

... then it sounds like polishing D5 is a better route.

BUT ... identify THE feature or requirement where moving to D6 means cleaner code. Basically, at that point, the D5 branch is locked, and new development occurs on D6. Then, when anyone asks "when is feature X going to be implemented?", you can point to the decision that moves to D6.

@Boris Mann It sounds like a

Macronomicus's picture

@Boris Mann
It sounds like a lot of what Adrian mentioned in the first post proves that D6 stands to be leaner/cleaner/meaner code.

@anarcat
I say we jump Louise

@adrian
I vote for D6 ...since I really want to use this and all I know/use is D6
Im not sure I want to install a d5 site to manage all d6 sites, just seems antithetical & might break my work-flow
Mainly because Im not familiar with the "kludges" & "hacks" you mentioned were necessary to replicate native D6 behavior, the thought of that makes my brain hurt.

You wouldn't happen to have a pre 6.x-0.2-dev laying about would ya?
I swear I'll help with bug reports or whatever I can!

^_^

Goal should be fastest route possible to clean code

Rainy Day's picture

Speaking as a wannabe Aegir end-user and not as an Aegir developer (but as someone with a software engineering background)…

It’s not that difficult to install D5 only to run the Aegir front-end, and if that strategy leads to fewer code branches to maintain, then it might result in faster development of the Aegir system with fewer errors overall.

On the other hand, i am inclined to hold off deploying Aegir until a stable D6 front-end solution is available. Not only is that easier and cleaner on my end, but it sounds like the Aegir code will be cleaner as well (with all those kludges and hacks vanquished). So i am hesitant to deploy a D5 Aegir front-end, even though it is able to support deployment of D5, D6 & D7 sites. I worry about those aforementioned kludges, and the pain i’ll go through when upgrading later. D6 has (finally) come of age, and it is definitely time to retire D5 ASAP.

If my vote matters, then it is cast in whatever direction leads to the quickest path to a stable, mature D6 front-end.

D6 and forward

jwoolson@drupal.org's picture

Like RainyDay, I'm a wannabe Aegir end-user and have been migrating all our sites to D6 for the last 4-5 months.

Jonathan Woolson
webmaster@fredonia.edu
www.fredonia.edu

Essentially what it comes down to ...

adrian's picture

Is that keeping the D5 front end for the 0.2 release allows us to develop the new features on a solid base, instead of a constantly shifting one.

The 'kludges and hacks' i referred to were essentially API's I had to invent to solve problems that were already solved in Drupal 6.

As with our 0.2 release, we will be focusing on standardizing on upstream API's, as ease of maintenance is of big importance to us. We are making excellent headway toward deeply integrating with Drush for this release, reducing the amount of custom code we use and reaching a point where all Drush commands will be able to be integrated into Aegir, and vice versa. (say hello to integration with drupal cvs and ala cart module installation, via the drush pm command set .. for instance)

Another good reason for sticking to D5 at the moment, is that we are very close to being self hosting. Which means that the 0.1 -> 0.2 upgrade will be the last one where you have to do actual manual intervention.

By keeping the 0.1 to 0.2 upgrade on the same version, we reduce the complexity of the problem (ie: you won't be upgrading drupal at the same time, just replacing the hostmaster profile, downloading the new drush and provision and running update.php).

After that point, you should be able to upgrade your Hostmaster in almost exactly the same way as any of your other sites (ie: drush.php provision migrate hostmastersite.com /new/release/of/drupal).

The amount of progress we are making is really astonishing, and when we talk about an 0.2 release, we are talking about a matter of weeks. If we had to roll in the 0.3 update that timeline could escalate into months.

So we're sticking to 'release early, release often'.

--
The future is so Bryght, I have to wear shades.

Always trust the cooks :-)

MichaelCole's picture

Hi, I'm glad you're writing this thread Adrian. It's nice to know the direction and have the conversation.

For me, I see Ubercart going 6.x "soon", and that will be the signal for me to start converting all my sites to 6.x and retire 5.x as (personally) relevant. Ubercart 6 may also have the recurring billing features Aegir is looking for.

I've making all my new sites in 6.x, and planned a Drupal 6.x site with Aegir 0.2 for signing up and offering services to my clients. So seeing 0.2 stay with 5.x is a little disheartening, but I totally trust you guys to know what's the best way forward.

I only wanted to mention that (to me specifically) a D5.x version of Aegir is less interesting. Of course, if I could have a D6.x "signup" site, that operates a hidden backend D5.x "hosting management" site, that provisions D6 client sites, then take as long as you want to convert hosting management from D5-6.

Or, if 0.3 has a scope of "Convert everything to D6.0" then that works great too. You guys know best =)

Thanks again,

Mike

You said it best

Macronomicus's picture

With all my reservations on downgrading to 5.x to get this to work .. I have to completely agree with your point ... they are cooking this beast up and their opinion is paramount because they truly know whats best. Besides how bad could it be? I guess I should try and see... lol. I guess I am off to install a 5.x drupal and take this baby for a spin!

PS. Ubercart has been 6.xbeta for a while now and is in working condition .. the folks working on it are doing a fantastic job!
Its amazing to see how quickly and professionally they are knocking it out, in fact all of drupaldom never ceases to amaze me.

Stay the course

Rainy Day's picture

Thanks for the clarification Adrian.

Then it sounds like keeping the D5 front end (for now) is the quickest path to a stable, mature D6 front-end.

Question: If one were to install D5 only to run the Aegir front end (with Aegir itself hosting only D6 sites), then it sounds like one wouldn’t need to be too worried about keeping the D5 system up-to-date (as most security fixes seem to address exploits by Drupal users). Since the front-end has a closed set of trusted users, and could be further isolated from the public behind a private DNS†, there would seem to be no opportunity for the bad-guys to gain a foothold. Would you concur?

Of course in that scenario, you would have to keep your D6 install up-to-date.

___________________________________ 
† A private DNS known only to your Apache server and your local /etc/hosts file.

It depends: I would always

anarcat's picture

It depends: I would always take care of a drupal install, wether it's local or not. The obvious advantage of splitting cleanly your d5 and d6 platforms like this is that you're going to have only a very few modules on the d5 platform and the brunt of your work will be maintaining the d6 platform. I consider the cost of maintaining a d5 platform + hostmaster profile + hosting module to be minimal (at least it's much less work than maintaining a fully-fledged d5 platform with views, cck, etc, etc). The good thing about hostmaster is that dependencies are minimal...

A.

Aegir hosting system

Group organizers

Group categories

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week