Drupal Release Cycle is challenging for large sites/clients

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Fintan's picture

For us (an end user who occasionaly develops for others) the concept of a six monthly update that does not have backward compatiblity is one of the biggest worries in using Drupal. While I understand the ideal and applaud it, it does bring up the issue of leverage in regard to the financial investment made.

We want to see a return on any investment we make so having to start an upgrade cycle every six months if going to be very hard to justify especially as the size of our sites mean that we are going to be constantly programming to stay still. Our most recent project has had over 400 man days in development and uses a huge number of contributed modules to update to V5 is going to be a major task.

The way I see it is:

V5 - released e.g. Nov 1
V5 - Reasonably stable - Dec 1
Most Key Contrib modules V5 ready Feb 1
3 Months testing and migrating = April 1
Next Version of Drupal due for release = April 1

We are thinking of skipping every 2nd version as a 1 yearly cycle may be achievable, but it is worrying that the release cycle is less that the time it takes to develop any decent size of site. Essentially the release cycle is inline with Fedora Style of release but fedora is not patently not suitable for large scale use and is a testing ground for new features that subsequently get put into enterprise ready releases.

I am guessing that a lot of people here will also have the same issue with any site that is medium to large in size, what approach's are people taking to this?

Comments

thats better

moshe weitzman's picture

nice topic :)

my .02 is that drupal's 6 month release cycle is perfectly reasonable. your proposal to upgrade once a year is also perfectly reasonable.

as an aside, organizations can make their upgrades go much much smoother by

  • use version control to keep track of your changes to core and contrib modules
  • change core and contrib modules as little as possible. that means understanding our APIs and using them as needed
  • contribute back your changes when they are generally useful. advantages are:
    • you will no longer have to maintain the fork. the community will take over
    • you can upgrade easier later becase there is no conflict. pick up improvements created by others
    • build up your karma within drupal project which helps when asking questions and requesting features and hiring developers, ...
  • stay in touch with drupal development. read [DrupalDigest](http://drupaldigest.com feeds). if drupal is central to your business, you ought to participate in, or at least be cognizant of, its upcoming improvements.

True

Fintan's picture

However Drupal.org needs to a decent structure in order for us to properly integrate what we are doing, hence I have agreed to sponsor Dereks work which will move things forward a long way but still not go far enough, but I hope this will come over time.

We have also discussed a module that I hope to get him to write once he is finished what he is on, basically we need a way of managing a large number of sites with a variety of contributed modules. So the module would simply reside on a given Drupal instance and submit back to a central source (lets call it a repository of module information) all the modules / version / patch's etc in use , the central repository then can compare these lists against drupal.org and see what needs to be done. But this needs his current work completed to really be useful so I will just have to wait a while.

Ideally I would like to take this to the next step and run it through the automated testing system. However despite all of this I still find it worrying to have such frequent upgrades that are not backward compatible even between minor revision numbers. All thats going to happen is we are going to automate the generation of a massive list of work that needs to be done especially if parts of core have been repudiated.

Submission of updates / modules is going to happen in the next few weeks but we are doing a lot of work on performance at the moment so until this is finished we going to hold off as it will undoubtedly require changes to be made.

I have already hired both Derek (sponsored his project) and Shakur (SOC) (working on one of our own projects) to do work for us, I have also agreed that we will provide at least one server (probably two) for testing purposes for the community, this will be sorted when Dries gets back settled back in next week. Karma is getting expensive ;-)

Version Control

blakehall's picture

Is there any documentation laying around about best practices for an individual's drupal version control setup...

I know of things like keeping contrib modules in sites/default/modules/ etc, but actually using something like CVS or SVN to keep track of major changes never really occurred to me (shows how much i know about version control right?).

I would think this documentation could be very useful (I just don't think I know enough to write it).

Thoughts?

Perhaps

ChrisKennedy's picture

http://drupal.org/node/5123 might be what you're looking for (and its parent page).

I respectfully disagree...

webchick's picture

The ability to break APIs (but not data) is absolutely critical in order for Drupal to remain competitive. If we were still trying to kludge around on v 1.0 of the node API, or with form_something functions instead of the Form API, we quite simply wouldn't be able to build the sites that we can today. The empowerment of Drupal developers to be able to think "outside" the box, envision those large-scale changes which will make the platform much more powerful, and actually put patches in core to get us there, are what causes Drupal to be such an awesome, flexible platform that we all know and love.

You will never, ever get a "by the clock" release cycle from Drupal. Drupal is at its heart an organic, volunteer-run project, so things only get done when there are people around who take an interest in and do them. And while there are people in the community paid to work on Drupal by various companies, they are being paid to satisfy their clients' itches, not yours or mine. The best (and really, only) way to ensure that a version of Drupal is stable and does what you need to do is to get involved with testing and development yourself. So I think it's awesome that you're taking this to heart and willing to put forward resources for automated testing of code, as this will certainly help improve the stability of the platform.

Every one of Moshe's points are very sound. Also, remember that just because there's a new version of Drupal does not mean that you need to upgrade to it! Only upgrade when it makes sense for you, either because the version you're using is no longer supported with security updates, or because you need a certain feature that's only present in the new version.

I understand your point however

Fintan's picture

The statement I made was not about whether or not it is right to break the api or not, I have spent years hacking interfaces into old cobol systems (ACMS as a SOA Broker is superb by the way) and the likes, I completely agree with breaking the API and think it is an imperative for future development that this is done. The issue is the frequency of this change, the larger the development the longer it takes to get a reasonable return on the investment. Breaking the API for minor releases does not recognise this issue.

If you have a website that has take 4 weeks to develop and takes a week to update to the latest version then its a no brainer to upgrade every six months, however if you have a site that takes 8 months to develop and takes 6 weeks to upgrade the commercials change. You need to make a return on the investment before you can consider upgrading. The point I am making is that a six month release cycle is obviously targetted at smaller sites as bigger sites could easily take six months to develop so you are out of date before you start.

As Drupal grows (and I truly belive it will) this issue will become a more serious issue, its fine for everyone to say that its the right technical thing to do but this ignores commercial realities and the real requirement for large commercial sites to get a financial return, by promoting this debate I just want to raise awareness that this is a genuine concern for large sites and should start to be considered when deciding on release cycles.

If sites dont upgrade then we are hit by issues of modules not being developed and the deveopment focus for core focused on the new version rather than patching the older version. All I am saying is as Drupal grows then it is important to realise that different dynamics exist when developing larger sites.

Until....

Anonymous's picture

Yea, thats all well and good until your are forced to upgrade due to security concearns or a recent vulnerability like the spate of XSS stuff from this year. Frequent upgrade required for large sites is a killer!

Tip of the hat, wag of the finger.

Anonymous's picture

I have to throw up some props to webchick because I know how active she is in the community.
The to agree with her, you could say the same thing about Firefox, Linux, or any Opensource project for that matter.
And yet the rate at which open source projects are being used is exponential.
You could say that a cycle like this isn't dependable, but the thing is how many times has microsoft delayed the vista launch date? How many game platforms or games for that matter have been pushed back?

Many big clients already use drupal (NASA, NATO, MTV, etc.)

Although I do wish that drupal had something more structured like this:
http://fedoraproject.org/wiki/Core/Schedule

It might help contributors push out a release earlier.

Some ways to lower the upgrade pain threshold in Drupal

zoon_unit's picture

If the upgrade process was less painful, then it would make more sense to do it. By focusing on a better upgrade path, we can have our cake and eat it too. Suggestions:

1) An automated update capability similar to Ubuntu, that searches for module and core upgrades and applies them automatically. Of course, this system should have a way to mark modules NOT for update and there should be a rollback capability. (ambitious, yes, but worthy) The ease at which CVS code can be updated via a simple Linux command proves the concept's worthiness.

2) Increased development of the panels, CCK, and views triad. By integrating these applications more closely into core and adding functionality, the need for many small modules goes away. If these apps have strong abilities to import and export metadata, then upgrading to a new version will consist of moving the metadata, rather than some custom module with all its inherent incompatibilities. Also, this approach means more security, as data flows through a couple of highly supported modules instead of through a hodgepodge of custom modules that may not be written as robustly.

3) The development of a forms designer to enhance views and CCK.

4) The development of "supermodules" similar to the enhanced voting approach webchick mentioned earlier. The key goal here is to eliminate module bloat. With less modules to upgrade, the upgrade process becomes easier.

I agree

Harry Slaughter's picture

I was just thinking about this very same thing.

As a developer, I find it exciting that Drupal moves so quickly. There are always tasty new features to enjoy.

But from the perspective of a site maintainer, this frequent shifting of the Drupal foundation could become very frustrating. Upgrades between major Drupal releases require a significant amount of planning and work.

The problem is even worse for those depending on 3rd party modules, which can take additional months to be ported to the current Drupal release.

Back to the other hand, there's no mandate that every site needs to be running the latest Drupal. Sticking with an older version should remain an option. The only thing "preventing" this is the fact that we only maintain patches for current and previous releases. I think if each version of Drupal had a 'guaranteed' lifespan, nobody would really have anything to complain about. Assuring security patches are available for a reasonable length of time (18-24 months?) for any release of Drupal would be super.

I just don't know whether this is practical and if there would be any interest in maintaining the older versions.

--
Drupal tips, tricks and services
http://devbee.com/ - Effective Drupal

--
Drupal tips, tricks and services
http://devbee.net/ - Effective Drupal

blather

moshe weitzman's picture

releases can be maintained indefinately if someone volunteers to do so. if you email dries with a credible request to maintain 4.5 or 4.6 he will accept.

..

greggles's picture

I agree with Moshe on this one...

And right here on groups.drupal - is the 4.6 maintenance coordination group: http://groups.drupal.org/drupal-4-6-maintainance Ber has said he is interested in this, he just needs some supporters.

Now is the time to get together if you are interested in it - 4.6 EOL is on the horizon.

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

I want to add a personal

keesje76's picture

I want to add a personal concern to this thread.

Though its 1,5 years old, I feel it's still relevant.

We are now halfway 2008, D6 is approx. 5 months out of beta, and some very important modules are still not released for D6, or are in beta stage.
Saying this, D7 is already on its way, with no choice than using D5 for production use...

It's not that easy, I know. I think there must be a better balance between release schedules and the drop of support for a Drupal release.

what you see now is that developers are anticipating on the delay of a release getting ready for production use. (no offence). Even today there are modules actively developed for D5, without being ported to D6. this is understandable, but contra productive on speeding up acceptance of a release.

Just my 0.02: Should a long term support release be a good idea? (indeed, just like Ubuntu LTS)
What do you think?

Regards,

Kees


Open Source Webdevelopment

Long term support was

catch's picture

Long term support was discussed on this thread in the development list recently: http://lists.drupal.org/pipermail/development/2008-April/029636.html

I personally think it could be a good idea. A longer release cycle is neither appealing to me, nor is it going to happen simply because every year or so someone starts a thread about it (although it's more like once a month). A longer support cycle for any given release, if there's resources to do it, doesn't interfere with core and contrib development but solves the issue for those who have it.

However, it'd require some of the people clamouring for longer release support cycles to 1. start talking seriously about what it means to support a core branch for an extra year - Drumm and others pointed to this in the development list discussion 2. find a way to gather and pool resources into making it actually happen.

Also - the original post on this thread talks about a 6 month release cycle - it's much closer to a year now, yet we're still having this discussion.

LTS seems like a good idea,

flickerfly's picture

LTS seems like a good idea, allow development to continue strongly and effectively, have regular testing periods and opportunity for growth. Meanwhile, LTS is standing strong, ready to leap forward when the next LTS is available. A site running LTS would typically want to have the same structure maintained for long periods of time without feature changes. As such, LTS would be about security and bug fixes in my mind and little else.

LTS from a commercial company

Boris Mann's picture

Scratch your own itch. I challenged people at Drupal Camp Vancouver to step up and gather funds to maintain old releases if they wanted to. But I don't currently see enough people being able to / wanting to maintain old releases in the community. If there is a big pain, then there may be a commercial opportunity for someone to take this on.

Oh, and the funny thing here is that Fintan, the author of this topic originally, has done a great post that pretty much reverses his position. See http://www.io1.biz/blogs/drupal-development-cycle-a-business-viewpoint

Heh. That's pretty funny,

catch's picture

Heh. That's pretty funny, and a great two links to put up the next time one of these threads starts.

However, qrios points out an

markus_petrux's picture

However, qrios points out an important issue...

1) if you start a new project that would take you a few months, if you choose D6, you may end your project before CCK, Views, Internationalition, which say is something you need...

2) if you choose D5, then you may get forced to upgrade to D6 sooner than what you would expect. A big problem if you had to develop a lot of functionallity yourself, for the given project (a few months work).

I mean, today is very difficult to choose which Dx choose for certain projects. There are non-core basic modules thay may need to impact on Drupal development cycle, perhaps.

The thing here is that both

catch's picture

The thing here is that both Views and CCK have undergone major, major rewrites in addition to a Drupal 5 - 6 port. If it was just a case of upgrading Views 1 and CCK 2 to D6 they'd both have been released months ago - and Views 2 and CCK 3(?) would be less far along than they are now, everyone would be waiting for them anyway, and merlin, KarenS, yched and others would be dealing with even more work on their plates than they already have maintaining two major branches.

In terms of impacting the core development cycle - the best bet is getting CCK and Views into core - or more likely the core of CCK and Views with the UI and some complexity remaining in contrib. Refactoring both modules for D6 is a big part of getting them ready for core in D7/D8 - alongside that, there's the DADS group, and Larry Garfield's new database layer.

Of course, this will require major core refactoring and API changes - which many people in these discussions want to slow down.

Upgrade burdens and LTS

tarvid's picture

The availability of security updates and bugfixes with low probability of blowing up existing Drupal installations would be a good thing. Then again getting my CVS act together would be a good thing too.

Generalizing a policy

HansBKK's picture

Here's a related post, feel free to comment there or here:

http://drupal.org/node/264495

It's obviously from my personal POV and coming from the current stage of the release cycle.

I'm considering creating a Getting Started handbook page What version of Drupal should I use? and would like to get some feedback on more generalized guidelines, something that will still apply in the future.


DrupALL vs core Drupal (nebulous I know but an important distinction)

The factors I've identified so far influencing the decision:

Experience level of the user-site-developer (drupal-specific vs more general LAMP vs most general webdev/sysadmin/programming/database).

Their comfort-level with risk - some people just like living on the cutting edge

Customer/employer issues - businesses with bosses and clients vs personal control

The clarity of the site's use-case and features wish-list. If well-defined, identifying the modules to be used (now there's a can of worms ) and letting that drive the choice. If not, IMO greater need for DrupALL.

Feedback please!

Agree 100%, however...

markus_petrux's picture

...when modules such as cck and views take months to reach stable stage, that's not just a matter of cutting edge liking, there will be a lot of time spent along the way, to keep in sync your work based on such critical modules...

...it depends on how much you invest on a given site, or how much are you able to. But, when you can't elaborate because something basic does not depend on what you're able to do (I mean, even if you contribute with bug reports, patches or whatever, that may not be enough for the work that is need to get certain modules out), then you may opt for a different platform. It's a reak risk, I think.