The Elephant in the Drupal Release Cycle: the users

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
damien_vancouver's picture

[ EDIT, Feb 24th: This thread is now retired. The discussion has continued with a proper proposal at: http://groups.drupal.org/node/212648. Please go there and review that proposal instead. ]

Hello everyone,

This was originally a very off-topic and long-winded post on http://drupal.org/node/1366940. That version was edited for brevity and for lack of a better place, I'm posting the longer op-ed piece here. Following discussions there regarding Drupal release cycles, and the idea of bugfix / stability releases every second release, I wanted to add these points about the silent majority: The users out there, and their needs.

I'm arriving late for the party here, but I'd like to add my $0.25. I'd like to remind everyone of the giant elephant in the room that hasn't yet been addressed: The 99.9% of the Drupal community that is made up of site builders, and end users (those people who actually pay for all of us to make a living by investing in sites that are built using Drupal coreand mostly a whole bunch of contrib modules).

             ..--""-.                  .-""--..
           .-'         \ ...----... /         '-.
         .'      .:::...,'              ',...:::.      '.
        (     .''''::;                  ;::''''.     )
         \             '-)              (-'             /
          \             /                \             /
           \          .'.-.            .-.'.          /            
            \         | \0|            |0/ |         /          {                                            }
            |          \  |   .-==-.   |  /          |          {   This was supposed to save me money!!??   }
             \          /;          ;\          /           {                                            }
              '.._      (_ |  .-==-.  | )      _..'           O
                  ""-//'        '` `-""              o
                       / /;   .==.   ;\ \         o o o o o
                 .---./
/   \  .==.  /   \ \
                / '.    -.__)       |"                  
               | =(-.        '==.   ;                    
         jgs    \  '.-.           /
                 _:_)   `"--.....-'

Some of those organizations are large sites, with appropriate development and maintenance budgets, who appreciate the true cost of their investment in open source, and who stoically accept the need to maintain, fix and promote. Then there are the other 90%+ of them, who are small organizations, drawn to Drupal because it is Free, or because they agree politically with Open Source, or because they are enlightened enough to understand it provides a lower TCO, or because they like the idea of working with software that is also a social movement. These people are the salt of the Drupal earth, they are our bread and butter as Drupal professionals, and they should be cherished and protected. Yet no one speaks out for them or discusses THEIR needs?? They are the giant elephant in the room here. They are definitely being ignored in this decision making process, other than to make plans on cool features that might entice them in future releases (a.k.a. "build it and they will come", circa dotcom 1.0).

The fact of the matter is, that the "word on the street" about Drupal is not as rosy as we might like it is. When I built my first Drupal 7 site (which I started last May), it was like pulling teeth to get my client to agree with Drupal 7. That is because the word on the street was that "drupal 7 wasn't ready" and "nothing works yet". What is all that referring to? It's actually nothing to do with core.. it's referring to the One Thing That Matters for the end users - the availability of stable contrib modules.

D7CX, while a nice idea, actually underscored the extent of the problem and provided fuel for the Microsoft marketing engine to flog sharepoint over Drupal. And this is our real challenge. The end users WILL abandon us if we continue to ignore their needs. They will flock to another CMS that positions itself as more stable, or most likely they will get sucked in by fear-mongering by Microsoft and the like and end up running sharepoint, because "at least then they can get support". (All these arguments I'm listing in quotes are actually quotes from customers by the way).

So back to my grass roots salt of the earth Drupal users and what in my mind is the real crux of the problem:

Let's say they were a long time Drupal fan, and they maybe got started with Drupal 5 for its excellent community features. They spent $20,000 (let's say) on their first Drupal site. A year later Drupal 5 is EOL and they are being encouraged to move to 6. Well, being a small organization, it takes them at least a year (and a lot of political wrangling) to find the $10,000 they now need to upgrade their $20,000 site to Drupal 6. Their upgrade finally happens a year and a half after when they SHOULD have upgraded.

Now fast forward a year to today. We have a giant, giant, GIANT user community of Drupal 6 sites now! What a great release that was! What amazing things got accomplished with it towards the end, once contrib stabilized!

But guess what.. here we are a year later and someone has to go down and inform that client they better start searching for budget again, because Drupal 6 is going to be EOL by the time they've found it..... at which time they can pay a bunch of money to port their site because a lot of contrib still won't be ready. Oh and by the way, it's more complicated this release, so you better find $15,000 this time, Drupal 7 is much more complicated and things are missing you used to use in Drupal 6 contrib, so $10,000 probably won't cut it.

It's common sense that this cycle is not sustainable. Isn't it? How long until they realize their $20,000 site has actually cost them $100,000 and then Sharepoint etc. starts looking like a cheap escape route for them? Microsoft is already gleefully telling them this, while buying them free lunch. The other alternative of course is people just can't upgrade, they can't afford it. Those of us who built their sites usually can't afford to just do it for them for free, and that's how we get a bunch of out of date sites that can be hacked. (also not good publicity for the project).

Do I have a simple solution to all of this? No, not really! I appreciate all the arguments above. I just wish that we (the decision makers) would consider the end users, and place their needs where they should be: ahead of ours as core developers and contrib maintainers. In fact their needs should come first . It is honestly ALL THAT MATTERS for the success of our project, and as a metric of our success, IMO. Features are cool and all, but the 99.9% are not core developers, they are average people, often going out on a limb against the advice of their superiors or dissenters within their organization, to adopt Drupal and embark on their personal journey of open source. New features in core do not help these people. Being taken care of for upgrade paths and stability should come first.

Here are a couple of possible solutions / changes I do have, requested on the 99.9%'s behalf:

  • Create a public unstable/testing release phase in the release cycle. "Release" new versions to this state, and encourage contrib developers to start porting at that release date. Then Revamp the CX pledge system into a pledge to get your module stable by the time the release becomes the stable release.

    In my mind the biggest issue for users is lack of stable contrib... So why would we not have Drupal 8 in "unstable" for the first year, with contrib appearing for it and people working out the kinks, but with the official "stable" version remaining at Drupal 7 so that's what new users get? I argue that anyone with a decent consultant or shop advising them is getting that anyway... the consultant/shop makes them hold off until contib is ready. I propose we add an extra (year?) long stage in the release cycle, where a new "testing" (unstable) release is available, but not the default. In a way we're not changing anything but calling the new release "unstable" for a year. Core developers go on doing their thing as now. But for that extra year we have the new release in "unstable" with site builders and contrib authors kicking the tires. We relax the patch rules for unstable - API breakage and stuff like that is OK. This will result in more actual usage of the new release and more stuff getting fixed early, so we don't end up where we are with Drupal 7 core bugs.. a ton of them and it's a giant pain in the butt to have to patch it against D8 and write a test there and then backport it all to D7.

    Had we had a year long drupal 7 public testing cycle with early adopters like me using it and fixing stuff, we could have saved a lot of that grief. We could even still have the "core" core people working on Drupal 9 in the meantime, which is honestly the only realistic way forward, we can't honestly expect core people to all stop and refocus on bugs for that year. But this approach will allow others in the community to step into that role. So with this approach, while contribby/testing people work on the drupal 8 testing release, and end users arriving at drupal.org for the first time still download the "stable" drupal 7. We could the adjust our marketing position to be Drupal: all about stability. Which in my mind would get us a lot more long term users than new features in core. Which frankly, end users don't care about, they just want to hear their upgrade path from the current version to the next is going to be smooth and not hugely expensive.

  • Longer support for old releases and more warning about EOL. Users need appropriate amounts of time to fetch budgets and plan the real cost of upgrades. We need to support D6 way beyond the release of D8 at the rate we're going, or we are going to lose a ton of users who decide it's less expensive/painful to switch to another CMS. The first suggestion on my list may allevaite this somewhat. Release cycles and dates should be driven by: stability of old release, and completeness of features. I refer to the Debian Linux project for an example of this done right. There's a simple reason it's the best IMO and it's not that their developers or project members are smarter than everyone else. As for stabillity of release, that should not be up to the people doing the patching to determine.

    It should also not be up to officers of Aquia, Inc. (no offense guys and gals, but.....) Their LEGAL responsbility is to their shareholders' bottom line. That is a HUGE conflict of interest IMO and puts the community and the people in question in a continual conflict of interest, which I'm sure they could all already speak to as not being a pleasant place to be.

    I realize that what I'm proposing could really take a lot of the fun out of it for some of the core people. This is a shame, but it's also an inevitable side effect of our success. We should celebrate, rather than ignore, our responsibilities to the end users. Remember how the mighty Linux got its start... Linus wanting a nice fresh new project to hack on without the responsibility of a user base. So what I'm proposing is not in the core developers' best interest at all. This is precisely why the project has grown too large for those people to be the decision makers on decisions that affect the rest of the community. Or if they are going to do so (in a BFDL kind of way, then it should be as Mr. Spock called it: "The needs of the many outweigh the needs of the few, or the one.").

To everyone involved with Drupal so far: thank you all for your outstanding contributions to the project; from writing code, to fixing bugs, to the much more important and unpleasant mundanities like figuring this kind of thing out. In doing so you are, and will probably remain, the unsung heroes. And as a nice side benefit, we can all keep making a living with Drupal and won't have to learn sharepoint or joomla. :)

Comments

Thanks :)

xjm's picture

Thanks for raising these points here. I agree that the upgrade path and the stability and availability of contrib are serious issues for a majority of sites--I myself am only now beginning to port my sites to D7, despite that I've personally worked on D8 already, for these two reasons.

On the other hand, I feel that there are a few misconceptions in your post... the "us/them", "core developers/users" dichotomy. I find this troubling. Core developers are members of the community. They are end users and site builders. They are contrib maintainers.

This paragraph in particular puts me on the defensive:

Longer support for old releases and more warning about EOL. Users need appropriate amounts of time to fetch budgets and plan the real cost of upgrades. We need to support D6 way beyond the release of D8 at the rate we're going, or we are going to lose a ton of users who decide it's less expensive/painful to switch to another CMS. [...] I realize that what I'm proposing could really take a lot of the fun out of it for some of the core people.

My concerns

  1. As I mentioned above, I am concerned about the notion of "the core people" as if it's an exclusive club or something. It's not, or they never would have let me in. ;)
  2. The idea of supporting D6 longer sounds great on paper and all, but...
    • By the time D8 is released mid-2013, D6 will be five and a half years old. That's longer than Ubuntu LTS. That's longer than your Apple Care Extended Warranty. That's a longer life than almost anything I can think of in IT, aside from perhaps the zombie that is IE6.
    • What incentive is there for maintainers to support D6 when their sites are not on D6? As a contrib maintainer, I already hate having to work on the D6 branches of my modules, just because the APIs in D7 are so much better. And I don't even have a production D7 site! How much worse will it be when I personally am not using D6?
  3. Finally--the reason I think many people reacted negatively to this offline earlier--your post seems to tell people what they can and can't do with the time they are donating to the project. I know it's probably not intended, but this is a huge button for most contributors.

In closing, regarding:

As for stabillity of release, that should not be up to the people doing the patching to determine.
It should also not be up to officers of Aquia [...] This is precisely why the project has grown too large for those people to be the decision makers on decisions that affect the rest of the community.

The decision makers in Drupal are people who, well, do stuff. It's not Acquia. It's not the DA. It's not some cabal of "core people." It's us. :)

It's worth saying...

arianek's picture

Fwiw, I think it's pretty difficult, if not naive, not to question who's making the decisions about what the direction of core development is these days. As much as some would like to believe that there hasn't been a huge power shift in the community, there certainly has. Exemplified by http://groups.drupal.org/node/210973#comment-694768 ie. even if it's not Dries who holds all the cards, "upstairs" clearly denotes a group of top level staff at a specific extremely large company.

Maybe the answer is that the rest of the community still has enough power to direct things as they wish (or go against those plans), but it's not wrong to question this or discuss the impacts of it. Seeing that those doing the high level planning and decision making for the project are certainly not at a company that would have trouble keeping up with things like major version upgrades, it's worth talking about how these issues impact companies/sites at various levels and sizes in the community.

Er. That's not what I meant

webchick's picture

Er. That's not what I meant by that comment. :) I meant that Dries had already signed off on the idea, and was involved in its development. That usually changes the nature of the discussion, because it means "if we figure out something here, then it'll happen" as opposed to "webchick is off her meds again." :)

Okay, but that's exactly how

arianek's picture

Okay, but that's exactly how it sounds to people outside that inner circle. I had the sense that conversations and brainstorming like this used to happen more online and more in the open where all the avid core contributors could participate (rather than a proposal being formulated at an internal meeting and then afterwards being discussed online/openly). Or am I missing something?

Ok, well I'll watch my words

webchick's picture

Ok, well I'll watch my words more closely in the future then. But in case it's not clear, that thread is brainstorming out in the open/online. It's not some kind of mandate, it's "Hey, we thought up this idea to solve this problem. What do you think?" The discussion there has been great, because it validated some assumptions, but also called into question some other ones, and we're working together to figure out a policy that best will meet everyone's needs.

And we as a community actually propose policy changes based on internal discussions all the time. Everything from the DA elections to issue thresholds has had the seed of its idea grown in some sort of "off the grid" communication, whether that be IRC discussions, e-mail chains, Google docs, code sprints, face-to-face meetings, etc. It's a common pattern to come up with an idea, bounce it off of 3-4 people first, and then put it out to the community when you have some basic level of validation that you're not smoking crack. ;)

This particular topic

arianek's picture

This particular topic http://groups.drupal.org/node/210973 seems like something that it would have been great to discuss in advance with the various teams that get impacted so heavily by the freeze deadlines even before tabling a proposal.

D7 freezes were incredibly frustrating - some teams (like Docs) were made to stick completely to the deadlines and complete work super early, while others who weren't under formal exceptions continued making changes for nearly an extra year! And I don't feel like that was ever addressed seriously, and the new proposal seems really fuzzy to me as far as keeping track of deadlines will go. (I'll post that back on the other thread where it belongs.)

My point was, though, that lots of off-grid discussions do still present some kind of opportunity for getting involved... I'm not saying this particular situation was necessarily ill-intended or anything, but it's really frustrating when it sounds like a small group are having internal company meetings about such overarching issues, especially when language like that is used. It's hard enough contending with the power shift that's going on with out it sounding so overtly that way.

Oh, one thing I forgot to mention

xjm's picture

I am hoping that some Drupal shops consider taking on paid extended support for Drupal 6. As you mention, D6 has a huge user base (although D7 already has a larger one, as of this week). So, I think consulting services to support D6 core and contrib after the official EOL would meet a community need and also be a good business opportunity.

Very good points

damien_vancouver's picture

@xjm, thanks for taking the time to make these insightful comments. No, I most certainly did not intend to start an us vs. them thing, or tell anyone what to do with their donated time... As usual, tact was not my strong point and I have angered the very people I need to convince. -1 out of 10 for execution.

Anyway....

The big downside to my plan definitely is, as you say, the extended lifespan for Drupal 6. An extension of the life cycle of the old release will involve maintainers consciously setting old versions to bugfixes/security fixes only, and support dropping off for that old contrib stuff. But we'd at least keep the infrastructure to support when critical fixes came out. Would everyone keep supporting their modules all the way to official EOL? Probalby not! But they would have a means to provide security updates and so on when those were necessary and thereby get them out to large numbers of users?

Regarding decision making and release dates:

The decision makers in Drupal are people who, well, do stuff. It's not Acquia. It's not the DA. It's not some cabal of "core people." It's us. :)

Well as far as I can tell, it's always Dries isn't it? New release cycle stuff is announced on his blog, including the determination that we've reached the plateau of productivity. I would argue the plateau of productivity should be evaluated by considering the completeness of core+contrib as a whole, not just the number of D7 installs - that statistic doesn't count finished D7 sites at their peak of productivity. I would further argue that one person should not arbitrarily decide, and further that it's not possible for any one person to have the necessary perspective to do so. I would be absolutely thrilled to find out we'd moved the decision of when Drupal was stable enough to release, and when we'd reached that peak of productivity, to the Drupal Association or some group that is representative of all users in the Drupal ecosystem.

I appreciate that Drupal is a do-ocracy and that the people who do the work get to decide. That seems fair to those doing the work, but that's kind of my point is that they may not see the forest for the trees, and realize their decisions are not in the best interests of the community at large. Should they receive special treatment anyway, as they are after all the ones volunteering to do the work? Aboslutely 100% yes. I'm all for that. But it seems to me these decisions get made based on core, by people who are working on core, in order to get new features in the next version of core out to the users. This would of course make perfect sense to those working on core as the best way forward to get new features out and fix problems sooner. Yet nowhere do I see discussion of the state of contrib, and how difficult it is to get to D7 still. Those are critical factors here. But they are contrib, not core, and these two things are treated as completely separate silos. It's worth noting that if you completely ignore Contrib, then our core release cycle and schedule (including our beta testing period) is completely 100% sane and works great.

After the amazing success of D6, we have 600,000+ Drupal sites out there. That is a whole lot of end users! I believe for most of these people what they want out of it most of all is stability and good value for their investment (be it time, money, whatever). I don't think the vast majority of them would agree at all that we've reached a plateau of productivity for Drupal 7.

Regarding my proposal for a "testing" branch or to just extend the beta cycle a lot: I believe that we are beta testing only a small part by just testing a new core without contrib. We should be beta-testing the real use-case of Drupal, which is IMO: site builders building sites using contrib.

Simply installing the beta of the next core, turning on some of the included modules and verifying that everything works is not telling us much we don't already know thanks to SimpleTest. I know as I tried it with a couple betas and RC's of Drupal 7. Everything looked fine. But lots of problems arose when I tried to actually create a real site with it. Contrb is only now reaching the point where an Average Jane can come along and set things up and have enough working contrib (and docs) available to easily build great sites. To coin a new term, I think where we are at now is that point - and it is a "plateau of accessibility". Maybe a good metric for that is the one that you mention, where D7 site installs are now greater than the D6 site installs. As for that elusive plateau of peak productivity... I think it actually comes later still, but we have to draw a line somewhere or we will have 10 year release cycles, old stuff going unpatched, more problems of that nature. We don't want Drupal 6 turning into our Windows XP, with people refusing/unwilling to upgrade a decade after it went EOL. (though I suspect it already is, as we'll see when we try and EOL it).

There's a great, well researched article about the state of D7 contrib here: http://drupal.cocomore.com/blog/state-drupal-7-contrib, where the author actually crunches all the numbers, decides that yes, it looks like time to move to D7, and then when he attempts it, guess what.... He ends up sticking with D6 because even still, it's not yet worth it to make the upgrade for a site that uses a lot of common contrib.

Finally, I really really like your idea of paid extended support for Drupal 6, but I wonder what form that could take. We'd need an official way for them to share the work for backporting things like security fixes. This implies an extended lifecycle of some sort. Maybe we need another state before EOL, "commercially support only" or something. But in that case, we need the infrastructure to remain open so that things that come out of it like security fixes can be shared to all.

LTS

LTS

damien_vancouver's picture

The Drupal-LTS group is currently our most attractive option for supporting D6 after the next Drupal major release. When D5 went EOL there was some good activity and discussion there and there is a really nice module (http://drupal.org/project/lts) that came out of it, which will be easily adapted to D6.

The module is a little light on security patches for D5 (only one) and its usage report is all zeroes. But... it's built, and the infrastructure and plan seem sound, and hopefully with the much larger install base of D6 sites, there will be enough drive for it to come together.

I remain optimistic for its future success and it should be the rally point for anyone who is still stuck at D6 (for whatever reason) when 8 is released.

Regarding this:

xjm's picture

Regarding this:

Well as far as I can tell, it's always Dries isn't it? New release cycle stuff is announced on his blog, including the determination that we've reached the plateau of productivity. I would argue the plateau of productivity should be evaluated by considering the completeness of core+contrib as a whole, not just the number of D7 installs - that statistic doesn't count finished D7 sites at their peak of productivity. I would further argue that one person should not arbitrarily decide...

I think there's a misconception here. Dries did not unilaterally make the decision, nor did he do so arbitrarily. As far as I understand, the announcement followed several conference calls between core maintainers and other key community members. Prior to that, there was an open town hall-type thing where all community members were invited to chime in about the D8 development cycle and process. Furthermore, the big picture plan for D8 has been part of an open discussion for over two years. So, when our BDFL announces a release date--which is appropriate in his role as a project lead--it's not coming out of thin air, sudden inspiration, or his role as Acquia CTO. It's coming out of a long community dialogue.

That said, I do think the process leading up to the release announcement could use more publicity and transparency, so maybe that's something we as a community can work to improve in the future.

damien_vancouver's picture

@xjm clarified,

I think there's a misconception here. Dries did not unilaterally make the decision, nor did he do so arbitrarily. As far as I understand, the announcement followed several conference calls between core maintainers and other key community members. Prior to that, there was an open town hall-type thing where all community members were invited to chime in about the D8 development cycle and process. Furthermore, the big picture plan for D8 has been part of an open discussion for over two years. So, when our BDFL announces a release date--which is appropriate in his role as a project lead--it's not coming out of thin air, sudden inspiration, or his role as Acquia CTO. It's coming out of a long community dialogue.

I guess my problem is that from the perspective of those who are not core contributors, or the "other key community members" consulted, it seemed very arbitrary and like no one asked us. In a bizarre Orwellian twist, it's not even possible to relevantly discuss these issues, as they are out of scope to the discussion by definition. I read Dries's D8 Development Cycle and Process announcement again, and you could pretty much add the phrase "Considering only Drupal core," to the start of every paragraph in there. In fact it explicitly says that in several spots. There's even talk of distributions but yet not one mention of the word contrib, or of the issue of contrib lagging behind so that end users are a year+ late adopting the new release and how it would be hard to make a meaningful distribution at release times without things like views and field tokens (to pluck some d7 examples).

For those who are not core contributors, this releasing-before-contrib is THE big issue with Drupal. It's the source of much pain and frustration. A new user who hears a new version of Drupal is out but knows nothing about it rushes to the site and downloads Drupal x.0 and tries to install it. Then they try and extend it, but can't. Are they understanding of why this is, which is surely not the fault of anyone working on core? Of course not. Instead they feel frustrated, angry, betrayed even by the very momentum that brought them there. Their conclusion: "Drupal sucks", or "doesn't live up to the hype", or "isn't ready", or has "crap documentation", etc etc etc.

Should the same user come along a year later instead, and download the latest version and use it, it all goes well for them. Rich contrib, lots of examples out there, lots of accurate documentation and recipes... they have an awesome time.

IMO the second experience should be the only one we want to happen. There is even commercial benefit to this, which is, a lot less people saying it sucks because they merely chose the wrong version == less uninformed dissing of Drupal == more celebration of the awesome of drupal == more work for all of us and less competition from closed source, other CMS's etc.

My modest proposal for dealing with this is we copy how Debian did it by introducing a "testing" phase to our release cycle, starting with D8 being released as "testing and D9 becoming "unstable". For anyone not sure about what this means, have a quick read of the introduction here: http://wiki.debian.org/DebianReleases.

It seems the biggest pushback by far to my proposal we adopt a Debian style testing phase starting with D8 was that it would drag out Drupal 6 even longer. After considering it, there's a smple solution: EOL Drupal 6 on the original schedule, when D8 makes it into the "testing" release. That is what Debian already does (they support the oldstable for "about a year" after the releases all move). Please don't write off this suggestion based on that.

My primary concern here is not extra EOL time for Drupal 6, it's more that when Drupal 8 is "released" to the general public, that there is contrib and documentation to back it up, which can ONLY happen with a year of testing. We do it anyway, we just did it for Drupal 7. We just don't officially acknowledge it, because only core is considered when planning. A side effect of this is longer release cycles yes, but I believe when you consider contrib as well, the release cycles are just that much too short anyway.

So, if this were to happen, when D8 is "released" mid next year:
D6 - goes oficially EOL, commercial support only, use LTS group which hopefully works out better than last time if we support it consciously as a project. Note I think the LTS group needs way way WAY more support and endoresement from the project come D6 EOL, or we will be sorry.
D7 - remains the Stable recommended download for people on d.org, with its rich contrib and documentation.
D8 - becomes "testing", which is not our beta period, but is actually post what used to be our 1.0 release, and carries on as testing until Drupal 8 + Contrib is at a stable state. Let's say that means the top 100 modules all have stable or RC's, as an example of what that criteria could be. Or as Debian does it, we make Drupal 8 stable once Drupal 9 reaches testing.

This is no extra work or compromise for anybody. Or if it is, please point out where, as I can't see it. D9 will be a year or two further out for "members of the public". But they can't even use it till then anyway, letting them play with it early is just going to cut them all up on sharp edges.

At the very least, even if these suggestions do not find traction, "other key community members" should include a representative group of the end-users, a term I use to include "suite builders" and those of less technical skill/experience with Drupal.... If from nowhere else, a committee of volunteer or elected D.A. members might be a good place to look for this perspective. We need to involve the people who can't just fix a problem themselves with a patch when they get stuck. Many of them do not / cannot understand issue queues or patches or our process at all, nor should we expect them to. But, these people outnumber the rest of us by several orders of magnitude, and are footing the bill for everything, so let's please consider their needs, and stop looking at Core like it's a standalone commercial product on a shelf. It's not, it's a (magnificent, athletic) naked body, and contrib modules are its clothes. It's hard to go out in public and get work done when you are naked. Even if you try, no one is going to take you seriously and many people will get outright offended. People will say, "The emperor wears no clothes", and yes, they would be right.

Not bad

Michelle's picture

I haven't been keen on some of the other stuff said with this but this plan is actually not too bad. The only real difference between it and how it's done now is the label we put on 8.0. As long as 9.x doesn't need to wait for 8.x contrib before it goes into code thaw, holding off a bit on waving the banners and blowing the horns on D8 until contrib can catch its breath a bit isn't a bad idea.

Michelle

I agree. This makes so much

arianek's picture

I agree. This makes so much sense when you look at it from outside our bubble (ie. most involved part of the contributor community).

x-posting from http://drupal.org/node/1410544#comment-5640366 (sorry, don't know where I should be discussing this!)

I was skimming this Debian stuff that damien_vancouver was posting about http://wiki.debian.org/DebianReleases#Introduction and it sounds like (despite being a fairly big change in how we think about our major versions), it could make a lot of sense. (Just skimming the intro will give you a quick idea of the system.)

[Like he says,] when D8 is released and D6 becomes unsupported, D7 remains the "stable" release which most end users use in production, D8 becomes "testing" which more advanced companies and those who are contributing work with, and D9 becomes "unstable" most recent version.

And everything trickles down in backports as usual. It seems like a much more accurate way of naming/thinking of things to what we actually do, how the contrib lag affects the broader community, etc.

also, barely anything needs to change

damien_vancouver's picture

I'm really glad to hear others like the idea. I was thinking about it in concrete "what would have to happen steps" and to my surprise, there really aren't many:

Naming it Testing

Instead of "Drupal 8.0 has been released!" we just say "Drupal 8.0 has been released for testing!". We link "testing" or make a page for information about what testing is (basically, a new name for the latest .0 release, an opportunity to catch contrib and docs up, a usable stable release for early adopters, etc etc).

Change http://drupal.org/start to show explain the new testing release

Below the Drupal 7 paragraph that's there now, we add a little section on Testing. Something like:

"Drupal 8 has been Released for Testing. The testing version is recommended for Drupal developers and more advanced users. Some programming may be required and things may change a little due to major bugfixes. Participating on Issue Queues may be required to reach success, but you will be using the very latest version of Drupal core with the newest features!" (but less cheesy)

Relax backport guidelines for Testing

As under discussion in http://groups.drupal.org/node/210973. These changes are the main technical difference between "testing" and "stable". Once the release becomes stable, we go for full freeze as we have now, to emphasize stability over everything else.

Change recommended version on the Drupal Project page

Set the recommended version to match stable. Actually it may make much more sense to call "testing" "available" and stable "recommended" so that everything matches our existing language. Although I like how "testing" implies a call to action to help test.

Celebrate and start playing with our new testing release!

Because we're probably all going to run testing, those who are reading this! A "second wave" of work starts, by the testing users, all of which fixes Drupal along the way. No one is sceaming murder about how things don't work, because.. they should be there to test, or at the very least, they were warned.

... and that's it, isn't it? No git changes, no version number changes, nada! Business as usual for everybody.. except that second wave of people. Maybe the second wave will become an entire new "generation" of drupal contributors - People who aren't quite ready to bust out modules or help with core issues, but who are perfectly capable of using issue queues and helping with general QA, and might even be comfortable enough to start produce patches if the environment was right. A big part of them feeling comfortable is them not feeling totally over their heads, like can happen now.

Can anyone think of other stuff we'd have to do?

So this really sounds less

catch's picture

So this really sounds less about the release cycle, and more about how the new release is promoted on Drupal.org (and elsewhere). From that point of view I think it's worth discussing. But I think it'd be better to put the specific proposal here into a new discussion. i.e. one that doesn't start by slag off people spending thousands of (usually) volunteer hours on core for not thinking about the millions of people using (and often making money off) their efforts who don't bother to help out.

agreed, where is the best spot?

damien_vancouver's picture

@catch, Agreed.

Could you or someone please point me at the appropriate place for the proposal? A new groups thread in a group, or perhaps the webmaster's issue queue? I will summarize that last version of the plan and its rationale there, in an appropriate way.

I think this might be tenable

xjm's picture

I think this might be tenable (and thank you for writing a specific proposal; that is much easier to respond to than these other really long posts of yours that I don't have time to read). ;) Few thoughts:

  • My big worry is that this would slow adoption and the porting of contrib modules even further. What incentive is there to get contrib modules ported (and fund getting contrib modules ported) during the "testing" phase that there isn't already during Beta?
  • The real root problem is also one of Drupal's greatest strengths: Anyone can contribute a module. People who contribute modules do so for free. Core is subject to intense peer-review and community governance, but contrib is YMMV and do what you're willing.
  • "Core developers," as a group, are not responsible for contributed modules, as a group. Individual developers, some of whom contribute to core and some of whom don't, are responsible for their individual modules.

No one is sceaming murder about how things don't work

This is probably why contributors have trouble addressing this question with cool heads. :) I'd tell those people: if something doesn't work, instead of complaining, step up and fix it!

Anyway. What I think we need is something like #d7cx, only much broader in scope and more broadly promoted. (I personally never once so much as browsed the codebase of D7 until after it was released, because I knew my department wouldn't be migrating within the year, and the only reason any of my contrib modules got ported was that people either sponsored me porting them or ported them themselves.) We need anything and everything we can do to get businesses to support and sponsor contrib developers.

My big worry is that this

damien_vancouver's picture

My big worry is that this would slow adoption and the porting of contrib modules even further. What incentive is there to get contrib modules ported (and fund getting contrib modules ported) during the "testing" phase that there isn't already during Beta?

I worried about this too, as a major flaw in this plan. We definitely need to weigh that possibility against the benefits.

I hope that most of them will merely carry on as before, at the same schedule. The main difference to them being: there's no lineup of angry/unreasonable/impolite end users demanding a new version of their module on release day. Instead these people are expected to help test and make the pledge of a stable 1.0 happen during the testing cycle, before the release hits stable. This is just my speculation though... it will have to be an identified risk we deem worth taking for the other benefits we gain.

As for a revamped #D7CX, I think this new system would help make that possible in a much more realistic way. The pledge could be something for both module developers AND users. ie. it's not a pledge to get it done by release day, it's a pledge to work with the user base to create a really fantastic and stable release by the time that major branch is marked "stable".

What I didn't like about the pledges for D7 was: the unreasonable stress they introduced to contrib developers. It turned out to be a lot harder than planned to get a stable release on release day. Even after those early releases, many of the bugs affecting contrib module stability don't show up until peolple start using it and combine with other modules. A great deal of contrib fixing went on just improving compatibility with other contrib, which is very important and necessary work to get stable, but isn't possible during core beta because it's just too early.

What I really liked about the pledges is how they visibly reveal that there is actual real work required to the end user. As well as involving the users in testing, the pledge system could be a good hook onto which we can hang some kind of corporate sponsorship dollars. That's a seperate discussion entirely but worth mentioning here because I think it could happen much easier with a formal, public testing phase.

A very real problem

fuzzy76's picture

This is a very real problem, and I think the current policy of only support the current major version and only one older is a bad decision. With the new policies discussed at http://groups.drupal.org/node/210973 there should be another set of rules for security patches only that would span a version or two further back. Ofcourse, it would be a difficult job to convince contrib module developers to do the same.

Sorry, but...

xjm's picture

Regarding:

This is a very real problem, and I think the current policy of only support the current major version and only one older is a bad decision.

Again, we are talking about over five years for the D6 life cycle. I'm not sure how anyone could expect volunteers to continue supporting software that long, much less longer. Paid long-term support really seems like a more reasonable solution to me than expecting volunteer developers to support code they wrote 5-7 years ago.

Edit: So, if you are concerned with this issue, I suggest getting involved with the group catch links above. :)

Contrib lag

Michelle's picture

Waiting for contrib to catch up could take forever. Everyone has a different idea of what is a "must have" contrib module. The really popular ones usually don't lag that much, anyway, unless there's something else going on like a Views rewrite or CCK getting half sucked into core.

Honestly, if every business that was stuck on D6 because there are modules they need not ready in D7 would pledge support for just one of those modules, contrib would catch up a whole lot faster. There are some sponsored modules, sure, but many, many modules are being done for free because someone either needed the functionality or thought it would be fun to do or some other reason other than getting paid. And there really isn't anything short of money (or someone else doing it) that will get those modules ported any faster than the person doing it for free feels like doing it.

Michelle

re: contrib lag

damien_vancouver's picture

Waiting for contrib to catch up could take forever. Everyone has a different idea of what is a "must have" contrib module.

I agree that everyone's idea of must have is different. However, we do have statistics by which we could make an informed decision of when that is. For instance, as reported in http://drupal.cocomore.com/blog/state-drupal-7-contrib in the 2nd section, the top 100 modules are now almost all at stable or RC releases. This in my mind would be an excellent metric to declare contrib "caught up".

The really popular ones usually don't lag that much, anyway, unless there's something else going on like a Views rewrite or CCK getting half sucked into core.

But... this does happen every release. Last release, Views was undergoing rapid change at release time. Field tokens were not available, and turned out to be virtually impossible to pull off without tons of extra work - there are still problems there including some nasty token recursion that it may not even be possible to fix.

Drupal 8 is going to contain major (exciting, awesome, superlative) changes from WSCCI... but that's going to involve major changes to the menu system and how requests are conceptually handled, which I'm sure will result in some slow contrib catch up again next time around.

Honestly, if every business that was stuck on D6 because there are modules they need not ready in D7 would pledge support for just one of those modules, contrib would catch up a whole lot faster.

I completely agree with you here. But that is sadly not at all how the real world is working out for us, and also can't happen efficiently without duplicated effort if we don't have project infrastructure surrounding it.

A client faced right now with the report that their site cannot be upgraded because modules are missing is not going to say "how about we pay to sponsor a module and just hope that every other company in our place does the same, and that by the time we finish sponsoring this one module, all the other ones we need will have had similar treatment and we can get our site built". Instead they say "say what? you told me that investing in Drupal was a good idea just two years ago, and now you're telling me I can't upgrade?!?" Suggestions to pick a lagging module and sponsor it and hope everything else just comes together won't go over well in that situation.

However - If there was project infrastructure and documentation and support surrounding this process, then maybe it is something that could happen. ie. "Join the universal movement to get contrib up to speed by sponsoring your module today" and "check in on the exciting progress at this progress meter page" and "here's a nice badge for your website to say what an awesome sponsor you are" etc. We could do it, but we can't get there from here by just ignoring the state and role of contrib and letting it sort itself out organically. It will die on the vine, just like the Drupal 5 incranation of LTS did. But the above suggestions are probably best hashed out in the LTS group, hopefully before Drupal 6's EOL.

.

Michelle's picture

I wasn't intending to start a discussion about putting in an infrastructure. The point, which I evidentially didn't make very well, is that too many companies just sit on a version because the new one "isn't ready yet" rather than jumping in and helping make it so. If every company that is "stuck" on a prior version would put in some time or money to move things along instead of waiting for other people to do it, new major versions would be ready much sooner and this wouldn't be such an issue.

This whole thing sounds to me like you're asking us to put the needs of those who don't contribute over those who do and that's never going to go over well.

Michelle

but they do contribute.. they contribute money.

damien_vancouver's picture

This whole thing sounds to me like you're asking us to put the needs of those who don't contribute over those who do and that's never going to go over well.

I'm not sure it has to be one over the other here, but yes I am asking that we consider their needs. WIth 600k+ sites out there, that means that millions of people are using Drupal. That's a lot of people!! I think that due to our success, we now have a moral obligation as a project to appropriately consider these people's needs when making decisions that affect them. While these people may not contribute patches or review issues, their patronage is ultimately footing the bill for Drupal on a macro economic scale. They are our benefactors. I believe that is a very important (if totally different) contribution to the project, and their needs should be considered as is appropriate to their numbers and role.

We don't have to put them first, that is not fair, but let's try and make it less painful for them.

Completely agree

arianek's picture

I completely agree with this. I've always felt that we don't actually focus enough on the needs of our "end users" (who I think of as being all of our clients, as well as site builders who don't have hardcore skills).

People like us, who are very engrained in the community and who do contribute to core/contrib, have the best shot at keeping up with the pace of development. But we really do leave a lot of those who helped build Drupal's demand, who pay our bills, and who keep our community growing, in the dust.

This takes shape in different forms, from prioritizing the needs of the developers over the needs of the end users when it comes to how we manage major versions (like Damien is saying), to how we onboard new end users - something that Dries and all of the surveys have clearly pointed at being a weak spot. Documentation (which is my pet project in the Drupalsphere) has always been fairly neglected, and yet it's consistently the #1 thing that people answering the surveys express wanting to be better. And it's one of the things that makes it hardest to onboard new end users, as they give up in frustration of not having adequate instructions to follow.

Questioning these processes isn't for "us" who are really tapped into things - we're the other end of the curve. And I think we really are neglecting the needs of a lot of those who are not contributing. As much as I believe wholeheartedly in contributing back the project (obviously), I don't think that only those who contribute should be able to drive the project. I think we should be also taking care of and nurturing those members of the community who we don't see and hear from all the time, but who we know are out there needing more support from us.

They're not contributing money to me. ;)

xjm's picture

Joking aside, I really think it's destructive to assign additional "moral obligations" to people who already donate so much time to Drupal because they believe in FOSS. That's simply not fair.

People and companies make money off Drupal. If a site uses Drupal but the company never files an issue, submits a patch, or contributes documentation, then how are they funding Drupal? They're funding themselves, and they could be using Wordpress or Rails or some closed-source solution for all the impact it has on the community and project.

Many wonderful businesses and individuals contribute back to the project. Many more don't. I'd rather talk about how we can move more people from column B to column A, than try to say that column A has moral obligations to column B.

+1000

Michelle's picture

I have no problem considering the needs of end users but there's a difference between keeping them in mind and making things more difficult for me in order to please them. They aren't paying me. If they are contributing back to Drupal in general, then I give them more consideration. If they are just taking from the project and not doing any giving, then I'm not going out of my way for them. Telling me I must do X whether I like it or not to make life easier for people who are neither paying me nor contributing to the project is just going to get my dander up. ;)

Michelle

in hindsight,

damien_vancouver's picture

I completely appreciate and agree with both of your positions, and apologize again for being so insensitive.

@xjm, I also do want to see people move from Column B to Column A too, and I think reducing the frustration of the newbies will foster an environment for doing just that. There's a lot more too it of course (good docs and web copy surrounding testing mainly, which I can help or lead creation of).

@Michelle, you're completely right, no you should certainly not have to do X whether you like it or not. That came out entirely wrong on my side. What I wanted is for us to acknowledge and consider these people and their problem, and hopefully find a non-painful way to help them out (without drastically affecting the contributors). I realize now that for this solution to feasible, it has to absolutely have zero negative impact on the contributors. In fact it should improve things for them or it's still not a good solution.

I think as a result of discussion here, we might be onto a good solution that meets those criteria. I am really anxious to get it off this horrible thread though and onto a new one.. but I don't want to start it in the wrong place. Is there a better g.d.o group than this one to start that new thread in? I'll start a new one with the pertinent details only, and retire this thread ASAP.

I think a lot of end users

arianek's picture

I think a lot of end users who don't contribute initially and have an easier/better initial experience with Drupal have a lot of potential to become contributors. So why shouldn't we want to cater to them more?

And as for eg. clients of the company I work for. They pay our team money to make them websites. And as part of making their websites, we contribute lots of tasty patches, patch reviews, documentation, etc. back to Drupal. And we also use some of the profits we make to fund contribution time on top of that. (IMO this is how all dev companies should work, but of course it's optional!) I'd say that's pretty directly funding development.

I think that both "random person with their little Drupal template site", and "client of Drupal shop who contribute back" would have a lot to gain from this kind of approach. And that for us as contributors, there wouldn't be a lot to lose by working this way. (I don't agree that it would necessarily harm contrib development - that's just an assumption.)

I'm not judging anyone's morals here (I would think that was obvious?) - I'm just saying I think we should make things easier and better for those groups if we can. It's not necessarily an either good for them or good for me kind of situation.

Sorry, I was replying to

xjm's picture

Sorry, I was replying to @damien_vancouver, who used the phrase "moral obligation."

arlinsandbulte's picture

Maybe this has already been said above (sorry, I did not read all posts)

In some ways, this was already done in D7... Drupal gardens was public WAY before D7 was, yet it used D7 unstable. And a lot of other sites were actively using D7 before the 7.0 'stable' release.
So, in a lot of ways, Gardens pushed D7-dev into the real world and a lot of bugs and changes were made because of that.
Doing this was a fairly new development in Drupal, largely attributed to the inclusion of the Testing Framework.
Next go around (D8), maybe the community will get even better at this and do it even more.

apology for offense

damien_vancouver's picture

Before this thread is put to bed I wanted to publicly apologize for my tone at the beginning of it.

What I meant to be provocative crossed way over the line to offensive for many very important people who have contributed more than words can say.

I got excited on a hot button issue and did not think nearly enough about the ramifications of how I was presenting it, or how inappropriately it would come across. That was really really stupid, not to mention counter-productive. I really do apologize for this and I promise to behave better in the future. Sorry.

Thread retired - please see new proposal instead

damien_vancouver's picture

I've created a proper proposal for the plan from this thread:

Proposal: Add a community testing period to each major Drupal release, before marking it as a recommended download.

Discussion can continue over there and this thread is now retired. Thank you.

Unofficial Drupal Roadmap

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: