Proposal: Loosen rules regarding patches that may be backported during a stable release (7.x+)

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

It's great when we get newly injected energy into usability efforts in Drupal, as happened with the Drupal Usability Study at Google last week. However, these patches always run aground of various "freezes" we put in place in the stable release of Drupal:

API freeze
Don't commit patches that break backwards API compatibility within a stable release, for the benefit of contributed module and theme authors.
String freeze
Don't change any user-facing text (e.g. anything in a t() function), for the benefit of translation teams.
UI freeze
Don't commit patches that affect the user interface, for the benefit of the documentation team, as well as to not invalidate tutorials, screencasts, books, etc.
Feature freeze
Don't commit patches that add new features; stick only to bug fixes. Additional functionality gets added in contrib.

As far as I can tell, these guidelines are not actually written down anywhere, but simply passed down word-of-mouth from core generation to core generation. I want to do two things in this thread:

1) Create documentation around these guidelines.
2) Propose that we change these guidelines to make them looser and allow more classes of patches to be backported, particularly minor UI tweaks and string improvements.

Here's a proposal (co-developed by Dries, Moshe, Alex, and I), with the items in bold being ones that we propose to change, the rest being documented to the best of my current understanding. My hope is that this can both help is iterate faster on the core product, and also provide additional incentive for people helping with Drupal 8 core development, as there'd be a chance of them seeing their changes in a version of Drupal sooner than 18+ months away.

Type of change Description Proposed guidelines
Data change Anything that would break the "we break your code, not your data" rule. TODO: Can't think of a good example, but basically something that would require manual data migration somehow. Critical issue only
API change Changing return values of a function/hook, adding new required parameter to function/hook. Would force contributed modules to change because something in core changed. Critical issue only
API addition Optional parameters on a function/hook, new hooks/helper functions. Would provide new capabilities which contributed modules could choose to rely on. Release notes mention
Data structure change Moving page elements to different fieldsets/containers, adding additional form elements, etc. Affects modules that do heavy altering of forms/pages. Release notes mention Critical issue only.
Markup change Changing output of theme functions, template files to remove wrapper markup, add . Requires changes to themes in order to stay in sync with core. Critical issue only
Markup addition Additional classes or attributes exposed to template files. TODO: Other examples? Themes could make use of them if desired, but would not be required to. Release notes mention
String change (admin-facing) Any changes or additions to any admin-facing string (string wrapped in t() or st()). Forces translators to translate new/changed strings, and multilingual sites see English strings until this is done. Release notes mention
String change (user-facing) Any changes or additions to any user-facing string (string wrapped in t() or st()); for example comments or the log in form. Forces translators to translate new/changed strings, and multilingual sites see English strings until this is done. Major/Critical string problem only. (e.g. typo, string is saying completely wrong information)
String change (error message) Any changes or additions to any error messages (string wrapped in t() or st()); for example form validation errors or error messages coming from exceptions. Forces translators to translate new/changed strings, and multilingual sites see English strings until this is done. Major/Critical string problem only. (e.g. typo, string is saying completely wrong information)
UI change Any changes or additions to the user interface, such as new administrative pages, re-ordering of fields, etc. Release notes mention, as long as the "conceptual" things for users to know remain the to grapple with, and the change is not systemic (e.g. switching the order of all form buttons).
Feature addition Any changes or additions to the user interface, such as new administrative pages, additional options on an administrative page, new permissions, etc. Release notes mention, as long as they are "opt-in." For example, a new checkbox that could optionally be checked, a new theme or module which could be enabled or not.

Whew! I think that's everything. What do you think?

Comments

+1

Crell's picture

Overall I am in favor of loosening these restrictions. My main concern is that Drupal is often not well-designed for such things, but that's a separate problem. :-) The guidelines above seem reasonable.

I will note two push-back points:

1) Adding functions or optional parameters: Cool, makes sense. However, classes present a slightly different problem. We can add methods to a class all day and break nothing, as long as the interface for existing methods doesn't change. Cool. However, changes to an interface (the PHP interface language construct, I mean) would force changes on any classes that implement that interface. So it's an API addition, but one that could necessitate changes elsewhere. On the other hand, a good general practice is to provide both an interface and a default base class, so if you're extending the base class then presumably the base class will be updated with a new default, so nothing breaks then. I'm not sure where I'm going with this, other than noting that we should think through where the line is between API addition and API change; it's not always a hard line.

2) Are there feature additions that are too big to backport, not because it's hard or breaks things but because we want people to have to upgrade to get them? I don't mean that in a malicious way, either. Windows' primary competition for years has been previous versions of Windows. :-) Also, the way you'd add a feature in Drupal Stable and Drupal Next may not always be the same, or even remotely the same. What I would not want to see happen is backport potential crippling the "best" implementation possible in Drupal Next.

Also note that since we allow direct DB access and direct access to virtually every data structure, all DB changes and all data structure changes are, or could be viewed as, API changes. We've done those before, and I'm not saying we should not continue to do so where it makes sense, but it is something to be aware of.

I think Push-Back Point #2 is

CatherineOmega's picture

I think Push-Back Point #2 is a good one. Anyone who's done any front-end work whatsoever can see the necessity of getting people to give up The Old Version, whatever that may be. However, maybe this could be the "final step" in deciding whether to backport. I can't think of a large enough feature offhand, but agree that if the implementation may not even be remotely the same, some serious consideration should be given, weighing of pros and cons, and so on. (If the implementation IS the same in Drupal Stable and Drupal Next, I say backport away!)

I agree it's definitely

webchick's picture

I agree it's definitely something to consider. However, to some extent this is mitigated automatically by the things we can't backport.

For example, we simply can't ship D7 with native, responsive HTML5 support like we hope to have in Drupal 8, because it would mean breaking markup for existing themes. We could ship with an optionally-enabled, more modern HTML5/mobile-friendly base theme in Drupal 7.18, though, which pulls in the markup changes in D8 core and lets people test them early.

Or take for example something like CMI or WSCCI. They will both require vast under-the-hood changes in order to make Drupal's configuration file-based and page serving logic, and therefore not be candidates for backport. But if successful, they will both make compelling reasons to upgrade to the next version.

Also, even in the case where we could technically backport, we could choose not to. Let's say the Drupal 8 Design initiative resulted in a new default theme design for Drupal 8—we could deliberately choose not to backport that in order to protect the branding of the new release.

LoMo's picture

Having a HTML5/mobile-friendly base theme in D7 core would be great, especially if it allows the community-at-large to pre-test some of the functionality in D8 core and helps iron out any issues that might have otherwise been overlooked. We need to make it easy to target mobile device now and while I know there are modules and themes which help with that, getting it into “core” in D7 will help revitalize Drupal and keep it a contender, especially for those considering using Drupal who might not yet understand how extensible Drupal is (i.e. given the lessons learned from last year’s usability studies).

About me:
  • Drupal evangelist
  • (Former) regular author of the Cocomore Drupal Blog
  • Systems tester / QA automation, Kairion GmbH / freelance

Also note that since we allow

linclark's picture

Also note that since we allow direct DB access and direct access to virtually every data structure, all DB changes and all data structure changes are, or could be viewed as, API changes.

I was thinking the same thing as I read the Data Structure one. I think we want to be particularly careful in judging whether these are actually necessary.

"API"

Crell's picture

Part of the difficulty there is that we do not clearly differentiate what is part of an API and what is not. Exposing bare objects means "everything is an API", which is problematic for many reasons, this included.

Arguably we should declare that the structure of certain tables is "not an API, you're not supposed to be touching this directly", and then make that modifiable. That's in practice what we've done in the past, informally. Of course, doing that retroactively could screw some people over anyway. (I've run into that with contrib modules from time to time, and it's highly annoying.)

API vs direct data access

DamienMcKenna's picture

Of course, given many of the core Drupal 7 data structures do not have ample CRUD APIs around them, any changes to data structures could be catastrophic for some sites.

Yeah to me if data structure

catch's picture

Yeah to me if data structure is being changed, then that's always going to be the return function of some function or hook somewhere, so I wouldn't really differentiate between that and an API change at all. The question then comes down to how likely we think it is that a module somewhere is relying on the thing being changed.

Agreed that data structures

webchick's picture

Agreed that data structures are the most problematic on the list, especially since something as seemingly innocent/"no-brainer" as adding a fieldset or moving a field into a different fieldset or adding a new checkbox to an admin form alters the underlying form/page array.

When Alex talked about this in reference to Drupal Gardens (which has something like 250 hook_foo_alter()s) he said that people doing that kind of precision custom alteration work would likely be reading the release notes (and poring over the diff between D7.X and D7.X+1) anyway, since there are so many things that could possibly break their stuff each point release (not to mention all of the contributed modules out there that don't have any of these notions of freezes). A mention in the release notes at least provides a pointer to what they need to change.

On the other hand, there are obviously sites out there that have been built by teams containing custom modules with alters, then the team who built it moved on to other things, leaving the poor schmuck at the company in a bad spot when a minor point release breaks their site. There are also contributed modules that make use of those alter hooks, too.

Yep, I'm going to easily say

Dave Reid's picture

Yep, I'm going to easily say that we can't just change these things around with a release notes mention. Most people do not have a dedicated developer pouring over release notes.

Senior Drupal Developer for Lullabot | www.davereid.net | @davereid

Overall, these guidelines

markdorison's picture

Overall, these guidelines seem pretty sound to me, though I would be interested in hearing dissenting opinions.

Oops, forgot to mention...

webchick's picture

This list was brainstormed last week together with Dries, Moshe, and Alex "effulgentsia" Bronstein, so there's already buy-in on this general idea from "upstairs." :) But we're definitely interested in getting perspectives from other people who actually build Drupal sites. ;)

I am in favour of this. I

timmillwood's picture

I am in favour of this. I would like to see many things added to existing drupal versions that build upon and improve existing features. As long as they don't break anything.

It'd be also great if we could give advanced warning of major changes incase there are any contrib modules which are affected by these changes.

themes / feature additions

xjm's picture

Most of this makes sense to me, but I'd like to make sure we are at least cautious about UI changes and markup additions. Those sorts of changes could easily toast existing themes if we're not careful, especially site-specific themes.

Also, I'd like to see "feature additions" scoped out a bit more... that could be, dunno, a slippery slope? Isn't our mantra generally "New features for stable releases can live in contrib"?

New guideline

Crell's picture

Perhaps that can be part of the guideline for what gets backported? There's a lot in core that cannot be fixed from contrib. Perhaps when evaluating a backport, we should look at it and say "so could this be easily done in contrib with a 4 line alter?" If it's straightforward to replicate entirely via contrib, we don't backport (because it can be backported via contrib). If it would be a pain to do in contrib, or not possible, then we can consider it for an in-core backport.

That's still a somewhat fuzzy line, but at least it's something to go by. I also like the idea of "backport only if the implementation is largely the same so it doesn't take much work".

The concern makes sense, but

webchick's picture

The concern makes sense, but be careful about the panacea of "oh, there's a contrib module for that." There are 15 THOUSAND contributed modules. :) And especially when you're new, finding the one out there that fixes the problem you're having right now is extremely daunting.

Note that this doesn't mean bringing in all of the top 50 modules to core. We'd still stay with the rough guideline of "If it's useful to 80% of sites" or whatever. But what it does mean is that if we put a nice small feature in D8 that doesn't hurt anything (say, like the automatic admin role we added in D7), as long as it can be cleanly backported in the same way, I think it should be on the table for consideration.

Examples of features I'd ideally like us to be able considering backporting to D7 (assuming they made it into D8) are things like Add a UI for if the site supports https, Include Pathauto in core, Convert Bartik's layout to be adaptive/responsive, and so on. All of these could be (and currently are) serviced by contrib. But offering them as part of the "out of the box" experience makes for a much friendlier Drupal.

The question of "Well how do we motivate people to switch to Drupal 8 then if D7 constantly has more backported awesome polish?" is still a very valid question, and something we need to consider carefully.

Backports module

David_Rothstein's picture

The concern makes sense, but be careful about the panacea of "oh, there's a contrib module for that." There are 15 THOUSAND contributed modules. :) And especially when you're new, finding the one out there that fixes the problem you're having right now is extremely daunting.

I cross-posted below while you were writing this, but I'll mention it here too: http://drupal.org/project/backports

That module needs lots of love but could easily become the semi-official repository for improvements that were committed to Drupal 8 to be backported to Drupal 7... So in that case it wouldn't be necessary for people to search for many separate contrib modules that solve each problem they're having.

Some ideas on the UX impact

Bojhan's picture

Sounds like a great idea! I always wished we would be more liberal in the way that UI changes are handled, because many are benefical and would not change the learning curve.
I would love to learn more about what counts as a ungrappable change. From my understanding something like;

I understand that we will make the guidelines up, as we go and knowing that my examples are correct, will help me understand how to apply the proposed guidelines.

We should set expectations, because many of of the fundamental UX issues our users encounter from; not being able to find pages, unable to use functionality, unable to grasp concepts like fields - will require us to radically change existing pages, such as the blocks UI, fields UI, module UI etc. We cannot fix those with the proposed guidelines, because they stretch beyond what we can do with small string/visual tweaks, nor will it likely to inject new energy in them.

However what will be possible, is that it will inject energy into initiatives such as the mobile iniative! Which essentially is a new feature? Because that would be amazing, its a way for us to get more UX activity on new opportunities - which is often hard to rally people around, because of the 18-month cycle.

I hope our users understand, why they do get one improvement but not another, as Crell notes it will blurr the line why one would upgrade.

Either way, really interesting idea! I would love to know, how this should solve some of the "freezes", from my perspective most of them apply to instances where the proposed rules do not apply - then again I am biased, and only see the big issues as frozen.

Awesome, thanks for your

webchick's picture

Awesome, thanks for your thoughts.

Yeah, Improve UX for machine names for fields is exactly what I had in mind as the type of change that we should be able to backport to D7. I think A cleaner look for the status report also applies, since it's a small visual change to a page that is only visited every so often. Other things I had in mind were Search entities (nodes, terms, etc.) within the administrative interface, Searchable modules page, Add a "Create content type" link to the content listing page and/or the add content page, etc.

We should set expectations, because many of of the fundamental UX issues our users encounter from; not being able to find pages, unable to use functionality, unable to grasp concepts like fields - will require us to radically change existing pages, such as the blocks UI, fields UI, module UI etc. We cannot fix those with the proposed guidelines, because they stretch beyond what we can do with small string/visual tweaks, nor will it likely to inject new energy in them.

I don't know that we need to box ourselves in quite like that. We've opened the door to changing the rules, so let's discuss "blue sky" a bit and think about what would actually work the best for the various people involved in Drupal.

According to Dries, even something like Visiblity of primary & secondary navigation would be a candidate for backport. Even though it's a fairly stark visual change, the actual mechanisms to navigate remain exactly the same (Click on the Foo tab, then the Bar link underneath it), so it doesn't require imparting new conceptual hurdles on users.

The reason not to do it would be that it changes Seven's underlying CSS, and if another admin theme is sub-theming off it that theme would break (thus an API/markup change). However, core not being able to make incremental improvements to its own admin/front-end themes severely cripples the kinds of things we could incrementally hack on. Not quite sure how to reconcile this conflict. Though I guess if we kept changing Seven and Bartik every point release that'd stop people from subtheming off them pretty quick. :D

However what will be possible, is that it will inject energy into initiatives such as the mobile initiative! Which essentially is a new feature? Because that would be amazing, its a way for us to get more UX activity on new opportunities - which is often hard to rally people around, because of the 18-month cycle.

Yep, that's exactly the goal. :) Also, to get some real-world testing of these new features way before Drupal 8 actually ships, and to hopefully make the jump from D7 to D8 a tiny bit less severe on people.

I hope our users understand, why they do get one improvement but not another, as Crell notes it will blurr the line why one would upgrade.

I addressed this to some extent up above. I think that the compelling reasons to upgrade to a new release will happen based on what we simply can't backport due to the other pieces (backwards compatibility, markup changes, etc).

Either way, really interesting idea! I would love to know, how this should solve some of the "freezes", from my perspective most of them apply to instances where the proposed rules do not apply - then again I am biased, and only see the big issues as frozen.

We'd do away with the term "freezes," IMO (except in the N months leading up to a new major release when we need to crank on bug fixes), and instead evaluate it on a patch-per-patch basis based on the guidelines in the table.

because many are benefical

tsvenson's picture

because many are benefical and would not change the learning curve.

Actually it would change it by (hopefully) make it less steep by being able to for example incorporate changes found in usability tests.

--
/thomas
T: @tsvenson | S: tsvenson.com

Making changes might make D8 adoption easier

circuscowboy's picture

If changes that are made to the underlying structure brought D7 closer to what D8 is going to be it could be a good thing. Even if modules were broken with easy fixes it would create a gradual transition rather than a sharp change when everything needs to get upgraded.

My point here is that people are that contribute have little bits of time and introducing changes that bring modules upgrade path closer would make the final transition easier. Keep the community on there toes rather than have things in contrib land get stagnant and only have to work hard every few years when a new core comes out.

If everything back-ported is well thought out - the change is going to happen sometime why not make it gradual.

All typo / minimum string

droplet's picture
  • All typo / minimum string changes could be backported directly. Integrating localize.drupal.org to list all changes. Update few words in every release is eaiser than tranlate/update all files in D8. I saw an issue is changed 10+ sentences, added a DOT at line end. If it backported immedately, we just fix it and add 10 DOTs. But until D8, it maybe tranlate all sentences again. Tranlation memory sometimes don't work smoothly.

  • Project pages missing Version Compatibility info.

Thomas_Zahreddin's picture

Hm, i do not see the need to seperate these two types of strings.

On the other hand i see the need to distinguish the error messages for admins form these for users.

I was pulling these

webchick's picture

I was pulling these definitions from Gábor's comment at http://groups.drupal.org/node/154304#comment-514814. I think it was in relation to all error messages, since whether you're an end user or an admin "エラー未定義の変数" doesn't help you resolve your problem. ;)

David_Rothstein's picture

If we allow data structures to change, then it seems to me this opens up the potential of introducing not just bugs but also security issues in contrib modules which are altering or using those data structures. I'm not that excited about a world in which Drupal 7.13 gets released and suddenly a security hole opens up in the 7.x-1.0 version of a contrib module I have running on my site... How will we avoid this?

In general, allowing data structures to change means that point releases of Drupal 7 will be riskier than they are currently. And if you look at the data, I'm not sure that's the direction we should be going in. According to http://drupal.org/project/usage/drupal, although it's been a couple weeks since the last Drupal core security release, only about one third of Drupal 7 sites have actually updated to the latest secure version. Granted that doesn't mean all those sites have actual security holes in Drupal core (since many of them might not be using the affected functionality), but it suggests that people are generally very hesitant to just drop a Drupal core point release into their site. So it seems to me like we need to be making these point releases safer and more plug-and-play, not less.

If the goal here is to allow usability improvements to be backported, then I like Crell and xjm's suggestion above; how about we only consider these riskier changes for backport if they are difficult or impossible to do via contrib?

And in terms of where they might live in contrib, I'll point out that the issue queue of http://drupal.org/project/backports is open and accepting patches :) It would be great to get more people working on that project and perhaps even to get it to the point where it can be promoted as the semi-official solution for people who want to see Drupal 8 changes on their Drupal 7 sites.

Well. I'm not sure if we can

webchick's picture

Well. I'm not sure if we can infer from the usage stats that the reason people aren't upgrading to the latest stable is because of stability. For example, right now Drupal 6 actually has more sites on the previous version than the latest version, and it's been the stable (as in the traditional definition of stable) release for over 4 years. And essentially any release of Drupal 7 is going to be more stable than any release of Drupal 6, by definition, because of the testing framework. (Which is why we couldn't really talk about changing this policy until D7+.)

I hypothesize that the lag in upgrade has a lot more to do that the core upgrade process is difficult and terrible for anyone not using Drush. I really hope Allow upgrading core with the update manager makes it into Drupal 8; that'd be our single biggest win against that trend (of course, the question is if it did get into Drupal 8, could we backport it to Drupal 7? :)).

The Backports module approach is interesting. But as a "real" site builder, I'm not sure I would ever install it. Looking at http://drupalcode.org/project/backports.git/blob/refs/heads/7.x-1.x:/bac... it seems like currently roughly 30% of the code in that module is working around the fact that these changes aren't in core and they have to be shoe-horned in through alters. However, for data structure changes that are deemed too severe to backport, this is a really good option.

Totally agree that allowing

David_Rothstein's picture

Totally agree that allowing the Update Manager to update core would be a huge win. But I'm not sure that's the main factor here... To find out, we can look at some contrib modules (since it's already possible to update those with the Update Manager).

Above, I pointed out that a couple weeks after the last Drupal 7 core security release, only about one third of sites had updated. So, we can look at some recent security releases of popular contrib modules for comparison. I found a couple:

  1. Webform had a security release on March 7, and current usage stats at http://drupal.org/project/usage/webform show that around one third of Drupal 7 sites which use this module have updated.
  2. CKEditor had a security release on March 14, and current usage stats at http://drupal.org/project/usage/ckeditor show that around one quarter of Drupal 7 sites which use this module have updated.

So roughly, the numbers seem consistent to me between core and contrib; regardless of how easy the update is from a technical standpoint, a significant number of people aren't taking these security updates in a timely fashion. (On the other hand, the numbers look slightly worse for the Drupal 6 versions of these modules, so we may be getting better.)

The main point is that overall, Drupal security updates are not being applied with the frequency we'd like. For a lot of other types of software, people just deploy updates and forget about it, but that's not generally the case with Drupal. Some of that is probably inherent to the way Drupal works, but my point is we really should be prioritizing stability, and getting core point releases as stable and safe as possible is an important example to set.

Current policy

David_Rothstein's picture

By the way, I just wanted to point out that the current policy is actually semi-documented at http://drupal.org/node/767608 (see the "Intention of the backport policy" section at the bottom), although not in anywhere near as much detail as in this post.

So it will be good to officially adopt the rules above for when, e.g., string changes can be made, since string changes aren't something that the current written policy mentions at all.

Thanks! I could not, for the

webchick's picture

Thanks! I could not, for the life of me, find that. :)

Completing unfinished features

tsvenson's picture

I am very happy about loosing up the freeze rules is now seriously discussed.

As far as I can tell, these guidelines are not actually written down anywhere

Explains why its been so darn hard to find them those times I tried to search or asked someone on IRC to point me to them :)

Completing unfinished features

One of the most important changes this would have is that we will have a mechanism in place to be able to complete features after a .0 release. In many cases these unfinished features aren't discovered until after the .0 release and users are beginning to build real sites with it.

Just for me to get a grasp of the limits here. Would this change for example mean:
- Tile can become a proper field?
- Much of the missing core tokens/features can be moved from the contrib token module to core?
- A large new feature set such as the File Entity module can me moved to core?
- Missing user fields exposed on the Manage Fields tab can finally get into D7 core?
To name a few examples.

Communicating changes

Reading the OP table, I'm not sure just mentioning changes in the Release notes is enough. First of, those notes needs to be reformatted so all changes that are not security/bug fixes are separated so they don't get buried in the usual quite long list. Its hard enough as it is to read for me.

Clear visual warnings will be needed in other places too, including the project page so users gets alerted about that a new point release contains changes that may affect the functionality of their site. Many user do not read, or fully understand, the release notes and this will mean we need to communicate these changes in a way they understand.

Automatic test tools

Would it be possible to use for example the coder module to review a module to see if changes in a new version affect a specific module? I haven't used it much myself, but from what I understand it can even do porting of a module between majors (to some extent).

Just thinking that it then should be able to keep track on changes between point releases and use that to present a report for existing modules.

Of course, the ultimate would be if all contribs could be scanned and maintainers could be alerted if their modules use anything that may be affected by these changes.

Other than that, I believe this is something that is going to be very useful for moving Drupal forward, especially since release cycles has gone from <1 year to 3+ years.

--
/thomas
T: @tsvenson | S: tsvenson.com

Ti[t]le can become a proper

webchick's picture
  • Ti[t]le can become a proper field?

That one I don't think so, since it would break themes, it would break modules printing out the node title, etc. So it'd violate the "API change" parts of this proposal.

  • Much of the missing core tokens/features can be moved from the contrib token module to core?

That one I think should absolutely be on the table, yep. We'd have to tread carefully since there'd need to be coordination between Token module releases and core releases so people didn't end up with two of the same thing and explosions, but I think that's a solvable problem.

  • A large new feature set such as the File Entity module can me moved to core?

Not sure about that one, would have to investigate more closely. If it merely added on capabilities to File entities, perhaps? But if it's fundamentally changing file entities, then no, because it would break modules that rely on them.

  • Missing user fields exposed on the Manage Fields tab can finally get into D7 core?

Yes, since that's more or less a bug, however that issue is already tagged for D7 backport.

Regarding release notes, right now I just use the template that was already established before my time. However, I'm happy to use a different release notes template that's more useful to site builders. Would you care to propose something in another discussion, and link it here?

I'm not sure about cluttering up the project page. I'd rather call it out on the release announcement post, improve the release notes template, and then drive people towards reading the release notes.

Unfortunately though, no, it is not super easy to scan all of contrib to check for how a proposed change affects them now that we've moved to Git. :( http://drupal.org/node/1057386 has some ideas. In general, though, I think for API changes we should continue to be very conservative regardless of the impact on contrib, because custom modules often make a lot of use of APIs (particularly alters) that contrib modules don't.

Thanks for the great answers.

tsvenson's picture

Thanks for the great answers. Made things much clearer for me. Dave Reid just told me File Entity can't get in there, but its not a biggie. Probably good it can evolve fully as a contrib and be mature for D8.

I would be happy to create a proposal for a new release notes template. Where can I find the existing and what would be the best place to submit the proposal, with link from here to it?

I still think there need to be some form of visual alert on the project page that at least gives users a clear indication that reading the release notes before performing the update is important.

--
/thomas
T: @tsvenson | S: tsvenson.com

Stable means stable

Damien Tournoud's picture

Stable means stable. It has to stay that way. Drupal is an open system in which it is legitimate to modify anything. It is the strength of Drupal (and arguably its main weakness).

Especially, it means that you cannot do that:

Data structure change: Moving page elements to different fieldsets/containers, adding additional form elements, etc. Affects modules that do heavy altering of forms/pages.

Most websites moderately-complex websites are heavily hook_form_alter()-ing the default UI of Drupal. You cannot suddenly add a new element or move the elements around.

That probably means that usability issues cannot be fixed in stable releases, yes. But those are stable for a reason.

Damien Tournoud

Yeah, I think the problem

webchick's picture

Yeah, I think the problem we're trying to outline here though is that the current level of stability we enforce on core is slowly killing us. It means we can only iterate on the core product once every 2+ years, and it's a HUGE struggle to find people willing/able to work on core because it involves many arduous hours of tremendous personal sacrifice, only to not be able to see the fruits of your labour for years down the road. It's simply not sustainable.

So the question is if we can cordon off some kind of "safe zones" in core that it's possible to iterate on over the years that lets us meet a happy medium between keeping stability for module/theme developers and site builders, but also providing more immediate benefit for core contributors to hopefully swell their ranks.

Fair point about the data structure change specifically, though. Any way we could come to a middle ground about that, though? http://groups.drupal.org/node/210973#comment-696458

The one category of backports

catch's picture

The one category of backports I think we could consider loosening up easily are documentation patches, since while they can change a lot of code, they actually make backporting future patches easier rather than harder by keeping the code bases as in sync as possible. They'll break all existing patches in the queue that touch that code, but those patches would need to be re-rolled for 8.x anyway, and there's zero risk of actually breaking something.

If we look at commits since September, the majority of Drupal 8 patches are already backported to Drupal 7:

~/www/8$ git log --since=03-01-11 --pretty=oneline | wc
1033 15132 143121

~/www/7$ git log --since=03-01-11 --pretty=oneline | wc
777

So the balance seems about right at the moment, and I agree with Damien - it's more important that people can rely on some level of core stability and their sites not blowing up or gaining unexpected changes with a core update. The other thing that causes attrition with core contributors is spending lots of time dealing with unexpected regressions or unintended backwards compatibility breakage and I'm more worried about trying to prevent people burning out quite so much than finding new people to replace them at the moment.

The one way I could see out of this would be an arrangement similar to the one there was with Pressflow in Drupal 6 (Pressflow for Drupal 7 is advertised but afaik doesn't have any real differences to core yet). i.e. have a branch that includes backports with a lower threshold than core itself. People can opt-in to use that, if it introduces incompatibilities with contrib modules (such as Pressflow's session handling in Drupal 6 did), then those can be patched in the contrib modules, but by nature it's got a smaller user base than the stable core release which is guaranteed to be stable and who are prepared for slightly more breakage to get the extra features. Then if something has been in that branch for months without causing any known side effects, it could be reconsidered for backport to the stable branch possibly. But doing this all in a single branch it really feels like things are close to, if not at the limit. I know I move every possible patch back to 7.x that I think could possibly go in.

Yep, I think the balance for

webchick's picture

Yep, I think the balance for backporting API and developer-related features is about perfect. What I'm essentially asking is how we can extend that (and in what ways) to end users and site builders.

Technically, stability means "Sorry, Drupal 7 didn't ship with a hook_widget_form_alter(), so go eff yourself." And largely, this is the attitude we took in Drupal 6 and below, mainly due to not having a testing framework which could catch at least the super-obvious problems. However, since the "go eff yourself" response tends to burn people out way more than coming across an unintended bug or two, we take a calculated risk with possible namespace collisions and we add the capability. And contributed module authors seem to love this, because it means they're not stuck with the same crappy limitations that Drupal 7 shipped with because we didn't have the foresight back then to figure out how it would be used it in the wild.

Adding new API capabilities to core in later point releases means better contributed modules with fewer workarounds, and happier developers who get little additional nuggets of goodness coming down in point releases. This is a good thing. So how can we provide some of the same for people working in the UI? I'd really love the answer to not be "Go eff yourself, unless you download this fork of core" (whether that be a Pressflow-like thing or via the Backports module). :\ But that does mean some amount of burden on the Translation and Documentation teams (both of which are CCed on this post), and it also means that despite best intentions, we might break a module doing something crazy alter-wise once in awhile. But the benefit of having happier site builders not stuck with the same crappy limitations since Drupal 7.0 seems worth at least looking at.

I agree that "Data structures" taken on its own though is way too broad of a category to call on the table for fixing, and there are many cases where this actually /would/ break stuff fairly significantly. Do you have any alternative suggestions of how we might codify what essentially amounts to a gut call atm?

142% in agreement

tsvenson's picture

As a site builder I am totally in agreement with your arguments here. I much rather go through a little pain/risk that introduction of new/slightly changed features in point releases than the pain and maintenance requirements workarounds means.

Its is also massively improving things for new projects as well as new users coming to Drupal. Having to start a new project by first adding a bunch of workarounds for well known shortcommings is just crazy.

--
/thomas
T: @tsvenson | S: tsvenson.com

Yep, I think the balance for

catch's picture

Yep, I think the balance for backporting API and developer-related features is about perfect. What I'm essentially asking is how we can extend that (and in what ways) to end users and site builders.

Most of those changes are going to be some kind of API change, so I think it's really the same discussion (although string changes and things that might invalidate screenshots are a slightly different category).

I don't really have any suggestions for how to detect the impact of API changes. drupalcontrib.org can be helpful for this stuff sometimes. Some of the work Lin Clark did on resurrecting greps of contrib might help.

While it wouldn't be easy to set up, especially for 7.x where things are already established, then it feels like it'd be easiest from the point of view of both flexibility and stability to allow core modules to have 7.x-2.x branches - then new installs or people who want new features could happily upgrade to the 2.x version (which matches 8.x as closely as practically possible without obvious API breakage), while sites that need absolute stability can stay on 1.x. That opens up the can of worms of keeping core modules in separate repos, but what's really being asked for here is the same flexibility for core modules that already exists for contrib. And people are already doing this, for example with http://drupal.org/project/simpletest.

allow core modules to have

tsvenson's picture

allow core modules to have 7.x-2.x branches

Wouldn't that risk creating a complex situation for contribs? Basically they will either have to maintain several 7.x-X.x versions or not take advantage of new features introduced in for example a 7.x-2.x version?

Then it could be really crazy to be on top when it comes to dependency between contrib modules.

--
/thomas
T: @tsvenson | S: tsvenson.com

Yes it would. But we have

catch's picture

Yes it would. But we have versioned dependencies for this reason which can be relied upon, better than breaking contrib modules with no opt out at all.

Bundle a backports module with core?

effulgentsia's picture

So how can we provide some of the same for people working in the UI? I'd really love the answer to not be "Go eff yourself, unless you download this fork of core" (whether that be a Pressflow-like thing or via the Backports module)

Would renaming Backports to ui_update or drupal_update and adding it, along with jquery_update, to core address this concern? So, no need to download anything, but enable this module if you want to run the latest stuff vs. don't enable it if you're concerned about integrations problems with your site's contrib/custom modules.

I'd definitely support

catch's picture

I'd definitely support jquery_update in core, the fact that major core versions end up with out of date and unsupported jQuery versions almost the instant they're released is really unfortunate.

I quite like the idea of a ui_update module as well, contrib module dependencies can handle that easily enough and it doesn't impose changes on anyone. It might get complicated of course and there'd still need to be criteria for what gets applied directly vs. what goes in the optional module. This still doesn't get around the versioning issue - i.e. if we backport a UI change once, then update it again six months later, then a module has no way to differentiate between the two, compared to if core modules could have minor version numbering. So lots to work out, but feels like the least worst option.

Either of these could also be enabled by default in the standard profile if we wanted them to, since they only affect new sites.

I really like the idea of

David_Rothstein's picture

I really like the idea of including a Backports or Drupal Update module in core (rather than relying on the contrib version) and turning it on by default in the standard install profile.

In fact, we might even turn it on by default in other situations too, depending on how stable we can make it...

I went ahead and created an issue in the Drupal core queue to discuss this further: http://drupal.org/node/1505456

Version-agnostic API consumers

sun's picture

The jquery_update proposal translates into "Libraries API in core" for me. You need the concept and infrastructure of Libraries API to allow all modules and themes to properly react on and integrate with whichever version is getting loaded.

While there's a feature request for Libraries API in core already, I frankly don't see it as mature enough yet - especially since 2.x is still in development and not battle-tested yet. I'd personally recommend to wait with that until we are confident that it works at a large scale - and that's to be proven via jQuery module.

Realistically, I don't really see it as an option for D8 - there's simply not enough time left. Targeting D9 sounds more reasonable to me.

Daniel F. Kudwien
netzstrategen

LibrariesAPI in core = doable

DamienMcKenna's picture

I replied on the LibrariesAPI-in-core issue, but in short: I disagree completely on not being able to have functionality similar to the module included in D8, it's not exactly an overly large module.

Do you agree with

effulgentsia's picture

Damien, do you agree with my proposed order in http://groups.drupal.org/node/210973#comment-701613? Please reply there.

Two Tier Release System.

alanmackenzie's picture

A slightly more complicated release system could side step this entire issue.

Ubuntu releases versions with Long Term Support as well as normal releases with a more limited time to live. The Linux kernel also has stable and unstable releases.

The exact details of how to do this are a discussion in their own right but I think allowing the end user to choose new features v stability is a better solution than an awkward compromise of the two.

When it comes to API changes, Linux kernel is no comparison

LoMo's picture

Applications written for Linux aren’t usually tied to one version of the kernel, as is the case with Drupal modules, so upgrading a Linux system is not as massive an undertaking as upgrading a non-trivial Drupal installation. That said, I do think the community should plan to provide “official support” for Drupal 6 (i.e. backporting of security fixes) after Drupal 8’s release since so many of the current D6-based sites (still about 300,000) may never be able to upgrade even if all the contrib modules they use have a clean upgrade path available; a lot of these sites have a number of custom modules which might require too much expense to justify porting for each major version of Drupal and I suspect that the vast majority of those with a long-time involvement in the community will have some kind of "stakeholder" relationship to such a D6 project.

In an article I recently wrote for Cocomore’s blog about the State of Drupal 7 / Contrib, I suggested (toward the end of the article), that future releases of Drupal could maintain a greater level of API compatibility (i.e. not between Drupal 7 and 8, but perhaps between Drupal 8 and 9), which could mean that you could see Drupal 8 as being an “LTS” release in the sense that Drupal 9 could potentially natively support modules written against the Drupal 8 API, without immediate need for them to be ported to Drupal 9. It might require an “API compatibility module” (or set of modules) be activated to allow for sites to be upgraded from Drupal 8 or to run with Drupal 8 modules before corresponding stable Drupal 9 versions are available. To avoid carrying “too much baggage” through multiple releases of Drupal, this could be done only for alternating releases or the API compatibility could extend back just one version. It could also be done with “data sharding” where, invisible to the normal user, a separate database or tables are used for one (older) version of Drupal which could be integrated into the new version of Drupal, which is currently not an easy task, but some sites are doing it to run Drupal 7 for most parts of the site and Drupal 6 where key features were not yet ready for D7.

Anyway, this is just an idea for the future of Drupal and I’m sure that there are better approaches and/or reasons to reject my ideas on this altogether, but if we are brainstorming what can be done to make upgrades less painful and help blur the lines between major releases of Drupal, I’m hoping something could come from discussing these kind of ideas within the scope of this conversation. For my desktop OS, I personally use Mac OS X and would probably almost never upgrade major versions of the OS if I knew that all my applications would need to be upgraded as a result; even if they were all free and open source. As it is, I have been using OS X long enough that I avoid using "dot-oh" versions, altogether, and normally wait at least several months to a year before I move to the next “big cat”. But at least then, even if a newer version of an application I use on occasion has not been rewritten for “OS X NewCat”, I can almost always still run it. The same is true for many applications written for Linux; if they ran in version X, they still run on version Y. But, so far, it’s not at all true for Drupal modules, which puts a hefty damper on the enthusiasm to upgrade. For larger “Drupal shops” which build many Drupal-based projects every year, it’s not such a big deal and they know when to suggest to clients that a project is worthwhile going through a major upgrade. But for many other stakesholders who potentially have all their work in one project and are more focused on the content than on continually building new Drupal sites, a misguided attempt to upgrade dampens enthusiasm for Drupal. If more features could be backported, it would make life a lot easier for those who cannot afford to go through a major site upgrade (loss of features or data or time required would be too great a barrier).

In short and more on-topic: I’m all for backporting whatever can be backported without breaking anything. I think there will always be plenty of interest in upgrading, but until major upgrades are a trivial matter, backporting of features is a great way to keep the community-at-large happy.

About me:
  • Drupal evangelist
  • (Former) regular author of the Cocomore Drupal Blog
  • Systems tester / QA automation, Kairion GmbH / freelance
damien_vancouver's picture

Borrowing from the very successful Debian Linux model, what if we introduced an extra release into the release cycle, "testing"?

On the planned Drupal 8 release date, it would become "testing". This would signal the start of the efforts for module maintainers to port their modules to the new version, and early adopters to kick the tires. Brave people could even use it to build sites for their customers. Freeze restrictions are relaxed as suggested at the start of this thread, so we can actually fix things effectively during that time it's in testing.

Core work can forge ahead with Drupal 9 as "unstable", with no contrib available, much like D8 is right now.

Drupal 7 meanwhile would remain as "stable" for another year (or however long it is). Drupal 7 contrib would get a chance to catch up, customers get more value and time to live out of their Drupal investment. It would keep all the restrictions it now has, which are safe and will not cause breakage.

When Drupal 8 it is declared actually stable, then it is released for a second time as Stable, Drupal 6 goes EOL, and Drupal 7 becomes "oldstable". The default version that users get when they visit Drupal.org and download becomes Drupal 8. More rigid API freeze restrictions go into place.

I wrote a much longer emotional op-ed of these ideas today which ended up at http://groups.drupal.org/node/211383. I apologize to those who are reading these ideas for a third time.

This is the current purpose of “beta” releases of Drupal...

LoMo's picture

I like some aspects of your idea, but I think that this could really be as easily implemented by extending the “beta” period before actually making Drupal 8 the “stable / recommended” version. Perhaps core could be kept in beta until the most-needed contrib modules are also reasonably stable. When Dries announced that Drupal 8 would be “released, to wild, international fanfare” in August 2013, he did not indicate that would necessarily be the "dot-oh" version of Drupal 8. I think it makes some sense to prolong the beta period until all major bugs are resolved and until “contrib” has had some time to catch up, even if that means delaying the 1.0 release for several months after August 2013. IMHO, we should not call it 8.0 just because it’s the release date, but because the product is ready for stable use.

On the other hand, the community pressure to get contrib fully functional is reduced if the release is still considered “testing” (or “beta”) so its a double-edged sword. :-/

About me:
  • Drupal evangelist
  • (Former) regular author of the Cocomore Drupal Blog
  • Systems tester / QA automation, Kairion GmbH / freelance

OK well then how about extending beta to a year?

damien_vancouver's picture

@LoMo, OK, agreed that is the purpose of Beta releases right now. But they are way too short to be useful to most site builders, because contrib lags behind core a year or so.

What if Drupal Beta 3 had been followed by more betas right up until the present time? Would it really have been so bad if we were preparing a release candidate now, after 19 betas?

Fixing bugs that required breakage would certainly be less painful and more stuff would be fixed in D7.

Contrib would have had time to catch up - a good thing.

Drupal 8's release (or rather, Drupal 8's being declared "stable" and ready for end-use) would be pushed back that extra year. At first that doesn't sound like a good thing, but it would also have an extra year of time to get polished before the new release. There would be time to fix bugs AND add new functionality, because we are not in such a rush. We can actually have our cake and eat it too.

It's not a perfect solution, but one benefit is that we don't have to make major changes to bring this about. We just have to slow down and acknowledge that real world customers need contrib, and let contrib catch up.

Yes, D9 will take a year longer to come to market, but will be much stronger as a result. It will be less frustrating for everybody, from core developers to core bug fixers to contrib module authors to site builders to the customers who invest in Drupal infrastructure to end users who'll enjoy the benefits of a stabler, more polished release. Meanwhile we don't even have to make any actual sacrifices regarding D8... no one has to put their dreams on hold here. Other than dreams of a D9 speedy release, which no one should have yet so we can't be squashing them making this choice now.

damien_vancouver's picture

I suggested yesterday that we borrow from how Debian has its unstable and testing branches, and others answered that this is currently what we do with Beta. This isn't exactly the case, and I'd like to clarify a bit more what is different.

Our beta periods are a limited time test of the upcoming release only. They do not involve contrib or much real world use. In Debian, end users have a choice of which version to run. The vast majority choose to run the current stable release. But power-users or developers can choose to sport the testing release instead, which is active the entire time the stable release is. So rather than a beta per-se, this is more an extra layer of development branch which can be chosen instead of the main stable branch. It's around for the entire lifetime of stable, and people can upgrade at any time. Because of this, it gets properly tested in real world situations with real world use cases and kinks are worked out earlier, without as rigid change control or them impacting as many people. New fixes are still committed to the newest repo (unstable) and then from there make their way back into testing, until the next release when testing becomes stable.

I still think some form of this Debian model could work well for us, not involve major sacrifices from anyone, and reduce a lot of pain for new users who end up downloading a release way way way too early to get anything done with it. I don't think this is the same as just "extending the beta", though it's not really any different as far as the developers go. It's about consciously deciding to stabilize the combination of contrb+core and promoting that new branch as "testing" for the first year or whatever. It makes the beta period much more accessible to a lot more people.

Yes this plan would result in a longer support cycle for D6 contrib developers, and yes I agree that 6 years is too long! But when they move off D6, they should just stop supporting the old modules in that case. Mark it security fixes only. Resist the temptation to port any new functionality to it (which won't be hard once you have moved off that version yourself).

To finish off, here's the official description of how it works from the Debian Wiki: http://wiki.debian.org/DebianReleases :

At any given time, there is one stable release of Debian, which has the support of the Debian security team. When a new stable version is released, the security team will usually cover the previous version for a year or so, while they also cover the new/current version. Only stable is recommended for production use.

There are also two main development repositories unstable and testing which are continually updated during the development of the next stable release. The latest packages arrive in unstable (which always has the codename "Sid"). Packages are automatically copied from unstable to testing when they meet criteria such as lack of release-critical bugs, and dependencies being satisfied by other packages in testing.

End users should generally choose to run either stable or testing. Stable is recommended for applications requiring production-level stability and security (servers, firewalls etc) and is also recommended for those who are new to Linux. Testing is recommended for slightly more advanced users who want newer software on their desktops and who are capable of reporting and fixing bugs to help Debian.

Let's keep overall release

webchick's picture

Let's keep overall release management questions to http://groups.drupal.org/node/211383 or a similar discussion. This thread is about something very specific, which is about defining the class of patches can be backported from the development release of core to the stable release of core in order to see those changes in the wild sooner, and hopefully get more people involved in core development (and maybe also help some with backwards-compatibility woes too, but not as a primary goal).

Is now the time to propose a

fuzzy76's picture

Is now the time to propose a superstrict extra column in the table for current-2 version for security patches only?

That has already been

webchick's picture

That has already been proposed. :) http://groups.drupal.org/node/152484

Where?

fuzzy76's picture

I didn't see anywhere in that discussion that actually suggested supporting more major versions than we do today.

Regarding backward compatibility

Crell's picture

For those who haven't read it, here's some of the architectural issues we face for BC and backporting:

http://www.garfieldtech.com/blog/backward-compatibility

Thank you, Larry...

LoMo's picture

Yes (and I assume you mean me, since I brought up BC) ;-) . Although I did miss your post at first since I was out of town at the time and took a while to catch back up with the “moving drop”.

I appreciate you confirming what I assumed: that we are too far along with Drupal 8 development and the changes are too fundamental to even consider maintaining any kind of backward compatibility. But if it's possible, in future, to not functionally break every contrib module, e.g. when upgrading from Drupal 8 to Drupal 9 (or possible to build a new Drupal 9 site, soon after its release, which includes some Drupal 8 versions of modules which don’t yet have “stable/native-Drupal-9” versions, that would be a great step, IMHO, toward keeping the community-at-large happy (even if might cause some grumbling among the “old-school” Drupal core developers). As I see it, the cost of “baggage” (server power and storage) is much lower than it was back when this was first brought up and hopefully the new Symfony 2 / WSCCI changes will also lighten the baggage load. What has not become cheaper are good coding skills and user frustration. ;-)

BTW, I don’t think anything in your blog post relates specifically to backporting, per se, though I guess the two have more than a bit in common.

About me:
  • Drupal evangelist
  • (Former) regular author of the Cocomore Drupal Blog
  • Systems tester / QA automation, Kairion GmbH / freelance

Seems like there is fairly

webchick's picture

Seems like there is fairly significant pushback on the "Data structure change" aspect of this, so I've reverted to "Critical issue only" for now. However, I'd like to dig into this a bit more, to see if we can possibly find some middle ground.

For example, I just committed a patch last night at http://drupal.org/node/1260752 which fixed the maxlength on a field element (bug) and removed its size attribute, for consistency ("feature"). This will of course break any module out there that is doing an alter on that specific locale administration page, on that specific field, and that specific attribute. But the chances of that, frankly, seem quite absurd to me.

Also, take something like Improve UX for machine names for fields. Arguably, this is a straight-up bug fix, because we have a machine name field type and are not making use of it, and it's certainly a UX trainwreck every single time. But once again, for a module out there that is doing an alter on that specific locale administration page, on that specific field, and that specific attribute, things "could" break. But the trade-off seems justified to me.

Is it possible we could split "Data structure change" into two (or possibly more) sub-categories? For example, admin page changes are ok, but anything front-end facing is not? Or, maybe asked another way, could the folks who are pushing back on data structure changes find a few specific examples in the issue queue of "This is royally going to screw us" type patches so we can try and derive the list from that?

(Re)moving elements > changing property values

effulgentsia's picture

Is it possible we could split "Data structure change" into two (or possibly more) sub-categories?

I would order as follows (from most destabilizing to least):

  1. Removing an element from a form / render array. This also includes moving an element from one part of the array to another (e.g., into or out of a fieldset), because this kind of move is basically the same as removing + adding.
  2. Removing the value of or changing the value of #type or other high-ripple-effect properties.
  3. Changing the value of #description or other low-ripple-effect properties. I would personally include #weight and #options in this category, but maybe others would consider those to be high or medium ripple-effect.
  4. Adding an element to a form / render array, or adding a property to an existing element.

In a very quick skim of Improve UX for machine names for fields, it seems that's #2 from above list.

We may want to consider the above categories in conjunction with site-visitor-facing vs. admin-only pages. For example, maybe #2 should be reserved for critical-only issues if front-end, but loosened to major (or even normal?) if admin-only. I would personally consider #3 and #4 to be okay to backport for both front-end and admin-only areas. But that's just one opinion.

Retroactive answer

David_Rothstein's picture

Or, maybe asked another way, could the folks who are pushing back on data structure changes find a few specific examples in the issue queue of "This is royally going to screw us" type patches so we can try and derive the list from that?

This is a retroactive answer, but http://drupal.org/node/1346760 ("Add a scalable weight select element"). This changed the #type of the "weight" form element in some cases and seems like it may have caused some breakage in the Context module as a result, although I'm not sure anyone would have guessed or predicted that beforehand.

I think this is a good example of how these kinds of data structure changes can break things sometimes, just given the way Drupal works and how many complex modules there are out there.

Maybe there's just something

arianek's picture

Maybe there's just something I'm missing or misunderstanding as more of a non-developer, but isn't this more "loose" guideline set a lot more fuzzy?

D7 freezes were incredibly frustrating - some teams (like Docs) were made to stick completely to the deadlines and complete work super early, while others who weren't under formal exceptions continued making changes for nearly an extra year! It was incredibly unclear where we stood as far as pushing back on the deadlines so we could continue working on things.

I don't feel like that was ever addressed seriously, and the new proposal seems like it's going to make all these deadlines even fuzzier... in concrete terms, how will people know when they really really need to abide the freezes?

I was going to reply to this

catch's picture

I was going to reply to this to say these guidelines are about what can be backported to the already released Drupal 7, not what can go in during 8.x code/feature freeze. Athough thinking about it, it would naturally affect the Drupal 8 code freeze guidelines as well (if something can be committed to 7.x, then why not 8.x), so this brings up a good point.

It's definitely the case that even now there are patches being committed to 7.x, that were bumped to 8.x during Drupal 7 code freeze. See http://drupal.org/node/765860#comment-5275470 vs. http://drupal.org/node/765860#comment-2965850. That is really more to do with the extreme pain of trying to get Drupal 7 out rather than which patches should and shouldn't have gone in at the time (IMO that one should have gone in at the time, and we got lucky with the backport over a year later in that no-one ran into the API change that we know of).

For the actual feature/code freeze, I opened http://drupal.org/node/1410544 to try to find an alternative to lots and lots of one-off exceptions and bargaining.

We'd still have the problem of figuring out which patches are and aren't disruptive - webchick's example of changing minor details on a locale admin form is a good one - compared to changing something on the node form, I've also snuck API changes into 7.x for APIs I'm 99.9% certain that no-one is using yet outside core to fix major bugs. However that's looking at the severity/impact of a patch rather than the class of it (i.e. not whether it's an API change or a feature addition, but will anyone be adversely affected by it).

Yay, thanks for the super

arianek's picture

Yay, thanks for the super relevant response! :)

FWIW, I haven't even contemplated working things to backport to D7, since as far as I understand none of the sort of docs/interface like changes that I might potentially work on would even be considered (and I haven't delved into D8 yet to work on things that wouldn't be for the purpose of being backported). (And I don't think this will change significantly?)

Obviously this kind of work was restricted because of trying to get D7 out the door, but it was silly and frustrating, since so many of those patches would've broken nothing (aside from needing translation updates), and really improved some of the small details that contribute to ease of use for the end users. So I'm really glad to see some loosening around that.

That said, I do (still) think that this proposal is kind of complex (or will be once we get that far down the line) and might lead to the same kind of mess as last time round. But, the issue you filed (Consider possible ways to shorten code freeze") is super awesome and exciting to me, as I think it brings us much closer to working in a more agile way where we're in a more ready-to-launch state when we get close to the end of the cycle, but there's still some built-in flexibility that applies across the board!

And now I'm gonna run over there and post a follow up comment!

Yeah, during the D7 code

webchick's picture

Yeah, during the D7 code freeze it became clear after awhile that unless the hammer was laid down on "feature-ish" improvements, bugs were never, ever going to get fixed and we would never, ever release. That hammer was actually initially laid down across the board, but it probably felt fairly arbitrary because in practice, we kept finding edge case after edge case during the course of hammering down on critical bugs that meant additional UI changes and expanding APIs beyond their pre-freeze capabilities. Catch's proposal is interesting because it provides constant incentive for fixing bugs, using the carrot of "if you help fix enough major/critical bugs, your feature could still get in." Much better approach than the stick, at least in theory, though I do worry about the impact on contrib if APIs are never really stable (but I'll take my comment over there :)).

What this proposal would mean in practice though for folks like the documentation team is that essentially we'd never shut off the ability to tweak help text or any other administrator-facing strings. If we came up with a better help system in Drupal 8, we could potentially backport it to Drupal 7 as an optional module so site builders could start using it a few months from now, as opposed to a few years from now. If explaining some convoluted part of the admin interface (like the Clean URLs page before 7.9) takes 50GB of text and 11 screenshots, we would no longer be stuck with that behaviour for 5+ years, we could simply just fix it, and make it more intuitive for people. All of which I think are good things.

The downside for the Documentation team would be that the Usability team could also potentially do things like make cosmetic changes to the installer, subtle tweaking of administration pages, adding new checkboxes here or there (or more maybe removing them ;D), etc. which could potentially invalidate written-down instructions (also books, videos) in minor ways. We could track these in http://drupal.org/list-changes for the benefit of the docs team staying on top of them. OTOH, now that docs are community-maintained, I'm not sure if this is as much of a burden on the team as it once might've been? But it would definitely be something to look at once the "official" docs initiative gets off the ground.

Timing?

Crell's picture

For string changes, the main concern is translation teams, right? What if we say that string changes are only allowed during the first week after a release? That way, translations teams know they have ~3 weeks to update anything that needs updating. If a string change misses that window, eh, it's only another month before it's legal again. Basically we unfreeze strings at the beginning of each point release to give translators as large a window as possible.

That is a brilliant idea

tsvenson's picture

That is a brilliant idea @Crell.

Actually I don't believe it is so much about missing the window as it is planning for the next. If your proposal, which I hope, becomes the rule then it is very easy to create a good workflow for all parties involved around not only committing string changes, but also pull in the translation phase in it.

Big +1 from me.

If this works out, it might be possible to apply the same model on other things as well.

--
/thomas
T: @tsvenson | S: tsvenson.com

That seems pretty reasonable

arianek's picture

That seems pretty reasonable to me on the docs/UI strings side of things, but I can't say how hard it will be to keep up on the translations side with the existing infrastructure.

What Jennifer (jhodgdon) and I had vaguely discussed around the Better Help System (for D8) and "Curated" or official docs (which would work together and totally change how "official" online and built-in project documentation is done for core and contrib - making it more structured and easier to maintain) was that in the future, it could be integrated with the translation system so that we could have translation sets be managed just like module updates in D7. Where we now see in Update Manager that we have new module updates awaiting, we'd see in our Translations Manager that we have new updates pending to our translations.

Ideally in the end it would be a smooth workflow where any code changes could be flagged for the per project documentation teams to be notified. And once they'd reviewed/changed (or if the change started as a docs change), the docs people would flag it as needing review for translation teams so they would get notified. And then if/when translations were done for a needed language, then they would flag it and push a notification into all the Drupal sites' Translation Managers notifying site admins to update their translations.

That's kind of just the pie in the sky brainstorm end-picture, but I think that what @crell's proposing would certainly fit well, starting with that plan of smaller windows/cycles for string changes.

ps. I ought to mention, work

arianek's picture

ps. I ought to mention, work on both those issues are on hold after the struggle we had with rolling out the community docs changes. I think (hope?) once there's some sign that the D.o infrastructure work will be easier, work on these will resume.

New hooks can nameclash.

donquixote's picture

webchick (original post)

API addition
Optional parameters on a function/hook, new hooks/helper functions. Would provide new capabilities which contributed modules could choose to rely on.

I think this will help a lot.
If we can not backport a feature, then at least we want to allow contrib to provide an equivalent. In many cases, this does require additions to the API, or these additions will make the task a lot easier.

New hooks and API functions are totally safe, except for: Nameclashes!

For newly introduced hooks, that would be:
1) An already existing function in a contrib module, which was not meant to implement any hook, does now suddenly become a hook implementation.
2) There is already a hook with that name in a contrib or custom module, probably with a different signature and purpose.
3) The hook is a prefix of an existing hook (invented in contrib or custom), or an existing hook is a prefix of the newly introduced one.

Btw, this problem is not with core alone, it also applies to any contrib module that wants to invent a new hook.
Also, any newly introduced function can result in a nameclash - not just hooks.


The solution is to use namespaces for hooks and procedural, as we already want to do for classes, and naming conventions that disallow any cross-module namespace pollution. If we have these in place (in D8, hopefully), we have one less reason for API freeze.

Namespaces for hooks and procedural have been proposed in several issues on d.o., with quite reasonable patterns being suggested.
jbrown, http://bluedroplet.com/blog/proposal-fixing-php-namespacing-drupal-8 (related issue on d.o. somewhere)
-> some time ago, and imo that was quite reasonable.
chx, http://drupal.org/node/548470#comment-5430732
-> not happy. Too many brackets, for my taste. Also, this thread has grown hopelessly long, and changed its topic too often. A lot of the older posts are now obsolete.
myself, http://drupal.org/node/1428592
-> fresh thread, no obsolete legacy posts, summary to distinguish from other proposals. Imo, this is the reasonable place to discuss this.

We've added a dozen or so

webchick's picture

We've added a dozen or so hooks to core since Drupal 7's release, and this has never once come up, which it typically doesn't as long as you're naming hooks well. I personally think that the namespace collision concern is grossly overblown, and it depresses me seeing how many very smart peoples' energy get wrapped up in these debates. :\

I personally think that the

donquixote's picture

I personally think that the namespace collision concern is grossly overblown

I have absolutely no data to challenge that.
I would imagine that most of the time we do not even notice a magic name overlap. There are so many bugs reported for core and contrib, that it would not surprise me if some of them are due to an unnoticed naming collision.
But then again, at least some of these collisions should have made noise. If in your experience that has not happened, then probably this is quite a decent indicator that it is less of a problem.
It just makes me nervous any time I want to invent a new module, function or hook name.

Older discussions

donquixote's picture

For anyone interested, here are some older discussions that I opened years ago. Maybe some of this is totally naive, and I would not suggest the exact same thing today. But maybe some of it does contain some useful ideas.
Introduction of new hooks in stable Drupal releases should be allowed and encouraged.
No more feature freeze for major Drupal releases (that is probably the super naive one)
Ping pong release cycles (for Drupal core)

Things to take away:
- If we can solve the nameclash problem (see above), then new hooks can solve a lot of problems. especially, many features can be "backported" in contrib, instead of messing with core.
- If we allow API extensions post release, we can design these hooks or API stuff on demand, instead of having to guess API needs before anyone is using it.
- API extensions do not always need to be backports. The problem we are trying to solve with a newly introduced flex point in D7 could be already solved in a better, but more complex way in D8.
- We could do releases between major versions, which would introduce new stuff, but would not break the API.

I think we're already doing

webchick's picture

I think we're already doing all of those things, as far as APIs are concerned. For example, Drupal 7.8 added hook_widget_form_alter(), Drupal 7.12 provides the ability to do master/master replication from DBTNG, and so on. This move has been really positively received by contributed module authors, because it means if they run into limitations, a fix is potentially only a few weeks out.

However, with the current policy on string, UI, and data structure freezes, our user-facing usability problems are trapped, frozen in time, for 5+ years. And without the ability to backport other features, such as new themes and modules, Drupal 7's out of the box capabilities, as far as end user are concerned, remain frozen in time for 5+ years.

This proposal is mainly about giving end users and site builders the same constantly evolving platform that module developers have benefited from since Drupal 7's release.

tsvenson's picture

@webchick: Just looked at http://api.drupal.org/api/drupal/modules!field!field.api.php/function/hook_field_widget_form_alter/7 and only the comment from Dave Reid says it requires 7.8.

api.drupal.org really should include a history section in the content body listing things such as what version it was included in and changes made down the line.

This information will especially be valuable for adding the core requirement for modules. In this case, a module using this hook should specify dependency on 7.8, not 7.x.

Comments is a bad place to to rely on for this, especially if its not the first comment.

--
/thomas
T: @tsvenson | S: tsvenson.com

That site is powered by the

webchick's picture

That site is powered by the API module, so please go and leave a feature request over there.

Done, see

tsvenson's picture

Done, see http://drupal.org/node/1456460. Feel free to put your weight behind it :)

--
/thomas
T: @tsvenson | S: tsvenson.com

If this is the case, then I

donquixote's picture

If this is the case, then I am all happy :)
I had some weak memory of situations where requests like this would be answered by "feature freeze, no chance". Maybe that was all for D6.
What I can currently think of, is secure password hashing in D6, making it easier for contrib to do something other than md5 (without the form_alter hack). This would probably require a bit more of code restructuring, not just adding a hook. So it was probably reasonable to deny that feature request for D6.

And I still support the rest of what you suggest in the topic starter.

Let's make this easier in D8

effulgentsia's picture

This thread should stay focused on D7, and we have some constraints we need to work within for that. But, while D8 is still in code thaw, there's things we can do to make D9->D8 backports smoother. Please share feedback about that in that issue, not here. Thanks!

Thanks all for the great

webchick's picture

Thanks all for the great discussion here. I've moved this table, plus a few notes from the discussion, over to http://drupal.org/node/1527558.

Just a heads-up for anyone

David_Rothstein's picture

Just a heads-up for anyone who's interested that I'm going to revive some of this discussion at CapitalCamp this weekend:

http://capitalcamp.org/content/drupals-backport-policy-how-enjoy-future-...

Translations

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week