What time should security releases happen? Can we pre-release? Can we work with WAF vendors?

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
greggles's picture

In an ideal world, what is the best time for a security release to happen?

Sometimes the security team doesn't have control because a project maintainer commits and makes the release node at a specific time. We can, of course, try to make it more clear that they need to commit and make releases before a specific time.

And often there is some control.

So, in our ideal world, what time would people want it to be released?

Can we do something to pre-release in different parts of the world?

Comments

I don't think there's a good

catch's picture

I don't think there's a good time - someone is always going to either be at the end of their day or already asleep. You can base it on perceived numbers/size of sites in various regions, but then it reinforces US/Euro-centrism.

Some kind of limited pre-release (especially for people in timezones where the release will happen outside 9-1pm (nice to give people time to actually patch and release, then monitor, then be able to eat dinner, after lunch is really the cut-off for convenience) seems the only way to deal with this.

Then this brings up who can you pre-release to. That's a much harder question but very quick ideas:

  1. Organisations that use Drupal for internal sites and sign a pre-release agreement that they won't disclose patches to third parties.

  2. Drupal consulting shops that sign a pre-release agreement that includes them also having an agreement with their clients that they won't disclose patches to third parties.

For this I'm only talking about 12 hours advance notice or similar to account for timezone issues.

With this, there's much more chance of a vulnerability being publicly disclosed prior to the official release due to a leak from somewhere.

However, if a vulnerability is disclosed a maximum of 12 hours prior to the official SA going out, how much worse is that than the SA going out when you're about to go to bed, then you waking up 9 hours later to a hacked site anyway?

We could always bring an SA release forwards if there's a leak, but you can't retrospectively warn people in advance.

I think the pre-release is a

mpdonadio's picture

I think the pre-release is a good idea. Besides my own servers and ones that I had direct access to, I had to draft instructions for clients who I don't have ssh access to so they could patch, have project managers send these off, then field calls / etc as people had to patch their own systems. That was the most time consuming part of my day.

I also had to notify IT directors at large organizations about the problem. I know one of them has hundreds of sites in-house, and they could have used the lead time to help plan.

I had to draft instructions

scor's picture

I had to draft instructions for clients who I don't have ssh access to so they could patch

This kind of preparation doesn't have to wait for the official SA to be out. Patching instructions don't depend on the nature of the SA. There will be patching to do for any SA. The only difference in the case of critical ones is that they have to be done asap. Patching instructions can easily be prepared in advance and all you have to do is to paste the link to the SA or the patch to apply. The Drupal Security team tried to give lead time in the hope that people would plan to be available to patch their sites.

I agree that time of day is

davidhernandez's picture

I agree that time of day is almost irrelevant due to the global nature of our community; however, day of the week is not irrelevant. Since the releases are already on a Wednesday, I don't have much to add. Mid-week is pretty standard update timing.

I would raise concerns about the pre-release idea, if it wasn't available to everyone. (with the signing of an agreement) I'm not sure we should get into the situation of determining what size organization, purpose, etc, determines you are worthy of privileged access.

We are talking about a real paper, signed, legal document managed by the DA's legal counsel, and not just an online form with a checkbox, correct? I assume that is something the DA would have to manage, and they may not be willing to take on that burden, and any implied liability. Especially, if they have to verify that shops are getting non-disclosure agreements from all their clients. Holly, or someone, would have to comment on how that might work, but I'm imagining a quagmire here. Or maybe I'm just overcomplicating it.

The legitimacy of the process would naturally limit the number of people willing to get added to the list, but it could still grow to a size that is difficult to manage. Would the security team bother trying to track down leaks if there are 500 people on the list?

This is all complete conjecture on my part though. I would instead ask, is there an existing model that exists for doing this? Do any other organizations, particularly open source ones, have a fair pre-release model that works, and doesn't bring along some costly administrative overhead?

I think it would be tough to

nmcclain's picture

I think it would be tough to align the elitism (and business advantage) of a select early notification group with the community's values.

I propose we'd get more "bang for our buck" focusing on a better way of pre-communicating risk / scope-of-impact / steps-to-remediate to the WHOLE community. Without disclosing the exact vulnerability, we should be able to provide super-clear guidance about WHO is impacted, WHAT the risk is, and HOW to mitigate (typically this will just be patching). I hate to say it, but Microsoft does a pretty good job of this - long before "patch Tuesday", they have released details about what products/versions are impacted and exact counts of critical/high/medium risk patches.

I guess I'd argue that we could meet MOST of the goals of an elite early release without actually disclosing the vulnerability. The exception is probably WAF vendors and large-scale hosting platforms, who could develop product-specific protections with advanced notice. I think it's fair to expect these organizations to contribute time and expertise to the Drupal Security Team, and to comply with those disclosure rules (which would still allow for early development - but not distribution - of WAF rules).

Pantheon Apparently has Pre-Release

Justin_KleinKeane's picture

I commented on Pantheon's site about their pre-response to Drupalgeddon (https://www.getpantheon.com/blog/what-we-are-seeing-drupal-sa-2014-005). Basically it looks like they took knowledge from the Security Team and proactively protected their own infrastructure before anyone outside the Security Team or the original reporter knew there was a problem.

Sorry if this post is choppy, but I'm sort of busy dealing with the fall out from this incident (because sadly I didn't know beforehand or pre-allocate resources to patch immediately after the SA arrived).

If what's posted on the Pantheon blog is true, then the Drupal Security Team misrepresented the confidentiality of security issues. Researchers should know that by reporting issues they reach a selected commercial audience with preference. This gives competitive advantage to those entities, which is decidedly un-open source and violates the terms on the Security Team page that says: "Do not share your knowledge of security issues with others."

The Drupal Security Team isn't an open group, you have to be invited. What incentive will there be for independent researchers, like myself, to report to Drupal Security Team, knowing that the reports will be preferentially disclosed to certain commercial entities prior to public disclosure or patch? It's all well and good that Pantheon was protected from the tidal wave of SQLi attacks, but what about the rest of us?

Here's my back of the napkin proposal. When a critical vuln is discovered everyone is given a heads up that on X date a critical vuln advisory will be released. This allows organizations to all prepare equally to roll out a patch on time without having to wonder if each Wednesday Drupalgeddon will drop. Currently, the time on advisories isn't even standard so you have to watch the announcements all day long. Switch to a specific time, say noon Eastern or something, and release reports then.

The Security Team needs to communicate that they don't give preference to any entity, commercial or otherwise, and that any vulnerability reports will be kept strictly confidential until a release is available. This would allay fears that by reporting an issue researchers are providing preferential commercial advantage to specific entities.

At the very least, if the Security Team does share 0-day info pre-SA, be up front about which companies will get the 0-day notice so it's clear that reporting an issue benefits Pantheon, say, but not RackSpace. After all, researchers have a stock portfolio to watch out for too.

A few quick thoughts. Release

greggles's picture

A few quick thoughts.

Release windows are announced in the core group and this one was described at https://groups.drupal.org/node/445893 as definitely happening (usually there's only an announcement of a window).

We are discussing noon (UTC-5) for at least core releases, but for contribs we have a bit less control. Do you think Noon eastern makes sense for the whole world? If so it would be good to know why/why not.

The list of team members is on https://security.drupal.org/team-members and clearly lists not just the person but also the company they work for. My sense is that, like almost everything in Drupal, being involved and helping to fix things pays you back with increased knowledge (among other things). The security team is a bit of a different situation since the information is confidential and usually embargoed. When you find issues do you discuss them with your team and/or the site-builders who asked you to review the module? How far do you share that information? What do you think is an appropriate limit?

The Drupal Security Team isn't an open group, you have to be invited.

This is only partially true. Applicants are vetted and I imagine you agree with that, but we invite the whole world all the time ;) For example in the join page. Or multiple times inside this podcast.

Scheduling releases

Justin_KleinKeane's picture

I think any arbitrary time for a release is fine. It might not be convenient for some folks, but the current system certainly isn't convenient for anyone. If the current release schedule is meant to provide a rolling time window for updates to make it easier for some people some of the time why not formalize it? Start with midnight UTC on Wednesday one week, then advance an hour until you cycle back to midnight UTC. Just publish the schedule and folks know. It's going to be inconvenient for some folks some weeks, but at least they'll know in advance.

Information sharing

Justin_KleinKeane's picture

When you find issues do you discuss them with your team and/or the site-builders who asked you to review the module? How far do you share that information? What do you think is an appropriate limit?

I've thought about this a lot. It seems to me that the Drupal Security Team has a few rules, and folks get upset when you don't adhere to them. While I may not agree that we're all bound by an explicit set of rules that are outlined in the Security Team process, it seems to me that if the Team takes the trouble to outline a set of guidelines they should be reliably adhered to. I think it may be unreasonable to expect that because a team member's work affiliation is public that there's then some implicit rule that they can share info with that company. The mix of some implicit understanding with an explicit guideline about how information is to be shared is duplicitous at best. If there is an implicit understanding, why not then make that explicit so it's transparent? Trust in a closed security process hinges on transparency and accountability, and inferences and implicit rules harm that trust.

When I find a vulnerability in a review of a module (or core) I tell my group, or whoever requested the review, that a vulnerability exists and I'm working with Drupal Security to resolve the issue. I don't tell them any more or less than that. When I've made disclosures in the past relations with the Drupal Security Team have become unpleasant very quickly (https://lists.drupal.org/pipermail/development/2009-May/032912.html). So it seems like there's a double standard that puts some folks in a privileged position, and those folks are a closed group (not just the Security Team, researchers as well). If you're going to expect researchers to keep discoveries confidential until a fix you must hold the Security Team to the same standard.

The fact that this SA has had real financial implications cannot escape this discussion. Certain organizations are paying a lot of money to clean up breaches. The organizations with advanced notice were spared that cost. If the behavior of the Security Team is going to result in financial advantage for their employers the Team should really be explicit about that so that I can short some stocks and buy others before I hand a vulnerability discovery over to the team.

While it is true anyone can ask to join the Security Team the vetting process is somewhat opaque. I say that the team is not open because when I asked to join I was told I couldn't. It's anecdotal, I know, but that's the best evidence I have for the assertion that the Security Team is not an open group.

When I've made disclosures in

scor's picture

When I've made disclosures in the past relations with the Drupal Security Team have become unpleasant very quickly

I believe those were not responsible disclosures? We prefer responsible disclosures. As far as I remember, you've adhered to the responsible disclosure program in the last few years.

While it is true anyone can ask to join the Security Team the vetting process is somewhat opaque. I say that the team is not open because when I asked to join I was told I couldn't.

I agree it used to be subjective and opaque many years ago. We've tried to streamline this process with the provisional team member program. When did you apply to join the team? I don't recall seeing any formal application from you.

The organizations with advanced notice were spared that cost.

There was a gdo announcement many days in advance and people talked about it on twitter, but we (Drupal security team) admit that we could have been more explicit about critical level of this SA. We are working on ways to enhance our communications. Note that we always recommend people to patch their site on Wednesdays as soon as the SA is announced. Sites that patched immediately or in the next few hours should be safe from this issue. (We've noticed the first attacks ~7h after the noon EST announcement).

The severity of this SA meant

pwolanin's picture

The severity of this SA meant that some companies (like Acquia here I work) took measures that we have never before for any SA because of the ease and consequences of exploits.

Usually, the most advance information I share with my colleagues might be a few hours ahead of time a definite yes or no about there being a release (but not the details) for core or a particular module.

The boring and stupid truth is that most weeks it's not clear until that close (hours) whether a given SA will actually happen since it needs both the patch finished and tested and maintainers being available.

The security team members are volunteers and while in rare cases the advance information might help my employer, in general it doesn't, and in general I don't get work time for working on security issues.

As I understand it, the

ianthomas_uk's picture

As I understand it, the security team didn't reach out to Pantheon and the other hosts, they found out because they have staff who are members of the security team - they got the benefit because of the contribution they are making to the community.

Those staff did presumably share it with more people than they would normally, which may even have broken confidentiality agreements, but doing so had big benefits for some in the Drupal community and very little risk.

doing so had big benefits for

greg.harvey's picture

doing so had big benefits for some in the Drupal community and very little risk.

Sorry, but I can't agree with that. Doing so leaked confidential information about a very serious exploit to an unknown group of people, without giving the pre-determined and trusted group of people time to do their job and provide a mitigation to the exploit.

I don't see a silver lining.

I think part of the issue is

davidhernandez's picture

I think part of the issue is the language in https://groups.drupal.org/node/445893 did not convey the severity of the SA. That is also assuming people are subscribed to the GDO group. An email sent to the security mailing list would have been good.

Many people who are not in the loop were caught off guard by how bad this exploit was, and how quickly they were expected to deal with it. People are not sitting at their computers waiting for SAs; in a position to drop everything and upgrade all their sites at a moments notice. If there was a pre-announcement saying something really severe is coming, and you should be prepared to deal with it immediately, people could've at least prepared themselves. Some people have to allot time in their schedules to deal with these things.

I think most people took this like any other SA/release and chose to deal with it the way they normally would, which is to upgrade when it's convenient for them. But then twitter, blogs, news, etc, lit up like crazy about how bad this was, and it caused everyone to scramble.

So you'd agree with Ned's

greggles's picture

So you'd agree with Ned's proposal, then? One thing he didn't address was the channel for sharing that information. It sounds like your proposal is via email to the security list?

My sense is that could work well, though I'm not sure about pre-announcing on the list and if that would really help. I think the biggest problem there is that often it's not clear what the final set of patches to go out will be until the Tuesday or Wednesday itself. We've tried several times to tighten that up, but so far it hasn't been super successful.

Mostly. So, we're also

davidhernandez's picture

Mostly. So, we're also talking about two scenarios; one is the regular release cycle, the other is the infrequent, highly critical vulnerabilities, like we had recently. I wouldn't have pre-announcements for everything, at least not to the mailing list, because people will start ignoring them. Critical pre-announcements need to be unusual enough that they attract attention. It would be great if those were sent to the mailing list.

I'm also thinking about what channels are most likely to reach the larger community. Mailing lists are pretty common for this type of thing, but for something critical try every channel available. There is the mailing list, twitter, gdo, planet, word of mouth, news media, etc. The media is less likely to run with a pre-announcement, but there are lots of other options.

Do you mean it isn't clear whether it will be contrib module A versus contrib module B in the announcement? I think those being released the way they currently are is fine, with no pre-announcement, though I don't know the last time (if ever) there was a highly critical contrib announcement. For core, even a one day heads up that a critical is coming would be helpful. For this last release there was no sense of the "OH MY GOD WE'RE ALL GUNNA DIE!" nature of the vulnerability, but that is what we got after it was out. From an IT infrastructure/deployment stand point, a pre-announcement gives you time to make preparations; move meetings, delay deployments, make staff available, etc, so you don't have to scramble to react.

mailing list, twitter, gdo,

greggles's picture

mailing list, twitter, gdo, planet, word of mouth, news media

We did twitter, gdo, planet and word of mouth. As you say, news media isn't likely to cover it. We have started the formation of a "security-press" team and I've at least asked if people think that would get picked up. So, I think the key difference you're suggesting on channels is to add the mailing list as a pre-announcement when we know it's a highly critical issue.

Regarding content: I certainly saw a lot of people get the right sense of the issue, but agreed it wasn't everyone. So that seems like it takes us back to exactly how we should change the content and, I guess, Ned's proposal that we start stating:

WHO is impacted, WHAT the risk is, and HOW to mitigate

I think it could help to speak in more specifics using the last release as an example. If we follow that guideline we would say:

All Drupal 7 sites are impacted. The vulnerability would allow an attacker to completely take over a site. Sites should mitigate the issue by upgrading Drupal core or applying a patch.

Thoughts?

Yes, getting the word out

davidhernandez's picture

Yes, getting the word out after the release was mostly handled fine. I'm referring to a pre-announcement, which could also happen on twitter, et al. That would have the biggest impact on conveying the gravity of the impending announcement.

WHO is impacted, WHAT the risk is, and HOW to mitigate

Yes, I agree with this. I would keep the language clear and concise, requiring as little foreknowledge as possible. I'm trying to find an example of a Microsoft announcement that uses this format, but I haven't found one. Does anyone have a good example? I only see the online advisories, like this one https://technet.microsoft.com/en-us/library/security/3010060.aspx , which is very verbose. (I'm not subscribed to any Microsoft mailing lists.)

Personally, I would avoid paragraphs, as they are harder to scan, visually. It also lumps the information together.

Who is impacted: Everyone using Drupal 7/Every site using Drupal 7/All version of Drupal 7 (just examples)
What is the risk: The entire website/Complete takeover of the website by an attacker
How to mitigate: Upgrade to Drupal 7.32 immediately or apply the supplied patch

Just to be sure, we did

greggles's picture

Just to be sure, we did pre-announcement via twitter, gdo, planet and word of mouth.

  • @drupalsecurity on oct 10 and oct 14 which included a release window - there were also tweets/retweets from team members and @drupal and similar. We haven't historically announced a release window on twitter except for the previous one where it was on a different date due to coordination with the WordPress release.
  • g.d.o which gets syndicated to planet - again, typically it's just a window and not a certainty
  • word of mouth...well... hard to link to ;)

I think the trickiest part of the bullets is the "Risk" one - even a XSS issue that requires "administer foo" access can technically lead to "complete takeover of the website." But we have those all the time and the mitigating factors often make them less important. I think we should probably rely on our new criticality scale which is a somewhat context-less number, but over time people will come to understand "20/25 is an UPGRADE NOW" situation while "5/25" might be a "Meh" depending on your config.

There is nothing in those

davidhernandez's picture

There is nothing in those tweets/posts that communicates severity, which is what I'm looking for. In fact, quite the opposite. They read rather run-of-the-mill. I'm not suggesting you do anything radically different, but a slight change of wording could better communicate the importance.

I agree with you that the Risk component is difficult to communicate. What amount/type of information is the Security Team comfortable releasing? Could a pre-announcement include the type of vulnerability and indication of mitigating factors? For example, "Tomorrow, we will be announcing a highly critical (20/25) SQL injection vulnerability with no mitigating factors." Is that too much information?

Yes, it is. The fear was that

Fabianx's picture

Yes, it is. The fear was that hackers would try to find the exploit before the release of the patch.

If you warn too much, you might get a 0-day, which would be even worse as then no-one has patched and all sites are hacked.

And given how long it took me and some others after the patch was released to find attack vectors that was a good decision.

However immediately after the bad guys "got it:, the severity should have been made higher again and not just 14 days later.

Obviously this whole issue

greg.harvey's picture

Obviously this whole issue got reopened today by the BBC deciding to feature this issue on the front page of their tech section (facepalm!) - I just wanted to pick up on some of the earlier points.

I also think pre-release is a bad idea. I totally agree with Ned and David above, that pre-warning could better indicate severity, but pre-releasing details? Nah.

I'm not sure the detail in David's last post is necessary - perhaps too much specific info is potentially giving "baddies" too much time and a specific target - "highly critical" in flashing lights and an ETA for the patch ought to be enough to make sure everyone's ready and taking it seriously.

I appreciate the people saying the reaction was stressful and painful, these things always are, and we were ready because someone on our team is also on the Security Team, and he pointed us at the Drupal Groups post (I would stress he expressly didn't tell us the nature of the bug, he simply told us it was bad, it would break and we should be ready). So we were ready, even though the post itself didn't transmit the scariness and alert level it ought to have done - we'd had that transmitted to us another way.

If the communications were just that little bit clearer and we really made sure there's no excuse for not being aware of them if you're a Drupal user, then I think things would've been better for the masses.

What scares me most about the latest incident though, is that clearly very sensitive information leaked out of the Security Team into large organisations, who prepared responses and "shields" for platforms, and so on. Which means that information was no longer under any kind of known control or classification. What happens when people leak that sort of info? Is there a policy for removing people who break the rules? There ought to be a policy - this is pretty serious stuff.

Change in tone from initial advisory to PSA

seanburlington's picture

The initial advisory was "Highly Critical" but still only rated 20/25 - there wasn't anything in there to signify the extreme criticality of this issue.

Those of us who weren't forewarned did not know to work out of hours to fix it.

We needed to take time to review the issue and demonstrate the urgency to allow a patch faster than usual change control policies would allow.

Still we didn't manage to patch with 7 hours.

Now, long after we are patched and secure our client sees an extremely alarming post, and we have to take more time to re-evaluate and reassure.

Sean Burlington
www.practicalweb.co.uk
London

SA Details in 'available update' alert?

brylie's picture

Hi all,
Can the Drupal "security updates available..." alert display information about security advisories? E.g. :
* Threat level
* Link to specific details

Brylie Christopher Oxley

Outstanding tasks

nedjo's picture

Judging from the lack of edits to the security team page on Drupal.org, it looks like there is still an outstanding need for concrete followup on this important discussion.

A main issue that several commenters raised is a big gap between the publicly communicated policy of the security team and actual practice, at least in the case of SA-CORE-2014-005.

The day after SA-CORE-2014-005 was posted, security team leader Michael Hess said that while it was "certainly okay to give their groups a heads up", security team members who learned in advance about a vulnerability were "not to share the details ... that this was this type of vulnerability and here's how we fixed it."

Yet specific details of the vulnerability are exactly the information that was shared with insider companies.

Yes, company names are listed alongside the names of security team members. But does this somehow convey that they are cleared to share confidential information with their employers?

Whatever the motivation of security team members in sharing confidential information, doing so had clear and significant commercial benefit. One blogger noted the "good news" that "we host most of our sites on Pantheon," a company that "had advance notice of the security announcement."

My feeling is:

  • It's important to hear some direct acknowledgement from the security team of the validity of issues raised by community members around the handling of conflicts of interest in relation to privileged information. Simply sidestepping these issues isn't going to increase community confidence.
  • Response to SA-CORE-2014-005 has effectively created a pre-notification system, but one that is unstructured, opaque, and not apparently based in consideration of how to provide the best protection to Drupal users.
  • Followup should include review and rewriting of the security team page on Drupal.org and other publicly accessible documentation to develop new policy or, at least, bring it in line with current policy. Potentially relevant ISO standards: ISO/IEC 30111:2013, ISO/IEC 29147:2014.

Hello Nedjo, We are still

mlhess's picture

Hello Nedjo,

We are still working on the policy changes. We plan to hold a private session and a public session (BOF) at DrupalCon about this exact topic. We want to make sure we get this right, and don't build a complex system where one is not needed and can not be maintained.
One of the other team members said you might be a good person to include in helping with the draft, if you are interested in this, please reach out to me via my contact form.

I think the people in this

greggles's picture

I think the people in this thread will be interested about the BOF event at Drupalcon LA. If you're going to be in LA, please come and let's discuss all these topics and more, to the extent we can :)

Details at https://groups.drupal.org/node/468388

DrupalCon LA

Justin Freeman's picture

I'd love to go to LA and discuss this at the BOF. But that's not on the cards unfortunately.

Can someone report back to this thread on the main points discussed in the BOF?

I think this issue is very important to a wide audience.

Agileware, Australian Drupal Developers
http://agileware.com.au

Yep, I'll do my best to take

greggles's picture

Yep, I'll do my best to take notes and post them (or see that someone else does).

That would be great, thanks

Justin Freeman's picture

That would be great, thanks G!

Agileware, Australian Drupal Developers
http://agileware.com.au

Issues to discuss

David_Rothstein's picture

I won't be at DrupalCon, so I wanted to add some thoughts here instead, on the following issues:

  1. Selective pre-disclosure of security issues to particular organizations (what happened in SA-CORE-2014-005, what should the policy be in the future)

    To move forward I think the community first needs more information about what happened with SA-CORE-2014-005. Many of us on the security team neither predisclosed anything to anyone, nor patched client sites before the advisory came out. For those who did, two things aren't clear to me:

    • Who outside the security team was told about the vulnerability, and how much were they told?
    • What kind of coordination was there around this? I don't think the broader security team was ever consulted about the pre-disclosures; it happened unofficially, outside the team.

    I think a policy going forward has to be very clear that early access to private security information is not intended to be a "benefit" a company gets when its employees are on the security team.

    And people outside the team should generally not get access at all. If they do, it needs to be based on a clear reason, and carefully coordinated among all members of the security team (not just a few members and a few companies).

    For example, we do currently give trusted non-team-members early access if they're helping solve a particular issue, or if they maintain a distribution based on Drupal or other open-source project that shares some code with Drupal and that is vulnerable to the same security issue, etc. But that's a specific policy that's intended to benefit the overall userbase of Drupal and other open source projects.

  2. Pre-announcement of security releases to everyone (how to do it, what information to disclose)

    This is a tough one.

    The intention of https://groups.drupal.org/node/445893 (and similar pre-announcements in other channels) was to convey to people who pay close attention (i.e. Drupal site owners) that they should be prepared for the release and that this was a different situation than normal. I know that many people saw these and did take the hint, but obviously many others didn't.

    The risk of going further (even if it's just to announce widely that the release will fix a "highly critical" security issue) is that the whole world then knows Drupal has a highly critical, unpatched vulnerability. And this particular vulnerability was not hard to discover. If I recall correctly, the person who discovered it found it on his first day doing a security audit of Drupal core. He was not experienced with Drupal, and he certainly didn't know in advance that a highly critical vulnerability existed.

    If we told the whole world in advance that there's a highly critical vulnerability out there, just waiting to be found, hackers might have decided to do a "security audit" of their own and found it in advance.

    How do we balance those risks in any future pre-announcements? Is there a time window for a pre-announcement that's far enough in advance that it would actually help people prepare, but not so far in advance that it seriously risks helping the vulnerability become a 0-day attack?

  3. What time should security releases happen (and should the release windows be shorter)

    For SA-CORE-2014-005 the security team announced a 2 hour window (https://twitter.com/drupalsecurity/status/522128826101669888) and stuck to it, although in the future we could do a better job of announcing that detail more widely.

    For normal issues we don't currently constrain it anywhere near that tightly.

    From suggestions above:

    Currently, the time on advisories isn't even standard so you have to watch the announcements all day long. Switch to a specific time, say noon Eastern or something, and release reports then.
    ...
    If the current release schedule is meant to provide a rolling time window for updates to make it easier for some people some of the time why not formalize it? Start with midnight UTC on Wednesday one week, then advance an hour until you cycle back to midnight UTC.

    I think these are interesting suggestions (although for the latter I would probably change it less often, and skip more than one hour each time it's changed). It would definitely make things more fair for people in different timezones.

    But with a mostly-volunteer security team (and mostly-volunteer maintainers), it might not be realistic. Frankly I don't think I want to volunteer to coordinate a release at 3am my time, in order to allow other people to get out of doing paid work at 3am their time :) I realize it's not that simple, but that's some of what we're talking about here.

    Is there a way to make it more realistic?

  4. How security release announcements should be worded to convey severity

    From a comment above:

    The initial advisory was "Highly Critical" but still only rated 20/25 - there wasn't anything in there to signify the extreme criticality of this issue.

    I think 20/25 was appropriate at the time, since the issue was not a 0-day vulnerability; it was not being exploited (and no exploit was publicly available) at the time of release.

    The wording of the security advisory was definitely intended to convey extreme criticality, though. For example:

    A vulnerability in this API allows an attacker to send specially crafted requests resulting in arbitrary SQL execution. Depending on the content of the requests this can lead to privilege escalation, arbitrary PHP execution, or other attacks.

    This vulnerability can be exploited by anonymous users.

    However it strikes me that this wording may have assumed a bit too much knowledge of Drupal? Without that knowledge the criticality may be less obvious.

    Going forward, what is the audience that we should be aiming at for the security advisories? Perhaps we should always try to have at least one sentence in there that is as non-technical as possible? The suggestions from greggles and others above could be a good starting point for that.

  5. How followup PSAs (like PSA-2014-003) should be handled

    One thing I definitely learned from this whole experience is that people find announcements like https://www.drupal.org/PSA-2014-003 beneficial, and expect those kinds of things from the security team.

    I'm not sure we previously considered that kind of activity (consolidating and broadcasting already-public "news" about an earlier security vulnerability) as part of the security team's work. But the security mailing list and announcement procedure is clearly a very broad channel, and it looks like a lot of people didn't hear that news until we put it in that channel.

    Going forward, we might consider taking that on as an official responsibility? If so, we could add this to the "Goals of the security team" section at https://www.drupal.org/security-team, and (obviously) we would then aim to get future PSAs out faster, once it is known that a vulnerability is being exploited in the wild.

Thanks @David_Rothstein for

dokumori's picture

Thanks @David_Rothstein for clarifying the key discussion points and sharing your thoughts on those.

I also won't be able to attend Drupalcon LA so am sharing my thoughts here:

on #2, pre-announcement:

  • whether or not to publish pre-annoucements: it depends. If we can mitigate the risk of giving attackers hints while increasing the chance of users to prepare to apply patch, we should do it

  • WHEN: Considering Drupal is used globally, one (business) day is probably the shortest notice we can give. @nmcclain mentioned about Microsoft's practice so I checked it and learned they used to give a 2-business days' notice although they are switching to a new model with which users can tailor security bulletin to get information that is only relevant to them.

  • WHAT: The more obscure we make it, the less useful the information will be for attackers. If we don't mention if the upcoming release is for core or contrib, it would give attackers less chance to discover the vulnerability.
    Having said, if we are publishing a highly critical security release for a contrib that is only used by, say 30 sites, pre-announcement would seem unnecessary for most people. But then how should we set the threshold?

On #3, the release time:

  • If we won't be pre-releasing like @catch mentioned (I prefer we won't), then we simply need to pick a time slot. Since this can never be fair for everyone, I'd prefer it to be arranged to benefit the largest number of users possible so we would be able to protect the largest number of sites possible. I suppose this can be determined by analysing the IP addresses collected by the Update Status module?

It may sound unfair, but there are many other unfairness which Drupal users have to put up with (e.g. security advisories being published in English only) and some of them are simply inevitable for various reasons.

on #4, wording of announcements

  • I believe site builders with less technical knowledge can benefit from explanations written in plain language. Since most vulnerabilities that our SAs cover fall under a handful of categories (e.g. XSS, CSRF, SQL injection, access bypass etc.), the descriptions can be included in the SA generator once finalised?

There's a new policy out for

greggles's picture

There's a new policy out for comment right now that I think many people on this thread would be interested in.

https://groups.drupal.org/node/472853

Please do use the context of this thread to review that policy and provide feedback on that post.

timing of releases around holidays

greggles's picture

It is Bastille day tomorrow which means a release toward the end of the day in France is not convenient.

Unfortunately, it is basically always a holiday somewhere, the list at the bottom of http://www.timeanddate.com/holidays/ shows holidays every day from now for the next 2 weeks. France is a fairly big group of Drupal users (4th biggest country by unique sessions in google analytics in the past month) and I recognize that Bastille day is celebrated in countries beyond France.

The Drupal Security Team policy about which holidays should be avoided when making a release is https://www.drupal.org/node/101497 and it talks about holidays that affect a significant portion of the Drupal user base. With only a small amount of analysis that list was created with Christmas and Gregorian New Year to be the 2 holidays of the year to avoid a release. Drupalcons are another time when the team does not make a release. If someone wanted to do a more formal study about which holidays we should avoid and present the results for discussion/adoption that seems very reasonable. For what it's worth, the 2 metrics that show a dramatic dip in Drupal involvement for Christmas/New Years are:

  • visits to drupal.org as measured in Google Analytics
  • Usage of modules like devel which are likely to be installed on development environments that do not report usage when the person building the site is taking a holiday.

We could also examine if other metrics might make sense for this analysis.

Just to add to this comment,

doubouil's picture

Just to add to this comment, a one-day holiday on a specific day of the week can have an influence : tomorrow Bastille Day is a Thursday, so some people took their Friday off to enjoy a long week-end.

I'd say the list of possible holiday to avoid release should include a contextual factor on the day of the week and overall practice around holidays.

Related to the time, if it

mpdonadio's picture

Related to the time, if it wouldn't be considered a disclosure, knowing whether putting a site in maintenance mode will protect it until patches can be applied, or if a site needs to go into full lockdown (security policy, iptables, htaccess deny/allow, etc). This would potentially help those who are in unfortunate timezones when something is announced, but can tolerate some downtime until the problem can be addressed.

That feels to me like it

greggles's picture

That feels to me like it gives away too much information, but I'm open to other perspectives.

I suggest that people just automate putting the whole site offline via webserver configuration if they won't be online to do the upgrades at the time of the release.