Follow up Drupageddon responsibly

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Bevan's picture

For most security advisories, announcing the vulnerability and fix is good enough. But Drupageddon is an exceptional SA; the Drupal community and its leadership need to communicate more clearly the severity of the impact of Drupageddon to owners and administrators of Drupal 7 websites, reaching out to them using every way we know how.

The aftermath of the announcement of Drupal core SA 2014-05 (Drupageddon) is extremely severe. Most Drupal 7 websites probably now have backdoors installed. Updating or patching Drupal does not remove backdoors that were installed before updating.

In global terms, Drupageddon is worse than Apache Heartbleed, because:

  • it is so easy to exploit
  • exploits can be automated easily (and have been)
  • there are so many Drupal 7 websites, including many that manage sensitive data.

Only 120k of 930k Drupal 7 websites that report to Drupal.org updated in the first 8 days after the announcement. I estimate the number of Drupal websites now compromised by Drupageddon exploits to be between 100k and 800k.

I doubt the owners and administrators of the majority of the websites with backdoors are aware their websites probably have backdoors.

Important facts

  • Automated Drupageddon exploits started compromising and installing backdoors on unpatched Drupal sites within hours of the announcement of Drupal core SA 2014 05.
  • Many of the installed backdoors remain available for the attackers to use even after updating Drupal core.
  • It is impossible to reliably detect if a website is compromised do to the variety of methods they use: adding files to the files directory and Drupal core's code base, inserting rows into the menu_router table, creating new admin users and roles, adding blocks that execute PHP, and probably more (views?).
  • Drupal 7 websites should be assumed to be compromised if they:
    • were not patched within a day or two
    • are publicly accessible (not protected by HTTP Basic Authentication or on a private network)
    • not hosted by a provider that blocked attacks via another means, such as Pantheon, Acquia, Platform.sh (and a few others?)

Recommendations

Owners and administrators need advice, which we can give them. E.g.;

  • Take the server offline
  • Assume all your data has been stolen without a trace. Advise your users and stakeholders accordingly.
  • Build a new VM/host server.
  • Restore the website from code, database and files directory backups from before October 15. Rolling back is the only way to be certain that no backdoors remain.

We can refer owners and administrators to tools and further information like:

Other noteworthy points we could share:

  • There are reports of compromised Drupal 7 websites being used to send spam.
  • Full-featured PHP shells were installed on some servers by Drupageddon exploits
  • Data is likely being copied without permission and without leaving a trace, making the event undetectable.
  • Attackers can use the exploit to write files outside of Drupal's directories if permissions are not locked down correctly (a common sysadmin error).
  • Attackers are likely using the PHP backdoors to gain access to other parts of the operating system via known but unpatched vulnerabilities in other software on the server.
  • I have seen about a dozen unique types of exploits. I am certain there are many more.

Next steps

I am happy to draft the announcement. If so, I have a few questions;

  • Is there consensus about using the nickname "Drupageddon"?
    I am in favour of it as the "armageddon" reference communicates the severity quite appropriately IMO.
  • Should such an announcement be a "security advisory"?

I think an SA is appropriate, given the new information it reveals. Even though there is not a "new" vulnerability per sé, the information significantly expands the scope of SA 2104-05.

Also, the fact it is information without a patch is consistent with other SAs that tell website administrators how to configure their website securely, which is also only information and recommendation.

If it is not an SA, the information needs to be distributed at least as widely as SAs.

Conclusion

Please publish widely the impact of Drupageddon exploits in order to control damage to:

  • Drupal's reputation
  • the safety of anyone using the internet
  • security of hundreds of thousands of systems

Neglecting to announce this information is irresponsible and hurtful to Drupal's reputation.

Comments

The Drupal security team is

scor's picture

The Drupal security team is not going to release a SA for this, it's a PSA that we need, if anything. Security team members have generally been avoiding the term Drupageddon/Drupalgeddon which is not reflecting well on the Drupal project. It might have been more appropriate if this vuln had been disclosed as a 0-day. Sites that followed the recommended process of patching on Oct 15th are safe (if they patched a few hours after the announcement). I agree though that they are things to be learnt with this SA for both the Drupal security team and the community, and that the current number of sites which are not on 7.32 is worrisome (though I'm sure some, but not all, might be patched).

As long as the PSA has reach

Bevan's picture

As long as the PSA has reach at least as wide as an SA, I think that is fine. The important thing is that website owners and administrators know that their websites are probably hacked even if they updated.

I understand "Drupageddon" reflects badly on Drupal, but I think the term grabs attention; and that is more important at this time. Nevertheless, I think its more important to get the information out quickly and effectively, rather than agree on a name. So call it what you will.

When will the security team get a PSA out?

Generally any SA or PSA is

pwolanin's picture

Generally any SA or PSA is released on a Wed, so I assume we'd be looking at the 29th as a possibility.

For anyone following along,

David_Rothstein's picture

For anyone following along, the PSA is out as of today: https://www.drupal.org/PSA-2014-003

Thanks to everyone who worked on it, especially Bevan.

Only 120k of 930k Drupal 7 websites that report to Drupal.org updated in the first 8 days after the announcement.

Those numbers from https://www.drupal.org/project/usage/2357213 may be misleading, for two reasons.

First, many sites applied the patch instead.

Second, many sites don't report back to drupal.org too frequently (the default is daily, but you need to be running cron that often - and also many sites switch the configuration to weekly). So I think that a site that reported to drupal.org on Monday October 13 and then updated right away on Wednesday October 15 would not show up as running 7.32 in that week's stats even though they did update in time.

The numbers from the week after (225k sites) are certainly an upper bound for the number of reporting sites that might have updated to 7.32 that first day. But that still doesn't take into account sites which applied the patch.

Thanks for the clarification;

Bevan's picture

Thanks for the clarification; I was wondering about that.

How to avoid security sensationalism?

brylie's picture

We also have a string of security related issues reported in the Free Software community recently. I agree that patching and updating is important, but also believe that instilling fear in site administrators and the general public is not a necessary prerequesite.

Speaking directly of this issue, on what do you base your estimate of 100K to 800K exploited sites?

Brylie Christopher Oxley

Avoid getting defensive

bigtrac's picture

I think sensationalism is not the issue.
The issue I would worry about it people losing trust in Drupal. We should be making sure people have confidence that this sort of thing will be improved so that the impact of similar security issues is minimised in future.
I was on holiday on the 15th and it just so happened for around 24 hours I had almost no WiFi connectivity. By the time I got the information it was too late and several of my sites were infected. (Luckily not my largest site as it was D6 and still in the process of being rebuilt).
But the website for another client was affected and since I was in the process of building it for them, the backups were too old. The whole site will need rebuilt. But the client is no longer keen on using Drupal.
Since I don't have a team of developers, and there was a 3 to 24 hour window to update, I cannot 100% guarantee to them that something similar won't happen again. (Obviously regular backups should help, but sometimes it doesn't happen even when it should. I'm aware I may be jumped on for not following best practice, but the facts are that lots of people are worse than me, and people make mistakes too.)

I can't help but feel that a different process for major security releases like this is required. It seemed that hackers got the best of this announcement, since it gave them a heads up which not everyone with an installation would receive in time.

Possibly in future there could be a warning that a major fix is coming? So that people have a bit more time to prepare and are ready to make the changes quickly, before the exploit is already known about in the wild. Not perfect, but better than announcing a security hole that is then exploited on a massive scale.

There's some discussion in

greggles's picture

There's some discussion in https://groups.drupal.org/node/446603 about the pre-announcement that did happen, what channels it went out on, what channels it should go out on, and what content it should have. I'd love to have your feedback on the proposals there.

So, I think it's clear there

pwolanin's picture

So, I think it's clear there are lessons to be learned about how to handle such a situation if it happens in the future.

However, let's be clear that this is the single worse vulnerability discovered since the Drupal Security Team was formed in 2005, so there was not really a clear roadmap for what to do in advance, how best to get the message out, or how quickly we'd actually see attacks.

One of the problems I see as

ktmom's picture

One of the problems I see as manager of a small website using Drupal, is most smaller sites/shops are looking for a quick solution to getting their clients/businesses online and don't have the resources or knowledge to monitor security. This doesn't make it Drupal's problem per se, but it does have an affect on perceptions. These managers are looking to solve an immediate problem without the time, knowledge or maybe willingness to ensure they are using a tool they are competent to use. Like so much in modern society, it seems we just want what is easiest in the moment.

I also think that these smaller site's managers may percieve that they won't be targeted because they are smaller. Surely hackers will go after government and corporate sites first giving them time to "get to" patching. I want to point out, our small site, with no data worth stealing, was attacked within 24 hours of the announcement. I would think that the ultimate goal is more to gain control of the server than the site.

I say again, I feel that it is the responsibility of the site owner/manager to be competent to put resources on-line. But if Drupal is concerned about the masses, maybe a plug-in in core that presents an alert or sends an email to the site administrator regardless of other notification settings with a pre-release message might help get the word out.

Sensationalism

Bevan's picture

Bryan Ruby at CMS report has a fantastic article on sensationalism.

I think mainstream media will always be sensationalist/alarmist because that sells to the mainstream audience. In this case, getting the word out to Drupal 7 website owners was the priority, and mainstream media coverage (emotional or otherwise) certainly helps achieve that.

I think all of them sensationalised it to some extent. Finding people to quote to sensationalise it more is not difficult with something this extreme. (This might be once in a decade or even lifetime sort of event!)

Small companies, limited resources

theargument's picture

I'd like to put in a thankyou to Bevan for his guidance on how to manage the fallout of this SA. The flow diagram and strategic/practical discussion of the problems has been really useful.

I have a small web design and development business - the datacentre where my sites are hosted only keeps backups for 7 days - and I now have around 20 compromised sites. Stupidly of course I wasn't a subscriber to the SA newsletter or I guess I would have heard of the issue sooner. As it was I first heard about it from the BBC over the weekend just gone. Am painfully aware that I've been caught napping but in all honesty I doubt very much whether I would have caught any of this in anything like good time even if I'd had a little more notice as I don't have the resources to continually monitor and update on a daily basis. With hindsight of course I see that I can't afford not to.

However, some system of alerting that gives genuine priority to, say, holders of Drupal.org accounts - would seem to be crucial. As it stands, an SA simply alerts hackers to a vulnerability and gives them a level playing field in the race to see who can hack or repair first. Which seems a little unfair. I have an account on Drupal.org but received no notification from that source - unless it got caught as spam... Admittedly hackers can have Drupal accounts too - but it would seem at least a step in the right direction in alerting genuine users ahead of the world in general.

I have updated all my sites and removed the megausers that had appeared in half of them - but am still left with the sure knowledge that all of them remain compromised and I have no option but to physically rebuild on new servers (which the datacentre will obviously charge me to create) and then transfer just those elements that I am reasonably sure are free of back doors - so really just the theme files that I hold locally and the contents of the files folder. Even then, as I understand it, my sites may not be safe?

The other things that concerns me is how safe are my D6 sites and Wordpress sites on these servers and should I be treating them with the same suspicion that I now view my D7 sites? Or am I 'reasonably' safe to transfer them as they are?

Anyway - thanks again to Bevan - back to building a brighter, newer more secure business...

Thanks @theargument for

David Sparks's picture

Thanks @theargument for taking the time to post this - and articulating what I'm sure many thousands like us will be thinking, but are too busy to express. Now, back to work recreating all that content since August's ad-hoc backup...

Thanks David. I guess if we

theargument's picture

Thanks David. I guess if we were bankers this would be a 'Lehman brothers' kind of day!

@theargument; Thank you very

Bevan's picture

@theargument; Thank you very much for your appreciation and for sharing your story. There are a lot of Drupal websites that will have stories like this; probably the "long tail" or "the 80%". My intention with the flowchart is to help those people.

This is not the right place to answer your questions. Also, you have an interesting story that I think is worth sharing as in example "worst case scenario" (or pretty-bad, at least). Could you post your questions on my blog (Drupal.geek.nz) or the SA's FAQ so I can respond there?

Thanks!

Benefit of hindsight

Bevan's picture

I forgot to respond to the comment about notification, which is on topic here. I agree it would have been fantastic to email all Drupal.org users about the original SA. I think that would have significantly reduced the impact of attacks. This probably didn't happen for a number of reasons; Drupal.org T+Cs, privacy policy, security team policy and most of all: I doubt many people expected automated attacks to happen so quickly and so broadly. I certainly did not.

In other words; that is a fantastic idea, now that we have the benefit of hindsight. Perhaps the security team can take that on board and provision for such a need in the future for such highly critical SAs.

Hindsight is always 20:20

theargument's picture

@Bevan I think Billy Wilder said that - though in this case it should probably be 'Hindsight is always 25:25'. Unsurprising I guess that policy in part responsible. While I appreciate that the speed of attacks is something you need to chalk up to experience - it can also serve to inform policy so it'd be good to see some changes in that direction to bring policy into line with conceivable events - which is what they serve to provide a framework for. We do, after all, live and learn.

Regarding the other questions - I will post now on your drupal.geek blog - many thanks for responding so quickly to this.

Bad Behavior, Honeypot, http:BL before 7.32?

decibel.places's picture

Does having these modules installed prior to 7.32 have any affect on exploits?

Great thread, will repost as part of our Podcast.

shanesevo's picture

Just finished a new podcast with a discussion of Drupageddon and just how serious this is.

http://www.commercialprogression.com/post/hooked-drupal-podcast-episode-2

Security

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week