Securing the files directory
Securing the drupal files directory
Hi everone, im having a bit of a nightmare securing the /files directory of my drupal site from google. I have lots of important user information in here (i.e. 000's of users CS's with contact numbers, address etc) that google is searching and indexing like mad on me.
I thought I had this secured taking the following methods:
1) robots.txt file - Disallow: /files/ and also
no CV's
Disallow: /files/
Disallow: /files/.doc
Disallow: /files/.doc$
Disallow: /files/.pdf*
Disallow: /files/*.pdf$
2) Under the drupal file system settings - Private - files are transferred by Drupal.
I have done some research and im getting the impression I should have placed my files directory one up from the root to start with, but doing this now doesnt seme to be an option for me with the amount of bad linking users have done amongst all the CVs in the directories.
Has anyone any ideas how I can prevent google indexing my files directory, and better stilll, remove all the links to date it has recorded about the drupal directory? Im very scared of this growing larger
Google webmaster tools
Google webmaster tools allows you to remove your files from its index via the tools section: https://www.google.com/webmasters/tools/sitetools
it can also analyze your robots.txt file and tell you which pages are currently being blocked by it.
The first rule of your robots.txt should be enough
When enabling private download, Drupal should create a .htaccess file in the files directory. Check if it exists and if it contains some restriction rules. That should be enough to secure your files directory.
thankyou scor
Scor, thank you so much, I never knew that option existed in google webmaster tools. I have submitted the drupal tigerenglish.com/files/ directory for removal and made sure the robots is blocking it.
I see the .htaccess file in the files directory but it only contains:
SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
Options None
Options +FollowSymLinks
Should I also add additional security in here?
FYI, Google does not remove
FYI, Google does not remove anything from its index if files are still accessible. At least, this is the error message I received the last time I tried. As for documents in /files - they should not be treated as private anyway, unless you turn on private uploads (which can affect performance of your site).