Posted by Macronomicus on August 9, 2009 at 10:56am
What would be the simplest way to block search engines from parsing Aegir management level Nodes?
Ive noticed some of the platform pages and other backend tasks came up in a google search I did.
Should Aegir do this automatically? Guess if we just set the hosting and tasks nodes as unpublished by default that would be easy enough but not really the most logical approach.
Comments
Your Aegir nodes are
Your Aegir nodes are accessible by the anonymous role?
Mine are denied by default, anonymous users have no access.
Anyway, that'd be my first 'simplest' suggestion to block googlebot visits! Block all visits to non-authenticated users, like it is by default. Otherwise if you want for some reason public access to your nodes but not to search-engine bots (?), look at tweaking your robots.txt file?
i think a .htaccess password
i think a .htaccess password would be good and also be a security improvement
--
http://aureli.us
@mig5 I havnt changed any of
@mig5
I havnt changed any of the settings related to access ... and can confirm they are not accessible by anonymous users. Still google parsed "some" of them anyhow.
@ben
What are you suggesting .. block access to the entire aegir site, but for admins?
In the google search results some of the verify tasks are coming up, however if you click them it says page not found ... here is a clip of one.
Verify mysite.com (Drupal 6.13) | My Site
Task Verify mysite.com (Drupal 6.13). Submitted by me on Mon, 07/13/2009 - 05:07. Status: Succesful. Executed: Mon, 07/13/2009 - 05:08 ...
www.myothersite.com/node/71 - Cached - Similar
Whats weird is that the verify tasks and referenced site node do not match or they are backwards.. somehow the bot is getting in and making weird assumptions.
I guess its not really divulging any important information though, and the pages aren't accessible so its not that big of a deal!
Just a little strange o_O
You don't have xmlsitemap or
You don't have xmlsitemap or some other module exposing the information to google?
No I left the aegir site
No I left the aegir site pretty spartin ... ive really not enabled much but the blog.
yes, unless you want
yes, unless you want anonymous users or anyone except yourself to have access to the site, its an easy and safe thing to write some extra lines to your .htaccess.
Baffling how a bot would get
Baffling how a bot would get hold of it, unless it wasn't protected in the past and what the bot has is cached stuff from an older crawl. In any case, since your stuff is protected from the anonymous role, you can consider making your robots.txt more strict disallowing sections (or all) of your site.. maybe?
Otherwise as ben said, protect the entire site with htpasswd / mod_auth in Apache so that one has to authenticate to even reach the site.
Ive discovered these are in
Ive discovered these are in fact old tasks none of the new ones are being cached by the bot ..so whatever it was is over now ... sweet!
Thanks for your thoughts everyone. Just one of those weird one off things I suppose.