Posted by drm on April 23, 2009 at 5:48pm
I want to disallow spider roaming on some of my sites. I've added a Disallow line with the full domain to the robots.txt file. When I use the syntax checker recommended in Drupal's default robots.txt, it gives me a warning, saying that some spiders prefer an internal path - no domain. But it also says that I have no error (just warnings).
What are folks experience? Will the main search engine bots pay attention to the domain paths?
Comments
I'm not sure what exactly
I'm not sure what exactly are your requirements, but you can check out the RobotsTxt module. I used on a multisite project some time ago: http://drupal.org/project/robotstxt
Mod_rewrite
We just solved that with some apache mod_rewrite rules. E.g:
RewriteCond %{HTTP_HOST} domainone.com$ [NC]
RewriteRule ^robots.txt robots_for_domainone.txt [L]
RewriteCond %{HTTP_HOST} domaintwo.com$ [NC]
RewriteRule ^robots.txt robots_for_domaintwo.txt [L]
So domainone.com/robots.txt actually returns the contents of a file named "robots_for_domainone.txt"