I run a number of sites through a Drupal multisite setup, using different subfolders, different subdomains and different domain names off of the same code base. In some cases, I own the domain names and in other cases I'm hosting a subdomain for a domain that my customer owns.
I'd like to be able to have the sites crawled by Google, Yahoo, Bing, etc., but I'm not sure what the best way is to setup XML Sitemaps and/or Robots.txt files for the different properties. I think I can only have a single Robots.txt file, and in that case, it would have to reference all of the properties per the standards for Robots.txt files. Is that correct?
As to XML Sitemaps:
-- First, I believe I can have a different XML Sitemap for each of the sites in my multisite implementation. Is that correct?
-- Where do XML Sitemaps get stored for multisite sites?
-- Is it possible or advisable to reference each of the unique XML Sitemaps in my Robots.txt file?
-- Should I do things differently for subfolder sites vs subdomain sites vs domain sites?
Thanks in advance for any recommendations and help!
Comments
With regards to the XML
With regards to the XML sitemap, are you using the XML Sitemap module? If so, this works by registering "sitemap.xml" as a standard Drupal menu callback that, when executed, returns a cached XML file that's saved in the site's files directory (which will be different for every site), so creating and serving a different sitemap for every site isn't a problem.
For robots.txt, you could create a module which does something similar, or perhaps do something like use daemon configuration (rewrite rules) to serve a file from sites/example.com/files/robots.txt when example.com/robots.txt is requested.
The Boise Drupal Guy!
Robots.txt per site in a multisite environment
The RobotsTxt module allows a robots file per site:
https://www.drupal.org/project/robotstxt