a noindex tag

I know I'm replying very late, but I found this thread while searching for the Meta tags, eh Node Words, module. For completion I just wanted to add:

In many situations you don't have access to robots.txt - you don't have access to the root folder of the domain where your Drupal is installed. Think personal home pages for students at universities, people using hosting services but without their own domain name and so on. Then the robots meta tag is the only option.

Hans

Perhaps my knowledge of robots.txt is not as strong as it should be, then!

I guess I was thinking for pages like:

I would like to have pages with this pattern in the index:
http://example.com/user/*

I would potentially like to remove pages like this from the index:
http://example.com/user/{any}/track
http://example.com/user/{any}/feed
http://example.com/node/{any}/feed

As far as I know, that kind of wildcarding would not be possible in robots.txt which would mean that I would need to have a

Disallow: /user/1/feed
Disallow: /user/2/feed
Disallow: /user/.../feed
Disallow: /user/n/feed

for every user, right?

I look forward to the insight from you and NikLP on this idea.

--
Knaddisons Denver Life | mmm Chipotle Log | The Big Spanish Tour

I think that's a fair point, but realistically, you're unlikely to build a site and then have an increasing number of pages that you don't want indexed - I suggest the probable outcome would be that you have "finished" a site, and then have a static list of pages that you do not wish to have indexed, and then can easily control that list via robots.txt.

Of course it would be easier from an administrator's (site manager rather than dev, here) point of view to perform this task at a node level from the admin pages. However, it's really just bloating the system in effect, as it will perform checks of one sort or another per node view, which impacts on "us" (and the users) rather than the target audience, ie the search engines, who, in regards to robots.txt, currently "take the strain" of the processing.

So ultimately probably not worth it. The only time I can see that option being really useful is if a site manager was really adding/deleting massive swathes of content (like sections or something) all the time, and there was a need to block one or two specific pages in each section. Even that's pretty tenuous at best, I think.