Comment Permalinks and Duplicate Content

I'm fighting this issue currently on my vBulletin forum where I have the canonical URL defined but I still have multiple versions of the page being displayed which have been disallowed in robots.txt. Currently I have about 19,000 pages that are being prohibited by robots.txt on a community with 10k threads and 190k posts. I haven't taken a major hit yet but I can't see that it's helping.

I also kind of question the usage of canonical reference since technically the comment page isn't going to risk being considered an exact duplicate...

Is it okay if the canonical is not an exact duplicate of the content?
We allow slight differences, e.g., in the sort order of a table of products. We also recognize that we may crawl the canonical and the duplicate pages at different points in time, so we may occasionally see different versions of your content. All of that is okay with us.

source - my link to Google's blog triggered the spam filter so you'll just have to take my word for that quote... :-)

Obviously it's going to be an exact duplicate of "some" of the content on the page but I don't think this usage is exactly what they had in mind since it's not a different version of a particular page. I'm interested to see what others have to say about this though.

As far as indexing goes though, I haven't noticed anybody indexing any of the pages I don't want to be indexed with my usage of canonical links and robots.txt exclusion

Hi Greggles,

I have two questions.

What happens when a cannonical link of a comment is used when the original node is deleted with all comments ?

I use comment page and it produces php error at present when calling deleted comment page in the Drupal 5.x version.
Comment page is here:
http://drupal.org/project/comment_page

Comment page also uses
comment/UID

Is it possible for you to allow updating the Drupal 5.x versions of comment page to your Drupal 6.x version as many will most likely update to Drupal 6.x once Drupal 7.x is released ?

Create a new path - "comment/CID" which points to comment_permalink

I think this is bad for SEO. Making new URLs for comments will probably negatively affect Drupal's SEO-friendliness in a significant way.
There IS such a thing as duplicate content and it can be very bad when done on a large scale.

Is the canonical link enough to avoid duplicate content with this page? In the issue where it was added Dries suggests that additions to robots.txt wouldn't help. Anyone have thoughts on that?

Don't rely on a canonical link tag. It's a Google thing, and Google is not the only search engine. It might not even be reliable with Google. Also, it's not good to have a lot of links on a page. Google will distribute PR to those new comment URLs even if they are blocked with robots.txt.

The hash ("#") can't be fixed? It's definitely not a good idea from an SEO perspective to make new URLs for comments.

While I agree with J. Cohen that creating a different path for each comment is a bad idea, I cannot agree with the comment on Canonical links.

The Canonical link tag is not just a Google tag and has been adopted by all 3 major search engines. I have used it extensively and it does what it is was intended to do with similar to exact content pages.

Bing adopts Canonical Link tag:
http://www.bing.com/community/blogs/webmaster/archive/2009/02/12/partner...

Yahoo adopts Canonical Link tag:
http://www.ysearchblog.com/2009/02/12/fighting-duplication-adding-more-a...

By nature, search engines want to index more useful content. URLs with parameters and hashtags used to cause issues with indexing this deep content. Using the robots.txt (which is also recognized by all 3 major SEs) instead of a canonical link is going against what they are trying to accomplish with adopting this tag across all 3 search engines.