Hi all,
I am new to nginx so hopefully someone here can help me.
Since my site will have a lot of images submitted by registered users I use CDN module to serve static files from Cloudfront and that is working well.
But there is one serious caveat: my site can be accessed from CDN subdomains thus creating a duplicate content issue which is a serious problem for SEO.
I've searched high and wide before posting here but with no success.
Is there a way to prevent search engines from indexing my CDN subdomains?
My CDN setup looks like this:
http://js.mysiteurl|.js .otf .ttf .woff .eot
http://css.mysiteurl|.css .less
http://img-cdn1.mysiteurl http://img-cdn2.mysiteurl|.jpg .jpeg .png . gif .ico .bmp .svg
Ideally search engines would still be able to index image files but site content would only be accessible via http://mysiteurl
Anyone has the code that can solve this problem please?
Comments
The way I read this, your CDN
The way I read this, your CDN basically works in such a way that cdn.example.com/node/101 is the same as example.com/node/101. In that case, it's not really an Nginx issue - it is exactly what the rel="canonical" metatag is meant to solve.
https://www.drupal.org/project/metatag
I am already using rel="canonical" metatag
@Brian I am already using rel="canonical" metatag with absolute url's which will resolve duplicate content issue however, I wanted to go one step further and make sure that cdn.example.com/node/101 is not indexed/accessible
Something like what was described at biztekmojo.com/00123/drupal-cdn-module-and-seo-avoid-duplicated-content which I tried to implement (despite all of the if statements) but it didn't seem to work as I can still access my content at cdn.example.com/node/101 even though I want Cloudfront to cache my static files only but it seems to cache and server the whole page from cdn.example.com/node/101
I've also tried
server {
listen 80;
server_name cdn1.uhaaa.net cdn2.uhaaa.net cdn3.uhaaa.net;
access_log /var/log/nginx/$host.access.log;
root /home/uhaaa22/public_html;
server_name_in_redirect off;
location / {
return 404;
# or
# rewrite ^ http://www.example.com permanent;
}
location ~ ^.+.(jpg|jpeg|gif|png|ico|js|css)$ {
# access_log off;
# error_log /dev/null crit;
expires 45d;
try_files $uri @drupal;
}
location @drupal {
###
### now simplified to reduce rewrites
###
rewrite ^/(.*)$ http://uhaaa.net/$1 permanent; }
}
which I found at https://groups.drupal.org/node/48182#comment-124088 but that did not work either (its a 4 year old recipe so that's probably why it did not work)
Any ideas please?
Drupal web developer and Search Engine Optimisation consultant at Webmar Design & Development Brisbane, Australia
Google+
is the example from
is the example from BizTekMojo doesn't work for you? It works perfectly fine for me.
http://www.biztekmojo.com/00123/drupal-cdn-module-and-seo-avoid-duplicat...
set $cdnyesno "no";
if ($http_user_agent ~* "CloudFront") {
set $cdnyesno "no";
}
if ($request_uri ~* "^.+.(css|js|swf|png|gif|jpg|jpeg|ico|pdf|zip)") {
set $cdnyesno "yes";
}
if ($cdnyesno ~* "no") {
rewrite ^/(.*) /$1 permanent;
}
Why not just use/modify your
Why not just use/modify your robots.txt file?
The Boise Drupal Guy!
use robots.txt?and how you
use robots.txt?
and how you will serve different robots.txt for CDN domains and different for www.domain.com ? using exactly same redirect rules as above, but do not apply to images/static files but to robots.txt file only? You still have same problem.
I don't necessarily think
I don't necessarily think it's a good solution, but there is a module showing how you can dynamically generate the robots.txt file https://www.drupal.org/project/robotstxt
If it doesn't support the exact use case it at least provides a proof-of-concept that delivering a different robots.txt based on server host name is possible.
knaddison blog | Morris Animal Foundation
well, controlling this at
well, controlling this at robots.txt level prevent only search engines that respect robots.txt while some scrapers doesn't.
Static only files via htaccess
If anyone is interested, I've just got this working with the following (D6 style) rule in htaccess.
# Only allow static files to be displayed on the static or CDN domain.# This is done by redirecting all attempts to access non-static files to the non-static domain name.
RewriteCond %{HTTP_HOST} ^static\.domain\.com$ [NC]
RewriteCond %{REQUEST_URI} !\.(css|js|ico|png|gif|jpg|jpeg|bmp|pdf|doc|docx)$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [L,R=301]
Add directly after "RewriteEngine on" so it occurs before checks for www vs non-www.
Presumably this could be done in reverse too, sending requests for static files from the www domain to the static domain.