robots.txt

Gracias por la ayuda de antemano. Mi proveedor me advirtió que varios spiders y bots (de spam y búsqueda) desestabilizan y provocan la caída de mi servidor por exceso de consultas, así que decidí prohibirles la entrada y agregué reglas a Robots.txt (que Drupal ya trae por defecto) contra UbiCrawler, Doc, Zao, etc, etc. El problema es que luego de agregar las reglas, Google Bot me envía advertencias de un supuesto bloqueo a dicho agent. No se si está bien añadir por ejemplo / o sin ese simbolo..

User-agent: UbiCrawler
Disallow: /

Saludos amigos, hace dos días modifiqué robots.txt y agregué Disallow para una lista de spider o bots que usualmente afectan los sitios webs, agregue para ello los bots no deseados después de un # no entry #

El problema es que de pronto Google me responde que he bloqueado el acceso a Google Bot desde User-agent: UbiCrawler, podrían explicarme como sería la codificación correcta=?

#

robots.txt

#

This file is to prevent the crawling and indexing of certain parts

of your site by web crawlers and spiders run by sites like Yahoo!

Hi,

I have a multisite for my mobile version, but I found the links to it in google. Duplicated content is really bad for SEO.

I cannot put robots.txt because it uses the same folder that already have robots.txt.

Any way to hide my mobile version site for the search engines?

Thanks

Hello,
My website has two languages : French and German.
I see in my log warnings, type "Page not found" like that :
http://www.mysite.com/de/robots.txt

My French language has no path prefix.
Example url : http://www.mysite.com/contact

My German language has a prefix path : de
Example url : http://www.mysite.com/de/Kontakt

Could problem comes from that difference between the 2 languages configuration?
If not, what could be the problem? How to solve it?

Since several people in our group are interested in SEO they may be running the Drupal Meta Tags module (http://drupal.org/project/nodewords). Apparently there is an issue in the module where, if certain config options are set, it will completely exclude your site from Google! Here's what my colleague Owen Barton just sent out about this:

I just discovered an unfortunate function in Drupal 5.x (Drupal 5.20) which creates multiple content in Google.

http://www.example.com/?q=Drupal

Where Drupal is an url alias.

http://www.example.com/Drupal
&
http://www.example.com/?q=Drupal
are offcourse the same but google catches both and indexes them.

adding Disallow: /?q= to robots.txt wil block these multiple urls.

Currently in Drupal 5.10 it produces multiple content in multiple urls:

domain/?page=16&from=1289
domain/?page=16&from=1357

Are currently indexed by Googlebot. But is being showed as double content for the same page in Google Webmaster Tools. In fact it displays the ?page=16

Similar to this ?page= produces fictional pages for the last page in tracker pages.

These pages are indexed by google:
domain/node?page=565
domain/node?page=751
domain/node?page=759
domain/node?page=787&%24Version=0&%24Path=/&%24Domain=.domainname.xx

But currently the last page is:
domain/?page=568

I enabled the module: refine by taxonomy, http://drupal.org/project/refine_by_taxo a while back and didn't think much about it until I discovered in Google Webmaster Tools that it produced some 50.000 additional pages which of-course was indexed by Googlebot !

My site has some 6.500 nodes at the time being covering politics in Denmark with the option for 12 taxonomies on each refine by taxonomy page. I have some 500 taxonomies defined. Refine by taxonomy is currently only avialable on Drupal 5.x