googlebot

Saludos amigos, hace dos días modifiqué robots.txt y agregué Disallow para una lista de spider o bots que usualmente afectan los sitios webs, agregue para ello los bots no deseados después de un # no entry #

El problema es que de pronto Google me responde que he bloqueado el acceso a Google Bot desde User-agent: UbiCrawler, podrían explicarme como sería la codificación correcta=?

#

robots.txt

#

This file is to prevent the crawling and indexing of certain parts

of your site by web crawlers and spiders run by sites like Yahoo!

Hi,

I have a multisite for my mobile version, but I found the links to it in google. Duplicated content is really bad for SEO.

I cannot put robots.txt because it uses the same folder that already have robots.txt.

Any way to hide my mobile version site for the search engines?

Thanks

Currently in Drupal 5.10 it produces multiple content in multiple urls:

domain/?page=16&from=1289
domain/?page=16&from=1357

Are currently indexed by Googlebot. But is being showed as double content for the same page in Google Webmaster Tools. In fact it displays the ?page=16

Similar to this ?page= produces fictional pages for the last page in tracker pages.

These pages are indexed by google:
domain/node?page=565
domain/node?page=751
domain/node?page=759
domain/node?page=787&%24Version=0&%24Path=/&%24Domain=.domainname.xx

But currently the last page is:
domain/?page=568