de/robots.txt not found

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
sahuni's picture
Posted by sahuni on May 15, 2012 at 7:48am

Hello,
My website has two languages : French and German.
I see in my log warnings, type "Page not found" like that :
http://www.mysite.com/de/robots.txt

My French language has no path prefix.
Example url : http://www.mysite.com/contact

My German language has a prefix path : de
Example url : http://www.mysite.com/de/Kontakt

Could problem comes from that difference between the 2 languages configuration?
If not, what could be the problem? How to solve it?

Categories: , ,

Comments

de/robots.txt not found - solution

Posted by onlineseo on May 15, 2012 at 3:31pm
onlineseo's picture

you have to upload robots.txt file to root folder,

And for http://www.mysite.com/contact and http://www.mysite.com/de/Kontakt

have to check from admin side, page available or not

thanks.

OSS
http://www.onlineseosubmission.com/

create a file in Notepad as

Posted by aditya@braintec... on May 15, 2012 at 3:55pm
aditya@braintechnosys.com's picture

create a file in Notepad as robot.txt and paste the code below :

User-agent: *
Disallow:

Save this file and upload to the root in your server .

And for http://www.mysite.com/contact and http://www.mysite.com/de/Kontakt

Please check if there are folders / file for contact page in your server .

Regards
Aditya khanna
www.braintechnosys.com

Well, I think my English was

Posted by sahuni on May 16, 2012 at 6:18am
sahuni's picture

Well, I think my English was not so good, because you don't answer to my problem.

I have a robots.txt at root, the default robots.txt we get when installing Drupal.
Just 1 robots.txt at root.

Question is why robots are sometimes looking to ....de/robots.txt?

I guess it's because of following fact :
I have a second domain name deutschname.com which is redirected to mysite.com/de/
So I guess when robots are looking for deutschename.com/robots.txt, they are redirected to mysite.com/de/robots.txt.
Can you confirm it's surely the reason why scrawler are looking for mysite.com/de/robots.txt ?

Perhaps, the fact that scrawlers do not find mysite.com/de/robots.txt is not a problem and I must let it down.
Can you confirm?

How can I know for sure that robots are looking for deutschename.com ?
Is not it best that robots do not scrawl at all for deutschename.com?
How can I stop it? Will it penalize the SEO for German language?

It's a lot of cascading questions, but at least for me, it makes sense to ask all of them.

You can get in touch with me

Posted by aditya@braintec... on May 16, 2012 at 7:07am
aditya@braintechnosys.com's picture

You can get in touch with me on skype : braintech_bdm02 for detailed discussion

Regards
Aditya khanna
www.braintechnosys.com

i like this post very much

Posted by albertstone on June 23, 2012 at 12:22pm
albertstone's picture

i like this post very much and it is very informative postSeo Dubai

Hi sahuni, i dont know if i

Posted by eule on June 23, 2012 at 2:34pm
eule's picture

Hi sahuni, i dont know if i right but have you installed the robots.txt module? http://drupal.org/project/robotstxt
i mean this is was you need for a multi language site

and u dont will penalize, but the spider can crawl everything,

best Regards
my site has not a multi language setup

Greets from Germany
Prepaid Tarifvergleich

No Eule, I did not. You give

Posted by sahuni on June 23, 2012 at 3:28pm
sahuni's picture

No Eule, I did not.
You give a good advice. I'll have a look asap.
Thanks for proposing