---

[ January Full List | Reply ]

---

Re: robots.txt error message From David A on 16 Jan '00 replying to robots.txt error message posted by Georgie Farmer >Hi, > >I get several error messages a week in my stats for "robots.txt". Could someone please tell me what this means? > >Cheers, >Georgie This file is used by SE spiders. Each time 1 visits it will request this file. The file is used to tell them what directories not to index User-agent: * Disallow: /images/ Disallow: /cgi-bin/ Disallow: /private/

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>Hi, >> >>I get several error messages a week in my stats for "robots.txt". Could someone please tell me what this means? >> >>Cheers, >>Georgie > >This file is used by SE spiders. Each time 1 visits it will request this file. The file is used to tell them what directories not to index > >User-agent: * >Disallow: /images/ >Disallow: /cgi-bin/ >Disallow: /private/