deadlock Message Board
[ January Full List | Reply ]

Re: robots.txt error message
From David A on 16 Jan '00
replying to robots.txt error message posted by Georgie Farmer

>Hi,
>
>I get several error messages a week in my stats for "robots.txt". Could someone please tell me what this means?
>
>Cheers,
>Georgie

This file is used by SE spiders. Each time 1 visits it will request this file. The file is used to tell them what directories not to index

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /private/




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message