---

[ March Full List | Reply ]

---

Re: robots.txt From jwj on 20 March '01 replying to robots.txt posted by inachu I have a site that I've been trying to get the robots to leave alone so I have been using the asterisk, as I assumed that was all you needed but slurp and gulliver keep showing up - I don't know if they're ignoring the request or simply coming to see if it's changed, so I now have the robots.txt file reading like this: User-agent: * Disallow: / User-agent: Slurp Disallow: / User-agent: Gulliver Disallow: / I'll check it next week and see if they're still sniffing. As someone pointed out a few weeks ago, cnn is covering all angles on their file http://www.cnn.com/robots.txt

Re: robots.txt - Inachu - 20/03/01

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>I have a site that I've been trying to get the robots to leave alone so I have been using the asterisk, as I assumed that was all you needed but slurp and gulliver keep showing up - I don't know if they're ignoring the request or simply coming to see if it's changed, so I now have the robots.txt file reading like this: > >User-agent: * >Disallow: / > >User-agent: Slurp >Disallow: / > >User-agent: Gulliver >Disallow: / > >I'll check it next week and see if they're still sniffing. > >As someone pointed out a few weeks ago, cnn is covering all angles on their file http://www.cnn.com/robots.txt