Jim's Message Board
[ Reply To This Message ]

Re: robots.txt Testimonials ...
From Max on 01 June '98
adding to robots.txt Testimonials ... posted by NightFly ~*

>Hi,

>I'm looking for feedback from anyone who has
>used robots.txt files with *any* success.

I've been using it for a long time, mostly to keep the little critters
out of the experimental parts of some sites. The search engine robots
obey the exclusion protocol. The ones scavenging email addresses from
web sites usually don't follow the rules in robots.txt and end up caught
in WebPoison scripts where they recurse themselves to death.

It works the way I want it to. What were you trying to do with it?


Replies from other people:


Reply to this message:

Who Are You?
Your name:
Home page title:
(optional)
Home page URL:
(optional)


Your Reply:
Subject:
Reply: