Re: Excite came snoopin again. Lesson to be learned

[ Reply To This Message ]

Re: Excite came snoopin again. Lesson to be learned

From Max on 22 July '98
adding to Excite came snoopin again. Lesson to be learned posted by Darryl Brown

>How do you protect your site from search engines that come to your server.

The best way is to keep the pages on your local drive until they are finished.
Second best: In the ROOT directory of the site, you use a ROBOTS.TXT file to exclude them.
The "PRAGMA" tags refer to caching by a visitor's site, not search robots.

Here's my robots.txt file that keep them from indexing places I don't want them
to go
# = comment
* = ALL ROBOTS should obey the directives
disallow = where they shouldn't go

# robots.txt for http://www.site.com/
User-agent: *
disallow: /forms/
disallow: /images/
disallow: /html/ # under construction
disallow: /temp/ # temporary features
disallow: /users/ # it's really wpoison to catch address harvesters
because they don't obey robots.txt

Replies from other people:

*Reply to this message:*
Who Are You?
Your name:
Home page title: (optional)
Home page URL: (optional)

Your Reply:
Subject:
Reply:	>>How do you protect your site from search engines that come to your server. >The best way is to keep the pages on your local drive until they are finished. >Second best: In the ROOT directory of the site, you use a ROBOTS.TXT file to exclude them. >The "PRAGMA" tags refer to caching by a visitor's site, not search robots. >Here's my robots.txt file that keep them from indexing places I don't want them >to go ># = comment >* = ALL ROBOTS should obey the directives >disallow = where they shouldn't go ># robots.txt for http://www.site.com/ >User-agent: * >disallow: /forms/ >disallow: /images/ >disallow: /html/ # under construction >disallow: /temp/ # temporary features >disallow: /users/ # it's really wpoison to catch address harvesters > because they don't obey robots.txt