Re: The robots are coming, but they don't stop.

[ Reply To This Message ]

Re: The robots are coming, but they don't stop.

From Max on 29 April '98
adding to Re: The robots are coming, but they don't stop. posted by berta

>This is not intended to be an answer to your question bu a question related with yours.
>How do you know that the robots had come to visit your site?
You see teeny-tiny footprints on your web page, the bowl of milk is empty, and the
cookies are gone.

You also will see a request for "robots.txt", which is a file they use
to find out which links they should not follow. If they don't find it
they will follow all links eventually.
(not all robots obey robots.txt but the big ones do)

If you know which IP address is the origin of a robot, you can track
individual pages that way by looking at the requests from that address.
Of course, this assumes you can look at the log filesand know what they
mean.

Replies from other people:

*Reply to this message:*
Who Are You?
Your name:
Email:
Home page title: (optional)
Home page URL: (optional)

Your Reply:
Subject:
Reply:	>>This is not intended to be an answer to your question bu a question related with yours. >>How do you know that the robots had come to visit your site? > You see teeny-tiny footprints on your web page, the bowl of milk is empty, and the >cookies are gone. > You also will see a request for "robots.txt", which is a file they use >to find out which links they should not follow. If they don't find it >they will follow all links eventually. >(not all robots obey robots.txt but the big ones do) >If you know which IP address is the origin of a robot, you can track >individual pages that way by looking at the requests from that address. >Of course, this assumes you can look at the log filesand know what they >mean.