Re: The robots are coming, but they don't stop.

[ Reply To This Message ]

Re: The robots are coming, but they don't stop.

From Max on 07 April '98
adding to The robots are coming, but they don't stop. posted by David Notestine

>Robots are coming but they only read my
>index.htm and the 5 pages generated by
>'pro artist'.
>In the last week, 5 or 6 have visited but
>none have analyzed my site.
Typical robot behavior is to visit the top-level
page first. They will come back at intervals
and follow the links from the index page: provided
there are links, and the site doesn't use frames
too heavily. They are programmed to space their
requests out to avoid overloading any server with
their activity.
I took the time to track their course through
a new multi-level site with about 50 pages by analysing
the log files for requests from known robot IP
addresses. I only submitted the index.htm page,
and had all the proper robot bait in place before I
submitted.
Here's a composite picture:
First, there was a request for robots.txt, followed
by a hit on the index.htm page. Within the week, there
were requests for the pages linked from the
index page, and they hit the third level pages
by the end of the fortnight. Every visit was
preceded by a request for robots.txt and about
five pages (maximum) were requested each day.

Frames make it VERY difficult for robots and repeat
visitors. Neither one can set bookmarks and
jump back to where they want to go after leaving the
site. Unless you have an absolutely overwhelming
need for frames, they are more trouble than they
are worth.

Replies from other people:

*Reply to this message:*
Who Are You?
Your name:
Email:
Home page title: (optional)
Home page URL: (optional)

Your Reply:
Subject:
Reply:	>>Robots are coming but they only read my >>index.htm and the 5 pages generated by >>'pro artist'. >>In the last week, 5 or 6 have visited but >>none have analyzed my site. > Typical robot behavior is to visit the top-level >page first. They will come back at intervals >and follow the links from the index page: provided >there are links, and the site doesn't use frames >too heavily. They are programmed to space their >requests out to avoid overloading any server with >their activity. > I took the time to track their course through >a new multi-level site with about 50 pages by analysing >the log files for requests from known robot IP >addresses. I only submitted the index.htm page, >and had all the proper robot bait in place before I >submitted. > Here's a composite picture: > First, there was a request for robots.txt, followed >by a hit on the index.htm page. Within the week, there >were requests for the pages linked from the >index page, and they hit the third level pages >by the end of the fortnight. Every visit was >preceded by a request for robots.txt and about >five pages (maximum) were requested each day. > Frames make it VERY difficult for robots and repeat >visitors. Neither one can set bookmarks and >jump back to where they want to go after leaving the >site. Unless you have an absolutely overwhelming >need for frames, they are more trouble than they >are worth.