The "robot-fooling" method - why it won't work for long

[ Reply To This Message ]

The "robot-fooling" method - why it won't work for long

From Max on 25 September '98
adding to Re: then why would they get listed 1st? posted by Terry Benge

>I'ts written, works well and cannot be detected.

I seriously doubt your claim, because you assume that the search
robot can itself be detected, and that their programmers are not
aware of these methods. Both assumptions are false.

It would be very easy to foil this technique. If the script relys
on the robot giving its name (useragent), one can change the robot so it
claims to be any of the popular browsers. If the script relies
on the request coming from a certain IP address, re-assign the
robot to a different one or run it through a proxy.

Only slightly harder to do, would be to have one robot requesting
the pages and a follow-up robot claim to be a browser. If
the pages are significantly different, there was a robot stuffing
script operating.

Replies from other people:

*Reply to this message:*
Who Are You?
Your name:
Home page title: (optional)
Home page URL: (optional)
If you'd like to get replies to your message by email, enter your address:
Your Reply:
Subject:
>>I'ts written, works well and cannot be detected. > I seriously doubt your claim, because you assume that the search >robot can itself be detected, and that their programmers are not >aware of these methods. Both assumptions are false. > > It would be very easy to foil this technique. If the script relys >on the robot giving its name (useragent), one can change the robot so it >claims to be any of the popular browsers. If the script relies >on the request coming from a certain IP address, re-assign the >robot to a different one or run it through a proxy. > Only slightly harder to do, would be to have one robot requesting >the pages and a follow-up robot claim to be a browser. If >the pages are significantly different, there was a robot stuffing >script operating.