Jim's Message Board
[ Reply To This Message ]

Re: Please help: Competitors Submitting Our Pages!
From Max on 17 August '98
adding to Please help: Competitors Submitting Our Pages! posted by Diana


>I frequently see references to the submissions on our server logs on days when I have not done it.
What are you seeing? If I submit YOUR pages from MY ISP, that transaction is between Excite and me.
You shouldn't be seeing anything. Are you perhaps seeing their robot indexing your site?
It can occasionally get "stuck" and repeatedly request the same page several times
Can you paste some of the server log entries?

>Also has anyone ever seen a reference to a site called seventwentyfour.com in their server logs.
>We have not requested this service.
Someone with the service has a link to YOUR site. The only way to test a link is to request the page
from the site that has it.

>I have written a robots.txt to disallow the LinkWalker robot from our site (was coming daily)
>but will this cause any problems getting listed anywhere else.
The problem is that LinkWalker will now report links to your site as "broken" (if it obeys robots.txt),
and webmasters might just delete the "broken" link rather than visit to see what's wrong. Let the critter
check links all it wants.
The exception to this would be is if is checking links SO often that it is loading down your server. The
early "change-detecting" robots were badly written, and would hit a site with so many requests
for full pages that it degraded service. From their web site, it appears that they only check once per day to
verify links, then feed the results from their database to their customers.



Replies from other people:


Reply to this message:

Who Are You?
Your name:
Home page title:
(optional)
Home page URL:
(optional)

If you'd like to get replies to your message by email, enter your address:


Your Reply:
Subject: