---

[ July Full List | Reply ]

---

Re: Is there a way to tell if I've been reviewed? From Brian at Shipping and Packaging Supplies Wholesale on 5 July '01 replying to Re: Is there a way to tell if I've been reviewed? posted by DRB >>Please excuse my stupidity, as I am sure this question is part of basic web site design, but I am a self-taught novice...I've heard people indicate that they know their site has been visited by a search engine spider / directory reviewer, but my question is: how do they know this ? Is there a program they have that tells them this information ? Thank you for your insight! > >Hope someone can help you, I just seen a way about 2 days ago >while surfing, you add a command before your url I beleive. Didn't bookmark it as I already am getting listed on search engines, for Yahoo it only took 3 days. A lot of the spiders have names, and the name shows up in your log files. Basically, the engine identifies itself. For example, the google spider is now called "Googlebot". Also look for requests for "robots.txt" file, as it is usually a search engine looking for that file to see what you want included or excluded by the engine. Just browse through your logs a bit and you will start getting some idea of whether or not it has been indexed. As for directories those are a little trickier, but even some of those identify themselves (I know Yahoo does, but cannot remember how it looks).

Re: Is there a way to tell if I've been reviewed? - Dale - 06/07/01

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>>Please excuse my stupidity, as I am sure this question is part of basic web site design, but I am a self-taught novice...I've heard people indicate that they know their site has been visited by a search engine spider / directory reviewer, but my question is: how do they know this ? Is there a program they have that tells them this information ? Thank you for your insight! >> >>Hope someone can help you, I just seen a way about 2 days ago >>while surfing, you add a command before your url I beleive. Didn't bookmark it as I already am getting listed on search engines, for Yahoo it only took 3 days. > >A lot of the spiders have names, and the name shows up in your log files. Basically, the engine identifies itself. For example, the google spider is now called "Googlebot". Also look for requests for "robots.txt" file, as it is usually a search engine looking for that file to see what you want included or excluded by the engine. Just browse through your logs a bit and you will start getting some idea of whether or not it has been indexed. As for directories those are a little trickier, but even some of those identify themselves (I know Yahoo does, but cannot remember how it looks).