>Google is my favorite of all search engines! Whenever I've done a search on that engine for anything what I'm looking for is 99% of the time in the first ten listings!
That's because Google uses links TO a page, natural language
processing, etc. to decide what to index and how to rank it.
They have the "finder" running around the web and a couple of
"indexers" that sit at home and analyse the data: 40-50 pages/second!
It is heavily dependent on document structure ... it's really
easy to please with good HTML and some nice content, but VERY
hard to fool with 'spamming' techniques. I didn't realize they
had turned it loose from the Stanford campus until it scavenged
half the pages on one site over the past week. I thought it was
an e-mail harvester.