Getting Found on the Web, the Essential Role of Search

The Internet promises, potentially if not yet actually, to make available the sum total of human knowledge to every individual. As such, it could become the greatest tool of empowerment, liberation and equalization in human history. However, the sheer volume of available information brings its own difficulties, not least that of information overload. A query of "Einstein's relativity" made to Google returned 1,800,000 results [Sept 20, 2008]. Obviously there is a need for more intelligent retrieval technology, as well as for user education. Although all the answers are out there somewhere, locating them is far from straightforward.

From its beginnings the Web has always been a disparate collection of pages of various types and degrees of structure created by a multitude of publishers residing on numerous servers around the world. Thus the role of the search engine has always been pivotal to the success of the Web. The search engine is essentially a complex program combined with a vast database that seeks to match searchers with the pages that most closely match their requirements. Given that the search engine is the launchpad for many Web sessions, any search engine that comes close to getting it right is guaranteed lots of eyeballs, the foundation for making big bucks online.

Although there is still plenty of room for improvement, Google has just about got the general search market sewn up. And doubtless Google already has the best geek brains working on improvements to its search offerings. However there is still a gap for quality niche engines directories etc that serve as authorities to what's available within their particular speciality. Perhaps such a resource would resurrect Yahoo! type categories alongside Web 2.0 user reviews.

For Web publishers, Google is God. Not least because most Web publishers rely on Google for much of their traffic. Google frequently returns millions of pages for common search terms. These are generally ordered by Google's Page Rank algorithm based on the number and "quality" (which itself is based on page rank) of inbound links. Please Google, and you've hit the jackpot. Make the best site in the world with top class content and if it doesn't show highly on Google for some popular search term(s) and it'll struggle to make an impact. It is possible to overcome Google's disinterest with expensive marketing, but at the risk that outlay will exceed profits.

The Anatomy of Search

A good search engine will consistently:
* return every resource of interest (ie eliminate or most likely minimize the green area)
* return only resources of interest (ie eliminate or most likely minimize the blue area)
* return resources in descending order of interest (ie be able to gauge the relevance numbers - for this particular search)

The next generation search engines will exhibit "intelligence". They will learn user interests and preferences, eg by collecting rankings on what the user looks at. They may also respond to a query by asking further questions.