by: Liji Elizabeth Thomas

Thousands of servers …billions of web pages…. the possibility of individually sifting through the WWW is null. The search engine gods cull the information you need from the Internet...from tracking down an elusive expert for communication to presenting the most unconventional views on the planet. Name it and click it. Beyond all the hype created about the web heavens they rule, let's attempt to keep the argument balanced. From Google to Voice of the Shuttle (for humanities research) these ubiquitous gods that enrich the net, can be unfair …and do wear pitfalls. And considering the rate at which the Internet continues to grow, the problems of these gods are only exacerbated further.

Primarily, what you need to digest is the fact that search engines fall short of Mandrake's magic mechanism! They simply don't create URLs out of thin air but instead send their spiders crawling across those sites that have rendered prayers (and expensive offerings!) to them for consideration. Even when sites like Google claim to have a massive 3 billion web pages in its database, a large portion of the web nation is invisible to these spiders. To think they are simply ignorant of the Invisible Web. This invisible web holds that content, normal search engines can't index because the information on many web sites is in databases that are only searchable within that site. Sites like www.imdb.com - The Internet Movie Database , www.incywincy.com - IncyWincy, the invisible web search engine and www.completeplanet.com - The Complete Planet that cover this area are perhaps the only way you can access content from that portion of the Internet, invisible to the search gods. Here, you don't perform a direct content search but search for the resources that may access the content. (Meaning - be sure to set aside considerable time for digging.)

None of the search engines indexes everything on the Web (I mean none). Tried research literature on popular search engines? AltaVista to Yahoo, will list thousands of sources on education, human resource development, etc. etc. but mostly from magazines, newspapers, and various organizations' own Web pages, rather than from research journals and dissertations- the main sources of research literature. That's because most of the journals and dissertations are not yet available publicly on the Web. Thought they'll get you all that's hosted on the web? Think again.

The Web is huge and growing exponentially. Simple searches, using a single word or phrase, will often yield thousands of "hits", most of which will be irrelevant. A layman going in for a piece of info to the internet has to deal with a more severe issue - too much information! And if you don't learn how to control the information overload from these websites, returned by a search result, roll out the red carpet for some frustration. A very common problem results from sites that have a lot of pages with similar content. For e.g., if a discussion thread (in a forum) goes on for a hundred posts there will be a hundred pages all with similar titles, each containing a wee bit of information. Now instead of just one link, all hundred of those darn pages will crop up your search result, crowding out other relevant site.

Regardless of all the sophistication technology has brought in, many well thought-out search phrases produce list after list of irrelevant web pages. The typical search still requires sifting through dirt to find the gold. If you are not specific enough, you may get too many irrelevant hits.

As said, these search engines do not actually search the web directly but their centralized server instead. And unless this database is updated continually to index modified, moved, deleted or renamed documents, you will land yourself amidst broken links and stale copies of web pages. So if they inadequately handle dynamic web pages whose content changes frequently, chances are for the information they reference to quickly go out-of-date. After they wage their never ending war with over-zealous promoters (spamdexers rather), where do they have time to keep their databases current and their search algorithms tuned? No surprise if a perfectly worthwhile site may go unlisted!

Similarly, many of the Web search engines are undergoing rapid development and are not well documented. You will have only an approximate idea of how they are working, and unknown shortcomings may cause them to miss desired information. Not to mention, amongst the first class information, the web also houses false, misleading, deceptive and dressed up information actually produced by charlatans. The Web itself is unstable and tomorrow they may not find you the site they found you today. Well if you could predict them, they would not be god!…would they?! The syntax (word order and punctuation) for various types of complex searches varies some from search engine to search engine, and small errors in the syntax can seriously compromise the search. For instance, try the same phrase search on different search engines and you'll know what I mean. Novices… read this line - using search engines does involve a learning curve. Many beginning Internet users, because of these disadvantages, become discouraged and frustrated.

Like a journalist put it, "Not showing favoritism to its business clients is certainly a rare virtue in these times." Search engines have increasingly turned to two significant revenue streams. Paid placement: In addition to the main editorial-driven search results, the search engines display a second — and sometimes third — listing that's usually commercial in nature. The more you pay, the higher you'll appear in the search results. Paid inclusion: An advertiser or content partner pays the search engine to crawl its site and include the results in the main editorial listing. So?…more likely to be in the hit list but then again - no guarantees. Of course those refusing to favor certain devotees are industry leaders like Google that publishes paid listings, but clearly marks them as 'Sponsored Links.'

The possibility of these 'for-profit' search gods (which haven't yet made much profit) for taking fees to skew their searches, can't be ruled out. But as a searcher, the hit list you are provided with by the engine should obviously rank in the order of relevancy and interest. Search command languages can often be complex and confusing and the ranking algorithm is unique to each god based on the number of occurrences of the search phrase in a page, if it appears in the page title, or in a heading, or the URL itself, or the meta tag etc. or on a weighted average of a number of these relevance scores. E.g. Google (www.google.com) uses its patented PageRank TM and ranks the importance of search results by examining the links that lead to a specific site. The more links that lead to a site, the higher the site is ranked. Pop on popularity!

Alta Vista, HotBot, Lycos, Infoseek and MSN Search use keyword indexes – fast access to millions of documents. The lack of an index structure and poor accuracy of the size of the WWW, will not make searching any easier. Large number of sites indexed. Keyword searching can be difficult to get right.

In reality, however, the prevalence of a certain keyword is not always in proportion to the relevance of a page. Take this example. A search on sari - the national costume of India –in a popular search engine, returned among it's top sites, the following links:

• www.scri.sari.ac.uk/ - of the Scottish Crop research Institute

• movie audiobooks

  Listen On The Net Need information on Internet Radio, audio or entertainment? Look no further - you've found a great repository for all of this information. If you want to listen to music or even to listen to free music just follow our sponsored links and you will find some great information. Listen On The Net

  
  Knowing you're getting value for money is very important in buying movie audiobooks, so by coming to this web site you can rest with the sound knowledge that you are getting the movie audiobooks you paid for.
  
  By buying through our recommend links you can also rest assured your movie audiobooks will be of the highest quality. How do we know? Because when we're shopping for movie audiobooks ourselves it's where we go.
  
  You might wonder why we're suggesting you click on some outside links rather than stay on our movie audiobooks site. Well it's because we've only just put this site up and haven't yet got it fully operational. Our aim is to be the best site for movie audiobooks info on the net, and I'm sure that one day soon we'll achieve it. So please bookmark us and come back soon. Meanwhile please follow one of the links.

  Listen On The Net Index