SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

About the author:

David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

---

Dont Focus Too Much on Your Internet Business Website Ranking

No doubt, having a high search engine ranking is very... Read More

How Search Engines Connect Sellers and Buyers

Maggie knows how to find what she wants. She lets... Read More

What are My Chances to Get the First Place in Search Engine Listings?

You must have heard the stories how people became rich... Read More

How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag

Duplicate content is one of the problems that we regularly... Read More

How Search Engines Work

Before anyone can start optimizing a web site, you must... Read More

Easy Steps to Get Onto Google Top Search Pages

To get on Google's top pages can be accomplished by... Read More

Keywords: The First Step To Recognition

Open Wordtracker [ http://www.wordtracker.com/ ] and you'll see... Read More

Put the Full Power of Google to Work with 11 Google Power Search Tips

Google has many ways to help you find want you... Read More

Keywords, Choose Them Wisely

By now you have likely heard that keywords and keyword... Read More

Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing... Read More

Whats the Big Hurry?

Quite often I get asked what the magic solution is... Read More

How To Get Your Web Site Banned ? or Search Engine Promotion, How Not to do it!

Search engines constantly strive to advance their technology and algorithms... Read More

How To Find Good Keywords

Good keywords are frequently searched for (high demand) but not... Read More

The Smart Way To Get Listed On Google!

So you want to get listed on Google? And you... Read More

Search Engine Traffic Myths, Time Wasters, and Pitfalls

Everyone wants to increase their rankings with the search engines... Read More

The Best and Easiest Google-Friendly Change to Your Web Site

No matter who you are or how much you pay... Read More

Maximising Google?s PageRank of Your Website to Maximise Traffic

Google uses PageRank to rank your pages. To maximize your... Read More

Why You Should NOT Submit Your Site On Search Engines

Before to answer to this question we have to know... Read More

Search Engine Marketing (SEM) - Houses on Sand

Do you depend on free search engine traffic for your... Read More

An Introduction to Google Sitemaps

... and why I 'm dying to get finally in... Read More

How to Create Killer Keywords to Get FREE Traffic to Your Website

When exploring good keyword choices there are several steps one... Read More

Gaining Additional PageRank

Google is the major search engine webmasters have to deal... Read More

Search Engine Optimization - A Beginners Guide

Getting your site listed in the top search engines, such... Read More

How to Succeed with the Search Engines

The Cold Hard Facts?..One of the most important factors in... Read More

Google Page Rank Is Dead - Part II

In part I - Google Page Rank Is Dead -... Read More

When Google PR Tool Bar is Down!

Where is the need to worry? Who needs a PR... Read More

Getting Listed in the ODP, Google Directory

First of all, the Google directory is really just the... Read More

What Makes The Perfect SEO Firm?

SEO companies come in all shapes and sizes. You've got... Read More

SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it... Read More

The Google Strategy

Webmasters across the Internet were totally floored by what happened... Read More

Achieving Better Search Engine Optimization

The search engine giants are locked in an all out... Read More

SEO Tips - Google Has Changed - Learn Why And What To Do

Google now checks the year your domain name was first... Read More

Which SEO Company/Firm to Choose for SEO Services?

In the last 2-3 years many new companies have mushroomed,... Read More