SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

About the author:

David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

---

Yahoos Back!

I was all set to write an article predicting the... Read More

MLM and SEO - Bad Business! No Business!

MLM has been around way before the Internet. It is... Read More

Designing a Website So the Search Engines Will Like You

Before you go and spend big money on a professional... Read More

Pay Per Click Versus ?Organic? Search Engine Listings - The Pros and Cons and Best Uses for Each

What's the difference?For those who aren't quite clear what the... Read More

Keyword Ownership: What It Is And Where Its Headed

Have you ever got one of those silly emails that... Read More

Google, Adsense, SEO, and How It All Works

Google uses an algorithm to determine the search engine results... Read More

Is New MSN Search More Precise? Just Ask Google

MSN finally unleashed its new search technology to the world... Read More

Link Popularity Explained and How To Build Links

Link popularity is the single most influential factor for determining... Read More

Microsofts New Search Engine

I tried out Microsoft's new search engine (beta version) the... Read More

Google Love - Five Top Tips To Make Google Love Your Site

1. Google love ...Google and all its programming is not... Read More

Diary of a Google Gazumpee

Back in November, when the Google Dance began, Barry Lloyd... Read More

How to Use the Google Patent to Get More Traffic

According to the recent release of the Google Patent Application,... Read More

Get Traffic You Need - Make Your Links Work

So you have built a nice web site with good... Read More

Whats Link Popularity?

Link popularity is just one of the ways you can... Read More

What All Top Google Websites Have in Common

While it is okay to admire and envy all those... Read More

Link Building: To Link, or Not to Link, That is the Question

Lately, there have been a lot of heated discussions regarding... Read More

Does The Number Of Links On A Page Affect Ranking?

Lots of research has focused on inbound links to a... Read More

Google Tests Expanded Search To Include Printed Works

Google Labs is currently testing Google Print, which returns results... Read More

How to REALLY Profit from SEO

I want to give you a few more things to... Read More

How to Build Massive Keyword Lists

As keyword marketing becomes more and more expensive and competitive,... Read More

Keyword Density And How To Use It To Keep Traffic Flowing To Your Site!

Generating high traffic to your web site can be costly,... Read More

Simple Way to Get High Pagerank

Everyone knows that if your website has a high Google... Read More

Why Optimize Your Site For Search Engines?

Sometimes a search engine optimization company will miss that glaring... Read More

Why Search Engine Marketing Has A Passion for Web Site Usability

Watching a recent football game, I imagined two very different... Read More

Onpage Optimization: Essential for Effective Offpage Optimization

Onpage optimization is the process by which various elements on... Read More

Submit Your WebSite: Add Your URL to Web Directories & Search Engines

Why get into Search engines and directories?Increase backlinks of your... Read More

Search Engine Optimization :: The Basics and Why Websites Need It

We all know that the most targeted traffic we can... Read More

A Three Day Marketing Plan for Better Google Rankings

If you're reading this article, you've probably discovered that simply... Read More

Optimize your Search Engine Placement Five Easy Ways

If you are like me, you created web site metatags... Read More

The Search Engine Showdown

If you're anything like me, you have a favourite search... Read More

SEO Blues

SEO, not again!, you may groan. The webmaster world is... Read More

Things You Must Realize When Searching

For the uninitiated, searching for web pages can seem a... Read More

Creating Search Engine Friendly Web Sites

With tons of competition and copycats online, you need a... Read More