SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

About the author:

David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

---

Why Pay-Per-Inclusion Search Engines are Dying

A Pay-Per-Inclusion search engine is a service in which a... Read More

Importance of Keywords in Anchor Text or Title Text

Keywords are indisputably, the single most important element of an... Read More

Developing A List Of Keywords For Marketing

Keywords aren't just some words that allow search engines, like... Read More

The Two Most Important Things You Must Do For Google Top Ranking

Attaining a top ranking in Goggle or any other major... Read More

Recommended Tools When You Put On The SEO Cap For Your Web Site

The name of the game is search engine optimization and... Read More

An Introduction to Google Sitemaps

... and why I 'm dying to get finally in... Read More

Search Engines: Tips and Strategies on Getting Listed and Ranking High for Newbies

You've got a website. You've put countless hours into it,... Read More

How Do I Submit My Site To Search Engines?

You've built up a good site, put in a lot... Read More

How to Avoid the Google Duplicate Content Filter?

More and more webmasters are building websites with publicly available... Read More

Here Today Gone Tomorrow

Its a matter of here today, gone tomorrow at the... Read More

Link Building Services

In today scenario when we talk about Search Engine Optimization,... Read More

SEO Tips - Google Has Changed - Learn Why And What To Do

Google now checks the year your domain name was first... Read More

New Free Online Tool for Keyword Research

I came across a new website the other day and... Read More

Linking Strategies to Skyrocket to the Top of Google

If you don't know already, one of the key success... Read More

Commonly SEO terms and Jargon used by Search Engine Gurus

What does all this SEO jargon mean.Have you ever been... Read More

How to Make More Money with Your Mambo 4.5.1 Site

For those of you who are not familiar... Read More

Fixed Point of Reference

According to author Frank Peretti, "A fixed point of reference... Read More

Anchor Text Optimization

Anchor Text (also called phrase linking) can significantly improve your... Read More

Google Rank Cake

6 cups thick content mix 1 jar word of mouth,... Read More

Basics to Search Engine Optimization

Search Engine Optimization popularly known as SEO.SEO is the process... Read More

How to Google; or How to be Easily Distracted

I set out with the intention of writing a self... Read More

The Disease of Google Banning: Are You At Risk?

Definition:Google banning occurs when Google removes your site from its... Read More

Keyword Targeting Strategy In Your Site

Once the keywords have been decided for the site one... Read More

PageRank for Websites: Is There More to the Web?

Google's PageRank has been around for years, and in the... Read More

How To Rank High On MSN Search

The new MSN Search is quickly gaining popularity among internet... Read More

.com Not Listed in Regional Yahoo? Don?t Despair!

If you're a non-American business with a .com web address,... Read More

Internet Marketing and SEO

Have you ever seen any email offers of getting you... Read More

Link Popularity: Why Its The Best Investment You Can Do For Your Business

More and more search engines rank your web pages based... Read More

Mindset from Yahoo!- Changing the SEO Game

Yahoo! has a developed a new concept in search engine... Read More

How to Get the Best Deal on Your SEO Project

If you own or manage a business Website, chances are... Read More

3.5 Tips To Help You Avoid Becoming The Next Search Engine Outlaw

Want to avoid being blacklisted by the search engines and... Read More

Speed Indexing - 3 Steps to Getting Your Website Listed in Google Quickly

Getting your website listed in Google quickly simply requires that... Read More

Quick Keyword Research that Works!!!

Keywords are terms or words related to particular topic. Keyword... Read More