Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!
There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.
Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.
Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.
Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.
Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.
If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.
After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.
Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.
In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.
Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:
Google Search - 47%
Yahoo! Search - 21%
MSN Search - 13%
All Others - 19%
These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.
There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.
Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!
Don't forget these key stats as you embark on your optimization journey...
Previous : SEO Expert Guide - Index of Contents
Next: SEO Expert Guide - Proposition Development (part 2/10)
About the author:David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.
Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.
After the latest PR update at Google and MSN's beta... Read More
You've got a website. You've put countless hours into it,... Read More
You need to be extremely careful with keyword research so... Read More
This is the first of a series of articles about... Read More
A good link building strategy has become an essential part... Read More
Sometimes our jobs as marketers means we need to look... Read More
What's all the talk about links we hear about? Reciprocal... Read More
There are methods to increase your search engine rankings which... Read More
The corporate fundamentals are par excellence! The product is unsurpassable... Read More
As soon as you register your domain name, submit it... Read More
A very provocative title, isn't it? Based on the latest... Read More
It's taken you 6-months of hard work, constant changes, reading... Read More
First of all you need to get a blog. If... Read More
6: The hidden benefits to having the right links. Every... Read More
Even though Google Revenues continue to soar, the hidden problem... Read More
Search engine optimization refers to the technique of making your... Read More
The attraction of Pay Per Click (PPC) online advertising is... Read More
Keyword density. When it comes to SEO copywriting, this has... Read More
SEO (search engine optimization) is a booming businees these days.... Read More
The Dream: You wake up one morning and notice your... Read More
The first task most netizens do when they log on... Read More
MSN finally unleashed its new search technology to the world... Read More
Designing a Better System for Search Engines and Information Distribution,... Read More
Before you make drastic changes to your website after a... Read More
Five FAQ about Google PageRank1. What is PageRank and why... Read More
It's unfortunate that many website owners are so hung-up on... Read More
Webmasters can spend most of their waking hours doing everything... Read More
Getting your website listed in the search engines is relatively... Read More
I have 'Googlebot' crawl my site every day like a... Read More
What are Search Engines?There are numerous different search engines, and... Read More
Keywords are indisputably, the single most important element of an... Read More
Being listed in search engines and ranked high on searches... Read More
What's the difference?For those who aren't quite clear what the... Read More
Part I of this article discussed some of the points... Read More
Oops! I meant "web content management system for windows." Do... Read More
One of the important factors in ranking high in search... Read More
It is a well known fact that writing, distributing and... Read More
Search Engine Optimization is a widely misunderstood industry. Many webmasters,... Read More
Online search giant, Google, often tests the waters for new... Read More
Being listed in search engines and ranked high on searches... Read More
Search Engine Optimization (SEO) is a very complex process. It... Read More
When you get an e-mail from SEO Company with content... Read More
If you want to increase traffic to your website and... Read More
To get on Google's top pages can be accomplished by... Read More
Top 10 search engine rankings. Everybody wants it but a... Read More
If I ask you how many web sites do you... Read More
If you imagine that building an optimized site is like... Read More
Welcome to part three in this search engine positioning series.... Read More
The chase for a high web ranking is constantly on... Read More
Before we explore the world of search engine optimization, it... Read More
There's no denying it: Search engines are a dominating force... Read More
About directories:A directory is simply a web site that contains... Read More
Stealth as in Spying. But Knowing What your Top Competitors... Read More
Let me preface this report by citing advertisers in 2004... Read More
There are methods to increase your search engine rankings which... Read More
No doubt, having a high search engine ranking is very... Read More
First of all, the Google directory is really just the... Read More
Keyword SelectionThe most important component of search engine optimization is... Read More
I get half of the world traffic for the term... Read More
There are many techniques that SEM/SEO experts use to optimize... Read More
Google's sweeping changes confirm the search giant has launched a... Read More
There are today search engine and internet marketing services, in... Read More
Anybody who has their own website or is involved in... Read More
Most Search Engine Optimization (SEO) experts agree that links back... Read More
News broke this week that Yahoo has purchased the Inktomi... Read More
Lots of research has focused on inbound links to a... Read More
Search Engine Optimization (SEO) |