Search Engine Robots - How They Work, What They Do (Part II)

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright � 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

---

9 Steps to Getting Better Search Engine Rankings

You finally have a website and you are ready to... Read More

Ten Steps To A Well Optimized Website - Step 2: Content Creation

Welcome to part two in this search engine optimization series.... Read More

Is a High Search Engine Ranking Important for YOU?

In the process of designing a website for clients the... Read More

SEO #1: Choosing THE Keywords to Optimize for

This is the first lesson out of 6 that teaches... Read More

Search Engines Are Doomed to Fail

A very provocative title, isn't it? Based on the latest... Read More

Google Sitemaps: 7 Benefits You Cant Ignore

Google Sitemaps enables Webmasters to Directly Alert Google to Changes... Read More

SEO = Search Engine Optimization, tips on successful page ranking

One of the key things to remember when developing your... Read More

Secrets on Website Promotion: How You Can Get a #1 Ranking for Your Website Name Within 30 Days

Launching a new website with enough acceleration to rise above... Read More

Search Engines 101 - Search Engines Explained

What Are Search Engines?A search engine is a database system... Read More

Google Love - Five Top Tips To Make Google Love Your Site

1. Google love ...Google and all its programming is not... Read More

Ten Steps To A Well Optimized Website - Step 3: Site Structure

Welcome to part three in this search engine positioning series.... Read More

Get Listed in Google Without Submitting Your Site

With Google delivering so much traffic, it is only normal... Read More

How To Find Good Keywords

Good keywords are frequently searched for (high demand) but not... Read More

Yahoo Dopey, MSN Goofy, Google is Mickey Mouse Lost in a Sandbox

Seventy-two days ago Googlebot first showed up and crawled over... Read More

Release from Google Sandbox Only to Search the Playground

The Google Sandbox Effect has been discussed at length in... Read More

Link Popularity

In today scenario when we talk about Search Engine Optimization,... Read More

How To Indirectly Get To The Top Of Search Engines

There are millions of web sites trying to get listed... Read More

Everything You Wanted To Know About Google -- But Were Afraid To Search For!

(A Reflective look at the little search engine that soared!)All... Read More

Website Ranking With an Internet Marketing Specialist

On the internet, competition is stronger than ever. There was... Read More

Googles PR System Explained

The complexities of Google's PR (Page Ranking) System have grown... Read More

Why You Need a SEO Maintenance Plan

A search engine optimization maintenance service plan will ensure that... Read More

Search Engine Marketing: 20 Nitty-Gritty Strategies To Compel People To Link To Your Website

You don't have to be a rocket scientist to know... Read More

Screwed: Is this an inevitability in the SEO World?

By about 2pm everyday, each of my team members has... Read More

How to Improve Your Search Engine Positioning and Increase Traffic Today

Every website has times when traffic is higher than others.... Read More

Google Has No Content On Page

One of the post from My SEO world Meet Up... Read More

Page Rank Purgatory - Simple Things You can Do to Keep Your Web Site Out of Search Engine Hell!

Are Meta Tags Really Dead?Right in their Guidelines Yahoo Tells... Read More

Do Search Engines Like Your Web Site?

Between 75% and 98.8% of visitors to Web sites come... Read More

Arrogant Overture Placing Greed Ahead Of Their Customers Needs

According to the dictionary, the definition of the word "overture"... Read More

Link Building: To Link, or Not to Link, That is the Question

Lately, there have been a lot of heated discussions regarding... Read More

The Google Sandbox Explained

Introduction The Google Sandbox is a metaphorical term... Read More

7 Simple Steps to Spy on Your Online Competition and Acheive a High Page Rank

My Grandfather ran a small Grocery Store and when you... Read More

Here Today Gone Tomorrow

Its a matter of here today, gone tomorrow at the... Read More

What is Google Talk ?

The advent of Internet has affected lives of almost the... Read More