Search Engine Robots - How They Work, What They Do (Part II)

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

---

10 Ways To Indirectly Get To The Top Of Search

There are millions of web sites trying to get listed... Read More

Easy Steps to Get Onto Google Top Search Pages

To get on Google's top pages can be accomplished by... Read More

?Web Content Management System fr Window?: Search Engine Typos

Oops! I meant "web content management system for windows." Do... Read More

An Easy Way Not to Get Banned by Google

Strategic search engine optimization involves far more than keyword research,... Read More

New Site and Sandbox: How to Get Rid of It

You have put lot of sweat in making your site.... Read More

Finding Targeted Keyword Phrases Your Competitors Miss

Finding Targeted Keyword Phrases Your Competitors MissFinding keyword phrases your... Read More

Would You Let A Dog Or A Butler Market Your Website?

The latest news to hit the Internet's 'water cooler' is... Read More

Which Search Engine Optimization Services to choose, Google OR Yahoo?

Search Engine Optimization is emerging as the most powerful form... Read More

SEM - Research Measures Success

SEM - Research Measures SuccessSearch engine marketing success comes from... Read More

The Search Doesnt End at Your Homepage

In a recent report for a new client, I wrote:... Read More

How to Use the Google Patent to Get More Traffic

According to the recent release of the Google Patent Application,... Read More

Why Does The Link Page Have A PR Zero?

For beginners, link exchange campaigns can be a minefield of... Read More

Hens Lay Eggs (HLE) by Silke Stahl

"HLE" is a bit of a joke term, (or possible... Read More

1 Simple SEO Strategy To Get More Visitors To Your Site From Google

Did you know that you can dramatically increase the number... Read More

Keyword Targeting Strategy In Your Site

Once the keywords have been decided for the site one... Read More

9 Ways to Keep Google Happy

A recent Google patent application has the SEO community buzzing.... Read More

One Way Link Building Secures Long Term Ranking Results

One-way link building is a great way to improve your... Read More

Link Building - The Waiting Game

Link building is a waiting game. Many clients have asked... Read More

21 Search Engine Terms Every Web Marketer Should Know Part 1

1. Search Engine - Is a database of web sites... Read More

Taking Advantage of Googles Sandbox Effect

Most new sites submitted to Google (at least within the... Read More

How Ive Maintained 7 Top Ten Google Rankings For Nine Months

Back in November 2004 I discovered a way to get... Read More

The Google Strategy

Webmasters across the Internet were totally floored by what happened... Read More

Building Link Popularity with Topical Articles

One of the important factors in ranking high in search... Read More

Google News - Just another article announcer?

In Google's recent battle towards becoming an international news center,... Read More

Shopping Carts and SEO

Shopping and the Web. They go together like Mr. and... Read More

The Smart Way To Get Listed On Google!

So you want to get listed on Google? And you... Read More

Optimizing Online Catalog Copy for the Search Engines

It only makes sense. You have an e-commerce catalog site.... Read More

Submit Your WebSite: Add Your URL to Web Directories & Search Engines

Why get into Search engines and directories?Increase backlinks of your... Read More

Google Ranking WITHOUT Ever Submitting To Google!

A while back, I read an article that explained how... Read More

Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools

In this three part article, you'll find many tools that... Read More

Adding City Names At The End Of Your Keywords Can Bring You More Profits

In recent times, I have been closely studying keywords that... Read More

Attack Smaller Searches To Get The Big Ones!

Searching online can not only be fun, but you sometimes... Read More

Search Engine Optimization With Sitemaps

I just wanted to share a little Search Engine Optimization... Read More