The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

---

Is New MSN Search More Precise? Just Ask Google

MSN finally unleashed its new search technology to the world... Read More

How Search Engines Work

Before anyone can start optimizing a web site, you must... Read More

The Modern Day Search Engine

The first task most netizens do when they log on... Read More

The Search Doesnt End at Your Homepage

In a recent report for a new client, I wrote:... Read More

Going To Market: Keywords and Backlinks, Part 1

As I waded into the 'Make Money Online' waters on... Read More

Google, Yahoo! and MSN Line up More Goodies for Internet Users

Internet users have never had it better what with Yahoo!,... Read More

Google Groups

Some very early users of the Internet - not the... Read More

How and When Should I Submit My Website to Google?

As soon as you register your domain name, submit it... Read More

SEO Expert Guide - Keyword Analysis (part 3/10)

If you imagine that building an optimized site is like... Read More

Google, Adsense, SEO, and How It All Works

Google uses an algorithm to determine the search engine results... Read More

Search Engine Optimization Tips For 2005 - Part Three

Welcome to part three of our series of articles on... Read More

Search Engine Optimization: What Is It?

Search Engine Optimization is the creation of a web page,... Read More

Search Engine Algorithm Quandaries

Before you make drastic changes to your website after a... Read More

Fixed Point of Reference

According to author Frank Peretti, "A fixed point of reference... Read More

Easy And Simple Steps To Get Listed In Search Engines

Search engines are one of THE best resource for free... Read More

Seecrets on Website Promotion: Search Engine Wars ? a Different Perspective

The objective of all search engine providers is epitomized by... Read More

DMOZ: Rotten To The Core

I knew things were bad at DMOZ. But I guess... Read More

Utilizing Popular Directories as Free Link Sources

If you're a webmaster, you've probably spent almost as much... Read More

SEO -- Tips to Optimize Your Webpage to Compete for a High Ranking

Search engine optimization (SEO) is very important to websites. If... Read More

Whats All This Hype About Links?

What's all the talk about links we hear about? Reciprocal... Read More

Playing in Googlebots Sandbox with Slurp, Teoma, & MSNbot - Spiders Display Differing Personalities

There has been endless webmaster speculation and worry about the... Read More

What are the Benefits Of Submitting To Directories?

Increasing your web traffic and rank in search engines is... Read More

Surviving the Search Wars ? Local Directories

The pursuit of online information has become an increasingly dynamic... Read More

Design A Spider Friendly Site

To be successful in the search engines it's important to... Read More

Attack Of The Killer Google Zombies!

Don't share this with people of a nervous disposition because... Read More

Six Reasons Why Your Alexa Rating Is Still Important

1. Additional Exposure For Your Site.At the end of 2003,... Read More

All about SEO or SFO?

First let's start with definitions:SEO: Search Engine Optimization, SFO: Search... Read More

Are You Making These Deadly SEO Mistakes?

Black Hat SEO: Web Spamming and Linking to Bad NeighborhoodsSo... Read More

Ten Steps To A Well Optimized Website - Step Seven ? Website Submissions

Welcome to part seven in this ten-part search engine positioning... Read More

Five Short Quick Tips on Web Page Titles

By Catherine FranzSearch engine spiders read and record page titles... Read More

Keywords, Choose Them Wisely

By now you have likely heard that keywords and keyword... Read More

Make The Search Engines Love Your Site

Most webmasters have no idea on how to make a... Read More

SEO Tips for Google

Getting a high ranking on Google is a big achievement.... Read More