The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

---

Find Best Keywords For Your Site

Keyword optimisation is probably the most important thing that you... Read More

Search Engines The Masters Of The Internet Universe ? Part 1

Trillions of Billions of content pages make up the wide... Read More

Do Search Engines Like Your Web Site?

Between 75% and 98.8% of visitors to Web sites come... Read More

If You Could Submit Just One Page To The Search Engines Which Should It Be?

Listen. Some make submitting pages to search engines sound like... Read More

Google Patent Application - User Data As Part of Ranking Process

In this third article, we continue to dig into the... Read More

Search Engine Optimization :: The Basics and Why Websites Need It

We all know that the most targeted traffic we can... Read More

Most Overlooked Search Engine Optimization Technique

Some of the basics when it comes to SEO and... Read More

Look Out MSN Search, Here Comes Gbrowser

It is official, the search engine wars are in full... Read More

Surviving the Search Wars ? Local Directories

The pursuit of online information has become an increasingly dynamic... Read More

Creating a Google SiteMap For Your Work At Home Business Web Page

Search engine traffic is the best traffic You can get... Read More

Dont Make the Top 30 SEO Mistake

SEO consultants will tell you that you need to be... Read More

Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing... Read More

How Search Engines Work

Before anyone can start optimizing a web site, you must... Read More

A Naive Mistake Cost Me My Google Rankings

Little do you know but you too could be making... Read More

Branding Versus SEO

Branding versus search engine optimization is a marketing dilemma that... Read More

Tread Towards A Successful ?Internet Research?

Internet is a terrific resource containing billions of web pages... Read More

Building Link Popularity As Easy As 123

The most difficult area of SEO is building link popularity.... Read More

Link Reputation - How to Improve Search Engine Rankings

One of the most overlooked strategies to improve search engine... Read More

Google WebSearch? & Google Adsense - Yahoo!

The makers of the world renowned search engine, Google, have... Read More

Seek Engines: What If Seek Had Bumped Out Search?

In the early days of the World Wide Web, when... Read More

7 Search Engine Optimization Strategy

Search engine optimization refers to the technique of making your... Read More

The Matrix Help For Keyphrase Analysis

Keyword analysis is a major part of search engine marketing.... Read More

Building Link Popularity with Topical Articles

One of the important factors in ranking high in search... Read More

Part I : Getting Free Hits Using These Simple Tips & Tricks

Search Engine OptimizationSearch engines still remain the #1 tool to... Read More

Search Engine Optimization: What Is It?

Search Engine Optimization is the creation of a web page,... Read More

SEO Expert Guide - Paid Site Promotion (Marketing) (part 7/10)

In parts 1 - 6 you learnt how to develop... Read More

Search Engine Optimization, Positioning: Is Chasing Google Algorithms Worth It?

Google is the undisputed heavyweight champion of search engines. Most... Read More

What In The World Is This Google Sandbox Theory Thing? And How Do I Beat It?

Ok, so over the past month or so I've been... Read More

Dealing With Search Engine Stress In A Home-Based Business

As a member of several search engine optimization forums, I... Read More

Pay-per-click ? the Ultimate Tool to Boost Affiliate Sales

The old ways are not always the best ways.The traditional... Read More

The Biggest SEO Scam of All

While there are many ethical SEO firms serving Internet users... Read More

Getting One Way SEO Links - The Easy and FREE Way

What we are hunting for?Unique Domains,One Way Links,Text Link copy... Read More

Things You Must Realize When Searching

For the uninitiated, searching for web pages can seem a... Read More