How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag

Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it's a problem that is easily rectified.

Your primary weapon of choice against duplicate content can be found within "The Robot Exclusion Protocol" which has now been adopted by all the major search engines.

There are two ways to control how the search engine spiders index your site.

1. The Robot Exclusion File or "robots.txt" and

2. The Robots < Meta > Tag

The Robots Exclusion File (Robots.txt)
This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. www.yourwebsite.com/robots.txt. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site's content.

The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.

Creating your robots.txt file

Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:

User-agent: *
Disallow:

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. By leaving the "Disallow" blank all parts of the site are suitable for indexing.

Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.

Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation

By naming the particular search spider in the "User-agent" you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.

That's all there is to it!

As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it's probably necessary to use a combination of the robots.txt and the robots tag.

The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the section of a web page. A simple example would be as follows;

In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.

In this second example I don't want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the "noarchive" directive.

What could be simpler!

Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.

Should you require further information about our search engine marketing or optimization services please visit us at http://www.e-prominence.co.uk ? The search marketing company

---

Yahoos Back!

I was all set to write an article predicting the... Read More

Search Engine Indexing - 3 Strategies Guaranteed to Skyrocket Your Success

In order to design a website that performs well with... Read More

How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag

Duplicate content is one of the problems that we regularly... Read More

Search Engine Wars - Quality Searches Vs Quantity!

It is no secret that Google and Yahoo are on... Read More

The Biggest SEO Scam of All

While there are many ethical SEO firms serving Internet users... Read More

Martial Arts Webmasters: Time to Optimize Your Site!

A few months ago I was looking through the search... Read More

Google?s Siren Call ? Is It Crashing Your Search Engine Marketing?

It's difficult to dispute the rational behind the rant since... Read More

All About Links -- Interview With Link Building Expert , Bob Gladstein

Julia: Welcome Bob. Thank you for taking the time to... Read More

Search Engine Visibility - The Mantra of Corporate Profitability

The corporate fundamentals are par excellence! The product is unsurpassable... Read More

Introduction to Google Page Rank (PR)

For anyone looking to enhance their Google Page Rank (PR)... Read More

How To Boost Your Keyword Density On Your Web Site To Gain Top Positions At The Search Engines

Let's talk about what keyword density is and how to... Read More

Are You Losing The Battle For Search Engine Traffic?

Search engine traffic should be a priority for any online... Read More

Ten Steps To A Well Optimized Website - Step Seven ? Website Submissions

Welcome to part seven in this ten-part search engine positioning... Read More

Sales And Crawlers, Update! Update! Update!

The importance to the algorithmic web crawlers that speed throughout... Read More

Arrogant Overture Placing Greed Ahead Of Their Customers Needs

According to the dictionary, the definition of the word "overture"... Read More

What Constitutes a Complete and Effective SEO Campaign?

Unfortunately, not many Search Engine Optimization companies know what this... Read More

SEO India: Tips & Techniques

SEO is all about "Individualism"According to me SEO is not... Read More

How Important is PageRank, Really?

Webmasters can spend most of their waking hours doing everything... Read More

Banned from Google and Wondering Why?

There are those that get on the computer one night... Read More

Googles New SEO Rules

Google has recently made some pretty significant changes in its... Read More

Monitor Your Visibility in Google, MSN, and Yahoo with these DIY SEO Tools

This is the second part of an article series in... Read More

How To Conceal Your Website?s ?Fingerprint? From The Search Engines

The internet marketing industry is now flooded with various page... Read More

Keep Your Content Fresh with this Quick and Easy Tip

Some of the Search engines want only original content. So... Read More

Duplicate the Exact Steps Used to Get a Number 1 Yahoo Ranking in Less than 30 Days

If you have ever been into a McDonalds you will... Read More

Are Your SEO Efforts Going To Waste?

Search engine optimization (SEO) is a long and complicated process... Read More

Tops In Toolbars?

Most internet marketers are aware of, and probably use, the... Read More

Get More Cosmetic Surgery Patients From The Web

It's no surprise that dominant cosmetic surgery practices also have... Read More

Beating the New Google AdWord Rules with Blogs and RSS

When Google Adwords first came on scene, it was not... Read More

How to Use Keywords to Improve Your Ranking in Search Engines

The importance of keywordsKeywords or key phrases you choose will... Read More

Now More Than Ever...You Need To Optimize Your Site for Search Engines

As the economy begins to recover in certain parts of... Read More

Search Engine Optimization: What Is It?

Search Engine Optimization is the creation of a web page,... Read More

Twelve Steps to Higher Search Engine Placement

Recent studies suggest that more than 80% of new visitors... Read More

Search Engine Optimization With Sitemaps

I just wanted to share a little Search Engine Optimization... Read More