Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

---

A Naive Mistake Cost Me My Google Rankings

Little do you know but you too could be making... Read More

Search Engine Optimization Lies & Misconceptions

In a perfect world, everyone would be honest.In a perfect... Read More

How Google Indexes Content From Your Web Directory

In a fluke, I was able to notice something about... Read More

Search Engine Marketing Hype Killing Small Businesses

Think about the first thing you ever heard about "marketing... Read More

Over Optimization and the OOP - Does a Penalty Exist?

If you have questions about whether or not the Over-Optimization... Read More

Search Engine Optimization and Web Site Usability

Build a Web site and the people will come.Ha! If... Read More

How To Measure Search Engine Marketing ROI

According to the Search Engine Marketing Professional Organization (SEMPO), advertisers... Read More

Maximize Your Search Engine Traffic - 13 Ways to Pull in More Visitors From the Search Engines

Maximizing traffic from the search engines to your web site... Read More

Search Engine Optimization - Enhancing Web Site Visibility

I've had several prospects and clients say to me "I... Read More

DIY SEO - Part 2 Tags n Things

Part 2 Tips 4 tagsMeta tags are the descriptive tags... Read More

Cracking the Google Code: Under the GoogleScope

Google's sweeping changes confirm the search giant has launched a... Read More

Search Engine Marketing 101 For Corporate Sites

When most people want to find something on the web,... Read More

Website Promotion: 10 Search Engine Optimization Blunders to Avoid

If you want to develop a successful search engine optimization... Read More

Diary of a Google Gazumpee

Back in November, when the Google Dance began, Barry Lloyd... Read More

PageRank for Websites: Is There More to the Web?

Google's PageRank has been around for years, and in the... Read More

Do the Robot!

Everyone should realize that the search engines (sponsored ads aside)are... Read More

Search Engines: Tips and Strategies on Getting Listed and Ranking High for Newbies

You've got a website. You've put countless hours into it,... Read More

Things You Must Realize When Searching

For the uninitiated, searching for web pages can seem a... Read More

Yahoos Back!

I was all set to write an article predicting the... Read More

Google Gunning For Directories?

Why is it that webmasters are so quick to blame... Read More

21 Search Engine Terms Every Web Marketer Should Know Part 1

1. Search Engine - Is a database of web sites... Read More

Search Engine Optimization for Beginners

If you are confused about terms like "search engine optimization"... Read More

Ten Steps To A Well Optimized Website - Step Seven ? Website Submissions

Welcome to part seven in this ten-part search engine positioning... Read More

What All Top Google Websites Have in Common

While it is okay to admire and envy all those... Read More

Speed Indexing - 3 Steps to Getting Your Website Listed in Google Quickly

Getting your website listed in Google quickly simply requires that... Read More

Dont Focus Too Much on Your Internet Business Website Ranking

No doubt, having a high search engine ranking is very... Read More

Search Engine Traffic Myths, Time Wasters, and Pitfalls

Everyone wants to increase their rankings with the search engines... Read More

What Constitutes a Complete and Effective SEO Campaign?

Unfortunately, not many Search Engine Optimization companies know what this... Read More

RSS Feeds - a Website Owners Friend in Disguise

We've all heard about it-it seems like all the buzz... Read More

Release from Google Sandbox Only to Search the Playground

The Google Sandbox Effect has been discussed at length in... Read More

How to Get a Website Indexed Fast

Get Indexed FastWhat does getting indexed mean?The search engines keep... Read More

Submitting Your Site To The Open Web Directory: Some Dos And Don?ts

One of the most important steps in any site's publicity... Read More

Google has an Achilles Heal - Will Their Competitors Notice?

Even though Google Revenues continue to soar, the hidden problem... Read More