|
Robot Manager
Pro
Reviewed
by: Robin
Nobles, send
e-mail
Published
by: Website
Management Tools, go
to the web site
Requires: Windows
98se/Me/NT4/2000/XP
MSRP: $129
(free trial version available)
Have
you ever wondered how far down into your site the search
engine spiders are going? Are the pay inclusion spiders
coming at regular intervals like they're supposed to? Wouldn't
it be great to see which spiders are visiting your site
and which pages they're crawling . . . at a glance? Robot
Manager Pro is undoubtedly one of the most helpful software
programs I've seen in a long time. It actually serves two
purposes:
- It
will create robots.txt files for you;
- It
will monitor spider activity to your Web sites.
What
is a Robots.txt File? It's a text file that "disallows" or
keeps spiders out of certain pages or areas of your web
site. For example, if you have a section of your site that's
devoted to your personnel, you can keep the spiders from
finding and crawling that entire section by including it
in a robots.txt file.
If you've
ever tried to create a robots.txt file by hand, you know
that it's nothing short of tedious, and it's so easy to
make a mistake. So having a program create the file for
you automatically is ideal. However, what's really exciting
to me about Robot Manager Pro is how it monitors spider
traffic. But first, let's look at the software as a whole.
The program is simple to use. On the left-hand side of
the screen, you'll see numbered areas. On the right-hand
side of the screen, you'll see a Quick Help section that
explains how to use that portion of the program.
What you're doing in steps #1-3 is creating your robots.txt
file. You'll tell the program which areas or pages that
you want to keep the spiders from crawling and the software
will create the robots.txt file for you and upload it to
the root directory of your Web site. Keep in mind that
you don't have to create a robots.txt file if you don't
want or need to. You can simply use the spider monitoring
portion by itself.
As
I stated previously, the most exciting part of the program
to
me is the way it monitors spider traffic. The
first time you download your log files to analyze your
traffic, it won't be quite as interesting as it gets each
additional time you use the program. Every time you use
the program, you'll open the initial file you created,
hit "refresh" to download the latest log files,
and let the program update your spider visits. If we were
to click on one of the spiders, such as Googlebot, we'd
see how many pages from this site that Google has spidered.
Google is a deep crawling engine and it really can find
most of your web pages if you provide text links to them.
If we were to click on any of the actual web pages, we'd
see a screen that tells us how often the spider has visited
the page.
Can you see the value in having this type of information?
- If
an engine hasn't found one of your important pages,
maybe you need to add more links to that page from
pages that are getting spidered more frequently.
- When
you create new pages, you'll be able to see at a glance
when the spiders find and crawl those pages.
- If
you use pay inclusion, you'll be able to monitor whether
the spiders are visiting like they're supposed
to.
- You'll
know exactly which pages of your Web site are being
spidered by which search engines and when.
If you're interested in tracking spider visits to your
site, Robot Manager Pro is a must-have program for you.
It's fast and easy to use, and it provides key information
about how the spiders are crawling your Web site. Website
Management Tools also offers a regular version of Robot
Manager, but it doesn't monitor spider traffic.
Letters to the Editor are welcome and occasionally abused in public. Send e-mail to: whine@kickstartnews.com
|
