Google, Yahoo, and other Bots.

"Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses."

Web Robots can be used maliciously by spammers but they are also used by search engines to rank your website. Each search engine uses their own bot and you can identify which bots are indexing your website by checking your access logs. You can control which files and directories these bots index and in some cases the speed at which they index by creating a robots.txt file and placing it in your "/html" directory. Using the links below you can experiment and find the right configuration for your robots.txt file. If you are having trouble you can always open a ticket and our support staff would be glad to help you.

Good starting point to learn about bots and robots.txt.

FAQ on Yahoo crawler:

FAQ on Google bots:

  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

What are the Metisentry's DNS Servers?


How do I configure a URL redirect? For example, when my customers type I want them to be taken to

This is referred to as a 301 redirect and can be done by editing the .htaccess file in your root...

What is a good FTP client to use?

There are a number of good FTP programs available freely on the web.    Some examples are WinSCP,...

I lost my (ftp, mail admin, miva, webmaster, mysql, plesk, root, etc) password, how do I retrieve it?

For password resets,  we recommend using the available "Password Reset"  or "Password Recovery"...

How often does Metisentry perform backups? How can I retrieve backed up data?

All Metisentry servers are automatically backed up each night to our backup servers.  From 2-10...