Robots.txt FAQ

How much do you know about the Robots.txt file?  It can be a handy tool for any webmaster to know more about.  Here are some of the most frequently asked questions I get about this special little file.
What is the robots.txt file used for?
In web site development, the robots.txt file is used as a [...]