Learn more about robots.txt

[priya]

Watch on how to block url other than robots.txt and more.

He has come up with this video to answer a couple of questions such as:

- Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?
- How do I make that url disappear from Google?

[Matt]

Watched this the other day. Very informative indeed, and anyone working the industry should watch it, as it gives a basic explanation for certain results that allows any employee to deal with the issue should the client bring it up.

[RayB]

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

[happygarg]

Robots.txt file is a file, which is initially checked by the search engine crawlers (robots), and based on it search engine decides, which pages have to do index, and which pages do not have to index.

Benefits of robots.txt file:
* Robots.txt is useful for saving bandwidth and keeping your logs clean.
* Prevents your system folders and files you don’t wish others to see from appearing in Google search results.
* Is equally helpful for concealing images and duplicate contents from search engines and readers.
* Gives more emphasis to those pages that you allowed to be indexed.
* Prevents server from continuously sending the 404 file not found error to search engines.
* Avoid indexing your duplicate and other important contents that you have preserved for future proposes.
* It is a great SEO tool for Search Engine Optimization and SEO’s