Robots.txt

What Is Robots.txt? Robots.txt is a text file which users put on their websites to tell search engines which pages on the site should not…