Robots.txt tells search engines what to crawl—or skip. Learn how to create, test, and optimize robots.txt for better SEO and site management. Robots.txt is a text file that tells search engine ...