What is a robots.txt file?
A robots.txt file is a small text file on your website that tells search engine crawlers which parts of the site they may or may not visit.
Robots.txt is a plain text file that lives at the top of a website, at yourdomain.com/robots.txt. It gives crawlers rules about which folders or pages to skip. It is often used to keep bots out of admin areas or search result pages. Robots.txt controls crawling, not indexing, so it is not a reliable way to hide a page from search results. A mistake in this file can block a whole site.
Why it matters for your business: One wrong line in robots.txt can stop search engines from reading your site. It is worth checking after any redesign or move to a new platform.
Related articles
More in Glossary
Still need help?
Ask our team