What Belongs in a Robots.txt File?

There are plenty of small jobs that sound easier than they feel. You know what the final result should look like, but you do not want to open a full desktop app just to make one change. That is where a focused browser tool can be useful. Instead of learning a large interface, you can work with the few controls that actually matter.

A robots.txt file is a set of crawl instructions for automated agents. It is not a security system and it should not be used to hide private information. A common mistake is blocking a path without thinking about whether search engines need to crawl resources on that path.

Keep the file simple. Start with the pages you genuinely want crawled and add a sitemap URL. After publishing, test your site with care so a broad rule does not accidentally block useful content.

A simple workflow

1. Decide what the finished result needs to do.
2. Use the smallest set of settings that gets you there.
3. Preview the result before publishing, printing or sharing it.
4. Keep the original source when it matters.