What robots.txt is for
Robots.txt is a plain text file that sits at the root of your site, at your domain followed by /robots.txt, and asks crawlers which parts of the site they should and should not fetch. It is a set of instructions for well-behaved robots, grouped by user agent: a User-agent line names the crawler, then Disallow and Allow lines list the paths the rules cover. Search engines check it before crawling, so it is the standard way to keep bots out of admin areas, endless filtered category pages, internal search results and other corners that waste crawl budget without helping anyone. Like the other tools on this page's shelf, it helps crawlers deal with your site sensibly; it does not improve rankings by itself.
The file is built entirely in your browser and downloaded straight from it. Nothing you type is uploaded anywhere.
What robots.txt is not
It is not security. The file is public by design, so anyone curious can read yours in a browser, and listing /secret-reports/ in it is closer to a signpost than a lock. Compliant crawlers honour the rules; scrapers, vulnerability scanners and anyone with bad intentions simply ignore them. If content is sensitive, put it behind authentication.
It is also not a noindex. Blocking a page stops Google fetching it, but if other sites link to that page it can still be indexed and shown in results as a bare URL with no snippet. Worse, because the page cannot be crawled, Google cannot see a noindex tag on it either, so blocking a page you want removed from search is actively counterproductive. The reliable route is the opposite: let the page be crawled and give it a noindex robots meta tag, or remove it, or protect it with a login.
The crawl-delay myth
Crawl-delay is the directive everyone copies from old tutorials and Google has never supported. Googlebot ignores the line completely, so adding Crawl-delay: 10 does nothing to the crawler most sites care about. The myth persists because Bing and Yandex do honour it, so it is not useless, just widely misunderstood. If Googlebot is genuinely straining your server, the supported answers are Search Console's crawl settings and serving 503 or 429 responses when overloaded, which Googlebot treats as a signal to slow down.
The sitemap line and a few sharp edges
The Sitemap line is the most quietly useful part of the file: it points crawlers at your XML sitemap, it must be a full absolute URL including https, and unlike the other directives it applies to all crawlers regardless of which User-agent group it sits near. You can list more than one. A few details worth knowing before you upload: the filename must be lowercase and the file must sit at the root, because a robots.txt in a subfolder is never read; each subdomain needs its own file; paths are case-sensitive, so /Private/ and /private/ are different rules; and when Allow and Disallow rules conflict, Google follows the most specific match, which is exactly how the WordPress preset above works. After uploading, fetch your domain followed by /robots.txt in a browser to confirm it is live, and use Search Console's robots.txt report to see the file as Google sees it.
Frequently asked questions
Does robots.txt stop a page appearing in Google?
No, and this catches a lot of people out. Robots.txt stops compliant crawlers fetching a page, but if other sites link to that page it can still be indexed and appear in results as a bare URL with no description. To keep a page out of the index, let it be crawled and add a noindex robots meta tag, or put it behind a login.
Is robots.txt a security measure?
No. The file is public, so anyone can read yours at your domain followed by /robots.txt, which means listing a private path actually advertises it. Well-behaved crawlers follow the rules; scrapers and attackers simply ignore them. Anything genuinely sensitive needs authentication, not a polite request.
Does the crawl-delay directive work?
Not for Google, which has never supported crawl-delay and ignores the line entirely. Bing and Yandex do respect it, which is how the myth survives. If Googlebot is hitting your server too hard, the supported routes are Search Console's crawl rate tools and returning 503 or 429 responses when you are overloaded.
Where does the file go?
At the root of the host, so https://example.co.uk/robots.txt, with the filename in lowercase. Each subdomain needs its own file, because the one on www.example.co.uk does not cover shop.example.co.uk. Crawlers only ever look at the root, so a robots.txt in a subfolder does nothing.
Is anything I type uploaded?
No. The file is built entirely in your browser with JavaScript and downloaded straight from it, so your rules and sitemap URL never leave your device.