User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
What is robots.txt?
A robots.txt file lives at https://example.com/robots.txt and tells crawlers which pages to index and which to skip. Always include your sitemap URL at the bottom.
Pair with other SEO tools
After generating your robots.txt, build matching XML with the Sitemap Generator, audit on-page tags using the Meta Tag Analyzer, and verify headings with the Heading Checker.
Related free tools
Free, private and instant — no signup needed.
Sitemap Generator
Generate an XML sitemap for your website instantly.
Meta Tag Generator
Generate SEO-optimised meta tags for any page in seconds.
Broken Link Checker
Check any webpage for broken links and 404 errors.
Heading Checker (H1–H6)
Check heading structure and hierarchy on any web page.
Meta Tag Analyzer
Analyze the meta tags of any live URL for SEO issues.