HomeToolsRobots.txt Generator

CSS WORKSPACE

Robots.txt Generator

Make a change, inspect the result, and take the output into your project.

How to use this tool ↓

Robots.txt Generator

Build a robots.txt file that tells search engines and AI crawlers which parts of your site to index. Use a preset or compose custom rules.

Quick presets

User-agent #1

Site directives

robots.txt119 bytes
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow: /

Sitemap: https://example.com/sitemap.xml
Agents
1
Rules
4
Size
119B

Place robots.txt at your site root (https://yoursite.com/robots.txt). Empty Disallow: means allow all.

A PRACTICAL GUIDE

Control crawling without mistaking it for security

Robots.txt gives cooperating crawlers instructions about which URL paths to request. It is public and is not a password system or a reliable way to remove an already known URL from search results. Use authentication for private content and appropriate indexing controls for public pages.

How to use this tool

  1. Choose the user agent and enter the paths that should be allowed or disallowed. Match the path structure of your own site.
  2. Add your sitemap URL if available. Review the rules carefully so that CSS, JavaScript, and important public pages remain crawlable.
  3. Download the file as robots.txt and place it at the root of the relevant hostname. Check its served contents after publishing.

The asterisk targets crawlers that follow the general rule group. The disallow line asks them not to crawl the preview path. It does not prevent a person from opening that URL, and the file itself reveals the path publicly.

Read the reference documentation ↗
Example to try in your project
User-agent: *
Disallow: /internal-preview/

Sitemap: https://example.com/sitemap.xml

Before you ship

Blocking a URL can prevent a crawler from seeing a noindex instruction on that page. Choose crawling and indexing controls deliberately rather than using robots.txt as a universal removal tool.

Explore a related tool