SEO and Website Guides

Robots.txt Guide: Rules, Examples and Safe Setup

By MayreDukaan Editorial TeamPublished 15 Nov 20264 min read

Illustration: a friendly robot at a garden gate reading a signpost, some paths open and one blocked

Learn how robots.txt controls crawling, where to upload it, and why Disallow is not a privacy tool. Build and review clear rules before publishing.

A robots.txt file tells cooperating crawlers which parts of a site they may request. Put it at the root of the relevant website host, and use it for crawl control—not passwords, private files, or a guaranteed way to remove pages from search results.

The robots.txt generator creates a file from crawler rules, allowed paths, blocked paths, and sitemap addresses. It does not upload that file or check your production server for you.

Crawling is different from indexing

Crawling means requesting a resource. Indexing means processing information for possible inclusion in search. Google explains that a blocked URL may still appear in results if other pages link to it, even though Google cannot crawl its contents.

For a public page you do not want indexed, use an appropriate noindex instruction and let Google access that instruction. For genuinely private information, require authentication. A robots file is publicly readable and can reveal the very paths you list.

Goal	Appropriate approach	What not to assume
Reduce crawling of selected paths	robots.txt rules	That blocked URLs vanish from search
Exclude an accessible page from search	A supported noindex instruction	That a blocked crawler can read it
Protect private information	Authentication and access controls	That robots.txt is a security barrier
Advertise a sitemap	A Sitemap line	That discovery guarantees indexing

Where the file belongs

For a site at https://example.com/, the location is https://example.com/robots.txt. A copy inside a folder is not the site’s root robots file. Rules apply to the protocol, host, and port where that file is served; do not assume another subdomain shares them.

Google’s creation guide calls for a plain-text file named robots.txt, using UTF-8. A document editor that saves rich formatting is the wrong tool. Confirm the published address returns your intended plain text rather than a login page or a branded error document.

Understand the main directives

User-agent identifies the crawler group. An asterisk selects the general group. Disallow names paths that group should not crawl; Allow can provide a more specific permitted path. Sitemap supplies the fully qualified address of a sitemap.

For a hypothetical example, choose the general crawler group, block /drafts/, and add https://example.com/sitemap.xml. Leave the public article paths available. This expresses a crawl preference; it does not make the drafts confidential.

A Disallow value of / blocks the entire site for that crawler. The MayreDukaan interface warns when it sees that value. Read the warning before publishing, especially when copying a file from a staging environment.

Create the file step by step

Open the generator and review the default all-crawlers group.
Enter allowed or disallowed paths, one per line, only where needed.
Add another crawler rule if you have a deliberate crawler-specific requirement.
Add your full sitemap address in the sitemap field.
Inspect the generated text and copy or download robots.txt.
Upload it through your hosting or deployment system to the correct root location.
Request the live robots.txt URL and compare its contents with the reviewed file.

The interface includes an optional crawl-delay field, but Google does not support that directive. Do not use it as a Google crawl-rate control. Support by another crawler must be checked in that crawler’s own documentation.

Review rules before changing a live site

Begin by saving the current production text in your normal change-management workflow. Identify the exact problem the new rule is intended to solve. “Improve SEO” is too vague; “stop crawling these duplicate filter paths” is a reviewable objective.

Check whether a blocked folder contains files needed by important pages. Google warns against blocking resources necessary to understand page content. Broad directory rules can have unintended consequences when stylesheets or scripts live below the same prefix.

Use the meta tag analyzer to inspect a fetched page’s robots metadata where accessible. This is separate from testing the site’s robots.txt rules: a meta tag report does not simulate every crawler’s access decision.

Common mistakes

A frequent error is adding noindex to robots.txt. Google does not support noindex in that file. Another is simultaneously blocking a page and expecting Google to read a newly added noindex tag on it.

Be careful with copied path names and capitalization. Google documents case-sensitive paths. A rule aimed at one spelling should not be assumed to cover every differently capitalized URL.

Finally, avoid publishing a list of sensitive administration locations as though the list protects them. Review access controls independently, and keep a rollback copy of any previous robots file.

Keep crawling rules simple

Use the smallest set of rules that serves a clear purpose, then review what is actually deployed. If you also need to publish a page list, follow the XML sitemap guide. Build your draft in the robots.txt generator, and treat publication as a deliberate site configuration change.

Frequently asked questions

Does Disallow remove a page from Google?

Not reliably. A blocked URL can still appear based on links. Use an accessible noindex instruction for search exclusion or authentication for private material.

Where do I upload robots.txt?

At the root of the relevant host, such as https://example.com/robots.txt.

Does Google use crawl-delay?

No. Google does not support crawl-delay in robots.txt.

Useful on MayreDukaan

Sources

Facts last checked against these sources on 15 Nov 2026, 07:43 Pakistan time.

Images: Illustration created for MayreDukaan (not a photo)