Robots.txt Definition: What It Is & How It Works

Robots.txt is a plain text file placed in a website’s root directory that tells search engine crawlers which parts of the site they are allowed to crawl and index. You create it, upload it, and the file gives you a simple, standardized way to control crawler access without needing to block them at the server level.

Formula

The robots.txt file itself has no mathematical formula, but the pace calculator above uses two standard formulas. Pace is the time divided by the distance: pace (min/km) = time (min) / distance (km). Speed is the distance divided by the time in hours: speed (km/h) = distance (km) / (time (min) / 60).

Variables:

• t – finish time in minutes
• d – distance in kilometers
• pace – average minutes per kilometer
• speed – average kilometers per hour

Worked Example

Using the default values from the calculator: finish time = 50 minutes, distance = 10 km.

1. Pace = 50 / 10 = 5.00 min/km
2. Speed = 10 / (50 / 60) = 10 / 0.8333 = 12.00 km/h

The calculator shows 5.00 min/km and 12.00 km/h, matching the manual calculation.

What Robots.txt Actually Does

Robots.txt is part of the Robots Exclusion Protocol, a de facto standard introduced in 1994. It is not a security tool and it does not hide pages from people. It only governs automated bots that choose to obey it. Major search engines like Google, Bing, and DuckDuckGo respect robots.txt, but a malicious bot can simply ignore it.

The file lives at the root of your domain, for example https://example.com/robots.txt. It contains one or more groups of rules, each starting with a User-agent line that names the bot, followed by Allow and Disallow directives. A simple example:

User-agent: *
Disallow: /private/
Allow: /public/

This tells all crawlers not to access anything under /private/ but to allow /public/. The order of Allow and Disallow matters in Google’s implementation: the most specific rule wins, and if two rules are equally specific, the Allow wins.

How to Create and Test a Robots.txt File

Creating a robots.txt file is straightforward. Use any text editor, write the rules, save the file as robots.txt (all lowercase), and upload it to the root directory of your domain. The file must be accessible at the top level, not inside a subfolder.

Common steps:

• Open a plain text editor (Notepad, TextEdit, VS Code).
• Write your rules, one per line.
• Save with UTF-8 encoding, no BOM.
• Upload via FTP or your hosting file manager to the public_html folder.
• Visit yourdomain.com/robots.txt to verify it loads.

After uploading, test it with a tool like Google’s Robots Testing Tool (now part of Search Console) or simply inspect the file in a browser. Also check that you did not accidentally block your own CSS or JavaScript files, which can harm rendering.

Choosing What to Block

Most sites do not need to block anything. The default is that crawlers can access everything they can find links to. Blocking is useful for a few specific cases:

• Duplicate pages like printer-friendly versions or faceted navigation.
• Private admin areas or staging environments.
• Search result pages that would create infinite loops.
• Large files like PDFs that you do not want indexed.

Do not block CSS or JavaScript, because search engines need them to render pages properly. Also avoid blocking entire sections unless you are sure you never want them indexed. If you want a page out of search results entirely, use a noindex meta tag instead, because robots.txt only prevents crawling, not indexing if the page is linked from elsewhere.

Common Mistakes and Misconceptions

One frequent error is using Disallow: / on a live site, which blocks all crawlers and effectively removes the site from search results. Another is putting the file in the wrong directory or naming it robots.txt.txt. A third is forgetting that robots.txt is public: anyone can view it, so never put sensitive paths there that you actually want to keep hidden.

A widespread misconception is that robots.txt is a security measure. It is not. It is a polite request. If you need to protect data, use authentication or server-side permissions. Also, robots.txt does not affect how your site appears to human visitors; it only affects bots that follow it.

Robots.txt vs. Meta Robots vs. Sitemap

Robots.txt controls crawling at the directory or file level. Meta robots tags control indexing at the individual page level. A noindex meta tag tells a crawler not to show the page in results, but the crawler still needs to fetch the page to see the tag. Sitemaps are a separate XML file that lists URLs you want crawled; they complement robots.txt by helping discovery.

Use robots.txt to prevent crawling of non-essential pages, use meta noindex to prevent indexing of specific pages that are already linked, and use a sitemap to point crawlers to your high-value content. They work together, not as replacements.

Costs and Maintenance

Creating and maintaining a robots.txt file costs nothing in money. It is a few lines of text. The real cost is in upkeep: when you change your site structure, move directories, or add new sections, you must update the file. A stale robots.txt can accidentally block new content or allow crawling of pages you meant to hide.

Check your robots.txt periodically, especially after major redesigns. Use your search console’s coverage report to see if any pages are unexpectedly blocked. Also verify that your Sitemap: line (if you include one) points to the correct sitemap URL.

Frequently Asked Questions

Does robots.txt stop Google from indexing my page?

No. Robots.txt prevents crawling, not indexing. If a page is blocked by robots.txt but is linked from other sites, Google may still index the URL without seeing the content. To remove a page from search results, use a noindex meta tag or remove the page entirely.

What does “User-agent: *” mean?

It means the following rules apply to all crawlers. You can also specify a particular bot, like User-agent: Googlebot, to give different rules to different bots. The asterisk is a wildcard for any bot.

Can I use robots.txt to hide my site from everyone?

You can block all well-behaved crawlers with Disallow: /, but people can still visit your site directly. If you want to keep the site private, use password protection or server-level access controls instead.

What is the difference between Allow and Disallow?

Allow permits a specific path to be crawled, while Disallow blocks it. In Google’s matching, the longest matching rule wins, and ties go to Allow. Other search engines may follow a simpler first-match rule, so be explicit.

Is robots.txt case-sensitive?

Yes, the paths are case-sensitive on most servers. /Private/ is different from /private/. Use consistent lowercase paths to avoid confusion.

Where should I put the Sitemap line?

You can put the Sitemap: directive anywhere in the file, but it is conventional to place it at the end. It is not part of any user-agent group; it applies globally.

Related

• Office Tab Feature Demo: What It Does
• How to Merge Two iTunes Libraries
• Document an Endangered Language with Community Consent

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0 | 100000
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2