Paste a robots.txt file, enter a URL path, and pick a crawler. The tester reports whether the URL is allowed or blocked, which rule matched, and why that rule won under Google's longest-path-match precedence. Everything runs in your browser: nothing leaves the page.
Parser notes
Sitemap directives found
Paste a full URL or a path starting with /. Query strings are respected when a $ anchor is present.
Every check below runs on the robots.txt content you paste, against the URL and user-agent you enter. The tester follows Google's public interpretation of robots.txt.
Longest-path-match wins. Ties break to the least restrictive rule, so Allow beats Disallow at equal specificity. The tester reports which rule matched and why.
Patterns like /*.pdf and /*.pdf$ compile the way Googlebot compiles them. The tester distinguishes /file.pdf from /file.pdf?ref=1 when a $ anchor is present.
Exact user-agent matches win over the wildcard group. Googlebot, Googlebot-Image, Bingbot, and a custom user-agent field all route to the correct rule set.
Unknown directives, malformed lines, and empty Disallow lines that quietly allow everything are surfaced as warnings alongside the test result.
Every Sitemap directive is pulled out and listed separately, because Sitemap is not scoped to a user-agent and does not affect path-level allow or block decisions.
Path matching is case-sensitive. The tester will not report /Admin/ as blocking /admin/, which surfaces the mixed-case rule bug that ships to production on real sites.
Semantics
A robots.txt file groups rules under User-agent lines. The tester picks the group whose User-agent line names the crawler exactly (case-insensitive). When no named group matches, the tester falls back to the User-agent: * group. A named group blocks the wildcard group from merging in, so rules must be duplicated when they need to apply to a specific crawler and the wildcard group.
Inside the selected group, every rule whose pattern matches the URL is a candidate. The rule with the longer pattern wins. When two rules tie on pattern length, the tie breaks to the least restrictive rule: Allow beats Disallow. This is the resolution Google implements, and the tester surfaces the winning rule with the reason it won.
The wildcard * matches any run of characters in the URL path. The end anchor $ pins the pattern to the end of the URL string, so Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?ref=1. All other regex metacharacters are literal in a robots.txt pattern: the tester escapes them when compiling.
An empty Disallow line ("Disallow:" with no value) means "allow everything for this user-agent." The tester flags this because the intent is almost always the opposite. Directive names are case-insensitive, but URL paths are case-sensitive, so a rule written for /Admin/ does not block /admin/ on a site that serves the lowercase URL.
These are the failure modes that show up on B2B and industrial sites during SEO audits. Each one is a bug the tester surfaces on the exact URL that triggered it.
Disallow: /Admin/ does not block /admin/. Robots.txt path matching is case-sensitive. Match the exact case the site actually serves, or add both variants.
A bare "Disallow:" with no value means "allow everything for this user-agent." The tester surfaces this as a warning because the intent is almost always the opposite.
Disallow: /*.pdf blocks /file.pdf and /file.pdf?ref=1. Add $ (Disallow: /*.pdf$) when the goal is to block only the exact file, not variants with query strings.
Rules written under User-agent: Googlebot do not apply to Bingbot. When a crawler-specific group exists, the wildcard * group does not merge into it. Duplicate the rules or use the * group deliberately.
Robots.txt blocks crawl, not indexing. A URL that is externally linked can still appear in search results when robots.txt blocks it. Use a noindex meta tag or header on the URL itself instead.
Distinction
Robots.txt tells a crawler whether it can fetch a URL. A Disallow rule stops the crawler from making the HTTP request. The URL can still be indexed when other sites link to it, because the crawler learns the URL exists from those external links and adds a listing to the index without ever reading the page body.
A noindex meta tag or X-Robots-Tag HTTP header tells a crawler that already fetched the URL to leave it out of the index. Noindex requires the crawler to reach the URL. When robots.txt blocks the URL, the crawler never sees the noindex signal, so blocking a URL in robots.txt to remove it from the index is the wrong tool. Serve the URL with a noindex header, and let it be crawled.
The robots.txt tester evaluates a URL path against the Allow and Disallow rules in a robots.txt file for a chosen crawler user-agent. It reports whether the path is allowed or blocked, which rule matched, and why that rule won on longest-path-match specificity. It also flags syntax problems, extracts every Sitemap directive, and warns on an empty Disallow that quietly allows everything.
Allow and Disallow conflicts resolve by longest-path-match: the rule whose pattern has more characters wins. When two rules tie on length, the tester breaks the tie by picking the least restrictive rule, so Allow beats Disallow at equal specificity. This mirrors how Google interprets robots.txt, so the tester result matches Googlebot behavior.
Robots.txt and noindex control different things. Robots.txt blocks crawling: it stops a crawler from fetching a URL. Noindex blocks indexing: it lets the crawler fetch the URL and read the tag that says "do not list this in results." A URL blocked by robots.txt can still appear in Google if the URL is linked from elsewhere on the web, because Google indexes the URL without ever reading the noindex tag.
User-agent matters because robots.txt groups rules by crawler. A file can have one set of rules for Googlebot, another for Bingbot, and a wildcard fallback for every other crawler. The tester selects the exact-name group first (case-insensitive) and falls back to the wildcard group only when no named group matches. Testing the wrong user-agent returns rules that apply to a different bot.
Path patterns are case-sensitive. Disallow: /Admin/ blocks /Admin/ but does not block /admin/. Directive names (User-agent, Allow, Disallow, Sitemap) and user-agent values are case-insensitive, but every character in a URL path is treated as significant. A common bug is a mixed-case Disallow rule that fails to block the lowercase URL the site actually serves.
The wildcard * matches any run of characters, so Disallow: /*.pdf blocks every URL that ends with .pdf anywhere in the path. The end anchor $ pins the pattern to the end of the URL, so Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?ref=1. Combining the two is the standard way to block file-extension patterns without accidentally blocking query-string variants.
Run the tester on your own robots.txt, then tell us what came back. We reply with a straight read on whether the crawl rules are helping or hurting the URLs that generate RFQs.