Robots.txt tester: can search engines crawl this page?
A robots.txt tester is a tool that checks whether a site's robots.txt file lets a search engine or AI crawler crawl a given page. Type a page address and pick a crawler: the tester reads the file, finds the rule that decides and tells you whether the page may be crawled.
This tool checks one thing. The AI Website SEO Checker checks your whole website: 23 checks, every public page, three pages read in depth and the keywords you can win.
Check your website SEO with AI →What is robots.txt for?
Robots.txt is a plain text file at the root of a website (yourbusiness.com/robots.txt) that tells crawlers which parts of the site they may visit. Crawlers read it before anything else. It is a request, not a lock: the standard itself says these rules are not a form of access authorization. Well-behaved crawlers like Googlebot and Bingbot follow it, while bad bots ignore it, so never rely on it to hide private pages.
How does this tester decide whether a page may be crawled?
It applies the rules Google publishes: find the crawler's group, find the rules that match the address, and let the most specific matching rule decide.
- It finds the right group. Robots.txt is split into groups, each starting with one or more User-agent lines. A crawler follows the group whose name matches it most closely (Googlebot-Image follows a Googlebot-Image group, then a Googlebot group, then the group for every crawler, *).
- It finds the rules that match the address. A rule matches when the address starts with its path. A star (*) stands for any text and a dollar sign ($) marks the end of the address.
- The most specific rule wins. Google uses the most specific rule, based on the length of its path. When an Allow and a Disallow are equally long, Allow wins. No matching rule means the page may be crawled.
These are the rules Google publishes and follows, and the same ones written into the Robots Exclusion Protocol standard (RFC 9309), so the answer here is the answer Googlebot gets.
What does this tester not check?
It answers one question, whether a crawler may fetch the address, so it cannot tell you whether the page is indexed or where it ranks. A page that may be crawled can still be kept out of results by a noindex tag, and a blocked page can still be listed. Bing, AI crawlers and others mostly follow the same rules, but each may read unusual cases its own way, and some ignore robots.txt altogether. A Crawl-delay line is reported but not applied, since Google ignores it. The tester reads the robots.txt of the exact address you type, and http and https, or www and non-www, can each have a file of their own. Like Google, it reads only the first 500 KB.
What mistakes do people make with robots.txt?
The common ones are blocking the whole site by accident, blocking CSS and JavaScript, using robots.txt to remove pages, and a robots.txt that answers with a server error.
- Disallow: / left over from the development site, which shuts the whole website out of search.
- Blocking CSS and JavaScript folders, so Google cannot render the page the way visitors see it.
- Using robots.txt to remove a page from Google. Google says robots.txt is not a mechanism for keeping a web page out of Google: a blocked page can still appear, without a description, if other sites link to it. Use a noindex tag instead, and let the page be crawled so Google sees it.
- A robots.txt that answers with a server error. Google then pauses crawling the whole site.
Robots.txt is also the best place to name your sitemap with a Sitemap line. Check it with the sitemap checker, and see how Google reads your whole site with the AI Website SEO Checker.
Robots.txt Tester: questions and answers
What does robots.txt do?
It tells crawlers which parts of a site they may visit. It does not remove pages from Google: a blocked page can still be listed if other sites link to it. To keep a page out of results, use a noindex tag and leave it crawlable.
Which rule wins when two rules match?
Google uses the most specific rule, the one with the longest path. When an Allow and a Disallow rule are equally long, Allow wins. This tester follows the same rules.
What happens when a site has no robots.txt?
If robots.txt returns "not found", crawlers may crawl everything. If the server answers with an error (a 5xx code), Google stops crawling the site for a while, so a broken robots.txt is worse than none.
Should I block AI crawlers like GPTBot?
It is a business choice. Blocking them keeps your content out of some AI training and answers, but can also keep your business out of the answers people read in AI assistants.
AI Web Maker