Aizii Learn · AI Search
← Back to Learn

How to allow Google-Extended in robots.txt

Add a group for Google-Extended with an Allow rule in [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain. Google-Extended is not a crawler. It is a robots.txt policy token with no user-agent string of its own, and it never appears in your server logs. Google says it controls whether content Google has already crawled may be used to train future generations of Gemini models and for grounding, which is handing content from the Google Search index to the model at prompt time. Allowing it is a training and grounding decision. It is not a search decision, and it has no effect on whether your site appears in Google Search.

Why Google-Extended gets mixed up

  1. People treat it as a ChatGPT switch. It is not. Google-Extended is Google's token. It says nothing about OpenAI, Anthropic, or Perplexity, and allowing it does not put your brand into ChatGPT, Claude, or Perplexity answers.
  2. People treat it as the Google search switch. It is not. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search. The crawler that feeds Search is Googlebot, and it has its own token.
  3. People grep their logs for it. That cannot work. Google says Google-Extended has no separate HTTP request user-agent string, and crawling is done with existing Google user-agent strings. Any log line that claims Google-Extended as the user agent is spoofed.
  4. People assume training and grounding are separate switches. At Google they share one token. Google's docs describe Google-Extended as covering both training future Gemini generations and grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. One rule covers both uses, so you cannot allow one and block the other with this token.

Step 1: Decide what you are actually deciding

Before you write the rule, name the decision in plain words. Allowing Google-Extended says Google may use content it has already crawled from your site to train future Gemini models and to ground Gemini answers. Disallowing it says the opposite. Neither choice changes your Search inclusion, your ranking, or your appearance in AI Overviews, because Google says those run through Googlebot and Search, not through this token. If your goal is to appear in Google Search or AI Overviews, this article is the wrong lever, and the Googlebot group is the one to check. If your goal is to stay available for Gemini grounding, where Google describes the token as governing whether crawled content may be used for grounding, then this token is the lever. Write the decision down before you write the rule, because the file will not explain itself to the next person who reads it.

Step 2: Check the file exists and read the Googlebot group

Open [yourdomain.com]/robots.txt in a browser. If nothing loads, create the file at the root of your site. Read what is already there before you add anything. Look for the Googlebot group first. If Googlebot is disallowed, Google-Extended has little to work with, because the token governs use of content Google has already crawled, and a blocked Googlebot crawls less of it. Also look for a User-agent: * group. A named Google-Extended group stands alone and does not inherit the wildcard group's rules, so check whether the wildcard group says anything you need to repeat.

Step 3: Add the Google-Extended group

Put the group near the top of the file, where the other AI tokens live, so it is easy to find and easy to audit later. Google's own example group uses the token with an allow and a disallow for a path, and the form below follows that shape.

User-agent: Google-Extended
Allow: /

That is the allow form. If your decision is to opt out of Gemini training and grounding instead, the form is the mirror image.

User-agent: Google-Extended
Disallow: /

Either form is a complete group. Do not write a half group with only the User-agent line and no rule, and do not add a Disallow for a path you meant to keep open. Because the token has no crawler of its own, there is no crawl to speed up or slow down, and there is no Crawl-delay line that applies to it. The group is a permission statement, and the file is where that statement lives.

Step 4: Do not confuse it with the search controls

This step is the one that saves the most confusion later. Google-Extended does not control AI Overviews or AI Mode in Search. Those are Search features, and Google points to its Search generative AI controls instead, including the snippet controls and noindex, for limiting what Search shows from your pages. Blocking Google-Extended will not remove your pages from AI Overviews, and allowing it will not put them there. Keep the two jobs separate in your head and in your file: Googlebot governs what Search can crawl, Google-Extended governs what Google may do with crawled content for Gemini training and grounding. If a template tells you Google-Extended is the switch for ChatGPT, that template is wrong, and the ChatGPT switch is OAI-SearchBot, which belongs to OpenAI, not Google.

Step 5: Verify the file, and accept there is no log check

Refetch [yourdomain.com]/robots.txt and confirm the Google-Extended group is present and spelled exactly Google-Extended. That fetch is the whole verification. There is no log check for this token, because it has no user-agent string and never appears in a log line. If you want evidence the token is being read, the evidence is indirect: Googlebot visits continue, your Search presence is unchanged, and the group is present in the live file. Do not grep for Google-Extended in your logs and do not add it to a log-based blocklist, because a token with no user-agent string cannot be matched that way, and any hit you find is spoofed. If a vendor dashboard claims to show Google-Extended visits, treat that claim with suspicion, because the vendor's own docs say the token sends no requests.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

Say you run a skincare brand

You sell serums and moisturizers, and your ingredient pages rank well in Google Search. A forum post tells you to block Google-Extended to keep your brand out of AI answers, so you add a Disallow group for it and wait. Nothing changes in Search, and your pages still appear in AI Overviews, because Google-Extended never controlled those features. What the block did change is the Gemini side: your content is no longer available for Gemini training and grounding, which is a real decision, but it was not the decision you thought you were making. You go back to the file, write down the two jobs separately, and make each choice on its own terms. Googlebot stays allowed, because Search visibility is what your store runs on. Google-Extended gets its own deliberate choice, allow or disallow, made for Gemini training and grounding and nothing else. The file now matches the decision, and the next person who reads it can see which job each group does.

What this guide does not cover

This guide covers one policy token. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover the Search controls for AI Overviews and AI Mode, which live in Search Console and in snippet and indexing controls, not in this token. It also does not cover the training controls of other vendors: GPTBot, ClaudeBot, and the Meta training tokens are separate decisions with separate docs, and none of them is answered by a Google-Extended rule.

Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.