Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# AEO for Gemini: the technical checklist

Gemini reads your brand through Google's index, not through a crawler of its own. Googlebot builds the Google Search index, and Gemini grounds its answers in that index. The control that decides whether your content may train future Gemini models and be used for grounding in Gemini Apps and Vertex AI is Google-Extended, a robots.txt token with no user agent of its own. The checklist in this guide is simple to state and easy to get wrong: keep Googlebot allowed, allow Google-Extended unless you mean to refuse it, and make the product facts true, complete, and readable in the first wave of HTML. No vendor publishes a formula for what Gemini recommends, and this guide does not claim one. It covers only the technical ground a brand controls.

This is a developer-led job with marketer checks. The developer owns robots.txt, the server, the firewall, and the markup. The marketer owns the product facts, the page copy, and the decision about training and grounding use. If you are both people, check Google-Extended first, because it is the one line in this checklist that a privacy checklist most often gets backwards.

## Where it goes wrong

Gemini checklists fail in a small number of repeatable ways, and the most common one is a single misunderstood token.

-   Google-Extended is blocked by mistake. A privacy checklist somewhere told the team to block it, on the belief it stops Google from indexing the site. Google says otherwise: Google-Extended is a standalone product token that manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not impact a site's inclusion in Google Search, and it is not used as a ranking signal in Google Search. Blocking it does nothing for Search inclusion and quietly cuts the brand out of Gemini's answers.
-   Googlebot is blocked by a blanket rule. A broad Disallow aimed at all crawlers catches the crawler that feeds the index Gemini grounds on. Google's AI features guide says a page must be indexed and eligible to be shown in Google Search with a snippet to be eligible as a supporting link in AI Overviews or AI Mode, and that there are no additional technical requirements. If Googlebot cannot crawl the page, Gemini has no index entry to ground on.
-   The firewall challenges Googlebot. A bot rule, a challenge page, or a block on Googlebot's IP ranges stops the crawler before robots.txt is ever read. Googlebot is often spoofed, so the fix is not to trust the user agent string. Verify a request by reverse DNS: the hostname must resolve to a googlebot.com address and forward-resolve back to the same IP. A run of genuine Googlebot hits returning 200 on key pages is the evidence that matters.
-   The product facts are wrong or late. A price in the page copy that disagrees with the structured data, a size that is gone but marked in stock, a shipping promise the checkout cannot keep. Gemini grounds on the index, and an index entry that disagrees with itself gives it nothing to cite with confidence. The fix is one true price, one true stock state, and one true shipping promise, in words and in code, at the same time.
-   The content arrives only after JavaScript runs. If the price, the stock state, or the product description loads only in the browser, Googlebot may read it only in a second wave of crawling, or not at all. Serve the product facts as server-rendered HTML, and check the raw HTML, not the rendered page in a browser, to confirm they are there.
-   The brand expects a result the same day. Google says crawling can take anywhere from several days to several months, depending on how often its systems determine a page needs to be refreshed. A fix is not instant, and requesting a recrawl in Search Console is the lever a brand has. Test, request a recrawl, then test again after Google has had time to return.

## The steps

**Step 1. Allow Google-Extended, unless the team means to refuse it.** The developer opens the robots.txt file at the site root and looks for a Google-Extended group. If it disallows the token and the brand wants Gemini to mention it, change it to an allow. Know what the decision means: allowing Google-Extended lets content be used for training future Gemini models and for grounding, which is handing Google Search index content to the model at prompt time. Blocking it does not reduce Search inclusion, ranking, or AI Overviews. So the only question is whether the brand wants its content usable by Gemini at all. If the team keeps the block, do it knowing it costs mentions, not knowing it protects the site from indexing. Do not look for Google-Extended in the server logs. It has no user agent string and never requests anything, so any log line claiming to be Google-Extended is spoofed.

**Step 2. Confirm Googlebot is allowed in robots.txt and at the edge.** The developer confirms robots.txt does not disallow Googlebot, and checks the firewall and the CDN rules so Googlebot is not challenged. Then the developer verifies in the logs and confirms the requests are genuine with reverse and forward DNS: the hostname must resolve to a googlebot.com address and forward-resolve back to the same IP. A block aimed at something else often catches Googlebot, so check the edge, because the file is what the brand declares and the edge is what enforces.

**Step 3. Make the product facts true, complete, and in the first wave of HTML.** The marketer writes the product facts once, in plain words: the name, the price, the stock state, the shipping promise, and the return policy. The developer puts the same facts in Product structured data on the page, with the price, the availability, and the shipping details matching the visible text. Then the developer fetches the raw HTML of a product page, with curl or view source, and confirms the price and the stock state are in it before any JavaScript runs. If they are not, the page needs server-side rendering for those facts. A page that looks right in a browser and wrong in raw HTML is a page Gemini cannot ground on with confidence.

**Step 4. Check the sitemap, the llms.txt file, and the Search Console setup.** The developer confirms the sitemap lists the product pages, the category pages, and the guides at their canonical addresses, and that robots.txt points at the sitemap. The marketer confirms the llms.txt file lists the pages that matter most, with a short note each. Then the developer confirms the site is verified in Search Console, checks the Sitemaps report for errors, and uses URL Inspection for a live test of a blocked page. If the report shows an error, fix the cause, not the report.

**Step 5. Ask Gemini the customer questions, request a recrawl, and retest.** The marketer asks Gemini the questions a customer would ask about the category, the products, and the location, and notes whether the brand appears and which pages it cites. If it cites a competitor, compare the two sites: is the competitor indexed, is its Google-Extended token allowed, is its product information in the first wave of HTML. Fix the gap the comparison shows, request a recrawl in Search Console, and retest after Google has had time to return. Google says crawling can take anywhere from several days to several months, so a retest the same afternoon is not a verdict.

## The machine-readable piece

This robots.txt sample names Googlebot and Google-Extended on their own lines, so the crawl decision and the training and grounding decision are separate and readable. Replace the placeholder with your own sitemap address. The Google-Extended group below allows the token. If your team decided to refuse training and grounding use, change that group to Disallow, knowing it costs Gemini mentions and does nothing for Search inclusion.

```
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://[yourdomain.com]/sitemap.xml
```

This Product JSON-LD block is the machine-readable version of one product page. It uses only fields from the Product structured data guide in Sources. Replace the placeholders with your real price, your real stock state, and your real shipping details, and make sure every value matches the visible text on the page. The priceValidUntil value is a placeholder shape: replace YYYY-MM-DD with the real date the price is valid until, or remove the field.

```
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "[Product name]",
  "description": "[One plain sentence about the product]",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "[Price]",
    "priceValidUntil": "YYYY-MM-DD",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  }
}
```

Check the file before you rely on it. Open your domain followed by /robots.txt in a browser and confirm the groups load. Fetch a product page as raw HTML and confirm the price and the stock state are in it. Search your server logs for Googlebot and confirm the source IP verifies by reverse DNS and the response is 200. If a log line claims to be Google-Extended, it is spoofed, because Google-Extended has no user agent string and never requests anything.

## What it looks like when it works

Say you run a garden center with an online shop, selling potting mix, planters, and seeds, with a guide on repotting houseplants. The developer finds a Google-Extended block in robots.txt that a privacy checklist added years ago, and the marketer finds product prices that render in the browser only. They allow Google-Extended so Gemini can ground on the content, and they serve the product facts as server-rendered HTML so the price is in the first wave. They confirm Googlebot is allowed at the edge, verify it by reverse DNS, and submit the sitemap in Search Console. They ask Gemini the questions a customer would ask about potting mix for repotting, request a recrawl, and ask again after Google has returned. The brand appears, and the pages it cites are the product pages and the repotting guide, not a stale listing. The fix was a token, a rendering change, and true facts, in that order.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The llms.txt format and a complete sample, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI systems read to verify facts, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

## What this guide does not cover

This guide covers the technical checklist for Gemini: Googlebot, the Google-Extended token, and the product facts. It does not cover the full technical audit that checks every crawler and every page, the transaction layer that lets an agent complete a purchase, or the per-engine playbooks for ChatGPT, Perplexity, and Claude. It does not cover what Gemini says about your brand over time, or how to correct a wrong answer once it appears. Those are separate jobs, and this guide is the map, not the whole trip.

## Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   Google, Google's common crawlers (specific): [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
-   Google Search Central, AI features and your website (specific): [https://developers.google.com/search/docs/appearance/ai-features](https://developers.google.com/search/docs/appearance/ai-features)
-   Google Search Central, Optimizing your website for generative AI features on Google Search (specific): [https://developers.google.com/search/docs/fundamentals/ai-optimization-guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
-   Google Search Central, Product structured data (specific): [https://developers.google.com/search/docs/appearance/structured-data/product](https://developers.google.com/search/docs/appearance/structured-data/product)
-   Schema.org, Product type (specific): [https://schema.org/Product](https://schema.org/Product)
-   RFC 9309, Robots Exclusion Protocol (general standard): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)
-   llms.txt proposal (specific): [https://llmstxt.org](https://llmstxt.org/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
