GPT-6 Astra reads your site twice

Two separate things have to be true before GPT-6 Astra can name your business, and they are controlled in two different places.

The first is the one everybody already knows about. OpenAI’s search crawler, OAI-SearchBot, has to be able to reach your pages. It reads robots.txt, and OpenAI’s crawler documentation is blunt about the consequence: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

The second is the one Astra makes matter. When somebody asks ChatGPT to actually do something involving your site, compare your pricing against two rivals, check whether you deliver to their postcode, work through your booking form, the fetch does not come from the search crawler. It comes from ChatGPT-User, and the same documentation says “because these actions are initiated by a user, robots.txt rules may not apply.” That request is decided by your CDN, your firewall and your bot rules, not by a file in your web root.

So the whole of the practical advice is three checks, in this order. Confirm robots.txt does not disallow OAI-SearchBot. Open your CDN’s bot settings and find out what it currently does to AI assistant traffic, because on 15 September 2026 Cloudflare’s defaults for that category change. Then make sure the answer to the question you want to win is in the HTML a bot receives, not assembled in a browser afterwards.

Everything below is the evidence for those three, plus an honest list of the things nobody outside OpenAI actually knows.

It is not a new discipline, and anybody selling it as one is selling you a rebrand. GPT-6 Astra SEO is the ordinary work of making a site retrievable and quotable, with one addition: your pages now get fetched by software that is halfway through a task for somebody, not only by crawlers building an index and people reading a screen.

The useful distinction is not SEO versus generative engine optimization. It is between three outcomes that people keep folding into one number:

• Being cited. Your page is one of the sources an answer was assembled from, and usually gets a link. This runs through OAI-SearchBot and ChatGPT's search index.
• Being named. Your brand appears in the answer text. This often comes from what other people's pages say about you, and can happen with none of your own pages cited at all.
• Being usable. An agent working on somebody's behalf can reach your page, parse it, and complete the step it needs. This is a plumbing question, and it is the one Astra puts pressure on.

Those move for different reasons and are fixed in different places. Ranking is not being named covers the first two coming apart from Google position. The third is what this piece is mostly about, because it is new and almost nobody is checking it.

GPT-6 Astra was announced on 3 September 2026. In OpenAI’s own words it “sets a new state of the art for computer use, browsing, software engineering, cybersecurity, science, and professional work.” The API model page puts numbers on the rest: a 1,050,000-token context window, a knowledge cutoff of 30 April 2026, and web search, computer use and MCP among its supported tools.

Read the benchmarks carefully, though, because the interesting part is where the gain is not. On BrowseComp, the public benchmark for finding hard-to-reach information on the web, Vellum’s write-up of the published scores puts Astra at 91.5% against GPT-5.6 Sol’s 90.4%. That is close to a wash. On OSWorld 2.0, the computer-use benchmark, it is 72.6% against 65.7%, and Astra completes tasks in roughly 40 minutes instead of 75.

That distinction is the whole story for site owners. The change arriving at your server is not smarter retrieval. It is a larger share of your traffic being an agent partway through somebody else’s errand, which fails differently from a reader and leaves no trace in your analytics when it does.

What OpenAI has not published is just as worth stating plainly. Neither the announcement nor the system card describes how Astra decides which sources to cite, how its search results are ranked, or what makes one page get quoted over another covering the same ground. Anybody telling you they have the ranking factors for Astra is describing a correlation study or guessing.

Most robots.txt files were written when “the OpenAI crawler” was one thing. It is three, they do unrelated jobs, and blocking the wrong one is the most common self-inflicted visibility problem we see.

The row that catches people is the middle one. Plenty of sites blocked every OpenAI user agent they could find in 2024 and 2025, reasonably enough, because the argument at the time was about training data. If that block caught OAI-SearchBot, your content has been ineligible for ChatGPT search answers ever since, and no amount of rewriting fixes it. The two decisions are genuinely separable: you can refuse training and still be cited.

Yes for search citations, and not reliably for agent fetches. That one sentence is the answer, and the split inside it is the part worth holding onto.

Blocking OAI-SearchBot takes you out of ChatGPT search answers, by OpenAI’s own statement. Allowing it does not put you in, it only makes you eligible. And a permissive robots.txt is not sufficient on its own: OpenAI’s publisher guidance also asks that your host or CDN allow traffic from its published searchbot IP ranges. A file that says yes in front of a firewall that says no is a firewall that says no.

For ChatGPT-User, robots.txt is the wrong lever entirely. OpenAI treats those fetches as user actions rather than crawling, so the decision moves down the stack to whatever inspects requests before they reach your application. Which is exactly where the next thing happens.

Cloudflare sits in front of a large share of the web, and on 1 July 2026 it announced a new way of sorting AI traffic into three behaviours rather than one switch. Its changelog defines them as Search, which “indexes content for later question-answering with expected referral traffic”, Agent, which is “real-time automated activity on behalf of users, like chat fetchers”, and Training.

From 15 September 2026, for new domains onboarding to Cloudflare, “bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed.” Existing customers can opt out of the new defaults before that date.

Line that up with the bot table. Cloudflare’s bot reference lists OAI-SearchBot under AI Search and ChatGPT-User under AI Assistant. The categories that get different treatment are precisely the two halves of the split this article opened with.

To be exact about the scope, because it matters and the headlines were loose about it: the new defaults are for new domains, the Agent block is on ad-displaying pages, and every existing customer had a window to opt out. This is not a switch that silently disconnects the whole web from ChatGPT on the 15th. It is a direction of travel, and it is the first time the agent fetch has been given its own default separate from the search crawl.

If you are not on Cloudflare, the point still stands, it just has a different owner. Bot protection, rate limiting, JavaScript challenges and CAPTCHAs all sit in the same position in the request path, and all of them treat an agent that is doing exactly what it was asked to do the same way they treat a scraper.

This is the quietest useful fact on the model page, and it decides which of your problems is fixable this month.

Anything about your business that became true after 30 April 2026 is not in the model. A price change, a new location, a rebrand, a product you launched in June: Astra can only know those by fetching them. If the fetch fails, or if the fact only exists on a page that renders client-side, the model answers from what it learned before the cutoff and does it confidently.

That gives you a triage rule that costs nothing to apply. Ask Astra a question about your business and look at whether the answer shows sources.

• Sources shown: the answer was assembled from a live search, so your pages are in play. Fixing the page can change the answer, on a crawl-and-index timescale of weeks.
• No sources: the answer came from the model's weights. Rewriting your page does nothing to that until the model is trained again, which you cannot schedule and will not be told about.
• Wrong facts with sources shown: the worst case and the most fixable. Something reachable is saying the wrong thing about you, and it may not be your own site.

Sorting your questions that way before you change anything is the difference between a fix that can work and a fix that never could. Measuring AI visibility goes through the same sort in more detail.

There is no published ranking system for ChatGPT citations, so everything in this area is correlational. The largest recent dataset comes from SE Ranking, covering 216,524 pages across 129,000 domains, reported by Search Engine Journal. The researchers say explicitly that their findings are “directional correlations rather than proof”, and that caveat should travel with every number below.

• Referring domains was the strongest single correlate: sites with over 350,000 averaged 8.4 citations, against 1.6 to 1.8 for sites with up to 2,500.
• Freshness showed up clearly: pages updated within three months averaged 6 citations, against 3.6 for older pages.
• Length and structure both tracked upward, with pages over 2,900 words averaging 5.1 citations against 3.2 for pages under 800.
• Google position tracked too, but weakly, and the authors are careful to say this does not show ChatGPT relies on Google's index.

Read that honestly and it says something slightly deflating: almost every strong correlate is a proxy for being a large, established, frequently updated site. None of it is a technique. A small business cannot acquire 350,000 referring domains by restructuring its headings, and a study that cannot separate “big sites get cited” from “these tactics cause citations” cannot tell you which one you are looking at.

What it does support, weakly but consistently, is the freshness finding, and that is the one you can act on without being large. A page that says what is true now, on a URL a crawler can reach, is the cheapest thing on the list.

No, and there is no evidence it does anything for citations at all. We went through the two log studies in Nobody is reading your llms.txt: 97% of llms.txt files received zero requests in May 2026. No major assistant has committed to reading the file, and Astra’s launch changed nothing about that.

We publish one anyway, and the reason is not visibility. It is a short, accurate description of the product that a person or an agent can read without parsing a marketing page. That is worth having. It is not an optimisation, and treating it as one is how an afternoon disappears.

No. Nobody can, and the shape of the system makes the promise incoherent rather than merely difficult.

An assistant assembles its answer fresh each time. The model samples its own output, and where it searched the web first, the pages it got back are not guaranteed to be the same set. Two identical prompts an hour apart can name different brands. Being cited is a rate, not a state, which is why a screenshot of one good answer proves very little and why Proving a change moved an AI answer spends its length on run counts and holdouts.

What you can do is stop being ineligible, and then measure the rate. That is a smaller promise and it is the only one with evidence behind it.

Ordered by how sure we are it matters, strongest first. The top three are plumbing and take an afternoon. The rest is ordinary content work that happens to also serve assistants.

• Open yoursite.com/robots.txt and search it for OAI-SearchBot. If it is disallowed, that is your single largest fix and it is one line. Allow about 24 hours for OpenAI’s systems to pick up the change.
• Check what your CDN does to AI assistant traffic. On Cloudflare that is the AI crawler controls in the dashboard; elsewhere it is your bot rules, rate limits and challenge settings. You are looking for whether a user-initiated ChatGPT fetch gets through.
• Fetch your own key pages the way a bot does, with curl and no JavaScript, and read what comes back. If your pricing, hours, location or product facts only appear after client-side rendering, they are not there for anything that does not run a browser.
• Give every important fact a stable, server-rendered URL. Pricing on a pricing page, hours on a contact page, specifications on a product page. An agent completing a task needs to land somewhere specific, not on a homepage that carousels through claims.
• Make the first two sentences under each heading answer the heading. This is the one structural habit with a plausible mechanism behind it: a self-contained passage is quotable without the rest of the page, and quotable passages are what get lifted.
• Say what changed and when, in text. Freshness is the strongest correlate you can act on at any size, and a page that dates its own claims is easier to trust and easier to quote.
• Fix what other people's pages say about you. Directory entries, review profiles, roundups. Being named often depends on those rather than on anything you own, and it is the slowest of these to move.

Notice what is not on that list. No schema type has been shown to influence whether ChatGPT cites you, no keyword density target exists, and there is no submission endpoint. Structured data is still worth having for Google’s rich results, which is a real and separate benefit. Ship it for that reason, not for this one.

Stating this plainly is the difference between an article you can act on and one you will regret acting on.

• How Astra chooses which sources to cite. Undocumented, in the announcement and in the system card.
• Whether Astra's web search draws on the same index earlier ChatGPT search used, or something new. Not stated.
• Whether markup, llms.txt or any other machine-readable signal influences source selection. No evidence either way, and absence of evidence is where most GEO advice quietly lives.
• How much of the traffic reaching your server today is an agent rather than a reader. Server logs can tell you by user agent, which is more than analytics can.
• Whether agent fetches will keep being treated as user actions rather than crawling. This is a policy position, and policy positions move.

On that fourth point, your logs really are the better instrument right now. The AI traffic GA4 cannot see covers why the analytics you already have miss most of this by construction: an agent completing a task rarely produces the click that GA4 is built to count.

Astra will be replaced, on the evidence of the last two years, within months. The two questions underneath it will not be.

Can a machine acting for a customer reach your pages and read them without running a browser? And when somebody asks the question your business exists to answer, how often does the assistant name you rather than a competitor? The first is a plumbing check you can run yourself this afternoon. The second is a rate, so it needs the same prompts run repeatedly over weeks before it means anything.

Running that loop on a schedule is the whole of what we built. How it works sets out what a check pulls, how the prompt set is built and what a rewrite changes. If you would rather see your own numbers than read about the method, start free: a free report costs nothing and needs no account, and it tells you whether an agent can read your site at all, which is the part of this you can fix before the 15th.