We ran the same prompts through four AI engines. They barely cited the same pages.

The question that the score cannot answer by itself

A recent paper called Scoring Without the Engine does something useful: it asks how far you can get by grading a page without repeatedly querying a live AI engine. Its strongest result is also the reason we wanted to run this study. Generic page quality had little relationship with citation order. Once the evaluation included the actual query, relevance became much more informative.

But the experiment deliberately fixes the candidate source set. That lets you study which of several already-exposed pages an answer model prefers. It does not measure the step before that: which pages the production system searches for, retrieves, reranks and puts in front of the answer model in the first place.

That is not a small distinction. If two engines never expose the same pages, a perfect model of citation preference inside a fixed source set still cannot tell you which pages will be visible end to end.

We sent the same prompts to four live engines

We used a fixed benchmark of 15 commercial prompts around AI-search and visibility software. On 6 June 2026 we collected the citations returned by ChatGPT, Microsoft Copilot, Google and Perplexity. The snapshot contained 589 citation observations, 528 unique URLs and 356 domains.

The first comparison is deliberately simple. For the same prompt on the same day, how many exact cited URLs did two engines have in common?

Almost none. Mean pairwise Jaccard overlap was 0.0079, or about 0.8%. The median was zero. In 84.9% of engine-pair comparisons there was no shared cited URL at all.

The top of the list was even more different

Ten prompts were observed on all four engines, which gives 60 pairwise engine comparisons. Across those comparisons, top-five exact-URL overlap was zero every time. Reciprocal-rank-weighted overlap, which gives more weight to higher citations, averaged 0.0027.

This matters because a low all-list overlap could in theory be driven by a long tail of incidental references. It was not. The pages closest to the answer were at least as engine-specific as the rest.

List length does not explain it

ChatGPT returned more citations per prompt than Copilot, so raw overlap alone is not enough. We added a matched-size baseline. For each of the ten prompts present on all four engines, we kept the observed four-engine URL universe and each engine's exact list length, then calculated the overlap expected if those lists were drawn randomly from that universe.

The baseline expected mean Jaccard overlap of 0.1272. We observed0.0072, only 5.7% of that amount. The random baseline expected zero overlap in 12.3% of comparisons. We observed it in 86.7%.

One engine is a poor proxy for the others

On the ten prompts available on all four engines, a single engine covered only 11.4% to 42.6% of the combined four-engine URL set. The union was 2.43 times larger than the broadest single-engine source set on average. Across the June 6 snapshot, 96.4% of exact URLs appeared in only one engine.

Even at the domain level the pattern remained. About 89% of observed domains appeared in only one engine. Monitoring one answer surface and calling it "AI visibility" therefore makes a strong assumption that this sample does not support.

The source set moves over time too

We also compared the same prompt inside the same engine from June 5 to June 6. Mean URL-set turnover was 67.0%. ChatGPT's average turnover was 82.6%, Copilot's 76.6%, and Perplexity's 45.5% in the available pairs.

Cross-engine differences were still much larger than this day-to-day movement. The point is not that citations are random. It is that both the engine and the observation date are part of the measurement.

Page fit, exposure and citation are different things

The paper formalizes the distinction with two stages. First a page has to be exposed: searched, retrieved, reranked or otherwise made available to the answering system. Then it can be selected or cited conditional on that exposure.

A query-to-page score can be valuable. It tells you whether a page is a good fit for a request. It may also help predict selection after the page is already in the candidate set. But it does not observe the first stage, and the first stage is where our four engines diverged most visibly.

The practical measurement stack is therefore:

1. Page fit: does this page answer this request?
2. Observed exposure: does the engine actually surface the page or domain?
3. Conditional selection: when exposed, does it survive into the cited answer?
4. Final visibility: does the brand or source appear in the answer a person sees?

Combining those into one score makes the number easier to sell and harder to interpret.

What this study does not prove

• The dataset contains 15 prompts in one commercial category. It is an audit, not a universal estimate for every query class.
• The observed citations are the end product of the source pipeline. They do not reveal every hidden retrieval candidate considered internally.
• The June 5 to June 6 comparison is short-horizon and should not be read as a long-run stability estimate.
• The matched-size baseline is a diagnostic for list-size effects, not a behavioral model of how any engine chooses sources.
• Low cross-engine overlap does not mean that page quality or query relevance are unimportant. It means they are not the whole production process.

Our fifth paper

This is the fifth paper in the measurement program we started this summer. Answer-Reconstruction Search Density measured the query and source work compressed into one answer. The Prompt Is Not the Query measured how the request is distributed across a conversation. Beyond the Final Prompt tested whether that conversation context changes the answer. Purchase Advice and Observable Buyer Responses looked at what happens after commercial recommendations appear.

This one moves outside the conversation and into the source layer. The same question does not produce one stable pool of "AI sources." In this benchmark, each engine built something much closer to its own source universe.

The submitted preprint is available here as a PDF. The arXiv submission is in processing and we will replace that link with the public arXiv identifier as soon as it is assigned.