A 40% Citation Overlap Study: What It Does and Doesn’t Show

Gemini local citations overlap only about 40% of the time when you repeat a query, or so goes the sentence carrying Steady Demand’s Grounding Drift study through the trade coverage we read this week, and it is almost right. The 40% is real, it is the study’s most quotable number, and it comes from a local-SEO agency’s dataset of 14,472 citations across 1,487 Gemini queries, 50 US metros and 10 local-service categories. But the study’s own pages say the figure is a cross-phrasing agreement rate, not an ask-the-same-words-twice rate. Those are two different measurements, and the vendor ran both.

The distinction matters because run-to-run consistency is becoming a budget argument. If an AI answer engine names a different plumber every time a homeowner asks, the case for optimising toward it changes, and so does the case for measuring it. A number that describes how three wordings of one question agree with each other does not tell you how stable a single wording is. The study contains a separate, much smaller experiment that does, and that experiment returns a range rather than a point.

So this is a methodology read, not a rebuttal. Steady Demand is a local-SEO agency publishing marketing research, which is the first caveat and travels with every number below. It is also unusually transparent: it names its formulas, runs a non-AI control, audits its own taxonomy and lowers one of its own headline figures. The job here is to separate what the published method can support from what the headline implies, using the same lens we applied to a different citation-decay study three days ago.

Start with the funder, because every number inherits it. The study is published by Steady Demand under the title What AI Actually Cites, with a companion methodology page, a Grounding Drift deep-dive and an interactive dashboard. The page titles of those study pages describe the company as a Google Business Profile and Local Services Ads management and local SEO agency, and the named author, co-founder Ben Fisher, is presented as a Google Business Profile Diamond Product Expert. We found no funding or sponsorship statement on any of the four pages. This is vendor marketing research by a firm that sells local visibility services, which is not a reason to dismiss it and is a reason to read it carefully.

The design, as disclosed: the 50 largest US metropolitan statistical areas by Census population, ten local-service categories, and three fixed phrasing templates per metro-and-category pair. The templates are best {vertical} in {city}, {state}, top rated {vertical} near {city}, {state} and who is a good {vertical} in {city}. That is a 1,500-query matrix (50 × 10 × 3), of which 1,487 completed; the vendor says 13 failed every retry against the Gemini API and were skipped rather than sampled out. It also says 498 of the 500 metro-and-category groups have all three phrasings. Those two statements do not quite reconcile: 13 missing queries have to fall in at least five different groups, so at most 495 groups can hold a complete set of three (our arithmetic: 13 ÷ 3 rounds up to 5). We report both as the vendor states them, with that gap flagged. Collection ran on July 27–28, 2026.

The ten categories, named on the methodology page though not in the coverage we read: Plumber, Roofer, HVAC Contractor, Electrician, Locksmith, Pest Control, House Cleaning, Personal Injury Lawyer, Dentist and Auto Repair Shop. The 50 metros are described by rule (largest first, each represented by its principal city and state) but are not enumerated as a list on any page we checked, so we do not infer them here.

Two disclosures deserve credit before any critique. First, the citation definition is precise: the API returns grounding_chunks (the cited source plus a Google-hosted redirect URI), grounding_supports (which sentence a citation backs) and web_search_queries (the literal sub-searches Gemini generated). Domains are aggregated from the title field of each chunk rather than the redirect wrapper in uri, which the vendor says it confirmed against live responses before collecting anything. Second, the overlap maths is named: Jaccard similarity on cited-domain sets, computed pairwise and averaged, in a “loose” full-set form and a “strict” does-the-top-recommendation-match form. Both are specified precisely enough to be checked against the API’s own output rather than taken on trust.

The study’s homepage callout reads: “Ask Gemini the exact same question, word for word, twice in a row — and the cited sources still only match about 40% of the time. We call this Grounding Drift.” The martech.org write-up that carried the study on August 20, under Danny Goodwin’s byline, restates the ask-twice framing without qualification, and without noting that the publisher is a local-SEO agency. That is also the framing most likely to travel, because it is the one the study’s own homepage leads with.

The Grounding Drift deep-dive says something narrower. Its scope note states that the 40% is the cross-phrasing agreement figure from the full 1,487-query, 50-metro dataset. In that dataset, each metro-and-category pair has three differently worded queries, each run once. The 40% is how much the cited-domain sets of those three wordings agree with each other, averaged over the groups that hold all three phrasings — the vendor says 498 of 500, with the caveat from Section 01 attached. Nothing in the main dataset asked the same words twice.

The literal ask-twice measurement exists, but it is a different experiment on a different scale, covered in Section 03. The point is not that the vendor hid anything; it disambiguates the two on its own pages. The point is that the homepage teaser and the coverage we read collapse them into a single sentence, and the sentence that survives is the one that describes the smaller experiment using the larger experiment’s number.

Both are legitimate things to measure, and for a local business both are uncomfortable. But they answer different questions, and a slide that says “identical queries overlap 40%” is citing Measurement A for Measurement B’s question. If you need the ask-twice figure, the honest version is a range with a sample size attached. If you need the 40%, the honest version names the three phrasings.

The identical-wording experiment, as the deep-dive describes it: six fixed queries, all of the form “best X in New York, NY” for dentist, auto repair, house cleaning, HVAC, plumber and personal injury lawyer, run across four rounds with zero wording change, 72 repeat-test calls logged in total. That works out to three calls per query per round, which matches the vendor’s reference to the next day’s three repeats being compared with one another. It is one metro and six distinct queries, against the main dataset’s 1,487 across 50 metros (our arithmetic: 1,487 ÷ 6 ≈ 248, so the main dataset carries roughly 250 times as many distinct queries; the 72 is a count of calls, not of queries).

The overlap depends on when you ask. The chart below is the vendor’s own Jaccard figures on cited-domain sets, by interval. The bars are agency-stated; we have not re-run the calls.

Two readings of that chart. The generous one is that the vendor’s “40–46%” region is honest about its own spread. The stricter one is that the region excludes the chart’s lowest bar: 26.5% is the overlap between three repeats run on the same day as each other, which is exactly the ask-twice scenario the homepage describes. A single New York day, six queries, and the answer to “ask twice, how much matches?” was closer to a quarter than to 40%. With 72 calls, no one should treat either end as a population estimate. That is the vendor’s own advice, and it applies to the headline too.

The deep-dive also reports where the instability comes from. The search strings Gemini generated internally for the same query (the web_search_queries field) overlapped far less than the resulting citations: 0.011 to 0.056 Jaccard across the four rounds, against 0.265 to 0.463 for the cited domains. The vendor’s reading is that the drift originates in query generation, not in the live web changing between calls. That is a plausible mechanism and a useful one for practitioners, because it means the variance sits inside the model’s retrieval step rather than in your listing.

The most persuasive part of the study is the part the coverage we read left out. Steady Demand ran a direct non-AI control: the identical 500 metro-and-category combinations, on the same repeated-query cadence (up to five timed rounds, five minutes apart), against Google’s own local pack in parallel. The local-pack data came through a third-party SERP data provider’s endpoint rather than direct scraping, which the vendor notes would breach Google’s terms.

The result, on the “strict” metric of whether the top listing or top recommended business matched round to round: Google’s local pack returned the same top listing 90.2% of the time across 4,466 pairwise round comparisons; Gemini returned the same recommended business 7.9% of the time across 5,000 pairwise comparisons. Both figures are for combinations that completed all five rounds.

This is what makes the drift finding more than a curiosity. The same cities, the same categories, the same timing, the same underlying Google index: one surface held its top answer nine times in ten and the other held it less than one time in twelve (our arithmetic: 90.2 ÷ 7.9 is a little over eleven). Whatever is unstable is specific to the generative synthesis layer, not to local search data as such. The vendor could have led with this, and the coverage we read should have.

A reproducibility claim stands or falls on its protocol, so the table below checks the seven questions we ask of any repeat-query study, plus three openness questions, against the vendor’s own pages. One complication the coverage we read flattens: this is four experiments sharing one report. The main 1,487-query dataset, the six-query isolate test, the local-pack control, and a Gemini-versus-ChatGPT comparison each answer some protocol questions differently, so the last column says which sub-study each answer applies to.

Read as a whole, the table is more favourable to the study than the headline treatment deserves. Seven of ten questions are answered in full or in useful part, including the two that decide whether anyone could repeat it, the citation definition and the overlap formula. The gaps are the familiar ones for vendor-published benchmarks: no list of the sampled units, no raw data, no external check. What the table cannot fix is the row that matters most for the headline, row 1: the main dataset contains no identical repeats at all, so it cannot by itself support an ask-twice claim of any percentage.

The study’s other headline is share of voice: roughly 60% of Gemini’s local citations point to the business’s own website, more than directories, review sites and forums combined. That directional finding looks solid. The precise figure does not hold still, and the vendor is the one who says so.

The methodology page describes a taxonomy audit of the fifteen highest-frequency domains that had been defaulting to the “business’s own site” bucket. Nine of the fifteen were misclassified: two city magazines, two lead-generation sites, a review-management vendor, two membership directories and others. After correcting the rules, the own-site share moved from 61.1% to 58.7% “at the time of this audit”. The report page rounds to 60%. The interactive dashboard displays 59.9%, which is 8,662 of 14,472 citations (our arithmetic: 8,662 ÷ 14,472 = 59.85%, so the rounding checks). The pages do not say which of 58.7% and 59.9% is current, or whether the dashboard was rebuilt after the audit.

Two things are worth crediting here. The vendor caught its own classification error and published the lower number, which is the opposite of the usual incentive. And the confidence intervals are bootstrapped at the query level, not the citation level, because citations arrive in clumps of roughly eight to ten per query and are not independent. That is the right call, and the reason for it is stated on the page rather than left implicit. The vendor even names a selection bias in its own significance test on phrasing volatility by vertical, noting that the pair it compared was chosen after seeing which two of the ten categories were the extremes. Naming it does not remove it, but a reader can only discount what has been disclosed.

What the Reddit figure does not tell you is anything about ChatGPT. The same report runs the 1,487-query design through ChatGPT’s search mode via a third-party endpoint, collected July 30, and finds only 8.3% average Jaccard overlap between the two engines’ cited domains and the same top business named 4.2% of the time. That is engine-to-engine disagreement, a third kind of measurement. The vendor also notes an August 20 re-collection of the ChatGPT side after reports that ChatGPT Search had sharply cut Reddit citations, with Reddit’s share of ChatGPT citations falling from 41.7% to 0.0% between the two pulls. Alongside it the vendor reports a same-day 40-query Gemini spot-check that returned a 6.7% Reddit share and calls that “consistent” with its published 13.7% baseline. Read plainly, 6.7% is less than half of 13.7%; with 40 queries and no interval published for the spot-check, it is too small to confirm or contradict the baseline either way, and “consistent” is the vendor’s characterisation rather than a test result. In any case that update is about ChatGPT; it is not evidence about Gemini’s drift rate.

Here is the study reduced to sentences you can put in a deck, with the version the data supports beside the version that travels.

On the corroboration: the deep-dive cites a SISTRIX study by Johannes Beus, published May 1, 2026, which tracked 82,619 prompts and 1,548,213 snapshots across six countries and three platforms for 17 weeks and reported week-over-week domain churn of 5% for Google AI Overviews, 56% for AI Mode and 74% for ChatGPT Search. It also cites a paper by Grossman and colleagues accepted to ACM SIGIR 2026 on generative search showing materially lower response consistency than traditional search. Both are presented by Steady Demand as context, and the vendor is careful to say its own metric differs from SISTRIX’s in kind and timescale. We have not independently pulled either source; treat them as what the study cites, not what this post confirms.

On Google: we checked the Gemini API’s Grounding with Google Search documentation page specifically for this post. It covers how grounding works, pricing, supported models and how to display citations, and says nothing about whether repeated identical calls return consistent sources, nor about determinism or sampling in the grounding step. That is an absence on one page, not a statement that Google has never addressed it anywhere. It does mean a practitioner has no first-party number to put beside the vendor’s.

And on the obvious comparison: this is not the same finding as the citation-decay study we reviewed on August 17. That one, from a different vendor, measured domain survival over months in a single Australian insurance category on ChatGPT and Google, and its headline number described which cited domains were never cited again. This one measures run-to-run and phrasing-to-phrasing agreement inside a two-day window, on Gemini, across US local services, using Jaccard overlap. Decay over time and drift within a window are different phenomena with different maths, and a chart that puts them on one axis is wrong before it starts.

Assume, provisionally, that the direction holds: Gemini’s local answers lean on the business’s own site, Reddit punches above the directories, and which sources get cited varies substantially between asks. Three practical conclusions follow, none of which require believing any particular decimal.

The first conclusion is the one that changes how you measure. Our share-of-voice tracking framework makes a version of this move already: a fixed prompt panel, re-run on a weekly cadence and read as a trend rather than a single snapshot. This study is an argument for pushing the same discipline down to the individual prompt, because a citation that appears in one run out of four is a 25% citation, not a win. The second conclusion is reassuringly boring. Everything we know about what still drives local visibility on Google itself and about what correlates with being cited at all points at the same place this study does: the business’s own pages. If your local-SEO programme is already treating AI answers as one more reader of a strong site rather than a separate channel to be gamed, this study is confirmation, not a pivot. That is how we scope agentic SEO engagements for local and multi-location clients: measure AI citation as a sampled rate, and spend the effort on the pages the engines are reading.

Looking forward, the interesting question is not whether Gemini’s local citations drift but whether the drift shrinks. The vendor’s own mechanism, variance in the generated sub-searches, is the kind of thing a model update could change in either direction, and a snapshot from two days in July cannot tell you the trend. If Steady Demand re-runs the same design in a later window and publishes both, that second run will be worth more than the first; a repeat of the identical experiment is the one thing that would let anyone say whether 40% is a ceiling, a floor or a coincidence.