Gemini Notebook’s Fetcher Ignores robots.txt: Now What?

Gemini Notebook’s fetcher ignores robots.txt — and that is not a bug, a loophole, or a policy change. It is how Google has always documented its user-triggered fetcher class. What changed on July 17, 2026 is the name: the Google-NotebookLM user agent became Google-GeminiNotebook, following the July 16 rebrand of NotebookLM itself, and every access-control rule keyed to the old string now has a shelf life.

The stakes are concrete. Gemini Notebook’s Discover Sources feature can scrape up to 10 articles for a single user query and return an AI summary that sends zero referral traffic back to the sources. Google states the product now reaches 30 million-plus users and 600,000-plus organizations, and Search Engine Journal’s framing — the outlet’s risk assessment, not Google’s — is that the rebrand “may expose your site to more AI scraping” simply because more people will use it.

This playbook covers what actually changed and what didn’t, why robots.txt is architecturally inert against this entire fetcher class, how Discover Sources differs from a manual source add, a side-by-side matrix of which Google agents obey robots.txt at all, and the controls — firewall rules, .htaccess UA blocking, IP-range blocking — that actually work. Every claim below is sourced from Google’s crawling documentation, Google’s product blog, Search Engine Roundtable, and Search Engine Journal.

Two announcements landed a day apart. On July 16, 2026, Google’s product blog announced that NotebookLM is now Gemini Notebook — in Google’s words, it “remains a standalone product focused on being your premier research tool, but it will now do more across the Google ecosystem.” The rename ships with feature additions: a secure cloud computer for writing and executing code inside notebooks (rolling out to Pro users over the coming weeks, per Google), notebook sync with the Gemini app, and planned integration with Google Search’s AI Mode.

On July 17, Google’s crawling documentation changelog followed: the product’s fetcher user agent was updated from Google-NotebookLM to Google-GeminiNotebook, with the old value supported through a transition window that sources place around August 2026 — no specific day has been published. If your log analysis, firewall rules, or bot analytics match the literal old string, that is the deadline that matters.

What did not change is just as important. The fetcher itself is not new: Google first added Google-NotebookLM to its user-triggered fetcher documentation in October 2025, roughly nine months before the rename. Its behavior — fetching only URLs a user’s action requests, ignoring robots.txt — is identical under the new name. Search Engine Journal put it bluntly: “Gemini Notebook is the exact same thing. There is no change to how it works or what it does.” The thing that changes is visibility. A product folded into the Gemini brand, surfaced across the Gemini app and eventually AI Mode in Search, will plausibly route more user research sessions — and therefore more fetches — through your content than a standalone tool called NotebookLM ever did.

Google’s crawling documentation sorts its bots into classes, and the class determines whether robots.txt applies. Per the official docs, “user-triggered fetchers are initiated by users to perform a fetching function within a Google product.” Because each fetch is a direct response to a specific user request — a person pasted a URL, clicked a button, asked for sources — Google’s position is that these fetchers generally ignore robots.txt rules. The logic: the human triggered the request, so the robots exclusion protocol, written for autonomous crawlers, does not govern it.

The practical consequence trips up experienced SEOs: a robots.txt directive targeting Google-GeminiNotebook is not rejected, logged, or errored. It is simply never read. A rule like this does nothing at all:

# Inert — user-triggered fetchers never read robots.txt
User-agent: Google-GeminiNotebook
Disallow: /

The narrow claim matters. robots.txt is not broken and is not useless: it still fully governs classic crawlers like Googlebot, and the Google-Extended token remains the documented opt-out for AI-training use of your content. Our robots.txt and meta robots reference covers that layer in full. What this post covers is the class of Google bots that layer cannot reach — and Gemini Notebook’s fetcher is the highest-traffic member of it right now.

Nor is it alone. Google’s user-triggered fetcher list includes Google-CWS (Chrome Web Store), FeedFetcher-Google, Google-Agent, GoogleMessages, Google-Pinpoint, GoogleProducer, Google-Read-Aloud, and the Google-Site-Verification tool — all grouped under the same ignores-robots.txt behavior. For this entire class, Google’s own guidance is that site owners who want to restrict access must use firewall rules, .htaccess UA blocking, IP-range blocking (Google publishes the fetcher IP ranges as JSON files), or the experimental Web Bot Auth protocol.

Both of Gemini Notebook’s fetch paths use the same Google-GeminiNotebook user agent, but they carry very different risk profiles, and conflating them leads to bad blocking decisions. The classic path is a manual source add: a user pastes your URL into their notebook, the fetcher retrieves that one page, and the user works with content they deliberately chose. That is close kin to a person reading your page — they knew your URL, they may well have visited it, and the fetch is strictly per-request, per-URL. The fetcher never scans or discovers pages on its own initiative.

Discover Sources is different. The user types a topic, and the feature scrapes up to 10 articles for that query, then produces an AI summary — with no referral traffic back to any of the sources. The user never curates the URL list and does not necessarily see which pages were fetched before the summary lands. An entire research session can run on your reporting, your data, and your analysis without a single visit registering in your analytics.

Search Engine Journal flags a second-order risk worth naming precisely, and attributing precisely: content ingested this way can be repurposed by users into audio podcasts and video explainers — Gemini Notebook has long supported Audio Overviews and Video Overviews — without attribution back to the original site, and if those derivative assets are republished, they could compete with the source in search. That chain — scrape, summarize, remix, republish — is SEJ’s risk framing, not a Google statement. But the underlying mechanics it rests on (10 sources per query, zero referrals, built-in derivative formats) are documented product behavior.

The honest counterweight: this is also how modern research gets done, and being one of the 10 sources a strategist’s briefing draws on has real influence value even when no click follows. We wrote about the workflow side in how marketers use Gemini Notebook (formerly NotebookLM) as a research workstation — the same mechanics that feel like a threat from the publisher’s side are the reason the tool has 30 million-plus users. Section 07 turns that tension into an actual decision.

The coverage of the rename treats Gemini Notebook in isolation. The more useful view is the category: which Google agents read robots.txt, and which are exempt by class. This matrix compiles Google’s crawling documentation into one reference — the rows in the second group are the ones your robots.txt file cannot see.

Compiled from Google’s crawling documentation (user-triggered fetchers and common crawlers pages), retrieved July 21, 2026. Trigger descriptions summarize each product’s documented function.

Read the second group as a category and the strategic picture sharpens: Google now operates roughly ten agents that your robots.txt file cannot govern, and the AI-era additions — Gemini Notebook’s fetcher, Google-Agent — are the ones fetching editorial content specifically. The trend line is hard to miss. Each new user-facing AI surface Google ships arrives as a user-triggered fetcher, robots.txt-exempt by classification, and the control burden shifts one layer down the stack — from a text file every SEO can edit to firewall and server configuration that many marketing teams cannot touch without engineering support.

Here is the failure mode nobody will see coming, because it fails silently. If you blocked NotebookLM’s fetcher at the firewall or in .htaccess after Google documented it in October 2025, your rule almost certainly matches the literal string Google-NotebookLM. Google says it will keep supporting that legacy value only through a transition window — reported as lasting until around August 2026, within weeks of the July 17 announcement. No specific cutoff date has been published. Once traffic fully moves to the new string, your old rule stops matching anything — not because Google removed the fetcher, but because the fetcher renamed itself past your pattern.

Two implementation details make careless matching fragile. First, Google documents platform-specific variants of the new UA — distinct mobile and desktop, Chrome-version-tagged strings — so an exact-match rule against one literal string can miss real traffic. Match on the Google-GeminiNotebook substring, not the full UA line. Second, this is the second naming change for this one fetcher in under a year (documented October 2025, renamed July 2026). Hardcoding a single UA string anywhere in access-control logic is fragile practice; during the transition, every rule needs to match both strings, and your log dashboards need both too or your bot-traffic reporting develops a quiet gap.

The same fragility cuts the other way for measurement. If you are tracking how often AI tools fetch your content — a signal worth having before you make any blocking decision — a log query keyed only to the old string will show NotebookLM fetch volume decaying to zero over the coming weeks while actual fetches continue under the new name. Update the queries before the transition completes, or you lose the trend line exactly when it gets interesting.

One scoping note first: this section is about controlling one robots.txt-exempt fetcher class. The broader policy question — which AI crawlers to allow for training, which for search citations, how robots.txt and llms.txt fit together — is a different decision layer, and we maintain it separately in our AI crawler access-control decision matrix. Everything that matrix governs works through directives that compliant crawlers read. Everything below exists because this fetcher class reads none of them.

Option 1 — .htaccess or server-config UA block

The most accessible control on Apache-family hosting. Match the UA substring and return 403. During the transition window, match both strings:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Google-GeminiNotebook|Google-NotebookLM) [NC]
RewriteRule .* - [F,L]

Search Engine Journal published a worked .htaccess example for the Gemini Notebook UA specifically; the pattern above is the standard substring-match form of the same approach. On nginx or at a CDN or WAF layer, the equivalent is a user-agent-contains condition returning 403 — same logic, different syntax.

Option 2 — IP-range blocking

User-agent strings are self-reported, and anyone can spoof Google-GeminiNotebook in a request header. For verification or harder enforcement, Google publishes the IP ranges its fetchers operate from as JSON files. A firewall rule keyed to those published ranges blocks the traffic regardless of what UA it claims — at the cost of maintaining the range list as Google updates it.

Option 3 — Web Bot Auth

Google’s documentation also points to the experimental Web Bot Auth protocol, which uses cryptographic signatures to authenticate bot traffic rather than trusting UA strings. It is early and not a practical dependency for most sites in 2026, but it signals where this is heading: verifiable bot identity at the request layer, because self-declared UA strings were never a security boundary.

What not to bother with

robots.txt directives aimed at this UA (never read), meta robots tags (the fetcher is not indexing your page, so index-layer directives are beside the point), and blanket Googlebot blocks (which take your search visibility down with them). Keep the robots.txt layer for the bots that honor it — that file still does real work for Googlebot and Google-Extended, per our complete robots.txt reference.

Now the strategic question, because the technical ability to block does not make blocking right. The calculus differs sharply by business model — and blocking a research tool that 30 million-plus people use to build briefings, podcasts, and study guides is not a free action. Being among the sources an AI summary draws on is a form of reach, even at zero referrals; being absent from it is invisibility, not protection.

Looking forward, the projection we would plan around: the user-triggered fetcher class grows. Gemini Notebook is being wired into the Gemini app and Google Search’s AI Mode, Google-Agent already sits in the same exempt category, and every agentic product across the industry has the same architectural excuse — a human asked, so robots.txt does not apply. By 2027, access control for content sites will look less like a robots.txt file and more like a firewall policy: identity-verified bots, published IP ranges, per-class allow and deny decisions maintained at the CDN. Teams that build that muscle now — starting with something as small as a dual-string UA rule — will make the later calls from data instead of panic. If you want a second set of hands on that posture, our agentic SEO service runs exactly this kind of AI-access audit, and our web development team implements the firewall and server-layer rules where marketing teams typically lack access.