Meta's New Terms on October 30: What Its Crawler May Read

On October 30, 2026, Meta's revised Commercial Terms take effect for every business that uses Facebook, Instagram, WhatsApp or Messenger commercially: running ads, managing a Page, listing products or using Ads Manager and Meta Business Suite. The terms add a clause authorising Meta's crawlers to read the websites associated with those businesses and use what they find "to provide, develop, improve, research, and maintain advertising and business products". There is no opt-out other than not using Meta's business products.

The clause made news because of a word that is no longer in it. The first version, reported by Nieman Lab on October 15, let Meta use the crawled content to "train" its products, with no limit to advertising. After publishers objected, Meta revised the text the next day. Legal experts quoted by Nieman Lab say the revision still grants broad rights. Meta says nothing has changed in how its crawlers behave and that robots.txt still works.

We read the October 30 terms in both the US and UK English editions, most recently on October 20, along with Meta's own crawler documentation. This post summarises them; it is not legal advice.

Meta's Commercial Terms sit on top of its ordinary user terms and apply whenever a Meta product is used for a business purpose. The preview lists what counts: using ads or marketing messages, selling or listing products, developing apps, building or using a product catalogue, managing a Page, managing a group for business, using any business product such as Ads Manager or Meta Business Suite, and using Meta's measurement services. That covers most companies with a Facebook or Instagram presence.

Meta says the October 30 version consolidates terms that previously sat in several product-specific documents. Nieman Lab reported that Meta's current terms already allowed scraping of sites tied to business accounts in one narrow case, inside the terms for generative AI ad features, where the content could be used only to optimise the business's ads. The new clause moves that permission into the general Commercial Terms and widens what the content can be used for.

Why it matters even if you never read terms: the change takes effect by continued use. A business does not sign anything on October 30; it keeps running ads and is bound. The failure case is an agency that manages twenty client ad accounts and assumes the clients have read this. They have not, and it is their websites the clause reaches.

The clause sits in section 1, now titled "Licenses & Rights You Provide Meta". In plain terms, as read on October 19, it says the business authorises Meta, or any entity acting on Meta's behalf, to use automated software, including scrapers, bots, spiders, crawlers and user agents, and to create test credentials, in order to access, index, cache, analyse, collect, retrieve or crawl two kinds of pages. First, domains and pages associated with the business's products or services. Second, pages made available through the business's or users' use of Meta's products. Meta may pull updates from those pages periodically.

The purpose is stated as providing, developing, improving, researching and maintaining "advertising and business products". The clause spells out what those include: helping businesses create and deliver ads, giving users information about a business's products and services, and connecting businesses with interested users. It also covers maintaining the integrity and security of Meta's products.

Two phrases do most of the work. "Associated with your products or services" is broad: a shop's product pages clearly qualify, but so, arguably, does a publisher's entire site when its Page promotes the articles. "Create test credentials" could, Cornell law professor James Grimmelmann told Nieman Lab, technically let Meta sign up as a user to get past registration walls, which matters for sites with gated content or member areas.

Nieman Lab published the original wording on Thursday, October 15. Its final sentence said Meta may use any content from those pages "to provide, develop, improve, research, train, and maintain the integrity of" the Meta Company Products, services and features covered by the terms. Meta published the revision the next afternoon, October 16, and TechCrunch reported the change on October 17.

Language editions of a terms page do not always change at the same moment, so we checked both. When we re-read them on October 20, they matched: the UK edition uses British spelling but the same revised clause, with no "train" and the advertising-and-business scope. Check the edition for your market on the day you review it; terms pages can change without notice.

Meta's statements have been consistent on two points. The first is that the clause is about product information, not news. Andy Stone, Meta's vice president of communications, wrote on X, as quoted by The Verge:

The second is that robots.txt, the file sites use to tell crawlers what they may visit, still works. A Meta spokesperson told Nieman Lab that the initial consolidation had not made clear that robots.txt preferences are respected, and that content owners can still disable the crawler through robots.txt. Meta's spokesperson Daniel Roberts told The Verge that the update does not change how Meta uses web crawlers.

The legal experts Nieman Lab spoke to were less reassured. Their points: "develop" is broad enough that AI training for advertising or business products could arguably fit inside it, and "business product" is not a defined term; the "associated with your products or services" scope is unchanged and could let Meta scrape entire sites; and because robots.txt states a preference rather than an enforceable right, the terms could be argued to supersede it. Nieman Lab also notes that the revised terms contain no language committing Meta to respect robots.txt. None of that has been tested, and Meta says it is not the intent.

Our reading is narrower than either side. The text as written today limits use to advertising and business products, plus integrity and security, and Meta's public statements say robots.txt is honoured. A business that does not want its content used to train Meta's foundation models can block the crawler Meta says it uses for that, covered next, rather than rely on the contract wording alone. That block covers one documented crawler; it is not a contractual limit on how Meta uses content its other crawlers collect.

Meta's web crawler documentation names five crawlers and what each is for. The page says Meta prefers industry-standard robots.txt over non-standard signals such as NoAI tags, and asks site owners to allow up to 24 hours for robots.txt changes to take effect.

Meta has not said which crawlers the clause relies on, but its documented purposes line up. Crawling for "advertising and business products" is what Meta says meta-externalads does. Training AI models is what it says meta-externalagent does, and that crawler is separately blockable. The two that may bypass robots.txt do so for named reasons: security checks on shared links, and fetches a user explicitly asked for. A blanket block on everything from Meta would also break link previews when your pages are shared, which is rarely what a business wants.

Publishers are the loudest group because their product is the text itself. Nieman Lab notes that virtually every news organisation has at least one Meta business account. The Verge points out that Meta has struck AI licensing deals with some outlets, including CNN, Fox News, USA Today and People Inc. A clause that let Meta use any publisher's site for free would undercut that market, which is why the wording mattered.

Online shops are the group the clause is most clearly written for. Meta's stated aim is accurate product information for ads and recommendations, which in practice means prices, availability and descriptions read from product pages. For many shops that is a benefit: ads that match the live site. The failure case is a shop that blocks every Meta crawler to make a point and then finds its catalogue ads drifting out of date.

Agencies sit in between. A business that uses Meta's products on behalf of a third party agrees to make sure that third party abides by the terms and warrants that it has authority to bind it, and the content licence covers material an agency provides for a client. That makes the agency the party most likely to have clicked through, and the one best placed to tell the client what the clause means for the client's website.

The terms cannot be negotiated, but the crawling can be shaped. The robots.txt lines below follow Meta's own documented format; adjust the paths to your site.

User-agent: meta-externalagent
Disallow: /

User-agent: meta-externalads
Allow: /
Disallow: /account/

For a deeper treatment of which AI crawlers to allow and why, our AI crawler access decision matrix covers the major vendors side by side, and our robots.txt reference covers the syntax. If you would rather we audit the crawlers hitting your site and set the rules for you, that is part of our agentic SEO service.

Meta is not alone in redrawing the line between showing a site and using it. Google began paying some publishers when their content shapes AI answers, which we covered in Google's AI contribution pilot, and Google is now testing ads inside AI Mode shaped by what users connect, covered in our AI Mode ads post. The common thread is that platform terms, not only court cases, are where the rules for AI use of web content are being set.

The practical lesson from our own 30-day crawler log study applies here: what a vendor says its crawler does and what arrives in your logs are two different records, and only one of them is yours. Write the rule, then check the logs.