Should your small business block AI crawlers? Meta's new terms and Google's AI Overviews setting

What's happening on Nov 20

Meta's Business Help Center says: "From November 20, 2026, updated versions of the Meta Commercial Terms and the Self-Serve Ad Terms will take effect." The Verge reported the terms were originally supposed to take effect Oct 30. Meta spokesperson Daniel Roberts told The Verge they "will go into effect on November 20th" "to allow more time for us to explain to our partners and stakeholders what's changing and what's not."

Roberts says the extra time is to explain "how Meta uses web crawlers and the fact that content owners will still be able to use robots.txt to signal preferences." Robots.txt is still your tool. Read Meta's terms yourself before Nov 20.

Meta's five crawlers, in plain words

Meta's Meta Web Crawlers page names five.

Meta's instruction is short: "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." Its own example blocks meta-externalagent and meta-externalads. Meta also says to "allow up to 24 hours for changes to robots.txt to take effect."

Which ones to allow and which to block

This part is Kit&File's view, built on Meta's descriptions.

• Keep facebookexternalhit allowed. It builds the preview when you share your own pages on Facebook, Instagram or Messenger. Our reading of Meta's description is that blocking it can break those previews.
• Decide on meta-externalagent. Meta says it's for "training foundation AI models or improving products by indexing content directly." Block it if you don't want that.
• Decide on meta-webindexer. Meta says allowing it "helps us cite and link to your content in Meta AI's responses." Block it and you may lose those citations.
• Think before you block meta-externalads if you run Meta ads. Meta says it's for "improving advertising and other business-related products and services." Meta's Business Help Center adds: "If you disallow Meta-ExternalAds from crawling relevant areas of your website, such as product pages, certain ads and business features and potential performance gains may not be available for you."
• Accept that robots.txt won't stop everything. Meta says meta-externalfetcher may bypass it for fetches a user asked for.
• Write the lines, upload, and wait up to 24 hours.

One example verdict, not a rule: block training, keep previews.

Copy-ready robots.txt lines

Both examples use only Meta's tokens and follow Meta's format.

# Example A: block Meta AI training, allow everything else
User-agent: meta-externalagent
Disallow: /

# Example B: block Meta AI training and Meta AI search indexing
User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Disallow: /

Your robots.txt sits at the root of your domain (yoursite.com/robots.txt). If your site builder manages it, use the builder's setting. Don't add a block for facebookexternalhit.

Google's AI Overviews switch

In Search Console, go to Settings > Search generative AI and choose Exclude. Google's Search generative AI control covers "AI Overviews", "AI Mode" and "Generative AI features in Google Discover." As of August 31, 2026, Google says it's live for all websites.

With Exclude, Google says "You won't receive any traffic or impressions from these features." Google also says "this control isn't used as a ranking or inclusion signal affecting other parts of Search." Content drops out "within 1-2 days," though some takes longer.

Check the Generative AI performance report first, as Google suggests. It shows what you'd give up. Include is the default, and the control is site-wide.

Google-Extended is a different switch

Google-Extended is a robots.txt token. Google's common crawlers page says it manages whether your content "may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding." It adds that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

The Search Console control doesn't touch training. Google says: "This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended."

So the Search Console control decides what shows in Search's AI features. Google-Extended covers Gemini training and grounding. Googlebot is Search itself. Don't block Googlebot.

Page-level controls: nosnippet and friends

Google's robots meta tag page says nosnippet "applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode."

The catch: it also removes your normal snippet in search results. For one section of a page, use data-nosnippet. Google's AI features page lists "nosnippet, data-nosnippet, max-snippet, or noindex" as the controls.

Before you block anything

This is Kit&File's view. If you want your business named in AI answers, blocking works against you. Our guide to getting named in AI answers is coming soon.

Blocking is a trade. Make it on purpose.

Put the guide to work

For the rest of your marketing basics, the Marketing Power Kit ($29) is an individual kit with a 30-day money-back guarantee. The guarantee covers individual kits only.

For a person to look at where AI fits in your business, AI Fit Flash is a $199 prepaid service.

Sources

• Meta for Developers, Meta Web Crawlers
• Meta Business Help Center, Updates to Meta's business terms
• Meta Business Help Center, About Meta-ExternalAds
• Google Search Console Help, Search generative AI control
• Google Search Central, Control what you share with Google Search
• Google Search Central, Google's common crawlers
• Google Search Central, AI features and your website
• Google Search Central, Robots meta tag
• The Verge, Meta delays new business terms after publisher pushback on AI scraping, Oct 23, 2026

Less content. More use.