Daily SEO Note — November 2, 2026: One robots.txt Line for AI Training

Audit window: 72 hours, 2026-10-30 00:00 UTC to 2026-11-02 06:00 UTC, extended from the usual 24 hours because today is Monday. Everything below traces to a primary source with a date and a rollout status. One story dominates both tracks, and it is the first time the training-opt-out question has had a single answer that more than one vendor agreed to.

1. SEO for Content Writers

The most consequential editorial change of the window: on Friday, October 30, 2026, Google and Microsoft each committed to a shared robots.txt signal that lets a site state one AI-training preference for every participating provider. That turns a per-vendor engineering chore into a single editorial policy decision — and because Google wired the signal to Google-Extended, which also governs grounding in Gemini Apps, the decision now has a visibility price attached to it.

Google Will Honor One AI-Training Opt-Out, and It Is Wired to Gemini Visibility

Google announced plans to support a shared AI training control in robots.txt, based on work from the IETF's AIPREF working group. Instead of maintaining separate instructions for every AI developer, a site will be able to express a broad preference once. The post is dated Friday, October 30, 2026, and is signed by John Mueller. Rollout status: announced intent only. Google says it aims to support the control "along with other AI providers in 2027," and the updated Google crawlers documentation — last changed 2026-10-30 — narrows that to early 2027. Nothing is enforced today.

Who it affects: all content, on every site. The editorial sting is in the coupling. Google says that when a site uses the shared token to opt out of training and Google-Extended is not explicitly specified, it plans to read that as if the publisher had disallowed Google-Extended. Google-Extended is not a training-only control: it also covers grounding, and Google states that sites which allow it are eligible to appear in responses and receive clicks via grounding in Gemini Apps. A blanket training opt-out, therefore, is also a decision to leave Gemini Apps answers. Google Search ranking and inclusion are explicitly not affected.

What to do differently in the next brief: stop treating "allow AI" as one yes-or-no switch and write it down as a policy with a shape. The signal is path-scoped, not just site-wide, so the practical editorial unit is the content type — a brand or press section can stay open to training while a proprietary research library stays closed. Decide that per cluster before anyone edits robots.txt, and record the decision in the content brief template so it survives staff turnover. What to stop doing: stop citing Google-Extended as a "training" control in your own explainers; on Google's own documentation it is a training and grounding control, and describing it the old way now understates what readers give up.

Microsoft Splits Training From Grounding, With a March 1, 2027 Deadline

Microsoft published Your content, your choice: clearer preferences for training and grounding on October 30, 2026, authored by Krishna Madhavan. It adopts the same shared robots.txt training token and then adds something the shared signal does not have: a separate, independent AI Grounding preference set per page. Rollout status: dated and scheduled. Microsoft begins honoring the new preferences as it crawls on February 1, 2027.

Who it affects: anyone who currently relies on NoArchive or NoCache to hold content back from AI use. Those tags stop being read as a training or grounding preference after March 1, 2027, and both new preferences default to Allow when nothing is published. That is a silent reversal: protection you set years ago lapses into permission unless someone republishes it. Microsoft also introduces DisallowExtended for grounding, which keeps content eligible inside Microsoft-operated surfaces such as AI Mode, AI Overview, Copilot and Microsoft 365, while excluding grounding Microsoft provides to third parties. Microsoft's own framing is blunt: Disallow or DisallowExtended will significantly reduce visibility in AI-generated experiences, and neither choice affects eligibility for ordinary Bing search results.

What to do differently: run an inventory this week of every page and template carrying NoArchive or NoCache, and treat it as a republish queue with a March 1, 2027 due date rather than an archive. Note the intermediate date too — Microsoft opens temporary grounding transition preferences in Bing Webmaster Tools on December 15, 2026, and only for sites already using NoArchive or NoCache as of that date. Also note the enforcement mechanic that trips editorial teams: publishing a preference is not the same as activating it, because Microsoft has to recrawl the page to discover it. Details and the companion FAQ are in Bing Webmaster Tools help.

Silence Is Consent: Both Announcements Default to Allow

Worth stating plainly, because it is the part that gets misread. In the IETF Internet-Draft that defines the shared token, a preference exists only if a group naming that token exists. If the file contains no such group, no preference has been expressed — and a wildcard group is explicitly excluded from matching it. Microsoft states the same outcome in product terms: both AI Training and AI Grounding default to Allow when no applicable preference is published.

The editorial consequence is that this is a decision you make by not making it. A site that has spent two years adding vendor-specific disallows has not expressed a preference under the new scheme, and an existing blanket wildcard block does not carry over. Put the question on the next editorial calendar review with a named owner, rather than waiting for a vendor deadline to answer it by default.

No Core Update, No Spam Update, No SERP Feature Change in This Window

Verified empty, so it gets one line rather than padding: the Google Search Status Dashboard logged no ranking or crawling incident between October 30 and November 2, 2026. The most recent ranking event remains the September 2026 spam update, which began 2026-09-24 and completed 2026-10-08. No rich result type was added, restricted or deprecated in the window, and Google's AI features and your website guidance still carries a 2025-12-10 last-updated stamp and still lists only Google-Extended, nosnippet, data-nosnippet and noindex. Volatility chatter circulated on the forums over the weekend; it is unconfirmed and is deliberately excluded from the checklist below.

Apply to Your Next Brief

• Add an "AI usage" line to the brief template with three explicit values — training, grounding, and the path scope they apply to — and require it to be filled before the draft is assigned.
• Decide training and grounding separately. They are independent preferences at Microsoft, and conflating them is what costs you AI citations you did not mean to give up.
• Treat a site-wide training opt-out as a Gemini Apps exit. If grounding visibility matters for a cluster, scope the opt-out to paths instead of the whole site.
• Open a republish queue for every page or template using NoArchive or NoCache, due before March 1, 2027; after that date those tags no longer express an AI preference to Microsoft.
• Diary two dates now: December 15, 2026 for Bing Webmaster Tools transition preferences, and February 1, 2027 for Microsoft honoring the new signals.
• Do not publish explainers that describe the shared token as live. Both vendors have announced intent with 2027 targets; writing it as shipped will need a correction.
• Keep Google-Extended described as a training and grounding control, matching Google's documentation, in anything you publish about AI controls.

2. SEO for Developers

The most consequential engineering change of the window is the policy-token group: a robots.txt construct that reuses RFC 9309 syntax but deliberately breaks one of its matching rules. Your existing wildcard block does not express a training preference, and the enforcement calendar that makes this matter is already dated — Microsoft starts honoring it on February 1, 2027. Everything else in the window is a patch queue: a breaking image-pipeline security release, a CDN rule that does not actually fix it, and a cache migration with a new bill attached.

The __usage_ai-training Policy-Token Group in robots.txt

Identifier and date: draft-kelley-aipref-training-preference-signal-00, October 2026, an IETF Internet-Draft — a work in progress that expires 3 May 2027, not a ratified standard. Vendor status: Google has announced intent with an early-2027 target per its crawler documentation; Microsoft begins honoring it February 1, 2027. Nothing enforces it today.

Breaking or non-breaking: non-breaking to deploy. The draft introduces no new directive names and no new grammar, so existing parsers treat a policy-token group the same way they treat any group for an unrecognized user agent — they ignore it. The symptom if you ignore it is not an error but a default: from February 1, 2027, Microsoft reads "no applicable preference" as Allow, and a site that assumed its old vendor blocks carried over will have expressed nothing.

Three semantics from the draft will catch you if you skim it. First, a wildcard group MUST NOT be interpreted as matching the policy token, so an existing User-agent: * block conveys no training preference at all. Second, consumers match the token exactly apart from case, so a typo silently means "no preference" rather than failing loudly. Third, the token is not a crawl directive: per section 4.4 it does not stop a fetch, and if you want to limit crawling you still need groups keyed to real crawler product tokens. Paths, longest-match precedence and group structure are unchanged from RFC 9309, and within a policy-token group Disallow expresses a preference against the use while Allow expresses one in favor. Proprietary tokens such as Google-Extended keep working alongside it, and consumers may document that they treat those as more specific.

# Traffic management: unchanged, still governs crawling (RFC 9309).
User-agent: *
Disallow: /admin/

# Usage preference: AI training. NOT matched by "User-agent: *".
# Disallow = preference against training; Allow = preference in favor.
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

# Vendor-specific controls keep working and may be treated as more specific.
User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

Microsoft's Page-Level AI Training and AI Grounding Preferences

Identifier and dates: Bing's training and grounding preferences, published October 30, 2026. December 15, 2026 opens temporary grounding preferences in Bing Webmaster Tools for sites using NoArchive or NoCache as of that date. February 1, 2027 is when Microsoft begins honoring the new preferences as it crawls. March 1, 2027 is the deadline after which NoArchive and NoCache are no longer read as AI preferences. Any temporary preference expires September 30, 2027.

Breaking: yes, for any site relying on NoArchive or NoCache. The symptom if ignored is a silent widening of permission rather than an error — both preferences default to Allow, so content previously withheld becomes eligible for training and grounding once the old tags stop counting. Two asymmetries matter when you implement: AI Training can be set in robots.txt or per page, and the page-level setting overrides robots.txt for that page, while AI Grounding is page-level only with no robots.txt equivalent at all.

The file or setting to change is the document head, or the equivalent HTTP response header: X-Robots-Tag: Bingbot: ai-training:disallow and X-Robots-Tag: Bingbot: ai-grounding:disallow, with ai-grounding:disallowextended for the narrower grounding option. Two operational notes from Microsoft's guide: these settings govern use of content, not crawl access, so an Allow does not override a robots.txt rule blocking Bingbot; and publishing a preference is not activating it, because Microsoft must recrawl the page to discover it. IndexNow can raise the chance of a recrawl but guarantees no date, which is exactly why the temporary Bing Webmaster Tools preference exists. Full matrix and FAQ: Bing Webmaster Tools help.

<!-- Page-level AI Training: overrides the robots.txt preference for THIS page -->
<meta name="Bingbot" content="ai-training:disallow">

<!-- Page-level AI Grounding: no robots.txt equivalent exists -->
<meta name="Bingbot" content="ai-grounding:disallow">

<!-- Narrower grounding option: Microsoft-operated surfaces only,
     excludes grounding Microsoft provides to third parties -->
<meta name="Bingbot" content="ai-grounding:disallowextended">

Nuxt Image 2.2.1 and ipx 4.1.0: Security Fix With a Breaking Input-Format Change

Version and date: @nuxt/image v2.2.1, published October 30, 2026, with v1.11.2 on the 1.x line; patched ipx releases are 2.1.2, 3.1.3 and 4.1.0. The release notes cite GHSA-qf89-6xm4-66wm for ipx and GHSA-p5vf-pjf5-ggj3 for @nuxt/image. The flaw is an integer overflow during multi-page image decoding in ipx: a crafted source image could corrupt server memory through a single unauthenticated request to the /_ipx route. Sites where an attacker cannot control any image source are not affected. Rollout status: released and installable now.

Breaking: yes, and the break is in the SEO-visible path rather than the security fix. /_ipx now accepts only JPEG, PNG, WebP, GIF, SVG, AVIF and HEIC as input and returns 403 to anything else before decoding it. The symptom if ignored is broken images in rendered pages and dropped image results for any pipeline that transforms another source format — TIFF being the common one. The second trap is resolution: ipx is an optional dependency, so upgrading @nuxt/image does not guarantee a patched ipx. Run npm ls ipx and confirm ^4.1.0 actually resolved before you call this done, then redeploy, because a rebuild is what puts the new handler in place.

export default defineNuxtConfig({
  modules: ['@nuxt/image'],
  image: {
    ipx: {
      // Only add formats you genuinely transform. Everything outside the
      // default web set now 403s before decode (@nuxt/image 2.2.1).
      allowedInputFormats: ['jpeg', 'png', 'webp', 'gif', 'svg', 'avif', 'heic', 'tiff'],

      // For untrusted/remote sources, harden further: blocks decoders for
      // all other formats process-wide.
      restrictDecoders: true,
    },
  },
})

// Verify the optional dependency actually resolved to a patched build:
//   npm ls ipx   -> expect ipx@^4.1.0 (or 3.1.3 / 2.1.2 on older lines)

Cloudflare's Emergency WAF Rule Is Not the Fix for That Bug

Identifier and date: WAF Release 2026-10-30 — Emergency, October 30, 2026. It adds "Nuxt Image / IPX - Explicit GIF or Animation Modifier Exposure" to the Cloudflare Managed Ruleset. Two properties make it a trap rather than a shield: the default action is Disabled, so it is not protecting anything until you enable it, and Cloudflare states the rule offers partial coverage and recommends patching to the latest Nuxt Image or IPX regardless. Non-breaking to enable; the symptom if you treat it as the remedy is an unpatched origin behind a rule that was never on.

The setting to change is the Managed Ruleset override for that rule ID in your zone, and the honest answer is to patch first and enable the rule as depth rather than instead. Two adjacent items from the same October 30 changelog are worth a glance while you are in there: Browser Rendering's Quick Actions default user agent moved from Chrome 119 to Chrome 152, which breaks any rule matching the old string, and a Workers subrequest change now reflects the original end-user connection in Bot Management fields for BYOIP zones. Both are the kind of edge-layer drift that silently changes what your crawler-verification and bot rules actually see — and the same review is the moment to confirm the CDN is not quietly overriding your robots.txt intent.

Vercel Data Cache Becomes Runtime Cache, With Reads and Writes Now Billed

Identifier and date: Data Cache is moving to Runtime Cache, October 30, 2026. Data Cache leaves beta and merges into Runtime Cache. Next.js fetch caching and unstable_cache move across on every version with no code changes. Teams created on or after October 30, 2026 are on Runtime Cache by default; Hobby teams migrate November 29, 2026; Pro teams are notified of their own migration date and Enterprise teams are contacted directly.

Non-breaking in code, breaking in budget — which is why it belongs in a technical SEO review rather than only a finance one. Data Cache reads and writes that were previously unbilled are now charged at Runtime Cache rates, starting at $0.40 per million reads and $4.00 per million writes and varying by region. The symptom if ignored is a new cost line that scales with exactly the behaviour aggressive revalidation encourages: a large sitemap regenerated often, or short revalidate windows across many routes, is now a metered pattern. The setting to look at before your migration date is the project's Data Cache usage in Observability; if the numbers surprise you, the lever is revalidation frequency and cache granularity, not switching caching off.

Quiet Elsewhere in the Engineering Track

Stated rather than padded. Next.js shipped no stable release in the window — only 16.5.0 canaries 31 through 36 between October 29 and 31, including a metadata-prerender revalidation fix, so there is nothing here to upgrade a production Metadata API or app/sitemap.ts setup for. Schema.org's most recent release remains 30.1 from 2026-09-16, so no vocabulary change touches your JSON-LD this week. No Chrome stable milestone landed in the window, and no Core Web Vitals threshold or measurement change was published. No crawling incident appeared on the Search Status Dashboard.

Ship Today

1. Patch the image pipeline: upgrade @nuxt/image to 2.2.1 or 1.11.2, confirm ipx resolved to a patched build with npm ls ipx, add any non-default source formats to image.ipx.allowedInputFormats, and redeploy.
2. Crawl your own image routes after that deploy and look for new 403s on /_ipx; that is the breaking input-format change showing up as missing images.
3. Enable the Cloudflare emergency rule as defence in depth, knowing it is disabled by default and partial — never as a substitute for the patch.
4. Add the __usage_ai-training group to public/robots.txt as a deliberate decision, remembering that your User-agent: * block expresses nothing under the new scheme.
5. Keep the vendor-specific groups — Google-Extended, GPTBot, ClaudeBot — in place; the shared token is unenforced until 2027 and vendors may treat proprietary tokens as more specific.
6. Inventory NoArchive and NoCache across templates and open the migration ticket now, dated before March 1, 2027.
7. Review Data Cache usage in Vercel Observability before your team's migration date, and check revalidation frequency on high-route-count sitemaps.
8. Verify the robots.txt your CDN actually serves matches the file in your repository, and re-check any bot rule that matches a hardcoded Chrome user-agent string.