Daily SEO Note — October 17, 2026: Cloudflare Starts Grading Whether Verified Bots Obey Robots.txt

SEO for Content Writers

The one change that should alter an editorial decision today came from a CDN, not from Google. On October 16, Cloudflare began continuously re-checking whether verified bots honor the preferences sites publish, and scoring them on it. Google's own Search Status Dashboard shows no ranking or policy update inside the last 24 hours, so nothing in this section is a reaction to an algorithm change.

Your crawler preferences now decide who is allowed to quote you

Cloudflare's October 16 post describes a system that keeps checking whether a verified bot does what it said it would do, rather than treating verification as a badge awarded once. The two signals it starts with are whether the bot respects robots.txt and whether it slows down after an HTTP 429 response. The result is published as an explainable trust band on Cloudflare Radar. Cloudflare says the system is in internal use for now, with customer-selectable thresholds planned but not yet available, so this is a change of direction rather than a switch that flips today.

For editorial teams the consequence is indirect but real. The file that decides whether an AI assistant may fetch your article is the same file now being used as evidence of whether that assistant behaves. That makes a blanket "block all AI" rule more expensive than it looks, because the crawlers that produce citations are not the crawlers that collect training data. OpenAI's crawler documentation describes OAI-SearchBot as the fetcher behind ChatGPT's search answers and GPTBot as the training crawler, and states that each setting is independent: a site can allow one while disallowing the other.

What to do differently: in the next brief, name the answer surface the piece is meant to be quoted in, then check that the matching search-side fetcher is allowed before the piece goes live. What to stop doing: stop writing "AI crawlers blocked" as a single line in a content policy. It collapses a training decision and a visibility decision into one, and only one of them costs you citations.

Google removed the Indexing API's quota escalation route

Google's documentation changelog records an October 16 update to the Indexing API quota page, described as reflecting the current quota limits and process, with an outdated form removed. The page as it now stands states a default of 200 publish requests per day per project, 180 getMetadata requests per minute, and 380 requests per minute across all endpoints, with the daily quota resetting at midnight Pacific Time and all use free of charge. What is no longer on it is any route to ask for more.

This lands on one vertical rather than on all content. The companion usage page still restricts the API to pages carrying JobPosting markup or BroadcastEvent embedded in a VideoObject, which means job boards, recruitment desks, and livestream or event publishers are the teams that feel it. For every other kind of content the API was never an option, and sitemaps plus ordinary crawling remain the route.

What to do differently: treat 200 submissions per project per day as a fixed editorial budget, and rank postings by value before submitting rather than firing the API at every new URL. What to stop doing: stop planning a launch around a quota increase. The form that used to grant one, and the approval path for job-posting and livestream use, are no longer documented.

No confirmed Google ranking change today

The Search Status Dashboard lists no October 2026 ranking update. The most recent entry remains the September 2026 spam update, which began on 24 September 2026 and ran for 13 days and 16 hours. Against that, community trackers reported a volatility and chatter spike on October 16. That report is unconfirmed, has no corresponding dashboard entry, and is recorded here only so it is not mistaken for a verified event.

There is a second reason to read rank data carefully at the moment. The rank-tracking vendor Nozzle published research on October 10 reporting that a share of the Google result pages it scrapes come back altered, and that the altered pages look complete and ordinary. Google has not commented, so treat the finding as the vendor's own measurement. The editorial consequence is narrow and worth stating plainly: confirm a suspected drop in Search Console query data before commissioning a rewrite, because a tracker graph alone no longer settles the question.

Apply to your next brief

• Name the answer surface the piece should be quotable in, and confirm the matching search-side fetcher is allowed before publishing.
• Split crawler decisions three ways in the content policy — training, search, and user-triggered — instead of a single "AI" toggle.
• Job and livestream desks: plan against 200 Indexing API submissions per project per day, and rank postings before submitting.
• Drop any brief that assumes an Indexing API quota increase is obtainable; the documented request path is gone.
• Confirm suspected ranking movement in Search Console query data before briefing a rewrite.
• Record which AI fetchers the site allows in the brief template, so writers and engineers work from the same list.

SEO for Developers

The consequential engineering change today is that your robots.txt and your rate-limit responses became inputs to somebody else's enforcement system. After Cloudflare's October 16 announcement, a malformed robots file or a limiter that answers 403 instead of 429 no longer merely fails to stop a crawler — it fails to register as a preference at all. Nothing shipped in the frameworks, Core Web Vitals tooling, or structured-data vocabularies inside the window.

Cloudflare re-checks verified bots against robots.txt and HTTP 429

Rollout: published 16 October 2026 at 13:00 UTC. Non-breaking, and not yet customer-configurable — Cloudflare states the system is in internal use while it builds history, with trust bands surfacing on Cloudflare Radar and a customer-selectable trust threshold planned. The symptom if ignored is quiet rather than loud: your stated preferences never become evidence, so crawler behavior against your origin is judged on a weaker signal than it could be.

The file to change is public/robots.txt. Cloudflare's post names RFC 9309 as the expectation it measures against, and notes that robots.txt is a floor rather than a complete grant: anything it does not mention is not automatically permitted. Give each fetcher its own group and keep the training decision separate from the search decision, so the file states a real preference instead of a blanket.

# Search fetchers — allowed, these are what produce citations
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Training crawlers and training opt-out tokens — disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

# Everything else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Shed crawler load with 429 and Retry-After, not 403

Same source and date as the item above; this is the second of the two signals Cloudflare measures. A 429 tells a crawler to slow down, and with Retry-After it says for how long — and Cloudflare now watches whether the bot's request rate actually changes afterwards. A 403 carries no such instruction, and a 503 claims the server is broken rather than busy. Edge rules that shed bot traffic with 403 therefore throw away the one response that both protects the origin and registers as a preference.

The setting to change is whatever terminates rate-limited requests: Next.js middleware, a Cloudflare Worker, or the action on a WAF rate-limit rule. Set the status to 429 and always attach Retry-After in seconds. Cloudflare's post is explicit that consistently slowing down or pausing raises a bot's trust band while repeatedly ignoring the signal lowers it, which only works if the signal is actually sent.

import { NextRequest, NextResponse } from 'next/server'
import { isRateLimited } from './lib/rate-limit' // your limiter: KV, Redis, Durable Object

const RETRY_AFTER_SECONDS = 120

export async function middleware(request: NextRequest) {
  if (await isRateLimited(request)) {
    // 429 is the only status a crawler can act on. Do not use 403 or 503 here.
    return new NextResponse('Too Many Requests', {
      status: 429,
      headers: {
        'Retry-After': String(RETRY_AFTER_SECONDS),
        'Cache-Control': 'no-store',
      },
    })
  }

  return NextResponse.next()
}

export const config = { matcher: '/:path*' }

Google rewrote the Indexing API quota and usage pages

Rollout: recorded in Google's Search documentation changelog on 16 October 2026 as an update to the Indexing API quota page to reflect the current quota limits and process, with an outdated form removed. Non-breaking as a code change; breaking as a capacity assumption, because the documented way to request more than the default is gone.

The quota page now states 200 publish requests per day per project, 180 getMetadata requests per minute, and 380 requests per minute across all endpoints, resetting at midnight Pacific Time, with current usage visible in the Google API Console. The usage page adds two details worth encoding in the client: quota is counted at the URL level, and batching does not reduce consumption, since a batch of 100 calls still spends 100 URLs of quota.

The symptom if ignored is a submission job that starts failing partway through the day once the project's 200 publish calls are spent, with no escalation path to request more. The fix is to make the cap explicit in the submitter rather than discovering it from the API, and to stop rather than retry when quota is exhausted.

// Scope: the Indexing API may only be used for pages with JobPosting markup,
// or BroadcastEvent embedded in a VideoObject.
// Quota is counted per URL; batching does NOT reduce consumption.
const DAILY_PUBLISH_QUOTA = 200
const ENDPOINT = 'https://indexing.googleapis.com/v3/urlNotifications:publish'

export async function submitUrls(urls: string[], accessToken: string) {
  const budget = urls.slice(0, DAILY_PUBLISH_QUOTA)

  if (urls.length > budget.length) {
    console.warn(
      `Deferring ${urls.length - budget.length} URLs: daily quota is ${DAILY_PUBLISH_QUOTA}.`
    )
  }

  for (const url of budget) {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${accessToken}`,
      },
      body: JSON.stringify({ url, type: 'URL_UPDATED' }),
    })

    if (res.status === 429) {
      // RESOURCE_EXHAUSTED. There is no documented form to request an increase.
      console.error('Quota spent for today — stopping instead of retrying.')
      break
    }
  }
}

Two WordPress advisories land on the SEO delivery layer

Published 17 October 2026. CVE-2026-107763 affects Fastcache by Host.it in all versions up to and including 1.7.5, rated Moderate at CVSS 4.3: a missing authorization check lets any authenticated user at subscriber level or above add or remove the plugin's page-cache rewrite rules in the site's root .htaccess. That is a caching plugin handing write access to the file that governs what crawlers are served, reachable from the lowest account tier a site offers.

The second, CVE-2026-104734, affects The Kirki in all versions up to and including 6.3.2 and is rated High at CVSS 7.2 — PHP object injection, though it requires high privileges and the advisory notes that no known POP chain ships in the plugin itself. Both advisories list patched versions as unknown at publication, so the available mitigation is to restrict registration, audit the plugins, and monitor the files that control crawling.

#!/usr/bin/env bash
# Alert on unexpected changes to the files that control crawling and redirects.
set -euo pipefail

WATCH=("/var/www/html/.htaccess" "/var/www/html/robots.txt")
STATE="/var/lib/seo-guard/hashes.sha256"

mkdir -p "$(dirname "$STATE")"

if [ -f "$STATE" ] && ! sha256sum -c --status "$STATE"; then
  echo "SEO guard: a crawl-control file changed unexpectedly" >&2
  sha256sum -c "$STATE" || true
  exit 1
fi

sha256sum "${WATCH[@]}" > "$STATE"

Everything else in the stack: no change in the window

One line, as promised: no framework, Core Web Vitals, or structured-data release inside the 24-hour window carried an SEO-relevant change. For the record, current versions are Next.js 16.4.2 (15 October, already covered on this site), SvelteKit 3.1.0 (14 October), Astro 7.4 (15 October), web-vitals 6.2.3 (5 October), Lighthouse 13.5.0 (18 September), and Schema.org 30.1 (16 September); the Vercel and Cloudflare changelogs carried no SEO-affecting entries on 16 October.

Ship today

1. Validate public/robots.txt against RFC 9309 and give each AI fetcher its own group, separating training from search.
2. Change rate-limit responses for crawler traffic from 403 or 503 to 429 with a Retry-After header.
3. Hard-cap Indexing API submissions at 200 per project per day and stop, rather than retry, once the API reports quota exhaustion.
4. Confirm every URL your Indexing API job submits carries JobPosting markup or BroadcastEvent inside a VideoObject.
5. On WordPress: audit Fastcache (1.7.5 and below) and Kirki (6.3.2 and below), restrict subscriber registration, and add a hash monitor on .htaccess and robots.txt.