Daily SEO Note — August 29, 2026: Google Confirms AI Overviews Now Expand Themselves

1. SEO for Content Writers

Google confirmed on August 28, 2026 that AI Overviews now open in their expanded state on some queries, with no click on Show more. The confirmation came as a spokesperson statement, not a blog post or a documentation update. If you write anything that competes for the space under an AI Overview, this is the one change today that should alter how you plan a page.

AI Overviews expand themselves, and Google's own documentation has not caught up

Google's wording is narrow and worth quoting exactly: "For some queries, AI Overviews may dynamically expand for topics where our systems determine it's most useful for people." The company added that "our research has shown that with this dynamic experience, users find Search more helpful and engage deeper in follow-up exploration," and that the expansion stops if the reader has already started scrolling.

Two things follow for an editor. The first is mechanical: on an affected query the AI answer occupies more of the first screen from the moment it loads, and the Ask anything box arrives already open, which routes the next step into AI Mode rather than into a blue link. The second is that this is not documented anywhere you can cite to a stakeholder. Google's own AI Overviews help page still instructs readers to "select Show more at the bottom of the AI Overview to open the detailed view." Rollout status: live on an undisclosed share of queries, confirmed by statement, undocumented.

What to do differently: stop writing the opening of an article as though it will be read after a click from position one, and start writing the first 60 words as the thing that gets quoted instead. A definition sentence that stands alone, a comparison the model can lift as a row, or a number only you measured are what survive into an expanded overview. What to stop doing: treating a rising impression count with a falling CTR on a head term as a title problem. On these queries it is a layout change, and rewriting the title will not recover the click.

Your AI content policy became a published declaration, not a private setting

Cloudflare launched BotBase for Operators on August 28, 2026, a directory in which the companies running crawlers declare, on the record, what their bot does and how it uses what it takes. Operators state a behavior (indexing, acting as an agent, data collection, model training, SEO support), an operator type (direct or intermediary), and a content use expressed in the Content Signals vocabulary.

This affects all content, not a vertical, and it matters to writers because the Content Signals vocabulary splits two decisions that are usually collapsed into one argument. Cloudflare's own example of a site's stated preference is search=yes, ai-train=no: be findable and quotable in AI search answers, but stay out of model training corpora. Those are separate switches, and somebody at your organization has already set them.

What to do differently: before you commission another round of first-hand testing or proprietary data intended to earn AI citations, ask which way those two signals are set on your domain. Commissioning original research as a citation asset while the site declines AI search crawlers is money spent against your own configuration. What to stop doing: describing the choice internally as "blocking AI." There is no single switch, and the search and training decisions carry opposite consequences for visibility.

PDFs thinning out of results — unconfirmed

Practitioners reported on August 28, 2026 that Google is surfacing noticeably fewer PDF files, including documents on government domains, with Search Console click and impression drop-offs offered as evidence. Label this unconfirmed. Google has said nothing, and the Search Status Dashboard logged no incident across August 21 to 28, 2026. It is a signal to check, not a finding to plan around.

If any part of your library reaches readers as a PDF — whitepapers, annual reports, spec sheets, price lists — pull a Search Console filter on .pdf URLs this week and look at the trend yourself rather than relying on the anecdote. The durable editorial answer does not depend on how this resolves: a PDF that matters deserves an HTML page carrying the same content, with the PDF offered as the download rather than as the indexed asset.

Nothing moved in the ranking, spam, or Discover systems

No core update, no spam update, no Discover or Top Stories change in the window. The documentation changelog logged nothing after the August 28 favicon entry covered in yesterday's note, and the AI features page is still unchanged since December 10, 2025, so the controls governing your appearance in AI Overviews are the same ones you had yesterday: nosnippet, data-nosnippet, max-snippet and noindex.

Apply to your next brief

• Write the first 60 words as a quotable, self-contained answer. On an auto-expanded query that passage is the whole of your visibility, and it has to make sense with no headline above it.
• Add one thing to every brief that cannot be synthesized: a measurement you took, a price you checked, a test you ran. Restated consensus is exactly what an expanded overview replaces.
• Plan for the follow-up question, not just the query. The Ask anything box now opens with the overview, so the second question is where readers go next — cover it in the same article rather than in a separate post.
• Stop diagnosing head-term CTR decline as a title-tag failure until you have checked whether the query carries an AI Overview. Rewriting a compliant title against a layout change wastes a cycle.
• Find out how your domain answers search=yes / ai-train=no before commissioning content whose purpose is to earn AI citations, and put the answer in the brief.
• Audit any content that only exists as a PDF and schedule an HTML equivalent. Treat the reported decline as a reason to check your own data this week, not as an established fact.

2. SEO for Developers

The day's real engineering item is a denial of service in the library that parses robots.txt for most Python crawlers. It is not a new vulnerability, but it entered GitHub's reviewed advisory database on August 28, 2026, which is the moment Dependabot starts telling you about it.

Protego 0.6.2: a hostile robots.txt can hang your crawler (CVE-2026-55520)

GHSA-wjmf-p669-5m5p is CVE-2026-55520, a regular expression denial of service in Protego, scored CVSS v4 7.1 (High) and classified CWE-400 and CWE-1333. Affected: Protego <= 0.6.1. Patched: 0.6.2, which is the current release on PyPI. Note the dates carefully — the advisory was published June 25, 2026, but was reviewed and last updated August 28, 2026. Nothing about the flaw changed yesterday; its status in the database did, and that is what puts it in front of your dependency scanner today.

Protego turns robots.txt wildcards into regular expressions, converting each asterisk into a lazy .*? pattern. A Disallow or Allow value carrying many asterisks compiles to a regex that backtracks exponentially, so the hang happens during matching rather than parsing. The advisory's proof of concept is a Disallow directive with twelve asterisks matched against a 60-character URL, which never returns. Non-breaking patch release; the symptom if ignored is a crawl worker pinned at 100% CPU and stalled indefinitely on a single host, with the cost scaling by URL length.

The blast radius is wider than the package name suggests, because Protego is the default robots.txt backend in Scrapy — ROBOTSTXT_PARSER defaults to scrapy.robotstxt.ProtegoRobotParser. Any in-house site auditor, broken-link checker, or SERP scraper built on Scrapy fetches robots.txt from hosts it does not control, which is precisely the exposure. A crawler restricted to your own domains is not at risk from a third party, but is still worth patching.

# CVE-2026-55520 / GHSA-wjmf-p669-5m5p - ReDoS in robots.txt wildcard matching
# Affected: Protego <= 0.6.1   Patched: 0.6.2
pip install --upgrade 'Protego>=0.6.2'

# Scrapy pulls Protego in transitively - check the resolved version, not the pin
pip show Protego | grep -i version
pip list 2>/dev/null | grep -iE 'protego|scrapy'

# Confirm which parser your crawler actually uses (default: ProtegoRobotParser)
python -c "from scrapy.settings import default_settings as d; print(d.ROBOTSTXT_PARSER)"

Cloudflare opens BotBase — and this is the day to diff your served robots.txt

Cloudflare shipped BotBase for Operators on August 28, 2026, under Protect & Connect → Application Security → BotBase. It gives crawler operators a searchable directory, a submission form and a review status, and it automates verification of IP lists, reverse DNS and Web Bot Auth signatures that a human previously checked by hand. Non-breaking, and nothing to change on your side to consume it — but it is a reliable place to look up what a user agent in your logs actually claims to do before you write a rule about it.

The operational warning sits in the adjacent feature. Bot Preference Sync, covered in Cloudflare's August 21, 2026 post and generally available from the Free tier upward, prepends Cloudflare-managed directives to your robots.txt so the file reflects the AI bot policy set in the dashboard. It is opt-in and enabled by default for new customers. That means the robots.txt in your repository and the robots.txt on your origin's edge are now two different documents, and the one crawlers read is the second.

This is the CDN-overriding-your-intent case, and it is worth a regression check rather than a one-off look. Fetch the live file, diff it against the one in version control, and assert that the training and search decisions are the ones you meant. Content Signals expresses them separately — Cloudflare's worked example of a site's stated preference is search=yes, ai-train=no — so a policy set in the dashboard by one team can quietly contradict a robots.txt maintained by another.

#!/usr/bin/env bash
# Fail the build if the edge is serving a robots.txt we did not commit.
set -euo pipefail
ORIGIN="https://example.com/robots.txt"

curl -sS --fail -A 'Mozilla/5.0 (compatible; robots-regression-check)' "$ORIGIN" -o /tmp/robots.live.txt

# Cloudflare Bot Preference Sync PREPENDS managed directives - a diff is expected
# once it is enabled. Read it, then update the committed baseline deliberately.
if ! diff -u public/robots.txt /tmp/robots.live.txt; then
  echo 'WARN: served robots.txt differs from the committed file.'
fi

# Assert the two AI decisions explicitly, so a dashboard change cannot flip them silently.
grep -qi 'search=yes'   /tmp/robots.live.txt || { echo 'FAIL: AI search opt-in missing'; exit 1; }
grep -qi 'ai-train=no'  /tmp/robots.live.txt || { echo 'FAIL: AI training opt-out missing'; exit 1; }

# Sitemap must still be reachable and must not 3xx into a redirect chain.
curl -sS -o /dev/null -w 'sitemap: %{http_code} redirects=%{num_redirects}\n' \
  --fail 'https://example.com/sitemap.xml'

Quiet surfaces, checked and empty

Next.js stable is still 16.3.3 from August 25; August 28 produced only 16.4.0-canary.10 and canary.11, whose routing and cache-key fixes are not shippable to production. Astro has nothing after 7.2.9 on August 27. Lighthouse cut no release. Vercel's changelog carried no entries touching redirects, rewrites, middleware, ISR, caching or image optimization. The Google common crawlers documentation is unchanged since July 14, 2026, Schema.org is still on release 30.0 from March 19, 2026, and the Cloudflare changelog logged no bot, cache rule or WAF entries in the window. No structured data feature changed eligibility today.

Ship today

1. Upgrade Protego to 0.6.2 anywhere a crawler fetches robots.txt from hosts you do not own. Check the resolved version rather than the declared pin, because Scrapy pulls it transitively.
2. Add a timeout around robots.txt matching in any long-running crawl worker. Patching removes this instance of the hang; a wall-clock bound removes the class of failure.
3. Diff your served robots.txt against the committed file. If Bot Preference Sync is on, the edge is prepending managed directives and the two have already diverged.
4. Put that diff in CI as a regression monitor alongside sitemap availability and canonical drift, so a dashboard toggle cannot change your crawler policy without a failing build.
5. Confirm the search and training signals separately with the team that owns the Cloudflare account. Blocking AI search crawlers while commissioning content to earn AI citations is the contradiction worth catching this week.
6. If you serve indexable PDFs, pull a Search Console filter on .pdf URLs and verify the trend yourself before treating the reported decline as real.