The web getting ready for AI agents — where the transition is actually happening

In short

The 2026 web is getting ready for AI agents, but not in the way popular discourse usually imagines. Google has not retreated: its global market share stands at 90.04 % in April 2026 (Statcounter), its query volume grew 21 % in 2024, and its search revenue rose 15 to 17 % year over year in Q3 and Q4 2025 (Alphabet earnings). Sites have not all switched to headless architecture: WordPress still holds 42-43 % of the web (W3Techs). AI agents will not be browsing the web on their own tomorrow: the best agent evaluated on a benchmark of realistic office tasks tops out at 30.3 % (TheAgentCompany, CMU). And there is no universal standard: seven competing protocols coexist, with zero ratified W3C/IETF RFC.

What is really changing is measurable elsewhere: the massive erosion of the outbound click from search result pages (zero-click at 58-69 %, organic CTR down 34 to 61 % on SERPs carrying AI overviews), the bimodality of web architecture (composable enterprise vs consumer WordPress), the fragmented emergence of agentic standards (MCP, A2A, AGNTCY) and the structural tension between exposure to agents (Microsoft NLWeb) and closure/monetisation (Cloudflare Pay Per Crawl). This article maps what is actually moving, source by source.

In plain terms: the web ↔ AI agents transition is real, but it is bimodal, bidirectional, complementary and fragmented. Not a massive, universal substitution.

Understanding the shift: outbound-click disintermediation

Picture a newsstand at the corner of a busy street. For decades, passers-by stopped, scanned the covers, bought a paper, walked away with it in hand. The newsagent — the search engine — lived off a commission on that transaction. Now imagine the newsagent puts up a display board in front of the stand that summarises each paper’s front page in large type. Passers-by read the summary, nod, and walk on — without opening a paper, without stepping into the stand.

This is what we call outbound-click disintermediation: the engine has not lost its audience, it has simply reduced the need to click through to the sites that produce the information. The newsagent keeps the footfall; the publishers lose the sale.

Three empirical orders of magnitude make the shift measurable.

Zero-click search. For every 1,000 US Google queries, only 374 clicks reach the open web (SparkToro/Datos clickstream study, July 2024). The European ratio is similar (360/1,000). On news queries, SimilarWeb measures 69 % zero-click in May 2025, up 13 points in a year. When an AI Overview — an AI summary at the top of the results page — appears, zero-click rises to 83 %.

Organic CTR in freefall. Four converging studies measure the drop in click-through rate on SERPs carrying AI overviews: Ahrefs -34.5 %, Authoritas -47.5 %, Seer Interactive -61 % (September 2025 update), Kevin Indig beyond -50 %. This is not a one-off effect: the convergence of four different methodologies documents a structural break.

Google → publisher traffic collapsing. 2.3 billion monthly visits to news sites in mid-2024, fewer than 1.7 billion in May 2025 — -26 % in eleven months according to SimilarWeb (relayed by AdExchanger). Documented individual cases: CNN -27 to -38 % year over year, Forbes -50 % in July 2025, Business Insider -40 to -48 %.

In plain terms: this is not a market-share transfer between engines. It is the erosion of the outbound click — the commission stops being levied on the site visit and is captured inside the engine’s summary. Publishers downstream bleed.

§1 — Headless bimodality: enterprise vs consumer

Before going further, some vocabulary for the technical passages that follow.

Vocabulary

• Headless: a web architecture in which content (back end) is separated from presentation (front end). Content is consumed through an API and the rendering is chosen freely on the client side (site, mobile app, AI agent).
• Composable: generalisation of the headless philosophy to the whole stack — every component (CMS, e-commerce, search, payment) is interchangeable and exposed by API.
• API-first: a design in which the API is conceived before the visual rendering; every consumer — human or agent — reaches the same content through the same documented channel.
• MACH: an acronym for Microservices + API-first + Cloud-native + Headless. It names a family of architectures championed by the MACH Alliance, a vendor consortium.

On the enterprise side, the transition is documented. Gartner’s Magic Quadrant for Digital Experience Platforms 2025 (as summarised by CX Today) projects 70 % of organisations under a composable mandate in 2026, against 50 % in 2023 — a 20-point rise in three years. The MACH Alliance, in its 2025 annual report (561 enterprise IT decision-makers surveyed), reports 87 % of companies having implemented MACH and 93 % of adopters declaring their return on investment met or exceeded. The headless commerce market grew from 1.7 to 2.1 billion dollars in 2025 to a projection of 5.5 to 7 billion in 2030-2032, a CAGR of around 21 % (Coherent Market Insights).

On the consumer and SMB side (small and medium businesses), the resistance is documented just as clearly. WordPress still holds 42.2 to 43 % of the total web according to W3Techs (May 2026), with a slow decline — roughly 5 points over four years. Wix and Squarespace combined account for ~9 % of the web and keep growing. These are integrated, non-headless platforms gaining ground alongside composable stacks: the transition is not one-directional.

In plain terms: the consumer web looks like a residential neighbourhood — detached houses dominate, inherited from past decades, and even new builds still favour simple formats (WordPress, Wix). The enterprise web looks like a high-end office district — the demand for composability and scalability pulls toward API-first stacks. The two neighbourhoods coexist in the same city.

One methodological caveat deserves naming: most figures on MACH adoption come from the vendors themselves (MACH Alliance, BigCommerce, Coherent Market Insights). Structural marketing bias pushes amplitudes upward. The directional convergence remains robust — Gartner DXP 2025, an independent analyst, confirms the trajectory — but the precise numbers (87 %, 70 %) should be read as upper bounds.

Three operating heuristics complete the matrix. Start from real complexity, not imagined complexity: a monthly blog with five articles gains nothing from a composable stack. Do not migrate for the sake of migrating: a WordPress site that serves its purpose needs no switch until a genuine multi-channel requirement appears. Measure the governance cost: composable means more parts, therefore more contracts, more audit surfaces, more versions to track.

§2 — Google did not retreat: it adapted

One recurring narrative predicted that language models would overturn Google’s monopoly. The empirical 2024-2026 figures tell a different story.

Google remains massively dominant. Its global market share stands at 90.04 % in April 2026 according to Statcounter. In 2024, its query volume grew +21.64 % year over year (SparkToro/Datos study), crossing 5 trillion annual queries — roughly 14 billion a day. On the revenue side, Alphabet reported search revenue up 15 % year over year in Q3 2025 (56.6 billion dollars) and 17 % in Q4 2025 — primary data extracted from the quarterly earnings calls.

Consumer LLMs remain a marginal volume. The empirical ratio between Google and ChatGPT stands at 373× more search-like queries on the Google side (SparkToro/Datos, March 2025). All consumer LLMs together (ChatGPT + Perplexity + Claude + Copilot + Gemini) account for less than 2 % of the total search market in 2025-2026. ChatGPT announced 800 million weekly active users in October 2025 (Sam Altman, TechCrunch) and Perplexity claims 780 million search queries in May 2025 (CEO Aravind Srinivas, Bloomberg Tech Summit) — impressive figures that remain modest against Google’s volume. And the LLM pool’s growth is slowing: SparkToro notes that no month since September 2024 has shown a factor above 1.1× on the segment.

Google adapts rather than loses. AI Mode — the conversational engine embedded in the results — has been rolled out in 40 languages, counted 75 million daily active users in Q3 2025 and saw its query volume double over the quarter. AI Overviews are deployed at scale and capture commercial intent before the click.

The real scenario = asymmetric coexistence. Pew Research, in its representative academic study of June 2025, measures 34 % of US adults having used ChatGPT (vs 17 % in 2023, a doubling in two years), 58 % of the under-30s, and 28 % of employees for work. But the same SparkToro panel (August 2025) confirms that 95 % of Americans still use a traditional search engine every month — a figure stable since 2023. The two uses coexist; the LLM adds, it does not replace.

In plain terms: Google did not lose its monopoly, it transformed its engine. The value of commercial intent is captured inside the AI Overview, and the commission that funds the open web is intercepted upstream of the click. Publishers are not absorbing a market-share transfer between engines; they are absorbing a reduction in the quantity of clicks available across the whole web.

§3 — AI agents: impressive in demos, fragile in production

Three definitions first, to frame what follows.

Vocabulary

• Web AI agent: a program driven by an LLM that controls a browser or a web API (clicking, typing, reading the DOM, calling a tool) to accomplish a human-defined task.
• Computer use: an extension of tool use in which the agent directly controls a machine’s screen — it “sees” through screenshots and acts through click coordinates and keystrokes.
• OSWorld, WebVoyager, WebArena: public benchmarks measuring an agent’s ability to complete tasks on a real operating system or website.

State-of-the-art capabilities in 2026. Benchmark progress is spectacular. On OSWorld, the score went from around 20 % in December 2024 to 72.6 % for Simular Agent S in December 2025 — a very brief overtaking of the human score measured on the same benchmark at 72.36 %. Anthropic published Claude Sonnet 4.5 at 61.4 % on OSWorld-Verified in September 2025, a gain of 19.2 points in four months. OpenAI Operator reaches 87 % on WebVoyager and 58.1 % on WebArena. Google Project Mariner climbs to 83.5 % on WebVoyager. On the open-source side, the browser-use project counts 92,500 GitHub stars in April 2026.

Commercial players in 2026. Four offerings are on the market: OpenAI ChatGPT Atlas (a Chromium-based browser launched 21 October 2025, with an integrated agent mode), Anthropic Computer Use API (launched 22 October 2024, improved on Claude 4.5+), Google Project Mariner (public since May 2025 on AI Ultra US), Perplexity Comet (global free tier on 2 October 2025, “millions” of sign-ups per Perplexity). The visible cost: Operator is included in ChatGPT Pro at 200 dollars a month, and Claude Opus 4.6 bills 5 dollars per million input tokens and 25 dollars per million output tokens. Against a Google search that is essentially free for the end user, the economic gap remains considerable.

Documented empirical limits. Three academic papers let us quantify the fragilities.

TheAgentCompany (CMU, December 2024, arXiv:2412.14161): on realistic office tasks — opening tickets, writing a contextualised email, manipulating a shared spreadsheet — the best agent (Gemini 2.5 Pro) completes 30.3 % of tasks. Claude 3.7 Sonnet 26.3 %. GPT-4o 8.6 %. Llama 3.1 405b 7.4 %. The study also documents behavioural hallucinations: an agent renames another user so that they bear the correct recipient’s name, having failed to find them — “functional” behaviour that no human would ever have chosen.

OSWorld-Human (June 2025, arXiv:2506.16042): on identical tasks, agents are 40 to 170 % less efficient in number of steps than humans, and their latency regularly reaches “tens of minutes” for tasks that take a human a few minutes; 75 to 94 % of total time is spent in LLM planning and reflection calls.

WASP (April 2025, arXiv:2504.18575): around 86 % of prompt injection attacks partially succeed against the web agents tested. CVE-2025-53773 documents a remote code execution (RCE) vulnerability in GitHub Copilot through prompt injection.

On commercial adoption, Gartner places AI agents at the “Peak of Inflated Expectations” in its 2025 AI Hype Cycle. The CIO Survey 2026 counts 17 % of organisations having deployed agents in production and 60 % planning deployment within two years — but also 40 % of agentic projects predicted to be cancelled by 2027.

In plain terms: 2026 web AI agents resemble a brilliant but distracted intern — able to book a simple flight or summarise three articles, lost as soon as the form changes, vulnerable to any instruction hidden in an email, and expensive full-time.

Three operating heuristics for teams assessing adoption:

• Choose the perimeter, not the agent. Performance depends as much on the task as on the model. A well-bounded task (controlled resources, human validation on output) tolerates an imperfect agent; an autonomous task with side effects does not tolerate a 30 % failure rate.
• Total cost, not LLM cost. Operator plus human review plus error correction can exceed the cost of a dedicated human in the first months. ROI is not computed on the token price alone.
• Security before scaling. WASP’s 86 % partial injection rate means an agent connected to mail or a browser without supervision is exposed to attacks today. See the agentic red teaming article for the audit methodology.

§4 — Emerging standards: fragmented and bidirectional

Four dynamics shape the landscape of standards meant to let agents talk to sites — and the reverse.

Vocabulary

• MCP (Model Context Protocol): an open protocol launched by Anthropic in November 2024 to standardise the connection between an LLM and external tools. Conceptually, a “USB for agents”.
• A2A (Agent-to-Agent): an agent interoperation protocol initially driven by Google, merged with IBM ACP under the Linux Foundation in September 2025.
• AGNTCY: a multi-vendor open initiative (Cisco, IBM, Google, LangChain) for agent coordination at the infrastructure level (directory, identities, signals).
• NLWeb: a Microsoft initiative (May 2025) for a native conversational layer on websites; created by RV Guha, who also gave us RSS, RDF and Schema.org.

MCP as the de facto standard. Launched on 25 November 2024, MCP has accumulated 85,100 GitHub stars on the modelcontextprotocol/servers repository, with an estimated 10,000 to 16,000 MCP servers in circulation (Pento.ai and thenewstack.io) and more than 97 million monthly SDK downloads. Major adopters: OpenAI (Agents SDK, Responses API and ChatGPT desktop, March 2025), Google DeepMind (Gemini, April 2025, confirmed by Demis Hassabis), Microsoft (Windows 11, Build May 2025), Cloudflare, IBM, AWS. Anthropic donated it to the Linux Foundation / Agentic AI Foundation in December 2025, a move that seals the protocol’s open character.

But MCP is not flaw-free. Three Remote Code Execution CVEs (CVE-2025-68143, 68144, 68145) were discovered by Solo.io in Anthropic’s official Git MCP server, exploitable through prompt injection. The de facto standard has documented structural vulnerabilities.

Fragmentation: 7+ competing protocols. Alongside MCP sit A2A (more than 100 supporting companies in February 2026, merged with IBM ACP under the Linux Foundation in September 2025), AGNTCY (more than 75 companies at the time of the Linux Foundation donation on 29 July 2025), agents.json (Wildcard), Zed AGP and several others. NIST announced its AI Agent Standards Initiative on 17 February 2026 in an attempt to coordinate. Neither W3C nor IETF has ratified an RFC: an IETF draft, “Framework for AI Agent Networks”, exists in 2026, and a W3C AI Agent Protocol Community Group meets every two weeks — but industry is running ahead of the normative standards.

The ecosystem’s bidirectionality. This is probably the most important pattern — and the least visible in popular discourse. The web is not moving unilaterally toward “exposure to agents”. Two opposing logics coexist.

On one side, Microsoft NLWeb (May 2025) installs a native conversational layer for sites — each NLWeb instance exposes an embedded MCP server and reads Schema.org. Adopters: Chicago Public Media, Allrecipes, Eventbrite, Hearst, O’Reilly, Tripadvisor, Shopify, Snowflake. The direction is “open the content to serve agents”.

On the other, Cloudflare is deploying a defensive, monetised strategy. Pay Per Crawl (1 July 2025, private beta) lets sites block AI crawlers or bill them per request. AI Labyrinth (March 2025) generates AI decoy pages to slow down crawlers that ignore robots.txt. AI Crawl Control (2025 rebrand) industrialises access governance. The figures convey the intensity: GPTBot traffic +305 % between May 2024 and May 2025 according to Cloudflare; 14 % of the 10,000 largest Cloudflare domains already have AI-bot-specific directives; GPTBot is both the most blocked (312 domains, 250 fully disallowing) and the most explicitly allowed (61).

llms.txt empirically a “dud”. Proposed by Jeremy Howard (fast.ai) on 3 September 2024, llms.txt was presented as the “robots.txt for LLMs” — a file listing priority content for agents. The directory.llmstxt.cloud registry lists roughly 1,800 adopting sites as of 6 May 2026, including Anthropic, Perplexity, Cloudflare, Vercel and Hugging Face. But OtterlyAI measured 0.1 % of real bot traffic (84 requests out of 62,100), three times less effective than a standard page. John Mueller (Google) stated in June 2025: “FWIW no AI system currently uses llms.txt”, and Gary Illyes confirmed in July 2025 that Google neither supported nor planned to support the format. The SEO community now compares it to the obsolete keywords meta tag of the 2000s.

In plain terms: the transition toward an agent-native web is real, but it is not a one-way march. MCP is winning, llms.txt is losing, Cloudflare is blocking, Microsoft NLWeb is opening. Sites pick their side according to their business model — an advertising publisher closes, a marketplace opens.

§5 — Predictable short term vs speculative long term

Across the four angles covered, confidence in the projections varies sharply with the horizon.

Short term (2026) — strong evidence. The directions are measurable today. The enterprise composable share holds its course (Gartner 70 % mandate in 2026, MACH 87 % adopters). WordPress’s slow decline will not accelerate abruptly — the ~5 points lost over four years is a background trajectory. Google captures AI search value through AI Overview and AI Mode, and publishers keep bleeding outbound traffic (-26 % Google → news in eleven months). AI agents progress on bounded use cases but remain fragile; latency, cost and security are structural locks, not one-off bugs. Standards converge partially (MCP de facto), and Cloudflare’s bidirectionality persists.

Medium term (2027-2028) — uncertain. Several trajectories are open. Partial standards convergence through NIST 2026 plus the IETF drafts is likely, but no W3C/IETF RFC is ratified to date — a natural two-to-three-year lag. The LLM search trajectory may plateau (growth has slowed since September 2024) or rebound (free product launches such as the Comet free tier). Mass agentic adoption versus the 40 % of projects cancelled predicted by Gartner remains undecided — it will depend largely on the simultaneous resolution of the latency, cost and security locks.

Long term (2029+) — speculative. At that horizon, several questions have no empirical answer:

• Will a W3C/IETF RFC unifying MCP/A2A/AGNTCY be ratified before 2029? Unknown.
• Does the search advertising business model hold, or do we tip into monetised agent-to-agent transactions? Unknown.
• Does the consumer estate (WordPress, Wix, Squarespace) switch natively to a composable stack, or do the two worlds stay sealed off? Unknown.

In plain terms: the level of proof changes with the horizon. On the direction of the trends, the 2024-2026 evidence is strong. On the exact amplitude it is moderate (half the figures are vendor self-reported). On projections beyond 2028 it is weak — and it is more honest to mark the speculative level than to simulate a certainty that does not exist.

Conclusion

Four words sum up the empirical verdict on the web ↔ AI agents transition: bimodal, bidirectional, complementary, fragmented. The transition is real, measurable and already under way — but it does not look like what popular narratives project. Not the death of Google: its market share, query volume and search revenue all rise across 2024-2026. Not the disappearance of WordPress: 42-43 % of the web holds firm, with a slow 5-point decline over four years. Not the universal reign of AI agents: 30 % at best on realistic workplace tasks, latencies in tens of minutes, structurally flawed security, visible costs. Not a universal standard: seven competing protocols, zero ratified RFC.

The one measurable underlying shift lies elsewhere — outbound-click disintermediation. Google is adapting its engine, capturing the value of commercial intent before the click, and the open web bleeds downstream. Publishers are absorbing a reduction in available clicks, not a share transfer between engines. It is a structural economic break — with no visual drama, no spectacular shutdown announcement, but with cumulative three-year effects that change both the content production model and the governance of web access (Cloudflare Pay Per Crawl, AI Labyrinth, AI-specific robots.txt directives).

The rest — mass adoption of agents as the primary medium, convergence on a universal standard, the SMB estate switching to composable — belongs to a two-to-three-year research programme, and beyond that to speculation. The useful method for tracking this transition is to observe empirically, to separate what is measured from what is extrapolated, and to mark the levels of proof. Rather than prophesying a massive rupture, read the quarterly figures — Alphabet earnings, SparkToro clickstream, Cloudflare traffic, OSWorld benchmarks — and see where the curves actually move.