Agentic browsing
Updated 2026-09-15
Agentic browsing is when an AI agent visits pages, clicks and reads on a user's behalf rather than answering from a search snippet. The agent, not the person, is the one reading your site — and what it can parse decides what the user is told.
What it changes
Three practical consequences. Access rules decide participation: an agent blocked at robots or by aggressive bot protection simply reports what it could reach elsewhere. Parseability decides accuracy: content behind interactions an agent cannot complete is invisible to it, the same way it is to a retriever. And measurement is immature: agent traffic is hard to attribute, appears in logs as ordinary requests or with self-declared user agents, and should be read as a directional signal rather than a metric.
Illustrative case (example)
An agent asked to compare two products fetches both pricing pages. One states prices in text; the other renders them in a widget after a click. The agent reports the first accurately and describes the second as "pricing not published" — an entirely avoidable loss.
Common mistakes
- Bot protection tuned so aggressively that legitimate assistant fetches are blocked along with scrapers.
- Key facts locked behind interactive components no agent can operate.
- Reporting agent traffic as a conversion channel before it can be attributed reliably.
Frequently asked questions
- Should sites block AI agents?
- It is a business decision with a measurable cost: blocking an agent that would have recommended you removes you from that recommendation. Decide per agent type, and revisit it as the agent list changes.
- How do we prepare for agentic traffic?
- The same way you prepare for retrieval: facts in text, no essential information trapped behind interactions, stable URLs, and clear entity statements.
