AI agents can browse websites, fill forms, and perform tasks. But the real challenge isn’t getting an agent to click a button. It’s making that interaction reliable, secure, and authorized.
Imagine asking an AI agent to book a train ticket.
It opens the website, searches for your destination, selects a date, and prepares to book.
Then it hits a CAPTCHA.
Or a login wall.
Or a page that changes its layout halfway through the task.
Or a security system that mistakes its activity for a bot attack.
Suddenly, your supposedly autonomous assistant needs you to take over.
This isn’t just a problem with one AI model. It’s a collision between two systems built around very different assumptions: an internet designed primarily for human interaction and AI agents designed to interact with it on our behalf.
And that collision is becoming increasingly important.
1. Websites aren’t obligated to welcome AI agents
The promise of AI agents is compelling.
Instead of manually comparing products, filling out forms, or navigating multiple websites, you describe the outcome you want and let an agent handle the steps.
But websites have their own incentives and responsibilities.
They need to protect user accounts, prevent fraud, control automated traffic, respect licensing agreements, and maintain their infrastructure.
Those requirements don’t disappear because the software making a request uses an LLM.
In October 2026, The Verge reported that major services were restricting AI agents for reasons ranging from licensing to security and website policies. Financial services have also raised concerns about agents accessing customer accounts.
Read the report from The Verge.
From a website owner’s perspective, an unfamiliar automated session might be a legitimate assistant — or an attempt to scrape data, abuse an account, or make unauthorized transactions.
The website often cannot safely assume the former.
The important distinction is that an agent’s ability to access a website does not automatically mean it has permission to perform every action on that website.
That is a product-design problem, a security problem, and an infrastructure problem.
2. Making an LLM smarter won’t solve every browser failure
A common approach to improving browser agents is to improve the model, provide better prompts, or give it more tools.
These techniques can help. But they don’t address every failure mode.
Consider four different problems:
Problem 1: The interface changes.
A button moves, a modal appears, or a website changes its layout. An agent relying on an outdated assumption may click the wrong element or fail to find it.
Problem 2: The website rejects the session.
A security challenge, rate limit, or authentication requirement can prevent the agent from proceeding. Better reasoning cannot grant permission that the website has withheld.
Problem 3: The page contains malicious instructions.
A webpage might contain text designed to manipulate an AI agent into ignoring its original task or disclosing information. Browser agents must treat website content as untrusted input, not as an authority that can override their instructions.
Google’s Chrome security team has discussed indirect prompt injection and the need for layered defenses in agentic browsing. Read Google’s security overview.
Problem 4: The agent makes a mistake but doesn’t recognize it.
It might submit an incorrect form, select the wrong item, or believe a task is complete when the final state says otherwise.
These are different problems. They need different solutions.
A single retry mechanism or a more powerful model won’t reliably fix all four.
3. The missing layer: reliable agent execution
A browser agent shouldn’t be judged solely by whether it can produce a sequence of clicks.
It should be judged by whether it reaches the intended outcome, respects permissions, and correctly identifies when it cannot proceed.
That requires an execution system around the model.
I would break this system into five components.
A. Observation
The agent needs an accurate view of the current page.
That might include a screenshot, accessibility information, DOM structure, visible text, or a combination of these signals.
It also needs to distinguish what it believes is happening from what it has actually observed.
B. Action
Every browser action should be explicit and traceable.
The system should know which element was targeted, what action was attempted, and whether the browser accepted it.
For consequential actions, the system should enforce appropriate approval requirements rather than trusting the model to decide everything.
C. Verification
After an action, the agent should check the resulting state.
If it clicks a button to submit a form, it should verify the submission rather than assuming success.
This is a fundamental difference between an agent that merely executes instructions and one that can demonstrate that the intended result occurred.
D. Recovery
When something fails, the agent needs to identify the failure category.
A missing element might justify re-observation. A temporary network error might justify a limited retry. An authorization failure should trigger a stop or an appropriate request for human intervention.
A CAPTCHA or explicit access restriction should not automatically trigger an attempt to circumvent the site’s controls.
E. Observability
Developers need to reconstruct what happened during a run.
That means recording relevant actions, observations, timestamps, errors, state transitions, and model decisions, with sensitive information redacted.
Without this information, debugging an agent can become an exercise in guessing.
Cloudflare’s Browser Run illustrates the growing infrastructure around agentic browsing, including browser sessions, recording, replay, and human intervention.
The broader lesson is that a reliable agent needs more than a model and a browser. It needs a system that can observe, verify, recover, and explain.
4. We need better ways for websites and agents to communicate
Today, developers often have to choose between different interaction methods.
They might use a documented API, interact with a website through browser automation, or use an agent-specific interface when one is available.
Each has trade-offs.
APIs can provide structured and predictable interactions, but they might not expose every workflow a user can perform.
Browser automation can interact with existing interfaces, but it is vulnerable to layout changes, session restrictions, and ambiguous page states.
Agent-specific interfaces could expose structured actions with clearer permissions and more predictable outputs. But they also require websites and agent platforms to agree on how those actions should be represented and secured.
This is why emerging browser-agent interfaces are worth watching.
For example, WebMCP documentation from Chrome discusses security considerations for structured tools exposed to agents through the browser.
I don’t think the answer is to make every website accessible to every agent.
Website owners should retain control over access, data, and consequential actions.
Instead, we need clearer ways to express:
- Which agents are making requests.
- What a particular agent is authorized to do.
- Which actions require explicit user approval.
- How websites can safely expose supported workflows.
- How agents should respond when permission is denied.
The goal should be predictable, authorized interaction — not unrestricted automation.
5. How should we measure a browser agent?
A successful demo is a weak measure of reliability.
A task that works once might fail on the next attempt because the page changed, a request timed out, or the model made a different decision.
I would evaluate browser agents using several metrics:
Task success rate: How often does the agent actually complete the intended task?
Recovery rate: When a recoverable error occurs, how often does the agent recover successfully?
False success rate: How often does the agent report success when the intended outcome was not achieved?
Human intervention rate: How often must a person take over?
Latency and cost per successful task: How much time and money does it take to produce a verified outcome?
Safety violations: Does the agent attempt actions outside its authorization or mishandle untrusted webpage instructions?
These metrics should be measured across repeated runs, varied page states, and realistic failure conditions.
A useful experiment would compare a basic browser agent with one that includes state verification, failure classification, bounded retries, and explicit human approval for consequential actions.
The experiment should report its task set, number of runs, failure categories, and limitations. Otherwise, the results could be misleading.
I haven’t presented benchmark results here because that comparison would need to be run and measured.
6. The future is not necessarily a world full of unrestricted agents
It’s tempting to imagine a future where every AI agent can access every website and complete every task without interruption.
But that isn’t necessarily a desirable outcome.
Websites need security. Users need control. Businesses need to manage costs and protect data. Agents need predictable interfaces and clear authorization.
The more useful future is one where legitimate agents can interact with participating websites through well-defined, secure mechanisms — and where unsupported actions fail clearly instead of unpredictably.
That also changes how we should build agent products.
Instead of asking only, “Can my agent navigate this website?”, we should ask:
- Can it explain why a task failed?
- Can it verify that its actions succeeded?
- Can it recover without repeating dangerous actions?
- Can it respect website restrictions?
- Can a developer reproduce and debug the failure?
- Can the user understand and control what the agent is doing?
These questions are less flashy than a demo of an AI shopping for you.
But they are much closer to what it takes to build software people can trust.
Conclusion
The next phase of browser agents will not be won by models alone.
It will depend on execution reliability, verification, security, observability, and better interfaces between websites and autonomous software.
The internet doesn’t simply need AI agents that can click faster.
It needs ways for agents, websites, and users to interact without sacrificing reliability or control.
The real milestone isn’t an AI agent that can use the internet once. It’s an agent that knows what it is allowed to do, can verify what it has done, and knows when it must stop.
That’s the engineering problem worth solving.
What do you think is the biggest bottleneck for browser agents today: model intelligence, website restrictions, or execution reliability?
Further reading