Browser Agent: When You Need One, How to Think About Reliability, Why It's Dangerous

Task: every Monday, export a report from three SaaS systems and combine them into one spreadsheet. Path 1 — API: one system has one, the second only on Enterprise, the third none. Path 2 — scraping: works until the layout changes, then breaks silently. Path 3 — a browser agent that literally opens the site, clicks buttons, and downloads the file. Where does the agent win? When there's no API, when the UI is stable for humans but unstable in markup, when steps depend on context ('notification popped up — close it'). Where does it lose? When speed matters (10–100× slower than API), when thousands of repeats per hour are needed, when debugging each step costs more than writing an integration.

Browser agents fall into two schools. First — vision-based: the agent takes a screenshot, the vision LLM looks at the image, decides 'click the yellow button on the right'. Second — DOM-based: the agent extracts page structure, the LLM decides 'click the element with id=submit', the framework dispatches the handler. Vision works where DOM doesn't help: canvas, complex SVG, elements drawn over images. But it's more expensive, slower, and hallucinates coordinates on fresh layouts. DOM is faster and more precise, but goes blind on visual elements without semantics. Modern frameworks (browser-use, Stagehand) combine both: DOM as the base layer, vision as a safety net for the hard spots.

The agent clicked 'Submit'. Task done? Not necessarily. Maybe a validation error appeared. Maybe a spinner loaded and the real result comes in 5 seconds. Maybe the page redirected, and what you see is already a different page. The main source of bugs in browser agents isn't the clicks — it's understanding what actually happened after the click. Naive approach — 'wait 2 seconds after the click'. Breaks on the first slow response. Slightly better — wait for a specific element. But what element? If after Submit you expect 'Thank you for your order' but it's not there — is that an error or just not rendered yet? The right pattern is explicit success criteria per step. 'After Submit, expect element X (success) or element Y (error). Wait up to 10 seconds. If neither — unknown state, escalate.' Without this, errors accumulate at each step, and three steps in the agent is clicking blind. Golden rule: every agent step must have a verification predicate, not a fixed delay.

When a browser agent breaks, the first instinct is to fix the code: add a timeout, a handler, a try/catch. Almost always the wrong layer. Most failures are either an instruction problem (the agent misunderstood the task) or an environment problem (popup, A/B test, network hiccup). Fixed on top, not in the code. Instructions: spell out the obstacles. 'If you see a cookie popup — close it. reCAPTCHA — stop and call a human. A/B variant — find the button by text, not selector.' Two-minute edits that save hours of debugging. Determinism: a preparation phase before start. Clear cookies, set the locale, inject the session token. Fewer surprises, less mid-task breakage.

The last topic — and the most important. A browser agent has access to the internet, your cookies, your tabs, your clipboard. One mistake — and the agent logs into someone else's account, publishes a post instead of saving a draft, sends money to the wrong place. Base rule: no prod access by default. Isolated session — fresh browser profile, sandbox, proxy, test account. Prod credentials only once you've confirmed the agent does what it should. Second: explicit boundaries. Domain whitelist, human approval on payments/deletes/messages, step logs with timestamp and screenshot. Never give the agent full access to a wallet or primary account — tests cover the known, prod brings the unknown.