Why Barkan reads the screen, not the docs
Documentation describes the product in general. The DOM describes it for this user, right now. A look inside how Barkan turns a live page into something a model can act on.
Gabriel Lancelot — — 7 min read
When we started building Barkan, the obvious architecture was the one everyone else had shipped: embed the product's documentation, retrieve the relevant chunks for a question, and let a model write an answer. We built that first. It worked well enough to be demoed and badly enough to be unshippable, and the failure mode was always the same — the answer was correct about the product and wrong about the user. This post is about the decision that followed: ground every answer in the live rendered interface instead, and what that actually takes.
The failure that changed the architecture
The question that broke the doc-trained version was mundane. A tester asked "how do I add a second seat?" and got a clean, six-step answer lifted straight from the help center. Step four said to click Add member. On the tester's screen, that button was greyed out with a tooltip explaining the Launch plan caps at one seat.
The answer was not hallucinated. It was true, in general, and useless in particular. The model had no idea the button was disabled, because nothing in the documentation could tell it what this account's screen looked like right now.
That is the core limitation of any assistant whose knowledge comes from prose about the product: it knows the product as designed, not the product as rendered for this user. And the gap between the two is exactly where users get stuck.
What "reading the screen" means
It does not mean screenshots, and it does not mean dumping the HTML into the prompt. Both are tempting and both fail — screenshots lose the structure a model needs to act, and raw HTML on a modern app is hundreds of kilobytes of framework noise with the useful signal buried in it.
Instead, when a user asks something, the widget captures an enriched snapshot of the rendered document and sends that, alongside the question, to the API. Roughly, it contains:
| Layer | What it carries | Why it matters |
|---|---|---|
| Interactive elements | Buttons, links, inputs, with a stable reference and an accessible label | So the model can point at and act on a specific control |
| Relationships | Which label belongs to which input, which button belongs to which form | Turns "the email field" into a concrete element |
| UI facts | Disabled states, selected tabs, badges, counts, validation errors | The "greyed out on Launch" information the docs never had |
| Content blocks | Visible headings and text, deduplicated and truncated | Enough context to understand the page, not the whole page |
| Form summaries | What is filled, what is empty, what is invalid | Lets the model resume a half-finished workflow |
| Active surfaces and scroll state | Open modals, drawers, the current viewport | Distinguishes "not on screen" from "does not exist" |
| Page meta | Route, title, allow-listed data attributes | Cheap, reliable orientation |
The whole thing is built to be small, stable and honest. Small so it fits in the context budget with room to think. Stable so the same element gets the same reference across turns, which is what makes pointing and multi-step actions possible. Honest so the model never sees a control that the user cannot.
Three engineering problems this creates
Grounding in the DOM solves the "wrong about the user" problem and immediately creates three others. They are all worth it, but they are real.
1. The interface moves
A documentation index changes when someone edits a doc. The DOM changes when anything happens: a dropdown opens, a toast appears, a list finishes loading. A snapshot taken a second early describes a page that no longer exists.
We handle this in two ways. The snapshot is captured after the page settles — we wait for in-flight network activity and layout to go quiet, with a cap so a noisy page cannot stall the answer forever. And in Do Mode, every action is followed by a silent re-capture before the next step is decided, so the model is always acting on the page as it is, not as it was.
2. Stable references on an unstable tree
Telling the model "click the third button" is fragile. Telling it "click the element with id b17" only works if b17 means the same thing on the next turn. Modern frameworks re-render aggressively, so we cannot lean on DOM identity.
Our references are derived from what a human would use to recognise the element — its role, its label, its position among its siblings, its enclosing landmark — and we keep a short-lived map from reference to live node. When the map goes stale, the action fails loudly and the model re-reads the page rather than clicking the wrong thing. A failed action you can see is far better than a successful action on the wrong element.
3. What not to send
An enriched snapshot of a real product contains real data: customer names in a table, an invoice total, an email in a form field. Sending all of it to a model by default is not acceptable, and "we need it for context" is not a good enough reason.
The snapshot is minimized on the client before it leaves the page. Visible text is truncated and deduplicated, input values are summarized as filled / empty / invalid rather than copied, and only an allow-list of data attributes is forwarded. The goal is for the model to know that there is a table of customers with 48 rows and a search box above it, not to know who the customers are.
Showing instead of telling
Once the model is grounded in the same interface the user is looking at, something becomes possible that no doc-trained assistant can do: it can stop describing and start pointing.
When the reply references an element, the widget moves a cursor to it on the actual page and waits. Across a multi-step workflow, that cursor walks the user from control to control — including across page navigations, because the snapshot is rebuilt on the new route. The instruction and the interface become the same object, and the translation step that makes documentation exhausting simply disappears.
Every assistant can tell you where the button is. The difference is whether it can see that you are already looking at the wrong page.
Where the docs still matter
None of this means documentation is useless to the model. It means it has a different job. The docs carry intent — what a feature is for, when to use it, what a setting means — and the screen carries state. A good answer often needs both: the knowledge base explains that webhooks retry three times, and the screen shows that this webhook's last delivery failed.
So Barkan does retrieve from the knowledge base, but it retrieves after it has read the screen, and the screen wins whenever the two disagree. If the docs say there is an Add member button and the screen says it is disabled, the answer is about the disabled button.
What we would tell another team building this
- Ground in the rendered interface, not the docs. "True in general" is the most expensive kind of wrong.
- Send structure, not pixels or raw HTML: interactive elements, relationships, UI facts, summarized content.
- Capture after the page settles and re-capture after every action; the DOM is a moving target.
- Minimize on the client. The model should know the shape of the data, not the data.
- Point, do not describe. Once you can see the element, you can show it.
The install is still one line
A reasonable worry is that "reads the rendered interface" implies deep integration. It does not. The widget is a single script tag in the layout you already render; it mounts its own root in a shadow DOM, observes the page from inside the browser, and needs no route annotations or component wrappers.
<script async src="https://trybarkan.com/widget.js" data-barkan-site="site_your_key"></script>Everything described above happens in that script. The product it is installed on does not have to know Barkan exists.
Frequently asked questions
Why not just train an assistant on the product documentation?
Documentation is general and goes stale. The rendered interface is specific to the user's account, plan and current state, and it is always current by definition, so grounding answers in it removes a whole class of confidently wrong replies.
What does Barkan capture from the page?
An enriched, privacy-minimized snapshot of the rendered interface: interactive elements with stable references, labels and relationships, visible content blocks, form summaries, active surfaces and scroll state — not a screenshot and not raw HTML.
Does reading the DOM work on single-page apps?
Yes. The snapshot is taken from the live document at the moment of the question and refreshed after each action, so client-side routing and dynamic rendering are handled the same way as a full page load.