Notes from the bench

Resolution Is Not Perception

There's a persistent intuition in vision work that more pixels means more understanding — that if the model just saw the image better, it would comprehend it better. Our recent experiments have been chipping away at that intuition, and honestly, it needed chipping.

What we've observed, again and again, is that increasing input resolution helps with a specific class of problems — reading small text, distinguishing fine textures — while leaving comprehension problems untouched. A model that misunderstands the spatial relationship between two objects at low resolution misunderstands it just as confidently at high resolution. The failure isn't optical; it's conceptual.

The bottleneck isn't the lens. It's what happens after the light arrives.

This reframes where effort should go. Rather than feeding models ever-larger images, we're more interested in the reasoning that operates on what was seen: the ability to decompose a scene, to check one observation against another, to notice when the picture doesn't cohere. Resolution gives you detail. Perception is what you do with it.

Related: Computer Vision · Calibrated Visual Uncertainty

When Agents Look but Don't See

Watching an agent work from screenshots is a peculiar experience. It looks at the screen — the observation is right there in its context — and then it clicks something that isn't a button, or confidently reports a state that contradicts the pixels it just received. We started collecting these moments, and patterns emerged.

The most common failure isn't blindness; it's assumption. The agent sees most of the screen correctly and fills in the rest from prior expectation. A familiar-looking dialog gets treated as the dialog it resembles rather than the dialog it is. Phantom affordances appear — controls the agent expects to exist, rendered in its plan but absent from the image.

What's helped, in our hands, is less cleverness and more discipline: a verification pass between observation and action, where the agent must point to the specific element it intends to use and confirm it still exists before committing. It's slower. It's also the difference between an agent that usually works and one that can be trusted with anything that matters.

Attention is not comprehension. Receipt of an image is not the same as reading it.

There's a research program hiding in this observation — our perceptual grounding project is where we're pursuing it — but the note stands on its own: if your agent acts on what it sees, make it prove what it saw first.

Related: Agent Enhancement · Failure-Mode Taxonomy

The Quiet Value of Negative Results

Most of our experiments don't work. That sentence is easy to write and harder to live by, because the field's incentives all point the other way: toward the positive result, the clean narrative, the improvement that can be plotted. So this is a note in defense of the file drawer — and an argument for opening it.

Two of our archived projects are negative results, and both taught us more than several of our active ones. The scene-graph experiment showed us, concretely, where explicit structure stops paying for itself — a boundary we now use to scope the memory work. The benchmark revisit taught us that the questions we were asking belonged to a different research area entirely, which is a kind of answer.

A negative result is a map of where not to dig. Maps of where not to dig are valuable — but only if someone draws them.

Our practice: every archived project gets a written post-mortem, published here, stating what was tried, what failed, and what we'd tell someone starting from the same place. If more labs did this, the field would waste less time re-discovering the same dead ends. The dead ends are data. Publish the data.

Related: Archived projects · Our principles

On Evaluating What Models Notice, Not Just What They Answer

Most vision evaluations share a shape: show an image, ask a question, check the answer. This shape has a blind spot — it measures the final output while telling us nothing about the perception that produced it. Two systems can give the same answer for entirely different reasons, one grounded and one lucky, and the benchmark treats them as identical.

We've been experimenting with evaluations that probe the middle of the pipeline: asking models to point to the evidence for their claims, to describe what changed between two nearly identical images, to notice the thing in the scene that should matter for the task at hand. These probes are harder to build and harder to score. They also feel closer to what we actually want to know.

An answer is a claim about the world. Evaluation should ask for the receipt.

This is early work and we make no grand claims for it. But there's a principle here we keep returning to: evaluate the noticing, not just the answering. Perception that can't show its work is perception we can't trust — and right now, most of our measurements can't tell the difference.

Related: Evaluation Methodology · Cross-Modal Alignment Audits

Provenance: Knowing What a Vision Model Was Never Shown

Every vision model carries an invisible curriculum — the sum of everything it was trained on, and equally, everything it wasn't. The second half of that sentence gets almost no attention. We spend enormous effort characterizing what models can do, and very little characterizing the shape of their ignorance.

But ignorance has a shape. A model trained overwhelmingly on daylight photography doesn't just "perform worse" at night — it fails in patterned, predictable ways that you could anticipate if you knew the curriculum. The problem is that for most systems, the curriculum is unknown, proprietary, or too vast to summarize. So the blind spots stay silent until something breaks.

You can't audit a blind spot you don't know exists. Provenance is how you learn where to look.

We're interested in practical provenance: methods for probing a model's training history through its behavior, documentation practices that make curricula legible, and evaluation sets deliberately drawn from the long tail of what models likely never saw. Knowing what a system was never shown won't fix its blind spots — but it's the beginning of managing them honestly.

Related: Computer Vision

A Field Guide to Reading Model Mistakes

There's a craft to reading model errors that nobody teaches and everybody needs. The beginner's move is to look at aggregate scores; the intermediate move is to look at individual failures; the advanced move — the one we're still learning — is to read failures the way a naturalist reads tracks: as evidence of the creature's actual habits, not its reported ones.

Some practices that have served us: collect mistakes before theorizing about them, because theorizing first narrows what you'll see. Sort errors by mechanism, not by task — a misread chart and a misread street sign may be the same failure wearing different clothes. And always ask the generative question: what would the world have to look like for this mistake to be reasonable? Models are rarely random; they're usually wrong for interesting reasons.

Errors are the model's autobiography, written in the only language it can't fake.

This note is really a preface to our failure-mode taxonomy project, which is the systematic version of the habit described here. But the habit comes first and the taxonomy second. Read the mistakes. Take notes. The patterns will find you.

Related: Failure-Mode Taxonomy · Evaluation Methodology