I rewatched The Man from Earth recently, and what stayed with me this time was how remarkably ordinary the professor remained despite the rather extraordinary predicament of being immortal. Fourteen thousand years of existence, an almost preposterous accumulation of experience, and yet none of the superhuman intellect or infallible judgement one might expect from a man who had outlived civilisations. It wasn’t the impossibility of his lifespan that held my attention this time. It was the stubborn ordinariness of his mind. And, rather unexpectedly, it sent my thoughts towards an almost inverse predicament we now face while building software for AI agents.

Read more: Teach the Agent Why

In the movie, professor John Oldman, suggests that he has been alive for fourteen thousand years. Naturally, the company begins interrogating him. Imagine the knowledge! The people he would have met, the civilisations he would have watched rise and fall, the peculiar privilege of witnessing history before it ever became history. Surely a man with that much experience must have become something of a superhuman intellect?

But John is almost disappointingly modest about it. He isn’t a genius. He simply had more time. His memory is still human, with his understanding still shaped by what people of each age were capable of knowing. Fourteen thousand years of living did not turn his mind into an impeccably indexed library, nor furnish him with the judgement to make perfect sense of everything in it.

And on this rewatch I found myself thinking about the rather peculiar inverse of John Oldman’s predicament. We are now building software for an intelligence that can arrive with an extraordinary breadth of knowledge, without ever having lived with our software as we have. It may know every tool. It may even know what every command does. But will it recognise why somebody invented those commands in the first place, that too in context of the present predicaments?

Where knowledge ends and instinct begins

Think of an experienced developer encountering a problem. There is something almost inexplicable about the speed with which they occasionally recognise the shape of it. A dependency is holding up another team’s work, and they reach for a mock. A change feels innocuous, yet they insist on an independent contract. Somebody wants to patch the symptom, and they go looking for the assumption that made the symptom possible.

None of these decisions requires knowledge of an obscure command. Indeed, an inexperienced developer could be taught every command involved and still miss the moment at which it would have been useful to reach for one. What the experienced developer carries is a sort of accumulated intuition for the possibilities — not merely how a tool works, but the situations in which the world becomes a little less troublesome because that tool exists.

And for all our enthusiasm about AI agents, that distinction is easy to overlook. A capable model may already know more about a tool’s syntax than its creator. It can read the help text, inspect the repository, call an API and discover a missing flag without much ceremony. Yet this is not the same as having acquired the very particular sensibility through which an experienced user recognises when a capability matters.

The model may infer that for itself, of course. Often it does. But when it does not, giving it another hundred lines explaining the machinery is not necessarily going to help. What is absent may not be knowledge of the machinery at all.

It may be the imagination of what becomes possible because the machinery is there.

The philosophy hiding inside a product

I’ve begun to think that a good product is, among other things, is an opinion about how some part of the world ought to work. Its features are merely the visible evidence of that opinion.

In context of, say APIs, a mock server, fis not particularly interesting because it knows how to return an invented HTTP response. A capable programmer could arrange that in an afternoon. The more interesting promise is that one team needn’t sit idle while another finishes its work; that an unfinished dependency needn’t hold an entire piece of software hostage. It is a small capability, but it carries a rather large idea about how work ought to proceed.

And an independent API contract is not important because it is another file to keep updated. Its importance lies in allowing the expected behaviour to have a life of its own, outside the implementation. Without that independence, code and tests can agree with one another quite enthusiastically and still be wrong. Again, the feature is small compared to the philosophy behind it.

This is the sort of thing the good software teach us, often without ever announcing that they are teaching. Through defaults, affordances, the paths they make effortless and the ones they make awkward, we gradually learn the opinions of the people who built them. Eventually those opinions become part of our own way of working. We stop thinking about which feature we are using and begin recognising the occasions on which it belongs.

But an agent may not have enjoyed that long courtship with the software. It may meet the product for the first time through a description, a schema, a document retrieved halfway through a task, or a few lines of instructions deposited into its context. It can inherit the capability almost instantly; the sensibility that normally accompanies it is another matter entirely.

So perhaps the thing to communicate is not simply the feature, but the bet behind the feature. What frustration did its builders notice? What trade-off were they refusing to accept? Which possibilities, previously inconvenient or invisible, were they trying to bring within reach?

These are not decorative stories to put around a product. They are the reasons a capable intelligence might choose to use it.

What words can change

There is an intriguing consequence here. The material we produce for agents — descriptions, examples, skills, documentation, even the public writing around a product — does more than convey facts about software. It participates in deciding which facts become relevant to a particular task.

That deserves some care. I don’t mean that one can activate a literal region of a silicon brain with an especially stirring value proposition, nor that agents will invariably obey whatever somebody has written about a product. A model’s reasoning is far less obliging than that, and quite rightly so. But the information it encounters can make one possibility salient, draw attention to a trade-off, or put an otherwise distant consequence within the horizon of its next decision.

A list of capabilities answers the question What can I do? A well-articulated product philosophy helps answer a more consequential question: What might be worth doing that I have not yet thought to do?

This is also why I have become suspicious of exhaustive instructions. The more eagerly we prescribe a journey, the more we risk replacing the agent’s own judgement with our imagined version of the task. There is a place for strict procedures — authentication, permissions, irreversible operations and other narrow bridges over decidedly unpleasant ravines. But when many paths are possible, the more valuable contribution is often to explain the considerations that make one path sensible. Then leave some room for intelligence to do what intelligence is supposed to do.

Nor is this an invitation to write marketing prose for machines. Quite the opposite! A grandiose promise may make for a stirring landing page and an utterly dreadful premise for somebody about to act upon it. If we want agents to make better decisions, the value we describe must be real, the limitations intelligible, and the circumstances in which the product is not the right answer just as admissible.

I find one question particularly useful when looking at anything written for an agent: What would an AGI do differently because we told it something? If the answer is nothing, we may have written a perfectly good piece of documentation, but not necessarily something that improves a decision. If the answer is blindly follow an elaborate workflow regardless of the task, we may have achieved something rather worse.

The ambition is subtler than either. To leave the agent with a better sense of which possibilities deserve its attention, and why.

Teaching the reason, not the ritual

We have spent decades teaching humans to use software. We built interfaces to reveal capabilities, wrote manuals to explain them, and let experience do much of the remaining work. The best users eventually developed a feeling for the product — something difficult to pin down in a feature table, but unmistakable when you watched them work.

Our newest users arrive by an altogether different route. They may possess tremendous general knowledge and still lack the particular intuition that makes one tool valuable in the middle of an unfamiliar problem. We can tell them what commands to invoke, certainly. But doing so merely gives them another set of movements to imitate. It does not necessarily give them a reason to make those movements when the script runs out.

I keep coming back to John Oldman. He had fourteen thousand years to accumulate experiences, and still had to decide which parts of that extraordinary history meant something to the present. Knowledge, however abundant, does not arrive already arranged into relevance.

Knowing every instrument is not the same as recognising what it makes possible

It is almost comic that we now find ourselves on the other side of that thought experiment. An intelligence that knows so much, and a new responsibility for us — to communicate the possibilities we imagined when we built our software, so that it might recognise them in situations we never imagined ourselves.

Perhaps the most valuable thing we can tell an agent about a product is not what the product does.

It is what becomes worth doing because the product exists.

Two related thoughts preceded this one: The Terminal as an API explores why old programmable interfaces have found a new user, and Discoverable Is Not Callable asks what that user changes about the products we build.