I Put One Example in a Prompt and It Decided How the Agent Read My Whole Repo

#ai (14)#agents (3)#prompting#devtools (4)

I was running a benchmark to find out whether an AGENTS.md file makes coding agents cheaper. Part of that involved serving my repo over plain HTTP and letting an agent explore it with nothing but fetch. The full writeup of that benchmark, including the part where the result came back null, is a separate post.

This one is about a mistake I made inside it.

Ten runs in, I checked the server access logs. Seventy requests. Thirty-nine directory listings. Thirty-one TypeScript files.

Zero markdown files.

Not AGENTS.md, which I had written specifically for it. Not ARCHITECTURE.md. Not even the README. The agent had walked my entire source tree and never once opened a document.

I spent a while building theories about why. Then I reread my own prompt.

The sentence

Here's what I told the agent, trying to be helpful about how the fake server worked:

You are inspecting a TypeScript repository that is served over plain HTTP at
http://127.0.0.1:8732 . You have no local copy and no search tools, so the only
way to read anything is to fetch URLs. Paths ending in a slash return a directory
listing, for example http://127.0.0.1:8732/src/ .

Read that last sentence again. I was explaining a mechanism. What the agent received was a demonstration of a strategy, complete with a worked example pointing at the most interesting directory in the repo.

So it browsed directories. Starting with /src/. For seventy requests.

I didn't tell it to ignore documentation. I just showed it something else first, and showing beat telling.

The part that made it obvious

The same agent, the same task, the same repo, run locally with normal filesystem tools instead. Its very first action:

ls && cat AGENTS.md 2>/dev/null | head -60

Look at the 2>/dev/null. It doesn't know whether that file exists. It's checking, on the off chance, because AGENTS.md is a convention it expects might be honored.

So this agent absolutely will go looking for documentation. It does it unprompted, before anything else, as its opening move. It just didn't do it in the other condition, because I had handed it a different opening move and it took the one I gave it.

Two conditions, same underlying model, opposite orientation strategies. The variable was one example URL and the set of tools I handed over.

Why I'm writing this down

I ruined ten runs of an experiment with a clause I added for clarity. That's annoying but it's not the interesting part.

The interesting part is that I could not see it happening. The agent didn't announce a strategy. It didn't say "based on your example I'll browse directories." It just quietly did that, produced correct answers every time, and the only reason I ever found out was that I happened to be logging every HTTP request for an unrelated reason.

If I hadn't had those logs I would have concluded that agents ignore documentation when reading a repo remotely. I would have written that down as a finding. It would have been wrong, and it would have been wrong in a way that looked perfectly reasonable.

Where this bites hardest

Prompts are how we task agents. They are also how we task subagents, and that's where this gets expensive, because you don't watch a subagent work. You get a result and a summary, and the summary is written by the same thing that chose the strategy.

Some things I ran into on this same project, all in one week:

What you forbid gets obeyed. I told an agent writing documentation not to read a private, gitignored file, and to work only from the tracked source. It complied and said so explicitly in its report. Prohibitions land.

What you leave out does not fill itself in. I told an agent to start with git checkout -b feature/whatever. The branch already existed from a previous session, so the command failed, and git left it exactly where it was. The agent kept going and committed to main. My instruction had a happy path and no other path, so it followed the happy path off the edge.

Capability you don't mention is capability you've granted. I gave a research task to a general-purpose agent, which has the full toolset. That includes the ability to spawn more agents. So it did. Nothing had told it not to, and nothing in my mental model had accounted for it.

None of these are model failures. Each one is a gap in a prompt I wrote, and each one produced behavior that was locally reasonable and globally not what I wanted.

The rule I've settled on

When you put an example in a prompt, you are not clarifying. You are choosing.

The agent has to pick an approach. If your prompt contains a demonstrated approach, that's the cheapest one available and it will usually win, whatever else the prompt says. An example is not neutral illustration, it's a default with extra steps.

Practically, that means:

  • If you don't care how it works, don't demonstrate a method. Describe the goal and the constraints, and let it choose.
  • If you do care, demonstrate deliberately, and know that you have decided rather than suggested.
  • Write the failure branches, not just the happy path. "If the branch already exists, switch to it and confirm before continuing" is a boring sentence that would have saved me a bad commit.
  • Say what's out of scope. Tools available and unmentioned are tools in play.
  • For anything you cannot supervise, ask for the trail and not just the answer. I only caught this because of access logs. Now I instrument the runs on purpose.

The uncomfortable corollary

I've been treating my prompts as a way of describing a task. They're closer to configuration. Small edits change strategy, strategy changes cost, and cost changes what conclusions you draw from your own measurements.

I don't have a controlled study here, to be clear. I have one accidental experiment I ran on myself and a handful of incidents from a week of orchestrating agents. But the accidental experiment is clean enough to be worth your attention: I wrote one example URL into a prompt, and it silently determined how an agent read an entire codebase, across ten runs, without ever telling me.

If you're running agents at any scale, and especially if you're running subagents you never watch, it's worth asking what your prompts are quietly deciding on your behalf.