I Tried to Prove Agent-Readiness Saves Tokens. 52 Runs Later, I Can't.

I built a scanner that grades a GitHub repo on how legible it is to an AI coding agent. Twenty-four checks, six axes, one score out of 100, a badge you can drop in your README.

The scanner and the benchmark harness aren't public yet. Both are still held together by hardcoded paths and assumptions about my own machine, and I'd rather clean that up than ship it as-is. They'll go up as a public repo once they're presentable, and I'll link them here when they do.

Then it occurred to me that I had no idea whether the thing it measures is worth anything.

So I tested it. Fifty-two agent runs later I still can't tell you it saves money, and I'm publishing the whole thing anyway, including the part where I got excited about a number and had to walk it back.

The thing I was afraid of

There's a 2026 study that found machine-generated context files made agents worse on five of eight tasks while costing twenty percent more. Developer-written files helped, but only when they were short.

That's the opposite of what everyone selling agent-readiness wants to hear, including me. So I set the experiment up to be able to tell me I was wrong, wrote the pass and fail thresholds down before running anything, and decided in advance to publish whatever came back.

Good thing, too.

Setup

I used Charlotte, my token-efficient browser MCP server, as the test subject. Two checkouts of the same commit:

charlotte-bare    what a cold agent sees today
charlotte-ready   same commit, plus AGENTS.md, ARCHITECTURE.md, CODEOWNERS

Same repo, same commit, same code. The only difference is three added files. If I'd compared two different repos I'd have been measuring codebase complexity and naming quality instead of the artifacts, and I'd never have known.

Before running a single agent I scored both checkouts with my own scanner to confirm the change actually registered. Bare came back 66 out of 100. Ready came back 95. Without that check a null result would have been meaningless, because I couldn't have told "no effect" apart from "the treatment never applied."

Every run was Claude Code in headless mode, which reports cost, turns, and a token breakdown. Identical prompts. The agent never knew which side it was on. I alternated which condition went first on each replicate, which turned out to matter: the first run of every batch was consistently pricier than the rest, in both arms equally. That's session cache warming, and if I'd run all the control runs first it would have looked exactly like an effect.

What I ran

Five arms, fifty-two runs. Every task had one objectively correct answer that I verified by hand in the source first, graded by string match.

ARM                          RUNS   CONDITION     MEAN COST   FIRST-READ TOKENS
Easy local (find a symbol)     12   bare            $0.1369              7,579
                                    ready           $0.1328              8,108

Hard local (trace a path)      10   bare            $0.1679              9,387
                                    ready           $0.1447              5,444

Hard local, terse artifact     10   bare            $0.1483              4,620
                                    ready           $0.1642              8,200

Remote, no README pointer      10   bare            $0.1667              5,377
                                    ready           $0.1944              8,833

Remote, with README pointer    10   bare            $0.1588              5,115
                                    ready           $0.2072              8,919

"First-read tokens" is cache creation, the count of content being ingested for the first time. It's the expensive kind of input and the best proxy I had for how much of the repo the agent actually had to read.

The easy task: name the function that hashes element IDs, its file, its algorithm. The hard task, taken from real planned work on Charlotte: trace the path from a mutating tool call through the shared render helper to the state differ, and name the predicate that tells a same-page mutation from a navigation.

All fifty-two runs got the right answer. Both conditions. Every arm. Hold that thought.

Where I went wrong

Four mistakes, three of which produced results I believed at the time.

I spent twenty runs testing a paradigm nobody uses. I argued remote reads would show the biggest win, since every file you don't read is a network round trip you don't pay for. The logic was fine. The premise wasn't. Claude Code, Cursor, Aider, Codex, Devin and Copilot's agent all clone your repo to a local filesystem and work there with ripgrep. Fetch-only orientation is a niche. Twenty runs answering a question that doesn't matter.

I assumed a file nobody links to would still get found. When the remote agent never fetched AGENTS.md, I decided it was undiscoverable and added a prominent README pointer. Reran it. Nothing changed. Then I read the server logs: across seventy requests the agent pulled thirty-nine directory listings, thirty-one source files, and zero markdown files. Including the README. A pointer inside a file nobody opens is not a pointer.

My own prompt probably caused that result. My remote prompt helpfully explained that paths ending in a slash return a directory listing, and used /src/ as the example. So I told it to go browse directories, and then acted surprised when it browsed directories. Can't separate the finding from the nudge. That one needs a rerun with neutral wording before I'd believe it. I pulled that thread separately, because it turned out to be the more interesting failure of the two.

I reported a 42% improvement that wasn't there. After the hard local arm I had bare at 9,387 first-read tokens against ready at 5,444 and got excited. Then I pooled every control run I'd collected on that identical task against that identical code:

bare condition, first-read tokens, ten runs

  2,554  ####
  2,807  ####
  3,010  #####
  3,715  ######
  4,706  #######
  5,203  ########
  9,316  ###############
  9,926  ###############
 13,380  #####################
 15,417  ########################

range 2,554 to 15,417. Six-fold spread. Mean 7,003.

verbose ready mean  5,444  <-- sits inside the bare range
terse ready mean    8,200  <-- also sits inside the bare range

Both treatment means land inside the control's own spread. There's no separation at all. My 42% was one sample that drew high meeting one that drew low. I should have pooled before I opened my mouth.

Why the measurement failed

The variance has a cause, and it's more interesting than the result.

Cost is dominated by one decision the agent makes near the end of a run: does it open a big file whole, or grep and take a slice? The file at the middle of my hard task is 713 lines and 25 KB. Read it whole and the run costs several times what a targeted peek costs.

That choice flips between runs for reasons that have nothing to do with the condition. So cost was never really measuring readiness. It was measuring a coin flip with a big payout stapled to one side.

Three things that survived

The useful findings didn't come from the aggregates. They came from instrumenting individual runs and capturing every tool call.

Agents look for AGENTS.md by name, before anything else. The first action in the ready condition, every time:

ls && cat AGENTS.md 2>/dev/null | head -60

That error redirect is the tell. Nothing in the prompt promised the file was there, so the agent wrote the command to survive its absence and tried anyway. It is betting on a convention before it has any evidence the repo honors one. In the bare condition that same opening move went to git log, which told it nothing useful.

If you're wondering whether to invent your own manifest format or use AGENTS.md, that's your answer. Tooling already reaches for that filename unprompted. You cannot buy that.

Agents truncate. That command ends in head -60. My AGENTS.md was 142 lines. It read the first 60 and never saw the rest.

I checked what made the cut: title, description, build and test commands, and a source map that got sliced off partway through. The section explaining how a request actually flows through the system, which was the single most relevant part for the question being asked, was never read at all.

Your AGENTS.md has a fold. Content below it may as well not exist.

Sometimes a small map replaces a big read. In the instrumented pair, both conditions ran nearly identical searches and then split at the very last step:

bare   ... grep, grep, grep, then Read: src/core/tool-helpers.ts   (713 lines, 24,848 bytes)
ready  ... grep, grep, grep, then sed -n 1,20p on that same file   (20 lines) plus one grep

to reach the same correct answer:
  bare    24,848 bytes of source
  ready    3,067 bytes of map, plus a 20-line slice

That's a substitution, not an overhead. Roughly 3 KB of orientation standing in for 25 KB of rummaging. It explains why the easy task showed nothing, since that one never required opening a big file in the first place.

It's also one observation, and I now know the strategy varies run to run. So it's a hypothesis, not a mechanism.

What I changed

I rewrote the artifact around what the agent actually consumes instead of what reads nicely to a human.

                    LINES    BYTES   FITS IN THE 60-LINE WINDOW?
original AGENTS.md    142    7,970   no, cut off mid source-map
ARCHITECTURE.md       261   16,512   never opened locally at all
rewritten AGENTS.md    59    2,855   yes, entirely

Everything load-bearing now sits above the fold: what the project is, the exact build and test commands, one line per directory saying what lives there, how a request flows, the two or three gotchas that would otherwise eat an hour, and a short list of where to go deeper.

Ordering matters as much as length.

I tested it. Same null. Turns and output tokens went in the terse version's favor, cost and ingestion went the other way, and given that six-fold spread none of it means anything at this sample size.

So what do I actually believe

I can't tell you AGENTS.md will save you money. I tried to measure it and I couldn't.

What I'll say is narrower. These files aren't free and they aren't universally good. They're a cost that buys you less searching, so they only pay off when the searching they replace is expensive. On a simple symbol lookup in a well-named codebase, ripgrep already wins and your AGENTS.md is dead weight in the context window.

The behavioral stuff is more useful right now than the cost question. Agents probe for AGENTS.md by name. They truncate long files. They navigate by directory structure and filenames. That last one might be the real lever, and it's a slightly uncomfortable thought: Charlotte's naming is genuinely good, which is probably why the bare condition did so well and why there was so little room for a map to help.

Next round stops measuring money entirely. Cost was always a proxy for that one big-read decision, so I'll measure the decision directly: did the agent open a large file in full, yes or no. Binary, visible in the tool-call stream, and across twenty or thirty runs per cell it gives a proportion with a real confidence interval instead of a mean over a six-fold spread.

I also need a task hard enough that the bare condition sometimes fails. Fifty-two correct answers out of fifty-two means nothing I measured could possibly have moved success. I've been measuring what an agent spends, not whether it gets there.

One repo. One language. One model. Small samples. Nothing here is settled.

I'm posting it at this stage because a null with its mistakes labelled is more useful to anyone else poking at this than a confident number would have been. If you're about to write an AGENTS.md, write a short one, put the map at the top, and don't let anyone tell you the savings are proven. Including me.