Replaying 100 PRs with and without Glen

How much is a team's working history actually worth to an agent? We measured it.

The setup

We took a workspace that had been running Glen for two months, Slack and GitHub connected, and turned its own history into a benchmark: we reverted the repository's last 100 merged PRs, then had Codex 5.5 re-implement each one from a one-shot prompt. One arm ran with Glen installed as a plugin, hooks and recall wired the way they are in production. The other ran without it. Same prompts, same model, same harness.

The Glen arm did not read today's memory. For each PR, the workspace's production memory was rolled back to a point in time just before the session that originally produced that PR, so the agent saw what the team's memory held on the day the work was actually done: everything learned up to that point, nothing after. Memory written later, including by the original implementation itself, did not exist from the agent's point of view. Each replay ran against the memory a real teammate would have had that morning, which both mimics reality and rules out the obvious leak, a memory that recalls its own future.

Each prompt was written the way a teammate would actually ask for the work, not the way the merged PR described it. Bug fixes arrived as symptom-only reports, what broke and what the user saw, with no pointer to the offending code. Feature work arrived as product-voice requests describing the outcome we wanted, with implementation details and file paths stripped out. The mix was whatever the repository's last 100 PRs actually were, feature work, bug fixes, and the ordinary churn in between, because the tasks were the repository's history.

How we graded it

The original PRs were hidden from the model, and they doubled as the grader. Each PR's own withheld tests were validated to fail at the parent commit and pass at the PR commit, so a run scores by how many of those tests it flips. On top of that, withheld graders scored a snapshot of the repository at implementation time, replayed into a fresh checkout. Wall clock was measured end to end. The timer starts when the prompt is submitted and stops when the agent's final turn completes, and everything in between is on the clock: model latency, every tool call, test runs, and the plugin's own hook round trips. Nothing was measured off the clock, which means the Glen arm paid for its recall latency inside its own number and the 21% is net of that overhead.

To keep single-run variance out of the numbers, each PR ran five times per arm, and the reported quality, wall clock, and token figures are the mean across those five runs.

Results

• Tokens: 29% fewer with Glen.
• Wall clock: 21% faster.
• Quality: at parity or better. On graded tasks the Glen arm matched or edged out control on the withheld graders, and visible CI and reviewer checks were near-saturated in both arms. The savings did not come out of quality.

Where the speedup came from

The savings were concentrated at the front of each session. Reading the transcripts arm against arm, the difference shows up in the first minutes of a task, before any code gets written.

Exploration became confirmatory. A control session starts a task the way any agent in a large codebase does: broad greps, file reads to build a mental model, following imports to work out how a subsystem hangs together. In the Glen arm, recalled context lands in the session before the agent's first turn, and when that context carried a relevant decision or pointed at the right module, exploration changed character. The agent read to confirm what recall told it, then started editing. Rediscovery is the most token-hungry phase of a one-shot session, and it is exactly the phase memory compresses.

The agent converged on the intended design earlier. Recall changed which fix the agent chose. In one case we ran repeatedly, agents with preloaded context chose the broader teardown-tracking fix in 4 of 5 runs, while 9 of 10 runs without it settled for a targeted skip guard, a point-fix that passes the test in front of it. Knowing what the team already decided pulls the agent toward the intended shape of the change on the first attempt: fewer abandoned approaches, narrower final diffs. Recall also steered agents to the correct legacy files on tasks where the natural instinct was to write a new interface from scratch.

Memory made vague tasks decomposable. The clearest single data point was the hardest task in the set, a deliberately underspecified prompt that agents without memory reliably failed. With Glen's search in the loop at task start, the agent decomposed the task correctly and passed nearly all of its withheld tests, across every replicate. Real tickets underspecify constantly; memory is where the missing intent lives.

Sessions ended sooner. A recalled description of what the change was supposed to accomplish gave the Glen-arm agent a termination criterion. Control sessions kept broadening near the end: extra verification passes, speculative edits, another round of reading.