"Delete Your Scaffolding" Is a Testable Claim

Boris Cherny built Claude Code. In July, after Opus 5 shipped, he described deleting more than 80% of its system prompt and watching the product get better without it. In a Y Combinator interview released a few weeks later, he went further: tear down your CLAUDE.md files, custom skills, and hooks every six months and rebuild from scratch, because most of those rules were compensating for judgment that older models lacked and newer ones have.

I run exactly the kind of thing he's describing. It's a skill called ponytail, by Dietrich Gebert, and it injects a "lazy senior developer" ruleset into every turn of every coding session: question whether the code needs to exist at all, reach for the standard library before custom code, prefer a native platform feature to a dependency, prefer one line to fifty. It exists to stop the model from over-building. If frontier models no longer over-build, ponytail is pure context tax, about 600 words added to every single turn.

So I had two options: argue with the guy who built the tool, or treat his claim as what it is, a testable prediction. Testing it took 66 model calls and an afternoon. An opinion about your AI tooling costs almost nothing to check anymore, and that fact should change how you hold these opinions.

The setup

Ponytail's repo ships its own benchmark: five everyday coding tasks (an email validator, a debounce handler, a CSV summer, a React countdown, a FastAPI rate limiter), run with and without the skill, scored on lines of code. The published numbers are from June 2026, run against Haiku, Sonnet, and Opus. There were no numbers for the Claude 5 generation. Fable 5 is the frontier model Cherny's thesis is about, and I'd been running ponytail on it untested.

I pointed the harness at claude -p with the model pinned to Fable 5, on a subscription plan, so the marginal cost of the whole experiment was zero. One methodological detail mattered more than I expected: ponytail was live in my own environment, so every benchmark call ran with clean-slate flags (no settings, no plugins, no MCP servers). Without that, the "baseline" arm would have had ponytail silently injected into it by my own session hook, and the whole comparison would have been garbage.

Two arms and three runs per task made thirty calls.

Round one: the effect is still there

Median code volume per task dropped 61%. Responses were 65% smaller and came back 30% faster, so the skill isn't burning its savings on extra deliberation. The null result that would have justified uninstalling didn't show up. Fable 5 without ponytail still pads a five-line ask with production notes and edge-case commentary, and writes two to four times the code.

The one inversion is the most interesting row. On the rate limiter, baseline reached for slowapi, the idiomatic dependency, and was done in 13 lines. Ponytail's ruleset forbids adding a dependency when the standard library can do the job, so it hand-rolled a sliding-window limiter and then, following another of its rules, appended a self-test. Both rules fired exactly as written. On tasks where a dependency is the lazy answer, the ruleset costs you lines instead of saving them.

Round two: is the short code worse?

Fewer lines proves nothing if the lines are wrong. Ponytail's repo also ships a correctness gate, so I ran all 30 outputs through it. Every failure it flagged turned out to be a false negative in the gate itself, and one of them was ponytail writing better code than baseline: its countdown computed remaining time from an end timestamp instead of decrementing a counter, which keeps working when browsers throttle background tabs. The gate's regex didn't recognize that as countdown logic.

Then I went looking for real damage. I ran both arms' email validators against 17 edge cases instead of the gate's five. Baseline scored 17/17 on all three runs. Ponytail scored 11/17 on all three: its one-line regex accepts leading dots, consecutive dots, hyphen-leading domain labels, and over-length addresses. That's a genuine gap. Two mitigations matter. The misses all run in the permissive direction, roughly the strictness browsers themselves apply to <input type="email">. And the shortcut was disclosed in the output itself, twice: a comment naming the upgrade path (email-validator) and a closing line listing what was skipped.

That left one hypothesis standing: ponytail is fine on toy prompts but cuts corners when correctness is on the line. So I built a second wave designed to punish laziness. Six tasks where shortcuts fail runnable tests: a filename sanitizer taking hostile input, bill-splitting in integer cents that must sum exactly, a duration-string parser, business-day date logic, an email validator with the strict rules spelled out in the prompt this time, and a TokenBucket class with five explicit requirements including thread safety. I wrote the graders before generating any outputs, so they couldn't bend toward either arm. Thirty-six more calls.

Baseline: 138/138. Ponytail: 137/138. The single miss was an input the prompt never mentioned (a domain label starting with a hyphen), in one run of three. Every requirement that was explicitly stated, the length limits, the dot rules, all five TokenBucket requirements, ponytail implemented in every run. The skill's own carve-out says "never simplify away anything explicitly requested," and under adversarial testing, that carve-out held.

My favorite detail from this wave: on bill splitting, ponytail wrote more lines than baseline, 6 versus 3. The extras were a two-line divmod solution plus three assertions proving it correct. Baseline shipped zero tests across all 33 of its runs. Ponytail shipped a runnable self-check with every non-trivial answer.

What the numbers settle

Across both waves: 155 distinct graded checks. Baseline passed all 155. Ponytail passed 148, and six of its seven misses came from a single underspecified prompt where it made a defensible, disclosed judgment call.

The shape of the result is the useful part. Ponytail's savings concentrate exactly where its risk doesn't. On vague prompts, where baseline over-builds freely, it cut code 61% with no functional damage. As specs tighten, the savings shrink toward 25%, then slightly negative on trivial tasks, because its mandatory self-tests add lines to already-minimal solutions, and its liberties disappear at the same time. It behaves like insurance against underspecification. Most real prompting is underspecified.

Cherny's July claims came out of this looking half right, which is better than most predictions manage. The teardown instinct is correct: I had no numbers for the current model and was running the skill on faith. The blanket prediction was wrong, for now, for this skill: Fable 5 still over-builds without it, measurably and repeatably. The part of his advice I've adopted isn't the deletion, it's the cadence. This scaffolding now carries an expiration date. The harness is saved next to a note in my agent's memory; when the next frontier model ships, the same 66 calls get re-run, and a null result means ponytail gets deleted without a eulogy.

One operational rule survives contact with all of this data. Ponytail's "Skipped:" lines, the ones listing what it chose not to build, are load-bearing. They were accurate in every output I inspected. If nobody reads them, you inherit the ceilings silently, and the email validator is what that looks like. The skill tells you where it was lazy; you still have to read it.

If you're running an agent with a pile of accumulated scaffolding, the move isn't to defend it or delete it on someone else's schedule. Find or write the benchmark that would prove your scaffolding useless, and run it against the newest model you use. Then put the result where your future self will trip over it. Mine took an afternoon, and the next one will take five minutes.