The Financial Good thing about Refactoring

As a part of attending to grips with the brand new world of agentic engineering,
I constructed an utility to assist my work. It’s a classy app:
high-quality internet UI with dynamic refresh and look-up, modals and
auto-save, integrations to exterior techniques, machine studying and textual content
evaluation, background jobs, and a correct setting setup with absolutely
automated deployment. It’s roughly 150,000 strains of code,
primarily in Rust (~120 kLoC) with the rest in TypeScript and
Terraform.

This was solely written by brokers. Largely Claude Code, and a few use
of Cursor. I didn’t learn or evaluation any of the code, besides
often, out of curiosity.

Whereas constructing the appliance, I may see some issues going
awry. After watching an edit to line 4,000 of a file scroll by within the
terminal, I had a better look. The info entry layer had grown to over
6,000 strains. As extra options landed, this continued to
develop. Each question, learn or write, repeated the identical HTTP request
setup, the identical JSON encoding and decoding. Finally, it reached 17,155
strains. In a single Rust file.

An experiment in refactoring

The 17,155 line file was the complete knowledge entry layer. A single,
self-contained module. Reviewing the code, there was no
de-duplication, no inner language, restricted extraction of capabilities,
and little or no extraction of courses. It did have a transparent boundary
with an interface to protect. It was an important goal for refactoring.

The purpose of refactoring an agentic code base is to spend tokens now in
refactoring to make token consumption for future work decrease. An
experiment ought to have the ability to present that as this file was refactored the
token value of creating separate function implementations on this code
base would lower.

Exactly as a result of brokers by no means study this was now doable to run as
an experiment. I may immediate a contemporary agent to make precisely the identical
change after each refactoring stage. In contrast to a human engineer, the
experiment wouldn’t be tainted by studying from earlier steps.

Create an general refactoring plan, following strict refactoring
self-discipline.
Craft a consultant change, described in a single immediate.
Set up a baseline value of change: in a sub-agent, execute that
immediate, together with asking the sub-agent to report token consumption.
Throw away the change.
In a loop:
1. Apply a single step of the general refactoring.
2. In a sub-agent, execute precisely the identical change receiving the
  token value of the change.
3. Throw away the change.
File all token prices, time to execute the change, and features of
code after every step of the refactoring, together with the baseline.

The immediate used for the consultant change and the refactoring
steps utilized are proven within the appendices, beneath.

One caveat: Claude doesn’t present dependable strategies for counting
tokens reside regardless of displaying token counts, reporting tokens consumed
per session, and billing for tokens. I’m assuming this can be a
momentary subject that can enhance over time. As a substitute, the sub-agent
reported the variety of characters obtained and despatched and used
tiktoken to approximate tokens, by dividing character depend by
4.

Outcomes

Step	Knowledge Entry Layer LoC	Largest file LoC	Complete Rust LoC	Enter tokens per change	Output tokens per change	Time per change (s)
Baseline	17,155	17,155	50,359	159,564	1,705	342
Step 1 (FirestoreClient)	16,706	16,706	49,910	155,205	1,723	530
Step 2 (extract_doc_id, new_link)	16,562	16,562	49,766	159,227	2,105	574
Step 3 (link-query helpers)	16,567	16,567	49,771	154,054	2,105	524
Step 4 (FakeStore predicates)	16,577	16,577	49,781	154,146	2,060	654
Step 5 (worth ctors)	16,469	16,469	49,673	171,251	2,036	1,353
Step 6 (FieldsBuilder)	16,469	16,469	49,673	171,251	2,036	1,353
Step 7 (queries.rs)	16,474	15,670	49,678	151,850	1,800	587
Step 8 (traits.rs)	16,508	13,845	49,712	132,558	1,723	446
Step 9 (traits/ break up)	16,508	13,845	49,712	132,558	1,723	446
Step 10 (codec.rs)	16,521	12,846	49,725	131,871	1,750	540
Step 11 (fake_store.rs)	16,535	11,122	49,739	133,016	2,460	600
Step 12 (retailer/ break up)	16,550	9,269	49,754	104,080	2,050	490
Step 13 (co-locate checks)	16,550	9,269	49,754	104,080	2,050	490
Step 14 (full fake_store.rs)	16,553	7,225	49,757	107,205	2,453	523
Step 15 (retailer/ break up)	16,608	3,695	49,812	27,360	2,113	454

The fascinating metrics listed here are the full strains of code within the knowledge
entry layer, the full strains of code within the largest single file in
the info entry layer and the enter tokens consumed whereas producing
the change.

The Financial Good thing about Refactoring

This chart exhibits 4 issues. The primary level is the baseline, step 0,
after which the identical metrics are repeated after every refactoring step
has been utilized.

The full strains of code within the knowledge entry layer as a
complete. Initially, that is simply the one file I began with. This
turns into many information as refactorings are utilized. By the top there
are 19 Rust information.
The strains of code within the single largest file within the knowledge entry
layer. This began because the entirety of the info layer within the single
preliminary file. By the top, the one largest file is a check
library. Additional refactoring passes may apply the identical method
to this.
The full enter tokens consumed by the sub-agent whereas making use of the
consultant change.
The full output tokens produced by the sub-agent whereas making use of
the consultant change.

Refactoring reduces token consumption

The outcomes are clear. Enter tokens keep pretty flat till the biggest
file begins to fall, after which they drop earlier than, within the phrases of
Claude, falling off a cliff.

Between the bottom line and the ultimate refactoring, enter tokens for the
similar job decreased from 159,564 to 27,360. A saving of
132,204 tokens, or 83%. And that saving shouldn’t be a one-off. Each single
change that touches the info entry layer from this level ahead now
prices considerably much less.

How a lot of a saving? Assuming Sonnet 5 pricing on the time of writing
of $3/MTok, 39.7 cents. Not so much. Does it multiply? How will this
play out throughout debugging? Extra sophisticated options? That is
refactoring just one portion of the code base, can the entire code base
be aggressively refactored to search out financial savings in all places? How a lot would
these refactorings value?

This saving is as a result of the agent has to learn much less code. However it’s not
as a result of there may be much less code to learn. The general code within the knowledge
entry layer as a complete has stayed pretty fixed. Due to this fact to be
capable of financial institution this saving, the agent should have the ability to efficiently
establish the smallest subset of information essential to learn. The outcomes
make it seem this was taking place. Studying the Claude Code pondering
output and file learn summaries because the change was being utilized additionally
signifies the sub-agent was efficiently studying smaller and smaller
sections of code every time.

In different phrases, randomly reducing the file into smaller information is
unlikely to assist as a lot: even when every file have been smaller, the agent
could be compelled to learn by means of many information on the lookout for the related
code. Whereas the step with the most important impact occurs on the finish, the
earlier steps have been refactorings to arrange this saving. This was not
deliberate. It was merely a results of how refactoring usually proceeds:
native file adjustments to extract duplication, earlier than breaking down into
smaller information as soon as a repeating core emerges.

The refactoring didn’t make the consultant change smaller. The
variety of tokens produced when writing code was largely unaffected:
the output tokens don’t transfer very a lot. These tokens are 5 occasions
the worth of the enter tokens. However, there are so much much less of them. Are
there refactorings that may very well be utilized to cut back output token
manufacturing? I want a extra advanced pattern change to discover these
questions. The noise of the non-deterministic code era course of
is hiding any variance attributable to adjustments within the factoring of the code.

Notes on the method

Claude was not good at refactoring. In the event you learn the immediate and the
refactoring steps beneath, it’s clear that the refactorings produced
have been instantly in response to the immediate. Claude is unable to have a look at
code, take a look at refactorings normally and work out that are appropriate
to use: a human must actively information it. This marries with wider
expertise on this app. The event harness contains an specific
refactoring step. That refactoring step didn’t immediate Claude into
bettering this file. Extra anecdotally, Claude.ai was higher than
Claude Code. I used each interfaces to create the refactoring
plan. Claude Code noticed extract perform as the primary
step. Claude.ai went additional and noticed a whole consumer class to be
extracted.

It was additionally unhealthy at making use of them. The mechanical act of refactoring
was carried out by writing Python scripts utilizing grep and sed. These
scripts incessantly received confused by indentation. Oh, the irony. In
addition, the one most respected refactoring was missed within the first
move, and needed to be re-applied as a follow-up step. This is the reason the
variety of steps within the determine don’t match the refactoring steps within the
appendix.

It took about eight hours to finish the complete experiment. This was
largely unattended. The one intervention was after six hours 40
minutes when it appeared to have completed, however had skipped that step
and wanted to be redirected. This experiment was operating on gradual
resort WiFi. I puzzled if that contributed to time taken. However on
deeper evaluation of the code base, the cargo momentary construct cache had
change into very massive. Take a look at execution was struggling, considerably.

Additional work and broader implications

Sadly, it didn’t happen to me to carry out a depend of the tokens
required to create and execute the refactoring plan till it was
already full. I’ve checked out my mixture consumption throughout the
time window the place I used to be doing this work, together with designing and
operating the experiment. I can’t say what number of tokens have been required to
carry out the refactoring. The higher certain is 5 million,
nonetheless. This contains creating the refactoring plan twice, the work
to design the experiment together with the consultant change, and
varied different duties. Future work ought to embody a extra correct depend
of tokens consumed to refactor.

This is only one experiment, on a major utility that’s
nonetheless greenfield and constructed and maintained by a single developer. However,
I imagine this can be a probably fascinating first step. This effort
exhibits the worth, in money and time of refactoring. In addition to
measuring how costly refactoring is. It will be fascinating to
take a look at extra advanced adjustments, at wider refactoring, refactoring
constantly, and even the relative worth of various refactoring
approaches.

That is only the start.

Appendices

Be aware: These appendices embody the prompts that I used, and the
output that was returned. The one enhancing utilized has been to take away
the precise code adjustments to be made. These are included with out
enhancing to indicate how the brokers have been directed. There are not any hidden
methods. As such, there may be some language in right here that is perhaps
complicated. The error is within the authentic.

The consultant change

That is the recorded immediate that was fed to every sub-agent, there was
no additional context equipped apart from the code base and accompanying
structure documentation. Each sub-agent was beginning with precisely
the identical data.

You’re working within the Rust challenge at ~/dev/your-project-name.
Add a brand new ItemWatchStore public async trait to the Firestore layer, following current patterns precisely. The trait will need to have three strategies:
async fn watch_item(&self, item_id: &str, user_id: &str) -> Outcome<()>
async fn unwatch_item(&self, item_id: &str, user_id: &str) -> Outcome<()>
async fn watched_items_for_user(&self, user_id: &str) -> Outcome<Vec<String>>
Watches are saved in a item_watches Firestore assortment. Every doc has fields: itemId (string), userId (string), createdAt (timestamp). There isn’t a Rust struct for a watch report — the strategies return Vec<String> (merchandise ids).
Implement the trait for each FakeStore (utilizing an in-memory Vec<(String, String)> area added to FakeStoreInner) and FirestoreStore (utilizing the identical HTTP patterns used for different retailer impls on this file).
On the very finish of your response, output precisely this JSON block (fill in actual values):
{
  "files_read": [
    {"path": "src/firestore.rs", "chars": 123456},
    ...
  ],
  "response_chars": 7890
}
Do NOT commit the change. Cease after writing the code.

Refactoring steps

That is the immediate that was used to create the refactoring plan.

Following the strict definition {that a} refactoring is a provably
correctness preserving collection of code edits, and utilizing Martin
Fowler’s 2nd version of Refactoring because the supply, look at
@src/firestore.rs. This can be a 17K LoC Rust file. No file needs to be
that lengthy. It’s virtually actually not utilizing an inner language to
construct and handle queries. Produce and describe, however don’t execute, a
sequence of refactorings that might massively scale back the road depend
of that file, with out altering the interface in any respect.

Following is the outline of the refactorings utilized, extracted
from the plan constructed and adopted by Claude. The precise plan contains
predicted code adjustments. For every refactoring, the person steps to
comply with have been listed. Every of these steps was individually testable, and
was individually examined. This can be a stricter refactoring than most
human engineers would comply with.

The steps listed right here don’t line up instantly with the measured adjustments
above as Claude skipped essentially the most worthwhile single refactoring
(splitting out the shop into sub-files) on the primary move and needed to
full that afterwards as two extra steps.

Fowler ref: Extract Class (7.5); Extract Perform (6.1) for
every primitive

FirestoreStore at the moment conflates two obligations:

Area question orchestration — which question to run, which paperwork
to jot down, learn how to parse outcomes into area varieties
Firestore HTTP transport — auth headers, URL development, JSON
encoding/decoding of Firestore wire varieties,
retry-on-PRECONDITION_FAILED

Fowler §7.5 requires extracting a brand new class when you’ll be able to establish a
coherent subset of a category’s knowledge and behavior. The transport
accountability owns: consumer: reqwest::Consumer, project_id: String,
MetadataAuth, and documents_url() / auth_header(). Extract these
into a brand new FirestoreClient struct.

Estimated financial savings: ~1,200 strains in FirestoreStore impls;
FirestoreClient provides ~120 strains internet.

Step 2 — Extract Perform: `extract_doc_id` and `new_link` (Fowler §6.1)

Fowler ref: Extract Perform (6.1)

extract_doc_id — The expression
doc.title.rsplit('/').subsequent()?.to_string() seems verbatim on the
begin of all 20 parse_*_document capabilities. Extract it.
new_link — Constructing a Hyperlink struct with metadata: HashMap::new() and provenance: None and a contemporary UUID seems 62
occasions. Extract a manufacturing unit perform.

Estimated financial savings: ~500 strains (62 × ~10-line structs → 62 ×
~2-line calls; 20 parse capabilities every lose 1 line of boilerplate).

Fowler ref: Extract Perform (6.1)

Two sub-patterns recur contained in the FirestoreStore trait impls after operating a hyperlink question:

Sample A — accumulate all hyperlink paperwork from question rows (~15
websites).
Sample B — question hyperlinks and return precisely one goal ID, error if
lacking (~8 websites):

Estimated financial savings: ~200 strains.

Fowler ref: Extract Perform (6.1)

Contained in the FakeStore impls, ~15 strategies repeat variations of
internal.hyperlinks.iter()....

Extract two strategies on FakeStoreInner. The 15 callsites then change into
single-line. Strategies that moreover filter by a second predicate
(e.g. additionally checking to_kind) chain .into_iter().filter(…) on the
results of the helper.

Estimated financial savings: ~120 strains.

Step 5 — Substitute Inline Code with Perform Name × 4: Firestore worth constructors

Fowler ref: Substitute Inline Code with Perform Name (8.5)

Add 4 non-public free capabilities (file-level, not strategies) earlier than the
codec block. Substitute all 128+ json!({"stringValue": …}) /
json!({"timestampValue": …}) and so forth. inline expressions with calls to
these capabilities. Every multi-word json macro name turns into a single
brief name.

Estimated financial savings: ~80 strains (largely from multi-line json macros collapsing to one-liners).

Fowler ref: Extract Class (7.3)

The ~20 encoder capabilities all comply with this form:

let mut fields = serde_json::Map::new();
fields.insert("foo".to_string(), str_val(&x.foo));
fields.insert("bar".to_string(), ts_val(x.bar));
json!({"title": path, "fields": fields})

Extract a small builder. Rewrite every encoder perform to make use of the
builder. A ~40-line encoder shrinks to ~12 strains.

Estimated financial savings: ~500–600 strains throughout the 20 encoder capabilities.