Till not too long ago I hadn’t tried operating fashions regionally for fairly some time, the frustration
had simply at all times been too excessive once I did it. A few month in the past although, I dove again in – there
have been simply too many claims on the market to disregard, about how far they’ve come, the way it’s now a lot
extra possible to run them, and the way a few of them have turn out to be actually good at coding. So that is
my private expertise with utilizing them, on and off, over the previous 4 weeks or so.
On this memo, I’ll begin with a extra normal introduction and undergo the components that affect the
viability of those fashions for coding. In a follow-up memo, I’ll describe my precise experiences in additional element.
Scope
My predominant curiosity is how helpful they’re for coding, and never simply auto full, however
agentic coding. Secondly, I am fascinated by how prepared and usable they’re extra
broadly, for builders who do not need to dive right into a bunch of specs and further tooling and
tweaks to make it work.
When it comes to {hardware}, I’ve been operating fashions on these two machines:
- Apple M3 Max, 48GB RAM
- Apple M5 Professional, 64GB RAM
Elements that affect viability
There are a myriad of things at play that may affect the outcomes, which makes it fairly
a tedious enterprise to guage which setup works finest underneath the useful resource constraints we
have. It additionally makes it very arduous to filter the indicators from the noise when folks share their
successes with these fashions on-line.
I discovered it significantly baffling that within the automated eval setup, one mannequin clearly delivered higher outcomes on the stronger machine (not simply pace, however higher code!), regardless of all different settings being the identical.
I will begin with a abstract, earlier than going into particulars of every of those components.
prefills
(system immediate, instruments, …)
various instrument schemas affect…
offers templates to enhance
higher {hardware} match results in higher…
differ in optimisation for…
Determine 1: The various components that may
affect the outcomes – faucet or hover over packing containers to focus on arrows.
- Runnability: RAM is the core constraint. I used fashions between 15-25GB, context
home windows of most 64K, harnesses OpenCode and Pi with zero Expertise and MCP servers energetic) - Response pace: Is impacted by many components, however was fairly good for some fashions, and leaps and bounds from the place it was a yr in the past. Setups: LM Studio + 4BIT quantization + M3 Max/M5 Professional + GGUF/MLX each
- Viability for agentic coding: Software calling was difficult nonetheless, the fashions usually failed, however can often self-recover from their failures. It is a key element of agentic coding particularly. With out it, you may nonetheless go the old style manner of copying and pasting from a chat window in fact. And small fashions are undoubtedly much more viable for auto full than agentic use.
- High quality of outcomes: Will depend on the duty (extra on that within the subsequent memo), however clearly nowhere close to the kind of functionality we will get from the massive fashions. Total it was very hit or miss. I solely checked out correctness of performance, did not go into code high quality.
RAM
The mannequin weights should match into the obtainable RAM, or extra particularly, VRAM. In the event that they
do not, the runtime both crashes (occurred to me as soon as!) or drops to unusably gradual
speeds.
On Apple Silicon, nearly all RAM is accessible to the GPU with no separate VRAM
restrict,
this may be totally different in different machine configurations.
Impacts: Mannequin runnability; Velocity of responses
My expertise: On the machine with 48GB, I ran fashions between 8 and virtually 30GB.
The 30GB stretches it a number of course, particularly when the context window will get added,
the 15-25GB dimension is extra snug and I haven’t got to shut fairly as many different
purposes. On the machine with 64GB, I as soon as ran a mannequin that was 48GB huge – which
went nice at first, however then it rapidly crashed…
Processing energy
Extra cores usually means quicker token technology, however structure issues too, and
newer chip generations can shut the hole even with fewer cores. It is a robust one to
examine between machines with out diving a lot deeper into the small print of every
configuration
and structure.
Impacts: Velocity of responses
My expertise: On each the M3 Max and the M5 Professional, all fashions I ran had fairly an
spectacular pace, in comparison with the place they have been a yr in the past. Velocity degrades although the longer a dialog will get. I’m okay utilizing them on the present pace for some duties – if the standard of the output have been acceptable.
Reminiscence bandwidth
Reminiscence bandwidth is a bottleneck for token technology, figuring out how briskly information strikes
between RAM and the compute models.
Impacts: Velocity of responses
My expertise: Each the M3 Max and the M5 Professional I used have an almost similar
bandwidth of ~300 GB/s, so I do not actually have a comparability to one thing else. However as
talked about earlier than, the pace of all fashions I attempted was fairly acceptable.
Variety of parameters
The parameter depend mainly represents the scale of a mannequin’s realized information and
capabilities. Extra parameters usually imply higher output high quality, but in addition a bigger
file
dimension.
Impacts: Quantity of RAM wanted; High quality of outcomes
My expertise: With 48GB, I used fashions round 30B parameters, +/- 5B. The
largest mannequin I loaded on the 64GB machine was Qwen3 Coder Subsequent 80B (MoE), which solved
the duty I gave it rather a lot higher than the smaller fashions – however then crashed after I
continued the dialog.
Reasoning capabilities
Reasoning fashions undergo a “chain of thought” course of earlier than responding, which helps
with complicated multi-step duties, however can even generate considerably extra tokens and gradual
responses down.
Impacts: Complexity of the duties; Velocity of response; Context window dimension (and
due to this fact want for RAM)
My expertise: All the fashions I attempted had reasoning capabilities, they usually have been switched on by default all through most of my experiments. Nonetheless, I
usually observed them getting into limitless circles inside the reasoning chain, particularly
the smaller ones. (“Wait, …”, “Really, …”, “However wait, …”) So I additionally did a couple of runs of my automated setup with reasoning off – and lo and behold, it isn’t solely quicker (which was to be anticipated), but in addition carried out the identical to barely higher! reminder that reasoning shouldn’t be at all times vital, and may generally even be counterproductive.
Software calling capabilities
For agentic use, a mannequin should be capable to reliably emit structured instrument name syntax that
matches the schema the harness expects. Fashions that weren’t particularly educated or
fine-tuned for instrument calling usually produce malformed calls.
Impacts: Skill to make use of agentic harnesses
My expertise: This was a typical problem with the fashions I attempted, although they
might usually self-correct and get better from a failed instrument name (e.g. utilizing flawed parameter
names like file.path as a substitute of filePath).
Format
GGUF is the usual format for llama.cpp-based runtimes like LM Studio and Ollama, and
has by far the most important mannequin library. MLX is Apple’s personal framework constructed particularly
for
Apple Silicon and may be quicker, however fewer MLX-formatted fashions can be found on the
second.
Impacts: Velocity of responses
My expertise: I attempted each codecs for one or two of the fashions, however I
personally did not really feel a lot of a distinction. That could possibly be because of the
unstructured sort of analysis I did – then again, as any person expertise
researcher would inform us, the
perceived pace is in the end what issues, not what the clock says…
Quantization
Quantization compresses mannequin weights to scale back the file dimension, buying and selling some high quality for
a
a lot smaller RAM footprint. The extent of quantization is often marked in mannequin names
and
descriptions as This autumn / Q6 / Q8, or 4BIT / 6BIT / 8BIT, with decrease numbers that means larger
compression. The latest buzz not too long ago have been QAT (Quantization-Conscious Coaching)
variants
of fashions, that are educated with quantization simulated in the course of the coaching. They’re
alleged to protect high quality higher than normal quantization.
Impacts: Quantity of RAM wanted; Velocity of responses; High quality of responses
My expertise: All the fashions I downloaded have been at This autumn / 4BIT, I did not get
round to making an attempt totally different variations but. I additionally have not gotten round to making an attempt a QAT
one.
Structure
MoE (Combination of Specialists) fashions have a big whole parameter depend however solely
activate a subsection of their weights at inference time, so a 35B MoE mannequin wants
considerably much less RAM and may run quicker than a 35B dense mannequin.
Impacts: Quantity of RAM wanted; Velocity of responses
My expertise: The Qwen3.6 35B MoE mannequin was by far giving me the perfect stability between variety of parameters and RAM utilization, and due to this fact runnability and high quality of outcomes. This could possibly be because of the MoE structure, I am unsure. The structure may also clarify my expertise of getting higher coding skills out of the mannequin on the 64GB machine than the 48GB – it is likely to be loading extra specialists there? I am unsure if that is true, but it surely’s the one wise speculation I’ve up to now.
Context window dimension setting
Context window dimension consumes RAM on prime of mannequin weights by means of the KV cache, which
grows with context size. The default dimension configured within the runtimes is way too small
for agentic coding, it needs to be set to no less than 32K, if not 64K.
Impacts: Dimension and complexity of duties; Quantity of RAM wanted; Velocity of responses;
Skill to make use of reasoning
My expertise: I attempted to see how little I might get away with. For small duties
I might generally work with 32K, however usually I needed to improve to 64K, in order that appears to be
a great default minimal. Because the fashions themselves have been already pushing the boundaries of
my obtainable RAM, I am unsure how how rather more I might nonetheless improve it, even on the
64GB machine… So whereas many of those fashions in principle help a bigger window,
truly utilizing it’s restricted by the reminiscence constraints.
Listing of fashions used
The thrill for coding has been throughout Qwen3 and Gemma 4, so these have been those I
went
for.
Qwen3
- Qwen3.6 35B-A3B MoE This autumn GGUF (22 GB)
- Qwen3.6 Coder Subsequent 80B MoE GGUF (45 GB)
Gemma 4
- Gemma 4 12B This autumn GGUF (7.5 GB)
- Gemma 4 26B 4BIT MLX (15.6 GB)
- Gemma 4 31B 4BIT MLX (29 GB)
Runtime
The runtime handles mannequin discovery, configuration, and loading. It additionally determines the
sensible query of how we combine our harness with the mannequin. Often this occurs by
beginning an internet server that gives a spread of typical APIs that the harnesses help,
and the localhost URL of that internet server is then configured as a mannequin
supplier within the harness. Most generally supported by harnesses is the OpenAI API, however Claude
Code e.g. expects Anthropic’s
Claude API .
Impacts: Ease of configuration and discoverability; ease of integration with
harnesses; pace of responses
Determine 2: The “Developer” view in LM Studio, displaying a
operating server and lots of the components described up to now (supplier url, mannequin dimension, APIs,
context window dimension configuration)
My expertise: Whereas I’ve used different runtimes up to now, I’m at the moment again
utilizing LM Studio, primarily for its person expertise. There are many ins and outs of which runtime is essentially the most optimised for which {hardware} and what sort of fashions, to get much more pace out of it. However considering again to the broader viability of operating native fashions, person expertise performs an enormous position
for that. For what it is price, essentially the most steadily talked about different from my colleagues was oMLX.
Harness (Claude Code, OpenCode, Pi, …)
Coding harnesses can range considerably in how a lot overhead they inject into the context
window (system immediate, variety of instruments), which turns into extra of an issue regionally the place
we’re
so useful resource constrained. Our personal expanded harness round that additionally makes a distinction,
e.g.
what number of abilities or MCP servers are energetic. An outline of every of them will probably be despatched to
the
mannequin, and once more, take up area within the context window.
I discussed above that small fashions nonetheless battle with instrument calling – and it most likely
would not assist that every harness has barely totally different schemas for the fundamental instruments. Let’s
take enhancing a file for example:
- Pi:
old_textandnew_text(see
right here) - OpenCode:
oldStringandnewString(see
right here) - Claude Code:
old_stringandnew_string(no less than that is what
it says once I requested it)
Lastly, not all harnesses simply help the mixing of native fashions. Open supply
instruments are often the go-to, however Claude Code may also be pointed at native suppliers. GitHub
Copilot appears to help it for his or her CLI, and I believe it is potential in Cursor as properly
to override the OpenAI base URL and level it to localhost.
Impacts: Dimension of context window wanted; Software calling success; Integratability
My expertise: In my makes an attempt, I used OpenCode and Pi. I averted Claude Code because it
apparently would burden the context
window fairly a bit.
Developing
In my subsequent memo, I’ll dive extra deeply into the forms of duties I gave the fashions, and what I skilled.
A preview of my total conclusions: Utilizing small fashions like that is nonetheless fairly messy and arduous to guage. It was a irritating expertise to come back to conclusions, because the outcomes rely on so many issues. I’d due to this fact say that it is nonetheless not prepared for a easy “plug and play” expertise for builders who do not need to spend an excessive amount of time on it.
Nonetheless, based mostly on this expertise I do have a go-to mannequin that I am utilizing regionally now, which is Qwen3.6 35B MoE. It supplied the perfect stability of functionality, pace and RAM footprint amongst what I attempted.
Extra to come back!