Can a Local LLM Run My AI Assistant?

TL;DR — I replayed 27 real tasks from my own AI agent against two local models, one hardware upgrade apart, scoring both against the same frozen Claude baseline. On a single RTX 3090 capped at a 16K context, a 30B model scored 22.8/100 to Claude’s 89.4, and leaked malformed tool-call syntax into a quarter of its answers — broken, not merely worse. On three 3090s with a 256K context, a 122B model scored 80.0, eliminated the malformed calls entirely (0 of 27), and cost $0.000969 per task against Claude’s $0.763 — roughly 787× cheaper. My agent now runs on the local model; I cancelled Claude Max and kept Pro. The honest caveat: model size and context budget changed together and I ran no control, so this measures a system upgrade, not an isolated model effect.

agent — a LangGraph react-agent wired into roughly 90 tools: email, calendar, notes, files, Office, WhatsApp, Discord, image generation, even spawning sub-agents for longer jobs. It ran on Claude from the day I built it, because Claude was the model I trusted it with.

A month ago I tried to replace that brain with a model running on my own hardware. It went badly, and I wrote up exactly how badly.

Then I changed the hardware and tried again, and got a different answer. Both attempts are below, measured the same way, because the distance between them is the actual finding.

Last time, on a smaller stage

The earlier piece — Local LLM Agents on an RTX 3090 — tested 5 local models across 2 agent frameworks on 17 tasks: 12 coding, 5 general-agent. qwen3-coder:30b topped that leaderboard cleanly. It was a fair test, and it told the truth about that scope.

Jarvis is a different scope entirely. Not 17 tasks in a clean harness — ~90 real tools, a personal-context system prompt, and years of actual messy requests logged in Langfuse, Jarvis’s self-hosted tracing. If the benchmark win was going to generalize anywhere, it should generalize here.

The setup: a replay, not a re-run

I pulled 28 real historical task prompts straight from Jarvis’s own Langfuse traces (90-day window), stratified across 7 categories — calendar, code, email, files, general, messaging, notes — 4 each.

Claude’s side of the comparison is its actual historical production answer, already logged in Langfuse. I did not re-run it. Re-running Claude through a sandboxed harness would mean feeding it fake stubbed tool data it never actually saw, which handicaps it for no good reason — the honest baseline is the answer it really gave, with the real data it really had. That baseline is frozen: it is identical in both rounds below. Only the local model is re-run.

The local side is a fresh run through a sandboxed replay harness: the real Jarvis LangGraph agent code, in-process, with every write-capable tool (send email, write to calendar, post to Discord/WhatsApp, write files) intercepted so nothing real happens. Read-only tools that touch real external systems — Outlook COM for email and calendar — are intercepted too, but the mock serves the real recorded output from that task’s original Langfuse trace when one exists, not a generic stub. The local model reasons over the same real inbox and calendar content Claude saw, not invented placeholder text.

The wrapper is default-deny: any tool not on a short explicit allowlist gets mocked. That detail earned its keep — between the two rounds Jarvis grew new tools, and default-deny meant they were intercepted automatically instead of quietly executing against my real accounts.

Quality is scored independently — not pairwise, to avoid position bias — by an LLM judge (claude-opus-4-8) on a 1–5 scale mapped to 0–100, applied identically to every answer in both rounds.

Worth flagging plainly: the judge is a Claude model, scoring Claude’s own answers alongside the local ones. That’s self-preference bias — a documented effect in LLM-as-judge setups, where a model tends to rate outputs from its own family more favorably. I can’t cleanly rule it out, and it probably flatters Claude. It’s a real limitation of this methodology, not a footnote to bury — and it matters more in round 2, where the local model is close enough for a thumb on the scale to change how you read the result.

Cost is the other half of the comparison, and it needs an instrument. Claude’s side is easy — Langfuse already recorded the real API billing for every historical call. The local side is electricity, which nothing bills you for per task. Every local run is logged as a priced experiment in HomeLab Monitor, the open-source monitoring stack I maintain: it attributes measured GPU energy to the window each run occupied and converts it at my actual dual-rate tariff. Neither side of the cost comparison is an estimate.

Three passes before the number was trustworthy

Worth a paragraph, because the numbers below are only worth publishing if the process that produced them holds up.

The first scoring pass had a silent bug: the judge model wraps its response in a list of content blocks, not a plain string, and the scorer’s JSON parser choked on ~40 of 54 judge calls — caught, logged, and quietly replaced with a neutral fallback score instead of erroring out loud. That run said Claude 71.6 vs qwen 43.5. Mostly noise.

Fixed and re-run, the gap got more extreme — Claude 89.2, qwen bucketed at exactly 15 or 45 on nearly every task. Investigating that found the real problem: 16 of the 28 tasks are calendar, email, notes, or messaging — categories where the safety wrapper correctly mocks the underlying Outlook COM calls, but the mock was returning a generic canned string instead of real data. qwen was reasoning over placeholder text while Claude’s historical answer was written against a real inbox. That wasn’t measuring model capability, it was measuring who got real data and who didn’t. Fixed by serving the real recorded tool output from the original trace wherever the mocked tool matched one the model actually called.

Re-run clean, then a third bug — a sustained Claude API rate limit during scoring silently neutral-scored another batch of judge calls. Added a real backoff loop, re-ran.

That neutral fallback deserves one more sentence, because it bit me again in round 2 — not as a bug, but as a thing I had to rule out. The judge returns exactly 70.0 when it hits an infrastructure error. In round 2, eight of 27 tasks scored exactly 70.0. Before believing any of it I checked every rationale string: all eight were genuine “3/5” judgements with substantive reasoning, zero infrastructure fallbacks, zero unparsed responses. A number that happens to equal your failure sentinel is worth verifying rather than celebrating.

Round 1: one 3090, and a model that couldn’t hold the room

The first attempt ran qwen3-coder:30b on a single RTX 3090 shared with everything else in the home lab. That sharing set a hard limit: with ~18 GB of weights resident, the context window had to come down to 16,384 tokens. Jarvis’s tool schemas and system prompt alone nearly fill that.

Claude averaged 89.4 / 100. qwen3-coder:30b averaged 22.8 / 100.

qwen never won a single category. Calendar and general — the tasks leaning more on reasoning over given context than on chaining tools correctly — were its best showing, and even those landed at a third of Claude’s score.

Where it broke

Quality score is the summary; it’s not the interesting failure mode.

qwen leaked a malformed tool-call tag into 7 of 27 final answers (25.9%) — instead of issuing a proper LangGraph tool call, it emitted raw text like ... straight into the response a user would have read.

Tool-overlap recall averaged 14.8%, measured only across the 18 of 27 tasks where the real historical trace actually used at least one tool. Most of the time qwen either called different tools than the ones that historically solved the task, or none at all.

And on 2 of 27 tasks, a genuine orchestration failure: pilot-17 (email, 24 tool calls) and pilot-27 (messaging, 27 tool calls) both got stuck calling the same already-answered tool over and over instead of synthesizing an answer and stopping. I checked the raw logs specifically to rule out the earlier mock-data bug: both tasks correctly received real replayed data. A stopping-condition failure, not starvation.

Two ways to lie about a mock

One task made both models actually call send_email(...) inside the harness — correctly intercepted, nothing sent, in either case. Claude then told the user the email had been sent. It hadn’t; that’s a fabrication. qwen correctly reported that the send didn’t go through.

It’s tempting to read that as “qwen is more honest.” It isn’t that clean — qwen is also the model leaking raw function-call syntax into user-facing text a quarter of the time, and looping twice. Both models mishandled the mocked environment, in opposite directions: one over-claimed success, the other under-delivered on basic task completion. Worth knowing, not worth a moral.

Same model, much harder room

Here’s what made round 1 a real finding rather than a cheap dunk: qwen3-coder:30b is the exact same model, same weights, same GPU, that scored 100% task success in my earlier controlled benchmark — 17 tasks, a much smaller tool surface, a purpose-built harness.

Dropped into Jarvis’s real environment, its reliability collapsed. Not because the model got worse. Because the room got much harder, and a model that’s excellent on a scoped benchmark is not automatically a safe drop-in for a large, real, production agent surface.

That was the honest conclusion, and the original version of this article ended there: Jarvis stays on Claude.

Then I changed the room.

Round 2: three 3090s, and a model that fits

The new box has three RTX 3090s — 72 GB of VRAM. That budget buys a different class of model: a 122-billion-parameter mixture-of-experts at Q3_K_M, about 53 GB of weights, served entirely on GPU with no CPU spill. And it buys the thing that turned out to matter most — a 256,000-token context window, a 16× increase over round 1’s 16,384.

Same 27 tasks. Same frozen Claude baseline. Same judge, same rubric, same harness.

80.0 / 100.

That’s 89.4% of Claude’s score, up from 25% in round 1. On 14 of the 27 tasks, the local model scored at or above Claude.

By category, the shape of the change is more interesting than the average:

general is the one category where the local model beats Claude outright — 90 to 85. notes and email land within a few points. files is where it’s still clearly behind at 64, and that’s the category most dependent on chaining several tool calls correctly rather than reasoning over content it was handed.

Reliability is where the change is starkest

The malformed tool-call leak went from 7 of 27 to zero. Not reduced — gone. In round 1 a quarter of answers had raw syntax bleeding into user-facing text; in round 2, none did.

Tool-overlap recall went from 14.8% to 38.0% — 2.6× better, and still the clearest remaining gap. The model now reliably produces well-formed tool calls; it still often picks different tools than the ones that historically solved the task.

The task that still doesn’t fit

One task never ran in either round. pilot-12 is a code dump: 336,906 characters. Round 1 excluded it against a 16,384-token window without ceremony.

Round 2 had 256,000 tokens to work with — and it still doesn’t fit. Ollama rejected it precisely:

request (281190 tokens) exceeds the available context size (256000 tokens)

The ceiling moved 16×, and the task overflowed it by roughly 25,000 tokens anyway. A useful reminder that “bigger context” is a moving target real work keeps outrunning — and the reason both rounds report 27 tasks rather than 28.

What this measurement cannot tell you

I want to be direct about the limits here, because the headline is tempting to over-read.

Two variables changed at once. Round 2 is not an isolated model upgrade. It is a bigger model and a 16× larger context window and three GPUs instead of one shared card. I did not run the control that would separate them — qwen3-coder:30b on the new box at 256K context. Without it, I can tell you the system got dramatically better; I cannot tell you how much belongs to the extra parameters versus simply no longer truncating Jarvis’s tool schemas and personal context at 16K.

My honest guess is that context is doing more of the work than parameter count, because round 1’s most characteristic failures — malformed tool calls, wrong tool selection, loops — are exactly what you’d expect from a model whose tool definitions are being cut off mid-schema. The fact that the malformed-call rate went to exactly zero points the same way: that reads more like “the schemas fit now” than “the model got smarter.” But that is a hypothesis I have not tested, and I’d rather label it as one than dress it up as a conclusion.

The judge is still a Claude model, with the same self-preference caveat as round 1 — now applied to a comparison where the local model is competitive, which is exactly where that bias matters most.

The local cost nearly went in wrong, and the reason generalises to any distributed setup. Energy gets attributed on the host being watched. In round 1 the machine driving the benchmark and the machine running the model were the same box, so that was correct without my having to think about it. In round 2 they aren’t — the harness runs on my desktop, the model runs on the GPU box — and the first pass duly attributed the work to the desktop’s idle card. The result came back as a tidy 0.0 kWh.

Nothing errored. I caught it only because zero was too clean to be true, and re-measured from scratch: sampling all three cards at 1 Hz, integrating power across each task’s exact window, and correcting for a 4-second clock offset between the two machines. If you take one habit from this piece, take that one — an instrument that agrees with your hopes deserves more scrutiny, not less, and “suspiciously round” is a reading worth chasing rather than celebrating.

What it actually costs

Priced at my home electricity tariff, the 27 tasks drew 0.1573 kWh across the three cards.

Two things worth reading off that table. The local model is still roughly three orders of magnitude cheaper than the API. And the upgrade was not free: the 122B costs 6.6× more per task to run than the 30B did. Going from “unusable and nearly free” to “usable and nearly free” cost real electricity — it’s just that “real electricity” here means fractions of a cent.

One caveat on that number: it’s gross, including idle draw. Those three cards pull about 107 W just holding the model resident, because I keep it loaded permanently for latency. If you’d only spin a model up on demand, your per-task cost would look different — lower marginal cost, higher latency.

What actually changed in Jarvis

Jarvis now runs on the local 122B model. It has for about a week.

I cancelled the Claude Max subscription and kept Pro. That’s the honest scoreboard: not “local models won,” but “local models got good enough that the top tier of my API spend stopped earning its keep.” Claude is still the better model on this benchmark — 89.4 to 80.0 — and it’s still what I reach for when I want an answer to be right the first time.

What changed is that the gap stopped being categorical. In round 1 the local model wasn’t a worse option, it was a broken one: a quarter of its answers had tool-call syntax bleeding into the text. You can’t put that in front of a real inbox at any price. In round 2 it’s a genuine trade — a few points of quality, on tasks I can identify in advance, for roughly a 99.9% cost reduction and my data never leaving the house.

The remaining gap is also specific rather than diffuse, which makes it manageable. files at 64 and 38% tool recall both say the same thing: this model is good at reasoning over what you hand it and still mediocre at deciding which tools to reach for. So the tasks I’d still route to Claude are the multi-step ones, and that’s a routing rule, not a subscription.

If you’re running the same experiment: the thing that moved the needle wasn’t picking a cleverer model. It was giving an adequate model enough context to see the whole tool surface at once. I’d test that before spending money on parameters.

Where’s your line — how many quality points is a task worth before you’d stop sending it out of the building?

All images in this article — charts and photographs — were created by the author.

Every energy and cost figure here was measured with HomeLab Monitor — an MIT-licensed, single-container stack I maintain for pricing local experiments. The shorter, more technical write-up of this harness lives on dev.to.