Contact Us

Local AI
Local AI

Article

Is Local AI Ready to Replace Frontier Models?

By

Andrew Duncan

Local AI

Article

Is Local AI Ready to Replace Frontier Models?

By

Andrew Duncan

What an in-house AI box can and can't do.

Why we went looking

Two things enterprise IT operations and security teams care about more than anything are cost and data protection of their systems respectively. Neither is new. What's new is that AI landed on both at the same time. 

For the first time in a while, a technology feels like it comes with data lock-in built in. Companies are spending real money to have the best AI available, and the best AI available runs on someone else's hardware under someone else's terms. That's uncomfortable when the spend keeps growing, and it's a real problem when the data isn't yours to move. 

Then there's the question of what happens to the data once it's there. Whether it could end up in the base knowledge of a future model, who can see it along the way, and what the exposure looks like if any of those changes. Most providers have reasonable terms on this today, but you're trusting terms rather than architecture, and terms get revised. 

This is a frequent theme, though it's rarely the whole company asking. More often it's one part of the business that has data that can't leave while the rest is perfectly happy in the cloud. 

The counter has always been price. Owning AI infrastructure meant enterprise servers running into six figures, so against a monthly subscription it was never a close call. 

That's the part that changed. Machines like the GMKtec EVO-X3, Ryzen AI Max+ 395, 128 GB of unified memory, a few TB of storage, lands in the $3–4K range depending on how you configure it. That's enough memory and enough bandwidth to run genuinely capable open models. Not cheap, but not a capital project either, and roughly what a single power user spends on frontier models in a year. So we bought one and spent a few weeks finding out: 

•

If in-house AI now costs about what one heavy user spends on frontier models in a year, is it worth doing?

•

Could we skip the monthly fees and just give developers machines that can run models themselves?

•

And if it can't replace what we use today, is there still a version of this worth owning?

I've been an engineer for 25+ years and have spent the last two using agentic coding tools daily. I wanted this to work, and I've tried to be honest about where it didn't.

The machine, briefly

It's a mini-PC with 128 GB of unified memory and roughly 256 GB/s of bandwidth. That's enough to hold a good 30B model at high quality with a 256K context window and still run image generation next to it. After tuning, it serves around 70 tokens/second on a fresh session.

Getting there took more work than expected. The inference engine turned out to matter about 4.5x more than the model choice did, and most of the advice online for this chip was wrong. That's all in a separate post: Make a Local LLM Feel Like a Frontier Model (coming soon).

Where it doesn't replace Claude Code or Codex

Caveat, my testing was all done through OpenCode, there are others. OpenCode, to me, is the most like Claude Code/CodeEx for open models. 

At 70 tokens/second, it feels fine. In a real session, it doesn't stay there. Throughput settles closer to 27 tokens/second once you've built up a working context, because generation slows as the conversation grows. 

That's the core problem. Agentic coding is a long chain of round trips against a context that only gets bigger, so most of the session is spent on the slow side of that curve. Frontier models are faster to begin with, and they don't fall off the same way as the session gets long. 

Capability is the other half. What fits on this box competes with Haiku and Sonnet, not Opus or GPT-5-class models. That's not a criticism; it's just the weight class. A good 30B mixture-of-experts model does well on classification, summarization, structured extraction, first-pass review, and small-scoped code changes. It does not do well on multi-step reasoning across a codebase it hasn't seen. 

Practically: for a scoped change in a file you already understand, it's close enough that I'd use it. For "figure out why this fails in production," it isn't. Compare it to the mid-tier models, and it looks good. Compare it to the top tier, and you'll be disappointed, though those are also the models you're least likely to be running in bulk anyway. 

One caveat: local agentic coding doesn't really work until you give the model a map of the codebase instead of expecting it to read its way to an answer. That's a real dependency, and it's covered in the technical post (coming soon).

Where it works well

Two areas, and both are unglamorous.

Everyday chat is basically solved. Pair a local model with a web chat UI and a tool gateway and it's close enough to a commercial assistant that I stopped noticing the difference. The gateway is why. A local model's training ended months ago, and it knows nothing about your business, but give it live web search, page fetching with a real browser, persistent memory,  

RAG over your own documents, and image generation, and the staleness mostly stops mattering. It looks things up, it remembers you, it reads your files. For general knowledge work on sensitive data, I'd hand this to a client today. 

Pipeline and automation work is where the real case is. I didn't expect this to be the headline, but it is. 

Everything that makes local AI painful for interactive coding doesn't apply to unattended work: 

•

Nobody is waiting. A triage job that takes 90 seconds instead of 15 is the same job.

•

Contexts are short and disposable. Evaluate one Playwright failure. Classify one error. Decide whether to escalate. Draft one issue. Each task starts fresh and ends in a few thousand tokens, so you never get near the depth where speed falls off.

•

Volume is high, repetitive, and often continuous. A queue that never empties is the one billing shape where fixed-cost hardware clearly wins over metered tokens.

•

The judgment required is mid-tier. Classify, triage, summarize, draft. That's what a local 30B does well, and it's what you'd send to Haiku or Sonnet rather than Opus anyway.

Playwright automation is the clearest example. Let a local model evaluate the failure, decide whether it's a real regression or a flake, open an issue with the trace attached, and escalate only the ambiguous ones to a frontier model. Same idea for first-pass code review: local handles style, obvious bugs, and missing tests, and you spend frontier tokens on the architectural questions. That's not local AI replacing anything; it's taking the high-volume, low-judgment tier so your paid tokens go to the work that needs judgment, and the sensitive data stays in the building.

It's not a single-purpose box

The spec sheet won't tell you this, but inference barely touches the CPU. Generation is bound by memory bandwidth and the integrated GPU, so 16 cores and 32 threads sit mostly idle while the model works. There's a lot of machine left over. 

Use it! On top of the model the box can run an entire supporting stack — reverse proxy, Postgres with pgvector, an embedding service, object storage, a self-hosted search backend, image generation and more. Image generation shares the GPU with the language model permanently and neither has to be stopped for the other. None of it dented the CPU. 

The overhead is real and something to consider in your evaluation: Think, an internal docs or wiki search index, a private container registry or package mirror, a shared dev database, scheduled ETL and reporting jobs, log and metrics collection, a staging environment. None of those need the GPU, and the GPU workload doesn't need them. 

Self-hosted GitHub or GitLab runners are the most obvious one. CI is bursty and schedulable, so it uses exactly the CPU capacity inference never asks for. And self-hosted runners keep builds inside the same boundary the local model already respects. A client who can't send code to a third-party API usually can't send it to a third-party CI cloud either. The privacy requirement doesn't stop at the model. 

The constraint isn't CPU though; it's host RAM, and that's where people get caught. Allocating 96 GB to VRAM leaves about 30 GB for the host OS and every container on it. That's fine for services and proxies, tight for anything memory-hungry, and it caused the two most tedious bugs of the whole build: a model that wouldn't memory-map because the host couldn't back the mapping, and an image generator that pinned 23 GB of unswappable RAM and took the machine down. Size your other workloads against 30 GB, not 128. 

This changes the capital math even though it doesn't change the token math. A $3–4K machine that's also your CI runner and your container host is a different purchase than one you have to justify against AI spend alone. 

The laptop option 

A shared box isn't the only shape this comes in. The same Ryzen AI Max+ 395 ships in laptops with the same 128 GB of unified memory, so putting a capable local model directly on a developer's machine is a legitimate approach rather than a hypothetical one. It's worth considering, with a few caveats.

The OS matters more than you'd think. Linux gets you the most usable memory and bandwidth, less baseline overhead, and more control over what you hand the GPU. Windows takes a bigger cut before you start, so if corporate policy keeps the machine on Windows, you're giving up capacity you paid for. That's a real constraint on a fleet device in a way it isn't on a box you control end to end.

It's a battery burner. These parts run up to around 160W sustained against roughly a 99Wh battery, so continuous inference isn't an on-battery activity. In practice, it's a tethered machine, a desktop with a built-in monitor. 

The memory split still applies. Our box gives three quarters of its memory to the model permanently. A laptop has to run everything else the person does, so realistically they'd run smaller or more heavily quantized models. 

Always-on work still wants a server. Pipeline jobs, scheduled analysis, CI, a shared chat endpoint, the document index, the embedding service. None of that runs on a machine that closes at 6 pm and goes home. And at fleet scale, a local model on every laptop means the sensitive content is copied onto every laptop, which is a different security posture than one machine in your own rack. 

Where a laptop makes real sense: someone whose driver is privacy and who needs to stay productive offline or away from the office. Consultants on client sites, field work, secure facilities, anyone regularly working without a usable connection. For that person, a machine carrying its own model beats a server they can't reach, and it's enough for a single developer who just wants private inference. The shared box wins once you have a team, shared indexes, or unattended work. 

The cost math, and why it probably isn't your reason

Call the box $3–4K depending on storage. One capital expense, depreciable, and after that a token costs electricity. 

Whether that's a good deal depends on something most teams haven't measured: how much of your AI spend is metered versus flat. Flat spend can't be displaced. A Claude Team seat is $20–$25/month standard or $100–$125/month premium with Claude Code included, and if an engineer already has one, moving test triage to a local box saves nothing. 

Heavy users go well past those tiers. An engineer running a lot of parallel agentic sessions, a review in one tab, a refactor in another, an investigation in a third, costs about $400/month in metered usage. So the box costs about what one heavy user spends on AI in a year, or a little less. 

That lets us ask a better question than "what's the payback," which would mean guessing how much work actually moves. Instead: what share does the box have to absorb to pay for itself in a year?

Engineers @ $400/mo 

Team spend / month 

Offload needed to cover a $4K box in a year 

1 

$400 

83% 

5 

$2,000 

17% 

10 

$4,000 

8% 

20 

$8,000 

4% 

8% for a ten-person team looks like an easy bar. But the table assumes the work can move, and mostly it can't. That $400 is Opus-tier spend because the work needs it — multi-step reasoning across an unfamiliar codebase, over a context that grows all session, with enough judgment to trust on a refactor. Both limits above hit hardest on exactly that work. You can't offload a task to a model that can't do it, so the eligible share isn't 8% by default. It might be close to zero.

The part that is eligible is work billed at Opus rates that never needed Opus — mechanical review passes, triage, classification, summarizing a diff. That's real, but fixing it is a routing change, not a purchase. Point that traffic at Haiku or Sonnet and you get most of the savings this afternoon, for free. So the box never actually competes against the $400. It competes against the much smaller Haiku bill that should have replaced it, and against that a $3–4K box is a hard sell.

One thing widens the slice, and you should measure it before writing any of this off: an engineer running five agent tabs isn't waiting on any one of them. That's what makes slower inference tolerable, and interactive-but-parked work sits a lot closer to the unattended tier than it looks. It helps the cheap API models just as much, though.

The one cost argument I'd defend is the multi-purpose one. Not "it pays for itself in displaced tokens," but "it's a capable server we needed anyway, and inference happens to use the part we weren't otherwise buying." The better reasons were never about money.

Four reasons that have nothing to do with cost

Roughly the order I'd raise them with a client.

1. Data stays yours.

If a client contractually can't send code, customer records, or contract language to a third-party API, the alternative to a $3–4K box isn't cheaper AI, it's no AI. There's no per-token price that competes with "the data doesn't leave the building." This is the reason to buy the box.

One task can only be done this way: redaction and de-identification. To have a third-party service find the sensitive parts of your data, you have to send it the sensitive data first. Privilege screening, PHI stripping, PII discovery, local inference is the only architecture where that job makes sense at all. 

2. It works with no internet.

Air-gapped environments, secure facilities, client sites with locked-down egress, a plane. It also makes you independent of things you don't control: provider outages, rate limits at peak hours, regional restrictions, a bad connection at the wrong moment. For some clients, that isn't a convenience; it's the only deployment they're allowed. 

3. The model can't be taken away.

Hosted models get deprecated, retired, re-tuned, and quietly updated. Weights on your own disk are a file you own. The model you validated is the model you'll be running in two years. That matters if you've tuned prompts against specific behavior, and it matters a lot in a regulated context where "nothing changed" has to be literally true. 

4. No refusals on legitimate work.

People are shy about raising this one, so I'll just say it. Hosted models are tuned conservatively for a general consumer audience, and that has real false-positive costs for professional work — security research and penetration testing, malware analysis, clinical content, legal discovery over ugly facts, moderation tooling that has to process the material it moderates, frank assessment of weaknesses in a client's own system. Hitting a refusal mid-engagement isn't an ethics win, it's a tool that stopped working on a job someone is paying for. 

The flip side belongs in the same paragraph. Take the guardrails off and the judgment is yours. When I set up image generation on this box I checked what it would refuse to produce and the answer was basically nothing, which is exactly why I kept the image tools off any shared endpoint. Unrestricted is a reasonable default for a professional using a tool. It's not a reasonable default for an endpoint with an audience. 

How you'd decide

Everything above is a throughput-and-feel assessment. I measured tokens per second carefully and judged quality by using the thing, which isn't rigorous. If you're putting a local model in front of real work, "it feels close enough" isn't a standard — you want a small eval set, numeric pass criteria written before you test, and the smallest model that clears them. The method is in the technical post.

The rule that came out of all of this: the task shape decides, not the model. One short call with a small prompt play to local hardware's strengths. A hundred chained calls against a growing context plays to all its weaknesses — that's the 70-to-27 drop, and it's why the same box looks strong on pipeline work and mediocre in an agentic coding session. Sort your workloads that way and most of the deployment question answers itself.

Where we landed

If the driver is data privacy, this hardware is a real answer today — chat, document analysis, RAG over internal content, and the automation tier. That case doesn't need qualifying.

If the driver is token cost, probably not, and for a structural reason rather than a math one. Heavy spend is Opus-tier spend on Opus-grade work, which is the work local hardware can't take. What it can take, you could have routed to Haiku for a fraction of the price and no capital outlay. Worth measuring at team scale, but not the reason to buy. 

If you operate offline, air-gapped, or under change control, or your work keeps running into consumer-tuned refusals, this is the deployment model and cost never really enters into it. 

If you want to replace Claude Code or Codex for daily development, not yet. I'd rather say that now than after someone has bought the box. 

The summary is that this hardware doesn't remove the need for frontier models. It changes what you spend them on, and it changes where your data has to go to get work done. The second part is what clients actually care about. 

What's next

Most of what made this usable wasn't the model or the machine, it was the tooling around it. Make a Local LLM Feel Like a Frontier Model is the technical companion: the hardware findings in full, why the inference engine mattered more than the model, and the gateway we built — how you give a model whose training ended months ago current information, persistent memory, RAG, vector search, and image generation server-side, so any client pointed at the endpoint gets all of it with no configuration. That's the piece we're packaging up for clients to use.

After that, the measurement work: golden datasets and real evals for the task types we'd hand to a local model. That's what turns "this box is interesting" into "this model clears your bar on your task."

Appendix: considerations, by industry

Optional reading. If you already know your own workloads, the rule above is enough.

The pattern is the same across all of these: the local model reads, structures, classifies, and routes. It doesn't decide. It turns unstructured mess into something searchable and sorted, then hands the judgment calls to a person or a frontier model. That's why mid-tier capability is enough. Ordered by how absolute the privacy driver is. 

Legal

The strongest fit, because privilege and confidentiality aren't preferences. A lot of outside-counsel guidelines now explicitly prohibit putting matter data into third-party AI services, which ends the conversation before cost comes up.

•

First-pass discovery triage — classify documents as responsive, non-responsive, or possibly privileged, and route the maybes to an attorney. Millions of documents, each a short independent context, nobody waiting on any single result. Probably the best-shaped workload for this hardware.

•

Contract clause extraction against a playbook — find indemnity, limitation of liability, assignment, and change-of-control language, flag anything that deviates from the firm's standard positions.

•

Transcript and deposition indexing — chunk, embed, and tag by topic, speaker, and exhibit reference so associates can actually search a matter.

•

Intake categorization and conflicts pre-screening — pull party names and matter type, route to the right practice group.

Not for legal conclusions, and not for anything citation-bearing that gets filed. Fabricated citations are the known catastrophic failure here and they could cost people their licenses. Retrieval and classification, not authority.

Financial services and insurance

Privacy, plus a governance angle that quietly favors local weights: when a model is part of a regulated process, "nothing changed this quarter" needs to be literally true. A hosted model that gets re-tuned underneath you are audit problems.

•

Communications surveillance — classify email, chat, and call transcripts for escalation to compliance. High volume, runs continuously, and touches MNPI and customer PII that can't leave. This is also the billing shape where fixed-cost hardware genuinely wins.

•

KYC “Know Your Customer”/AML “Anti-Money Laundering” document intake — extract fields from IDs, statements, and utility bills, classify document type, check that a packet is complete.

•

Claims triage — first-notice-of-loss classification, complexity and severity routing, fraud indicators flagged for human referral.

•

Complaint categorization — a regulatory tracking obligation that's pure classification work.

•

Internal policy RAG — a "how do we handle this" assistant over compliance manuals and prior guidance.

Not the underwriting or credit decision, and never the adverse-action reasoning. Fair-lending explainability and model-risk governance put those out of scope. It's a pre-processor and a router.

Healthcare

HIPAA makes the privacy case on its own, and de-identification is the task that can only be done locally. 

•

De-identification of clinical notes before anything else touches them.

•

Prior-authorization packet assembly — is every required document there, does the diagnosis code match the narrative

•

Referral and patient-message triage — route by specialty and urgency.

•

Coding suggestions offered to a certified coder, never applied automatically.

•

Internal protocol and formulary RAG for staff.

Clinician in the loop, always. No diagnostic output.

Retail and e-commerce

The privacy driver is softer here, but two others take over: volume, and competitive sensitivity around pricing, margin, supplier terms, and unreleased assortment plans. Weaker case, much larger one.

•

Catalog normalization at scale — category assignment, attribute extraction from messy supplier descriptions, duplicate and variant matching. Millions of SKUs churning constantly, which is the continuous workload where metered billing hurts most.

•

Image attribute tagging — color, pattern, material, style pulled from product photos. Per-image API costs add up fast at catalog scale.

•

Review and ticket theme clustering — embeddings that surface "customers keep complaining about this" without anyone writing the query first.

•

Search query understanding over your own logs, which say more about your customers than most teams expect.

•

Supplier invoice and PO extraction.

Customer-facing chat probably still wants a frontier model, since brand voice and edge cases matter and someone's waiting on it. The backend pipeline behind it is a good fit for local. 

One thing worth planning for 

Document-heavy intake shows up in three of these four — IDs, statements, invoices, product photography — and all of it leans on the model's vision capability. That's a real reason to pick a vision-capable model even if your primary use case is text.

What an in-house AI box can and can't do.

Why we went looking

Two things enterprise IT operations and security teams care about more than anything are cost and data protection of their systems respectively. Neither is new. What's new is that AI landed on both at the same time. 

For the first time in a while, a technology feels like it comes with data lock-in built in. Companies are spending real money to have the best AI available, and the best AI available runs on someone else's hardware under someone else's terms. That's uncomfortable when the spend keeps growing, and it's a real problem when the data isn't yours to move. 

Then there's the question of what happens to the data once it's there. Whether it could end up in the base knowledge of a future model, who can see it along the way, and what the exposure looks like if any of those changes. Most providers have reasonable terms on this today, but you're trusting terms rather than architecture, and terms get revised. 

This is a frequent theme, though it's rarely the whole company asking. More often it's one part of the business that has data that can't leave while the rest is perfectly happy in the cloud. 

The counter has always been price. Owning AI infrastructure meant enterprise servers running into six figures, so against a monthly subscription it was never a close call. 

That's the part that changed. Machines like the GMKtec EVO-X3, Ryzen AI Max+ 395, 128 GB of unified memory, a few TB of storage, lands in the $3–4K range depending on how you configure it. That's enough memory and enough bandwidth to run genuinely capable open models. Not cheap, but not a capital project either, and roughly what a single power user spends on frontier models in a year. So we bought one and spent a few weeks finding out: 

•

If in-house AI now costs about what one heavy user spends on frontier models in a year, is it worth doing?

•

Could we skip the monthly fees and just give developers machines that can run models themselves?

•

And if it can't replace what we use today, is there still a version of this worth owning?

I've been an engineer for 25+ years and have spent the last two using agentic coding tools daily. I wanted this to work, and I've tried to be honest about where it didn't.

The machine, briefly

It's a mini-PC with 128 GB of unified memory and roughly 256 GB/s of bandwidth. That's enough to hold a good 30B model at high quality with a 256K context window and still run image generation next to it. After tuning, it serves around 70 tokens/second on a fresh session.

Getting there took more work than expected. The inference engine turned out to matter about 4.5x more than the model choice did, and most of the advice online for this chip was wrong. That's all in a separate post: Make a Local LLM Feel Like a Frontier Model (coming soon).

Where it doesn't replace Claude Code or Codex

Caveat, my testing was all done through OpenCode, there are others. OpenCode, to me, is the most like Claude Code/CodeEx for open models. 

At 70 tokens/second, it feels fine. In a real session, it doesn't stay there. Throughput settles closer to 27 tokens/second once you've built up a working context, because generation slows as the conversation grows. 

That's the core problem. Agentic coding is a long chain of round trips against a context that only gets bigger, so most of the session is spent on the slow side of that curve. Frontier models are faster to begin with, and they don't fall off the same way as the session gets long. 

Capability is the other half. What fits on this box competes with Haiku and Sonnet, not Opus or GPT-5-class models. That's not a criticism; it's just the weight class. A good 30B mixture-of-experts model does well on classification, summarization, structured extraction, first-pass review, and small-scoped code changes. It does not do well on multi-step reasoning across a codebase it hasn't seen. 

Practically: for a scoped change in a file you already understand, it's close enough that I'd use it. For "figure out why this fails in production," it isn't. Compare it to the mid-tier models, and it looks good. Compare it to the top tier, and you'll be disappointed, though those are also the models you're least likely to be running in bulk anyway. 

One caveat: local agentic coding doesn't really work until you give the model a map of the codebase instead of expecting it to read its way to an answer. That's a real dependency, and it's covered in the technical post (coming soon).

Where it works well

Two areas, and both are unglamorous.

Everyday chat is basically solved. Pair a local model with a web chat UI and a tool gateway and it's close enough to a commercial assistant that I stopped noticing the difference. The gateway is why. A local model's training ended months ago, and it knows nothing about your business, but give it live web search, page fetching with a real browser, persistent memory,  

RAG over your own documents, and image generation, and the staleness mostly stops mattering. It looks things up, it remembers you, it reads your files. For general knowledge work on sensitive data, I'd hand this to a client today. 

Pipeline and automation work is where the real case is. I didn't expect this to be the headline, but it is. 

Everything that makes local AI painful for interactive coding doesn't apply to unattended work: 

•

Nobody is waiting. A triage job that takes 90 seconds instead of 15 is the same job.

•

Contexts are short and disposable. Evaluate one Playwright failure. Classify one error. Decide whether to escalate. Draft one issue. Each task starts fresh and ends in a few thousand tokens, so you never get near the depth where speed falls off.

•

Volume is high, repetitive, and often continuous. A queue that never empties is the one billing shape where fixed-cost hardware clearly wins over metered tokens.

•

The judgment required is mid-tier. Classify, triage, summarize, draft. That's what a local 30B does well, and it's what you'd send to Haiku or Sonnet rather than Opus anyway.

Playwright automation is the clearest example. Let a local model evaluate the failure, decide whether it's a real regression or a flake, open an issue with the trace attached, and escalate only the ambiguous ones to a frontier model. Same idea for first-pass code review: local handles style, obvious bugs, and missing tests, and you spend frontier tokens on the architectural questions. That's not local AI replacing anything; it's taking the high-volume, low-judgment tier so your paid tokens go to the work that needs judgment, and the sensitive data stays in the building.

It's not a single-purpose box

The spec sheet won't tell you this, but inference barely touches the CPU. Generation is bound by memory bandwidth and the integrated GPU, so 16 cores and 32 threads sit mostly idle while the model works. There's a lot of machine left over. 

Use it! On top of the model the box can run an entire supporting stack — reverse proxy, Postgres with pgvector, an embedding service, object storage, a self-hosted search backend, image generation and more. Image generation shares the GPU with the language model permanently and neither has to be stopped for the other. None of it dented the CPU. 

The overhead is real and something to consider in your evaluation: Think, an internal docs or wiki search index, a private container registry or package mirror, a shared dev database, scheduled ETL and reporting jobs, log and metrics collection, a staging environment. None of those need the GPU, and the GPU workload doesn't need them. 

Self-hosted GitHub or GitLab runners are the most obvious one. CI is bursty and schedulable, so it uses exactly the CPU capacity inference never asks for. And self-hosted runners keep builds inside the same boundary the local model already respects. A client who can't send code to a third-party API usually can't send it to a third-party CI cloud either. The privacy requirement doesn't stop at the model. 

The constraint isn't CPU though; it's host RAM, and that's where people get caught. Allocating 96 GB to VRAM leaves about 30 GB for the host OS and every container on it. That's fine for services and proxies, tight for anything memory-hungry, and it caused the two most tedious bugs of the whole build: a model that wouldn't memory-map because the host couldn't back the mapping, and an image generator that pinned 23 GB of unswappable RAM and took the machine down. Size your other workloads against 30 GB, not 128. 

This changes the capital math even though it doesn't change the token math. A $3–4K machine that's also your CI runner and your container host is a different purchase than one you have to justify against AI spend alone. 

The laptop option 

A shared box isn't the only shape this comes in. The same Ryzen AI Max+ 395 ships in laptops with the same 128 GB of unified memory, so putting a capable local model directly on a developer's machine is a legitimate approach rather than a hypothetical one. It's worth considering, with a few caveats.

The OS matters more than you'd think. Linux gets you the most usable memory and bandwidth, less baseline overhead, and more control over what you hand the GPU. Windows takes a bigger cut before you start, so if corporate policy keeps the machine on Windows, you're giving up capacity you paid for. That's a real constraint on a fleet device in a way it isn't on a box you control end to end.

It's a battery burner. These parts run up to around 160W sustained against roughly a 99Wh battery, so continuous inference isn't an on-battery activity. In practice, it's a tethered machine, a desktop with a built-in monitor. 

The memory split still applies. Our box gives three quarters of its memory to the model permanently. A laptop has to run everything else the person does, so realistically they'd run smaller or more heavily quantized models. 

Always-on work still wants a server. Pipeline jobs, scheduled analysis, CI, a shared chat endpoint, the document index, the embedding service. None of that runs on a machine that closes at 6 pm and goes home. And at fleet scale, a local model on every laptop means the sensitive content is copied onto every laptop, which is a different security posture than one machine in your own rack. 

Where a laptop makes real sense: someone whose driver is privacy and who needs to stay productive offline or away from the office. Consultants on client sites, field work, secure facilities, anyone regularly working without a usable connection. For that person, a machine carrying its own model beats a server they can't reach, and it's enough for a single developer who just wants private inference. The shared box wins once you have a team, shared indexes, or unattended work. 

The cost math, and why it probably isn't your reason

Call the box $3–4K depending on storage. One capital expense, depreciable, and after that a token costs electricity. 

Whether that's a good deal depends on something most teams haven't measured: how much of your AI spend is metered versus flat. Flat spend can't be displaced. A Claude Team seat is $20–$25/month standard or $100–$125/month premium with Claude Code included, and if an engineer already has one, moving test triage to a local box saves nothing. 

Heavy users go well past those tiers. An engineer running a lot of parallel agentic sessions, a review in one tab, a refactor in another, an investigation in a third, costs about $400/month in metered usage. So the box costs about what one heavy user spends on AI in a year, or a little less. 

That lets us ask a better question than "what's the payback," which would mean guessing how much work actually moves. Instead: what share does the box have to absorb to pay for itself in a year?

Engineers @ $400/mo 

Team spend / month 

Offload needed to cover a $4K box in a year 

1 

$400 

83% 

5 

$2,000 

17% 

10 

$4,000 

8% 

20 

$8,000 

4% 

8% for a ten-person team looks like an easy bar. But the table assumes the work can move, and mostly it can't. That $400 is Opus-tier spend because the work needs it — multi-step reasoning across an unfamiliar codebase, over a context that grows all session, with enough judgment to trust on a refactor. Both limits above hit hardest on exactly that work. You can't offload a task to a model that can't do it, so the eligible share isn't 8% by default. It might be close to zero.

The part that is eligible is work billed at Opus rates that never needed Opus — mechanical review passes, triage, classification, summarizing a diff. That's real, but fixing it is a routing change, not a purchase. Point that traffic at Haiku or Sonnet and you get most of the savings this afternoon, for free. So the box never actually competes against the $400. It competes against the much smaller Haiku bill that should have replaced it, and against that a $3–4K box is a hard sell.

One thing widens the slice, and you should measure it before writing any of this off: an engineer running five agent tabs isn't waiting on any one of them. That's what makes slower inference tolerable, and interactive-but-parked work sits a lot closer to the unattended tier than it looks. It helps the cheap API models just as much, though.

The one cost argument I'd defend is the multi-purpose one. Not "it pays for itself in displaced tokens," but "it's a capable server we needed anyway, and inference happens to use the part we weren't otherwise buying." The better reasons were never about money.

Four reasons that have nothing to do with cost

Roughly the order I'd raise them with a client.

1. Data stays yours.

If a client contractually can't send code, customer records, or contract language to a third-party API, the alternative to a $3–4K box isn't cheaper AI, it's no AI. There's no per-token price that competes with "the data doesn't leave the building." This is the reason to buy the box.

One task can only be done this way: redaction and de-identification. To have a third-party service find the sensitive parts of your data, you have to send it the sensitive data first. Privilege screening, PHI stripping, PII discovery, local inference is the only architecture where that job makes sense at all. 

2. It works with no internet.

Air-gapped environments, secure facilities, client sites with locked-down egress, a plane. It also makes you independent of things you don't control: provider outages, rate limits at peak hours, regional restrictions, a bad connection at the wrong moment. For some clients, that isn't a convenience; it's the only deployment they're allowed. 

3. The model can't be taken away.

Hosted models get deprecated, retired, re-tuned, and quietly updated. Weights on your own disk are a file you own. The model you validated is the model you'll be running in two years. That matters if you've tuned prompts against specific behavior, and it matters a lot in a regulated context where "nothing changed" has to be literally true. 

4. No refusals on legitimate work.

People are shy about raising this one, so I'll just say it. Hosted models are tuned conservatively for a general consumer audience, and that has real false-positive costs for professional work — security research and penetration testing, malware analysis, clinical content, legal discovery over ugly facts, moderation tooling that has to process the material it moderates, frank assessment of weaknesses in a client's own system. Hitting a refusal mid-engagement isn't an ethics win, it's a tool that stopped working on a job someone is paying for. 

The flip side belongs in the same paragraph. Take the guardrails off and the judgment is yours. When I set up image generation on this box I checked what it would refuse to produce and the answer was basically nothing, which is exactly why I kept the image tools off any shared endpoint. Unrestricted is a reasonable default for a professional using a tool. It's not a reasonable default for an endpoint with an audience. 

How you'd decide

Everything above is a throughput-and-feel assessment. I measured tokens per second carefully and judged quality by using the thing, which isn't rigorous. If you're putting a local model in front of real work, "it feels close enough" isn't a standard — you want a small eval set, numeric pass criteria written before you test, and the smallest model that clears them. The method is in the technical post.

The rule that came out of all of this: the task shape decides, not the model. One short call with a small prompt play to local hardware's strengths. A hundred chained calls against a growing context plays to all its weaknesses — that's the 70-to-27 drop, and it's why the same box looks strong on pipeline work and mediocre in an agentic coding session. Sort your workloads that way and most of the deployment question answers itself.

Where we landed

If the driver is data privacy, this hardware is a real answer today — chat, document analysis, RAG over internal content, and the automation tier. That case doesn't need qualifying.

If the driver is token cost, probably not, and for a structural reason rather than a math one. Heavy spend is Opus-tier spend on Opus-grade work, which is the work local hardware can't take. What it can take, you could have routed to Haiku for a fraction of the price and no capital outlay. Worth measuring at team scale, but not the reason to buy. 

If you operate offline, air-gapped, or under change control, or your work keeps running into consumer-tuned refusals, this is the deployment model and cost never really enters into it. 

If you want to replace Claude Code or Codex for daily development, not yet. I'd rather say that now than after someone has bought the box. 

The summary is that this hardware doesn't remove the need for frontier models. It changes what you spend them on, and it changes where your data has to go to get work done. The second part is what clients actually care about. 

What's next

Most of what made this usable wasn't the model or the machine, it was the tooling around it. Make a Local LLM Feel Like a Frontier Model is the technical companion: the hardware findings in full, why the inference engine mattered more than the model, and the gateway we built — how you give a model whose training ended months ago current information, persistent memory, RAG, vector search, and image generation server-side, so any client pointed at the endpoint gets all of it with no configuration. That's the piece we're packaging up for clients to use.

After that, the measurement work: golden datasets and real evals for the task types we'd hand to a local model. That's what turns "this box is interesting" into "this model clears your bar on your task."

Appendix: considerations, by industry

Optional reading. If you already know your own workloads, the rule above is enough.

The pattern is the same across all of these: the local model reads, structures, classifies, and routes. It doesn't decide. It turns unstructured mess into something searchable and sorted, then hands the judgment calls to a person or a frontier model. That's why mid-tier capability is enough. Ordered by how absolute the privacy driver is. 

Legal

The strongest fit, because privilege and confidentiality aren't preferences. A lot of outside-counsel guidelines now explicitly prohibit putting matter data into third-party AI services, which ends the conversation before cost comes up.

•

First-pass discovery triage — classify documents as responsive, non-responsive, or possibly privileged, and route the maybes to an attorney. Millions of documents, each a short independent context, nobody waiting on any single result. Probably the best-shaped workload for this hardware.

•

Contract clause extraction against a playbook — find indemnity, limitation of liability, assignment, and change-of-control language, flag anything that deviates from the firm's standard positions.

•

Transcript and deposition indexing — chunk, embed, and tag by topic, speaker, and exhibit reference so associates can actually search a matter.

•

Intake categorization and conflicts pre-screening — pull party names and matter type, route to the right practice group.

Not for legal conclusions, and not for anything citation-bearing that gets filed. Fabricated citations are the known catastrophic failure here and they could cost people their licenses. Retrieval and classification, not authority.

Financial services and insurance

Privacy, plus a governance angle that quietly favors local weights: when a model is part of a regulated process, "nothing changed this quarter" needs to be literally true. A hosted model that gets re-tuned underneath you are audit problems.

•

Communications surveillance — classify email, chat, and call transcripts for escalation to compliance. High volume, runs continuously, and touches MNPI and customer PII that can't leave. This is also the billing shape where fixed-cost hardware genuinely wins.

•

KYC “Know Your Customer”/AML “Anti-Money Laundering” document intake — extract fields from IDs, statements, and utility bills, classify document type, check that a packet is complete.

•

Claims triage — first-notice-of-loss classification, complexity and severity routing, fraud indicators flagged for human referral.

•

Complaint categorization — a regulatory tracking obligation that's pure classification work.

•

Internal policy RAG — a "how do we handle this" assistant over compliance manuals and prior guidance.

Not the underwriting or credit decision, and never the adverse-action reasoning. Fair-lending explainability and model-risk governance put those out of scope. It's a pre-processor and a router.

Healthcare

HIPAA makes the privacy case on its own, and de-identification is the task that can only be done locally. 

•

De-identification of clinical notes before anything else touches them.

•

Prior-authorization packet assembly — is every required document there, does the diagnosis code match the narrative

•

Referral and patient-message triage — route by specialty and urgency.

•

Coding suggestions offered to a certified coder, never applied automatically.

•

Internal protocol and formulary RAG for staff.

Clinician in the loop, always. No diagnostic output.

Retail and e-commerce

The privacy driver is softer here, but two others take over: volume, and competitive sensitivity around pricing, margin, supplier terms, and unreleased assortment plans. Weaker case, much larger one.

•

Catalog normalization at scale — category assignment, attribute extraction from messy supplier descriptions, duplicate and variant matching. Millions of SKUs churning constantly, which is the continuous workload where metered billing hurts most.

•

Image attribute tagging — color, pattern, material, style pulled from product photos. Per-image API costs add up fast at catalog scale.

•

Review and ticket theme clustering — embeddings that surface "customers keep complaining about this" without anyone writing the query first.

•

Search query understanding over your own logs, which say more about your customers than most teams expect.

•

Supplier invoice and PO extraction.

Customer-facing chat probably still wants a frontier model, since brand voice and edge cases matter and someone's waiting on it. The backend pipeline behind it is a good fit for local. 

One thing worth planning for 

Document-heavy intake shows up in three of these four — IDs, statements, invoices, product photography — and all of it leans on the model's vision capability. That's a real reason to pick a vision-capable model even if your primary use case is text.

Meet the Author

Meet the Author

Andrew Duncan

Andrew Duncan

Director of Software Engineering

Andrew Duncan is a Director of Software Engineering in Richmond, VA. He is a driven technologist focused on modern technology stacks and best practices. Andrew believes nothing is more rewarding than making software needs a reality with a focus on flexible, scalable and supportable code.

Andrew Duncan

Andrew Duncan

Director of Software Engineering

Andrew Duncan is a Director of Software Engineering in Richmond, VA. He is a driven technologist focused on modern technology stacks and best practices. Andrew believes nothing is more rewarding than making software needs a reality with a focus on flexible, scalable and supportable code.

Andrew Duncan

Andrew Duncan

Director of Software Engineering

Andrew Duncan is a Director of Software Engineering in Richmond, VA. He is a driven technologist focused on modern technology stacks and best practices. Andrew believes nothing is more rewarding than making software needs a reality with a focus on flexible, scalable and supportable code.

Digging In

Related Insights

Loading insights…

Digging In

Related Insights

Loading insights…

Digging In

Related Insights

Loading insights…
Pick your Vibe

© 2026 UDig LLC. All rights reserved.

Pick your Vibe

© 2026 UDig LLC. All rights reserved.