The problem isn’t compute, it’s memory
Frontier models of this generation are almost all Mixture of Experts (MoE). DeepSeek-V3 and DeepSeek-R1 activate about 37 billion parameters per token, out of the 671 billion they total. Llama 4 Scout activates 17B of its 109B total, and Llama 4 Maverick activates 17B of 400B. Per token, the compute cost is equivalent to that of a medium-sized model, which explains why they generate text at reasonable speeds in a data center.
Memory works differently: the full weights must be resident, whether you query them or not. An expert that is not activated in this turn still occupies its spot in VRAM. The sparsity of an MoE saves flops, not gigabytes.
The arithmetic makes it clear. Llama 3.1 405B in weights alone occupies about 810 GB in 16-bit, more than the 640 GB provided by the eight 80 GB GPUs of a standard node. DeepSeek published its model trained directly in FP8, and even then, the weights are around 670 GB.
Before looking at performance benchmarks, the deciding figure is how much resident memory the full model requires and how fast your hardware can read it. Compute is no longer the bottleneck in most of these systems: memory is.
VRAM arithmetic: weights, KV cache, and context
The basic calculation is simple. In FP16, 2 bytes per parameter; in 8-bit, 1 byte; in 4-bit, about 0.5 bytes plus some overhead for embeddings and normalizations. Scout, with its 109B total, needs more than 55 GB for weights alone in 4-bit, meaning it won’t fit even in a 32 GB RTX 5090. Maverick, in 4-bit, goes up to about 200 GB and is out of reach for any desktop.
The second item is the KV cache, which grows linearly with context length. Llama 3 70B, with its 80 layers and 8 KV head groups, spends about 320 KB per token in FP16; with a 128K context, that’s over 40 GB—more than its own weights quantized to 4-bit. If your application maintains long conversations or serves several users in parallel, the cache can end up weighing more than the model.
Meta boasted that Scout supports 10 million context tokens. That number is for exhibition and is served from data center infrastructure; on a workstation, the practical ceiling arrives much sooner, because every additional context token costs permanent memory throughout the session.
When comparing configurations, calculate weights plus KV cache with your actual usage lengths, not the 4,096 tokens of a demo benchmark. If you want to see which models fit in a specific GPU according to these criteria, we published a guide to local LLM models that explores VRAM ranges and use cases.
Llama 4 and DeepSeek: the price of entry
The hardware floor is set by the model creators themselves. In the April 2025 announcement, Meta made it clear that Scout fits in a single 80 GB H100 with Int4, and that Maverick also fits in a single H100 with Int4, although in BF16 it requires a host of eight H100s. At market prices, each H100 is around 25,000 or 30,000 dollars, so Scout with Int4 costs the price of a car, and a Maverick node at BF16 approaches a quarter million.
DeepSeek-R1, for its part, deploys comfortably on nodes of eight H200 GPUs with 141 GB of HBM3e. The known home-brew route is the 1.58-bit dynamic quantization published by Unsloth, which brings R1 down to about 131 GB and makes it executable with around 200 GB combining VRAM and RAM. According to Unsloth’s own tests, the model remains functional at that size, but you still need a server machine, not a desktop computer.
The no-purchase alternative is hourly rental. In specialized clouds like Lambda or RunPod, an H100 rents for around 2 or 3 dollars per hour; a full-time month is, rounding off, about 1,800 dollars. Owning the hardware only wins this round if you keep the GPUs saturated, and keeping eight GPUs saturated is literally called operating a small data center.
The pattern is consistent: the frontier is viable by the hour or by billed tokens, not as a home appliance. Whoever buys hardware for frontier models is buying a small datacenter, with all the costs that entails.
Quantizing to fit: the real toll
In dense models between 7B and 70B, dropping to 4-bit (Q4_K_M in GGUF, or AWQ and GPTQ variants) barely costs quality in general tasks, and this is what allows an 8B model to run on a 16 GB laptop. In that range, quantization is a net gain and actually sustains almost the entire current local ecosystem: Ollama, llama.cpp, and LM Studio operate at those depths.
Below 4-bit, the terrain gets messy. 3-bit and 2-bit quants degrade visibly, and in giant MoEs, the margin is even thinner because each expert is queried infrequently, and quantization noise is not compensated for throughout the inference. Unsloth’s idea addresses exactly this: preserving more bits in attention and shared experts, and compressing only the routed experts, which make up the bulk of the file.
Long-reasoning models add another unpleasant effect. R1 generates chains of thousands of tokens before giving the answer, and each token is conditioned on all previous ones, so a small bias introduced by imprecise bits can derail the entire chain. The longer the derivation, the more expensive the initial error.
The sensible escape route is distilled models. In January 2025, DeepSeek published six versions of R1, based on Qwen from 1.5B to 32B and Llama from 8B to 70B. Distill-Qwen-32B in 4-bit occupies about 20 GB, fits in a single 4090 or 5090, and retains much of the original’s reasoning style. The rule of thumb: if you need to quantize to 2-bit to make the giant fit, the problem isn’t your quantizer—it’s that you chose the wrong model for your hardware.
The full bill: hardware, power, and obsolescence
Electricity adds up quickly. A system with two high-end GPUs consumes around 1 kW under load; at 0.20 euros per kWh and continuous use, power is about 145 euros a month before counting cooling or the price of silence in your office. Amortization weighs more: 300,000 dollars for a node spread over three years is about 270 dollars a day, whether the days are spent under load or not.
Meanwhile, frontier APIs have continuously become cheaper. GPT-4o costs 2.50 dollars per million input tokens and 10 for output; o1 is 15 and 60; Anthropic’s Sonnets are 3 and 15; and DeepSeek published rates for R1 at 0.55 and 2.19 dollars respectively, with heavy discounts for cached input. With prices like R1’s, a few dozen million tokens per month bill for dozens of dollars, a figure against which any home hardware loses by a landslide if the criterion is the bottom line.
There is the elegant Apple route: the Mac Studio with the M3 Ultra reaches up to 192 GB of unified memory with 819 GB/s bandwidth, enough to load R1 with low quantization at chat speed. The max-memory configuration is around 9,500 dollars. It is the best "desktop capable of the impossible," and even so, it competes against a cheap VPS plus an API that bills cents.
Then there is obsolescence, which in this cycle moves by semesters. R1 overwhelmed everyone in January 2025, and in April Llama 4 arrived with a new architecture; the specific weight of a frontier model dilutes quickly. The hardware you buy today will be amortized far from the frontier—that is its natural destiny.
Local hardware almost never wins on price against the frontier API; it wins in other leagues, such as privacy, controlled latency, or offline availability. If you buy, let it be for one of those reasons, because the economic reason has already been decided against you.
What does actually pay off locally
Between 16 and 48 GB of VRAM, there is excellent life. Llama 3.1 8B, Qwen2.5 in its 14B to 32B sizes, Mistral Small (24B), and R1 distillations comfortably cover data extraction, summarization, technical writing, classification, and everyday reasoning in 4-bit quants. There, the purchase does pay for itself, especially if the machine was already amortized.
If your motivation is confidentiality and proprietary knowledge, the answer also doesn’t involve a 671B in the closet, but rather a medium model tied to your documents. A private RAG with a 32B that never leaves your network provides more value per euro than the largest of models running at 2-bit. We developed the full case for firms and consultancies in this article on local vertical AI and private RAG.
For a permanent and lightweight service, a modest VPS with Ollama and a small model handles bots and assistants for cents a month. The pattern that repeats in all deployments that make sense: the model is the smallest one that solves the task, never the largest that fits in your budget.
How to decide without burning your budget
The decision rule fits in two lines. If the task is solved with quality by a model up to 32B quantized to 4-bit, local wins: almost zero marginal cost, data at home, zero variable bills. If the task truly demands a frontier model, the API or hourly rental wins in 99% of cases, and the hybrid approach (sensitive tasks locally, frontier tasks in the cloud, with explicit routing) is the default design.
To decide with data, run this test in one afternoon. Take 50 real prompts that you currently send to the API, run them through a Distill-Qwen-32B or a Qwen2.5-32B in Q4 with Ollama on your GPU, and note the tokens per second, the successes you can judge yourself, and the measured consumption. Repeat the same 50 against the frontier model you use and note the projected monthly bill with your real volume.
With those two columns on the table, the result tends to be uncomfortable but clear: the difference in quality rarely pays for the difference in cost, and when it does, it’s better to rent an H100 for a few days than to buy one. Local shines where its conditions are met, and the frontier shines precisely where the price is based on usage.