Ternary LLMs Remove the Multiplier, Not the Cost

The four-hundred-megabyte model, and the one you can actually download

Every weight in Microsoft's BitNet b1.58 2B4T is −1, 0 or 1. Not a floating-point number rounded down to something small — one of three states. The model carries 2.4 billion parameters and was trained on 4 trillion tokens [2]. Stored that way, it occupies roughly 0.4 GB rather than about 2 GB, and it lands within 1–2 points of state-of-the-art full-precision models of its size on MMLU, GSM8K and HumanEval+ [3]. The idea also scales further than a demo. Running a 100-billion-parameter ternary model on a single CPU, the bitnet.cpp authors measured 7.45 tokens per second on an Apple M2 Ultra and 1.69 tokens per second on an Intel i7-13700H laptop [1].

Two years on from that result, the largest ternary model Microsoft actually ships is still the 2.4B one, joined in July 2026 by two ternary embedding models at 0.6B and 270M parameters [2]. One on-device state-of-the-union survey reports where everyone else has settled: 4-bit quantization is the standard deployment practice for on-device models, a 4x memory reduction at minimal quality loss [20].

So the demonstration is real and the adoption is not.

What actually disappears when a weight becomes −1, 0 or 1

The saving comes from a thresholding rule, and a plain one. A single threshold Δ maps every real-valued weight to one of three states — {−1, 0, +1} — according to whether it sits above +Δ, below −Δ, or inside the band between them [4]. Three states carry log₂3 ≈ 1.58 bits of information, which is where the awkward name comes from [3].

Storage is the lesser half of what changes. Multiply an activation by a weight that can only be −1, 0 or 1 and you are not really multiplying: you subtract, you skip, or you add. Ternary weight networks implement the weight-activation product as a conditional add, subtract or bypass, and the general-purpose multiplier drops out of the inner loop entirely [4]. Seen from the systems side, the hardware gain comes from substituting simple additions for multiply-accumulate, not merely from storing fewer bits [5]. The same fact is what makes circuit-level co-design worth attempting at all: ternary quantization eliminates multiplications and leaves only additions [6].

A weight set to zero is not filler but a connection switched off, so a ternary network is structurally sparse, and that sparsity is worth money on the right hardware: ternary accelerators report up to 10× inference speedup and 12× energy efficiency over conventional in-memory designs at 80% average sparsity [4]. Optimized ternary kernels on ordinary CPUs are reported to cut energy per token by as much as 82% against full precision [3].

None of which is free. A ternary model is not a compressed model but a different machine, and every bit taken out of the weights is repaid in another currency somewhere else in the stack: training tokens, kernel engineering, silicon area, model width.

The first bill: you cannot get here by compressing a finished model

The obvious way to obtain a ternary model is to take a trained one and round it. That does not work, and the point at which it stops working is well mapped. A systematic study of low-bit quantization-aware training finds 4-bit group-wise weight quantization at group size 128 to be near-lossless across both reasoning and non-reasoning tasks, while 3-bit and 2-bit quantization cause large accuracy drops, worst on reasoning [7].

Work in Nature's npj Artificial Intelligence names the shape of that failure: large language models undergo model phase transitions, with performance collapsing beyond critical compression thresholds [8]. The cause is distributional. Weights are heavy-tailed, so a quantization step wide enough to accommodate the outliers is too coarse for the dense middle, while a step fine enough for the middle clips the outliers into large individual errors [8]. Below 4 bits there is no setting of that dial that is good for both.

ENERZAi quantized Whisper Small two ways and published both numbers. Pushed to 2 bits by post-training quantization, the model's word error rate went to 37.06% [9]. Trained at 1.58 bits instead, the same model gave up 0.39 percentage points of word error rate against FP16, at an 8× compression ratio [9]. Those are the vendor's own measurements rather than an independent replication, and the gap between them is two orders of magnitude wide.

Which means ternary is not a deployment setting. It is a pretraining decision, and pretraining decisions are billed in tokens. BitNet's ternary models are trained from scratch on 4 trillion tokens, and BitCPM4 spends 100 billion tokens even when it starts from an already-pretrained model [10].

Why that bill is so large: the zero that traps the gradient

Tequila names the mechanism: deadzone trapping [10]. During quantization-aware training a large number of weights become stuck at the boundary of the zero region, where they receive no informative gradient. They stop learning. Convergence stalls, and the only remedy anyone has found is to keep feeding the model tokens until enough of the trapped weights work loose — which is why the token bill runs to hundreds of billions.

The zero that gave ternary its expressive edge over binary is the same zero that traps the optimizer. One feature, one mechanism, one pathology.

Tequila's fix follows from the diagnosis. It splits the deadzone into two distinct minima, 0− and 0+, reactivating dead weights as learnable dynamic biases that keep receiving gradient without adding inference-time complexity [10]. At group size 128, that brings LLaMA-3.2-1B and LLaMA-3.2-3B to competitive quality on 10 billion tokens of quantization-aware training rather than hundreds of billions [10].

CAT-Q attacks the same bill from the other end, by trying to avoid retraining altogether. It treats the scaling factor α and the threshold Δ as learnable parameters optimized during calibration, and replaces hard ternarization with a softened transition function built on a hyperbolic tangent. That makes the whole thing differentiable enough to optimize inside a post-training pipeline, for what CAT-Q states as more than 10× memory reduction relative to FP16 [11].

Both papers shrink the bill. Neither makes it disappear.

The second bill: the kernel is the product

A trained ternary model dropped into a stock transformer stack buys almost nothing. The format is not the speedup; the kernel is.

bitnet.cpp, built on llama.cpp, supplies the missing half — a mixed-precision GEMM library with ternary lookup-table kernels and an Int2-with-scale path — and reaches up to 6.25x over full-precision baselines and up to 2.32x over other low-bit baselines [1]. Quality holds while it does: on WikiText2, HellaSwag and WinoGrande the lookup-table kernels track Float16 almost exactly, and the Int2-with-scale variant is lossless [1].

The reason the kernel wins is not that it performs less arithmetic. Decoding is memory-bound, not compute-bound. A 2026 study of energy in LLM serving finds the decode phase fully memory-bound at every model size and sequence length tested, with energy per generated token largely invariant across float32, float16 and bfloat16 [12]. The same study measures up to 4× lower GPU energy from dropping precision during prefill, where the work genuinely is arithmetic [12]. Changing the numeric format without changing the bytes moved buys nothing at decode. Ternary wins because it moves far fewer bytes: constraining weights to −1, 0 and 1 gives 8× memory compression while holding 93–99% of full-precision accuracy [13].

This second bill is also paid unevenly, and the vendor says so. Microsoft's own README documents optimized kernels on x86 CPUs and limited ARM support, with the flagship ternary model reaching up to 6.17x on x86 inference against a full-precision baseline [2]. Platform coverage is itself a cost, and somebody has to pay it before the compression ratio means anything on your machine.

Where the second bill goes next: down into silicon

Even the lookup table is memory-bound. T-SAR finds that LUT-based GEMM and GEMV are bottlenecked on memory requests rather than compute, and fixes it by generating the tables in registers instead of fetching them. Memory request volume falls by 8.7–13.8× against a TL2 lookup-table baseline, and geometric-mean prefill speedups reach 8.8× on a workstation, 8.4× on a laptop and 12.4× on mobile, across models from 125M to 100B parameters [13]. The spread across machines matters less than the fact that it holds everywhere: the win comes from not moving the table.

Past that, the cost leaves software altogether. TOM stores an entire ternary model in on-chip read-only memory, sized against an edge target of 1000 tokens per second, and notes why the approach was previously impractical: an existing ROM-based accelerator spends roughly 300 mm² of chip area on 1.86GB of ROM [6]. BitROM makes the enabling condition explicit. A conventional compute-in-ROM design holding LLaMA-7B would need over 1,000 cm² of 14 nm silicon, about 273 times the area of ResNet-56 [14]. Compute-in-ROM was a dead end for language models until ternary weights shrank the stored model enough to fit — the architecture exists because the format exists.

Every one of these advances pays down the throughput bill. None of them touches what the format costs the model itself.

The third bill: the one no kernel pays down

That cost takes three forms.

The first is that reasoning is the sensitive axis. The same low-bit QAT study finds degradation under extreme low-bit post-training quantization concentrated on reasoning benchmarks rather than general tasks [7]. The intuition is mechanical: error compounds along a chain, so a single-step lookup absorbs a rounding error that a twenty-step derivation amplifies.

The second is that perplexity is not a safety metric. Work on alignment-aware quantization shows quantized models reverting to unsafe completions while perplexity stays low, and states the general form of the problem: optimizing for utility or statistical fidelity does not guarantee that alignment survives the compression [15]. That indicts the standard acceptance test. Check perplexity, check a benchmark suite, ship — that procedure is not measuring the property most likely to have broken.

The third is that some of the bits come back as width. In small language models, 1.58-bit quantization is reported to lift perplexity from 77.8 at 16-bit precision to 130.1 at a hidden size of 32; doubling the hidden size closes the gap, to 36.3 against a 16-bit baseline of 36.7 [16]. Spending parameters to buy back precision is a real cost and the compression ratio hides it completely. It also makes ternary a scale-dependent bet rather than a universal one, since the reported gap against full precision narrows as models grow [9].

Who can afford this, and who ternary is really competing with

A scale-dependent bet is a bet against something. Ternary's competitor was never FP16.

It is NVFP4: a 4-bit floating-point format with one sign bit, two exponent bits and one mantissa bit, giving fifteen distinct values [17]. Every block of 16 values carries a dynamically computed FP8 scale, where the earlier MXFP4 attaches a coarse power-of-two scale to every block of 32 [17]. NVIDIA's own report puts NVFP4 at half the memory of FP8 with a 2–3x boost in arithmetic performance, recovering near-BF16 accuracy on post-trained models through quantization-aware distillation [18]. But the decisive property is not the bit count. It is that Blackwell tensor cores execute the format natively across training and inference, at close to 16-bit accuracy [17]. Where the hardware already accelerates your format, there is no reason to remove the multiplier.

The split shows up in what practitioners report. A long Hugging Face forum thread on integer-only inference concludes that a fully integer path — the entire forward graph, normalization and softmax included, with no floating-point fallback — is not broadly production-ready in mainstream libraries, and that the datacenter mainstream is likely to be FP8 plus FP4-family microscaled formats, because that is what server hardware accelerates well [19]. The same thread locates the value of integer-native execution elsewhere: edge devices, CPUs, older NPUs, DSP-like accelerators [19]. That is precisely the constituency bitnet.cpp was built for.

There is one 2026 release that cuts against all of this, and everything known about it comes from the vendor. In April 2026 PrismML released Ternary Bonsai, an open-weight ternary family at 8B, 4B and 1.7B parameters, with the 8B model reported at 2.15 GiB packed against 16.38 GB in FP16 and a self-reported average of 75.5 across six benchmark categories [21]. A third-party runtime shipped support for it the next day [22]. Ternary can ship, and the tooling around it can move fast.

What would settle whether it ships well is a comparison nobody has published: an independent, matched-memory head-to-head of a ternary model against a 4-bit post-training-quantized model above 8B parameters. Every strong ternary result above is either first-party — Microsoft's own 2.4B model and its own framework [2] — or vendor-reported, like PrismML's self-measured family [21]. The correlation between ternary weights and small deployed models is thoroughly documented; the causal claim that ternary cannot scale is not, and nothing here tests it.

So the honest verdict is narrower than the question usually asked, and more interesting. Ternary quantization holds up. It holds up at the scales where it has been measured, on the platforms where someone bothered to write the kernels, at a price that moves rather than disappears.