Find the request volume where renting or buying GPUs beats paying per token — including the MLOps overhead most comparisons leave out. From Appendix G of Production LLM Architecture.
Each scenario pre-fills every field with defensible defaults. Adjust your volume below; everything else is optional.
Collapsed on purpose. A scenario above plus your volume is enough for a verdict; open this only to override a specific assumption.
The Mid tier is both a mid-range frontier API and hosted open-weights pricing — hosted Kimi K3 is $3/M input, $15/M output, $0.30/M cached, the same numbers. For most teams the real alternative to owning a rack is renting someone else's inference of the same open model, not a frontier API.
405B assumes int8 (fp16 weights alone would be ~810GB, more than 8x80GB=640GB fits) — flagged here because the dtype assumption changes the math, not because it's unusual practice.
2.8T MoE preset reflects Kimi K3 (Moonshot AI; open weights and technical report both released July 2026, arXiv 2607.24653). 104B of 2.8T params active per token — 16 of 896 routed experts (2.8T ÷ 896 ≈ 3.1B/expert × 16 ≈ 50B) plus a ~54B always-active shared trunk (2 shared experts, attention, router and embeddings), summing to 104B. That shared-trunk share is unusually high but in family with K2's lineage: K2 runs 32B active of 1T (3.2%), K3's 104/2800 is 3.7% — both a step above the ~1.8% a bare 50B-active figure (also circulating for K3) would imply. Weights ship MXFP4 with MXFP8 activations, quantisation-aware from the SFT stage onward, so 4-bit is the trained format rather than a post-hoc quantisation: self-hosted quality matches the hosted API. MXFP4 carries an 8-bit E8M0 block scale per 32 elements, so 4.25 bits/element amortised — 2.8T × 4.25 ÷ 8 = 1.49 TB, not the 1.4 TB commonly quoted from a bare 4-bit assumption. That is a floor; non-expert tensors ship at higher precision and the released checkpoint measures ~1.56 TB, which is this preset's default. Memory is now modelled as hybrid attention: K3 interleaves 3 Kimi Delta Attention layers (fixed recurrent state, context-independent) with 1 Gated MLA layer, plus one extra MLA layer at the end of the backbone — 24 full-attention layers of 93, so f = 0.258. The KV and state fields are dense-equivalent (the cost if every layer were that type); at f = 0.258 they resolve to ~27.6 KB/token and ~217 MB/stream. Both assume BF16 KV — halve them for FP8.
H100 and H200 have no native microscaling tensor cores, so MXFP4 kernels upcast to BF16 to compute. You keep the 4x memory saving and lose part of the bandwidth and compute benefit — the software fallback reaches the FP8 rung of the FP4/FP8/BF16 4x/2x/1x throughput staircase. Blackwell B-series and MI400 execute MXFP4 natively. Preset accordingly.
Workload-dependent defaults — real throughput depends on batching, framework and traffic shape, so adjust to your own benchmarks. Each preset's figure is now checked against the two hardware ceilings above at its own batch size, so no preset claims a throughput its silicon cannot reach. That check is why the 70B preset is 4x A100-80 at batch 40 rather than 2x at batch 10: below batch 35, 500 tok/s/GPU is arithmetically impossible for a 140 GB model.
Self-hosted cost is fixed (GPU + ops); API cost scales linearly with volume. The crossover point is your breakeven.
This is Appendix G. The book has 61 more tables like this — plus the GPU capacity planning (Appendix H) and reference architecture (Appendix I) that turn this estimate into a deployable system.
Buy on Amazon →