Self-hosting Kimi on consumer GPUs: an honest reality check

Kimi's selling point is long context and long-horizon agent work. People read that, look at their 24 GB GPU, and assume a quantised build will fit. Usually it will not — and the reason is not parameter count.

The weights are the small part

A mixture-of-experts model activates a fraction of its parameters per token, but all of the experts must be resident in memory. A 4-bit quantisation of a large MoE flagship lands in the tens to low hundreds of gigabytes depending on the release. That is multi-GPU territory, not a consumer card.

The distilled dense variants are the ones that fit. They are also not the same model: you get the family's style and instruction tuning, not its long-context capability.

The KV cache is the bill

This is the part people underestimate. KV cache scales with:

bytes ≈ 2 × layers × kv_heads × head_dim × tokens × bytes_per_element

At 1M tokens, even with grouped-query attention, a single long session can consume more memory than the weights. Serving a 128k-token request is already a different hardware class than serving 8k.

Practical consequences:

• 1M context on consumer hardware is not a thing. Even if weights fit, the cache does not.
• Long context and concurrency trade off directly. Two simultaneous 100k sessions double the cache.
• Quantising the KV cache (FP8) is now common in vLLM and SGLang, and it is the only reason long-context self-hosting is viable at all.

If your real requirement is "read a 200-page document", chunking plus retrieval is cheaper and often more accurate than brute-forcing the context window.

What actually fits, by budget

The configuration that works

If you are determined to self-host a Kimi-class model, the setup that survives contact with reality looks like this:

vllm serve <kimi-open-weights-repo> \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.92

• Two GPUs minimum for anything in the flagship class.
• FP8 KV cache is not optional at this context length.
• Prefix caching is what makes repeated document queries affordable.
• Cap max-model-len to what you actually use. Reserve the rest as headroom.

The cheaper answer

For most teams, the honest recommendation is: use the hosted long-context model for the 5% of requests that need it, and a local mid-size model for everything else. Long-context requests are rare and expensive; paying per token for them is usually cheaper than owning the hardware that can serve them.

Check current per-million pricing on the Kimi model pages before you commit to a build. If your monthly long-context spend is under the cost of a second GPU, the answer is already decided.