Infrastructure

Using Kimi K3 (a 2.8T MoE model) and its vLLM deployment recipe as a case study, this article breaks down what Hardware, Strategy, KV Offload, P/D Disaggregation, and other parameters actually control — their real-world effects, applicable scenarios, and common misconceptions.

The same LLM can demand over 100 GB of VRAM during training yet run inference on a single 24 GB GPU. The difference lies not just in compute, but in what data sits in memory and how fast it reaches the GPU. Starting from HBM, host memory, and SSDs, this piece unpacks the mechanics behind training, inference, and cache-based pricing.