What Kimi K3's vLLM Configuration Tells Us — What Those Parameters Actually Control

Open Kimi K3's vLLM deployment page, and you'll see a row of options: Hardware, MXFP4, TP, EP, KV Offload, P/D Disaggregation, and more. Just from the names, it's hard to tell what each one actually controls.

These parameters don't change Kimi K3's core capabilities. But they do determine: whether the model can fit into a GPU cluster at all, how the model and incoming requests get distributed across GPUs and nodes, and whether the deployment ultimately favors low latency or high-concurrency throughput.

Kimi K3 is a 2.8T-parameter MoE (Mixture of Experts) model. At this scale, memory, communication, and scheduling all loom larger in how configuration choices play out. Below, using the official recipe as our guide, we'll go through this vLLM configuration parameter by parameter.

The Model's "Weight": Why 1680GB

Kimi K3 uses a MoE architecture with native support for both vision and text. The model has 896 routed experts in total, with 16 activated per token, plus a shared expert.

The phrase "only 16 experts activated per token" easily creates the misconception that the model's memory footprint shrinks proportionally. In reality, while each token only invokes 16 experts during inference, all 896 expert weights must still be loaded into the GPU cluster at deployment time. Sparse activation reduces compute — not memory requirements. The hardware threshold doesn't come down because of it.

K3 currently ships a single model variant: MXFP4 weights, MXFP8 activations. This is not the more common post-training quantization (PTQ) applied at deployment time, but rather quantization-aware training (QAT) that directly produces a 4-bit checkpoint.

The page estimates weight occupancy at 1680GB:

2.8T × 0.5 byte/param × 1.2 headroom ≈ 1680GB

1680GB is only the bare minimum for loading weights. When the model actually runs, additional space must be reserved for KV Cache, communication buffers, CUDA Graph, the vision encoder, and potentially an enabled speculative draft model.

Hardware: 9 Options, Only 3 Viable on a Single Node

The vLLM page offers 9 hardware options.

NVIDIA:

AMD:

Most hardware options on the page assume 8-GPU nodes, with GB200 and GB300 as the exceptions. These use 4-GPU NVL4 units, where four GPUs are interconnected within a single module via NVLink, hence the "4×" rather than "8×" in the config column.

Roughly calculating against the 1680GB weight footprint, only B300, MI325X, and MI355X exceed this threshold with a single node's VRAM. All other hardware requires multiple nodes or multiple NVL4 units.

When interpreting Hardware parameters, it's important to distinguish between what a config option expresses and what the page has actually verified.

The first nuance: node count feeds into the generated launch command, but the frontend does not re-validate total VRAM for every multi-node combination. For example, 2×H100 yields 1280GB total, and 2×GB200 NVL4 yields 1536GB — neither can hold the weights, yet the page may still generate a launch command. So "Hardware × node count" expresses a cluster combination, not a guarantee that the combination has passed capacity checks.

The second nuance: the page's default selection only reflects the frontend's initial state — it's not equivalent to the recipe's recommended configuration. The recipe YAML specifies default_hardware: b300, but the frontend code doesn't read this field. If the URL carries no parameters and the browser has no relevant cache, the page defaults to H200 × 2 with Multi-Node TP. When interpreting these configurations, keep "what the frontend currently shows" separate from "what the recipe has verified"; the verified marker indicates whether a given configuration has been recipe-verified.

Hardware is also more than just a VRAM capacity choice — it determines the underlying software stack and execution path. NVIDIA uses CUDA, AMD uses ROCm, and the corresponding Docker images, inference kernels, and speculative decoding backends all differ. For example, Hopper (H100/H200) mandates the marlin MoE backend, Blackwell mandates TRT-LLM Ragged MLA, and AMD uses the AITER path. These are all recipe-level hardware overrides that users cannot modify. In other words, switching Hardware changes the underlying backend along with it — not just the VRAM numbers.

Strategy: How Model and Requests Are Distributed Across GPUs

Strategy addresses two layers of distribution: first, how to split a single large model across multiple GPUs — model parallelism; second, how to distribute multiple requests across different GPUs — data parallelism.

To build intuition, picture a GPU cluster as a restaurant kitchen: each request is an order ticket, and each GPU is a chef or a workstation. Model parallelism is multiple chefs collaborating on the same order; data parallelism is multiple teams of chefs handling different orders simultaneously.

The K3 recipe supports five strategies at the bottom layer, each with a floor — a minimum GPU count requirement. Think of the floor as the minimum number of chefs needed for that division of labor: if the user's GPU count falls short, the generator automatically raises it to a valid value.

TP (Tensor Parallel): Layer-by-Layer Model Splitting

TP splits every layer of the model across multiple GPUs, with the same batch of requests computed jointly by the entire TP group. In the kitchen analogy, it's like a single dish always being prepared by the same team of chefs: each station handles only part of the current step, and results must be combined before moving to the next.

With single-node TP, all communication stays within one machine — like chefs all in the same kitchen, with the shortest communication paths. Multi-node TP extends a single TP group across multiple machines — like a single order being fulfilled by chefs spread across different kitchens. For example, two 8-GPU servers correspond to TP=16. The head node serves the HTTP API, while the other nodes participate in compute only via --headless.

Every layer in cross-node TP requires synchronization, so high-speed interconnects like InfiniBand/RDMA are essentially mandatory. Plain Ethernet isn't impossible to run on, but it's like every step requiring a slow pass between kitchens — latency accumulates, and performance degrades noticeably.

TEP (Tensor + Expert Parallel): Further Splitting the MoE Layers

TEP builds on TP by additionally applying EP (Expert Parallel) at the MoE layers, distributing experts across different ranks. Think of MoE experts as different stations in the kitchen — say, hot dishes, cold dishes, and desserts. Non-expert components like attention continue using TP, while expert weights and computation are parallelized via EP. When a token is routed to a particular expert, the corresponding data must also be sent to that expert's GPU — like an order ticket being directed to the appropriate station based on dish type.

TEP involves both TP synchronization and MoE all-to-all communication — akin to frequent order and material handoffs between general-purpose stations and specialized expert stations — making it more sensitive to interconnect bandwidth and load balancing.

DEP (Data + Expert Parallel): Optimizing for High-Concurrency Throughput

In the multi-node DEP configuration currently generated for K3, TP=1. Each GPU maps to one local DP rank, the cluster's total GPU count is the DP size, and --enable-expert-parallel distributes experts across different GPUs. In the kitchen, DEP is more like opening multiple serving lines simultaneously: different DP ranks handle different orders independently, with expert-processing portions routed to the specialized stations distributed across GPUs.

For example, two 8-GPU nodes generate:

--data-parallel-size 16
--data-parallel-size-local 8
--data-parallel-hybrid-lb
--enable-expert-parallel

DEP allows different DP ranks to process different requests concurrently, making it better suited for high-concurrency throughput. One thing to note: the current recipe does not have each node first form a local TP+EP group, nor does it configure intra-node TP for multi-node DEP. Returning to the kitchen analogy, each node is not a complete kitchen capable of independently handling every step.

Additionally, when using DEP on 8-GPU nodes, the floor is 2 nodes — meaning at least 16 GPUs; for 4-GPU nodes (GB200/GB300), at least 4 nodes are required. This constraint should be factored into hardware selection planning.

P/D Disaggregation: Splitting Inference into Two Stages

Prefill/Decode Disaggregation splits a single inference into two GPU pools. In the kitchen, this is like separating prep and plating into two teams: the Prefill team reads the entire order at once and prepares the materials and records needed downstream; the Decode team takes over these prepared results and continuously produces output.

• Prefill: Reads the full prompt in one pass, processes the context, and generates KV Cache. It primarily affects TTFT (Time to First Token) — how long you wait after sending a request before seeing the first token.
• Decode: Generates tokens one by one based on the existing KV Cache. This stage is typically more memory-bandwidth-bound and primarily affects ITL (Inter-Token Latency) — the generation speed of subsequent tokens after the first one appears.

The request path looks roughly like this:

Request → Prefill pool generates KV Cache
       → KV Cache transferred via NIXL side channel
       → Decode pool continuously outputs

What travels over the NIXL side channel is the KV Cache corresponding to the request — not the model weights. In kitchen terms, it's passing the prepared materials and order records, not moving the entire kitchen setup.

Both Prefill and Decode can independently choose TP, TEP, or DEP, with 1 to 16 nodes configurable for each. The K3 recipe defaults to TEP for Prefill and DEP for Decode. The two roles use separate HTTP ports (default 8001/8002), NIXL side-channel ports (5557/5558), and cross-node coordination addresses, with a Router directing requests to the appropriate endpoint. Think of the Router as a dispatcher that routes order tickets to the prep team and then to the plating team in sequence.

The primary value of P/D Disaggregation is that it lets the two workload types scale independently. This way, complex orders requiring long prompt processing are less likely to clog the Decode team that's continuously serving, and resources can be tuned separately for TTFT and ITL. The cost is equally direct: you need an extra GPU pool and introduce more complex Router and KV transfer topologies.

KV handoff itself also consumes time and network bandwidth. So P/D Disaggregation provides resource isolation and independent scaling capability — it does not guarantee improved total throughput under all workloads. If KV Cache transfer itself becomes the bottleneck, throughput may actually drop.

KV Offload: It Moves KV Cache, Not Model Weights

KV Offload addresses KV Cache capacity and reuse — it cannot solve the 2.8T model weight capacity problem.

Model weights remain essentially resident in GPU memory throughout the serving lifetime. Continuing the kitchen analogy, they're like the full set of equipment and the complete recipe book that the kitchen must have on hand at all times. KV Cache, by contrast, is the prep work and temporary records generated while processing each order: the more complex the orders and the more orders being processed simultaneously, the more counter and storage space they consume.

The page lists four options:

• Off: KV Cache stays in GPU HBM (High Bandwidth Memory) — like keeping all prep work right on the counter at your fingertips, with the shortest access path.
• Simple: Offloads part of the KV Cache to local CPU memory — like temporarily moving items to the kitchen storage area, retrieving them back to the counter when needed.
• Mooncake: Uses external KV Cache storage and transfer infrastructure — like tapping into warehousing and distribution systems beyond the kitchen itself.
• LMCache: Uses a standalone caching component to store, retrieve, and reuse KV Cache across CPU, disk, or nodes — like using an independent inventory system to manage and reuse prep work.

Not all strategies support the same offload methods. The current actual support matrix is:

Mooncake is marked as unsupported across all 9 hardware options for K3. Enabling Simple also reserves roughly 220 GiB of CPU memory per rank, so you can't just consider the GPU-side benefit.

Offloading is best suited for two scenarios: first, when KV Cache capacity runs short — the prep counter can't hold any more; second, when many requests reuse the same system prompts, document prefixes, or agent contexts — like a batch of orders sharing the same base prep work. But if cache hit rates are low and KV Cache must frequently traverse PCIe or the network, it becomes a case of materials constantly shuttling between counter and warehouse — both latency and operational overhead rise.

The single most common misconception: enabling KV Offload still won't let a single 8×H100 node fit Kimi K3. Offload moves the prep work and temporary records generated per order — not the permanent equipment and complete recipe book that the kitchen must always have. In model terms, what gets offloaded is KV Cache, not the 2.8T parameter weights; model weights must still be accommodated by the GPU cluster.

Features and Advanced: The Override Order You Can Easily Miss

The page also provides four Feature toggles and several Advanced tuning knobs. The most important thing here is the parameter composition override order:

base → variant → strategy → hardware → Advanced → Features → KV Offload

Think of this chain as an order ticket getting annotated over and over: the further right a configuration sits, the later it's written. If a later note modifies the same setting, the later-written value wins. So Advanced overrides the fine-tuned values from hardware and strategy, and Features in turn override Advanced.

Features

• Tool Calling: Enables the API to accept a tools parameter and uses the Kimi K3 parser to structure outputs as tool calls. K3 occasionally produces output formats the parser can't handle correctly, so production deployments should add schema validation and retry logic.
• Reasoning: Uses the kimi_k3 reasoning parser to separate reasoning content from the final answer. It only changes the output structure — it doesn't enhance model capability.
• Spec Decoding: Loads Inferact/Kimi-K3-DSpark as an additional draft model. Each round proposes up to 7 candidate tokens, which K3 then verifies in batch. NVIDIA uses the FLASHINFER_MLA backend; AMD uses TRITON_MLA. This feature suits low-latency, small-batch scenarios, but the draft model consumes extra VRAM; on NVIDIA, max-num-seqs is also capped at 32. Under high concurrency, this constraint may offset the speedup.
• Text Only: Adds --language-model-only, skipping the vision encoder. This reduces resource usage and startup overhead, but the service will no longer accept image inputs.

Advanced: The Override Pitfall

Advanced provides four general-purpose tuning knobs. When manually enabled, they override the fine-tuned values from all preceding layers:

• max-num-batched-tokens=8192: Overrides PD Prefill's 16384, PD Decode's 32, and Blackwell's 32768 — effectively replacing three scenario-tuned values with a single generic one.
• max-num-seqs=256: Overrides AMD's default of 128 and PD Decode's 32. But if Spec Decoding is also enabled, the Feature layer will then override this with NVIDIA 32 or AMD 128. In that case, the override chain becomes Advanced → Features.
• gpu-memory-utilization=0.95: K3's base configuration already uses 0.95, so toggling this typically won't change the final command, but it does produce a different UI state.
• max-model-len=auto: Overrides Blackwell's default 1M context window limit, letting vLLM auto-determine the cap based on available KV Cache.

This override order means that between the toggle states you see in the UI and the final command, multiple parameter substitutions may have already occurred. When checking a configuration, use the composed argv and environment variables as the source of truth — don't rely solely on which pills are highlighted on the page.

After generating the command, also verify two things: whether contradictory boolean flags appear simultaneously (e.g., --enable-prefix-caching alongside --no-enable-prefix-caching); and whether any value was silently overridden by a later configuration layer.

Closing Thoughts

The vLLM recipe covers the primary configuration dimensions — hardware, parallel strategies, inference stage splitting, and cache management. Working through this configuration page, you can connect several core concepts in model loading, parallel computation, and inference scheduling:

• Hardware and precision formats determine how model weights are stored and loaded into the cluster. MoE's sparse activation reduces compute per token, not the total weights that must be loaded;
• TP, TEP, and DEP show how model computation, experts, and requests are distributed across GPUs, and surface the associated communication overhead and load-balancing concerns;
• P/D Disaggregation splits inference into Prefill and Decode stages, helping distinguish TTFT from ITL and understand why KV Cache must be transferred between the two resource pools;
• KV Offload draws a clear boundary between persistent model weights and per-request intermediate state, and clarifies the distinct roles of GPU HBM, CPU memory, and external caches;
• Features and Advanced demonstrate how the final command is composed from multiple configuration layers. The toggles on the page are inputs — what actually takes effect is the final generated argv and environment variables.

K3's checkpoint, Docker images, and specific commands will continue to evolve, but the questions behind the parameters remain relatively stable: how weights are loaded, how computation and requests are distributed, how inference state flows, and how the final configuration is assembled. Understanding these relationships and concepts makes it easier to reason about model deployment and inference.