Deploy Kimi K3 on GPU Cloud: Cluster Sizing & Setup (2026)

Moonshot AI published Kimi K3's open weights at 00:00 UTC on July 27, 2026, hitting the date it had set for itself (late evening of July 26 in US time zones, which is why both dates circulate). The model card repeats the line every giant open MoE release ships with: run it on a "supernode configuration with 64 or more accelerators." That's not a cluster-sizing plan, it's a footnote. Here's the actual arithmetic: 2.8 trillion total parameters, 104B active per token, a published checkpoint that is 1.56TB rather than the 1.4TB the obvious calculation gives you, and the concrete H100, B300, B200, and GB200 NVL72 configs that get you from zero to a serving endpoint.

Two numbers do most of the work in this guide. The checkpoint is 1,560.94GB across 96 shards, which is about 160GB more than 2.8T parameters at 4 bits predicts, and enough to make an 8-GPU B200 node fail to load the model at all. And the minimum Hopper configuration is four 8-GPU H100 nodes, 32 GPUs, the one SGLang documents in its cookbook and the most widely available way to get K3 running. The step-by-step bring-up for that cluster is in the walkthrough section below, from provisioning through a verified endpoint.

If you're coming from Kimi K2.7 Code or an earlier release (K2.6, K2.5), the jump to K3 isn't incremental. Total parameter count nearly triples, from 1T to 2.8T, and the deployment math changes with it. If you haven't committed to a model yet and want the broader self-hosting landscape first, the open-source LLM VRAM guide is worth reading before you provision a cluster for a 2.8T-parameter model.

What Kimi K3 Actually Is: 2.8T Total Params, 16 of 896 Active Experts

Kimi K3 is Moonshot AI's third-generation flagship: a 2.8 trillion parameter Mixture-of-Experts model with 896 total experts, 16 active per token, and a native 1,048,576-token (1M) context window. Moonshot unveiled it at the World AI Conference in Shanghai on July 16, 2026 and shipped API access the same day, with public weights following on Hugging Face eleven days later under a Modified MIT license.

Moonshot's own launch comparison table puts K3 at 77.8 on Program Bench, 88.3 on Terminal Bench 2.1, and 93.5 on GPQA-Diamond, ahead of the Claude Opus 4.8 figures it lists alongside them (71.9, 84.6, and 91.0 on the same three). Treat those as self-reported vendor numbers, not independently verified results: Moonshot ran both models on its own harness, and independent trackers report a different picture for at least one of the three. On GPQA-Diamond specifically, third-party tracking sites have logged Opus 4.8 scores closer to 93-94%, which would make that metric a toss-up or a narrow Opus edge rather than the clear K3 win Moonshot's table shows. No independent, apples-to-apples run of all three benchmarks across both models is publicly available as of this post's publication date. For a wider view of where K3 lands against the rest of the current open-weight field, see the open-weight frontier model showdown, and weigh any single vendor table accordingly. None of that changes the deployment math below: a bigger benchmark gap doesn't shrink the weight footprint you have to fit in VRAM.

Kimi Delta Attention (KDA), AttnRes, and Stable LatentMoE Routing

Three architectural changes separate K3 from the K2 family, and two of them matter directly for deployment.

Kimi Delta Attention (KDA) is a linear attention mechanism built to keep attention cost bounded as context grows toward 1M tokens, rather than scaling the way standard multi-head attention does. K3 does not use it everywhere: of the model's 93 layers, 69 are KDA linear-attention layers interleaved with 24 conventional MLA layers. That hybrid split matters operationally, because the two layer types hold different kinds of state. KDA layers keep a fixed-size recurrent state per request; the MLA layers keep an ordinary per-token KV cache. Any KV-cache or prefix-caching reasoning you apply to K3 has to account for both. Attention Residuals (AttnRes) works alongside it, selectively retrieving representations across depth instead of accumulating them uniformly through every layer. Combined, Moonshot reports these two changes deliver roughly 2.5x better overall scaling efficiency than Kimi K2.

Expert routing uses what Moonshot calls a "Stable LatentMoE" framework with quantile balancing: a token routes to an expert whenever that expert's router score lands in the top quantile for the batch. The mechanism is deterministic and hyperparameter-free, and Moonshot's stated goal is even expert utilization with zero dead experts, reinforced by a Per-Head Muon optimizer and a Sigmoid Tanh Unit (SiTU) activation designed to prevent the dead-neuron pathology that tends to show up in rarely activated experts at this scale.

Active Parameters: 104B Per Token, Confirmed

Early coverage of K3 disagreed wildly on this number, with trackers quoting anywhere from 32B to 104B. Moonshot's model card settles it: 104B active parameters per token, from 16 routed experts out of 896 plus 2 shared experts that fire on every token.

That figure is higher than a naive 2.8T x 16/896 calculation predicts (which lands around 50-60B) because the shortcut only counts the routed-expert slice. It ignores the two always-on shared experts and the rest of the always-active stack: attention, KDA, AttnRes, embeddings, and the MoonViT-V2 encoder. Every token pays for that stack regardless of which 16 experts the router picks.

For hardware sizing, the ambiguity never mattered much anyway. What has to fit in VRAM is every expert's weights, not just the 16 that happen to activate on a given token, because the router can send any token to any of the 896 experts on any forward pass. Active-parameter count tells you about compute speed. It tells you nothing about memory footprint. That distinction is the whole reason the VRAM math below runs on total parameters, not active ones, exactly as it does for DeepSeek V4-Pro's 1.6T MoE.

VRAM Math: The Real 1.56TB Checkpoint vs the 1.4TB Estimate

Start with the arithmetic, then correct it against the checkpoint Moonshot actually published, because the two don't match.

The MXFP4 row is the one that misleads people. The published moonshotai/Kimi-K3 repository is 1,560.94 GB across 96 shards, roughly 160GB more than 2.8T x 0.5 bytes predicts. The gap is everything in the checkpoint that isn't a 4-bit expert weight: MXFP8 activation scale blocks, BF16 and F32 tensors for embeddings, layer norms, the router, the attention and KDA projections, and the MoonViT-V2 vision encoder.

Plan around 1.56TB, not 1.4TB. That 160GB difference is exactly the margin that decides whether an 8-GPU B200 node loads the model or dies at 94% of the way through weight loading, and it is the single most common sizing mistake on this model.

Why Moonshot Trained for 4-Bit in the First Place

MXFP4 isn't a post-hoc quantization Moonshot bolted on after training in BF16. K3 was trained with quantization-aware training applied from the SFT stage onward, specifically so the released MXFP4 weights with MXFP8 activations serve natively at 4-bit without a separate calibration or quantization pass on your end. That's a meaningfully different starting point than K2.7 Code, where INT4 (AWQ) was an optional checkpoint alongside a native BF16/FP8 release. For K3, MXFP4 is the default. For the mechanics of block-scaled 4-bit formats and how MXFP4 differs from NVIDIA's NVFP4 hardware implementation, see the MXFP4 microscaling quantization guide.

The practical upshot: 1.56TB is the number you plan around unless you have a specific reason to run at higher precision.

MXFP8 (~2.8TB) and BF16 (~5.6TB) for Teams That Need Full Precision

Moonshot's public release is MXFP4 weights with MXFP8 activations, not a separate BF16 master checkpoint. If your team needs BF16 or MXFP8 weights, for further fine-tuning, for an accuracy baseline to compare quantization drift against, or simply because your serving stack doesn't yet support native MXFP4, check Moonshot's Hugging Face repository for an additional precision release before assuming one exists. The 2.8TB and 5.6TB figures above are what you'd need if such a checkpoint (or an upcast at load time) is available; they aren't a confirmed second SKU as of this post's publication date.

KV Cache Overhead at 1M Context and Why It Changes Your GPU Count

KDA's hybrid linear attention design exists specifically to keep the attention-side memory and compute cost from scaling the way standard full attention does as context stretches toward 1M tokens. That's good news for KV cache planning in principle. In practice, Moonshot has not published the exact per-token KV cache byte cost for K3, so there's no clean formula to plug into a spreadsheet yet.

What you can plan around: K2.6 and K2.7 Code, running Multi-head Latent Attention at a 256K context ceiling, needed roughly 40-80GB of extra headroom beyond base weights for a handful of concurrent long-context sessions. K3's context window is 4x larger. Even with KDA's architecture working in your favor, budget generously, well over 100GB beyond raw weights if you plan to serve concurrent long-context requests, until real production numbers are public. This is exactly why the GPU count you pick isn't just "however many GPUs fit the weights."

Reference Cluster Configs on B300, B200, and H100

Exactly one configuration runs K3 on a single node: 8x B300. Everything else, including all Hopper options, is multi-node. Here's what the arithmetic and the hardware actually support.

Why Blackwell's Native FP4 Tensor Cores Are the Better Match for MXFP4

K3 was trained with quantization-aware training so the released MXFP4 weights serve natively at 4-bit, no calibration pass required. That detail matters for GPU choice, not just VRAM math. Hopper-generation cards (H100, H200) ship a Transformer Engine with native FP8 support only, so serving MXFP4 on Hopper means dequantizing 4-bit weights to 16-bit inline during each matmul. Blackwell (B200, B300) ships a second-generation Transformer Engine with native FP4 Tensor Cores that execute K3's checkpoint format directly. For a model Moonshot built specifically for 4-bit serving, B300 is the GPU whose compute path actually matches the weights, not just whose memory happens to fit them.

Hopper still works, and cheaply, which is why it gets its own section below. Just don't expect Blackwell throughput from it.

Single 8x B300 Node (2,304GB) Is the Only One-Node Option

B300 SXM6 ships 288GB of HBM3e per GPU; an 8-GPU node totals 2,304GB. Against the real 1.56TB checkpoint, that leaves roughly 743GB for KV cache, activations, and concurrency headroom, on a single node, with no multi-node bring-up to manage at all. If B300 capacity is available, this is the config to reach for first. It is the shortest path from zero to a serving endpoint on this model.

8x B200 (1,536GB) Does Not Fit, and the Reason Matters

This is where the 1.4TB estimate bites. 8x B200 at 192GB per GPU totals 1,536GB, which clears the theoretical 1.4TB figure with 136GB to spare and fails the real 1,560.94GB checkpoint by about 25GB. You will get most of the way through weight loading and then hit an out-of-memory error, which is a slow and expensive way to learn the checkpoint is bigger than the arithmetic suggested.

Scale to 16x B200 across two 8-GPU nodes (3,072GB total) and the problem disappears: roughly 1,511GB of headroom after weights, using data-parallel-size 16 for the attention and KDA path combined with expert-parallel-size 16 to spread the 896 experts across the 16 GPUs, 56 experts per GPU. That is the DP16+EP16 shape the vLLM recipe points at for two-node MXFP4 serving.

Four 8x H100 Nodes (32 GPUs, 2,560GB): The Hopper Path

SGLang publishes an H100 cell in its Kimi K3 cookbook, and it calls for four 8-GPU H100 80GB nodes: 32 GPUs, 2,560GB aggregate. Against the 1.56TB checkpoint that looks like ~999GB of slack, but the real per-GPU picture is much tighter than that subtraction suggests.

In practice each rank ends up holding roughly 59.6GB resident, about 1,908GB across all 32 GPUs rather than 1,561GB. The extra ~350GB is per-rank duplication: every rank keeps a full copy of the attention, KDA, AttnRes, embedding, and MoonViT-V2 stack alongside its 28-expert shard, plus the repacked weight buffers the marlin kernel needs. On an 80GB card with ~78GB usable, that leaves only about 3.6GB for the KV cache pool and another ~3.4GB for KDA's recurrent state.

The mechanism that makes this work at all is --moe-runner-backend marlin, a W4A16 kernel that dequantizes MXFP4 weights to 16-bit inline during the matmul. It keeps the memory savings of a 4-bit checkpoint on hardware with no FP4 tensor cores, and pays for it in compute. A documented 4x8 H100 deployment measured single-stream decode at 5.8 tok/s and 14.0 tok/s aggregate across four concurrent streams. Treat those as the right order of magnitude rather than a guarantee: they came from H100 PCIe-NVLink nodes, and SXM5 with a full NVLink mesh should do somewhat better. For contrast, SGLang reports roughly 423 tok/s batch-1 decode on GB300 with DSpark speculative decoding, which is the gap between Hopper W4A16 emulation and native FP4 Blackwell with speculation. That is fine for evals, batch generation, agentic runs where the bottleneck is tool latency, or simply proving out the model before committing to Blackwell. It is not a high-concurrency production endpoint.

If you want the same 4x8 topology with real KV headroom, swap in H200 SXM5 instances on Spheron: 141GB per GPU puts 32 GPUs at 4,512GB, which turns that 3.6GB KV pool into something closer to 90GB per GPU. On-demand H100 access is the cheaper of the two and the one SGLang's cookbook documents directly.

GB200 NVL72 Slice Sizing for MXFP8/BF16 and Moonshot's "64+ Accelerators" Supernode Guidance

Once you move up to MXFP8 (~2.8TB) or BF16 (~5.6TB), even B300 node counts climb, two 8-GPU B300 nodes (16 GPUs, 4,608GB) clear MXFP8 with over 1.8TB of headroom, while BF16 needs closer to three B300 nodes. At that scale, a rack-scale system starts to make more sense than stacking standard nodes.

GB200 NVL72 packs 72 Blackwell GPUs at 186GB HBM3e each into a single rack with 13.5TB of unified memory addressable across all 72. Doing the bare weight-fit math: MXFP8's 2.8TB needs about 16 of those 72 GPUs; BF16's 5.6TB needs about 31. Both numbers are well under Moonshot's own guidance of "supernode configurations with 64 or more accelerators" for production serving. That gap between the bare weight-fit minimum (16-31 GPUs) and Moonshot's recommended floor (64+) is where KV cache headroom for 1M-context sessions, request concurrency, and redundancy actually live, not in the weights themselves. A full 72-GPU NVL72 rack clears the 64+ threshold comfortably, with the added benefit that GB200's native NVFP4 tensor core execution is hardware-compatible with K3's MXFP4 checkpoint format; see the NVFP4 vs MXFP4 guide for the format-level distinction if you're deciding between the two on Blackwell hardware.

GB200 NVL72 capacity on Spheron is reservation-based rather than listed at a live per-GPU on-demand rate; the GB200 NVL72 guide covers full rack specs, and GB200 NVL72 on Spheron is where you submit GPU count, timeline, and workload for a same-business-day availability check.

Spot vs On-Demand Tradeoffs for a Cluster This Size

Spot rates below were read from the Spheron GPU offers API on 30 Jul 2026. They are rounded, because precision here is false confidence: see the volatility note underneath.

The table is spot-only on purpose. On-demand capacity for these SKUs appears and disappears with availability: at the time of writing only B200 had on-demand offers listed, at $7.50/GPU/hr. When on-demand is available it runs meaningfully above spot and it is the right choice for a production endpoint, for the reasons in the next paragraph. Check current GPU pricing for what is actually listed when you go to provision.

Two things stand out. 8x B200 is absent from the table on purpose: it does not fit the checkpoint. And the single 8x B300 node is the configuration to want on architecture alone, independent of what the rates happen to be on any given day: one node instead of four, native FP4 execution instead of dequantizing kernels, and roughly 25x the per-GPU memory headroom of the H100 cluster (about 93GB free per B300 against a 3.6GB KV pool per H100).

Do not build a cost model on one reading of this table. Over a single afternoon while this guide was being written, H100 spot doubled, H200 spot nearly doubled, and on-demand offers for H100, H200, and B300 went from listed to absent. Pull live rates at provisioning time and re-check them before any commitment longer than a few hours.

Spot is 29-61% cheaper depending on the SKU, but the risk profile differs from a single-GPU workload. Every GPU in a data-parallel plus expert-parallel group holds a distinct shard of the 896 experts; reclaim any one of them and the whole serving group goes down, not just one replica. That applies to the single-node 8x B300 config too, since all 8 GPUs are still one DP+EP group. It gets worse as node count rises: a 32-GPU spot cluster has four times the reclaim surface of a single node, and re-forming the process group means re-loading 1.56TB from cache. For a production endpoint, on-demand is the safer default. Spot is worth it for evals, offline batch generation, or any run you can tolerate restarting.

Pricing fluctuates based on GPU availability. The prices above are based on 30 Jul 2026 and may have changed. Check current GPU pricing → for live rates.

Step-by-Step: Deploy Kimi K3 on Spheron with 4x 8x H100

This is the full bring-up for the 32-GPU Hopper configuration from SGLang's Kimi K3 cookbook, on Spheron. Budget about 50 minutes from provisioning to a live endpoint, most of it spent downloading 1.56TB of weights. Every command runs on all four nodes unless the step says otherwise.

If you have B300 capacity, skip to the single-node vLLM config in the next section instead. One node is less to go wrong.

Prerequisites

• A Spheron account with access to 8x H100 SXM5 capacity (app.spheron.ai)
• A Hugging Face account and access token with read scope for moonshotai/Kimi-K3
• SSH keypair registered before provisioning, per the SSH connection docs
• At least 1.7TB of free local NVMe per node, or a shared volume all four nodes can mount
• Docker with the NVIDIA container toolkit on each node

Step 1: Provision Four Identical 8x H100 Nodes

Provision four identical 8x H100 SXM5 nodes. On Spheron, log in and launch four 8x H100 SXM5 instances from the Spheron AI provider. They have to be identical: all 32 ranks load the same shard layout, and a mismatched node will fail the process group rather than degrade gracefully.

Record two things from each node once it boots:

Pick node 0 as the head node. Its IP goes into every node's launch command.

Open the full TCP range between the four nodes. NCCL and Gloo negotiate ephemeral ports during process-group formation and will hang on a partial range rather than fail loudly. Keep SSH (22) and the inference port (8000, head node only) scoped to your own public IP.

Step 2: Point Hugging Face and Docker at Local NVMe

The root disk will not hold a 1.56TB checkpoint. Move both the Hugging Face cache and Docker's data-root onto NVMe before pulling anything, on all four nodes:

# Substitute your instance's NVMe mount point for /mnt/nvme
sudo mkdir -p /mnt/nvme/hf /mnt/nvme/docker
sudo chown -R "$USER:$USER" /mnt/nvme/hf
export HF_HOME=/mnt/nvme/hf

sudo systemctl stop docker docker.socket
echo '{"data-root":"/mnt/nvme/docker"}' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker

Confirm Docker actually moved before you pull 13GB into the wrong filesystem:

docker info --format '{{.DockerRootDir}}'   # expect /mnt/nvme/docker

If you provisioned a shared volume instead of using local NVMe, mount it on all four nodes and point HF_HOME at it. See volume mounting for the setup. Shared storage saves you three redundant downloads; local NVMe gives faster weight loading on every restart. For a spot cluster that may restart repeatedly, local NVMe is usually worth the extra download.

Step 3: Download the Checkpoint on All Four Nodes in Parallel

The checkpoint is 1,560.94GB across 96 shards. Start all four downloads at once, not one after another:

python3 -m pip install -q huggingface_hub hf_transfer
export HF_TOKEN="hf_your_token_here"

nohup env HF_HOME=/mnt/nvme/hf HF_HUB_ENABLE_HF_TRANSFER=1 HF_TOKEN="$HF_TOKEN" \
  python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3', max_workers=16)" \
  > /tmp/dl.log 2>&1 &

HF_HUB_ENABLE_HF_TRANSFER=1 is what makes this finish in about half an hour instead of several hours. Expect roughly 30-40 minutes per node at sustained multi-gigabit throughput. Watch progress with tail -f /tmp/dl.log and confirm the final size before launching:

du -sh /mnt/nvme/hf/hub/models--moonshotai--Kimi-K3   # expect ~1.5T

A short download is the most common cause of a failed launch. If this number is materially under 1.5TB, re-run the same snapshot_download call; it resumes rather than restarting.

Step 4: Pull the SGLang Container

Run this concurrently with the download, on all four nodes:

nohup docker pull lmsysorg/sglang:kimi-k3 > /tmp/pull.log 2>&1 &

The kimi-k3 tag is built against CUDA 13. If your node image ships CUDA 12, use lmsysorg/sglang:kimi-k3-cu12 instead. Check with nvidia-smi and match the tag to the driver, or the container will fail on startup with a CUDA version mismatch. The image is about 13GB.

Step 5: Set the Per-Node Launch Variables

Only two values differ between nodes. Set these on each machine:

export RANK=0                       # 0 on the head node, then 1, 2, 3
export SELF_IP="10.0.0.10"          # THIS node's IP
export HEAD_IP="10.0.0.10"          # node 0's IP, identical on all four
export NIC="eth0"                   # network interface from Step 1
export HF_TOKEN="hf_your_token_here"

Getting RANK wrong is the failure mode to watch for. Two nodes claiming rank 0 means the process group never forms and all four sit at the barrier until --dist-timeout expires.

Step 6: Launch SGLang Across All Four Nodes

Define the engine arguments. These are identical on every node except $RANK:

K3_ARGS="--model-path moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tp-size 32 \
  --ep-size 32 \
  --nnodes 4 \
  --node-rank $RANK \
  --dist-init-addr $HEAD_IP:20000 \
  --moe-runner-backend marlin \
  --decode-attention-backend flashmla \
  --mem-fraction-static 0.85 \
  --dist-timeout 3600 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 8000"

Then start the container. Launch on the head node first, then the other three within a minute or so:

docker run -d --name sgl --gpus all --network host --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /mnt/nvme/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env NCCL_CUMEM_ENABLE=1 \
  --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --env SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
  --env SGLANG_K3_ATTN_RES_MODE=jit \
  --env SGLANG_MOE_FUSED_GATE_RADIX=1 \
  --env SGLANG_HOST_IP=$SELF_IP \
  --env NCCL_SOCKET_IFNAME=$NIC \
  --env GLOO_SOCKET_IFNAME=$NIC \
  --env NCCL_DEBUG=WARN \
  lmsysorg/sglang:kimi-k3 \
  sglang serve $K3_ARGS

What each of the non-obvious flags is doing:

Expect roughly 11 minutes from container start to a serving endpoint once weights are cached locally:

Follow the head node with docker logs -f sgl. The other three ranks stay quiet until the process group forms, which is expected and not a sign anything is wrong.

Step 7: Verify the Endpoint

From the head node:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [{"role": "user", "content": "Explain expert parallelism in two sentences."}],
    "max_tokens": 256,
    "reasoning_effort": "low"
  }' | python3 -m json.tool

A healthy response carries both content and a populated reasoning_content. If reasoning_content is empty or the chain of thought is leaking into content, --reasoning-parser kimi_k3 did not take effect. Check that you did not carry over kimi_k2 from a K2-family deployment, which is the single most common configuration error on this model.

Then confirm the memory picture matches expectations with nvidia-smi. On 80GB H100s you should see roughly 71GB of 81.5GB in use per GPU, with about 59.6GB of that being resident weights. Materially less means a rank failed to load its shard.

Step 8: Tear Down

Weights on local NVMe disappear with the instance, so capture anything you need first.

# On each node
docker rm -f sgl

Then release all four nodes from the Spheron dashboard. On a spot cluster this matters more than usual: an idle 32-GPU H100 cluster was costing about $93/hr to do nothing at the rates above.

What Goes Wrong, and What It Means

vLLM/SGLang Serving Flags for K3's Long-Context, Multimodal Requests

K3's weights landed less than 24 hours before this guide was written. Flag names below reflect vLLM's day-0 preview support and are likely to shift as the model moves from nightly builds to a stable release. Pin your vLLM commit hash in production and re-check the release notes before upgrading.

Tensor-Parallel, Expert-Parallel, and Data-Parallel Flags (Single-Node B300 Reference Config)

On a single 8-GPU B300 node, no Ray cluster or multi-node networking is required. Launch vLLM directly:

vllm serve moonshotai/Kimi-K3 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --host 0.0.0.0 \
    --port 8000 \
    --tool-call-parser kimi_k3 \
    --reasoning-parser kimi_k3 \
    --max-model-len 1048576 \
    --trust-remote-code

--data-parallel-size 8 replicates the attention/KDA path across all 8 ranks, one per GPU. --enable-expert-parallel shards the 896 experts across those same 8 ranks, 112 experts per GPU. This is the same DP+EP pattern used for wide expert parallelism on other giant MoE models.

If you're deploying on two 8-GPU B200 nodes instead, start a Ray cluster spanning both nodes and double every DP/EP value to 16:

# Node 0: start Ray head
ray start --head --port=6379

# Node 1: connect to Ray head
ray start --address='<node0-ip>:6379'

# Node 0: launch vLLM across both nodes
vllm serve moonshotai/Kimi-K3 \
    --data-parallel-size 16 \
    --enable-expert-parallel \
    --host 0.0.0.0 \
    --port 8000 \
    --tool-call-parser kimi_k3 \
    --reasoning-parser kimi_k3 \
    --max-model-len 1048576 \
    --trust-remote-code

That 16-rank config puts 56 experts on each GPU. vLLM's own K3 recipe treats 8x GB300-class GPUs as the baseline and recommends multi-node for production, so verify flag names against the recipe before you rely on this shape.

For a comparison of when vLLM's DP+EP path versus SGLang's --ep-size/--enable-dp-attention combination makes more sense for your serving stack, see vLLM vs SGLang. The equivalent SGLang launch for a single B300 node:

python -m sglang.launch_server \
  --model-path moonshotai/Kimi-K3 \
  --tp 8 \
  --ep-size 8 \
  --enable-dp-attention \
  --dp 8 \
  --context-length 1048576 \
  --port 30000

KDA Prefill Caching and Why Standard Prefix Caching Doesn't Apply Yet

Standard prefix caching, the kind that gives K2-family models a free speedup on repeated system prompts and shared tool definitions in agentic workloads, works by hashing token-aligned KV blocks and reusing them across requests that share a prefix. That mechanism still applies to K3's 24 MLA layers, which hold a conventional per-token KV cache. It does not map cleanly onto the 69 KDA layers, which maintain a fixed-size recurrent state per request rather than discrete per-token key/value pairs, so a cache lookup built purely for standard attention doesn't decompose the same way against them.

The practical consequence is that prefix-cache benefit on K3 is partial rather than all-or-nothing: you get reuse on the MLA layers and have to rely on engine-specific KDA state handling for the rest. Both vLLM and SGLang shipped KDA-aware caching alongside day-0 support, but don't carry your hit-rate assumptions over from K2.6 or K2.7 Code, which were pure MLA. If your workload leans on long, repeated system prompts or tool schemas, benchmark actual cache hit rates on K3 before sizing concurrency around them.

Tool-Call-Parser, Reasoning-Parser, and Max-Model-Len for 1M Context

K3 introduces its own kimi_k3 parsers. This is the one place where carrying a K2-family config forward will silently break things: kimi_k2 is still a valid parser name, so both vLLM and SGLang start without complaint, then emit chain-of-thought text inside content and leave reasoning_content empty. Tool calls fail to parse into OpenAI-format tool_calls the same way. Set both --tool-call-parser and --reasoning-parser to kimi_k3. The wire format of MCP and function-calling integrations built against K2.6 or K2.7 Code is otherwise unchanged, so only the parser names need updating.

--max-model-len 1048576 sets the ceiling at the full 1M-token window, but set it to your actual traffic maximum, not the architectural ceiling by default. Every token of unused context length is KV cache headroom you're paying for and not using; if your real requests cap at 128K tokens, set --max-model-len 131072 and recover the difference for more concurrent sessions.

For multi-file or multi-document requests near the top of that range, add chunked prefill to avoid a memory spike on the first forward pass (shown here for the single-node B300 config; use --data-parallel-size 16 on a two-node B200 deployment):

vllm serve moonshotai/Kimi-K3 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --max-num-batched-tokens 8192 \
    --tool-call-parser kimi_k3 \
    --reasoning-parser kimi_k3 \
    --max-model-len 1048576 \
    --trust-remote-code

MoonViT-V2 Vision Encoder Flags for Multimodal Requests

K3 ships with a MoonViT-V2 vision encoder, roughly 401M parameters, small enough that it doesn't meaningfully change the VRAM math above, but it does need its own flag to distribute encoding work across your GPU set. Following the same pattern K2.6 uses for the original MoonViT:

vllm serve moonshotai/Kimi-K3 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --mm-encoder-tp-mode data \
    --tool-call-parser kimi_k3 \
    --reasoning-parser kimi_k3 \
    --max-model-len 1048576 \
    --trust-remote-code

Send multimodal requests through the standard OpenAI-compatible chat completions endpoint with image_url content blocks. Confirm the vision path is active by sending a test request with a base64-encoded image before routing production multimodal traffic to the endpoint. For provisioning and SSH setup, docs.spheron.ai covers connection steps for both single-node and multi-node clusters.

Calling K3: reasoning_effort and the reasoning_content Contract

K3 always thinks. There is no way to turn reasoning off, no enable_thinking: false escape hatch, and no non-reasoning sibling model. What you get instead is a throttle: reasoning_effort, which accepts exactly three values.

The default matters for cost. Leave reasoning_effort unset and every request runs at max, generating reasoning tokens you may not need and paying for them in both latency and GPU time. On the 32x H100 config, where single-stream decode sits around 6 tok/s, the difference between low and max on a simple request is the difference between a few seconds and a minute. Set it explicitly per request type.

curl -s http://$HEAD_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [{"role": "user", "content": "Refactor this function for readability."}],
    "reasoning_effort": "high",
    "max_tokens": 2048
  }'

Responses split across four fields: content for the answer, reasoning_content for the chain of thought, tool_calls for structured tool invocations, and reasoning_tokens in the usage block, counted separately from completion_tokens. Bill against the sum, not completion_tokens alone, or your cost model will understate max-effort traffic badly.

Echo reasoning_content Back on Every Turn

This is the operational gotcha that catches teams migrating from the K2 family. K3 expects the complete previous assistant message on multi-turn calls, including reasoning_content and any tool_calls. Drop reasoning_content when you append to the message history, which is what most OpenAI-compatible client wrappers do by default, and generation quality degrades over the course of a conversation. Long agentic loops are where it shows up worst, because the model loses the thread of its own prior reasoning at exactly the point it needs it most.

If you have a message-history helper that strips non-standard fields, fix it before you put K3 behind an agent. This is not an optimization; it changes output correctness.

Kimi K3's real 1.56TB checkpoint means the cluster-sizing math matters more than the benchmark scores. A single 8-GPU Spheron B300 node fits it on native FP4 hardware, and four 8-GPU H100 nodes get you there for less if you can live with ~6 tok/s single-stream decode.

Spheron B300 instances → | Check H100 availability → | View all GPU pricing →