$ llm-deploy --estimate

LLM Deployment Planner

Estimate memory, parallelism and throughput for a model on a target GPU

INPUT · 部署参数
> Model
> Target GPU
> Devices
> GPUs per node
> Total params (B)
> Active params (B)
> Context length
Model max 1,048,576 tokens; lower it when VRAM is short
> Prefill length
> KV precision
> Weight precision
Concepts
Precision
Bytes per param: fp32=4, bf16/fp16=2, fp8/int8=1, fp4=0.5 — drives weight size.
Tensor Parallel (TP)
Splits each layer's matmul across cards; needs high-bandwidth interconnect.
Pipeline Parallel (PP)
Splits the layer sequence into stages; has cross-node bubbles.
Data Parallel (DP)
Multiple full replicas each serve part of the concurrency; near-linear throughput.
MLA vs GQA
MLA compresses KV to a low-rank latent (DeepSeek); GQA uses fewer KV heads.
Efficiency η
Conservative 0.7 / Typical 0.8 / Aggressive 0.9 — higher is more optimistic, less overhead.
TTFT / TPOT / Throughput
First-token latency (prefill compute-bound) / per-token latency / tokens per second (decode bandwidth-bound).
Deployment Plan
DeepSeek-V4-Flash (MoE 284B-A13B)MLAMoE
▸ GB10 (DGX Spark) · 1 devices × 1 GPUs = 1 cards
✗ Won't fit · min 3 cards
Current device · 1 devices × 1 GPUs = 1 cards
⚠ Insufficient cards: 1 < min 3, cannot fit one replica
1×TP · 3×PP · 3 cards per replica, up to 5 concurrent
Per-stream tok/s
21tok/s
Throughput
0tok/s
Total concurrency
0
TTFT
266ms
TPOT
47.6ms
Per-card usage
112.3GB
Card schemes
CardsFitsReplicas (DP)ConcurrencyThroughput
1 GPUs◀ Current deviceWon't fit———
3 GPUs✓ Deployable1526 tok/s
6 GPUs✓ Deployable21052 tok/s
12 GPUs✓ Deployable420104 tok/s
Tier comparison
TierCardsEfficiency ηPer-cardTTFTTPOTThroughput
Conservative4η=0.7112.6 GB304 ms47.6 ms29 tok/s
Typical3η=0.8112.3 GB266 ms47.6 ms26 tok/s
Aggressive3η=0.9113.2 GB237 ms47.6 ms31 tok/s
Per-card memory(Typical)
■ Weights: 94.7 GB
■ KV cache: 9.6 GB (1,048,576 tok)
■ Overhead: 8 GB
■ Total demand: 112.3 GB / Usable 114 GB
Unsharded total:284 GB(Weights)+ 28.9 GB(KV)+ 24.1 GB(Overhead)= 337 GB
Notes
  • › 参数量与激活参数由公开口径/checkpoint 体量推算,官方以模型卡为准
  • › 屋顶线估算,吞吐随 η 取值浮动,非实测;不含投机解码(MTP)加速
  • › KV 按 CSA+HCA 压缩注意力(时间轴压缩,等效 KV 极小),全层常驻计算
  • › PP 跨节点开销未计入带宽
  • › 总吞吐 = η × 带宽 ÷ (激活权重/并发 + 每路KV),decode 受内存带宽约束