Estimate memory, parallelism and throughput for a model on a target GPU
| Cards | Fits | Replicas (DP) | Concurrency | Throughput |
|---|---|---|---|---|
| 1 GPUs◀ Current device | Won't fit | — | — | — |
| 3 GPUs | ✓ Deployable | 1 | 5 | 26 tok/s |
| 6 GPUs | ✓ Deployable | 2 | 10 | 52 tok/s |
| 12 GPUs | ✓ Deployable | 4 | 20 | 104 tok/s |
| Tier | Cards | Efficiency η | Per-card | TTFT | TPOT | Throughput |
|---|---|---|---|---|---|---|
| Conservative | 4 | η=0.7 | 112.6 GB | 304 ms | 47.6 ms | 29 tok/s |
| Typical | 3 | η=0.8 | 112.3 GB | 266 ms | 47.6 ms | 26 tok/s |
| Aggressive | 3 | η=0.9 | 113.2 GB | 237 ms | 47.6 ms | 31 tok/s |