Loading calculator…

Inference Speed Calculator

Calculate AI model inference speed in tokens per second across different hardware configurations. Compare LLM throughput on GPU, TPU, and CPU with batch size optimization. Essential for AI deployment planning, cost estimation, and production serving infrastructure.

Inference Speed

Speed = (Memory Bandwidth / Bits per Token) × Utilization

Keterangan:

  • Speed
    Speed (tokens/second)
  • BW
    Memory bandwidth (GB/s)
  • Bits
    Bits per token (FP16=16, INT8=8)
  • Util
    Utilization (0.6-0.9)

How to Use

  1. 1

    Select Model

    Choose model and quantization (FP16, INT8, etc.).

  2. 2

    Select GPU

    Choose the GPU being used.

  3. 3

    Calculate

    Get tokens per second.

Contoh Perhitungan

Llama 3 8B on RTX 4090

Soal:

FP16, BW=1008GB/s, Util=0.8. Speed?

Penyelesaian:
  1. 1.Speed = (1008 / 16) × 0.8
  2. 2.Speed ≈ 50 tokens/s
Hasil:≈ 50 tokens/second

An RTX 4090 can run Llama 3 8B at about 50 tokens per second.

Pertanyaan yang Sering Diajukan (FAQ)

Does batch size matter?
Larger batches increase throughput but reduce responsiveness.
FP16 vs INT8?
INT8 is about 2× faster but accuracy drops slightly.

Kalkulator Terkait

LLM VRAM

VRAM requirements

Training Cost

Training cost

Referensi