Model Weights VRAM
Model weights = parameter count × bytes per weight. A 70B parameter model at FP16 (2 bytes/weight) needs ~140GB just for weights; at Q4_K_M (0.5 bytes/weight) only ~35GB.
NVIDIA VRAM Calculator
We earn commissions when you shop through the links below.
Estimate VRAM requirements for local LLMs. Calculate model weights, KV cache, CUDA overhead, and check fit against common GPUs. Works entirely in your browser.
All calculations happen in your browser.
No model weights or specifications are uploaded anywhere.
Model Weights VRAM
0 GB
KV Cache VRAM
0 GB
CUDA Overhead
0 GB
Total Minimum VRAM
0 GB
Recommended (25% margin)
0 GB
⏳
Select a model to check fit
How VRAM Is Calculated
KV Cache
Key-Value cache grows linearly with context length: 2 × layers × heads × head_dim × context_length. Longer contexts need proportionally more VRAM during inference.
CUDA Overhead
A fixed ~0.55 GB overhead accounts for CUDA context, driver scratchpad, and temporary tensors during inference.
Frequently Asked Questions
- It’s a close estimate based on standard VRAM formulas. Actual usage varies by inference engine (llama.cpp, Ollama, vLLM), kernel optimizations, and OS overhead.
- Real-world inference needs headroom for CUDA context, temporary activation tensors, and OS memory. The 25% margin helps avoid out-of-memory errors.
- Light quantization (Q4_K_M, Q5_K_M) typically retains most model quality while cutting VRAM needs roughly in half compared to FP16.
- Yes. Once the page loads, all calculations run locally in your browser.
Understanding Local LLM VRAM Requirements & Quantization
FP16 vs Quantized Weights
FP16 stores each parameter at 2 bytes. Q8_0 uses 1 byte, Q5_K_M uses ~0.67 bytes, and Q4_K_M uses just 0.5 bytes per weight — reducing model size by up to 75% with minimal quality loss.
NVFP4 on Blackwell
NVIDIA's native 4-bit format on Blackwell GPUs (RTX 50-series) provides native hardware decompression, delivering near-FP16 quality at 4-bit memory footprint.
Size Scaling Table
A 70B model needs ~140GB at FP16 but only ~35GB at Q4_K_M. Smaller 8B models fit on consumer GPUs like the RTX 4060 at 8GB VRAM.
How KV Cache Scales with Context Length
KV Cache Basics
Each token processed adds key and value tensors for every attention head across all layers. This memory grows with every token generated.
128k Context Impact
A 128k context window on a 70B model can require tens of GB just for KV cache — often more than the model weights themselves at low quantization.
Context vs Model Size
For long-context workloads, prioritize models with larger context windows and GPUs with more VRAM. For short prompts, prioritize parameter count and quantization tier.
Choosing the Right Hardware for Open-Source AI
Consumer GPUs
RTX 4060 (8GB) suits small 7B-14B models at Q4_K_M. RTX 4090 (24GB) handles 70B models at Q4_K_M for inference and light fine-tuning.
RTX 5090 & Beyond
RTX 5090 (32GB) enables larger models at better quantization tiers. NVFP4 support on Blackwell further reduces memory footprint by 2x vs Q4_K_M.
Mac Studio Unified Memory
Mac Studio M2 Ultra (192GB) and M3 Ultra (384GB) can load massive models into unified memory — ideal for 70B+ models at higher quantization tiers.