How much VRAM does this LLM actually need?

At 4-bit quantization (Q4_K_M) Llama 3.1 70B weighs about 40 GB. Add 2-4 GB for an 8k context KV cache and 1-2 GB of overhead, you land near 48 GB total. A single 48 GB card (RTX 6000 Ada, A40) fits it. Two 24 GB consumer cards (2x 4090, 2x 3090) fit it with tensor parallelism. A100 80 GB is the safe single-card choice.

Apple Silicon Macs use unified memory, so the GPU shares system RAM. About 75% is available for the GPU. An M2/M3/M4 Pro with 36 GB runs 7B-13B comfortably. M3/M4 Max with 64-128 GB handles 32B and 70B at 4-bit. M2 Ultra 192 GB and M4 Ultra 256 GB are the only single-machine options that fit DeepSeek V3 quantized.

Quantization reduces the bits per weight. FP16 is the native training precision (16 bits, no loss). Q8 halves memory with negligible drop. Q4_K_M (~4.85 bits) is the popular sweet spot, ~70% memory cut and a small quality hit. Q3 and below start showing real degradation, particularly on math and coding tasks. The MMLU drop from FP16 to Q4 is usually 1-3 points.

Every transformer layer keeps the keys and values for past tokens so it does not recompute attention. Memory scales linearly with context length. For Llama 3.1 70B at 32k context, the KV cache is around 10 GB on its own. Long-context models with high head counts (older MQA-free designs) burn KV cache the fastest. Group Query Attention (GQA) cuts this dramatically.

Benchmark numbers often quote the model weights only and assume a tiny context. This calculator includes weights, KV cache for your context length, activation memory, and a 1 GB engine overhead, then adds 20% headroom because real workloads spike. If you set context to 2048 and ignore overhead, you will roughly match the marketing numbers.

Two used 3090s give you 48 GB total at roughly the same price as one 4090, which has 24 GB. For models above 24 GB the 2x 3090 setup wins. Below 24 GB, the single 4090 is faster, runs cooler, and avoids the multi-GPU complexity. The break point is whether you want to run 70B-class models locally.

Yes, with friction. ROCm support has improved a lot. RX 7900 XTX with 24 GB is a viable LLM card on Linux through llama.cpp Vulkan or ROCm builds. Datacenter MI300X with 192 GB is excellent if you can get one. Day-to-day, most tooling (vLLM, transformers, ExLlamaV2) is more polished on NVIDIA.

Mixture of Experts models (DeepSeek V3, Mixtral) have many expert subnetworks. Each token uses only a few. DeepSeek V3 has 671B total weights but only 37B active per token. You still need to fit all 671B weights in memory because any expert can fire. Speed scales with active params, memory scales with total.

KV cache memory is independent of weight quantization. Setting it to 8-bit halves cache memory, 4-bit quarters it. Most modern serving engines (vLLM, llama.cpp, TensorRT-LLM) support FP8/INT8 KV cache. Quality impact is small on chat workloads, larger on reasoning.

Token generation is bound by memory bandwidth, not compute. Read the active weights once per token. Tokens per second roughly equals (bandwidth in GB/s) / (active params in GB). A 4090 at 1008 GB/s running 8B-Q4 (4 GB) gives ~250 tok/s peak, ~160 in practice. The calculator uses a 65% efficiency factor.

Yes, but mostly through KV cache reads. The first token in a long prompt takes a while (prefill is compute-bound). After that, every generated token reads KV from all prior tokens. At very long contexts, KV bandwidth starts dominating and tok/s drops.

No, this is for inference only. Training adds optimizer states (AdamW = 8 bytes per param at FP32 master), gradients, and activation checkpointing. A 7B full fine-tune wants ~80 GB. LoRA fine-tuning is much cheaper, ~10 GB for 7B on top of model weights.

Pick the closest size and architecture. The calculator uses exposed config (params, hidden size, layers, KV heads, head dim). For a model with similar architecture, results are within 5-10%. We will keep adding models, especially Mistral Large, Llama 3.3, and the new Chinese open weights.

Yes, with patience. Quantize to Q4 or lower (~40 GB), use llama.cpp on a machine with 64+ GB RAM and DDR5. Expect 1-3 tokens per second on a fast desktop, 5-10 on a server with high memory bandwidth. Apple Silicon Macs are far better than x86 CPUs for this because of the unified memory architecture.

Eighteen covers the spectrum from cheap consumer cards to top-tier datacenter and Apple Silicon. Adding more turns the list into noise. If your specific card is missing, find the closest match by VRAM and bandwidth, the verdict will still hold.