How VRAMForge Models GGUF Weights, Parallel KV-Cache & Roofline Tok/s

1. GGUF Super-Block Weight Equation (Effective BPW) — VRAMForge

Modern llama.cpp K-quants allocate 6-bit blocks to attention/output tensors plus ~6% ggml compute-graph metadata:

Weights_GB = (Parameters_Billions * Effective_BPW / 8) * 1.06
Q4_K_M = 4.85 bpw | Q5_K_M = 5.70 bpw | Q8_0 = 8.50 bpw | FP16 = 16.0 bpw

2. Parallel Grouped-Query Attention (GQA) KV-Cache Scaling — VRAMForge

Serving parallel streams across 16K–64K windows expands KV slots dynamically, moderated by prefix reuse (0.65 factor) and Q8_0 KV cache quantization:

KV_Cache_GB = (Context_Tokens / 1024) * KV_Per_1K_Model * Quant_KV_Scale * (1 + (Users - 1) * 0.65)

3. Memory-Bandwidth Roofline Token Throughput (Tok/s) — VRAMForge

Autoregressive token decoding streams the active weight matrix once per generated token, making sustained GB/s memory bandwidth the primary speed ceiling:

Autoregressive_Tok_s = Effective_Bandwidth_GBs / Active_Tensor_Footprint_GB
AMD EPYC / Ryzen DDR5 KVM & VDS: ~78 - 135 GB/s | NVIDIA A16 / L40S / A100: ~600 - 1,555 GB/s

Open Interactive VRAMForge Workbench →