> **TL;DR.** Most teams overpay for inference by 3-10x because they skip three things: quantization, KV cache configuration, and batching strategy. Fix those first. Hardware and serving framework choices matter but are secondary to getting the model configuration right.
Why Inference Is Not Just "Running the Model"
Training a model is a one-time cost. Inference runs continuously against real users, real latency budgets, and real billing cycles. A model that performs well on a benchmark can still destroy your unit economics in production if you're materializing full-precision weights, processing requests serially, or allocating KV cache naively.
The gap between a badly-served model and a well-served one is not 10-20% — it can be a 5-10x difference in throughput and cost per token. AI inference optimization is where that gap closes.
The core problem: transformer inference is memory-bandwidth-bound, not compute-bound, for most deployment configurations. You're waiting on VRAM reads, not GPU arithmetic. Every technique in this guide attacks that bottleneck from a different angle.
Quantization: INT8, INT4, and When to Use Each
Quantization reduces weight precision, which cuts memory bandwidth and VRAM requirements.
**INT8 (W8A8 or W8A16)**
- ~2x memory reduction vs FP16
- Negligible quality loss on most tasks (perplexity degradation < 1%)
- Safe default for production deployments
- Supported natively by vLLM, TGI, TensorRT-LLM
**INT4 (GPTQ, AWQ, GGUF Q4_K_M)**
- ~4x memory reduction vs FP16
- Quality loss is model-dependent — acceptable for most chat tasks, noticeable on complex reasoning
- Enables running 70B-class models on consumer hardware (2x 48GB GPUs instead of 4)
- llama.cpp Q4_K_M is the practical default for local/edge deployments
**The tradeoff matrix:**
Start with INT8. Move to INT4 only after measuring quality impact on your specific task distribution.
KV Cache: The Most Undertuned Knob
The KV (key-value) cache stores attention computation results across tokens so the model doesn't recompute them on every forward pass. It's what makes autoregressive generation tractable.
The problem: KV cache allocation is often left at defaults, which either wastes VRAM or causes costly evictions under load.
**What to tune:**
- **Cache block size:** vLLM uses paged attention with configurable block sizes. Larger blocks reduce fragmentation for long contexts; smaller blocks are more efficient for short requests.
- **GPU memory utilization:** vLLM's `--gpu-memory-utilization` defaults to 0.90. For mixed-length workloads, 0.85 gives headroom; for long-context workloads, you may need to tune this alongside `--max-model-len`.
- **Prefix caching:** vLLM supports automatic prefix caching (APC). If your system prompt is long and consistent across requests — like a RAG context, a coding assistant system prompt, or a customer service persona — enable prefix caching. First-token latency drops dramatically for cached prefixes, often 40-60% on real workloads.
```bash
vllm serve mistral-7b-instruct \
--gpu-memory-utilization 0.87 \
--enable-prefix-caching \
--max-model-len 16384
```
Prefix caching is the highest-ROI optimization for most production deployments. It costs nothing and requires no model changes.
Continuous Batching and Throughput vs. Latency
Static batching processes a fixed batch of N requests together. If request 1 finishes early, the GPU waits for the rest. This is how early inference servers worked and it's expensive.
**Continuous batching** (also called dynamic batching or in-flight batching) adds new requests to the batch as slots free up. vLLM, TGI, and TensorRT-LLM all implement this. It's the single biggest throughput improvement over naive serving and should be non-negotiable for any production deployment.
The latency-throughput tradeoff:
- **Low traffic:** requests are served near-immediately, latency is low, GPU utilization is low
- **High traffic:** requests queue slightly, latency increases, GPU utilization approaches 100%
- **Overload:** queue grows unbounded — you need autoscaling or request shedding
Set `max_batch_tokens` based on your latency SLA, not just throughput maximization. A batch that's 2x larger may double throughput but also double p99 latency. Measure both.
Serving Stack Comparison
The main open-source options in 2026:
**vLLM**
- Best general-purpose choice. Continuous batching, paged attention, prefix caching, OpenAI-compatible API.
- Supports most open-weight models. Tensor parallelism across multiple GPUs.
- Production-ready, actively maintained.
**TensorRT-LLM**
- NVIDIA's optimized runtime. Best raw throughput on NVIDIA hardware.
- More complex setup; model conversion required. Best for fixed-hardware deployments where you want maximum GPU utilization.
**llama.cpp**
- CPU-viable (though slow). GGUF format. Best for local development, edge deployment, and hardware without NVIDIA GPUs.
- Apple Silicon MLX or llama.cpp with Metal backend gets useful throughput on M-series chips.
**Triton Inference Server**
- Model-agnostic serving infrastructure. Supports ensemble pipelines (pre/post-processing + model). Better fit for multi-model systems than single-model serving.
**ONNX Runtime**
- Strong for smaller models, edge deployment, and non-transformer architectures. Less relevant for LLM-scale inference.
For most teams building on top of open-weight LLMs: start with vLLM. Graduate to TensorRT-LLM on NVIDIA if you need the last 20% of throughput and have ops bandwidth to maintain it.
Hardware Reality in 2026
**Cloud GPU (A100, H100, H200)**
- High throughput, high cost. Necessary for very large models or high-concurrency workloads.
- H100 SXM is the current performance reference. H200 improves memory bandwidth, which directly helps inference.
**Consumer GPU (RTX 4090, 5090)**
- 24GB VRAM handles 7B-13B models at INT4/INT8. 70B requires multi-GPU.
- Viable for low-traffic self-hosted deployments or development.
**Apple Silicon (M3 Ultra, M4 Pro/Max/Ultra)**
- Unified memory means 64-192GB addressable by the GPU. Runs 70B models comfortably at INT4.
- Throughput is lower than datacenter GPU but the cost-per-useful-token story is good for moderate loads.
- MLX is Apple's inference framework; llama.cpp Metal backend is also solid.
**Edge inference**
- Smaller quantized models (Phi-3 Mini, Gemma 2 2B, Qwen 2.5 1.5B) are increasingly viable on mobile silicon and specialized edge chips.
- Relevant for latency-sensitive applications where round-trip to cloud is unacceptable.
For teams shipping AI features on a budget: a single M4 Max or used A6000 often outperforms cloud on cost for moderate traffic. Do the math before defaulting to on-demand cloud GPU.
Pruning and Knowledge Distillation
These require more work than quantization and serve different goals.
**Structured pruning** removes entire attention heads or MLP neurons. The result is a genuinely smaller model that runs faster without quantization tricks. Tools like LLM-Pruner and SparseGPT automate this, but you need to fine-tune after pruning to recover quality.
**Knowledge distillation** trains a smaller model (student) to mimic a larger one (teacher). This is how many small-but-capable models are built — the student learns not just ground-truth labels but the teacher's output distribution. Distillation requires a training pipeline and labeled data, which puts it out of reach for most product teams. You're more likely to *use* a distilled model (e.g., a fine-tuned Phi or Qwen variant) than to run distillation yourself.
**When to consider these:** when quantization isn't enough and you need to reduce model size for hardware constraints, not just cost.
Cost Benchmarking and Where to Start
Rough cost anchors (durable order-of-magnitude, not specific provider pricing):
- Frontier model APIs: highest cost, zero ops burden
- Self-hosted large open models (70B+): moderate cost, moderate ops burden
- Self-hosted small open models (7B-14B): low cost, low ops burden if using vLLM
- Edge/local: near-zero marginal cost, upfront hardware
The optimization sequence that pays off for most teams:
1. **Enable prefix caching** — free, immediate
2. **Switch to INT8** — minimal quality risk, significant cost reduction
3. **Tune batch configuration** — improve throughput under load
4. **Consider INT4** — measure quality impact first on your eval set
5. **Right-size the model** — a well-prompted 14B model often beats a poorly-prompted 70B at 5x lower cost
6. **Evaluate hardware** — only after steps 1-5 are done
AI inference optimization is an iterative process. Profile first, optimize the biggest bottleneck, measure, repeat.
Next Steps
- If you're deploying models to production, see [AI Model Deployment 2026](/en/rehberler/ai-model-deployment-2026) for the full ops picture around serving, monitoring, and rollback.
- If you're a backend engineer integrating inference into a larger system, [AI for Backend Developers 2026](/en/rehberler/ai-backend-developers-2026) covers the API integration and reliability patterns.
- For SREs managing inference infrastructure at scale, [AI for SRE 2026](/en/rehberler/ai-sre-2026) covers observability, alerting, and capacity planning for model-serving systems.