TL;DR: A dual-socket AMD EPYC 9334 delivers 20–28 tokens/sec on Q4-quantized 7B–20B LLMs and sub-real-time TTS inference (RTF 0.16 for Kokoro). Throughput is roughly half that of an NVIDIA L4 GPU, but at a fraction of the per-hour cost. For batch, async, and lightweight TTS workloads, this AMD EPYC LLM inference benchmark shows CPU is the right call.
Why Inference Costs More Than You Planned For
Training is a one-time cost. Inference is not. Once a model is in production, it runs continuously. Cost per query scales directly with traffic, and for many teams, inference spend overtakes training spend within months of launch.
The hardware decision for inference is also different from training. Training needs large GPU clusters and high-bandwidth interconnects. Inference needs low latency, high throughput per dollar, and enough memory bandwidth to serve quantized weights efficiently. Those requirements do not always point to a GPU.
How LLM Inference Works (And Why It Matters for Hardware Choice)
When a prompt hits an LLM, processing happens in two stages.
Prefill. The model converts input into tokens, runs them through its layers, and builds a KV cache. Because all input tokens are known upfront, this stage is parallelizable and compute-bound. It ends when the first output token is generated.
Decode. The model generates each subsequent token one at a time, reading from the KV cache at each step. Large models have large KV caches, which makes this stage memory-bandwidth-bound.
These two stages have different performance profiles, which is why benchmarks report two different metrics.
Key Metrics for This AMD EPYC LLM Inference Benchmark
Time to first token (TTFT) reflects prefill performance. Lower is better.
Tokens per second measures decode throughput. Higher is better, particularly for batch and streaming workloads.
Real-time factor (RTF) is the standard metric for TTS models. Below 1.0 means faster than real time. Above 1.0 means it cannot keep up.
Hardware and Software Setup
All tests ran on a single HPE ProLiant DL385 Gen11 server with two AMD EPYC 9334 processors.
| Component | Specification |
|---|---|
| CPU | AMD EPYC 9334 (x2) |
| Architecture | Zen 4 |
| Cores / Threads per socket | 32 / 64 |
| Base clock | 2.7 GHz |
| L3 cache | 128 MB per socket |
| TDP | 210W per socket |
| Memory | 64 GB DDR5 |
Two benchmarking tools were used: llama-bench (part of llama.cpp) for local model evaluation, and OpenLLM with llmperf for API-level throughput testing.
Models Tested
| Model | Parameters | Type |
|---|---|---|
| DeepSeek-R1-0528-Qwen3-8B-Q4_K_M | 8B | LLM |
| Gpt-oss-20b | 20B | LLM |
| Llama-2-7b-Q4_K_M | 7B | LLM |
| Mistral-7B-Instruct-v0.2-Q4_K_M | 7B | LLM |
| Kokoro (ONNX Runtime) | 82M | TTS |
| Microsoft SpeechT5 | 150M | TTS |
| Coqui XTTS-v2 | 400M | TTS |
Test configuration: 512-token prompt, 128 tokens generated, 24 CPU threads for LLM. 180-character input, 32 CPU threads, 30 runs per model for TTS.
AMD EPYC LLM Inference Benchmark Results: TTFT and Throughput
Time to First Token (TTFT)
TTFT across Q4 models ranged from 3.6 seconds (GPT-OSS-20B) to 4.8 seconds (Llama-2-7B). For GPT-OSS-20B, switching to FP16 had minimal effect. For DeepSeek, the same switch pushed TTFT to 8.1 seconds.
llama.cpp AMD EPYC Tokens Per Second: Q4 vs. FP16
DeepSeek drops from 27.8 tokens per second at Q4 to 8.1 at FP16. For sustained batch workloads, Q4 quantization is the practical choice on CPU.
Memory and CPU Utilization
CPU utilization stayed between 20–30% across all runs. Q4 models leave substantial DRAM headroom for multi-tenant deployments. DeepSeek at FP16 consumed close to 16 GB, which limits concurrent instances considerably.
CPU vs. GPU Inference: Cost Per Token Comparison
Cost per token determines the right hardware. For reference, we ran the same FP16 throughput test on an NVIDIA L4 GPU. The L4 produced 16.7 tokens/sec on DeepSeek-R1-8B and 58.6 on GPT-OSS-20B, compared to 8.1 and 26.2 on the EPYC 9334.
The GPU delivers roughly double the throughput. Bare metal EPYC servers run at a fraction of the hourly cost.
| Metric | AMD EPYC 9334 (dual) | NVIDIA L4 GPU |
|---|---|---|
| DeepSeek-R1-8B throughput (FP16) | 8.1 tok/s | 16.7 tok/s |
| GPT-OSS-20B throughput (FP16) | 26.2 tok/s | 58.6 tok/s |
| Relative throughput | ~0.45× | 1× |
| Typical hourly cost | Lower | 3–5× higher |
| Cost-per-token advantage | ✅ Batch/async | ✅ Latency-critical |
If you are deciding between GPU options for latency-critical workloads, our bare metal GPU comparison covering H100, H200, and L4 walks through when each makes sense.
TTS Inference on AMD EPYC: RTF Benchmarks
Kokoro (82M) — RTF 0.162
Generates audio roughly six times faster than real time. Consistent performance between p50 and p95. The standout result in this test set.
Microsoft SpeechT5 (150M) — RTF 0.6
Memory footprint around 1.4 GB. A solid option for single-speaker synthesis with limited memory.
Coqui XTTS-v2 (400M) — RTF 1.41
Supports voice cloning and multilingual synthesis but cannot serve real-time audio on this hardware. Best fit for batch audio generation or overnight processing jobs.
EPYC LLM Inference Benchmark vs GPU: How to Make the Call
CPU inference is a good fit for:
- Batch summarization and document processing
- Audio transcription queues
- Overnight report generation
- Edge deployments where GPU cost is a constraint
- Lightweight TTS workloads that do not require real-time synthesis
GPU inference is still the right call for:
- Real-time, latency-critical workloads at scale
- High-concurrency serving where throughput is the ceiling
- Models above 20B parameters without quantization
Running batch inference or TTS workloads? See our bare metal dedicated server configurations and pricing.