Mixed
Most of the time that will give you half to a third of decode throughput vs the benchmark
Jabdoa2 on Hacker News
60k-140k prompt with a few k of output
In the thread “Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware”
Helped: Proper parameter tuning to outperform internet benchmarks.
Got in the way: Benchmarks do not reflect real agentic coding workload performance.
… We found that real usage is more 60k-140k prompt with a few k of output for realistic agentic coding workloads. Most of the time that will give you half to a third of decode throughput vs the benchmark. It also becomes much more about cache aware routing and kv cache management (how to offload and restore). …