Mixed
I started benchmarking models on 10 of my real tasks ranging from easy to hard but even with the same model the completion time sometimes varied by as much as 50% between runs
knapcio on Hacker News
completion time sometimes varied by as much as 50% between runs
In the thread “Claude Opus 5.5”, replying to this comment
Helped: Regular use of multiple models for different tasks developed general feel.
Got in the way: High variance in completion times made benchmarking 10 tasks meaningless.
Well, to be honest it’s all based on my subjective experience. I started benchmarking models on 10 of my real tasks ranging from easy to hard but even with the same model the completion time sometimes varied by as much as 50% between runs. That made me realize you’d probably need hundreds of tasks to get any meaningful numbers. …