Mixed
Both found it, in a similar amount of time. Several senior engineers had stared at the code for literally days and didn't find it
rstuart4133 on Hacker News
15172580 instead of 15172579
In the thread “The beginning of scarcity in AI”, replying to this comment
Helped: Finding subtle race condition bugs that senior engineers missed.
Got in the way: Producing good code; models only produce crap that passes tests.
… I've asked GLM-5.1 and Opus 4.6 to find a bug caused by a subtle race condition (the race condition leads to a number being 15172580 instead of 15172579 after about 3 months of CPU time). Both found it, in a similar amount of time. Several senior engineers had stared at the code for literally days and didn't find it. There is no doubt the models do vary in performance at various tasks, but we are talking the difference between Ferrari vs Mercedes in F1. …