Yes
It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic
springtimesun on Hacker News
In the thread “Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases”
Helped: Sandboxing agents with specific commit context and grading against accepted PRs.
Got in the way: Finding realistic examples because real dev flows rarely follow simple ticket-to-PR paths.
… Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space. …