It identifies the worst offending flaky tests, forms hypotheses on whether it's a testing issue or a production issue, then tries to divide-and-conquer until it gets minimal reproduction steps
SeanAnderson on Hacker News
shaved 20 minutes off their pipeline and improved pass rate by 35%
In the thread “Claude Code refuses requests or charges extra if your commits mention "OpenClaw"”, replying to this comment
Helped: Monitoring flaky tests, generating assessments, building training playgrounds for non-engineers.
Got in the way: Some bad solutions had to be thrown away.
… I came in, set Claude up, gave it read access to CI artifacts, had it build out some tooling to monitor the rolling pass/fail rate over the last 30 days, and let it loose. It identifies the worst offending flaky tests, forms hypotheses on whether it's a testing issue or a production issue, then tries to divide-and-conquer until it gets minimal reproduction steps. If it's not able to create deterministic reproduction then it'll make a best guess at fixing the issue and grind away at test re-runs all night until it can try to figure out if it fixed the issue with statistical confidence instead. …