Yes
With that said, it’s been pretty useful at uncovering simple mistakes before another dev has had a chance to review
jumploops on Hacker News
In the thread “Using Large Language Models to Catch Vulnerabilities”
Helped: Catching simple mistakes in PR diffs before human review.
Got in the way: Errs on the side of medium severity too often.
… In practice it’s mostly right, but definitely errs on the side of medium too often (which is reasonable without the additional context of the rest of the codebase). With that said, it’s been pretty useful at uncovering simple mistakes before another dev has had a chance to review. [0] https://magicloops.dev/loop/3f3781f3-f987-4672-8500-bacbeefc...