I ended up having to point it out directly the next day. It was a spurious -A NUM argument that would, in a production scenario, cause it to recognize valid data as invalid because the full context of the data would be missing
rjbwork on Hacker News
In the thread “Unpredictable black boxes are terrible interfaces”, replying to this comment
Helped: None stated; tool generated code but required human review.
Got in the way: Generated a spurious argument causing logic errors in production scenarios.
… It was near the end of the day so I asked them to follow up with me in the morning after they'd found and fixed the bug. I ended up having to point it out directly the next day. It was a spurious -A NUM argument that would, in a production scenario, cause it to recognize valid data as invalid because the full context of the data would be missing. All this to say, it is a very useful tool, but we should be highly suspicious of its output, and really make sure we understand what it's giving us. …