Yes
I've found that these models are extremely good at reasoning since late last year, but they're basically 200 IQ Amelia Bedelias
ndriscoll on Hacker News
In the thread “Google fixed more Chrome bugs in June than over the past two years, thanks to AI”, replying to this comment
Helped: Iterating on context docs and treating it as a partner.
Got in the way: Initial lack of context caused failures.
I've found that these models are extremely good at reasoning since late last year, but they're basically 200 IQ Amelia Bedelias. So my working model earlier this year was to give it a brief instruction and watch how it failed, which was generally a lack of context that it couldn't have possibly known. …