It turns out, not very good at all
From Jan Wildeboer
#Oops. A benchmark that checks how good "AI" coding works on real code from real companies. The TL;DR: Nope, it doesn't work. Not even close.
"Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. [...] We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions."
This is supposed to be one of the main uses of "AI". How soon before the bubble pops?
AI might be good at drawing elves or creating cat videos, but how lucrative a use case is that?
No comments:
Post a Comment