Do we do better with LLMs - or do they delude us into thinking so?

Are perceptions accurate when it comes to AI?

More and more, we're learning that Chatbots ("AI") mess with human perception.

It is hard for even the most experienced users to accurately assess the benefits of AI use.

Someone tried to measure how much AI speeds up coding when in the hands of experienced developers...

And it turns out that when experienced developers think AI gave them a speed-up (average: +20%), the cold, hard fact is that it slows them down (average: -20%).

The paper: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer

The blog post and an excellent review by Steve Newman

Thanks to METR for this important research 👏