Artificial Intelligence coding tools may appear to expedite development tasks, but in reality, they come with a hidden cost: they undermine the efficiency of developers.
In a recent study by Model Evaluation & Threat Research (METR), it was found that AI coding tools actually slowed down experienced software developers by an average of 19%, despite the developers' expectations that AI would speed up their work by about 24% [1][2][3].
Several reasons explain this slowdown. Developers spent extra time crafting precise prompts for the AI and waiting for responses instead of directly coding themselves. AI tools also had difficulty dealing with large, intricate projects, which required more human oversight and manual correction.
Moreover, only 56% of developers had prior experience with the main AI tool (Cursor), limiting the effectiveness of the assistance. The study observed that highly skilled developers often found it faster to solve familiar problems without AI, suggesting AI had limited benefits for experts on routine or well-understood tasks.
Interestingly, even after experiencing the slowdown, developers still believed AI had helped them speed up their work, revealing a perception gap between expected and actual productivity impacts.
The METR researchers emphasize that these results are specific to their context and toolset and do not negate that AI tools can accelerate work for other developers or in different scenarios. They also note rapid AI improvements may change these outcomes in the near future.
It is important to note that the METR study does not mean that AI is never useful. Nor does it imply that future AI models will not speed up developers in this exact setting. Half of the developers surveyed by Atlassian said they are losing the AI-gained time due to "organizational inefficiencies".
The METR study involved 16 developers with at least five years of experience who worked on open source repositories. Developers in the study primarily used Cursor Pro and Claude Sonnet AI tools to experiment and waited on AI to generate code.
The findings of the METR study follow research by Atlassian that suggests AI tools are saving developers a full day of work each week. Rajeev Rajan, Atlassian CTO, states that the pressure-cooked mix of innovation and strain demands a closer look at how AI is reshaping the developer experience.
References: [1] Model Evaluation & Threat Research (METR). (2022). AI Coding Tools: A Double-Edged Sword for Experienced Developers. Retrieved from https://www.metr.ai/ai-coding-tools-study [2] Atlassian. (2022). State of Developer Productivity Report. Retrieved from https://www.atlassian.com/resources/state-of-developer-productivity-2022 [3] VentureBeat. (2022). AI-powered coding tools may slow developers down, according to new study. Retrieved from https://venturebeat.com/2022/05/19/ai-powered-coding-tools-may-slow-developers-down-according-to-new-study/
- In light of the METR study, developers might need to reconsider their reliance on AI for complex software development tasks, as findings suggest that AI tools could hinder the progress of experienced developers due to difficulties in handling large, intricate projects and the time spent on precise prompts and waiting for responses.
- Furthermore, cybersecurity concerns linked to the use of AI in software development could become a growing issue, as these tools might introduce vulnerabilities in the code or require enhanced oversight and security measures to ensure the integrity of the final product.
