How do Generative AI tools impact software developer productivity and code quality? A recent large-scale study – one of the most empirically robust I’ve seen – tackled this question by analysing 218k+ developers across 880m+ commits (with control groups and everything).

The results? A modest but consistent 4% productivity boost without sacrificing code quality.

Other key findings:

– Moderate GenAI users emerged as the highest overall performers (i.e. less is more)
– Only 1% of developers committed GenAI-authored code without significant rework.

It’s a nice 4% gain right? Not exactly game-changing. but worth it if licence costs stay reasonable. Enough to reduce a team of 100 to 96 engineers (or boost output by 4%)?

Not that simple – the study is somewhat flawed because it’s measuring effort not outcomes and individuals and not teams. The latest DORA Report (the gold standard when it comes to measuring high performing tech teams) found AI Tools marginally improving individual productivity, but having a NEGATIVE impact on overall team productivity.

As Giovanni Asproni said to me on Bluesky:

“I think that better product management, with a decent strategy, and proper prioritisation and planning will increase output far more than 4% in most organisations. And it may actually help saving energy and CO2 emissions.”

I ran a LinkedIn poll asking software developers about their experiences using AI coding assistants with production code. The results were interesting 👇

Why production code? Developers spend most of their time writing it, and AI tools can handle non-critical activities like proof-of-concepts, solo projects, and quick experiments fairly well.

I received 50 responses – not a huge sample, but interesting trends emerged nonetheless. Also, LinkedIn polls aren’t anonymous (which I didn’t know!), so I could see who voted. It allowed me to exclude non-developers and gauge experience levels from LinkedIn profiles and add it to the data.

You can see the results broken down by answer and experience level in the chart attached.

What stood out to me?

– Only one respondent – an apprentice engineer – said they “automate most things.”

– 74% found AI assistants helpful, but 26% felt they’re more trouble than they’re worth.

– All who said AI assistants “get in the way” (24%) had 8+ years of experience.

– 80% of respondents had 8+ years’ experience (I know a few of them, and others’ profiles suggest they’re legitimately experienced engineers).

My thoughts? Unless they improve significantly:

1) They’re unlikely to shift productivity dramatically. From my experience, “task” work takes about 10-30% of a developer’s coding time. So, while helpful, they’re not game-changing.

2) That a significant portion – especially experienced engineers – find these tools intrusive is concerning.

3) Having heard stories of AI tools introducing subtle bugs, I was surprised not to see more votes for “considered harmful.”

4) The idea that AI tools might replace software engineers – or that even “conversational programming” is anywhere close – still feels a very long way off.

5) I worry about an apprentice software engineer getting an AI tool to do most of their work. Partly because more experienced developers are a lot more cautious and must have a reason for this, but mainly because they won’t be learning much by doing so.