This article is cross posted from the 7digital Blogs From the Geeks:
I see lots of articles and discussions on how you measure the productivity of a development team. Having worked with cycle time and throughput for a few years now I’ve come to the conclusion that when combined these very simple measurements are a highly effective gauge of productivity.
Cycle Time
We measure cycle time as the count of working days between work starting (“in progress”) and completion (“done” or “live”) and is shown below per item over time. The chart below also shows a trend line (going down, nice) and error bars for the standard deviation:
Ideally you want to see the trendline on your cycle time chart going down, but more importantly you don’t want to see it going up. Shorter cycle times suggests we’re delivering value to the organisation quickly and do not have money unnecessarily tied up in inventory (unreleased code). It also means we’re more predictable and able to respond quickly to change.
To be truly useful (and much more difficult to game) cycle time has to measure the time it takes for things to really be done, not some “potentially shippable” nonsense. It’s also very simple to measure – you just need to track when the work starts and when it ends. We say a piece of work starts when it’s taken off the top of the prioritised queue and done when it’s been released to production (and verified). Sometimes we’ve done the analysis and requirements before hand, sometimes we haven’t (I don’t really care to be honest as we never allow more than 5-10 items in our prioritised queue so it’s not like we’re doing loads of unnecessary up front analysis or consider it a huge time drain).
Throughput
We measure throughput as a count of items completed per month:
Obviously the more work we do (with the same man power) the better (but probably less bugs and more features).
Individually these measurements are useful, but can be misleading if considered in isolation – cycle time going down is great, but if you’re only delivering one piece of work a month it’s not so great. Throughput can go up or down, but could be because you’ve more or less people in your team.
It’s hard to say if productivity can really be measured by just cycle time and throughput and being witness to metrics being gamed in the past (or causing undesirable behaviour) I’m wary of trying to improve them directly. However as a couple of really simple measurements we can track over time they’re a really good guide to support or refute anecdotal evidence as to whether anything we’re doing is having an impact on our ability to deliver. A great example is the two charts above. In around Sept/Oct they show one of our team’s cycle times going down and throughput going up, which was largely a result of changing tack and spending a considerable amount of time improving the build process and removing/fixing flaky tests.
What can you try to increase your team’s throughput and reduce cycle time? What can these measurements tell us about the impact of X happening?
There are of course many other ways to judge the effectiveness of a team and I wouldn’t want to get too caught up with just these ones, but it certainly pleased me greatly to see how the team above’s measurements improved and it certainly corroborated with the mood of the team since they managed to turn things around.
I’m of the mind increasing throughput and reducing cycle time is A Good Thing™. It generally means we’re getting more done, more quickly.
In my last post, I railed against the list makers and pondered why people are still so keen to pursue such fine grained detail. I hinted at something I’ve been thinking about for a while, that it comes down to who your working for.
Probably the biggest benefit of working so intimately with our customer has been the amount of work we don’t undertake. The problem with this is it’s very difficult (read impossible) to measure. The customer may be very happy, but there’s nothing to show for it. In fact, you could argue this works against us as we’re doing ourselves out of a job.
Managers can only evaluate our productivity on the work we do. Where I work our project manager is required to issue a progress report every 2 weeks. It consists of our spending (which I feel is quite justified), a list of all the stories we’ve been working on, their status and our velocity (in story points). If something is taking a lot longer than expected she is required to explain why. A lot of time is spent analysing our progress this way, but very little time getting feedback from our customer (shouldn’t there be a customer happiness rating in this bi-weekly report?).
However, I have sympathy with my managers. It’s their job to report on our progress to their managers, justify our existence and protect us from people looking for excuses to cut budgets. This is where I feel the irrational desire for measurement emanates. Somewhere up the line someone who is too busy to get their hands dirty with the detail wants a really simple way of diagnosing the health of a project (“Costing 50K a month but only delivering 20 points? Hmm… clearly something wrong there.”). If the velocity suddenly drops, does this necessarily mean there’s something wrong? Unfortunately anecdotal evidence or gut feelings don’t translate very well to very hierarchical structures (which I think says more about the structure).
Imagine this scenario: A bank has a crucial trading system which requires no development work for the foreseeable future. You have a small team of domain expert developers who’ve worked on the system for a long time, but now having nothing to do (they’re not doing nothing though, being enthusiastic devs they’re investigating new technologies and better ways of doings things as well a spiking new ideas which could improve the system). A manager sees a report which shows they’re no adding “no value” and makes them all redundant. Very soon after there is a problem with the system, it falls over and all trading has to cease until its fixed. As all the domain experts are no longer there it takes a very long time and the bank loses tens of millions. If the manager had kept the team on they’d have fixed it within a fraction of the time and saved the bank a fortune. If, rather than looking at productivity reports, the manager had appreciated the value of the system to the customer I doubt he would have made the decision he did.
Customers couldn’t give a stuff about velocity, story points or any other spurious measurement technique you care to come up with. They do care about you asking them questions and always seem delighted to be involved when we want to show them something or give them some feedback (tip: it’s important that this never takes more than 5-10 minutes). If we’re having problems we don’t try and hide them, we explain them to our customer in a way they’ll understand (so far they’ve been very understanding). Ultimately (and directly in our case), it’s our customer who’s paying the bills, not our managers.
If our ultimate goal is to provide as much value to our company as possible then its certainly not by trying to measure the highly subjective productivity of the developers*. If there’s any measurement to be done we should be focusing our efforts on how much we’re satisfying our customer’s needs. There are many ways we could approach this such as seeing how they use our software (or, more importantly, don’t), asking them to provide feedback, imagining if something we built was taken away and how it would impact them, roughly estimating how many people it would take to do the jobs our systems replace and so on. However, I feel it will take some people a lot of convincing that this is the way to go…
*Which, as Fowler points out, we cannot measure anyway
