rob bowley

I ran my session on estimation again at XPDay 2008 last week. I’ve come up with a decent title for it now: “Dealing with the Estimation Fallacy”.

It followed the same format as the previous time I ran it so you can go there to find the slides etc.

Here’s the output from the discussion. I’m running this session again at SPA 2009 and after that I will collate the output from all three into something coherent and useful:

How do you keep the context stable?
The problem is managing the context. If the business says you need to do it sooner that changes the context.
What about if the business doesn’t have the full context? What about issues you don’t know about?
Why are we different to any other type of organisation?
It’s too difficult to estimate more than 2 months ahead
Familiar work is easy to estimate, the unknown is not (which is mainly what we do)
If we report back to the business regularly they can’t deny hard facts
Manage the risk by working iteratively
It’s quite common for people to start building skyscrapers before the architects finish designing the top

• but their customer’s not going to ask for another 20 floors and more space halfway through

We need to help the customer understand what they need
People need to know whether it’s a viable business case
Projects are never agile if you have requirements up front (fine if nothing changes though)
You should only commit to work three months ahead
“2 years” for a software project is too long - will not be competitive
Movies spend a long time in development, then they get the budget and the scope gets cut to fit. But, they are more predictable than software.
Too much money (e.g. banks) will never produce good software - scope to big.
Are big estimates/commitments ever believed by anyone? Often they’re only used to make projects viable
IT departments are not pushing back enough - should never suggest you can plan more than 3 months in advance
2 types of estimation:

1. The real business of estimation (e.g. stories, iteration scope, tasks ect)
2. The game of estimating within the business context

Agile projects are rarely ever really agile as they’re operating withing the context of the business/traditional practices.

Firstly I asked the audience why we estimate?

• Predictability
• To be able to choose between options
• Planning
• So we can manage budgets
• To justify development
• Manage expectations
• Learn more about the problem
• Synchronise with other projects
• Get a cost benefit ratio
• Prioritisation
• Customers like to know when it’ll be ready

I then did a short presentation (slides below) explaining why we’re no good at estimating.

Finally we had a group discussion where we discussed my findings and what we can do. Here’s the output:

People often care more about the deadline as they only use 5% of the features

We need to create a better contract with the customer

When we need to synchronise with other projects what choice do we have?

We rarely measure by what we’ve achieved.

• Should be more like “we need this much business value for this much money”

Trust doesn’t exist - contracts are defense against the risk

That’s life! E.G. 2012 Olympics site build - people need to come together on a deadline

Agile enables us to react and focus on what’s important - is it scope? is it cost? is it deadline?

Also, iterative development means at least we deliver every two weeks to at least have something.

Measurement should be based on what’s been done.

However, estimation is used to measure productivity - we need better forms of measurement.

Estimation is not risk management but it’s what it’s used for.

Should the customer estimate value?

The closer you get to the end of a project the greater the pressure to estimate accurately.

Also is the impact of pressure from customer to estimate more accurately.

When integrating with 3rd parties need to estimate.

Need to recognise risk when estimating.

• Sponsors need to understand risk.

Time/duration of estimate = ^risk.

It’s about how we manage the estimate not the way we estimate where the answer lies.

• Managing risk/quantifying risk

If customers are more aware of risk they’ll make better decisions.

A good model would be to recognise the risk in the contract e.g.

x% risk = x cost

xx% risk = xx cost

Resources

Slides

Click here for ppt >>

Click here for pdf >>

Links

The Black Swan: The Impact of the Highly Improbable

Oliver Burkeman on why everything takes longer than you think

Planning Fallacy

Overcoming Bias: Planning Fallacy

Optimism Bias

Dept. of Helth Guidelines for Optimism Bias

Cone of Uncertainty Controversy

Predicting Velocity When Team Membership Or Size Changes Frequently | Mike Cohn’s Blog

Use Risk Management to Make Solid Commitments

There’s a lot more on my delicious bookmarks tagged for estimation

Mike has written a very thoughtful article here on my current favourite subject.

“Building a house” is perhaps the most overused software analogy out there. Sure, there are many overlaps and it’s a concept that non-technical people can grasp easily when we need to explain something, but it simply doesn’t add up. I’ve ran into this analogy frequently in my current obsession with estimation (someone even used it to comment on one of my recent articles) so I’ve been compelled to take this on as well ;-). Below is is a typical example of an argument I hear all the time. It was posted in response to this article on InfoQ:

If a house was like software and I was a contractor given the above conditions this is what I would do: I would build something that was usable and fulfilled the absolute minimum requirements as soon as possible (e.g. a wooden shed extension with a camping stove), see what the customer thinks and then rebuild it, say, every two weeks adding more and more of the requirements and responding to their feedback as we go, making sure that at the end of every two weeks the customer still had something they could use. In the end we’d have fulfilled as many of the requirements possible given the allocated money and time frame. We wouldn’t have gone over budget, been able to respond to unexpected events (e.g. finding out the foundations are not sufficient) and the customer could change his mind (on windows, room layouts etc) as we go.

Crucially, what the author of the above comment fails to grasp is the difference between incremental and iterative development (see end of article). Developing software (notice how no one says “building” software) is not like building a house - we have the luxury that it’s relatively inexpensive (if we’ve followed good design principles) to go back and change it as often as we like. I say, tell me honestly how much you’ve got to spend/how much time you have and I’ll give you the best possible value for those conditions. If an unexpected event occurred which means we can’t deliver it, well, it would have happened if we’d estimated it anyway. At least my way (well, it’s not my way is it…) we’re more likely to find out sooner and hopefully save the customer some money/embarrassment.

If you’re going to argue with me (which I dearly hope you will) please do not use this tired analogy. At least now rather than engaging with you I can point you this way and I don’t have to repeat myself

Further reading:

Jeff Patton explains on his blog the important difference between iterative and incremental development. - an exert from an excellent presentation I saw him give at XPDay 2007
The Software Construction Analogy is Broken. A very thorough article posted on kuro5hin a while back.

In my last post, I railed against the list makers and pondered why people are still so keen to pursue such fine grained detail. I hinted at something I’ve been thinking about for a while, that it comes down to who your working for.

Probably the biggest benefit of working so intimately with our customer has been the amount of work we don’t undertake. The problem with this is it’s very difficult (read impossible) to measure. The customer may be very happy, but there’s nothing to show for it. In fact, you could argue this works against us as we’re doing ourselves out of a job.

Managers can only evaluate our productivity on the work we do. Where I work our project manager is required to issue a progress report every 2 weeks. It consists of our spending (which I feel is quite justified), a list of all the stories we’ve been working on, their status and our velocity (in story points). If something is taking a lot longer than expected she is required to explain why. A lot of time is spent analysing our progress this way, but very little time getting feedback from our customer (shouldn’t there be a customer happiness rating in this bi-weekly report?).

However, I have sympathy with my managers. It’s their job to report on our progress to their managers, justify our existence and protect us from people looking for excuses to cut budgets. This is where I feel the irrational desire for measurement emanates. Somewhere up the line someone who is too busy to get their hands dirty with the detail wants a really simple way of diagnosing the health of a project (”Costing 50K a month but only delivering 20 points? Hmm… clearly something wrong there.”). If the velocity suddenly drops, does this necessarily mean there’s something wrong? Unfortunately anecdotal evidence or gut feelings don’t translate very well to very hierarchical structures (which I think says more about the structure).

Imagine this scenario: A bank has a crucial trading system which requires no development work for the foreseeable future. You have a small team of domain expert developers who’ve worked on the system for a long time, but now having nothing to do (they’re not doing nothing though, being enthusiastic devs they’re investigating new technologies and better ways of doings things as well a spiking new ideas which could improve the system). A manager sees a report which shows they’re no adding “no value” and makes them all redundant. Very soon after there is a problem with the system, it falls over and all trading has to cease until its fixed. As all the domain experts are no longer there it takes a very long time and the bank loses tens of millions. If the manager had kept the team on they’d have fixed it within a fraction of the time and saved the bank a fortune. If, rather than looking at productivity reports, the manager had appreciated the value of the system to the customer I doubt he would have made the decision he did.

Customers couldn’t give a stuff about velocity, story points or any other spurious measurement technique you care to come up with. They do care about you asking them questions and always seem delighted to be involved when we want to show them something or give them some feedback (tip: it’s important that this never takes more than 5-10 minutes). If we’re having problems we don’t try and hide them, we explain them to our customer in a way they’ll understand (so far they’ve been very understanding). Ultimately (and directly in our case), it’s our customer who’s paying the bills, not our managers.

If our ultimate goal is to provide as much value to our company as possible then its certainly not by trying to measure the highly subjective productivity of the developers*. If there’s any measurement to be done we should be focusing our efforts on how much we’re satisfying our customer’s needs. There are many ways we could approach this such as seeing how they use our software (or, more importantly, don’t), asking them to provide feedback, imagining if something we built was taken away and how it would impact them, roughly estimating how many people it would take to do the jobs our systems replace and so on. However, I feel it will take some people a lot of convincing that this is the way to go…

*Which, as Fowler points out, we cannot measure anyway

I felt compelled to respond to Jeff Atwood’s recent post written after reading Behind Closed Doors: Secrets of Great Management. In brief, Jeff explains that one of the main messages from the book is that as developers suffer from chronic “nearly done” or “90%” syndrome the only way for project managers to schedule and plan effectively is to get developers to list everything they need to do to finish a chunk of work and then estimate how long each of those tasks will take. If developers don’t have this list then their first task must be to make a list before they do any coding.

Well, that’s great and it would be a very effective strategy if the developer knew every single one of these tasks before they started. As I said in a previous post on estimation, the critical flaw is we don’t. We may think we know and might admit to being able to estimate 90% of these tasks, but we can’t estimate tasks we did not know we had to do and there’s no accountability for the unexpected (which as I’ve said before has an annoying habit of cropping up on an incredibly regular basis).* And what about invention?! The idea that having a list of all the things you need to do to complete a task will mean developers can provide accurate estimates is frankly laughable.

I usually agree with most of the stuff Jeff says so I was surprised to read this and even more surprised to find a similar article from Joel with lots of very disturbing graphs and charts explaining how you can accurately track velocity. There must be greater forces at work here. It’s certainly not driven by the need for increased productivity as it takes a lot of time to estimate in such detail (incredible detail if you use Joel’s “Evidence Based Scheduling” approach), time that would be in my opinion, much better spent just getting on with the work.

For me it all boils down to who your working for, your customer or your manager. I intend to cover this paradox in my next post…

* The view from this book (I haven’t read it and am only going off Jeff’’s article) also contradicts McConnell’s findings that estimates do not improve by putting more effort into creating them.

I’ve had a lot of reasons to think about estimation recently and I’ve come to a firm conclusion - it’s a complete waste of time. There are so many things you could be doing that will add value to your project - estimating adds nothing. In fact it has has the adverse effect of making your project less likely to succeed. I will explain why:

We cannot predict the unpredictable

More often than not, the factors that have the biggest impact on the duration of a project we simply did not see coming. Afterwards we look back and say “ah, well we know that for next time so we won’t make the same mistake again”. We investigate the reasons things went wrong, blame people or processes and move on to the next challenge confident this time it will be a success. This is the fatal flaw. What we don’t recognise is that the problem was not the particular event that delayed the project, but that something unexpected happened at all. This is an extremely costly mistake which eventually ends with people losing their jobs and a lot of money being thrown away. Some people may argue that when they estimate they allow for this by applying a “margin of error”. How much then? 5, 10, 20 percent? The problem with these unpredictable events or Black Swans is that no margin of error could possibly account for them, especially so if the object of your estimate is to win business or commit your organisation’s finances for the next X months. Unfortunately its in the nature of our business that we will constantly be presented with “unknown unknowns” and the sooner we realise this the better.

Even without these “unpredictable” events. We are useless at predicting the future

Until recently, I was a believer in McConnell’s Cone of Uncertainty which argues that the further away you are from a project deadline the more exponentially unreliable your estimates will be (this is not improved by putting more effort into the estimation process). Well I now think this is invalid. For one thing the graph is symmetrical. If this was based on reality it would mean we overestimate as much as we underestimate. If that was the case we would deliver early on as many projects as we deliver late (stop laughing). Also, it suggests that our estimates get better as the project progresses. Even with iterative development and when we estimate at the last responsible moment (e.g. the week before) and assuming no big surprises came our way (which always do), I have found we are mostly way out (I would consider anything above a 10% error margin to be enough to make it a worthless exercise). On the project I’ve been working on for over a year now, with roughly the same team (a really good team, the best I’ve ever worked with), the accuracy of our estimation has not improved in the slightest.* All we can say is (assuming no Black Swans come our way which as I’ve stressed, always do) the closer we get to the finish line (i.e. the less work there is in the backlog) the less there is to go wrong.

It is not in the interests of the customer

If the idea is to give our customers something they can use to forecast budgets then we’re not doing it. As we cannot predict the future, what we end up giving them is next to useless, in fact its likely to have a detrimental effect by lulling them into a false sense of security and dissuading them from allowing for uncertainty in their budgeting.

Dr Dobbs’ Journal did a survey on how we define success. They found:

• 61.3 percent of respondents said that it is more important to deliver a system when it is ready to be shipped than to deliver it on time.
• 87.3 percent said that meeting the actual needs of stakeholders is more important than building the system to specification.
• 79.6 percent said that providing the best return on investment (ROI) is more important than delivering a system under budget.
• 87.3 percent said that delivering high quality is more important than delivering on time and on budget.

So why are we so obsessed with it? The most common criticism I hear of agile methodologies is if a customer is given the choice between a company that says they’ll deliver in X months and cost £X and one that will not promise anything (sic) they’re bound to go with the former. Well, the survey above would suggest otherwise, as would I. In my last job I was in the position of choosing an agency to build a website and can assure you the last thing on our mind was how good they were at meeting deadlines. We were most impressed by the agency (sadly now defunct) who, for their pitch, did research into our customers and actually started building the site rather than knocking up any estimates.

What about when projects deliver on time and on budget?

Whilst some projects do deliver on time and on budget much of this can accounted for by chance rather than excellent estimation skills. These projects get scrutinised for what went so well (at least they should if your organisation is in any way decent) and the lessons are taken away to the next project. However whilst some of the lessons learnt may well be valid, no consideration is given to the enormous impact of blind luck! Adversely to when projects go bad, people and processes are given too much credit for success. This all results in a confirmation bias. Every time you do this is like looking for a higher piece of cliff top to balance on the edge of.

Conclusion

Estimates are good for one thing - showing how pointless estimating is. We are able to use them track a project’s progress and show where events took it on a different course that no one had expected.

Only working in an iterative process where you’re presenting your productivity to the customer on a regular basis are they going to be in a position where they can make informed decisions on the effectiveness and ongoing viability of the work being undertaken. Fail faster, fail better.

* Instead of trying to improve our estimates (again) we decided to spend less time doing it. In our sprint planning meeting we no longer break our stories down into tasks. Therefore we do not measure our progress during the sprint in such detail. So far this has had no adverse effect, but has had the effect of freeing up many hours of development time.