a very interesting post and I can see that you speak from personal experience so this is clearly valuable learning that can and has been usefully and successfully applied. And I also recognise that if you have no need to estimate then there’s no value in doing so. But I disagree with much of what you’ve written based on my own experiences.

One of my key learnings from the early days of XP was that its approach to ‘the only constant is change’ wasn’t to try to prevent change in a ‘freeze the requirements’ sense, nor to say there’s nothing we can do about change, but rather to equip the team with the tools that allow them to calmly and quickly respond to change. So the value of an iteration plan (and the estimates that inform it) is to say: “all things being equal we think we can deliver this”. When change happens those estimates allow us to rapidly judge the impact and respond accordingly (which might be to scrap the iteration, carry on until the end regardless, or do something about the change right away). This also holds for release plans, estimated at a much lower degree of accuracy. You’re absolutely right that we can’t predict the unpredictable but estimation and planning aren’t about predicting; they’re about saying “given what we know today this is how we think things will play out, when something changes, these plans will change”.

On the subject of budgets, they are a fact of life in any non-trivial business and I can’t agree that the only response software teams can give is “we are no good at working within budgets”. Agile teams are best managed by burn rate and a simple application of burn rate/velocity (“in the last three months we’ve spent this to deliver this”) can be to judge whether a budget should be increased or decreased or the capacity altered accordingly. Of course, this is generally pretty crude but then so are most budgets.There is the question of ROI vs. budgets but without some ability to estimate and plan, even rather crudely, how will a business ever get to choose which model it wants or needs to apply?

What about when projects deliver on time and budget? Well I’ve seen teams consistently do this, time after time, without the need to pad or fudge. Was it blind luck? Maybe, although I don’t think so. What I believe is the investment was made to do estimates, get things wrong, learn through reflection and gradually improve (which included the ‘customers’ of those estimates getting used to the fact that they weren’t always accurate and far more useful in terms of understanding ‘net effect’ than knowing that Task A would definitely be done by Tuesday morning). Over time those estimates became more and more reliable and the ability to respond quickly and calmly to change, by being able to assess impact and re-jig work accordingly, improved.

For me this is just like TDD. In the early days with a new team they struggle to write the tests first (or at all). They wonder why all this effort is being made for an activity that has little obvious value (because the system is so small it is easy to manually test or predict the effect of a change). But over time tests become easier to write and as the test set grows with the system their value becomes more obvious. We invest in the TDD process because we know it will ‘come good’ and I think estimation is very similar in that respect.

Best,

Paul