In the short time I’ve been involved with the community I’ve seen almost no discussion or articles which involve any contribution from people outside of the IT department and none of the methodologies/name clouds I can think of appear to have been developed or evolved with any collaboration from the people driving the work.

I don’t mean to disparage all the hard work and enthusiasm people (I include myself) have put in to trying to make things better, I just think the fact that we’ve left out the most important people from our discussions has caused us a lot unecessary of pain.  Agile adoption is never a smooth affair, especially the process of “convincing” people outside of the development department. If we involved our customers more in the process of forming our principles, tools and methodologies we would surely get where we all want to be a lot more quickly. Remember, people don’t resist change, they resist being changed.

Here are some examples of what I mean:

• Speaking to my CEO in the pub the other day he said he often finds it “patronising” when we try and impress on him the importance of the the things we’re trying to do.
• When I first started with Scrum I countlessly came up against resistance and cynicism when trying to encourage customers to participate in traditional Scrum meetings such as retrospectives, planning and stand ups. Nothing Scrum teaches prepares you for this (I’ve done both CSM and Estimating and Planning courses and read a lot otherwise).
• The terminology we use (e.g. Scrum, Sprints, Stories, Kanban, eXtreme Programming) is totally immature in some people’s eyes.  One useful takeaway from the session was to stop using inappropriate terminology in front of customers.

I have an enormous amount of respect for our CEO and others within our organisation for putting the trust they do in us even though they (understandably) find so much of what we do beguiling and irritating. In my experience you would be very lucky to be able to work with someone who is prepared to take that kind of risk – in most places I’ve worked you just don’t stand a chance.

For me, the flaw lies deep within Agile. It was never designed to address the needs of customers. XP and Scrum were designed for fixing dysfunctional environments. The terminology was designed to appeal to developers. When most of the Agile principles and methodologies were developed the need was different and they don’t appear to have evolved. New methodologies suffer the same – why does there have to be an introductory guide to Kanban for Managers? If we’re having to try and sell this stuff to people I think we’ve already lost most of the battle.

In the last 15 years or so most of the “microeconomic” problems with software development have been solved. The majority of people writing software may still not be doing it well, but the answers are there if you care to look. The big problems are still out there though and as a community we need to start addressing them and I think the only way we can do this is by getting our customers more involved in the debate.

This post is really a rallying call to all those that feel the same as I do. I’m keen to start doing something about this, but where do we start?

*I use the term customer here to mean anyone who is a customer of a development team.

Experience often counts for little in software development. If you’ve spent your whole career in the same shop with little exposure to other languages or people outside your organisation it’s quite possible that some 21 year old upstart with a copy of Clean Code under his or her arm will wipe the floor with you when it comes to effectively writing and maintaining software.

Granted, working with good or even great developers will mean a lot will rub off on you. I’ve learnt countless lessons from the people I’ve worked with, but if I look around me people are no older than 30 at most with an average of around 5 years developing software in probably no more than 3 different organisations.

People like Martin Fowler, Eric Gamma, Kent Beck, Robert C Martin, Craig Larman and Michael Feathers have been at it for 25 years or more and in that time have slowly built up the kind of reputation you only get from regularly being right.

Also granted, blogs are are an invaluable resource, but are rarely little more then a meme in someone’s head and give you nothing like the deep contextual insight you can get from a well written book. There is also little to assure you that the author is anymore likely to have a better idea than yourself. Believe me when I say there are many people blogging who rarely live up to the practices they preach and are no more or less likely than you to know the right or wrong way of doing something.

I have learnt a lot from both colleagues and blogs, but both pale to what I’ve learnt from the books I’ve read. I can comfortably say there is no way I would be where I am today without them and I strongly believe it will earn you more money. When you think of all the things you could do to try and put more folding stuff in you’re back pocket it’s a relatively simple win!

I’ve been inspired to write this after reading Eric Evans Domain Driven Design which has gone right into my top 5 books of all time. Why is it so good? It’s not because Eric has necessarily been born with some supernatural instinct for writing great software or that Domain Driven Design is going to save the planet. It’s because it’s full of the lessons that Eric has learnt in his long and illustrious career, carefully woven into a highly readable narrative. There’s nothing particularly new here. Like all the great books I’ve read it is no more than a distillation of the practices in the industry which through time have proven to be the most effective. I remember Martin Fowler once saying that people often asked him what would be the future of software development. His answer was that to see the future you only have to look to the past.

Below is a list of the books which significantly changed the way I think and work more than any other’s I’ve read. No doubt you’ve heard of them all already and are on a list of books to read in the back of your mind somewhere and I’m sure there are plenty of others that had as significant an impact on people as these have on me. However I think few will argue that any of these books do not deserve their place on this list. All I can say is get on and read them if you haven’t already.

Refactoring: Improving the Design of Existing Code by Martin Fowler
Domain Driven Design: Tackling Conplexity in the Heart of Software by Eric Evans
Working Effectively with Legacy Code by Micheal Feathers
Agile Software Development: Principles, Patterns and Practices by Robert C Martin
Clean Code: A Handbook of Agile Software Crafstmanship by Robert C Martin
xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros
Lean Software Development by Mary and Tom Poppendieck
The Toyota Way: 14 Management Principles from the World’s Greatest Manufacturer by Jeffry LIker

*I am in no way professing to be a top class developer

Certainly not enough to be able to draw a clear line between the two and start comparing the benefits of one against the other. Daniel Jones, the author of The Machine That Changed the World did a keynote at XPDay2008 and said from what he saw there wasn’t really much difference, apart from the name. In fact, when they were looking for a name for Lean they thought about using Agile.

Agile came from Lean, the related processes came from people studying Demming, The Toyota Production System or through convergent evolution. The two guys who invented Scrum are Japanese and that’s no coincidence.  When I started reading about Toyota and Demming it all simply made sense to me and explained where Agile came from.

Then I found an article written by Craig Larman on the history of iterative development and that wrapped it up: http://www2.umassd.edu/SWPI/xp/articles/r6047.pdf

There is no point spending any time discussing and trying to find differences between what is essentially the same thing just written in a different way.

It reminds me of how the Jews and Christians went their separate ways and then came the Muslims and then they split up into the Sunni and Shia’s. They’ve all been fighting bitterly for the last 2000 years but all worship the same god. Go figure!

Recently there’s been a lot of (and subsequently criticism of) debate on message boards and blogs about the relative benefits of one paradigm versus the other.  Personally I don’t care much for subscribing to any particular paradigm and am much more interested in what works and what doesn’t (and in which circumstances) so my response is to publish what my team and company is actually doing right now. This is a snapshot of our current practices. Ask me again in 6 months and hopefully I’ll show you something very different.

The inspiration and influences for the way we work mostly come from Agile, Scrum, Extreme Programming, Software Craftsmanship, Lean, Lean Software Development, Kanban Software Development and The Toyota Way. It is all and none of these things.

Context

We are a small to medium size “start up” organisation working in the new media industry. The company employs around 60 people mainly based in the UK. The development department numbers around 20 co-located people. Agile practices are a relatively new introduction and the previous approach was of the typically chaotic type familiar to young businesses.  We mainly work in .Net using C# but also dabble in Ruby, Javascript and UI languages. The rest of the article mostly relates specifically to one of the teams working within the department but also addresses some of the practices of the department as a whole.

Team

The team is made up of 4-5 developers and a tester. There is no project manager at the team level – in the spirit of self-organisation (principle 11) the duties traditionally the responsibility of a PM are shared between the team members. The Product Owner role is shared between the stakeholders within the organisation for the products the team is responsible for.

Iterations

We currently work in 1 week Iterations. It’s a new team who are also new to many of the agile concepts and doing this enables us to control the amount of work in progress, focus on delivery, improve our discipline and most importantly have short feedback cycles so improvements can be discussed and applied frequently. The downside is the overhead created by the amount of meetings. Once we’re comfortable the team is working well together we will have the opportunity to change this if desired (e.g. Continuous Flow, changing the iteration length, changing the frequency of meetings).

Meetings

Each iteration we have the following meetings:

Work Prioritisation – occurs iteration minus one. Stakeholders come together to raise and prioritise work not yet commited to.
Requirements Gathering – occurs ad hoc when necessary. All the team is required to attend along with the customer/s to bash out requirements for work prioritised in the prioritisation meeting.
Planning – occurs at the beggining of the iteration. Prioritised features (MMFs) which have been analysed are broken down into stories, discussed, estimated and commited to based on our current velocity (avg. over last 6 weeks)
Stand Up – occurs daily at 10am at the task board. Anyone outside the team is welcome to watch
Retrospective – occurs at the end of the iteration. Any actions from the meeting are to be completed by the end of the next iteration.

Requirements

Features are requested at the prioritisation session and use the User Story format.

More detailed requirements are gathered during the requirements meetings mentioned above, with the customer/s and all team members present. We use whiteboards to bash out the requirements and convert them into acceptance criteria using the “Given, When, Then” format. We have a rule that no work can be commited to unless we’re happy we have a clear understanding of the requirements.

Task Board

The task board is essentially a Kanban board with each stage of the delivery process separated into columns. We have an implicit limit of 2 stories in active, but otherwise have not applied limits to any other columns. Features (MMF) are blue, the stories which make up the MMFs are yellow, bugs are pink and quick support tasks white. When a story is commited to, the feature card is moved into “commited”, above the titles of the columns and tracks the last related story.

Measurement & Metrics

We use an Excel spreadsheet to hold the product backlog and track the data from the Kanban/Task board. Whenever an MMF moves to another column the date this occured is recorded. You can download a copy of the spreadsheet here (you may want to check the calcs on the CFD, not sure they’re right). Among other things it calculates average cycle time, average velocity and projections based on velocity. I’ve tried a few bespoke tracking tools (such as Mingle) and found nothing is as powerful and flexible as Excel.

We have a manual Cumulative Flow Diagram (CFD) which each team member takes turns to update daily so everyone shares the responsibility (it is also their job to update the Excel spreadsheet each day). The CFD diagram only tracks the value delivered to the business (one unit = one MMF. Measure the output, not the input) and is also represented in the Excel spreadsheet. Why have both you may ask? Visibility.

We have some rudamentry code metrics set up through our continuous integration framework such as NDepend output and test coverage but are working towards something more visible and useful.

Estimation

Still very much a necessary evil.

For comitting to work for an iteration we use Story Points using the fibernache (1, 2, 3, 5, 8…) sequence and achieve them by playing Planning Poker with everyone who may be involved in the work required to take part. We will only estimate (and commit to) work we have already analysed and gathered requirements for.

For longer term planning, as we don’t yet have enough information to be able to use cycle time for projecting work completion, using the velocity based on points completed per iteration has proven a very powerful toool to be able to give the rest of the business a better idea of our capacity and timescales (previously they had none). However this has well known drawbacks and we must be careful it does not get abused, as I have seen before (such as gaming of estimates, whether intentionally on subconsciously). Also, as we need to understand and have gathered the requirements to be able estimate this way it means there is very limited scope to how far into the future we can do this with any degree of confidence (as requirements will change). Once we have a reasonable amount of data in the system we will be able to use average cycle time, which will be much more powerful.

Coding Practices

Apart from the rules we’ve commited to as a team, Pair Programming, Unit Testing, Refactoring and the best working principles and practices of the software industry are encouraged from the top of the department and applied rigorously but pragmatically.

At the request of the department members (as a result of a disucssion on collective responsibility) we created a development standards document which includes topics such as naming conventions and testing. As much as possible the document is vague on implementation details to prevent it from holding us back when better working practices come along. We use shared Resharper and Visual Studio settings to help us keep to these standards.

As mentioned below we also frequently hold sessions to improve our skills.

Automated Testing

All new or modified code is covered by unit tests, integration tests (such as database interaction) and automated acceptance tests which test against the acceptance criteria (this last one is quite new territory).

Continuous Integration and Deployment

All projects are under continuous integration (we use TeamCity) and we are working towards having all deployments doing the same. We have monitors on the wall which show all the currently failing builds. Do I need to mention we use source control?

Roles and Responsibilities

Every role in the department is covered by a document explaining their roles and responsibilities. They are written in a way which encourages self-organisation and collective responsiblity. You can download them here. I will be talking about these descriptions more in a future article.

Learning Culture

Each week two hours are set aside for learning sessions such as coding dojos and presentations (we’re currently running a design patterns study group). Outside of these developers are actively encouraged to take the time to learn new practices during working hours (within reason). We have a library of books available on a range of subjects which are at the disposal of everyone. More often than not if there’s a book that someone would like to read the company will purchase it and add it to the library (books are pretty cheap in the grand scheme of things).

Continuous Improvement

Outside of retrospectives we have a monthly departmental session where the most pressing problems are discussed and actions taken away. However there is no limit or retriction to when improvements can be made and everyone is encouraged to take the initiative when they see a problem that needs addressing.

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity–the art of maximizing the amount of work not done–is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

The Five Principles of Lean

1. Value – specify what creates value from the customer’s perspective.
2. The value stream – identify all the steps along the process chain.
3. Flow – make the value process flow.
4. Pull – make only what is needed by the customer (short term response to the customer’s rate of demand).
5. Perfection – strive for perfection by continually attempting to produce exactly what the customer wants.

The Seven Principles of Lean Software Development

1. Eliminate waste
2. Amplify learning
3. Decide as late as possible
4. Deliver as fast as possible
5. Empower the team
6. Build integrity in
7. See the whole

I. Having a long-term philosophy that drives a long-term approach to building a learning organization

1. Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals

II. The right process will produce the right results

2. Create a continuous process flow to bring problems to the surface

3. Use “pull” systems to avoid overproduction

4. Level out the workload (heijunka). (Work like the tortoise, not the hare)

5. Build a culture of stopping to fix problems, to get quality right the first time

6. Standardized tasks and processes are the foundation for continuous improvement and employee empowerment

7. Use visual control so no problems are hidden

8. Use only reliable, thoroughly tested technology that serves your people and processes

III. Add value to the organization by developing its people and partners

9. Grow leaders who thoroughly understand the work, live the philosophy, and teach it to others

10. Develop exceptional people and teams who follow your company’s philosophy

11. Respect your extended network of partners and suppliers by challenging them and helping them improve

IV. Continuously solving root problems to drive organizational learning

12. Go and see for yourself to thoroughly understand the situation (Genchi Genbutsu).

13. Make decisions slowly by consensus, thoroughly considering all options; implement decisions rapidly (Nemawashi).

14. Become a learning organization through relentless reflection (hansei) and continuous improvement (Кaizen).

• No multi-tasking
• All new or changed code must be thoroughly unit tested
• No more than 2 pieces of work in “Active Work” at any time (our team is 3 developers)
• We always work on the highest priority task
• Our definition of done is “In UAT”
• Leave it in a better condition than you found it
• No hidden work
• No overtime
• No disruptions
• Don’t SysTest your own work (we don’t have a tester yet)

Firstly, if you haven’t read it, I’d highly recommend Agile Retrospectives: Making Good Teams Great by Darby, Larsen & Schwaber. A great introduction to retrospectives with a lot of plans and ideas to help you out (although I have to say I find many of them too long for my liking).

However, beyond that book there really isn’t much else available so I’ve started a wiki to collect retrospective resources. Without further ado I’m delighted to announce:

The Agile Retrospective Resource Wiki or http://retrospectivewiki.org

A plea for contributors

Everything else you need to know is on the wiki, but I would like to take this opportunity make a plea for contributors. The content is currently a bit limited as I’m the only contributor so far so if you’ve got any cool retrospective plans, tips, tricks or anything else please add them.

Along with Matt Wynne I’ll be running the Retrospective Surgery session mentioned on the wiki at SPA2009 next week so I expect we’ll be getting some juicy new content imminently.

“In sales, they teach that the first objection is often a false objection–misdirection rather than a real concern. “Does Agile scale?” can be one of these false objections.

I wasn’t really being asked “Does Agile scale?” (By now, though, we know the answer: yes.) What I was really being asked was, “Does Agile work in large, dysfunctional organizations? Can I keep doing all of the ineffective things I’m required to do and still say I’m Agile? I can’t have a co-located team–it’s out of my control. I can’t have active, involved product managers–they’re too busy. I can’t create cross-functional teams–it would disrupt our reporting hierarchy. How does Agile scale to mysituation?”

It doesn’t… at least, not well. Welcome to Mediocrity.”

I’m slightly more sympathetic than James (who has clearly become frustrated seeing the same pattern over and over again) having worked in a large organisation adopting agile and see how we can quickly get to agile mediocrity, but struggle to go any further. Typically the catalyst for change is some enthusiastic developers and a forward thinking dev manager bold enough to take a chance that the Agile gumpf these guys keep going on about might just solve a few problems. This has some success, which gets jumped on by higher management who are lured in by the buzz and enthusiasm and it gets rolled out to other teams, however then it starts hitting walls. Firstly is the resistance within the department (and this is relatively easy to get over), but then it becomes clear (to the agile teams) that to achieve real results change needs to occur further afield and here’s where it all starts to seize up.

Agile is a bit like dropping a bomb in an organisation in that it creates shock waves which spread outside the IT department and have the potential to change the entire organisation (after all agile is the child of lean production which can be applied everywhere ). The problem is the wider the shock waves spread the more difficult the change is. Large bureaucratic organisations will always suffer from more inertia and change will be slower. I guess here is where a lot of people might get frustrated and start saying things like “it doesn’t scale”. The big mistake is blaming agile for your organisational inefficiencies. It’s not the fault of agile that your company is a lumbering old dinosaur that shares the characteristics of a supertanker when trying to change direction

In my mind this is largely a management problem. They’re often very keen on Agile when it doesn’t mean they actually have to do anything. A self-motivated software team trying to make things better? Great! Restructure the organisation so the delivery team is sitting and speaking with the customer, employ in-house testers and allow time for learning in working hours? Mmm not so attractive.

It’s not sufficient just to roll out Scrum or install a Kanban board. As I’ve said similarly before to succeed with agile you need to look past the latest fashionable methodology and really understand the underlying principles. The Agile Manifesto and the principles behind it would be a good place to start.

Rather than explaining why else I think rewriting a legacy system is generally always the wrong strategy see what Bob Martin, Chad Fowler and Joel Spolsky have to say.

But anyway, what’s this got to do with Real Options analysis and planning? Taking a Real Options approach would very rarely if ever end in the decision to go ahead with a massive rewrite. The calculated risk level to return on investment involved is highly unlikely to be favourable due to the protracted timescale (compared to the incremental addition of features which will be adding value in a short period of time). Also, if you were pursuing multiple options concurrently – aware that one would be dropped once there was a clearer idea of the most appropriate strategy – you’re less likely to get into the situation where you’ve invested so much into one option the momentum and expense (with nothing to show) make it difficult to admit defeat and drop it. With Real Options you’d never put all your eggs in one basket, which is exactly what the “Big Rewrite” does do.

“Real options recognize that today’s investments give investors the choice of pursuing further investments later, if conditions appear favorable, or abandoning the project if the environment has deteriorated. The capital investment made today provides future flexibility that can and must be valued, but is often missed by traditional DCF or ROI measures. Borrowing from both finance and strategy, real options can provide a way to analyze the value of investing in initiatives made under uncertainty.” – Dave Latimer, the Global Financial Services Sector Lead for the Institute for Business Value

Real Options resources:

Wikipedia: Real Options Analysis
“Real Options” Underlie Agile Practices – Chris Matts and Olav Maassen
Calculating value during uncertainty: Getting real with “real options”(pdf) – Dave Latimer
blog.mattwynne.net : My Real Options Story

One of the most interesting things he said was that when they were coming up with the name for what they were trying to promote they considered “agile”, but thought it would be too difficult to sell and so decided on Lean instead. Also, he didn’t have much understanding of modern software development, but at a glance saw very little difference between what we called Agile and he called Lean.

So when we talk about Lean Software Development being an Agile methodology we’ve got it the wrong way around. Agile is Lean, Scrum is Lean, XP is Lean. You were already doing Lean Software Development, you just didn’t know it!

In my mind it’s time to drop the titles (which all carry too much stigma) and simply start referring to it as professional software development.