It seems these days there are countless methodologies, processes and practices to choose from when developing software and, somewhat ironically, the list seems to be growing at the rate of Moore’s law. I’ve read about, discussed, been on courses and been to conferences about a lot of them and the thing I’ve consistently found most useful is talking to other practitioners about what they’re doing and what’s working (or not working) for them.
Recently there’s been a lot of (and subsequently criticism of) debate on message boards and blogs about the relative benefits of one paradigm versus the other. Personally I don’t care much for subscribing to any particular paradigm and am much more interested in what works and what doesn’t (and in which circumstances) so my response is to publish what my team and company is actually doing right now. This is a snapshot of our current practices. Ask me again in 6 months and hopefully I’ll show you something very different.
The inspiration and influences for the way we work mostly come from Agile, Scrum, Extreme Programming, Software Craftsmanship, Lean, Lean Software Development, Kanban Software Development and The Toyota Way. It is all and none of these things.
Context
We are a small to medium size “start up” organisation working in the new media industry. The company employs around 60 people mainly based in the UK. The development department numbers around 20 co-located people. Agile practices are a relatively new introduction and the previous approach was of the typically chaotic type familiar to young businesses. We mainly work in .Net using C# but also dabble in Ruby, Javascript and UI languages. The rest of the article mostly relates specifically to one of the teams working within the department but also addresses some of the practices of the department as a whole.
Team
The team is made up of 4-5 developers and a tester. There is no project manager at the team level – in the spirit of self-organisation (principle 11) the duties traditionally the responsibility of a PM are shared between the team members. The Product Owner role is shared between the stakeholders within the organisation for the products the team is responsible for.
Iterations
We currently work in 1 week Iterations. It’s a new team who are also new to many of the agile concepts and doing this enables us to control the amount of work in progress, focus on delivery, improve our discipline and most importantly have short feedback cycles so improvements can be discussed and applied frequently. The downside is the overhead created by the amount of meetings. Once we’re comfortable the team is working well together we will have the opportunity to change this if desired (e.g. Continuous Flow, changing the iteration length, changing the frequency of meetings).
Meetings
Each iteration we have the following meetings:
Work Prioritisation – occurs iteration minus one. Stakeholders come together to raise and prioritise work not yet commited to.
Requirements Gathering – occurs ad hoc when necessary. All the team is required to attend along with the customer/s to bash out requirements for work prioritised in the prioritisation meeting.
Planning – occurs at the beggining of the iteration. Prioritised features (MMFs) which have been analysed are broken down into stories, discussed, estimated and commited to based on our current velocity (avg. over last 6 weeks)
Stand Up – occurs daily at 10am at the task board. Anyone outside the team is welcome to watch
Retrospective – occurs at the end of the iteration. Any actions from the meeting are to be completed by the end of the next iteration.
Requirements
Features are requested at the prioritisation session and use the User Story format.
More detailed requirements are gathered during the requirements meetings mentioned above, with the customer/s and all team members present. We use whiteboards to bash out the requirements and convert them into acceptance criteria using the “Given, When, Then” format. We have a rule that no work can be commited to unless we’re happy we have a clear understanding of the requirements.
Task Board
The task board is essentially a Kanban board with each stage of the delivery process separated into columns. We have an implicit limit of 2 stories in active, but otherwise have not applied limits to any other columns. Features (MMF) are blue, the stories which make up the MMFs are yellow, bugs are pink and quick support tasks white. When a story is commited to, the feature card is moved into “commited”, above the titles of the columns and tracks the last related story.
Measurement & Metrics
We use an Excel spreadsheet to hold the product backlog and track the data from the Kanban/Task board. Whenever an MMF moves to another column the date this occured is recorded. You can download a copy of the spreadsheet here (you may want to check the calcs on the CFD, not sure they’re right). Among other things it calculates average cycle time, average velocity and projections based on velocity. I’ve tried a few bespoke tracking tools (such as Mingle) and found nothing is as powerful and flexible as Excel.
We have a manual Cumulative Flow Diagram (CFD) which each team member takes turns to update daily so everyone shares the responsibility (it is also their job to update the Excel spreadsheet each day). The CFD diagram only tracks the value delivered to the business (one unit = one MMF. Measure the output, not the input) and is also represented in the Excel spreadsheet. Why have both you may ask? Visibility.
We have some rudamentry code metrics set up through our continuous integration framework such as NDepend output and test coverage but are working towards something more visible and useful.
Estimation
Still very much a necessary evil.
For comitting to work for an iteration we use Story Points using the fibernache (1, 2, 3, 5, 8…) sequence and achieve them by playing Planning Poker with everyone who may be involved in the work required to take part. We will only estimate (and commit to) work we have already analysed and gathered requirements for.
For longer term planning, as we don’t yet have enough information to be able to use cycle time for projecting work completion, using the velocity based on points completed per iteration has proven a very powerful toool to be able to give the rest of the business a better idea of our capacity and timescales (previously they had none). However this has well known drawbacks and we must be careful it does not get abused, as I have seen before (such as gaming of estimates, whether intentionally on subconsciously). Also, as we need to understand and have gathered the requirements to be able estimate this way it means there is very limited scope to how far into the future we can do this with any degree of confidence (as requirements will change). Once we have a reasonable amount of data in the system we will be able to use average cycle time, which will be much more powerful.
Coding Practices
Apart from the rules we’ve commited to as a team, Pair Programming, Unit Testing, Refactoring and the best working principles and practices of the software industry are encouraged from the top of the department and applied rigorously but pragmatically.
At the request of the department members (as a result of a disucssion on collective responsibility) we created a development standards document which includes topics such as naming conventions and testing. As much as possible the document is vague on implementation details to prevent it from holding us back when better working practices come along. We use shared Resharper and Visual Studio settings to help us keep to these standards.
As mentioned below we also frequently hold sessions to improve our skills.
Automated Testing
All new or modified code is covered by unit tests, integration tests (such as database interaction) and automated acceptance tests which test against the acceptance criteria (this last one is quite new territory).
Continuous Integration and Deployment
All projects are under continuous integration (we use TeamCity) and we are working towards having all deployments doing the same. We have monitors on the wall which show all the currently failing builds. Do I need to mention we use source control?
Roles and Responsibilities
Every role in the department is covered by a document explaining their roles and responsibilities. They are written in a way which encourages self-organisation and collective responsiblity. You can download them here. I will be talking about these descriptions more in a future article.
Learning Culture
Each week two hours are set aside for learning sessions such as coding dojos and presentations (we’re currently running a design patterns study group). Outside of these developers are actively encouraged to take the time to learn new practices during working hours (within reason). We have a library of books available on a range of subjects which are at the disposal of everyone. More often than not if there’s a book that someone would like to read the company will purchase it and add it to the library (books are pretty cheap in the grand scheme of things).
Continuous Improvement
Outside of retrospectives we have a monthly departmental session where the most pressing problems are discussed and actions taken away. However there is no limit or retriction to when improvements can be made and everyone is encouraged to take the initiative when they see a problem that needs addressing.
One of the keynotes at XPDay 2008 was from Dan Jones, the author of the books The Machine That Changed the World and Lean Thinking and one of the team of people who came up with the term “Lean Production”. It was quite humbling to be in the same room as the living person who has probably been more influential than anyone else to modern business processes in every industry and all over the world.
One of the most interesting things he said was that when they were coming up with the name for what they were trying to promote they considered “agile”, but thought it would be too difficult to sell and so decided on Lean instead. Also, he didn’t have much understanding of modern software development, but at a glance saw very little difference between what we called Agile and he called Lean.
So when we talk about Lean Software Development being an Agile methodology we’ve got it the wrong way around. Agile is Lean, Scrum is Lean, XP is Lean. You were already doing Lean Software Development, you just didn’t know it!
In my mind it’s time to drop the titles (which all carry too much stigma) and simply start referring to it as professional software development.
Scrum is getting a real bashing at the moment as you can see here and here and I think it could do with some defending.
I was on a team that adopted Scrum and it really empowered us. After a while, pair programming, TDD and refactoring became common-place because we learnt through the iterative process that they helped us write better software. People saw the positivity in the team and the productivity improvements and it lead to a fundamental shift in the department’s culture. Now all our teams are doing some form of iterative development, BDUF has gone out of the window and practices such as TDD and pair programming are actively encouraged by the management! People at all levels can see the difference it’s made to our productivity and the reputation of our department.
So Scrum can be very successful. It would never have got this far if people weren’t doing Scrum and getting positive results, in fact agile would not have become so big if it wasn’t for the success of Scrum so we’ve got a lot to thank it for.
However, here is where I think some of the criticism sticks:
Three people on the team above (including myself) went on the Certified Scrum Master course ran by Mike Cohn who is an excellent teacher, but I’ve since been on a “Scrum Estimating and Planning” course which seemed more geared towards telling people what they wanted to hear rather than any fundamentals of agile and the problems it’s trying to solve. As James Shore suggests in his article, the quality of the teachers on the CSM courses has a huge impact.
As company’s will always prefer to send people on a course rather than create a learning culture within (especially if they’re very small, which is more forgivable) the 2 day Certified Scrum Master course will be with us for a while yet and most significantly, will continue to be the main entry point to the agile world. People will continue to adopt Scrum because they’re failing with whatever they’re currently doing and continue to fail because the Scrum trainer will teach them how to write a story or run a stand up but not address the root problems, partly because you could never go into such detail in such a small amount of time and partly because that’s not the point of the course.
“Certified Scrum Master” suggests that you’re some kind of guru (it certainly sounds a lot more impressive then my BA(h) in Business and Quality Management which took me 4 years to get a mediocre grade in). Perhaps a name change would make it more credible. If organisations were not led to believe this 2 day course is going to solve all their problems then they wouldn’t be in the situation they’re in.
Edit: Since writing this article (a few hours ago) I’ve had a significant change of heart and feel I’ve incorrectly laid blame at the foot of the Scrum Alliance. It’s not their fault that their Certified Scrum Master training (which is designed to teach people how to become Scrum Masters, not solve all their problems) has become so popular. However, it is a bit unfortunate that the name could so easily be misinterpreted to mean more that it is and it’s also unfortunate that organisations hook on to the most simple looking solution to their problems (and no, I haven’t been threatened by Scrum mafiosa or bottled out I’ve just changed my mind).
It’s startling to see how much momentum has been gathering around Lean Software Development and I don’t think it’s a coincidence that at the same time many people seem to be falling out of love with Scrum.
At last year’s XPDay there were no sessions on Lean methodologies, this year there are three (the one I’m doing with Matt Wynne, Karl Scotland’s on Kanban, Flow and Cadence and a keynote from the Lean Enterprise Academy).
There’s something about this that makes me feel slightly uneasy. What’s so wrong with Scrum? Well, I’d be the first to stand up and say that Scrum has it’s failings – they’re the same reasons it’s become so popular. The simplicity, clear definition and business-friendliness of Scrum make it easy to sell (arguably, unlike XP) and waterfall or traditional development dynamics (project manager, use-case, Gantt chart, meeting) can be easily translated into Scrum’s characteristics (scrum master, story, burn down, stand up). Unfortunately all too often this is what happens resulting in what’s become know as Cargo Cult Agile, or WAgile, the underlying concepts and belief systems being mostly ignored. When I did Mike Cohn’s Certified Scum Master Course his principle lesson was “Inspect and Adapt” which he repeated endlessly throughout the course, but when I look on the Wikipedia entry for Scrum this doesn’t even get a mention so it’s easy to see how this happens.
So what’s so different about Lean? Well for a start there aren’t any clearly defined rules, but instead principles such as Eliminate Waste and Build Integrity In. However it strikes me that there’s just as much room for abuse with concepts such as Kanban, Minimally Marketable Features, Cumulative Flow Diagrams and so on. Consider this exert from The Toyota Way (which I’ve shamelessly stolen from James Shore’s article on Kanban Systems – I am actually reading this book as we speak, but haven’t got very far yet):
“…TPS experts get very impatient and even irritated when they hear people rave and focus on kanban as if it is the Toyota Production System. Kanban is a fascinating concept and it is fun to watch… When is the kanban triggered? How are the quantities calculated? What do you do if a kanban gets lost? But that is not the point… The challenge is to develop a learning organization that will find ways to reduce the number of and thereby reduce and finally eliminate the inventory buffer… So kanban is something you strive to get rid of, not to be proud of.”
Many of the reasons people aren’t being as succesful as they’d like with Scrum are exactly the same reasons they won’t be any more successful with any other methodology. People tend to focus on tools because it’s a lot easier than trying to tackle the often very difficult, challenging and more fundamental problems they grew from. Real change is hard and takes time, a very long time in some cases.
If you’re failing with Scrum don’t think lean, kanban, extreme programming or any other colour of agile will save you. Essentially if you’re failing it’s because you’re doing it wrong. However if you’ve found Scrum is working really well for you and has brought enormous benefits maybe you should come to the talk Matt and I are doing at XPDay and see how we evolved to a more lean process.
Update: since writing this James Shore has written a frighteningly similar post on The Decline and Fall of Agile, but I guess great minds think alike huh? ;-P
