XPDay London ’09: Call for sessions

Submissions are now open for programmed sessions at XpDay London 2009, to be held 7th and 8th December 2009. http://www.xpday.org/

You are invited to propose a session for the first day of the conference. They’re are particularly interested in the following

  • Experience reports—share your stories of challenge and success with Agile and Lean techniques. Experience reports will be intensively shepherded by experienced practitioners.
  • Hands-on technical sessions—share techniques and practices in practical sessions: workshops, tutorials, simulations
  • Practitioners’ advances in the art—share the techniques of expert Agile and Lean practitioners, work with them to move the craft forward.

The second day of the conference will be an OpenSpace session with topics selected at the end of the first day. Programmed sessions are most suitable for topics requiring some set up or extensive preparation.

To submit a session, please go to http://xpday-london.editme.com/XpDay2009Submissions

Submissions will be accepted until Friday 14th August.

What’s the difference between Lean and Agile?

There isn’t any.

Certainly not enough to be able to draw a clear line between the two and start comparing the benefits of one against the other. Daniel Jones, the author of The Machine That Changed the World did a keynote at XPDay2008 and said from what he saw there wasn’t really much difference, apart from the name. In fact, when they were looking for a name for Lean they thought about using Agile.

Agile came from Lean, the related processes came from people studying Demming, The Toyota Production System or through convergent evolution. The two guys who invented Scrum are Japanese and that’s no coincidence.  When I started reading about Toyota and Demming it all simply made sense to me and explained where Agile came from.

Then I found an article written by Craig Larman on the history of iterative development and that wrapped it up: http://www2.umassd.edu/SWPI/xp/articles/r6047.pdf

There is no point spending any time discussing and trying to find differences between what is essentially the same thing just written in a different way.

It reminds me of how the Jews and Christians went their separate ways and then came the Muslims and then they split up into the Sunni and Shia’s. They’ve all been fighting bitterly for the last 2000 years but all worship the same god. Go figure!

Snapshot of my team’s current practices

It seems these days there are countless methodologies, processes and practices to choose from when developing software and, somewhat ironically, the list seems to be growing at the rate of Moore’s law. I’ve read about, discussed, been on courses and been to conferences about a lot of them and the thing I’ve consistently found most useful is talking to other practitioners about what they’re doing and what’s working (or not working) for them.

Recently there’s been a lot of (and subsequently criticism of) debate on message boards and blogs about the relative benefits of one paradigm versus the other.  Personally I don’t care much for subscribing to any particular paradigm and am much more interested in what works and what doesn’t (and in which circumstances) so my response is to publish what my team and company is actually doing right now. This is a snapshot of our current practices. Ask me again in 6 months and hopefully I’ll show you something very different.

The inspiration and influences for the way we work mostly come from Agile, Scrum, Extreme Programming, Software Craftsmanship, Lean, Lean Software Development, Kanban Software Development and The Toyota Way. It is all and none of these things.

Context

We are a small to medium size “start up” organisation working in the new media industry. The company employs around 60 people mainly based in the UK. The development department numbers around 20 co-located people. Agile practices are a relatively new introduction and the previous approach was of the typically chaotic type familiar to young businesses.  We mainly work in .Net using C# but also dabble in Ruby, Javascript and UI languages. The rest of the article mostly relates specifically to one of the teams working within the department but also addresses some of the practices of the department as a whole.

Team

The team is made up of 4-5 developers and a tester. There is no project manager at the team level – in the spirit of self-organisation (principle 11) the duties traditionally the responsibility of a PM are shared between the team members. The Product Owner role is shared between the stakeholders within the organisation for the products the team is responsible for.

Iterations

We currently work in 1 week Iterations. It’s a new team who are also new to many of the agile concepts and doing this enables us to control the amount of work in progress, focus on delivery, improve our discipline and most importantly have short feedback cycles so improvements can be discussed and applied frequently. The downside is the overhead created by the amount of meetings. Once we’re comfortable the team is working well together we will have the opportunity to change this if desired (e.g. Continuous Flow, changing the iteration length, changing the frequency of meetings).

Meetings

Each iteration we have the following meetings:

Work Prioritisation – occurs iteration minus one. Stakeholders come together to raise and prioritise work not yet commited to.
Requirements Gathering – occurs ad hoc when necessary. All the team is required to attend along with the customer/s to bash out requirements for work prioritised in the prioritisation meeting.
Planning – occurs at the beggining of the iteration. Prioritised features (MMFs) which have been analysed are broken down into stories, discussed, estimated and commited to based on our current velocity (avg. over last 6 weeks)
Stand Up – occurs daily at 10am at the task board. Anyone outside the team is welcome to watch
Retrospective – occurs at the end of the iteration. Any actions from the meeting are to be completed by the end of the next iteration.

Requirements

Features are requested at the prioritisation session and use the User Story format.

More detailed requirements are gathered during the requirements meetings mentioned above, with the customer/s and all team members present. We use whiteboards to bash out the requirements and convert them into acceptance criteria using the “Given, When, Then” format. We have a rule that no work can be commited to unless we’re happy we have a clear understanding of the requirements.

Task Board

The task board is essentially a Kanban board with each stage of the delivery process separated into columns. We have an implicit limit of 2 stories in active, but otherwise have not applied limits to any other columns. Features (MMF) are blue, the stories which make up the MMFs are yellow, bugs are pink and quick support tasks white. When a story is commited to, the feature card is moved into “commited”, above the titles of the columns and tracks the last related story.

Measurement & Metrics

We use an Excel spreadsheet to hold the product backlog and track the data from the Kanban/Task board. Whenever an MMF moves to another column the date this occured is recorded. You can download a copy of the spreadsheet here (you may want to check the calcs on the CFD, not sure they’re right). Among other things it calculates average cycle time, average velocity and projections based on velocity. I’ve tried a few bespoke tracking tools (such as Mingle) and found nothing is as powerful and flexible as Excel.

We have a manual Cumulative Flow Diagram (CFD) which each team member takes turns to update daily so everyone shares the responsibility (it is also their job to update the Excel spreadsheet each day). The CFD diagram only tracks the value delivered to the business (one unit = one MMF. Measure the output, not the input) and is also represented in the Excel spreadsheet. Why have both you may ask? Visibility.

We have some rudamentry code metrics set up through our continuous integration framework such as NDepend output and test coverage but are working towards something more visible and useful.

Estimation

Still very much a necessary evil.

For comitting to work for an iteration we use Story Points using the fibonacci (1, 2, 3, 5, 8…) sequence and achieve them by playing Planning Poker with everyone who may be involved in the work required to take part. We will only estimate (and commit to) work we have already analysed and gathered requirements for.

For longer term planning, as we don’t yet have enough information to be able to use cycle time for projecting work completion, using the velocity based on points completed per iteration has proven a very powerful toool to be able to give the rest of the business a better idea of our capacity and timescales (previously they had none). However this has well known drawbacks and we must be careful it does not get abused, as I have seen before (such as gaming of estimates, whether intentionally on subconsciously). Also, as we need to understand and have gathered the requirements to be able estimate this way it means there is very limited scope to how far into the future we can do this with any degree of confidence (as requirements will change). Once we have a reasonable amount of data in the system we will be able to use average cycle time, which will be much more powerful.

Coding Practices

Apart from the rules we’ve commited to as a team, Pair Programming, Unit Testing, Refactoring and the best working principles and practices of the software industry are encouraged from the top of the department and applied rigorously but pragmatically.

At the request of the department members (as a result of a disucssion on collective responsibility) we created a development standards document which includes topics such as naming conventions and testing. As much as possible the document is vague on implementation details to prevent it from holding us back when better working practices come along. We use shared Resharper and Visual Studio settings to help us keep to these standards.

As mentioned below we also frequently hold sessions to improve our skills.

Automated Testing

All new or modified code is covered by unit tests, integration tests (such as database interaction) and automated acceptance tests which test against the acceptance criteria (this last one is quite new territory).

Continuous Integration and Deployment

All projects are under continuous integration (we use TeamCity) and we are working towards having all deployments doing the same. We have monitors on the wall which show all the currently failing builds. Do I need to mention we use source control?

Failed builds monitor

Roles and Responsibilities

Every role in the department is covered by a document explaining their roles and responsibilities. They are written in a way which encourages self-organisation and collective responsiblity. You can download them here. I will be talking about these descriptions more in a future article.

Learning Culture

Each week two hours are set aside for learning sessions such as coding dojos and presentations (we’re currently running a design patterns study group). Outside of these developers are actively encouraged to take the time to learn new practices during working hours (within reason). We have a library of books available on a range of subjects which are at the disposal of everyone. More often than not if there’s a book that someone would like to read the company will purchase it and add it to the library (books are pretty cheap in the grand scheme of things).

Continuous Improvement

Outside of retrospectives we have a monthly departmental session where the most pressing problems are discussed and actions taken away. However there is no limit or retriction to when improvements can be made and everyone is encouraged to take the initiative when they see a problem that needs addressing.


Principles

The Principles behind the Agile Manifesto

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Business people and developers must work together daily throughout the project.
  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Working software is the primary measure of progress.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity–the art of maximizing the amount of work not done–is essential.
  • The best architectures, requirements, and designs emerge from self-organizing teams.
  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

The Five Principles of Lean

  1. Value – specify what creates value from the customer’s perspective.
  2. The value stream – identify all the steps along the process chain.
  3. Flow – make the value process flow.
  4. Pull – make only what is needed by the customer (short term response to the customer’s rate of demand).
  5. Perfection – strive for perfection by continually attempting to produce exactly what the customer wants.

The Seven Principles of Lean Software Development

  1. Eliminate waste
  2. Amplify learning
  3. Decide as late as possible
  4. Deliver as fast as possible
  5. Empower the team
  6. Build integrity in
  7. See the whole

The 4 Sections and the 14 Principles of the Toyota Way

I. Having a long-term philosophy that drives a long-term approach to building a learning organization

1. Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals

II. The right process will produce the right results

2. Create a continuous process flow to bring problems to the surface

3. Use “pull” systems to avoid overproduction

4. Level out the workload (heijunka). (Work like the tortoise, not the hare)

5. Build a culture of stopping to fix problems, to get quality right the first time

6. Standardized tasks and processes are the foundation for continuous improvement and employee empowerment

7. Use visual control so no problems are hidden

8. Use only reliable, thoroughly tested technology that serves your people and processes

III. Add value to the organization by developing its people and partners

9. Grow leaders who thoroughly understand the work, live the philosophy, and teach it to others

10. Develop exceptional people and teams who follow your company’s philosophy

11. Respect your extended network of partners and suppliers by challenging them and helping them improve

IV. Continuously solving root problems to drive organizational learning

12. Go and see for yourself to thoroughly understand the situation (Genchi Genbutsu).

13. Make decisions slowly by consensus, thoroughly considering all options; implement decisions rapidly (Nemawashi).

14. Become a learning organization through relentless reflection (hansei) and continuous improvement (Кaizen).

Kanban is just a tool, so why is it being treated like a methodology?

I was throwing some shapes on Twitter recently about some concerns I have with the current Kanban craze. Unfortunately I think the cursed 140 character limit meant my points got misinterpreted and may have lead people to think I’m anti-Kanban which is not the case in fact it’s quite the opposite. I’ve been using Kanban boards for over a year and half and jointly ran a presentation at XPDay2008 on evolving from Scrum to Lean which focused heavily on the use of Kanban boards.

The thing that’s making me itchy is how Kanban has somehow been elevated into a methodology unto itself. We don’t have “Scrum and Sprint” conferences or XPandPairProgrammingDay so why do we have Lean Kanban Conference Miami or the UK Lean and Kanban Conference? Also, pretty much everywhere you see someone talking about Lean software development the title of the blog or presentation also includes Kanban in the same breath? More than that I see a lot of discussion around Kanban in blog posts and Twitter but very little on Lean or the Lean Software Development principles.

I’m sure proponents of Kanban will say no one is suggesting Kanban is a methodology and I would agree I’ve not seen anyone say it is. The problem is interpretation. People have a habit of focusing on rules and methodologies because they’re a lot more easy to tackle than the problems they we’re created to solve. Scrum has been enormously successful (if you consider wide adoption a measure of success) but very few teams are doing it well as James Shore has been writing eloquently about recently because it does not force you address the real issues. The beauty of Lean software development is it is just a set of principles. It intentionally avoids prescribing how to do something. Obviously this causes problems as most people don’t want to get involved in the difficult stuff they just want to be told how to do it. Consider this reply I received on Twitter:

erwilke @robbowley “I think we’re trying to avoid kanban being seen as a stand-alone methodology, but people don’t “get” it as a set of tools”.

Maybe there’s a good reason why people don’t get it – because it you need to understand where it’s coming from. Focusing on Kanban and ignoring all the rest however, that’s easy!

Elevating Kanban to the prominent position it is now in makes me feel like history is going to repeat itself. I prophesied this some months back. It has been the most popular post on my blog by a long way.

If you’re getting into Kanban, be warned. Kanban is just a tool and in my opinion no more important than say, pair programming, unit testing or domain driven development. It is certainly a lot less important than the white elephant in the room which very few people seem to be addressing which is building the right thing in the first place. As Peter Drucker famously said: “There is nothing so useless as doing efficiently that which should not be done at all”.

Kanban is a small part of something much, much bigger, see the whole.

 

*Edit* Some responses to this article:

Is Kanban Just a Tool? – David Anderson

It is Not What It is that Really Matters - Israel Gat

Kanban: It’s a Tool, and There’s No Such Thing as “Just” a Tool – Project Management Revolutionary

Depend in the direction of stability

As a general rule of thumb you should depend in the direction of stability (The Stable Dependencies Principle (SDP)). A package should only depend upon packages that are more stable than it is. If something is changing a lot it we should not depend on it. If it isn’t then we can comfortably reference it as a versioned package/assembly.

“But I want to add something to the GeneralFunctions/Domain/Utilities project”
Why? If you are the only one using it then there is no reason for it to be there.

“But someone else may want to use it in the future”
The possibility that someone may is not a good enough reason to put it there. Follow the principle of You Aren’t Gonna Need It (YAGNI). 

“But what if I need to change something that already exists?”
“Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification” – The Open Closed Principle

Creating a TeamCity currently failed builds page

Frustrated with seeing too many failed builds and wanting to make the issue more visible we are planning on putting an LCD monitor on the wall to display all the failing builds. However, unfortunately TeamCity does not have such a page available. There are various ways of receiving the status of builds such as through RSS feeds and the Status Widget, but the information is no more than you can see on the overview page and if you have a lot of builds it’s not easy to see where the problems are.

First, I experimented with using Linq to mine the RSS feeds and was getting somewhere but it wasn’t exactly a five minute job. Then Kirill Maximov, a JetBrains TeamCity developer responded to my cries of help on Twitter saying I could modify the External Status Widget and a whole world of fun was opened up to me.

How to create a failed builds page

  1. Create an html page that includes the External Status Widget as described here. By default this page will only show the current status of all projects as you see on the overview page, but we only want to know about failed builds.
  2. Add the wrapping <c:if test=”${not buildType.status.successful}”></c:if> to the file *TeamCity*/webapps/ROOT/status/externalStatus.jsp (it is probably worth making a backup of this file first).
  3. Thats it! However as you have now realised you can do a lot more customisation. If you have a root through some of the jsp files you’ll see there are a lot of features you can take advantage of to customise further.

You can download my html page and modifed externalStatus.jsp files here

Team principles

On my new team we’ve just started our first proper iteration. We’ve agreed to commit to the following principles:

  • No multi-tasking
  • All new or changed code must be thoroughly unit tested
  • No more than 2 pieces of work in “Active Work” at any time (our team is 3 developers)
  • We always work on the highest priority task
  • Our definition of done is “In UAT”
  • Leave it in a better condition than you found it
  • No hidden work
  • No overtime
  • No disruptions
  • Don’t SysTest your own work (we don’t have a tester yet)
Have you done something similar? What are your team’s principles?

TDD keeps you focused

One of the less stated benefits of doing TDD is that it concentrates your mind on the problem.

Recently I was looking at a section of code which was creating a Lucene search string from criteria passed in from the user. I needed to add some new functionality, but it was a bit higgledy-piggledy so following the broken windows principle I decided to refactor it to the builder pattern before I added my new feature.

I ran NCover on the code and it reported 70% coverage which considering what I knew about the history of the system was surprisingly high. I decided there was enough coverage, paired with a colleague and dived in. At first we were doing well. Our changes broke the tests a few times which reassured us about the coverage, but after a while it felt that we were losing a bit of steam. We’d done a lot of refactoring but it didn’t feel like we’d got very far.

Skip to a couple of days later – I’d not been able to touch the code and neither had my colleague who had now been taken off to deal with another project, so I pulled in the other dev on my team who is probably the most fanatical TDD nut I’ve ever met. Immediately he was uncomfortable, he’d quickly spotted the tests were really only integration tests (something I had put to the back of my mind) and got jittery as we continued from where I’d left off. I didn’t like to see him in so much pain so quickly relented and let him write new tests even though I felt it was just going to slow us down, maybe even to the point where we wouldn’t get it finished. However he assured me that although it felt slow now we’d be racing along soon enough.

Not only was he right about that, but as we were doing it the objective was so much clearer. Writing the new tests meant we were forced to really think about each move we made so our refactoring had clear direction. We spent about the same time working on the problem but probably made twice as much progress as my previous attempt and ended up with a proper unit test suite around the code.

Some day soon people are probably going to do away with the need to do TDD, which I think will be a shame as it never ceases to amaze me how many benefits it has.