“The best architectures, requirements, and designs
emerge from self-organizing teams”

– Principles Behind the Agile Manifesto

I’ve often had people say to me that we’re not really self-organising because we’re not all taking part in all the decisions made at our organisations, such as how the company is run or what products we should be making. This is a misunderstanding of how self-organisation occurs. The fact is we’re constantly self-organising, it’s just most of the time we’re completely oblivious to the fact it’s happening.

For example, within a group of people (such as a workplace) there are many social sub-groups, such as the early morning runners, the ones that go to lunch together and the after work drinkers. They all naturally self-organise around shared goals & desires. Even within a team where a highly autocratic management approach is in place, members of the team will self-organise around anything they can (such as making the tea or going out to buy some biscuits). The thing with self-organisation is it’s actually harder not to be doing it!

If you define self-organisation as when everyone in a team is able to control everything that affects it where do you stop? Your team? Your department? Organisation? City? Country? See where I’m going?

Self-organisation always occurs within boundaries defined by it’s context. For example, a football team might be a football pitch for 90 minutes, a group of friends on holiday the holiday villa they’re staying in or your department at work by the responsibilities it has. Something or someone always sets these boundaries. If they did not exist self-organisation couldn’t occur.

What causes self-organisation?

A complex system* will always be vulnerable to external events which are outside of it’s control, such as the opposite team, the weather or your customers or stakeholders. Self-organisation emerges when it becomes important to respond to these uncontrollable events in a manner which benefits the system. The ability to do so results in the system being adaptive. We know that software development is complex so we can determine we’re in a complex adaptive system. It’s a huge subject and not the particular focus of this article so I won’t try to go into any more detail, but if it’s a new concept to you I strongly encourage you to investigate further.

How to take advantage of the fact that it occurs

When self-organisation is on your side it can be extremely valuable. If you’re working in a complex adaptive system it’s shown that by far the most effective way to allow that system to operate is to allow it to self-organise, because that’s what it will naturally try to do.

The problem is it’s not always on your side (if you’re an oppressive Arab dictator, for example) and will often work against your goals and objectives. I’ve often seen self-organisation criticised because this happens. I’ve also seen it criticised because a team is unable to self-organise effectively when allowed to do so. In both cases you may be tempted to blame the individuals in the team. This would be unjust because in reality it is the system to blame (and probably you as the controller of the system).

To enable self-organisation to work in your favour you need to understand and ideally define the system – set it’s boundaries and try to organise it in a manner which will encourage self-organisation to occur in a way which supports your objectives. This is often called Systems Thinking. Again this is a large and well documented subject so there’s no need for me to elaborate further (if you’re looking for a good place to start though I thoroughly recommend Jurgen Appelo‘s excellent book, Management 3.0).

*It’s widely held that software development in general exhibits the properties of being complex (as described in the Cynefin Model).

It may surprise some, but I’ve been at this agile lark for less than 4 years (or maybe that’s a lot?). I’ve been motivated by Jason Gorman’s recent post and case study (and some subsequent twitterings) to share some of my thoughts from working at BBC Worldwide¹ at that time, a particular project I worked on which has influenced me far more than anything I’ve been involved with before or since and how it all changed me (for the better).

The Online Catalogue (OLC) project

Shortly after I started at Worldwide I was moved into a team doing a re-write of the Online Catalogue (now renamed BBC Wordwide Sales – essentially a sales device so broadcasters can find and view details on shows produced by the BBC they could purchase and broadcast on their own channels). It was the company’s first foray into Agile and both a great success and complete disaster.

It was a success because we were a great group of people being given the opportunity to determine how we wanted to work. We made many, many mistakes (inc. classics such as “done” being development complete i.e. not tested or released), but crucially we were learning and improving all the time.² It was an extremely empowering and motivating experience and created an incredible bond between the people in the team (although I don’t see them often I still consider many of them great friends today). The highly collaborative nature of the way we worked meant I learnt more in the first year than all of my career up to that point. The wonders of TDD, refactoring and pair programming were truly opened up to me. It wasn’t so much a jump as a giant leap. The news of our successes spread quickly and it wasn’t long before Agile software development was the de-facto approach.

No doubt you’re probably far more interested in why it was not a success and you’d be right to be (I learnt far more from what went wrong that what went well).

Crucially before development started a year had already been spent on analysis resulting in a use-case requirements document a good inch thick so we weren’t iterating on functionality, only incrementing in two weeks sprints (and knowing the audience I don’t think I need to explain further). Also the expectations on the project’s delivery were very optimistic. The combination of these two and the political and bureaucratic culture of the organisation meant failure was baked in from the start.

Shortly after development began it was evident it was going to take a lot longer than expected, but by that time the project had gathered too much momentum and too much money had been spent. At each management review the estimated completion date got longer and longer but each time no one was prepared to cut the scope or do the honourable thing and kill it. As is typical with death marches failure isn’t an option until it’s the only one left. The initial estimate was 3 months and the project was eventually completed 18 months (and well over a million pounds) later. The product itself? To it’s credit it was a much better engineered application than it’s predecessor and was a good foundation for future development, but in many ways it had less functionality than the product it was replacing.

The subsequent witch hunt was extremely unpleasant with the blame eventually being laid at the foot of the Lead Developer and Project Manager. Their estimates weren’t good enough. Fortunately our PM (who was actually excellent and by far the best I’ve ever worked with) had already left, which unfortunately meant the Lead Developer bore the brunt of it and his contract wasn’t renewed, which was undoubtedly a political gesture to higher management. There was no interest or barely any investigation into the root causes. When we did eventually do a project review the findings were filed away. It didn’t matter any more as blame had already been assigned and everyone had moved on.

I wouldn’t say we were completely without fault, there were many things within our power we would have done differently if in the same situation again, but I could say that about all the projects I’ve worked on since.

The community

Pete Camfield and Jason Gorman built a brilliant community, among other things introducing lunchtime sessions where they managed to get in many well known faces to do talks, but also encouraged us to do our own. I did my first ever presentation on “Test Smells” inspired by the xUnit book and have since presented at numerous conferences and events, something I can’t imagine I’d ever have had the courage to do without this opportunity. We also started a design patterns study group and a library (which I proudly curated).

Additionally the lead developers, coaches and some others (like myself) who were motivated to change things got together regularly to discuss how to do so (during the time I wrote an article which I think sums up the mood quite well).

Of course it’s no community without brilliant people, of which there were many (I daren’t name check for risking excluding or offending anyone).

Why I left

I think I’ve already alluded enough to the fact there were significant systemic issues with the organisation. I guess it’s somewhat inevitable (and not exclusive to BBCWW) that larger companies suffer somewhat from bureaucracy and the political environment that generates.

Most people I worked with were decent and just trying to do a good job, but the nature of the environment brought out the worst in people (in the same way buying a house in the UK seems turns everyone into cold-hearted monsters)³ and this went all the way up the chain. We were able to optimise the way we worked within teams but efforts to change things beyond this were exhausting and mostly futile. It was ultimately an environment which suited politically minded career types rather than people who wanted to make a difference. I eventually got too frustrated with banging my head against the ceiling.⁴

How it changed me

The combination of my experiences on the OLC project and the environment created by Pete and Jason changed me unrecognisably. I learnt that in an agile environment being a super brain developer (which I’ll never be) is really not that important. It matters far more if you give a shit and want to make things better. As my confidence in my abilities grew I became more vocal about what was wrong, at the beginning within our team, but soon (as most agile teams quickly realise) about what was wrong with the organisation which was causing many of our problems. Among other subjects my experiences on the OLC project inspired countless articles and conference sessions about (and a fascination with) estimation and predictability.

Most of all it showed me that, unlike my previous belief, most organisations are not ran very well, in fact they’re being run very badly, which means billions upon billions of pounds are increasingly and needlessly being wasted on software projects and thousands upon thousands of people are being stressed, frustrated and depressed by being involved in them. It showed me that there is a better way and that I get huge satisfaction from being involved in trying to improve that situation (well, at least not making it any worse).

Conclusion

Agile environments make people care. It means you’re much more exposed to your customers and stakeholders and spend a lot more time (than most people in your organisation) engaged in retrospecting and discussing problems. I’ve also noticed recently that people are much more critical on agile teams – something I thought was a bad sign but I’ve now realised the opposite is true⁵

It has it’s downsides of course. More emotional engagement means more emotional attachment. I ride the peaks and troughs a lot and find it difficult to switch off.

It’s not as much the project or organisation it makes me care about as much as the people I work with, it just makes it all more human.

Footnotes

¹ BBC Worldwide is the commercial arm of the BBC, a wholly owned subsidiary and quite separate from the publicly funded corporation (although all it’s profits go back to fund the BBC).

² Matt Wynne and I presented our experiences of evolving from Scrum to Lean at XPDay 2008. Here’s a write up and here’s the slides.

³ This kind of behaviour is well documented by complexity sciences but most famously by the Stanford Prison Experiment.

⁴ I’d like to think this doesn’t have to be the case. I think at least part of the problem at BBC Worldwide was it suffered from having too many long termers (10 years was quite normal) meaning people were quite protective of their positions or not keen to make any noise/get off their comfy chairs with the consequence the status quo was very difficult to change. Also, within the development team the ratio was weighted too far towards contractors than permanent staff (I was perm) with the result that people either didn’t care or weren’t around long enough to make it stick. I understand the IT department has recently been through a major overhall with many people having left via redundancy and new management at the top, so I doubt my experiences are still relevant, however I’ve not heard anything to say it’s improved either.

⁵ A recent experience report published by Benjamin Mitchell found that a team involved in a System’s Thinking based approach were the most motivated but also more critical (than other areas of the business) of how well they were satisfying customers.

Ivan Moore wrote an article a while back on how he runs pair programming interviews. After introducing them to the recruitment process at my company I have to agree with Ivan that they’re incredibly valuable – among other things they’re a great way for a potential employee to show they have a developer conscience. I wanted to share some of our learnings too.

Real Code

We use real code and real problems we’ve already had to solve. I know others like to use programming exercises, but we like to see how candidates deal with problems which are more relevant to the job they’re applying for such as dealing with legacy code and unit testing.

Code Review

We start by walking the candidate through the code and it’s context, give them 10 minutes to review the code by themselves and then discuss their thoughts. Outside of the actual pairing exercise this is a great opportunity for them to show us how well they understand OO principles and practices by identifying any code smells. We’re also looking out for how well they explain their answers as an ability to communicate well is essential.

The Exercise

Next we roll up our sleeves and start programming. We set the candidate a challenge (in the same code base), work with them as much as possible without making it too easy, explain anything they’re unclear of and help them with any tools they’re not familiar with. Whenever we come to a tricky or challenging situation we ask them what options we have. If they’re totally stuck we will suggest possible solutions, but ultimately they’re the one making the decisions. The exercise continues until the allotted time runs out (1 hour normally).

It’s about what you do not how you do it

We explain to the candidate at the beginning of the interview that there is no finishing line. What we’re looking for most is how they approach a problem and how well they work with others. Often people who get the furthest do so by making risky refactorings and poor design decisions.

Pair pair programming

It’s always good to have more than one opinion so generally there will be two people in the interview aside from the candidate. One takes part in the exercise whilst the other acts as an observer. This allows the navigator to focus on the exercise. After the interview we get together and discuss our thoughts. Often one of us will have forgotten or not noticed something from the interview and this ensures we get the most balanced and fair view possible.

Does it work?

So far everyone who we’ve recruited who has taken part in the pairing exercise has lived up to, if not excelled our expectations of them based on how they performed. One drawback is although we do try to keep the interview as relaxed as possible it can be quite intimidating for some, however having done quite a few now I cannot imagine recruiting a developer without using this technique as part of the interview process.