It may surprise some, but I’ve been at this agile lark for less than 4 years (or maybe that’s a lot?). I’ve been motivated by Jason Gorman’s recent post and case study (and some subsequent twitterings) to share some of my thoughts from working at BBC Worldwide1 at that time, a particular project I worked on which has influenced me far more than anything I’ve been involved with before or since and how it all changed me (for the better).
The Online Catalogue (OLC) project
Shortly after I started at Worldwide I was moved into a team doing a re-write of the Online Catalogue (now renamed BBC Wordwide Sales – essentially a sales device so broadcasters can find and view details on shows produced by the BBC they could purchase and broadcast on their own channels). It was the company’s first foray into Agile and both a great success and complete disaster.
It was a success because we were a great group of people being given the opportunity to determine how we wanted to work. We made many, many mistakes (inc. classics such as “done” being development complete i.e. not tested or released), but crucially we were learning and improving all the time.2 It was an extremely empowering and motivating experience and created an incredible bond between the people in the team (although I don’t see them often I still consider many of them great friends today). The highly collaborative nature of the way we worked meant I learnt more in the first year than all of my career up to that point. The wonders of TDD, refactoring and pair programming were truly opened up to me. It wasn’t so much a jump as a giant leap. The news of our successes spread quickly and it wasn’t long before Agile software development was the de-facto approach.
No doubt you’re probably far more interested in why it was not a success and you’d be right to be (I learnt far more from what went wrong that what went well).
Crucially before development started a year had already been spent on analysis resulting in a use-case requirements document a good inch thick so we weren’t iterating on functionality, only incrementing in two weeks sprints (and knowing the audience I don’t think I need to explain further). Also the expectations on the project’s delivery were very optimistic. The combination of these two and the political and bureaucratic culture of the organisation meant failure was baked in from the start.
Shortly after development began it was evident it was going to take a lot longer than expected, but by that time the project had gathered too much momentum and too much money had been spent. At each management review the estimated completion date got longer and longer but each time no one was prepared to cut the scope or do the honourable thing and kill it. As is typical with death marches failure isn’t an option until it’s the only one left. The initial estimate was 3 months and the project was eventually completed 18 months (and well over a million pounds) later. The product itself? To it’s credit it was a much better engineered application than it’s predecessor and was a good foundation for future development, but in many ways it had less functionality than the product it was replacing.
The subsequent witch hunt was extremely unpleasant with the blame eventually being laid at the foot of the Lead Developer and Project Manager. Their estimates weren’t good enough. Fortunately our PM (who was actually excellent and by far the best I’ve ever worked with) had already left, which unfortunately meant the Lead Developer bore the brunt of it and his contract wasn’t renewed, which was undoubtedly a political gesture to higher management. There was no interest or barely any investigation into the root causes. When we did eventually do a project review the findings were filed away. It didn’t matter any more as blame had already been assigned and everyone had moved on.
I wouldn’t say we were completely without fault, there were many things within our power we would have done differently if in the same situation again, but I could say that about all the projects I’ve worked on since.
The community
Pete Camfield and Jason Gorman built a brilliant community, among other things introducing lunchtime sessions where they managed to get in many well known faces to do talks, but also encouraged us to do our own. I did my first ever presentation on “Test Smells” inspired by the xUnit book and have since presented at numerous conferences and events, something I can’t imagine I’d ever have had the courage to do without this opportunity. We also started a design patterns study group and a library (which I proudly curated).
Additionally the lead developers, coaches and some others (like myself) who were motivated to change things got together regularly to discuss how to do so (during the time I wrote an article which I think sums up the mood quite well).
Of course it’s no community without brilliant people, of which there were many (I daren’t name check for risking excluding or offending anyone).
Why I left
I think I’ve already alluded enough to the fact there were significant systemic issues with the organisation. I guess it’s somewhat inevitable (and not exclusive to BBCWW) that larger companies suffer somewhat from bureaucracy and the political environment that generates.
Most people I worked with were decent and just trying to do a good job, but the nature of the environment brought out the worst in people (in the same way buying a house in the UK seems turns everyone into cold-hearted monsters)3 and this went all the way up the chain. We were able to optimise the way we worked within teams but efforts to change things beyond this were exhausting and mostly futile. It was ultimately an environment which suited politically minded career types rather than people who wanted to make a difference. I eventually got too frustrated with banging my head against the ceiling.4
How it changed me
The combination of my experiences on the OLC project and the environment created by Pete and Jason changed me unrecognisably. I learnt that in an agile environment being a super brain developer (which I’ll never be) is really not that important. It matters far more if you give a shit and want to make things better. As my confidence in my abilities grew I became more vocal about what was wrong, at the beginning within our team, but soon (as most agile teams quickly realise) about what was wrong with the organisation which was causing many of our problems. Among other subjects my experiences on the OLC project inspired countless articles and conference sessions about (and a fascination with) estimation and predictability.
Most of all it showed me that, unlike my previous belief, most organisations are not ran very well, in fact they’re being run very badly, which means billions upon billions of pounds are increasingly and needlessly being wasted on software projects and thousands upon thousands of people are being stressed, frustrated and depressed by being involved in them. It showed me that there is a better way and that I get huge satisfaction from being involved in trying to improve that situation (well, at least not making it any worse).
Conclusion
Agile environments make people care. It means you’re much more exposed to your customers and stakeholders and spend a lot more time (than most people in your organisation) engaged in retrospecting and discussing problems. I’ve also noticed recently that people are much more critical on agile teams – something I thought was a bad sign but I’ve now realised the opposite is true5
It has it’s downsides of course. More emotional engagement means more emotional attachment. I ride the peaks and troughs a lot and find it difficult to switch off.
It’s not as much the project or organisation it makes me care about as much as the people I work with, it just makes it all more human.
Footnotes
1 BBC Worldwide is the commercial arm of the BBC, a wholly owned subsidiary and quite separate from the publicly funded corporation (although all it’s profits go back to fund the BBC).
2 Matt Wynne and I presented our experiences of evolving from Scrum to Lean at XPDay 2008. Here’s a write up and here’s the slides.
3 This kind of behaviour is well documented by complexity sciences but most famously by the Stanford Prison Experiment.
4 I’d like to think this doesn’t have to be the case. I think at least part of the problem at BBC Worldwide was it suffered from having too many long termers (10 years was quite normal) meaning people were quite protective of their positions or not keen to make any noise/get off their comfy chairs with the consequence the status quo was very difficult to change. Also, within the development team the ratio was weighted too far towards contractors than permanent staff (I was perm) with the result that people either didn’t care or weren’t around long enough to make it stick. I understand the IT department has recently been through a major overhall with many people having left via redundancy and new management at the top, so I doubt my experiences are still relevant, however I’ve not heard anything to say it’s improved either.
5 A recent experience report published by Benjamin Mitchell found that a team involved in a System’s Thinking based approach were the most motivated but also more critical (than other areas of the business) of how well they were satisfying customers.
Very useful experience report, Rob. Sorry to read your last-minute update about things getting worse.
But always love to hear about people learning and growing – even when there’s an associated downside. Thanks for sharing! 🙂
– Bob (@FlowchainSensei)
P.S. No “tweet this” button?
Interesting post, especially given as I’m the new lead on the OLC project!
I can’t talk for past process or delivery, but will admit the current process is far from perfect. It’s something I’m working on taking more control of though. Especially a t-shirt size estimating model (I *hate* being asked to estimate in days). There are days when I wish I could turn green and hulk smash the place up, so I can see why you left.
Looking at the codebase, I do find it a tad complicated, over engineered is probably a better term. I’ve never seen 8 databases for a small admin website, let alone similar db counts per app!
I’ll maybe right a post myself in 6 months, let you know if it’s still the same (or not)!
Hi Rob,
The codebase was definitely over-engineered. It suffered from a lot of BDUF and someone had been drinking the SOA kool aid too, which in hindsight was quite unnecessary and added much pain. The complexity certainly didn’t help speed things up and I think it could have been delivered in less time if a more simple approach was taken.
Unfortunately by the time most of us started working on the project it was well too far down the road and too much had been invested for anyone to be persuaded cuts in scope and complexity.
Having said that I think we were all quite proud of the code and the general consensus was it was the best thing any of us had written. Of course it’s a long way from perfect (as I’m sure you’ve found out) but I’d certainly be interested to know how well it’s standing the test of time.
Rob
p.s. I decided not to mention the technical decisions on architecture because I don’t think they had a huge impact in the long run. You’ll always look back and see what could’ve been done better – something you’ll never really know until after the event.
To Rob and Rob
The lesson I’m sure you’ve both learned about years in IT projects is that unless the project is either straightforward according to them team expertise level (been there done that story) or that you’ve managed to be agile in your approach all the way through, you will only know at the end of the project how simple it could have been made.
Actually, I’ve only recently experienced that very feeling having worked for an investment bank when, coming as an outsider because I came late in the project, I guessed that over 50% of the codebase and complexity could be cut without impacting on features.
Now as to whether there is truth in that projects fail because too much kool aid of X and Y is drunk I would agree. But as the controversial figure I often like to see myself as, in that kool aid list I’d add things like TDD and refactoring and maybe others, too. Rob said it himself, the team was great but sadly the bottom line is that the project was deemed a failure by the business (substantial cost and deadline overrun). Shouldn’t this be seen as symptomatic of IT (“we have great fun, we get well paid but the business doesn’t trust us!”) that, in some ways, the Agile movement is trying to solve in breaking strict interpretations of Waterfall.
Remember, we remember much more by failures than by successes, and we can take better corrective actions to small failures than bigger ones.
I’ve learned that and I’m sure many have, too.
That’s called experience.
Daniel
“Remember, we remember much more by failures than by successes, and we can take better corrective actions to small failures than bigger ones.”
should read
“Remember, we learn much more by failures than by successes, and we can take better corrective actions to small failures than bigger ones.”
The code is generally good. It’s mainly from an architectural perspective it’s overly engineered. I can see what you mean about SOA (I think someone read about the command pattern around that time too!). The system is generally running very well though. It’s just awkward to make changes to, and can be hard to debug issues when the issues are between systems.
@Daniel I agree to an extent, but you talk of trust. An estimate of 3 months for an 18 month project, that won’t instill trust! Estimates are largely wrong by nature, but they shouldn’t be that wrong. It may not be developer fault… forced estimates to meet budgets, scope creep, being asked to estimate the whole project at the beginning a la waterfall (things always get forgotten you can never estimate the whole lot). If you’re trying to do Agile but other parts of the business aren’t, then you’ll always run into problems. But as developers we’re not paid to write code, we’re paid to deliver so are largely accountable. Also you mention TDD/BDD, this (and things like logging, performance) should be kept well away from the kool aid list. They’re non functional requirements of any good project. It’s probably a bad term anyway (kool aid), SOA isn’t a bad thing, nor is the command pattern. They just need to be used correctly, and not because it’s something newly learnt.
Totally concur on the community there btw, one of the things I really like about the place. Some very talented individuals (Simon has just left, big shame). The patterns group still lives on every Wednesday too.
As a heads up to how it’s changing. Many of the apps will be merged into a singular admin app (putting in CMS too). And the admin app will be moved up to live alongside OLC (no more webservice calls and less duplicate data). We’re already merging vs projects, and plan to merge db’s down to around one per application. We’ll also be trying to remove the subsystem layer as it’s a bit RPC, and use the domain/model more directly.
Rob,
The technical changes you’ve described (removing a lot of the complexity) sound like a really good move, good luck 🙂
I’m also pleased to hear the DPSG sessions are still going and there’s still a great community.
I notice that, as usual, there is no mention of consciously trying to deliver real value to stakeholders early and at every iteration. This is the fundamental sickness of Agile Today. Plenty mention of code, none of value to stakeholder. Plenty mention of learn only at the end, none of learning what´s real at each increment.
If you had focussed on delivering measurable high value, prioritised early, then the whole discussion of 3 months or 18 months would be moot. You have a big bang mentality, not real Agile.
You need to continue iterative delivery until value for delivery cost is no longer positive. Who cares then about estimates.
You have either learned your agile from poor teachers or failed to implement the good stuff they taught. As I have long predicted we will continue to build a bad rep for agile until the Next Big Religion cathches our fancy.
I am happy you all felt so nice about your teams, but how did the stakeholders feel? Not happy bunnies if I read between the lines.
Grow up and serve the people who pay you.
Stop blamng the stupid organisation, and start delighting them with results.
See Demming, “Radical Management” for a mature criticism of Agile Manifesto (page 88), and some really mature advice (from great agile people massively!) on focussing on delighting the customer incrementally. That is what is was supposed to be all about, in principle.
http://www.gilb.com/tiki-download_file.php?fileId=456
What are the Dangers of Current Agile Practices,
“Values for Valueâ€
http://www.gilb.com/tiki-download_file.php?fileId=448
Value-Driven Development Principles and Values – Agility is the Tool, Not the Master
July 2010 Issue 3, Agile Record 2010 (www.AgileRecord.com)
http://www.gilb.com/tiki-download_file.php?fileId=431
Tom,
Thank you for you (rather acerbic) comments, but I think you’ve completely missed the point of my article, (or I’ve not explained it well enough). We (the community of people who cared about trying to make things better) fought extremely hard to try and get projects done in the manner you’re describing, but unfortunately none of us were in positions of sufficient authority and otherwise unable to persuade others that there was a better way (hence we were only able to achieve local optimisation within our teams). As I explained, the nature of the environment meant there was a high degree of politics which encouraged people to be more protective of their positions and careers than trying to improve the way we worked. People who suited this kind of environment (wallflowers and manipulative psychopaths) thrived whilst anyone who was earnestly trying to make things better got extremely exhausted trying to do so.
I stuck around for a long time even though I really wasn’t enjoying it because I felt we could improve things, but eventually (after finding even the managers I thought were supportive and openly talked about and advocated lean, kanban, agile etc were not prepared to stand up and support us when it really mattered) it was time to leave. Thankfully I’m now at a (much smaller) organisation where the level of bureaucracy, politics and bullshit is much much lower and we’re working much more closely to the way you’re describing, improving all the time, and it feels great.
@Tom to be fair you’ve made a few assumptions. It was delivered to completion in 18 months, that doesn’t mean it wasn’t done iteratively. Just the estimates/scope were off.
And regarding estimates (I don’t favour them, but), internal customers often want to know them. Purely from an opportunity cost perspective, do you build feature A or B? Given how the estimates weren’t hugely accurate though one could argue the value they brought.
It’s all good and well to say get on with it, but when you’re one man in a team of people/departments, not all of which want to follow an agile process, and you don’t have the authority to push change, then it’s always going to be a struggle. Your environment will affect you and how you work. I know, I’ve just switched from a startup (Huddle) to the BBC and can notice quite a difference. Noteably the layers of bureaucracy you have to jump through to get a simple thing done. Especially in a place like the BBC, certain habits and processes are very deeply ingrained and can take a long time to change them.
@Rob the weekly session are really good. Be sure to keep pushing them wherever you go. It keeps the dev team close knit. Of all the places I’ve worked, it’s by far the best in terms of regular sessions, full attendance, and learning new technologies.
@Rob (January 1st, 2011 at 1:08 pm)
The command pattern is a very useful one when you don’t know how the domain will evolve to accept those commands.
This enables you a greater level of flexibility and provides a natural anti-corruption layer between the UI which is the part of any system that gets the most input and the core of your system, AKA the domain.
This pattern is that the core of a trendy architectural pattern called CQRS (see Greg Young blog on Codebetter.com).
What you’re basically saying is that you’re migrating from a relatively deep layered (“overly complex”) architecture to a shallow one.
This is OK by me although none of these styles come without any risk.
Now about your statement : “They just need to be used correctly, and not because it’s something newly learnt.”. I kind of agree but you’d be rich by now if you could apply the right architecture/patterns all the time 🙂
I’ve more than once now seen “extra” software complexity just to shield developers from unclear requirements. And when you come late to a party it’s easy to have plenty of “WTF!?” going in your head 😉
Daniel
I won’t weigh in too much, as my exposure to the project Rob is talking about was pretty light. Also, I have also left recently and there are many people still there with more understanding.
I would just like to mention that within big organisations, it may feel futile sometimes instigating change. Day to day, it may appear that you are just pouring sweat down a hole.
But as I sat there, working through a variety of mostly successful projects (showcase, love earth, windmill road, listener, topgear) I directly benefited from the ice breaking happening elsewhere. Sure, I had my share if hassles, but not nearly what would have been possible if the soldiers elsewhere weren’t slashing away. In many ways I feel a bit of regret that I didn’t get involved more, and know Jason, Rob, Pete etc better.
But the lesson is that esp in big organisations, don’t give up on the right thing. Inertia is good and bad. It can feel like you are pushing a yak up an icy slope. But remember, there are dozens of
people that may be benefiting from your work, not only now but for years to come.
Big orgs employ many people, that’s why they are hard to change. But that’s why they can also be the seed for a tremendous amount of good.
Ah Yes. The OLC3 project. I remember that. I was one of the people who had to spend time dissecting the documents to glean meaningful information. I also remember hearing that originally OLC3 was intended simply as a ‘conversion to .NET from classic ASP’ and that the business sponsors assumed they were just getting a re-skin, not a whole root and branch re-engineering. It’s a bit of a hearsay story, but it’s an interesting one.
What isn’t hearsay is that the previous version was built by 3 people in 9 months. So one can see why the sponsors were somewhat peeved.
What I would gleen from the project is that ‘like-for-like’ replacements are actually some of the most difficult projects to involved in, because people have emotional attachment to want went before and also think the replacement is easy because you have something existing to refer to. The other thing is that I think valid and provocative new approaches were brought in – for the better.
However. The untimate problem was that OLC3 was a waterfall project that had aqile techniques applied to it. That doesn’t make it an agile project. Furthermore I would argue that many projects that are ‘agile’ are really nothing of the sort because of the way decisions, expectations and committments have been agreed. If I may plug my own blog I have written about what I mean. Don’t be put off by the title of the article…
http://www.mikedorey.f2s.com/blog/index.php/2010/12/30/are-you-really-agile/