(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article is also available as a podcast).
Note: This article is a reprint of an original that was published on 5/25/2022 in the walled garden of an intranet, which is where Wednesday Wisdom lived in the days before it escaped into the wild.
In the late 1990s, I worked on a huge project that sought to redo the entire global transaction services of a large Dutch bank. The bank had developed big international ambitions and so it thought it needed a big overhaul to create an ultra-modern globally integrated set of services for its large customers. It hired the most expensive consultants it could find and they set about designing a shining city on a hill.
Book tip for the Dutch readers: De Prooi.
There are lots of reasons why this was one of the most memorable projects I have ever worked on, but one reason stands out in particular: It failed hugely. The bank spent hundreds of millions of euros and the project went absolutely nowhere: We rearranged lots of toner on blank pages, but not one useful system was built, let alone rolled out. Eventually, the main contractor was fired and the project got scaled down significantly. Even then it struggled mightily, but something eventually happened that they could chalk up as a (minor) success of sorts.
IT projects are extremely prone to failure. A 2009 IDC report stated that 25% of IT projects experience utter failure. The rest does not fare much better: About 50% of projects require rework and 20-25% fail to provide a return on investment. If there is any rule, it seems to be that the bigger the projects are, the harder they fail. According to Gartner (who doesn’t love them :-) projects with budgets of over 1M USD are 50% more likely to underperform than smaller projects. They also concluded that 31% of projects under investigation were deemed complete and utter failures, 52% were partially unsuccessful, and only 16.2% of projects were completed successfully (source).
These are sobering numbers and they should influence our behavior.
Personally, I am very interested in failure, to the point that at in the past I co-ran the Boston chapter of F*ckup nights. This is a worldwide series of events where people with experience came to talk about things that went horribly wrong in their business or project. Because, remember: Experience is what you get when you were expecting something else…
If you have never been to a F*ckup nights event, find your local chapter and attend a few; it’s a lot of fun.
Why do things fail? And why do huge IT projects in particular fail so often?
Fun side story: In law school I had to take a class on how to argue a case. The case I got assigned to defend was that of an IT contractor that had messed up a project. I felt right at home and obliterated opposing counsel…
Before we get into the finer details of why projects fail, let’s begin by calling out an uncommon cause of failure: A lack of qualified people.
Of course, trying to pull off a project with below-par staff is doomed from the start, but in my experience this rarely happens. Of the failing projects that I partook in, I have a lot of doubt whether better engineers could have pulled it off. On the aforementioned global transaction services project I worked with some of the smartest people I have ever worked with, all experts in their field, and all with impressive amounts of knowledge, skills, and experience. But still, the project failed hugely, despite having more than enough talent. This is of course a known phenomenon in sports too. If the talent of individual players was the only critical factor, surely the French soccer team Paris-Saint Germain (PSG) would have won a major tournament by now.
But, they’ll have another shot at it on May 31st. Allez!
Outcomes do not (solely) correlate with the expertise of the people working on the project. If it would, most projects would probably not fail, or at least not to the extent that they do. Individual expertise surely matters, but what does it actually matter for? And what are the other things that matter?
In my experience, projects typically also do not fail because people have no idea what needs to be built. The idea that projects need to start with a requirements gathering phase has been solidly entrenched for decades now. All the way back in college, the first project we did for our Practical Software Development class, started with requirements gathering; the idea is that old 🙂. Sure enough it’s a tricky job and hard to get right, but I daresay projects do not fail because we have no idea about the requirements. If anything, a lot of work has been done to make this part better, for instance by recognizing the role of the Subject Matter Expert whose job it is to ensure that whatever is built “meet the needs of the stakeholders, legislation, policies, standards, and best practices”.
That said, a failure to completely grok the details of a problem is still a major obstacle in writing correct programs, something I wrote about somewhere else.
So even if we know what needs to be built and we have the right people to build it, a lot of projects still fail. Why is that?
In my experience the major reason for failure is too much ambition coupled with an underestimation of the complexity of the road to get there.
Projects invariably start with a lot of energy. There is a problem and we are going to fix it, because we got ideas! A lot of projects start with an unconstrained phase where people are brainstorming ideas on how to solve things. However, they typically do not brainstorm on how to solve it; to the point that they do it is usually a complete technology push where architects get to propose their hobby horse “du jour”, like data streaming, Spanner, or a brand new yet to be built workflow system.
Most brainstorming discusses the future ideal situation.
This is an example of a common problem with humans, which is our tendency to focus on the fun part of any activity that we are engaging in. The future ideal situation is a fun thing to think about. We are completely unconstrained and can therefore imagine a perfect solution, a system that works flawlessly to solve the set of problems at hand. It is here where Powerpoint architects excel, drawing impressive diagrams with boxes and arrows and employing the latest technologies and patterns.
Unfortunately, we are almost never in a green field situation where we can build a solution in perfect isolation. And even if we could, by the time we are done building that solution, the problem will have morphed to something else altogether for which our brand new solution does not work very well.
Since we are not in a green field situation we usually need to fix the problems at hand by iterating from one working situation to another one. The need for iteration constrains us mightily in all sorts of annoying ways. For one thing it turns the project into a much larger endeavor than anyone imagined because there is all this work involved in ensuring that the system we have continues to work. It also means spending time and energy on migrating from one state to the next. I daresay this is nobody’s favorite part of any project.
Figuring out the myriad of details that will allow us to evolve the existing system into a better state, one small step at a time, is important and often not fun. And since it is not fun, not a lot of people want to work on it or think about it. Surely it is much more fun to build a new workflow engine than it is to figure out the nitty gritty of how today’s systems and processes actually work (which is usually completely different from how people think things work :-)
The users, developers, and administrators of any system that is in production are like people living in mud huts in a swamp. The mud huts are rickety, the mosquitoes are a menace, and there are swamp fires everywhere. Additionally, some of the bigger buildings that are foundational to the entire endeavor are sinking into the ground. Clearly what we need is a shiny new city! So, the city planners get to work. Before long we are discussing whether the public transport system in the shiny new city needs to be a subway, a tram, or electric buses. This gives rise to many meetings, documents, and heated discussions. In the meantime, the people still living in the swamp are drowning in the mud.
The need to iterate from one working situation to the next could preclude certain ideal futures. To stick with the swamp metaphor: The need to keep the swamp working for all people who work and live there today, might preclude us from building a subway. Can you imagine building the underground system of London today (assuming that the city is exactly like it is today, but without the Tube)?
Of course some high level idea about the future situation is required, but the level of detail needs to match the timeline. If we guess the shiny new city is three years out, then the only thing we need now are sketches of what needs to go where approximately.
What kills many projects is a mismatch between the level of ambition with respect to the goal and a neglect of concrete idea on how to get there. Most projects that fail have a huge ambition level coupled with a fantastically clear vision on the future ideal situation, but without a plan on how to get there in small steps that iterate from one working situation to the next.
Sure, the nitty gritty of the small steps to the bright new future is not a lot of fun (unless you are me, since I have a warped idea about what is considered fun), but let me repeat two mottos that I discussed in earlier articles:
Work is not fun, which is why we call your salary “compensation”.
Ideas are cheap, execution is what matters.
The beauty of coupling a high level idea of the future with concrete plans on how to get there is that you start making things better immediately. Additionally, it will be easier to adapt the future situation to the ever changing current circumstances.
Your value to the company is not that you can design the shiny city on the hill, that’s the easy bit. Your value is that you know how to get there while keeping the swamp that you are in right now functioning.
Share this post