Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?
This week we are covering a reader question. F. contacted me and asked: “Hi Jos! Talking about anxiety, do you have any advice on how to manage expectations and deadlines with managers? I work as a software engineer in Big Tech and recently my manager is stressing me a lot about giving accurate ETAs and being able to meet them.”
At first, I thought this was maybe a question about how to plan accurately, but after reading the question again, it became obvious that it was in fact a question on how to manage upwards for software developers. With that cleared up, let’s dive into it…
Right off the bat, we need to realize that software development is incredibly hard and expensive. By and large, you should not do it unless absolutely necessary.
Reading tip: No, you don’t want an app, which I wrote 3-4 years ago when yet another hopeful person asked me to help them develop software for their company and I had to explain (again) why this was much harder and much more expensive than they thought it would be.
You and F. might think you are in the business of developing software, but you are not. You are in the business of solving certain types of problems and writing software is how we do that. And because writing software is expensive and everyone needs to keep their costs down, it stands to reason that you should develop as little software as possible.
I assume that both F. and their manager know that no software project in the history of the entire universe was ever executed according to plan. Lots of reasons for that, the number one being that even though the production of software might look like a manufacturing problem, it is in fact not. Instead, it is a weird combination of research, design, discovery, making a piece of art, and building a working prototype.
All the aforementioned things are hard to do according to a plan because of the unknown unknowns, both in terms of what we want to build and how we should build it. To a large extent, this is why Agile development was created. People who insist on an accurate delivery date for a piece of yet-to-be-written software are either completely in denial about the basic nature of software development or they are willing to make significant compromises on requirements.
In every software development project there are three major dimensions that are constrained: Time, money (typically meaning: manpower), and requirements. At least one of these needs to be variable. Demanding “an accurate ETA” means that time is fixed and since Fred Brooks already taught us that adding manpower to an already late software project makes it later, the requirements are really the only candidate for the variable dimension. It is easy to plan a software project accurately if you are deciding up-front you are going to be happy with whatever can be produced by the delivery date.
If F.’s manager is in denial about the fundamental nature of software development, there is really only one thing to do, which is to find another team on the double. If a manager requires a fixed delivery date but is not willing to compromise on the requirements, there is no way that you are going to meet their expectations because, no matter how brilliant you are, you cannot change the physics of software development.
While writing this, it dawned upon me that there is of course a way to make time, money, and requirements fixed, which is if you manage to set the accurate ETA so far in the future that it is entirely conceivable that you will manage to deliver the requirements by that time. If you get a year to put a “Hello, world” program into production you will probably not fail. This approach is, of course, entirely fictional.
If F.’s manager is not totally incompetent, they will know about the fundamental nature of software planning and then the question becomes what they actually mean when they say that they want an “accurate ETA”. This question of meaning is much harder than it seems, because people are terrible at expressing themselves correctly, as witnessed by everyone rolling their eyes when the local nerd patrol answers “Yes” to the question: “Are you not hungry?”
So what does insisting on an “accurate ETA” mean? One possibility could be that F.’s manager is of the opinion that it is totally doable to meet the requirements within the time and people allotted to the project. If F. is of the opinion that this is not achievable we have a difference of opinion and the recommended strategy is to figure out where this difference of opinion comes from. In my experience, these types of differences of opinion usually stem from the fact that F. and their manager live at completely different layers in the problem fractal.
The nature of the problem fractal is that a lot of problems look incredibly easy when viewed from a high enough level, but at the same time seem really complicated when viewed at a (much) lower level. Managers often live in a weird state of knowing enough about computers to know that most problems are solvable, but not knowing enough about computers to know how incredibly hard seemingly simple sounding problems are. “Ok, so you need these low-latency caches in multiple regions to be consistent, how hard can that be?” Answer: “Very hard”, but that might be hard to explain to someone who only has a helicopter view of the problem.
Software developers typically only have a vague understanding of the problem they sign up to solve and only during execution do they figure out how hard it actually is. The only remedy for this is a planning process that is so exhaustive that you basically build the entire thing in order to figure out how to build it, then throw it away, and build the real thing. Nobody does this anymore. Mind you, we used to do this, but then customers were upset because they thought they already had a working version that we knew to be only the prototype. This led to many prototypes being deployed in production, with unsurprising results.
Instead of doing this exhaustive planning, we start with a vague understanding of what we need to build and how we need to build it, and then push someone for a date. Any software engineer in their right mind would refuse this, but eventually nobody can withstand the pressure to pick a date, any date, and eventually one gets thrown around.
Now we have a problem because, more likely than not, you get the go-ahead on the project only because of this entirely fictional date. Eventually, we find out that things are going to take much longer and hence will be much costlier than expected. This leads to significant frustration all around, especially in the manager, who in a sane organization will be responsible for this overrun. Plus they probably committed to other teams that this new thing would be ready by a certain date. And of course there is the budget and, not to be forgotten, the opportunity cost of the overrun.
The best way to prevent this problem is to never commit to a date until you are absolutely certain that you can do it within the time allotted. This is usually impossible. Even if you think you got line of sight to the solution, problems crop up left and right because our industry sucks and for one reason or another we constantly forget that.
Here is an example: Some time ago, I was (tangentially) involved in a security project to enable SSD encryption using features from the Opal Storage Specification. “How hard can that be?”, you would be forgiven to think. We have been encrypting stuff for decades now and the problem is well understood. Having the hard disk do the encryption is entirely possible and the interface to control that should not be rocket science. This project sounds like a week or two of work at best.
Oh, you naive flower child, how mistaken you would be, because of course the storage vendors managed to completely eff up the implementation of Opal again, leading to a variety of deep problems, including vendors implementing the standard incompletely, implementing it incorrectly, or just implementing it badly, leading to significant security vulnerabilities. Things are in fact so dire that the smart money will advise you not to use Opal but to use Linux’s LUKS disk encryption instead, which is a bit slower but has the upper hand on Opal because it does not completely suck.
Software development is problematic because every piece of software is a one-off. We are never able to learn from previous software development projects because every project is an entirely new universe of problems using different software and different people. Let me remind you again: Software development is not a manufacturing process.
If F. and their manager live on different planes of existence when it comes to technical detail, that is an information gap that needs to be fixed by F. and by good project management, including regular status updates including progress, newly identified risks, and the progress of the mitigation of previously identified risks. These status updates will also need to identify the extent to which the timeline is under pressure, so that they can discuss what to do about that at the earliest opportunity. If the timeline cannot be met, F.’s manager should not learn that only when the day of the “accurate ETA” rolls around!
The problem with status updates like these is that they are usually depressing and people hate giving out and receiving bad news. This then leads to the concept of “Watermelon reporting”, which happens when a project officially looks green on the outside, but is red all the way through once it is cut open.
The project management aspect of software development is not fun but, as I am wont to say: “That’s why we call your salary compensation”. Really, if work was fun, you would have to pay the company to do it as they’d be selling tickets at the entrance for the privilege of coming in to work. To make matters worse, the brunt of this sort of project management can typically not be delegated to a project manager because they live in the same information black hole as the manager. Project managers depend on accurate status updates from the software developers and, from experience, have a really hard time getting these for the reasons outlined above.
In a successful software development project, managers must be made participants in the process, get a say in how to address newly identified risks, and co-decide how to deal with timeline pressure: The responsibility for either delivering on time with reduced requirements or extending the timeline must be shared!
If F.’s manager is not interested in playing their part in the software development process, I revert to my earlier advice on finding another team quickly. It is impossible to work with a manager who does not want to be part of the solution, but does want to criticize you when you are not meeting goals that you were probably only partly responsible for defining. So if your manager asks you for an “accurate estimate”, whatever you do, do not provide a date until there is an agreement on the process of sharing responsibility for managing the process of developing that software.
Of course, it is also entirely possible that F. is an underperformer and that the manager is the good guy in this whole story. But let’s be honest and feel free to draw upon your personal experience here, how likely is that to be true?
