(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article is also available as a podcast).
Note: I regularly have thoughts and opinions that are not a good fit for Wednesday Wisdom. To cover these, I started a new (irregular) publication called Thursday Thoughts which will host these more opinionated pieces on technology and maybe even politics. The first article is called “The EU and AI: Happy with Bronze” and it is about the EU’s ambition to come in third in a race where winner takes all. Find it here.
With that out of the way, here is this week’s Wednesday Wisdom:
One of the nice things about getting older is that it creates clarity. There are for instance many things that I used to get upset about, but that these days I just don’t give a shit about anymore. With a bit of luck I have about twenty or so years to go on this planet and I just can’t be bothered anymore by many things that I have seen happen all my life and that, despite the best efforts of a lot of people, are still happening on a daily basis.
I first had that insight a few years ago when I was dating someone a bit younger than me.
Not that much younger though, because if there is one thing that I really never ever want to do, it is explaining to a love interest who Pink Floyd is…
My dear lady friend was complaining about some stuff that was happening at work and I suddenly had this flash of insight: “Yeah, I used to be upset about that too, but now I am not anymore, because apparently I stopped caring.” It was a huge relief.
It’s not that I have gotten completely cynical; I still get upset, but these days it is mostly with myself. All things considered, that seems to be a great target for upsetness because, guess what, me is something I can change. I was for instance upset with myself the other day because I failed a practical flight test. Apparently, I am not good enough yet, but that is something I can work on and improve.
Another piece of clarity that dawned upon me in the last few years is that I am terrible at owning real estate. This is quite hilarious because between the lovely Mrs. Wednesday Wisdom and myself, we own three properties and we are always talking about buying more, because, as it turns out, we are really good at buying real estate. But then once we have it, we are really bad at doing all the work that is required to keep it in mint condition.
Owning is work. I learned this from my favorite aunt and uncle when they got a letter from the city some decades ago that informed them that per applicable law they needed to do some work on their condo. The letter was really nice about it, but reminded the recipients that “owning means maintaining”, so they were on the hook for some work that would set them back a couple of grand.
The “owning means maintaining” insight is precluding me from seriously considering buying a plane. I love flying and I would love to own my own plane, but if there is something that needs to be meticulously maintained, it is an airplane. The good thing about owning an airplane is that you are really not allowed to do any of the work yourself (bar putting some air in the tires and oil in the engine). The bad thing is that as the owner or operator you are responsible for keeping your plane legal by meeting a very strict maintenance schedule and equally strict airworthiness directives issued by the FAA.
Getting older made me realize that I am a person of intense but short-lived attention for things that (currently) interest me. The moment the novelty wears off, my mind wanders off too and I start building an equally intense and short-lived passion for something else.
My current interest is getting my ham radio license because that seems just fun and a great hobby for a nerd.
It is a miracle really that I managed to raise a child and am managing to stay married because both of these require long-term effort. But then again, the former is something you don’t have to do all by yourself and with respect to the latter: With Mrs. Wednesday Wisdom you really never have a dull moment 🙂…
The “owning is maintaining” mantra also holds for software.
Some years ago I joined a startup and during my ramp-up I wrote a piece of software that did something useful but also potentially dangerous. I took lots of care to make sure that my code did the dangerous things correctly and safely, and after an extensive canary process we released it into production, where it behaved wonderfully well. After some time, I transferred ownership to the appropriate team. In its new home, a junior engineer was assigned to take care of my software and unfortunately it was immediately clear to me that they didn’t really understand how this thing worked, nor where the given the time and incentives to make that so. I was there for a bit to hold the junior’s hand, but eventually I went on to do other things.
A week ago I got news through the grapevine that, despite my best efforts, my code had recently gone rogue and that some of the bad scenarios it was designed not to let happen, actually happened. I don’t have any of the details, so I don’t know what transpired, but what I assume happened is that the code functioned to everyone’s satisfaction for years and then people stopped looking at it closely.
Every piece of software has sleeping bugs in it and has unwritten (and often badly understood) expectations about the exact implementation of the services it depends on. As the environment changes, the still working code ambles towards the edge of the cliff and then finally, Wile E. Coyote style, it tumbles into the abyss below. Many catastrophic software failures resemble the process of a company going bankrupt: Slowly at first, then suddenly.
When analyzing software failures of this nature, there is a lot of emphasis on the final step over the edge of the ravine, but I am just as interested in the ten steps before that fateful moment. Sure, that last step sealed the code’s fate, but it was the ten steps before that brought it to the place where the final misstep could happen. Careful attention to what happens during the everyday running of your code can find these steps in the wrong direction as they happen. Unfortunately, not a lot of people look closely at the behavior of working code, so they miss the signs that the code has started the long walk towards the edge. This brings to mind a lemma by Sydney Dekker: Usually, everything that can go wrong doesn’t, and then people learn the wrong thing.
As a student pilot, I am taught to think about my decisions in terms of “outs”. If I make a decision to do or not do something, how many “outs” do I have left if that decision turns out badly? If you look at airplane crashes, the focus in the popular press is usually on the proximate cause of the crash, which is the last “out” that went away. For instance: The pilot and plane were caught in bad weather and therefore they didn’t make the landing and crashed because of windshear or downdrafts or something like that. But if you look a bit deeper, you often see a series of steps, each of which took away an “out” and brought the whole situation to the point where there was only one “out” left. When that last “out” went out of the door, the crash occurred.
A recent episode of the “I Learned About Flying From That” podcast describes a scenario where the passengers showed up later than expected but the pilot still decided to leave. Because of high fuel prices at the Leeds airport, they decided not to take on fuel. Now it was getting dark and because of the late departure, the incoming bad weather was more of a factor. And because of higher landing fees at airports that support instrument approaches, they had decided not to file an instrument flight plan. Each of these decisions took away an “out” and eventually they barely made it back to the ground alive, ultimately depending on friends from the flying club to line up their cars and shine some light on the otherwise unlighted grass runway where they had to land. They were lucky.
Software works like that too. If you are trying to write reliable software, you take a “belts and suspenders” approach that makes sure that no single thing breaks your code. But, time will eat away at your carefully designed provisions, taking away one “out” at a time. Eventually, your code has become dependent on a single thing working and when that dependency fails, you now have a total system failure. This is where careful ownership comes in. “Owning is maintaining”, so as an owner for a piece of software you are responsible for making sure that you know that this is happening and then addressing it by creating new “outs”, putting in new belts and new suspenders in all the right places.
Unfortunately, this is not popular work. As a profession we have serious Shiny Object Syndrome and most people would rather work on something new, hip, and happening than spend time keeping the existing code base in a healthy state (meaning: with ample “outs”). Our recognition and promotion processes are counterproductive in this respect too. “Designed a new system and put it in production” will get you promoted, whereas “kept our entire code base humming along and there were zero SEVs” does not.
In my career I have come across lots of teams that like having ownership but that seem to be unwilling to do the work that is implied by ownership. The amount of times that I have been told that I cannot see or touch something because “we own this” can no longer be counted. Yes, I know that you own this, but it is broken or I need a feature and you are not doing the work! Much too often, ownership is about status, not about signing up to do the work.
“We are the foo team, we own the bar system.” Yeah, fine, great, who cares. Ownership should not be static. You should be valued for your work, not for the number of acronyms on your wiki pages. For the systems you own, do you ever look at the logs and metrics? If not, you are not the owner, you are, at best, the lazy guardian of a bunch of orphans, anyone of which could become the next Voldemort.
Share this post