On Raising Flagpoles

A treatise on dealing with details

(Like this article? Read more Wednesday Wisdom!)

The following story might be apocryphal, but I like it nonetheless: The brother of a friend of mine was up for a promotion in the New Zealand Royal Navy. As the final step in the process he had to appear in front of a panel of officers to answer questions and perhaps game out a few scenarios. One of the staff officers posed the following:

“Imagine you are a lieutenant and you are tasked with raising a flagpole. You have some wood, two shovels, various other tools, a sergeant, and two enlisted men. What is the first order you give?”

My friend's brother thought about it for a second and said: “Sergeant! Raise that flagpole!”

I always thought this was a brilliant response: Of course this is the right answer. Raising a flagpole isn’t that hard, the sergeant and the two enlisted men can totally be trusted with this. Heck, I would be quite surprised if the sergeant in turn wouldn’t have said something similar to the enlisted men. This really isn’t something that requires the lieutenant to spend time on.

I, on the other hand, am a detail-oriented person. I am both a computer scientist and a lawyer, and in both of these professions details matter an awful lot. Getting the details right is the difference between a piece of software working or not (or winning in court or not). Of course I pay attention to details, details are the bread and butter of what we do here.

However, in most of the projects I am involved with, my work is not about the details at all. Somebody better pay attention to the details, but mostly that somebody is not me.

As a very senior engineer one of my problems is to know when to care about details, which details to care about, and to have a clear idea why I need to care about them. You cannot let go of all the details all the time. But you can also not pay attention to all the details all the time. Choices have to be made.

The easiest details to let go of are the unimportant ones. These are the things where it doesn't really matter if we get them “right” or not. 

For instance, I read a fair amount of design docs written by non-idiomatic speakers of English and some of them are full of grammatical, style, and spelling errors. I sometimes start correcting or commenting on them, but I almost always give up after at most two paragraphs. Why? Well, first of all, I don't want to be an a--hole and secondly: I got bigger fish to fry. I care about grammar and spelling a lot because ultimately it goes to document correctness, but by and large I understand perfectly well what is meant. If it's really bad, I might refer the author to someone else to help them.

Sergeant: Fix that document!

The next level of details are the ones that are not unimportant but also not crucial. Code reviews come to mind. I care a lot about idiomatic and clear code because that goes directly to correctness, our ability to debug, future maintenance burdens, and performance. However, when looking at a piece of code, I can often not afford to go into that level of detail. Every now and then I do, mostly just for the fun of it, but again, I usually have bigger fish to fry. If things are really egregious I should refer the code author to an experienced team member or to an expert in the language.

Sergeant: Review this code!

Then there are the details that really matter… Once I was guiding two teams to work together on a new service. Team A needed some data from a data store that Team B was responsible for. Unfortunately, the data structure team A needed was not readily available and needed some expensive pre-processing. Obviously, team B would need to run some preprocessing steps on the data and then store the results somewhere (this is life in the NoSQL world). To ensure that we got the requirements right I proposed specifying the API and getting agreement on that before starting any design and code work.

Total rocket science, I know...

APIs are contracts and it is extremely important to get them right, including all the details. One of the main reasons to get all the details of a contract right at the start is that you often only start looking at the contract closely when things do not go as planned but you are already committed. When everything and everyone is hunky dory and is meeting expectations, you never really look at the fine print. But as soon as things are no longer awesome, that's when the fine print comes in. 

Come to think of it, they should call it the "not fine print", because it's the print you read when things are not fine.

Sergeant: Write that API spec!

However, since APIs are very important to get right, down to the details, I applied another another principle I learned form an armed forces specialist (a colonel in the Dutch Royal Air Force): “Trust is good, verification is better.”

The first draft of the API was a text document with a very loosely specified set of JSON structures that conveyed something like: You can call our API with this struct , which might contain a few members like that, and then in return we give back another struct, which looks a bit like this, and might have these members. From this loose specification it was not at all clear which request members were required, which ones were optional, what the types were, which fields would always be present in the response, which fields might be present, which error returns were possible, or what the maximum latency was.

In other words, none of the fine print was there. My lawyer heart cringed…

So, I intervened and convinced everyone that we should specify the API very crisply in a modeling language that allowed for exact specification of the API, including input, output, and errors. This specification included the domains (types and allowed values) of the fields, which immediately prompted discussion on these allowed values. “Why can this field only be 1 to 4? Why not 5? And what would happen if in the future we would want it to be 5?” These are great discussions to have up-front! Paying attention to the details at this point in time means you are setting the right expectations. These are details that I should be involved with.

Sometimes the details matter somewhat but I decide not to care because a) everyone has a right to their own destiny, b) you need to allow people to make mistakes (how else would they learn), c) they might be right, and d) it would take too much time for me to do all the work myself.

Staying with that same example: Team B needed to preprocess their data set and then store the preprocessed data somewhere for easy retrieval through the newly minted API. But where? What kind of storage system should be used? It's a big and important detail because, ultimately, whether the whole solution works or not depends on whether that infrastructure actually works well enough for the use case.

But here is the crux: This is one of these details where there are great answers, good answers, suboptimal answers, and bad answers. As long as we don't choose a bad answer, everything will work just fine. Additionally, if it turns out the chosen storage solution is not as awesome as we need it to be, we can always go back and reimplement it using some other database. Sure, that'll cost time and money, but on the whole it is preferable to be probably right-ish right now and start working today, than to overindex on making the absolute best choice and miss the window of opportunity.

In the parlance of my current employer: This decision is a two-way door: If we pass through it and decide that it was the wrong option, we can go back through it and choose another one.

So even though I love looking at technical infrastructures, enjoy figuring out exactly how everything works, and then make the best decision, this is not really a detail I need to be involved in; the only thing I need to make sure is that we don't make an absolutely bad choice. 

Sergeant: Choose that infrastructure (but show me the plan before you start writing code against it).

My personal problem with details is that it is really easy for me to get lost in them. Getting the details right is fun and rewarding and I am good at it, but ultimately it is not what I am paid for. So when the tech lead of the aforementioned project asked me if I wanted to be involved in the implementation, I unfortunately had to reject his kind offer because although it would be fun, it would also distract and take away time from the things that I need to accomplish in order to get a decent performance rating.

The lure of details is that, for me, losing myself in the details is often the easier path.

When faced with a number of things that I could be working on, I have a tendency to work on the things that I know how to do, and that is often figuring out the technical details. In the most positive interpretation of that behavior this allows me to get the details out of the way quickly, after which I can start paying attention to the hard things. But that is really a fallacy; there are an infinite amount of details to take care of and if I go down that road, I will never get to the hard things.

Details are fun and emotionally rewarding but they are the fast food of engineering: Lots of easy calories and readily available, but ultimately very bad for you.

The story that I started the article with offers a great meme to help me stay on the straight and narrow. What is happening here? Am I raising a flagpole by myself? Are there people who are better suited to doing that work? Am I getting my hands dirty without actually adding a lot of value?

I can raise all the flagpoles I want in my spare time, but at work it is mostly someone else’s job to engage with the fun bits of our profession.