I never approve of anything (except of you)

Most approval processes are a train wreck...

(Like this article? Read more Wednesday Wisdom!)

In the dim and distant past, when dinosaurs still roamed the Earth, I was working a contract at the global transaction services division of a large Dutch bank. The bank had a very elaborate system of production change management which required a lot of approvals from every corner of the organization. This made changes slow and cumbersome. It was not uncommon for me to spend an entire morning on a task such as creating a new directory on two production machines.

THIS IS NOT AN EXAGGERATION!

When asked what I did for a living I used to say: “I solve simple problems in a complex organization.”

Whenever we wanted to install some software on a production machine we had to write an extensive instruction for the system administrators who had to perform the actual installation. That manual was hard to write because we didn’t quite know the setup of the production machines, to which we did not have access (because: security). Whenever there was an error in the installation instruction, we would get a report back indicating at which step the instruction had failed and what the error message was. The underlying problem could be as difficult as a shared library version conflict (potentially hard to solve) or as simple as “/usr/local/bin” not yet existing on a particular machine.

When this happened we had to augment the installation instructions with extra steps and then jump through all of the approval hoops again. Consequently it would sometimes take days for us to install a very simple piece of software on all machines.

You’d think that a strong approval process like this helps lower risk, but really it does not. An approval is only as good as the person who gives it and in this case most of the people who needed to approve had no idea what we were trying to do or how computers actually worked. The approval process didn’t mean that things would go right; it didn’t catch mistakes preventively. Getting approvals was just a chore, a bunch of hoops to jump through without any benefit.

The approvals process didn’t help overall system security either. In order to help us write better instructions we negotiated a “read-only” userid on the production machines. Once that happened I found a setuid shell script, which of course helped us tremendously…

My boss at the time once spoke the following magic words: “I never ask you how you do what you do, because once I know, I have to do something about it” 🙂.

Approvals seem like a decent idea: Someone wants to make a change which impacts other people and which might break something. By forcing that someone to explain exactly what they are going to do and then having knowledgeable people approve that course of action you hope to reduce the probability of an incident. Additionally, approvals are a way to give the people who are formally responsible some practical authority over a change.

I have seen approvals work well in situations where there is a fragile common resource that is critical for the correct operation of the entire system. For instance, at one company I worked at there was a component that saved the logs from all production instances into a centralized cluster where they were backed up, indexed, and made accessible for analysis. This infrastructure was essential for the most critical business process of all: Billing.

Since nothing is ever perfect, it was possible for a set of tasks to go wild and to overwhelm the log saving process with requests or exhaust the available disk space, killing the logs cluster. To prevent this from happening there was a review and approval process you had to go through before you could start writing logs. I always thought this made a lot of sense because the logging infrastructure was complicated and it was not always obvious to everyone how to use the system correctly. Furthermore, the consequences of a mistake could be very bad. This particular approval process was not terrible, as the SRE team had done a great job outlining what you needed to tell them and once you filed the ticket a very knowledgeable person would look at it and respond within three business days.

There are a few elements that made this process work well: It was clear what the approval was for and what information you needed to provide to help the reviewer make the right decisions. Once you filed your request there was a service level on the response time, which made it easy to include the approval in the overall project plan. And, most importantly, the people who were reviewing your request were knowledgeable, which helped make the process efficient and which ensured that the approval meant something.

A lot of approval processes don’t work that way. And they definitely did not work that way in the aforementioned bank, where it was entirely not clear who the approvers were, what they would be looking for, or how long they would take. Additionally, more often than not, the approvers were not necessarily always very knowledgeable, leading to delays and sometimes very dumb questions.

To add insult to injury, there is some pernicious corporate effect at work in most approval processes: Since an approval process makes you responsible for any incidents, the best answer to any request for approval is to reject it.

“We are the knights who say …. No”

In these big bureaucratic processes, the various approvers typically have no stake in the outcome of the project whose changes they are asked to approve. Consequently they couldn’t care less if the project happens on time or not, or if it happens at all. What they do know is that if they approve and something goes wrong, there is a chance that things come back to them. From these parameters the optimal behavior follows: Just say no!

If for some organizational reason you can’t say “no”, there are three intertwined strategies that are almost as good:

1. Do an extremely thorough job reviewing the request, taking all the time you need.

2. Ask for more information. And then some more…

3. Identify every possible problem, no matter how small and no matter how low the probability of it occurring, and ask for changes.

Surely if you are thorough and need more information that just means that you are trying to do a good job, right? Who can fault you for that?

In one organization I worked for we had an extremely well run production change request (PCR) system: You entered your PCR into a system, outlining what you were going to do, when you were going to do it, and what the impact was. Then on Monday morning there was a one hour meeting where all the PCRs for that week were discussed (briefly). No objections meant approved. If you had an objection but weren’t at the meeting? Too bad for you, as that meant your objection went unheard. The focus of that meeting was to disseminate information and give people a chance to ask questions and figure out what they needed to do to protect themselves from any adverse impact.

Even a set of well-run approval processes can create barriers that are, in aggregate, insurmountable.

At one company we had accumulated so many reviews and approvals for the launch of a new system that timelines had become ridiculous, even though most of these processes were good and made (at least some) sense. At some point we ran a project, aptly called “Train Wreck”, that sought to reduce the launch review and approvals checklist to more manageable proportions. A lot of the processes were replaced with automated “self-service” systems or with better controls in fragile infrastructure so that they would be better able to protect themselves against rogue clients.

Outright removing reviews and checklists requires trust in people and a healthy appetite for risk.

I have both 🙂…

Recently I set up and ran the engineering-wide design review process for the organization I worked for. I made it clear from the outset that this was not compulsory and not an approval process. I want teams to come to the design review if and when they think it is valuable for them. I also want them to weigh the various pieces of advice they get against their timeline and their other goals, and not see these as edicts from some council of overseers sitting in an ivory tower. At the end of the day I want every team to be in charge of their own destiny and not beholden to the opinions of a bunch of people who do not have a stake in the outcome of their project and who are not responsible for the tradeoffs that every project needs to make.

You can always ask for my opinion and advice, but I will not approve or disapprove of what you want to do. Unless it all goes wrong of course, in which case I wholeheartedly disapprove ex tunc 🙂.

---

You are at the end of the article. Please choose one or more of the following buttons to continue your adventure:

---