Interviewing has become a mess

And I am not sure who to blame. Entropy, maybe? Or corporate gravity?

(Like this article? Read more Wednesday Wisdom!)

Recently there has been an abundance of posts and articles decrying the amount of effort involved in modern tech interviewing. And not only for engineers; on LinkedIn we can read posts where people applying for various other tech roles are asked to do multi-day take-home projects. To make matters worse, it appears not uncommon for positions to be closed without any explanation after all the effort has been put in. Clearly something is going on…

For most of my professional career, tech interviewing was not a thing. I got my first job by studying the org chart of the bank I wanted to work for and sending a letter to HR saying that I was interested in a role in a particular team. In response, I got an invitation for a chat with my future manager in which we discussed my qualifications and why I wanted to work there. No technical questions, no coding challenges, no behavioral interviews, nothing. 

Though I did go through a fun full-day psychological evaluation where it was determined that I was compos mentis enough to be a systems engineer in an IBM mainframe environment.

That was pretty much deal for my subsequent jobs; interviews were very light chats that focused more on motivation and establishing rapport than about evaluating technical skills. Eventually, many years later, I was interviewing for a consulting role at a healthcare tech company and one of their engineers came in and asked a few technical questions. I remember being upset: “Wait, what now? Surely my resume speaks for itself!?”

In my own little consulting company tech interviewing wasn’t much better. We did usually discuss some technical matters, but there was no systematic evaluation of technical knowledge and experience. It helped that we mostly hired through recommendations from people we knew, but we were not very diligent in affirming the technical qualifications of the people we hired. Looking back that seems weird, but it was the nineties and we were on a holiday from history 🙂.

This all changed in the early 2000s and I think Google is mostly responsible for this. When I was invited for job interviews by Google in 2006, I had heard that these interviews were super hard and technical, but two-decades-younger-naïve-me did not research, had no idea what to expect, and consequently did not prepare. Fortunately I can bullshit with confidence and think on my feet so I managed to squeak past the hiring committee.

Personally I think that the deciding factor was a surprise answer to the Robots on a Plane interview question. When it was pointed out to me that a circular search pattern doesn’t work because the planet has an infinite size and memory in the robot is limited, I responded that I would then first invent infinite memory, which would be doable because I had infinite time. This got a laugh out of the interviewer because he apparently had never heard that particular answer before 🙂.

On another note: It is a terrible interview question, but I have used it myself as a time-filler when I had already made up my mind and had all the data I needed for the hiring decision.

Google turned tech interviewing into a system, complete with manuals, guidelines, beginner courses for new interviewers, advanced courses for experienced interviewers, question banks, site-based interviewing clubs, feedback templates, and whatnot. Their goal was to take interviewer variability out of the loop; a candidate should have approximately an identical chance of passing regardless of the interviewer assigned and the questions asked.

I am not authoritatively stating that Google invented this style of interviewing; I got some anecdotal evidence (remember: The best kind of evidence) that Microsoft did something similar already in the nineties. I do think that Google took the method to its logical extreme and popularized it throughout the industry.

Google’s was a valiant attempt at creating a better interviewing system for tech roles, but ultimately it was self-defeating for the same reason that all systems involving humans are self-defeating: The actors in the process are motivated to pass the test and hence evolve to explicitly address the test, thereby making it less reflective of the thing you actually want to get an insight into. Remember: You get what you measure, so if you measure the ability to hack out algorithms in Python to solve toy problems on a whiteboard in 45 minutes, you get people who have trained for that specific thing. But, I am running ahead of myself a bit.

For most things in life, ranging from baking bread to interviewing, using a method is obviously much better than just relying on inspired chaos. Google’s results seemed to validate their method: When I started at Google in 2006, I was easily the dumbest person in a building full of geniuses, all of which had managed to pass the interviews. Clearly, Google’s system only allowed the smartest people to pass. Consequently, Google’s interviewing method was adopted by many other companies and especially by tech companies in Silicon Valley. Changes were made, but I still recognize Google’s interviewing system everywhere I go.

Google’s interviewing method was not secret and so people started to prepare. Initially this took the form of candidates sharing interview questions on various sites but by now there are books, courses, websites (obviously involving AI), forums, and even entire companies that will help you prepare for success in the interviews. I myself add to this madness by doing mock interviews for friends and acquaintances to help them succeed.

All of this preparation has impacted the field in two ways, which conspire to drive down the overall quality of the workforce.

First of all: Mediocre candidates now frequently do well, courtesy of solid preparation They might for instance spend lots of time on LeetCode or read books like Alex Yu’s excellent System Design Interviews. The overall quality of interviewers was never that great, but together with well prepared candidates, the signal value of the average interview has dropped significantly. A mediocre interviewer who asks a mediocre, but well-prepared, candidate to invert a binary tree is not going to come to a good conclusion about this candidate’s skills. Same if you do a mediocre job asking the URL Shortening system design question.

Secondly: Good candidates frequently do not do well, for the simple reason that they cannot always reproduce the stock answer to a stock question. Good interviewing will erase that sad fact, but good interviewing is rare. It is for that reason that whenever I interview, I prepare like a mofo. For my last series of interviews I solved most of the 2023 Advent of Code and I went through a number of the hard problems on LeetCode. I have been coding for over forty years, but it is really necessary to do all of this preparation. Mutatis mutandis, when I do a triathlon, I also still train, even though I have been able to walk, bike, and swim quite competently for over fifty years. Like triathlons, tech interviewing requires muscle and even the winners need to put in the training hours.

In interviewing, raw talent rarely cuts it, unless the interviewing is excellent, which it rarely is.

The deeper problem, and this is something that Google already figured out more than a decade ago, is that interview performance is not indicative of job performance. There are lots of reasons for that, but especially for coders it makes a lot of intuitive sense. Look at the interview questions: In over thirty five years of professional software engineering experience, I never had to balance a binary search tree or create an iterator of iterators. If I would have had to, I’d probably copy and paste a working version from somewhere and adapt it, just like I did the two times in my life when I had to implement binary search by myself (because for some reason there was no implementation available in any library that was available in the particular environment I was operating in). And neither have I ever, or am I ever going to, develop my own key-value store. 

Overall job performance relies on many skills. Coding and design skills are obviously essential, but so are writing skills, verbal communication skills, time management skills, project management skills, clock reading skills, giving feedback, taking feedback, and setting priorities. Unfortunately, these skills are very hard, and maybe even impossible, to test for in a standard 45 minute interview. Even the behavioral interviews that were designed to dig out some of the “soft” skills now can be prepped for.

In short: The “old” school interview system does not work anymore. 

The solution that many companies have come up with is the take-home assignment. This can take the form of writing a larger body of code, including documentation and a presentation, which the candidate then will have to give in front of a panel of interviewers. Another example is the Amazon writing exercise which requires you to write a one-page document on a particular topic. In the product management space it is not uncommon to have to develop a product strategy as a take home exercise. When interviewing at Confluent I was asked to make a design for running Kafka as a Service on AWS (despite not knowing a lot about either Kafka or AWS at the time). 

These take-home assignments are not seldom significant time sinks. As a candidate you want to put your best foot forward and that often leads to spending lots of time doing the best possible job. That is all fine and dandy as long as you get the job, but stories abound of people being asked to do significant amounts of unpaid work and then not getting the job or the position being withdrawn without any explanation.

It is totally understandable that employers want to make the best possible hiring decision. Software engineers are highly paid workers and it often takes months to ramp them up. That is a significant investment, so you want to hiring related risk as much as possible. Furthermore, even though the difference in total cost of employment between an average and an excellent engineer might be a factor of two to three, the value of their output scales much faster. An excellent engineer is not just twice as productive as an average engineer, they are often five to ten times as productive. Of course the employer wants to make the best possible choice.

The answer to making better decisions is better interviewing, but that is really hard to do and would take a significant investment in the interviewing method and the skills of the interviewers. I still believe that five good interviews are enough to tease out whether the candidate is any good or not, but instead many companies seem to have decided that instead of doing better interviewing, it is easier and cheaper to make the candidates do more work and use that to get more signal on the candidate’s qualifications.

Please note that I am completely disregarding the fact that in many companies the hiring bar has been dropping as well. There are many reasons for that, but at the root of it is that in most companies not everyone needs to be a genius and a lot of managers would rather have a muppet than nobody.

Unfortunately, even with take-home assignments, most hiring decisions come down to a crapshoot. With the best possible interviewing you might be able to get a good handle on first-order job-related knowledge and skills, but the overall ability to get stuff done is so intangible that you will only find that out after a few months. As I wrote last week, whenever I dealt with underperformers, the problem was typically not that they didn’t know how to code.

For that reason, many traditional companies hire new-grads through trainee programs that often take months, including internal apprenticeships with different departments. It’s a good, but expensive, way to hire great people, but, among other problems, it doesn’t scale very well. I have seen this work in tech, but is usually only done for specific segments of the populations, such as military veterans returning to civilian life.

So, what to do? In the olden days we used to use something called the probationary period. This is a three to six month period in which the employer can “test drive” a new employee. At the end of the probationary period there is a formal decision point to continue with the employee or not. In countries with good labor protections (not the US), the probationary period often comes with a much lower barrier for both the employer and the employee to break the labor contract. It is not uncommon for the full benefits and protection of employment to kick in only after passing the probationary period.

The advantages and disadvantages of the probationary period are obvious. In the tech industry, the probationary period is all but gone. Even in countries where it still technically exists, it is a “wax nose” (Dutch saying meaning that it may look like a nose, but it is very soft and not functional). Reinstating the practice is probably impossible because given the (still fierce) battle for talent; there is really no motivation for good candidates to take the risk of probation if they can also get a permanent position with an employer that doesn’t do probationary periods.

All in all we are in a terrible state right now. There is at the same time an abundance and a shortage of talent in the market: An abundance of average candidates and a shortage of great ones. Unfortunately, in most companies the interviewing standards are just not high enough to distinguish between the two. However, given the seeming overall abundance of candidates, there is a tendency for interviewing to get out of control, with employers hoping that this will improve interviewing signal and thereby the quality of the hiring decision. All candidates suffer from this.

Maybe the only way out of this conundrum is to get much better at dealing with underperformers.