Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?
Completely unrelated: In 1982, when I knew approximately nothing and was writing code in BASIC and assembler, my math teacher introduced me to structured programming and a Danish programming language called Comal. In 1990, when I still knew approximately nothing, I had some free time on my hands and I started developing a Comal interpreter in C. Since then I have been tinkering with it on and off. It is nothing to be proud of, because the architecture and implementation (still) represent my 1988 understanding of how interpreters work, but fwiw, here it is: https://github.com/josvisser66/opencomal.
For the fact that ChatGPT has made writing good interview feedback easier than ever, I am still seeing more than my fair share of terrible feedback, so I thought I’d spend a few words on this lovely Wednesday morning to explain why great interview feedback still matters and how to write it.
As we all know, there is a lot of turmoil in the market, with layoffs and everything, but hiring is still a thing. Companies might not be hiring literally everyone they can find anymore, but new companies are still being founded, companies are still growing, and companies are busy retooling (meaning: Swapping mediocre expensive employees that they hired during the lean times with better and less expensive employees that are available right now). With an abundance of candidates on the market, the chances of hiring a good candidate are better than ever before. Unfortunately, your actual chances of hiring good candidates are only as good as your interview process. Also unfortunately, from first-hand and anecdotal second-hand experience, I have sadly to conclude that there is still room for considerable improvement, something I wrote about for the website interviewing.io.
As an indicator, the kind people over at interviewing.io told me that they tried to break into the market for interviewer training, but they had to let that idea slide because there was not enough interest.
Tech company hiring processes are broadly the same across the industry: The candidate is exposed to 5-7 interviews, dedicated to topics like coding, system design, a technical deep dive on a previous project, a behavioral interview, and perhaps a culture fit interview, where a manager or tenured employee tries to establish whether a candidate fits into what the company thinks their culture is. Each of these interviewers write feedback on the interview. This feedback then goes to a committee that makes the hire/no-hire decision. Sometimes these committees are made up of senior engineers and managers, sometimes these committees are made up of the interviewers plus the hiring manager, and sometimes a “bar raiser” is part of the committee (a practice I highly recommend).
No matter how the committee is established, the common factor is that all members of the committee have only the feedback to know what happened in all or most of the interviews. If the interviewers themselves happen to be in the committee, they only have first-hand experience of what happened during their own interview and so they are just as dependent on feedback as the non-interviewers in the committee.
All companies I know of require the interviewer to assign a score to the candidate’s interview performance. This score represents the interviewer’s overall decision with respect to how this candidate’s performance relates to the hiring bar.
Dad joke: Where do you take a recruiter on a first date? Answer: The hiring bar! Ask me how I know :-)
The scores are typically in the range 1-4, with some companies allowing decimal numbers, leading interviewers to fret about whether this candidate deserves a 2.2 or a 2.3. Whole numbers 1-4 typically align to “strong no”, “no”, “yes”, and “strong yes”. The hiring committee gets both the written feedback and the scores.
The relevant insight for hiring committee operation is this one: Hiring committees hire on feedback, not on scores! This might seem like a bold statement (and to be sure, it is typeset in bold), but it is easy to understand: If the hiring committee would hire on scores alone, we could replace the entire committee by a very small shell script. Instead, the hiring committee should read the feedback to form their own opinion on how the candidate performed during the interview and then come to a decision. This process also compensates for any misconceptions that the interviewer might have about what an acceptable answer is. These misconceptions are all too frequent, because most interviewers want to be nice, mistaking being a nice interviewer during the interview with being a strict (but fair) evaluator of the candidate’s performance after the interview is over.
Straying from the narrative for a bit: It is important to be nice. Once, long ago, I was chastised by a recruiter because a candidate had told them in the post-interview debrief that they had been afraid of me. Surprised, I turned to my friend and colleague Toni, who I had interviewed: “Toni, I interviewed you, am I not the nicest interviewer you have ever met?” Toni, with his typical candor (he is Finnish), replied: “You were the only interviewer I was afraid of.”
Candidate experience is important because if a candidate is good, you want them to join the company and that is much easier to achieve if they liked the interviewers. If they are not that great, you still want them to say (or at least think) that the interviewers were nice and fair, because there is literally nothing to be gained by them thinking that all the interviewers were assholes.
During the COVID pandemic, the lovely Mrs. Wednesday Wisdom was in a position to overhear my interviews when we all moved everything to Zoom. “Oh, that went well”, she once remarked after an interview had concluded. “No it didn’t”, I replied, “this candidate was terrible!” “But you were so nice to them”, she said, somewhat surprised.
Mission accomplished!
The lesson is this: Be nice to the candidate during the interview, but be strict in evaluating the candidate’s performance because, obviously, you want to hire only the best candidates, especially in a market where there are enough of them available. Remember this: A’s hire A’s, B’s hire C’s…
Back to feedback: In an earlier Wednesday Wisdom article I have argued that being able to write well is an important career skill. One of the places where you get to use (or at least, got to use) that skill is when writing interview feedback, because without well-written feedback, how are the committee members going to figure out what actually happened during the interview and whether the interviewers did a good (enough) job and were correct in their assessment?
Unfortunately, I still see a lot of bad feedback that fails to describe exactly what happened during the interview, what the interviewer thought, and why the interviewer thought that. Here is a (fictional) example (that nonetheless is very close to some of the worst feedback I see):
The candidate did a good job. They solved part 1 of the question without problems and had made good headway on part 2 when we ran out of time, though I am certain that they would have completed it if they had more time. They asked good questions and responded well to my hints. Score: 3.0; hire!
Mind you, this is not the conclusion, this is often the entire feedback!
Imagine that you are on the hiring committee for this candidate. What can you conclude from this feedback? Answer: Nothing. Either you believe the interviewer on their word that this candidate is worthy of hire or you don’t, but you have no evidence either way, other than perhaps knowing the interviewer personally and having a subjective opinion about them as an evaluator.
Good interview feedback reads like a short story of what happened during the interview, including meaningful things the candidate said or produced (code, diagrams), what the interviewer thought of that, and why they thought that, preferably with references to established quality guidelines, hopefully coming from the interview question scoring rubric.
If your interview is a coding interview, you should include code the candidate wrote in your feedback to back up your assessment that something that the candidate did was good or bad. If your interview is a system design interview, you want to outline the design decisions that the candidate made and whether they are good or bad and why. If you gave the candidate a hint, you want to describe what the hint was and how the candidate responded. If you asked the candidate a meaningful question, you want to include what that question was and how the candidate responded.
Obviously, the feedback does not have to be a play-by-play, as it is intended to be a summary; but it does need to have all the required evidence to back up your core claims. It does not have to be verbatim, but it does have to be persuasive. And if the hiring committee has any questions, you might want to make sure that you have all the information available in the form of notes and maybe a link to the coderpad that can be looked at if there ever is a need to dive into the details of what happened.
Which brings us to the topic of interview notes.
To be able to write great interview feedback you need to keep great notes during the interview. I am a big fan of recording interviews, but regardless of whether that is possible or not, I always keep sumptuous notes that includes the things the candidate said, the things I said, my running commentary of how things are going, code snippets the candidate produced, timestamps (approximately every five minutes), and all the other goings on that are important enough to influence my evaluation (like the connection being broken or the candidate having to look something up on Google).
This comes down to a lot of typing during the interview and I thank my parents for having sent me to a Scheidegger typing course back in secondary school. One thing about this is that it might distract the candidate during the interview. To allay this somewhat, I always tell candidates that they will hear a lot of typing in the background, but that this does not mean that I have lost interest and am Facebooking my wife, but that I keep notes to help me write good feedback.
After the interview, I go over the notes and add any additional insights or comments that I only thought of after the interview or that I didn’t have time to write down during the interview.
With that out of the way, the time has come to write the feedback. This used to take me about one-and-a-half hours because good writing takes time. This time was then earned back with interest during recruiter review and hiring committee meetings because processing well written feedback is much faster than processing bad feedback.
Fortunately, these days, we have ChatGPT, so instead of spending an hour or more crafting a well written story, I throw the interview notes (including all code the candidate wrote), the candidate’s resume, the job description, and the question rubric into ChatGPT Pro and tell it to write feedback that I can paste into our Applicant Tracking System (ATS). Five minutes later I have pretty good feedback, including ChatGPT’s score suggestion. I then spend another five minutes going over the feedback, suggesting improvements and maybe telling ChatGPT to modify the score (rarely). Another five minutes later, I have something I can copy and paste into the ATS. All of this together takes fifteen minutes max.
With ChatGPT, there is really no excuse anymore for bad interview feedback because the model is more than capable when it comes to writing good feedback provided that you kept great notes during the interview.
Back to the interview notes for a second: Whenever you write, you always write for an audience and when you are going to create interview feedback using ChatGPT, the model is the audience for your notes. That means that you need to write and structure your notes so that the model can make sense of it, for instance by clearly separating the things you said and the things the candidate said, or separating things that were said and things that you thought. Simple markers like “TC:” and “Me:” help tremendously with that.
Good writers (like myself) will tell you that they can write better than ChatGPT, which is undoubtedly true for them given enough time. But ChatGPT can write very good feedback in a matter of minutes and, obviously, quality is fitness for use. Plus that not everyone is a good writer. As a friend of mine (an Oxford grad) puts it: “I love ChatGPT, it takes my colleagues’ bad English and converts it to good English.”
Great interview feedback remains as important as ever and with AI there are no more excuses for writing bad interview feedback.
