Great article. One thing to note about the 'interview performance is not indicative of job performance'. This could also partly be the same phenomenon as "height doesn't determine how well you'll do in the NBA"... of course it does, but if you've made it to the NBA, the effect vanishes (because a smaller player who still made it must be very good otherwise).
I'm actually intrigued by the idea of a company that requires a probationary period for all hires. If indeed this resulted in a high ratio of high performers, the opportunity to work in such a team would be appealing. The concern that it might be unattractive to potential new talent seems like it would apply disproportionately in a way that would actually serve as a helpful filter (candidates who at least feel confident that they'll be able to prove their worth on the job will be undeterred from applying).
I'm less confident that this would not result in a crazy number of lawsuits. :-(
Great article as usual. Just one data point here, but I've always refused "try before you buy" jobs. To me, hiring is always a leap of faith (for both sides), so if they won't jump, neither will I.
In addition, any company that requires take home assignments is biasing its candidate pool heavily. As a mother of 2 young children, who typically holds a full time job, I will simply not apply. I've seen my most talented friends react in the same way.
I find the push to remove interviewer variance absurd (I get why, but it's unattainable with humans and the very idea is preposterous), and the result - like you said - about as good as a coin flip. Your odds of false positives and false negatives are ~~ the same.
I have done a fair share of interviews, at Google and outside of Google, in both the recruiter and the hiring manager roles, hiring devs for my team. (And also from the other side of the table when looking for a job myself :-))
After seeing how different companies hire, and experiencing various styles, I still think the take-home exercise is the best final filter for candidates. If you do it correctly, it's a game changer. First thing is to prepare the task well, so you can see not just the code, but all the other skills as well - how they think the problem through, what questions do they ask during the task, how well they manage the time, how they communicate, how they present the solution, and how they explain their choices in the subsequent Q&A. I have seen so many candidates who looked promising after the initial interview and bombed horribly during the take-home task to be convinced otherwise.
What really worked well for me in terms of the take-home task:
- Pay for the candidate's time, make it a paid work. It shows them you are serious and value their time.
- Give them something real to do - small fixes, new or enhancement of an existing feature if you can give them access to your codebase.
- Add a small research task to it, or prepare the main coding task in a way they have to do a bit of a research into something they haven't worked with or seen before.
- Give them access to the future team mates, so they can ask questions during their work on the take-home task - invite them to a special channel in Slack etc.
- Split the task to mandatory and nice to have features and watch what they pick to implement. Ask them why they made the choice they made.
- Allocate certain time for it and give a firm deadline (we usually used tasks people could do in a few hours, or in about 20 hours if they worked on the nice to have stuff).
- Pay-per-hour, with a hard stop at certain budget. You can see how they think about work and their time, how much time they spent on it, and you can compare it with how much work they were able to do in that time frame.
- Design the task in a way that it might be tight to complete it in the allocated time, so they need to make compromises - you'll see how they prioritise and communicate about any potential time issues.
- Final presentation - ask them to prepare a final demo and research task presentation, but leave the format up to them, with one condition - put a time limit on the presentation. This tests their verbal and written communication skills well.
- Challenge their decisions during the follow up Q&A - you'll see how they can take feedback, react to questions, explain stuff.
- Ask them for the feedback on your codebase or the task itself - you'll get the feedback, often quite interesting from the best candidates, and you will see how they can give feedback about something they worked with
Preparing a good take-home task takes time and several iterations, but it's well worth the time spent on it.
I have hired a few engineers who had their code in production before they officially started in my team, because their work on the task was so good it did not need a lot of effort to clean up and release.
Plus the few days of working with the codebase and the team serves as a good onboarding, whey they started, they knew a few people, and were familiar with a bit of the codebase, so it was not a cold start.
Obviously, we only invited people to the take-home task as a final step for promising candidates, and our success rate was about one hire in two invited.
What does it have to do with the home-take task? We never had any issues with handing over a few hours of work with a 7 day deadline. If candidates had any issues with the deadline, they said so and we did accommodate it. I’m a father of 3, and have done take-home tasks while also in a fulltime job. It has never been an issue. If a candidate can’t find a few hours for a paid work, are they really interested?
It's not a few hours, it's a few hours on top of all the other hours, and most likely you're not the only place people are interviewing for.
My partner and I are both software engineers, with full time jobs and full lives. Last time I switched, I only interviewed for this one place and it was a heavy burden on them, and that was just the classic round of interviews. Essentially, some candidates would walk away from what you're suggesting the moment you outlined the process. They'll never get to the point that they're really interested.
Looking for a job is a fulltime job :-) Every time I switched jobs I sent out 30-50 applications, had to go through a lot of screening calls, interviews and it always came down to 3-5 final take-home tasks of a various scale from a few hours of work to about 20 hours. It only happened once that I had to do two take-home assignments in one week, but as I mentioned, you can always ask for time extension or if they have a set timeframe you can ask to start later. Yes, it is challenging for two fulltime parents, but not imposible. I have done it a few times (3 times in the last 8 years, we are both working fulltime and have 3 little kids). As I said, if a candidate does not want to put in some effort, it's also a signal for the hiring manager.
I think it's good to acknowledge that your proposed solution does have a bias, but to Jos' broader point, there'll always be inherent biases to any human-based processes. I think Tali makes good point, and it's better to conceded that your approach favors younger folks with fewer obligations than people who have busier lives.
Similar to Jos' point about probationary periods, good people might just choose to optimize their time for companies without this take-home requirement. From a candidate's perspective, take-home interviews don't necessarily make the company a good place to work for, and good candidates have plenty of options. At best, this raises the bar, but hiring only younger candidates and selects against the excellent 10x'ers.
Great article. One thing to note about the 'interview performance is not indicative of job performance'. This could also partly be the same phenomenon as "height doesn't determine how well you'll do in the NBA"... of course it does, but if you've made it to the NBA, the effect vanishes (because a smaller player who still made it must be very good otherwise).
I'm actually intrigued by the idea of a company that requires a probationary period for all hires. If indeed this resulted in a high ratio of high performers, the opportunity to work in such a team would be appealing. The concern that it might be unattractive to potential new talent seems like it would apply disproportionately in a way that would actually serve as a helpful filter (candidates who at least feel confident that they'll be able to prove their worth on the job will be undeterred from applying).
I'm less confident that this would not result in a crazy number of lawsuits. :-(
Hi Jos,
Great article as usual. Just one data point here, but I've always refused "try before you buy" jobs. To me, hiring is always a leap of faith (for both sides), so if they won't jump, neither will I.
In addition, any company that requires take home assignments is biasing its candidate pool heavily. As a mother of 2 young children, who typically holds a full time job, I will simply not apply. I've seen my most talented friends react in the same way.
I find the push to remove interviewer variance absurd (I get why, but it's unattainable with humans and the very idea is preposterous), and the result - like you said - about as good as a coin flip. Your odds of false positives and false negatives are ~~ the same.
I have done a fair share of interviews, at Google and outside of Google, in both the recruiter and the hiring manager roles, hiring devs for my team. (And also from the other side of the table when looking for a job myself :-))
After seeing how different companies hire, and experiencing various styles, I still think the take-home exercise is the best final filter for candidates. If you do it correctly, it's a game changer. First thing is to prepare the task well, so you can see not just the code, but all the other skills as well - how they think the problem through, what questions do they ask during the task, how well they manage the time, how they communicate, how they present the solution, and how they explain their choices in the subsequent Q&A. I have seen so many candidates who looked promising after the initial interview and bombed horribly during the take-home task to be convinced otherwise.
What really worked well for me in terms of the take-home task:
- Pay for the candidate's time, make it a paid work. It shows them you are serious and value their time.
- Give them something real to do - small fixes, new or enhancement of an existing feature if you can give them access to your codebase.
- Add a small research task to it, or prepare the main coding task in a way they have to do a bit of a research into something they haven't worked with or seen before.
- Give them access to the future team mates, so they can ask questions during their work on the take-home task - invite them to a special channel in Slack etc.
- Split the task to mandatory and nice to have features and watch what they pick to implement. Ask them why they made the choice they made.
- Allocate certain time for it and give a firm deadline (we usually used tasks people could do in a few hours, or in about 20 hours if they worked on the nice to have stuff).
- Pay-per-hour, with a hard stop at certain budget. You can see how they think about work and their time, how much time they spent on it, and you can compare it with how much work they were able to do in that time frame.
- Design the task in a way that it might be tight to complete it in the allocated time, so they need to make compromises - you'll see how they prioritise and communicate about any potential time issues.
- Final presentation - ask them to prepare a final demo and research task presentation, but leave the format up to them, with one condition - put a time limit on the presentation. This tests their verbal and written communication skills well.
- Challenge their decisions during the follow up Q&A - you'll see how they can take feedback, react to questions, explain stuff.
- Ask them for the feedback on your codebase or the task itself - you'll get the feedback, often quite interesting from the best candidates, and you will see how they can give feedback about something they worked with
Preparing a good take-home task takes time and several iterations, but it's well worth the time spent on it.
I have hired a few engineers who had their code in production before they officially started in my team, because their work on the task was so good it did not need a lot of effort to clean up and release.
Plus the few days of working with the codebase and the team serves as a good onboarding, whey they started, they knew a few people, and were familiar with a bit of the codebase, so it was not a cold start.
Obviously, we only invited people to the take-home task as a final step for promising candidates, and our success rate was about one hire in two invited.
How many of them were parents? Women? Currently employed?
In addition, much like the argument about the quality of interviewer, this relies on the quality of the evaluator(s) in much the same way.
Pardon, but I remain unconvinced.
What does it have to do with the home-take task? We never had any issues with handing over a few hours of work with a 7 day deadline. If candidates had any issues with the deadline, they said so and we did accommodate it. I’m a father of 3, and have done take-home tasks while also in a fulltime job. It has never been an issue. If a candidate can’t find a few hours for a paid work, are they really interested?
Ugh, I typed a response and it got deleted 🫣
It's not a few hours, it's a few hours on top of all the other hours, and most likely you're not the only place people are interviewing for.
My partner and I are both software engineers, with full time jobs and full lives. Last time I switched, I only interviewed for this one place and it was a heavy burden on them, and that was just the classic round of interviews. Essentially, some candidates would walk away from what you're suggesting the moment you outlined the process. They'll never get to the point that they're really interested.
Then again, with the market as it is right now...
Looking for a job is a fulltime job :-) Every time I switched jobs I sent out 30-50 applications, had to go through a lot of screening calls, interviews and it always came down to 3-5 final take-home tasks of a various scale from a few hours of work to about 20 hours. It only happened once that I had to do two take-home assignments in one week, but as I mentioned, you can always ask for time extension or if they have a set timeframe you can ask to start later. Yes, it is challenging for two fulltime parents, but not imposible. I have done it a few times (3 times in the last 8 years, we are both working fulltime and have 3 little kids). As I said, if a candidate does not want to put in some effort, it's also a signal for the hiring manager.
I think it's good to acknowledge that your proposed solution does have a bias, but to Jos' broader point, there'll always be inherent biases to any human-based processes. I think Tali makes good point, and it's better to conceded that your approach favors younger folks with fewer obligations than people who have busier lives.
Similar to Jos' point about probationary periods, good people might just choose to optimize their time for companies without this take-home requirement. From a candidate's perspective, take-home interviews don't necessarily make the company a good place to work for, and good candidates have plenty of options. At best, this raises the bar, but hiring only younger candidates and selects against the excellent 10x'ers.