The only way to provide this evidence would be to to conduct an RCT in which you hire some percentage of people who fail the test as a control group. The nascent psychologist in me loves this idea, but it seems unethical, and presumably this is why nobody does it.
But I disagree that we need ”evidence.” It’s flat out obvious. If you give your job candidates a discrete math problem orders more difficult than the usual ticket, you minimize the risk of hiring someone who can’t do the job.
You're also maximizing the odds they will get bored and feel baited. With all due respect, you could have picked a better example to illustrate your point.
This is exactly why you can't make such bold claims. You're looking at this from one perspective without considering all the variables which go into the workplace equation. So far, the best correlator to job performance is giving people work corresponding to what they will be doing at the work place. For various reasons, this isn't really happening in software development, making people all to eager to grasp at proxies which they swear will do the same thing. Maybe whiteboarding is in line with the day-to-day at FAANGs, but tons of companies ask questions which are not in line with the day-to-day at all. Both vertically and horizontally. It creates dissatisfaction, burnout, bad cultures, sometimes even animosity, as if hiring managers are playing pranks on candidates.
If you're only looking at this from the perspective of "get someone who can do the job", you're going to miss the forest for the trees. As others say, the company is also being interviewed.
I like this point. This is an especially big problem in my area (data science), in which people think they are being hired to do deep representation learning, and are subsequently mortified to learn that the job is principally about transpiling ”product manager” to ”SQL”. The cure is to be up front with job candidates. Tell them the job involves no deep learning. Explain that you give hard interview problems because you need to know that they can reason about eg asymptotic complexity on the rare times where it arises.
> So far, the best correlator to job performance is giving people work corresponding to what they will be doing at the work place.
I don’t necessarily disagree, but I believe you want to optimize for the hardest day rather than the average day. If your team is optimized for who can best solve an average problem, you will eventually have a problem that nobody can solve, and the business will suffer greatly as a result. Hence the discrete math problems.
I think this is a huge mistake. I work in data science, and nearly universally my colleagues are motivated by opportunities to learn. If you're testing for the very rare day, so that your employee has the right answer in their front pocket when that happens they are never going to feel challenged, feel like they're growing, and they're going to jump ship.
I think you're doing snobby filtering, and it's not to your benefit.
On the hard days, the answer has never been seen before, and therefore can’t be in anyone’s pocket. You need to know that you have employees who are smart enough to discover it. Technical interviews are how you find them.
The observation that your employees are motivated by learning opportunities is useful for how you manage them, but this doesn’t tell us how to hire people who can do the job.
There are a few problems with this. If your "hardest day" deviates far from your "average day", while being very infrequent, you're still not doing a good job enticing the candidate. Worse, by looking for people who have already acquired the skills to combat the worst you can throw at them, you're severely stagnating their growth.
Then comes the cherry on top, if your company is doing it while not providing absolutely stellar benefits to function as external motivation, you're further incentivizing people to work on ditching your place. Many companies do not give total comps in line with the skill level tested for, setting themselves up for failure in this regard. Altogether, you're still looking at this primarily from the job interviewer's perspective, not the perspective of the candidate which translates into the company's long term (lack of) health.
>you will eventually have a problem that nobody can solve
No. Someone has to be the first person to solve a question. This strategy doesn't take into account the capacity of individuals to learn and overcome challenges. What's more, most individuals will not be on the highest level, allowing mentors to train them to higher levels.
>I don’t necessarily disagree
It's not about disagreement. Schmidt and Hunter did a meta-analysis on this way back in 1998, and psychology has yet to find the silver bullet. Look at table 1[1]. Most studies checking the market find similar things. It is ridiculously difficult to find correlations between job performance and selection criteria. All corporates, including FAANG, are not exempt from making the very human mistake of doing something which seems great, but ends up being part of survivorship bias. In something as multi-faceted as the job market, combined with the young and everchanging nature of software development, making statements on correlations shouldn't be taken for granted.
Notice how the study above points out years of experience as one of the worst predictors. Most of the job market still uses YoE as if it is gospel. The market is nowhere near as rational as people like to believe it is.
[1]: https://www.researchgate.net/publication/232564809_The_Valid...
You might enjoy Work Rules by Laszlo Bock. They actually did do this at Google to validate their interview approach.
In all seriousness, thank you for bringing this up - it’s an interesting story and I will probably look into it. But I also think it’s pretty obvious that coding interviews work extremely well, and any discussion of ”how can we create a procedure that is better for everyone” that doesn’t acknowledge this is hopeless.
