In my last job search (~3 years ago) I was presented with many requests to complete a design challenge. I rejected them outright and the responses I got (typically from a 3rd party "design recruiter") were quite astounding. Some acknowledged and moved on, but there were several who clearly expressed frustration, disdain, sometimes almost anger. One dropped me from another role I was working with them on for refusing.
Now it's a total red flag for me. But judging from Blind posts it's still a common practice.
Interestingly, I always took restrictions on time seriously. When I was told I should spend max 3 hours, I stopped latest at 3 hours, often earlier and discussed shortcommings of the solution based on spent time in the interview.
That strategy is easier when you are not that interested in that particular job though.
I am in Product now, but when I was in SRE, I got one like this. "We don't want you to spend more than 3-4 hours on this", where "this" was:
Build a log parser for streaming Apache CLF. The parser should:
Keep a rolling monitor of the top 10 requests, displaying their velocity in req/sec as a rolling average.
Display aggregates of visitor counts over the last hour and day.
Have high watermark alerts when ingress traffic as a whole, or to hotspot URLs hit thresholds, and then be able to de-alert when traffic dropped.
Scaffolding to deploy same.
Unit tests and documentation for same.
Ability to ensure URLs were safe, stripped of any GET parameters.
Not overly complicated, but you're not busting that out in 3 hours in any polished manner, if at all.
Allo
I just didn't proceed with the interview after that, if the staff can't manage to plan how long a test takes, there's no chance that they can plan anything accurately in the company either.
Counterpoint: those fresh out of school are likely to do the best in some "invert a binary tree" leetcode style interview, with that fresh in their memory. Those who are older (and thus more likely to have families) who actually have experience at the craft are most likely to be able to bang out an actual coding task quickly.
It's been a long time since I've been on the job market - but even a decade ago, I had third-party recruiters who would forward me a take home test... and another successful candidate's solution, "for reference".
And even if candidate doesn't get external help, I'm extremely sceptical about time limits. Nothing stops someone spending 6 hours on a "2 hour" take home test, producing a better solution than someone who followed the time limit strictly.
It's pretty hard to extract a signal under such circumstances. Unless you're hiring for a job where a willingness to bend the rules to get results is desirable - used car sales, for example.
Unless you can ... demonstrate you only spent two hours. It's pretty rare you can do that, but... several years back I got a callback on an interview, and they forwarded me the 'coding test' portion. I got it around 1:30p, and I emailed them back the finished product around 3:45p, so.. a bit over two hours, including download and setup of their test environment.
I advanced further, but they ultimately chose someone else. I think I may have been a bit too senior/advanced for them, or.. maybe I was just a jerk in the interview, or... someone else was just a better fit? I'll never know.
I do know it was one of handful of interviews in the last 10+ years that was actually done well. On the technical side, they had a test git repo that actually worked - fork the repo, pull down, run the docker compose and everything just worked. I spoke to the woman who'd set it up and it was obvious she'd put a lot of time in to making it all as smooth as possible, and it was. I wanted to work there just because she'd put that much effort just in to 'a test'.
I will say that this is something you definitely should not be, especially if it's a larger corp. You should be political, passionate, approachable and friendly. It doesn't matter if you are more technically skilled. It might be even worse for interviewers if you are technically skilled and arrogant about it, because it would make their work lives worse. Even though I appreciate technical skill and passion, I wouldn't hire anyone who was actively arrogant or similar about it. I look to create a team that in theory would be supportive or empathetic towards low performers, although of course preferably we would find a solution to low performers or understand why they are performing low if possible, because ultimately I do want high performing team.
But most things we interact with are done half-assedly, or are MVPs, or are deliberately made to a stupid spec, or don't prioritize the user, etc... there are a million ways to mess it up - usually by not even trying - and the result is draining.
I did a stint of hiring for a software company over a decade ago where they did use a take home test. Now we did get the odd person cheat on it, but I can think of only a handful of occurrences where that had happened - at least in a way where it had meaningful impact - out of the probably hundreds of people we interviewed.
The take home test was simply there to figure out whether or not it was worth inviting an engineer in for interview, at which point we of course did an in person coding assessment. If you cheat at the take home test, what happens at the interview? You get found out is what happens, and it can lead to quite an awkward conversation. If you don't get found out perhaps it doesn't matter anyway because you've just got through a coding interview anyway. That's not to excuse cheating but to point out that it's perhaps only a risk in terms of technical skill if it's your sole point of assessment of those skills.
But, anyway, as I say: I don't much like take home tests for other reasons. And I don't much like platforms like Codility either (with these I found the signal to noise ratio to be particularly poor). The reason I don't like any of this stuff though is that, if you're hiring, most likely it disadvantages you as an employer versus other companies who don't put candidates through these tests.
Unless you work for a company where everyone wants to work you have to ask yourself this question: why would anyone willingly go through our burdensome and tedious selection process when they can get a job on a similar, or maybe even better, salary elsewhere much more easily?
It's hard to extract a signal about prospective work quality from anything but a trusted referal or a probationary hire. It's not like there's a great alternative just being ignored.
Some orgs feel more comfortable assuming that whiteboard performance represents general aptitude rather than drill practice, some are more comfortable assuming that small assignments represent work simulation rather than plagiarism or misrepresentation. Both approaches are swamped by noise, but when you're sieving innumerable applicants and are accountable to an image of fairness and standardization, you have to use something.
So far, it's been extremely successful for me in predicting whether a candidates going to get weeded out by the rest of the interviewing process or not.
I once got rejected from a YC company because I responded more than three hours after a two hour take home test was given.
So you certainly could stop them. Doesn't really leave the best impression of the company though.
And if you're curious I wasn't actually working on it for three hours, I just started late :)
I hate it when I have to immediately open it and finish it, like a timed coding test, although I don't bother with those because I don't think I've ever passed one...
Realistically, people are going to spend more time on their take-home assignments, because they perceive there's an advantage to doing so. It's not hard to imagine how this spirals out of control over time, a feedback loop of candidates trying to do more and more free, meaningless work as the bar is raised. A three-hour assignment that takes an entire weekend, a three-hour assignment that takes a week, etc. But they shouldn't be openly encouraging this death spiral, they should be trying to keep it under control.
Better to say "if it looks like you spent 20 hours on a 3 hour assignment, we'll think of you as someone who isn't good at estimating, and wastes a lot of time fiddling with things that are out of scope".
It's almost the opposite of what the job actually entails. I rarely estimate that a project will take 3 hours and then I have 48 uninterrupted hours in which to accomplish it, no biggie if it takes more than 3. The job is typically the opposite, where I estimate it will take 3 hours and then I realized that I need to accomplish 40 hours of work in the 20 meeting-free work hours I have this week so I need to find a way to finish it in 90 minutes.
I noticed the same thing. It seems take homes are mainly geared towards pre-selecting young single people with a lot of free time to code on the side besides their main job and other responsibilities, or who have no other responsibilities.
Now I'm also single and I have the time to invest in them, but I question the future of my career in tech, how will I be able to compete for new jobs later when I won't be, and take homes will be normalized? I don't remember my friends in other careers ever having to do unpaid work before getting a job. Perhaps I chose the wrong career.
What sucks even more is when you sink many hours in them and then just get ghosted, not even a rejection message, nada. I've already been burned twice by this. Or when the recruiter just spams you the take home link without first having a talk with you about the job details, the compensation range, if you're a good fit, etc. It's "first you solve this take home for us, then we can have a talk". Feels incredibly cold and inhumane.
My counterpoint to this is that takehome tests are great for folks who don't necessarily do well on brainteasers, but are otherwise strong functional developers. Despite having a family and children, I always prefer a takehome assessment. (And, just as an FYI, I'm 36 and a single parent to a young grade-schooler)
I might be biased, in that I feel like I do poorly in the live programming / brainteaser style interviews, but do strongly when I have a takehome assessment. Live programming is like a completely bizzaro-land version of programming, where a real person is staring you down while you type, observing your every interaction, and deciding your fate based on a random 60 minutes of the highest-stress part of your day. ("Oh, you googled a for loop syntax, you are clearly an idiot lying about your experience" when in fact, it's more like, "I get thrown 6 different programming languages every single week, each with similar but slightly varied syntax for every single thing, and having a person stare at me and decide the entire future of my career is a little bit stressful and anxiety-inducing").
With a takehome assessment, I can think about it for a while, I can write it all upfront, I can try a few different implementations and pick the one that feels best, I can accurately and intelligently explain exactly what I did and why I did it, I can talk in detail about the problem and potential tradeoffs. I can wrap the whole thing in nice automated testing, I can setup basic CI/CD for it on GitHub Actions or equivalent. I can demo my commitment to strong well-written clear documentation and dev experience, right in my branch Pull Request -- and all of that will likely be a closer match to what real-world day-to-day work at the new company might be like, than solving your LeetCode brainteaser.
I don't love "unpaid work before getting a job", yes that part sucks. But it sucks less than having a random stranger decide your an idiot, based on watching you sweat for an hour. Even if I'm going to get ultimately rejected anyway, the takehome route still feels better.
I’m often worried that I’ll look like a dummy when I have to look up basic loop syntax. I’m not dumb, and I’m somewhat experienced… I just write code in Python, Matlab, Fortran, Bash, whatever else. It is shocking that the basic shape of syntax can be so different!
And also the degree to which it doesn’t actually matter. It’s just when starting a new file. My brain needs a little bit of syntax around to get locked into the language I think.
For me personally I prefer it to live coding as I can actually think because there's no stress but usually the "1-2h task" in reality is 8h if you want it done properly (good code, documentation, tests, project & build setup, etc.). If the job isn't top tier it is simply not worth it, at least in the previous market.
Still beats memorizing the optimal algorithms to LC questions.
Which tells you exactly how exploitive they will be with that free time. You'll onboard, it'll be good for a bit, then suddenly there will be surprise crunch-mode that will never end.
> when you sink many hours in them and then just get ghosted, not even a rejection message
This will manifest in a thank you of a layoff after all those crunch hours.
> first you solve this take home for us, then we can have a talk
"Cool, I can take on that contract. I'll put together a quote for you and I'll get started. Who should I send my W-9 to?" Many will, of course, balk at this, which tells you everything you need to know about them. The good ones will pay it. Set your boundaries.
If your idea of hiring is to spam me after I apply in Workday with a multi hour coding project to see if I merit the time speaking to even a recruiter, no, that job app is going in my circular file.
Sadly how people recruit is completely broken for most companies that are not FAANG etc. (It is broken for FAANG as well but in a different way).
As I often say "you need to win, not win an argument". Similarly you want candidates that can be good productive members of your team at the salary you are paying and not the candidates who can nail your interview process.
The problem with take home tests is not the cheating. But rather failure of the recruiter to actually evaluate the assignment. Do you care about the end result or do you care about the art of craft ? Do you care about readability of the code or performance of the app ? Is the assignment similar to the work candidate will end up doing at workplace ?
My suggestion generally has been:
1. Look at candidate's past work. Has candidate worked with reputed companies ? 2. Talk with candidate with general technology topics around their work. Something like "what do you like about react?" 3. Give candidate a very simple codesignal test. Nothing to fancy. Say if you want react engineers just test their ability to implement simple components.
This vastly kills a lot of inferior talent in my opinion.
4. Give a very simple problem without any "trick" or "iq test" and ask the candidate to code it in front of you. Also let the candidate use internet, documentation and AI tools.
In this day and age asking candidates NOt to use AI tools like asking a candidate to write code without using keyboard. You want people who can use things like Github copilot.
One of them was causing such friction with endless meetings and mail chains, that a project she coordinated that was a quagmire for six months was finished in the two weeks after she left.
I can spend time answering your take-home, or trying to get other interviews. From what I've seen on take homes: Other interviews are strongly EV+.
Before you go: But you never got a job off a take home, that's why. I did. It was one of the worst managed misfits I've had. I am sure others thrived there, for me it was a living hell, that didn't even pay that great.
So... Much like requiring a suit to interview and a few other things. It goes in my bin of "Your company has performed self selection."
Note: I will leetcode, at least there is a human there, we're talking, interacting, and I am learning about the company.. But most of all, that feel of: I put up and hour, you put up an hour. Let's talk. Is very real.
Which... doesn't necessarily correlate to someone being good at programming or web development or what not. Or whether they'll be good at the job in general, given that most engineering roles don't have someone screaming "get this done in under an hour or else!" with no time to think things through properly.
At the same time though, I can also see why they do that, since otherwise it comes down to "whoever's got the most free time and willingness to grind away at the problem", turning an hour long exercise into a 3 day one in the process.
We're biased, but we think the old form of take-home assessments (+ classic Leetcode tests, etc) are completely broken. Beyond reasons you all mention - they're completely unreliable today in the age of ChatGPT, etc. Way too easy to cheat.
We're seeing candidates copy takehome instructions into an LLM, paste the solution and submit in <5 minutes. It's hard to write a problem that's (1) low enough scope to solve in a short time, but (2) hard enough that LLMs can't step towards sovling it.
At Ropes - we're using LLMs to evaluate how candidates code, not just the final solution. So HM's can step in and look at a timeline of actions taken to reach the solution, instead of just the final answer. How do candidates debug? What edge cases do they consider? Etc. We think these answers hold real signal and can be answered for the first time async.
We're trying to make this better for candidates too. E.g. (1) shorter assessments, (2) you can often use your own IDE, (3) you're not purely evaluated on test cases, etc. But we're not yet perfect. If this sounds interesting / you have strong thoughts I'd love to talk to you - email is in my bio.
> Rather, they were the design studios, the lean web development agencies, the mobile dev teams based out of Eastern Europe.
Given a team of... let's make up some numbers.... {8} devs working on client projects that have {250} candidates to evaluate, how much time do the {senior} members of the team have to evaluate them? Note that taking {two} senior devs in such a shop out for {30 minutes} for each candidate for a live coding interview is {=about a week and a half} of nothing but doing interviews for each of them - removing them from other revenue generating tasks and a reduction of {at least 1/4th} of the work for that duration.
Extending this beyond 2 weeks of time means that the best candidates are likely to have found other opportunities.
Take home tasks shifts that time to a standardized "I can look at this sample code in 2-3 minutes and get a feel of if the candidate will be producing something maintainable or hot garbage."
At that point, one can narrow down the pool of candidates down to a small number that can be interviewed and decided with some expedience.
The answer there is often "select only the ones that show the most things on the resume without evaluation of their practical skills" ... and we get to the complaints then of "but people lie on their resumes all the time" and "but my skills at coding aren't things that I can reflect on the resume?"
And when you're spending 3-4 minutes per resume, you don't have time to open up a GitHub link and try to figure out if the person who this is wrote the code or if they just forked another repo ... and what contributions are theirs.
For evaluating a coder, the time imbalance is heavy on one side or the other.
Having two people do seven hours of interviews a day for two weeks is also in the "not feasible" category. As its the company that is setting up the criteria for the interview, they've got the first move and are setting up a process that they can fairly evaluate the candidates that apply and go forward with that step in a way that is most likely to eliminate the most risky candidates and find the candidates that have the skills but don't present well on a resume.
- Whiteboard coding tasks put a LOT of pressure on you to perform while being watched in a stressful high-stakes situation, which is a situation that the extreme majority of SWEs won't deal with, and many simply can't. They also tend to test for algorithms (inverting a binary tree, etc.), something that is unrelated to most actual jobs.
- Takehome assessments are unfair to those with a family that might not have the free time to do them, and if you're applying for multiple jobs, can easily become a full-time job on their own.
- Performing a 5+ round interview gauntlet is frustrating and absurd, and can make the hiring process take well over a month.
> If the very first thing in your application process is a 6 hour long unpaid assignment, then it’s hard to imagine you’re not wasting someone’s time because at this stage you’ve barely done any filtering.
I pretty much walk away at this point. It's like asking a person for sex after buying them a drink. (And if I have to explain why, you clearly don't get it.)
> But if the takehome is offered, say, after 3 rounds of interviews and will be discussed in detail during your final round with a panel of two senior engineers, I think that’s fair and provides a strong signal.
Exactly. That's the point that I'm willing to invest in a takehome.
And how are these big tech firms spending $billions on recruiting and interview time, as if the education credential is worthless, but still preferring to use it as a first-line filter?
But I don't think this type of evaluation is a valuable signal anymore in the age of AI. Anyone could nowadays finish your take home assignment in a fraction of the effort it would take them without AI. This might not be relevant after they're hired, since they would likely also use AI assistants, but for evaluation purposes you want to assess how someone thinks about problem solving, not how they generate the solution, and then confidently walk you through it.
The best interview style for software developers IMO is the code review challenge. Show them a piece of code from a hypothetical PR of a junior developer, and ask them to a) describe what the code does, b) point out any issues they see, and c) fix the issues. You can have several difficulty levels depending on the role and expected candidate experience, or you can cherry pick these from your own codebase. The benefits of this approach are that it's a simulation of a real-world task they would be doing on a daily basis, it's a collaborative effort and showcases their communication skills, while also allowing them to evaluate yours, it's much less stressful than white boarding and quizzing sessions though still involves some coding, and unlike take home assignments, it can be completed in an hour or 90 minutes at most.
(For me, at least. I don't parallel-task very well in the first place, and being under the magnifying glass makes it significantly worse. I have trouble, say, even making sense of the words someone else says when I'm already focused on coding/reading/writing something. If the interviewer interjects a thought or question about what I'm doing, I often have to stop, struggle to ~park my train of thought, and then ask them to repeat what they said. If I make the mistake of asking them to repeat before the train's parked, I may have to ask them to repeat it again.)
I think I'd also look pretty favorably on an interviewing process that gives a few format choices and lets me choose what I'm most comfortable with.
Also it completely eliminates the tool selection and test system portions of the exercise because I pick all of that stuff and make sure anything I don't care about is automated or is out of scope of the solution.
Done correctly you can usually get through it in less than an hour, leaving about 10 or 15 minutes for the candidate to ask me some questions.
I am totally surprised that "reading code" is not really appreciated even at FAANG companies though very likely that is one of the most important things they will end up doing.
the project was to make a django app for storing your pokemon application, and a script to download the list of pokemon from an api. So atleast it was fun, but i hated turning people down who took the time to do it.
Also, enough with the fucking Roman numerals, Jesus Christ.
I like respecting candidates' time, because I want the strongest candidates to not filter themselves out at the mention of a take-home. That's why I ask them to schedule 4 hours with me where at the start they will get the problem description and at the end they will submit what they have - this way they aren't pressured to put in more time because they know for sure that their competitors didn't - and I get to compare apples to apples.
I also designed the rules to minimize the perverse incentives set by the clock by giving multiple ways to get recognition for partial solutions.
This is the content of the document the candidate gets initially (they only get the actual problem later):
Meta
This is a real task we encountered and solved at Finubit. Out of respect for your time, we limit this task to 4 hours (of wallclock time, to remove the incentive to spend more than that to gain a competitive advantage). It took us much more than four hours to get the perfect solution, so don't worry, we don’t expect a full production-ready solution to 100% of the problem.
We ask that you approach this task as you would the first 4 hours out of as much time as you need to finish the task to a reasonable quality standard. We do want to see a working Proof Of Concept at the end of those 4 hours, like we would ask of any team member picking up a long, open-ended research task. The POC can be very limited in functionality, but functionality that it promises will be held to a standard of quality.
If you encounter ambiguities or mistakes, assume the most reasonable way to resolve them and make a note to check with Product that your understanding was correct. If you disagree with Product about something, make a note to push back against the design decision they made (but in the meantime assume the current spec, unless it's grossly wrong). If there's a subcomponent in your design that you don't know how to implement but believe to be solvable, it's OK to assume the existence of a black box that solves it.
Please take a moment to set an alarm clock. When the clock strikes 4 hours, please send us your code and research notes, and then go over the code and document everything you’d change before you consider the code production-ready and send us the annotated code.
We will be available to answer clarification questions if needed.
Then there is a checklist to reduce loss on technicalities (forgetting to set an alarm or send annotations).
(If you're in Israel, and you want to try it for fun, shoot me an email)
