It's certainly another interesting idea, but from the top of my head, I find several drawbacks: The first is NDAs, as not all code may be exposed to non-employees. Secondly, it will disrupt your team for a bit, because at least one developer should be available at all times and can't really join discussions or go into deep focus time. Finally, and most importantly, I think it's impossible for such an interview to have internal consistency: Every candidate will probably solve a different problem (or do you keep a bug open just for interviews?) in a very different social situation. If you're more the nervous type, you suddenly have a whole room full of full-blown developers see you sweat. Internal consistency, however, is one of the most important themes in interviews for me. Everyone deservers teh same chance.
> Paying every candidate is an expense, but is it less expensive than hiring the wrong candidate and then needing to fire them? It seems like other comments here have already pointed out that they pay something like 160k annualized salary hourly rate, which seems pretty reasonable.
But is it less expensive than other methods of interviewing? If a classic whiteboard has a slightly worse chance of hiring the wrong person, but has no additional cost, it might still be better, all in all. Additionally, the expense depends on how many you interview for a given position (is there a filter interview beforehand or do you invite every reasonable resume?). A better idea that we sometimes use it to hire uncertain, but hopeful candidates for a paid internship. If they're good, the get a full contract; if not, we haven't lost much. Also, it's an easier decision after several weeks than after four hours. However, we can't do this for everyone.