One of the main questions I ask in interviews is basically "we have a data pipeline with goal X and constraints A, B, and C. How would you design it?" Depending on how they do, we'll discuss various tradeoffs, other possible goals/constraints, and so on.

This is based on a real system I designed and have been maintaining for ~5 years, and is also very similar to other systems I've run at previous jobs.

About half the candidates complain that it's not a realistic question.

As you climb the engineering IC ladder your responsibilities involve larger and larger tasks and timescales. It's not possible to measure whether someone can make the right decisions about the architecture of a service as requirements change over years in a 1-day interview. Anything that isn't "be the engineer maintaining this service for 1 year" is going to be a proxy for that, which can be learned and gamed.

What would be a "proper" interview in your opinion?

> Yes, ask give interview tasks which are realistic depictions of the actual job tasks.

I think this is exactly what a lot of companies try to do when interviewing. Depending on how much time they want candidates and interviewers to spend on the task, this ranges from leetcode-style problems to bigger coding challenges (possibly with some debugging or collaboration involved).

What would you suggest concretely?