A custom engine is never _needed_ technically speaking since COTS engines are often customizable to the point where you can do anything you want with them. That doesn't mean that they don't influence the design of games, though.
There's a talk that Casey gives where he explains how he implemented the movement system for The Witness, in which he shows examples of Unity-based "walking simulator"-type games dealing with limitations of the engine in ways that The Witness was able to totally avoid. This allowed the game's artists to be creative with set design without worrying as much about performance issues or collision bugs, thus potentially opening up more design space.
https://www.youtube.com/watch?v=YE8MVNMzpbo
Here's some praise of The Witness by Fabien Giesen:
> That’s where I am right now. I have never seen another game as cohesive as this. Not even close. Without getting into specifics or spoiler territory, I have never seen a game with such manifest intention behind every single detail, nor one where all the details cohere this well into a whole. This goes for the design and gameplay itself, but also across traditional hard boundaries. Game design and art play off each other, and both of these are tightly coupled with some of the engine tech in the game. It’s amazing. There is no single detail in the game that would be hard to accomplish by itself, but the whole is much more than the sum of its parts, with no signs of the endless compromise that are normally a reality in every project.
https://fgiesen.wordpress.com/2016/01/30/thoughts-on-the-wit...