The core difference is that when Twitch plays an ad, they'll never send you that part of the video. [1] So buffering doesn't help. If YouTube would do this, you could have a custom player that preloads enough of the video so that all ads could be automatically skipped and as a viewer you wouldn't notice their existence. However, on Twitch, even if you're willing to give up the live factor and would buffer, you still would have missing parts of the video where an ad was placed. So you would lose content. [2]

--

[1] They do send a super low resolution stream with no audio, but not the actual quality you are watching in.

[2] Right now Twitch does not stop sending content in markets where they don't have ad inventory. So there are actually browser extensions already that will use a custom foreign market server to proxy the video during ad breaks, so that you can still have an 100% effective ad blocker.

For livestreaming it's easier because you can't skip forward anyway

The server can't completely ignore the player, as it would have to adjust the embedded timestamps to be consistent with the skip operations, or otherwise the player won't show the video. In other words, the server would have to act as if the ad is embedded wherever you skip to in the video. Not only would it mean that users lose their position in the video when they try to skip, since after the ad it would continue in a different place, but it would also mean that timestamps don't uniquely identify positions in the non-ad parts of the video anymore, which is a nonstarter in many ways.