Plain <video> elements are easy to download, but not great for streaming, which is what most people are doing nowadays. Much of the JS complexity that gets layered on top is to facilitate adaptive bitrate selection and efficient seeking, and the former is especially important for users on crappier internet connections.

I'm not a fan of how much JS is required to make all that work though, especially given the vast majority of sites are just using one of two standards, HLS or DASH. Ideally the browsers would have those standards built-in so plain <video> elements can handle them (I think Safari is the only one which does that, and they only do HLS).