The issue is that you're performing a test which involves a whole bunch of interconnected layers and then assigning one part of that interconnected system as where the benefits lie.
Why not the TLS implementation? The video drivers and their kernel interfaces? The OS' process scheduling? A dozen other things that might be responsible for the perceived performance difference to one degree or another?
It feels like gamers who blame 90% of multiplayer issues on "the netcode".
It'd be different with data looking specifically at TCP connection establishment timing without a bunch of other stuff involved.