The default socket receive and send buffers are ~200KB, so you would actually need 400 GB of memory in order to have each of those 1048576 file descriptors connected to a unique socket.

And if you were keeping them open for 5 minutes as suggested, that would still limit you to only 3400 clients / second.

I do actually agree that they need a longer idle timeout on these connections, but I just wanted to point out that comparisons with the processing power required to set up a TLS connection aren't apt.