Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to.
Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.