You're not thinking creatively enough. I'm only interested in ESP, not IKE. Consider having the TLS handshake negotiate the use of ESP, and when selected the system would plumb ESP for this connection using keys negotiated by TLS (using the exporter). Think ktls/kssl but with ESP. Presto -- no orchestration of IKE credentials, nothing -- it should just work.
The real key is getting ESP HW offload.