Before post-quantum cryptography concerns, KEM were indeed mostly built on top of Diffie-Hellman key agreement, but you could also build one on top of RSA, or on top of some lattice constructs. But you wouldn't build one yourself, there are good constructions to choose from! The OP actually has a 3-part series on KEMs, although I don't think it addresses post-quantum issues [2].
[1]: https://www.latacora.com/blog/2024/07/29/crypto-right-answer... [2]: https://neilmadden.blog/2021/01/22/hybrid-encryption-and-the...