I say this as someone who works on PKCS#11 code: It's not really possible to have a productive conversation with someone that is missing key domain experience and knowledge, but is so certain of their correctness anyway.
More concretely, a forked daemon only needs to support RSA and other crypto operations without revealing their keying material. They don't need a full TLS/SSL stack.
That said, there's absolutely no additional complexity in having both Apache and the hypothetical daemon using a full TLS/SSL crypto library. Any __TEXT pages will be shared, and duplicated __DATA and base-line library allocations are essentially zero.