I'm a little bit too familiar with podman. LXD is more mature and actually implements all of the hardening features I mentioned. podman
could implement them in theory, but doesn't. Its default security posture is very similar (though not the same as) Docker. Don't get me wrong, I do want to see podman succeed -- but I don't like the amount of unneeded hype around it. It's effectively a Docker rewrite by Red Hat (and other folks) that has some fairly important improvements, but it's not a revolutionary new concept. As for buildah, I am too biased to respond to that question.
Oh, and most of the Docker CVEs found in recent years -- including those I've found -- have also impacted podman. The most brazen example is that podman was vulnerable to a trivial symlink attack that I fixed in Docker 5 years ago[1,2]. It turns out that both Docker and podman were vulnerable to a more complicated attack, but the fact that podman didn't do any special handling of symlinks is just odd.
[Disclaimer: The above is my personal opinion.]
[1]: https://github.com/containers/libpod/pull/3214
[2]: https://github.com/moby/moby/pull/5720