Sadly, not possible on a shared residential connection. I tried the setup where the server was also the router, and that was much less convenient in case of failures than having a dedicated router.

But yes, the abundance of routers is definitely a problem here, with an OpenWRT and a FritzBox in the mix. Perhaps I should try to get IPv6 running and forget about NAT woes.