Especially since we are talking 10-year increments here, for the pandemic to have a real impact on census stuff, it would have to have killed many people _who were not otherwise going to die over the next decade_. Most likely, we front-loaded a lot of these expected deaths to the early part of the decade. That is to say, at some point in the next 10 years the census will be an accurate representation of living voting-age people in the USA.
I didn't balk at 500k deaths. In fact, I tried to make sure that nobody would accuse me of downplaying the deaths/impact of the pandemic.