That's not really what those people were saying.
It doesn't matter if the number is 5% with a low infection rate, or 0.5% with a very high infection rate: both will lead to very large numbers of dead people unless we take severe action early on.
And so far they've been right. Covid-19 is causing huge amounts of excess mortality, even when compared to a really bad flu year.