The original problem was that racists were not just taking race into account but were disproportionately penalizing certain races. They would literally just refuse to do business with black people. And then once that was made illegal, they would refuse to do business with people from black neighborhoods (redlining), i.e. use location as a proxy for race so they could continue to refuse to do business with black people under that pretext.

Normal Bayesian statistics doesn't do that because it's missing the actually racist piece of it, which is giving disproportionate weight to race (or something that correlates with race) so that you refuse disproportionately many people of a particular race for no legitimate reason.

The unfairness never came from taking into account some factor that correlates with race, or even race itself to the extent that it actually correlates with outcomes. It came from using a factor to deny service even though it didn't correlate with outcomes, or if it did then still not proportionately to the huge negative weight assigned to it. It came from giving race, or a proxy for race, disproportionate weight. Giving it proportionate weight isn't unfair, it's the only thing that is fair.