(IANAL disclaimer) The freedom of association has been implied by analogy to be protected by the explicitly-enumerated freedom to peaceably assemble in the First Amendment, in some specific cases. But it's a loose protection by analogy; the letter of the First Amendment doesn't even mention a freedom of association. The SCOTUS has not ruled that the freedom of association extends to a freedom to exclude in public accommodation; in layman's terms, the reasoning is "You have the freedom to serve as many people as you want, but that does not imply a freedom to not serve other people if you want to operate under the corporate charter that the government grants."
This overview describes things much better than I can. https://constitution.findlaw.com/amendment1/first-amendment-...
> The court was wrong and will be wrong again.
No doubt, but I don't think you'll find a receptive ear (in the public or in legal circles) to the notion that the Court has been wrong regarding upholding the Civil Rights Act.