It's like the fearmongering around MDMA, which claims that there will be delayed harmful effects from meddling with your brain chemistry, but hundreds of millions of users and decades later where is the widespread harm? Arguably it proves that it's one of the safest drugs on earth.
Of course controlled studies are rigorous, but widespread use must also lead to widespread harm if it exists. There may be hidden harm because it's a small effect on a large population or only harms a small population, but that just means it's relatively safe.