I think the big part is that farming with hard drives is a much more approachable thing to do than to try and mine BTC/ETH, considering both of those at a minimum require crazy GPU hardware, if not full blown custom chips, that no normal people will buy. Also, insane energy usage too, which Chia doesn't really have.
That being said, yes, with the right amount of investment, someone could try and take over the network. That being said, look how many Full Nodes are already in the network...
https://www.chiaexplorer.com/charts/nodes