The issue with free cheap energy such as fusion isn't its direct byproducts, but its impact on society.
It will fuel an economic growth without precedent, which in turn will increase the consumption of material goods, which in turn will increase the extraction of natural resources. Those resources will become rarer and more difficult to extract, thus leading to more destructive extraction strategies.
Energy puts a cap to our ability to destroy the planet in order to satisfy our material needs. To top it all off, everyone around here loves to build new stuff, but finding ways to get rid of it without creating waste isn't as trendy.
I don't believe one second that our civilization is capable of controlling itself. And don't tell me about multi-planetary scenarios or whatnot. They are neither realistic (regardless of the energy availability) nor desirable.