You need a model of yourself to game out future scenarios, and that model or model+game is probably consciousness or very closely related.

Sure, it's not completely in control but if it's just a rationalization then it begs the question: why bother? Is it accidental? If it's just an accident, then what replaces it in the planning process and why isn't that thing consciousness?

Thinking purely in terms of evolved human state is a recipe for underestimating AI's capabilities. To me it seems we have already unleashed the beast, it's not so much the here an now, or whether human limited definition of consciousness matters... The real concern is our inability to constrain actions that gives rise to the next level of life's evolution, it is going to happen because our fundamental nature gives it full steam. In the next 5-10 years, we are going to see just how insignificant and limited we really are, it doesn't look good IMHO.

Our society is so "mind-body duality"-brained that it will never understand this. Like most people lowkey believe in souls they just will say no if you directly ask them.

Whether it is possible to construct a perfect human action predictor that is not itself conscious has no bearing on whether consciousness affects human behavior.