Now, the question is: would you trust it? As a human, a manager, a president? With the current generation, I treat is as a dumb but quick typist, it can create code much faster than I can, but the responsibility to verify it all is entirely on me. We would need decades of proof such an AGI is reliable in order to actually start trusting it - and even then, I'm not sure how safe it would be.
You have no option but to trust an ASI as it is all-powerful by definition. If you don't trust ASI, your only option is to prevent it from existing to begin with.
Edit: please note that AGI ≠ "human intelligence," just a general intelligence (that may exceed humans in some areas and fall behind in others.)
By this definition a calculator would be an AGI. (Behold -- a man!)
I don't understand this sentence. I don't trust Generative AI because it often spits out false, inaccurate or made up answers, but I don't believe my "only option is to prevent it from existing".
Making the central planner an AGI would just make it worse, because there's no guarantee that just because it's (super)intelligent it can empathize with its wards and optimize for their needs and desires.
Most humans can not lie all the time. Their true intentions do come out from time to time.
AGI might not have that problem - AGI might hide its true intentions for hundreds of years.
It is an argument about signal bandwidth, compression, and noise.
