But underneath all the bullshit is something truly useful, so I wouldn't necessarily say that Confidently Incorrect™ is ChatGPT's primary ability.
Coincidentally, over the past few weeks of this ChatGPT craze, I've needed to create a lot of fake data to seed a database. Normally not a big deal, but these records need to have foreign keys pointing to one another. I wondered whether ChatGPT could help me out, so I briefly described the fields I needed, their format and data type, the type of information they contained, etc etc. It did it almost perfectly, and fixing the little errors was trivial. I was dreading having to do this because it's such a pain in the ass.
To me that kind of thing is going to be the most useful application. Everyone's freaking out (in both the "scared" and "excited" sense) over AI's ability to replace creativity, but I'm focused on it's ability to replace toil.