To be clear, a lot of them aren't "AI scientists," just hypemen, many of them who came into the "tech" world chasing crypto.

Many, many AI scientists are very clearly aware of this problem. AI safety researchers have been predicting exactly the failure modes we see in the various incarnations of GPT for years. Issues like alignment and over-fitting are incredibly hard problems to solve, and there's still a lot that isn't known, but they are recognised in the field. The problem is business leaders, investors and governments are utterly clueless on any of this, hence the Google Bard and Microsoft Sydney debacles when they were rushed into the public view.

There is also the risk that the recent massive success of LLMs will bring a lot of excited bandwagon seekers into the field that don't have or appreciate the background on AI safety.

A half working system is good enough for real life tasks when combined and used by and in conjunction with attentive half working human collaborators.

What about helping correct the grammar and style of an article? Writing out boring code? Creating a test for code you already have? These are all tasks that have important economic consequences. AI is not half-bad at them, it is actually approaching human levels of performance.

If we continue to improve AI, maybe one day its performance in these critical tasks you describe will also surpass humans.

I’d take a half-working system over what we have now, which is a system I’d put at around a third-working.