I can see two major delaying factors here:
1. Current generation LLM technology won't scale to true AGI. It's missing a number of critical things. But a lot of effort is being spent fixing those limitations. But until those limitations are overcome, humans will be needed to "manage" LLMs and work around their limitations, just like programmers do today.
2. Generalist robotics is far behind LLMs for multiple reasons, including insufficient sensors and fine motor control. This would require multiple scientific and engineering breakthroughs to fix. Investors will, presumably, spend a large chunk of the world's wealth to improve robotics to replace manual labor. But until they do, human hands will still be needed in the physical world.
The real danger is if AI passes a point where it starts contributing substantially to its own development, speeding up the pace of breakthroughs. If we ever hit that tipping point, then things will get weird, and not in a good way.