So, nothing about my mental model of the world thinks that AI recursively self-improving (limitlessly) is possible but wouldn't a basic sort of math-y version of this be that most improvements AI can find in itself would be linear, but the marginal cost of improvements would be exponential?
I get that none of this stuff is like solid, but recursive self-improvement feels similar to a perpetual motion machine.
Base assumptions about the fundamental nature of the world have changed in the past (I like some of the examples in `sapiens`, even arguing that the idea of being able to improve our environment as a species was not believed for most of history) so I could be wrong, but fast takeoff feels like a fantasy.
I also think about cookie clicker math. A new technological paradigm may give us exponential improvement, but we eventually hit the next exponential. i.e. when you buy a big upgrade in cookie clicker, it can start making your current exponential (say if you're at millions of cookies) really fast, the wheel spins really fast until you hit about 1 billion in which case the wheel slows down to a crawl.
