The assertion is "growing the economy does not benefit the average person; the rich take all the gains". Well, let's think about moving in the opposite direction: "Shrinking the economy does not hurt the average person; the rich will take all the damage".
When I put it that way, the statement seems (at least to me) much less believable. So we have a statement about the shape of the productivity-vs-personal-well-being graph that seems believable if we move one direction, but seems unbelievable if we move the other direction. That makes me much more skeptical of the whole idea.
I think it's more likely to be the case that we aren't recognizing all the ways that productivity is enhancing the average person's well-being.
