The impact to the whole workforce, industry and/or economy as a whole is a second order effect of the real ethical issue of providing a worse service for so cheap it's almost free such that the market won't bear significantly better service provided by humans. As I see it, the ethical concerns are not about specific people being out of a job, but with setting an expectation that it's not worth providing real, useful service (using actual people) because to do so would be a cost higher than phoning it in with AI.