The problem for me is that your conceptualization of worker's rights negatively impacts poorer people, stuck in poorer countries, who need that work more for basic survival.
So if we're forming normative opinions based on morality (reducing suffering or whatnot), why would I support that? And if we're not not appealing to morality, then what are you appealing to, and why would I want to go along with it?
The only argument I see is a nativist and protectionist one, which uses the superficial facade of morality as a misdirection.
My intention is not to punch down on anyone. I support some level of taxation and wealth redistribution. I support anti-slavery legislation.