I can't agree with your generalization that people who faced more challenges are likely to be better at doing the job. It seems possible those early life difficulties could be traumatizing, leaving those folks less resilient.
The US military used to think successful soldiers with childhood trauma, had coping skills that protected them in deployment. When they ran the studies, they found they were completely wrong - people with childhood trauma, regardless of their military success, were multiple times (4-6x odds ratios) more likely to develop PTSD, (re-)start smoking, or misuse substances.
It's a neat narrative: go through hardship + come out the other side = better coping skills / more productivity. However, humans are complex and often fragile.
I accept the argument that a person who has experienced more hardships has accomplished more to reach that same point. That could be justified if your hiring principles are "who has earned this spot more". It doesn't necessary follow that their trajectory has a steeper slope from the point of hiring.