The people who study this stuff seriously end up concluding that cultural and domestic factors are the biggest predictor. There are plenty of minority groups who at one point didn't have any stuff, and were discriminated against (Jews, Irish, Italians, Chinese, Japanese, etc.). The main difference seems to be cultural values that prioritize the nuclear family and educational attainment. The SAT isn't racist, poor black people who study do far better than rich white people who don't.
If America was so racist, the single most successful ethnic minority wouldn't be Nigerians. It has nothing to do with race and everything to do with the culture, family, and values you grew up with.