That's a fair concern, but to me it seems like an "easy" yet flawed solution to systemic oppression, kind of like affirmative action. For starters, by drawing districts specifically along racial lines for minorities, you are de facto also drawing the other districts along racial lines for white people. I also think that more diversity in congressional districts is a good thing, as it avoids "lock-in" districts where the incumbent has essentially a 100% chance of winning every election due to homogeneity of their voters. Of course, the other side of the coin is that minority populations can get "diluted" to the point where they can be safely ignored by politicians.

I believe the real solution to our problems is to move away from geographic representation in its entirety. It's too easy to cheat, and by design it only ends up allowing representation of around half the constituents in its area. I'm not sure what could replace it; perhaps a parliamentary system based on opt-in voting blocks could work. But in its present form it does seem deeply flawed.