I'm speaking about impressions, not the actuality of it. One person's casual attire, which is the norm for a lot of interviews now, might be interpreted as "gang clothes" even when they're clearly not.
Like is a simple Kangol t-shirt "gang clothes"? By the same token a Lulu Lemon or Izod shirt is, they're just different "gangs". People have a way of reading into things. A well-dressed black person might look "like a drug dealer" while a scruffy white dude might look "like a real programmer". It's a matter of perception. Our pattern matching is often badly distorted by the media.
Improving is always a good idea, but if it completely handicaps entire groups of people it's not necessarily an improvement, it's just shifting the artificial rejection criteria.