A real life example: In many (all?) countries, it's common to pay more for your car insurance if you are young, if you are male, or if you recently acquired your driver's license. One might frame this as a prejudice towards young, male drivers as being reckless. But statistically they are just that, so the prejudice is balanced (within a reasonable tolerance). It's unfair to the careful young, male driver, but well, life is not always fair.
Collectivism is just fine when you are the one on the receiving end. Either we set absolute lines of discrimination across all industries or we disallow it entirely.
That's the computer science problem being worked on here.
Not everyone shares your politics.
I'm perfectly happy with algorithms detecting that certain people are more likely to be safe drivers than average, and giving them lower rates, and concentrating premiums on the groups more likely to be in accidents, even if I don't understand why Armenians (in your example) get in more crashes.
The problem being worked on here is "what if Armenians shouldn't get car loans because they don't pay them back as much as other groups?" I.e., algorithms rightly classifying people leads to results that we believe are "unfair".
This research is illustrating that you can't simultaneously have accuracy and fairness. You need to explicitly decide how much accuracy you are willing to give up to get fairness. I.e. it's computing the tradeoffs needed to evaluate the ethical question: how many Armenian deadbeats should you extend credit to in order to be "fair"?
Go play with the simulation to see. The various fairness criteria all achieve lower than maximal profits.
If a computer program spots an irrelevant unhelpful correlation its generalization error will noticeable go up. If it is an irrelevant "helpful" correlation it means there is a problem with the data (such as leakage), not with the algorithm. If there is a problem with the data, all bets are of, both for black and white box models.
A blackbox model will probably not find that being an Armenian alone will lead to more crashes. Being non-linear in nature it will find interactions (young male Armenians are more likely to crash than young males in general). If the importance of such a feature is not significant enough to distinguish it from noise, then regularization may automatically remove it.
Even if we observe that 99 out 100 Armenians crash their cars, and you decline someone a loan, because he/she is Armenian, you may just have discriminated against the 1 Armenian who is a safe driver. Young male drivers who drive safely have a worse time getting loans, because their group (the set of young male drivers) spoiled it for them. So their only hope of getting a loan is you adding more features (like nationality), to be able to distinguish them as safe drivers, not removing them and lumping them into the status quo.
To use your Armenian example, it could be that while being Armenian doesn't actually affect your driving, a "true" model could still end up being bad for Armenians if being Armenian is correlated with the things that actually do affect crash risk.
But what about this: we don't need to solve all social ills every single time. What if we let the algorithm correctly decide crash risk, and if we notice that it unduly impacts Armenians and that's not an outcome we want, we via a separate channel compensate the Armenians? That is, acknowledge the fact that Armenians may crash more, but give them a government subsidy to offset the higher premiums, and work to bring the premiums down (i.e.: fix the underlying issues that being Armenian is correlated with).
It's related to something I've been thinking about lately with regard to minimum wage. I like the idea that everyone should have a livable income, but tying the implementation to businesses that have low wage jobs seems like mixing concerns. For example, my company doesn't have any minimum wage jobs, but shouldn't my company chip into this social ideal same as any other?
What if we let the businesses pay whatever the market will bear, and if we decide that as a social concern people should get more than that, we subsidize them from the government, which is wear this concern is coming from in the first place.
Even if that was true[1], where are you getting the data from? Choosing which types of data to use is just as important as the model.
[1] as others have already pointed out, it depends on the technique/etc
However with a lot of ML classes we are told to look for the simpler explanation which would be nationality in this case, in the absence of collection of the real reasons.
Not sure how that can be gotten around. I'll read the paper.
My male Montana cousins always had slightly cheaper insurance than I did, but my female Montana cousins more than made up for it.
We accept that younger drivers pay more for car insurance. And that disabled people pay more for their health insurance. And that the government (in the UK) has a special business grants scheme for ethnic minority entrepreneurs.
Obviously discrimination depends on context. If a car insurance firm had a special policy for ethnic minorities, people would (rightly) be outraged. But in the context of a government intervention, based on the evidenced disadvantages that ethnic minorities face, discrimination is accepted.
This does not seem fair to me, because if this is applied then your race (group) would determine your credit score threshold which feels discriminatory to me.
I feel that, by definition, it is not discriminatory only if none of your attributes that you could be discriminated against (gender, race, ...) are taken into account at all.
Maybe the concept of 'equal opportunity' is just some compromise between discrimination and making less informed decisions.
For example, let's take blacks in the US. The data tells you that a black person is more likely to be a criminal than a white person. There are two possible reasons for this: (1) blacks are more prone to crime, or (2) blacks are more likely to live in circumstances that make them criminals. With access to only anecdotal data, I strongly believe that (2) is true, and that if you took into account enough circumstances (e.g. single parent, school district, income level, parents' wealth) you'd be able to remove race from your model and still arrive to the "equal opportunity" result. That way, you wouldn't discriminate based on race, but you would still help the people most needy of help (poor, uneducated, etc.). I think the same applies to colege applications and sex wage gap, which is why I strongly oppose any kind of affirmative action.
Life expentancy sex gap might be a different case. IIRC, men die earlier because of some behavioural/social tendencies (working dangerous jobs, supressing emotions, risky behaviour (speeding, smoking)), so maybe there are some non-discriminatory (or at least less-discriminatory) ways of determining life expectancy (e.g. testosterone level, job description, ...). However, there are also biological differences that seem very strongly linked to the quality of being male (i.e. the Y chromosome). E.g. prostate cancer is less lethal than breast cancer, and women are more likely to get MS than men. These particular examples suggest that men should live longer, but I'm guessing there might be other gender-specific ilnesses that might reduce their (our) expected lifespan. If that's the case, I don't think it's too discriminatory to have different insurance fees for different sexes.
In all such scenarios, however, there could still be broader societal goals that would override specific instances of (non-)discrimination. For exapmle, AFAIK women's health is more expensive (because of pregnancy), but having more children benefits everyone in the society (in the West), so it makes sense to "discriminate" against men by letting women pay less for their health insurance.
I'd like to add a third possible reason for your consideration. Since "criminality" i.e. guilt of committing a crime is determined after a process engaging the law enforcement and justice systems, we have to examine whether there are inherent biases in those systems that result in skewed statistics. For instance, do police officers selectively target blacks for monitoring and investigation? Are blacks discriminated against in the courtroom as a result of procedure or human nature?
This is pretty much the only way to help people who were born addicted to meth inside a trailer in the mountains who also happen to be blessed with being white.
The problem is, when given access to a large number of classifiers, some of which have inevitably been affected by a pre-existing racial bias, a black box machine learning algorithm will likely become discriminatory as well if race is not in some way represented and equalized.
For instance, many justice systems in the U.S. use machine learning software to determine the likelihood that a criminal will reoffend, and use that prediction to determine sentencing. Race is never used explicitly as a classifier, but the program ended up being significantly more likely to rate blacks as more likely to reoffend [1]. Classifiers like "had parents with previous criminal convictions" can be misleading when blacks are more likely to be convicted for the same crime as whites. It doesn't mean that the white person's parents didn't engage in criminal activity or other reprehensible behavior that might cause their child to become a violent, repeat offending criminal - just that they were able to get away with it more easily because of a biased system.
Machines end up just as biased as the data they've been trained on, so if we are going to use computers to judge things that have such a significant impact on people's lives, we can't risk racism slipping through the cracks.
[1] https://www.propublica.org/article/machine-bias-risk-assessm...
Past payment history: if you're black, prior discriminatory behaviors may have limited your ability to open credit accounts, and thus you have less history to go on. Available balance has the same reasoning. Zip code correlates with race.
I'd hope very few people are including an "is_black" feature in their classifiers. If you eliminate anything that is informative towards race though, you're likely going to have a classifier that doesn't work very well.
The problem is that we have datasets that have arisen from a history that included significant racism, both overt and latent. There is no way to separate those effects from the data. You either get an "optimal" classifier that is racially biased in ways we don't want, or you get one that intentionally gives up some perceived performance in favor of fairness.
In practice, the first variables to be used for classification are the ones that have a biggest effect. Then you're going to use them (from order of importance) as eliminatory or classificatory
> because if this is applied then your race (group) would determine your credit score threshold which feels discriminatory to me.
But the opposite is also discriminatory, which is what the article is showing. Because then you're using the same ruler to evaluate different groups, and of course the minority person with 2 jobs can't match the credit score of an Ivy-League educated WASP
[1] https://www.amazon.com/Weapons-Math-Destruction-Increases-In...
[2] http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...
Personally, I find this as the key outcome here. The accountability is on the people/systems who make the decision and that leads to an appropriate incentive. Win-win as a start.
It's a sensitive topic, because sometimes we're actually tampering with the data, trying to eliminate known human or selection biases.
The first defence against discrimination is, in my honest oppinion, for everyone working with data to be aware of these problems. To know that, besides ROC, precisions and recalls, we should measure the impacts of the models in sensitive demographics (gender, race, nationality, sexuality).
And one of the things that I learned (in [1]) is that, even if you're carfull with the features you use, you might still have a negative effect.
One needs to understand how these features interplay with each other. For example, you may not directly use a protected class feature (race) to make your prediction but you might end up using a secondary or tertiary variable (like location) to end up learning a protected class feature due to statistical correlations.
I would imagine that if we are separating people into groups based on demographic or social factors to make decisions, then those decisions may have an impact on that entire group. (In this example, maybe granting more loans to the blue group alters the group characteristics and leads to higher profit in the long term despite higher immediate risks).
Is there an area of ML research that considers this kind of concept?
My guess is that at most companies spending time on making an algorithm non-discriminating will be viewed as a waste of time and money.
Banks are built on trustworthiness. Having your bank's name in the headlines for discriminatory practices can have a severe negative impact on trustworthiness. They have a whole teams of people devoted to this topic and every year at most banks every employee has to learn about, "reputational risk."
Having worked at big banks for over a decade now I am 90% certain that discrimination by banks at this point is primarily due to carelessness.
Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate.
Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch?
Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables.
It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan").
It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate.
[1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
[2] https://www.youtube.com/watch?v=furfdqtdAvc
[3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236
Secondly, it isn't mathematically possible to be "nondiscriminatory" - there are multiple definitions of that term and they are mutually conflicting. For example, as this article shows, "equal outcomes", "equal opportunity" and "equal treatment" (group unaware) don't make the same decisions.
So no matter which definition of "fair" you choose, some intrepid reporter can choose a different definition and then write a clickbait article calling you racist.
It is an interesting question. However, there are ways to answer this question as we have been measuring discrimination before algorithms. Algorithms are a way of reaching a decisions. Therefore, we can measure discrimination if we audit decisions generated by algorithms. In other words, look at the input-output of the algorithm and measure the impact. In the US, the doctrine of disparate impact has long been used a guideline to evaluate discrimination [0].
[0] https://en.wikipedia.org/wiki/Disparate_impact#The_80.25_rul...
What is important to note here is that we need to tweak the mathematical model to the culture we want to achieve. In other words, the objective function of the optimization problem needs not only match the current state of the world, and provide an hindsight in one's own economic interests, it also needs to take into account the culture that we want to reflect, and influence. Otherwise, these statistics are just a giant status quo amplifier.
What I mean, say the model identifies that a certain group has a greater risk due to systemic problems. If you change something about the group, you can change the calculated risk without changing the model. And this may very well be a better way to achieve the outcome you want.
Specifically, by preventing insurance companies from using a more accurate mode, what you're demanding is that the random people who happen to have taken the same insurance packages but are not part of the group should make an extra contribution to fix these systemic problems.
But why them? Shouldn't we all contribute instead, hopefully using a fair system for assessing how much should each pay?
Instead of tweaking the model, you can change the inputs by providing a state-backed guarantee to the underprivileged groups. Isn't it more fair overall?
Specifically, a machine learning will produce different numbers for 2 individuals with the exact same characteristics except the race. And that is the problem that needs to be addressed.
Let's put it in another context. Let's say I'm a white athlete, and I'm very good at running the 100m race. Actually I run just as fast as a black person who is my main rival. Now if someone has to select one of us to go to the Olympics, they should toss a coin to decide who goes. If you use a ML algorithm, it would absolutely send the black person, because no white people has won a 100m race in the last 20 olympics. That's the kind of bias ML does and that needs to be addressed.
Good point. Systems should work for people, not the other way round.
The key thing to understand is that we're talking about discrimination that is usually unintended and unexpected by the designers of these systems.
This is presented as if it's an unambiguous fact, when it's largely a political stance.
That is surprising. And given this discussion about only wanting stats that have a valid explanation as well as fitting the facts, is interesting.
We detached this subthread from https://news.ycombinator.com/item?id=13006361.
It is a sensitive subject I suspect
And I cannot find the original parent comment in the discussion and I look like a total dick with my comment coming out of nowhere.
I will try and understand what moved where and get back to you
Aha:
"""A great example is how very resentful many young white men of college age are that universities are requiring them to take sensitivity courses designed to reduce the instance of campus rape, but strictly speaking men of that age are the overwhelming majority of bad actors in that environment. Statist"""
It is only my memory but I am fairly sure that the above comment said "strictly speaking young white men ..." which what prompted my comment. Maybe I transposed the white in the first part of the paragraph to the final part. Not sure. Please do check, let me know if I was being a fool or not.
Darn I need to md5 hash parent comments I reply to now :-)
I should have gone with "citation please"
Anyway, I have written up a sort of overly long comment which won't fit in 2000 chars so I can't submit it and so this is the link
http://www.mikadosoftware.com/articles/HNdisaster
I apologise for thinking this was not a slippery slope and for not reviewing my text for misunderstanding. And I apologise for putting a piece of text up there that is so blatantly... horrific. It's awful to have that in my permanent history even if I know how it was a mistake.
As it says in the article I am going to go and have some reflection time.
What a cock up.
See you in a few weeks.
Have you seen statistics to the contrary? Why would it be surprising if you don't have an informed prior in the opposite direction?
As in I am surprised that the parent claimed ... err the parent I replied too seems to have vanished and my comment is looking pretty weird
Anyway from memory he raised that young white men had a higher rape stats than any other group - I was surprised by the inclusion of race in that
Having read that, what would you assume is race and economic background is?
After criminal proceedings he was sentenced to community service.
Now, what would you assume is race and economic background is?
PS: Bias is insidious and really hard to control for.
White, because most people in the US are white.
Here's the big fallacy: "the two groups have different thresholds, meaning they are held to different standards." They are not held to different standards because they're different groups, but because of other reasons that indicate different loan default rates. So you cannot call this "discrimination". This is how things should be.
If you don't want to be banned, you're welcome to email hn@ycombinator.com. We're happy to unban people if they give us reason to believe they'll only post civil, substantive comments in the future.
