Morally Invalid Statistical Predictors
Not enough discrimination?
Aleks pointed me to this article by Stan Liebowitz on the recent financial crisis:
At the crisis’ core are loans that were made with virtually nonexistent underwriting standards - no verification of income or assets; little consideration of the applicant’s ability to make payments; no down payment. Most people instinctively understand that such loans are likely to be unsound. But how did the heavily-regulated banking industry end up able to engage in such foolishness? . . . it was the regulators who relaxed these standards . . . a “landmark” 1992 study from the Boston Fed concluded that mortgage-lending discrimination was systemic. That study was tremendously flawed - a colleague and I [Liebowitz] later showed that the data it had used contained thousands of egregious typos, such as loans with negative interest rates. Our study found no evidence of discrimination. . . . Ironically, an enthusiastic Fannie Mae Foundation report singled out one paragon of nondiscriminatory lending, which worked with community activists and followed “the most flexible underwriting criteria permitted.” That lender’s $1 billion commitment to low-income loans in 1992 had grown to $80 billion by 1999 and $600 billion by early 2003. Who was that virtuous lender? Why - Countrywide, the nation’s largest mortgage lender, recently in the headlines as it hurtled toward bankruptcy. . . . This damage was quite predictable: “After the warm and fuzzy glow of ‘flexible underwriting standards’ has worn off, we may discover that they are nothing more than standards that lead to bad loans . . . these policies will have done a disservice to their putative beneficiaries if . . . they are dispossessed from their homes.” I [Liebowitz] wrote that, with Ted Day, in a 1998 academic article.
There’s a lot of interesting stuff here, from the frustration of an academic Cassandra whose 10-year-old warnings have been ignored, to the interplay between economics and politics in setting bank lending rules. Liebowitz presents the story as if it’s obvious that Countrywide etc. were making bad loans (in his words, “Sound crazy? You bet”). It’s hard for me to believe that the president of the Boston Fed and the activists at ACORN were strong enough to muscle the 1995 Congress into passing such a bad law. There must have been some other arguments or interests in the law’s favor. Liebowitz must be oversimplifying the political aspects of his story (although I can see how he might do that in his frustration of having predicted the problem in 1998 and having seen nothing done about it).
It all comes back to statistics (of course)
My real interest in this story (and, I assume, the reason why Aleks sent to me) is statistical. Liebowitz is talking about rules for statistical inference, prediction, and decision making that aren’t allowed to use certain information, even if that information has predictive power.
To remove this from the politically-charged area of racial discrimination, let me give an example from education. Suppose you are giving final grades in a college calculus class, and you can use the following information: homework grades, midterm exam score, and final exam score. The goal of the grade is to assess how capable the student is at calculus. Presumably the best estimate will be some weighted average of homeworks, midterm, and final.
Now suppose you have some students whose final exam scores are missing–for simplicity, imagine they are missing completely at random, and for some reason you can’t have the students retake the exam. For these students, you can estimate what their final exam scores would’ve been, by fitting a regression of finals on midterms and homeworks, using the other students in the class.
OK, fine. Now imagine that you have one more bit of information available on the students–their math SAT scores. And further suppose that this variable adds predictive power: that is, in a regression of final exam scores on midterms, homeworks, and SATs, that the coefficient of SAT is clearly positive. The question is: should you use it? What if you have two students with identical homeworks and midterms, but one got a 500 on his SAT and the other got a 700? Should you impute a higher final exam score to the kid with the 700 SAT? This would be a better predictor, but somehow it doesn’t seem fair. If anything, it almost seems unfair in the other direction, that the overachiever who did so well despite his 500 SAT would get punished. But, really, homework and midterm exam scores are noisy, and more information should be better.
Now you can consider sex and race as additional predictors and you see the problem. In many settings, information is available that you can’t use because of some fairness rule. (The #1 example, maybe, is pre-existing conditions in medical insurance.) No easy answers, but to me it helps to think how it plays out in an apolitical measurement setting.
And now for something completely uninformed
To get back to the mortgage mess . . . my own, completely uniformed, take is that moral hazard has a lot to do with it. I can’t find the article where I read this–I think it was in the Times–but I saw some discussion of how various industry and government people were trying to put together a solution that would allow for some relief, but still allow people to get the profits if house prices went back up again. Hey . . . where’s the money for that coming from??

