Can AI Discriminate? 💳
The potential for Artificially Intelligent bias, and what we can do about it.
3 min readMay 12, 2021
By: John Cameron
TL;DR
Goldman Sachs is being investigated for potential bias within its credit-limit algorithms after multiple prominent cases emerged from couples claiming that the husbands received credit limits that were orders of magnitude larger than their wives, despite the minimal disparity in asset ownership and bank activity. This list includes Steve Wozniak, one of the original co-founders of the Apple Computer, and his wife, Janet Hill.
The Breakdown
- With the continued rise in the prominence of Artificial Intelligence (AI) and Machine Learning (ML), traditional lenders are continuing to increase their usage of algorithms in order to determine credit as a means of reducing overall costs, boosting loan applications, and improving overall accuracy
- Many Big Tech companies (Amazon, Apple, Facebook, and Google) have already begun making inroads into the financial services sector, offering various loans and payment options directly to their users
- One particular instance of this is Apple Card, a joint venture credit card from Apple and Goldman Sachs released in early 2019, designed primarily to be used with Apple Pay through Apple’s ecosystem of devices (iPhone, iPad, Apple Watch, & Mac)
- Initially deemed by executives from both companies to be one of the most successful launches ever, a number of recent cases involving high-profile figures in the tech space have uncovered a potential bias built into credit limit algorithms used to set the maximum amount of credit extended to users
- David Heinemeier Hansson (creator of Ruby on Rails and Cofounder at Basecamp) posted a series of tweets back in November of 2019 detailing the eye-popping 20x disparity in credit provided to him versus his wife, despite filing joint tax returns indicating it was his wife who had a better credit score
- His comments were further echoed by Steve Wozniak, (one of the original creators of the Apple Computer alongside Steve Jobs), who discovered a 10x difference in credit limit despite no separate assets nor bank or credit accounts between him and his wife
- Consequently, a Wall Street regulator has now opened a probe into Goldman Sachs Group Inc.’s credit card practices, conducting an investigation to determine whether New York law was violated through discrimination against a protected class
The Significance
- Another similar case was opened in late 2019, when health giant and largest US insurer, UnitedHealth Group Inc. was accused of an algorithm that allegedly favored white patients over black patients
- Cases such as these lie at the increasingly prominent intersection of ethics and AI, and serve as a conversation-starter in discussions on what fairness entails within the greater context of AI
- Indeed, there are noticeable disparities across some demographics (hence why younger people tend to have lower credit scores compared to older ones), however, the question of whether these should be built into algorithms is far more intricate because, inevitably, some demographic attributes, unlike age, are largely immutable
- All these serve to further confirm the importance of delving into the development of XAI (Explainable AI), as a means of moving away from the black-box nature of many algorithms today that do not allow their outcomes to be broken down and reasoned through more precisely
