In a world where AI’s reach continues to expand, the challenge of managing algorithm bias has become increasingly apparent. The recent missteps by Google’s Gemini chatbot highlight the complexities and unintended consequences of attempting to correct bias in AI systems. This post delves into the intricate nature of bias, exploring why ‘correcting’ it is far from straightforward. We’ll look at some philosophical dilemmas, societal implications, and the potential for AI itself to be part of the solution by uncovering and understanding the roots of bias. Ultimately, this piece poses the critical question: Can AI help us heal society, or are we at risk of engineering a distorted reality?
The Gemini Incident: A Lesson in Overcompensation
I am sure that everyone has heard by now of the somewhat problematic attempts of Google’s conversational chatbot, Gemini, to be more inclusive in its image generation efforts.
Concerned by the tendency of other text and image generation tools to exhibit bias, Google fine-tuned Gemini to present a broader range of outputs aimed at ensuring inclusive representation across various prompts. However, this well-intentioned effort led to numerous embarrassing instances that quickly spread across social media.
For example, the pope was depicted as an Asian woman, and the Founding Fathers, Vikings, and soldiers in Nazi Germany were portrayed as Black, leading to significant distortions of historical figures and events.
This overcompensation for bias extended beyond image generation, affecting Gemini’s text outputs as well. The chatbot was notably one-sided in its discussion of controversial topics, such as providing arguments in favor of affirmative action to drive diversity in higher education while refusing to provide arguments against. It also declined to generate content for a job description related to a fossil fuel lobby group, justifying its refusal by stating that lobby groups typically work in the public’s interest whereas fossil fuels are “bad.” At times, it simply declined to answer prompts entirely if deemed too controversial.
Google quickly acknowledged the problem and made efforts to rectify it, saying that their attempts to be inclusive “failed to account for cases that clearly should not show a range of people” and that the model became “way more cautious” than they had intended.
This may be an extreme example, but it raises important questions around dealing with AI bias. We know that there is bias in AI output, and we know that this bias can be severely detrimental to affected parties. It must be addressed, but what is the best way to tackle it?
Google showed us that the answer is not as simple as ‘correcting for bias.’ The nature of bias is that it is a highly complex social construct. It is rooted in an intricate interplay of factors, including individual personal experiences, cultural norms, and historical context. Because of this, what is considered biased can vary significantly across different communities, cultures, and group affiliations.
‘Correcting for bias’ will always be from a one-sided view of bias. As Patrick Lin, Professor of Philosophy at California Poly State University, says of the Gemini blunder: “Everyone was shouting about the bias baked into the system, but in doing so they ended up revealing their own biases.”
So, if bias is going to be ‘corrected’ in the way Gemini tried to, according to whose worldview is this to be done? And to what extent can it be done before it becomes more like social engineering?
This was clearly a concern with the Gemini blunder. Right-wing critics were quick to jump on it, claiming Google was pushing its own, left-wing political agenda. Even left-wing papers acknowledged the partiality and worried about the control such a tech giant had on shaping our information environment.
Ethical Frameworks and Unconscious Bias
In dealing with AI bias, how do we determine whose view is right when bias is pervasive and everyone’s perspective is inherently subjective? And even if we agree on a moderate approach to addressing bias, can we realistically manage it effectively?
Serap Keles has thought a lot about this and advocates a Virtue Ethics moral doctrine to address the issues of bias. Keles acknowledges that a “shared ethical framework” may seem idealistic but believes that certain human values transcend cultural and temporal boundaries, as they emerge from shared human experiences. This perspective is supported by the observation that independent philosophies, despite their distinct origins, have often converged on similar ethical principles. Keles contends that using universal virtues as a moral paradigm for AI (such as justice, fairness, kindness, empathy, and compassion) can prevent ethical relativism, where standards are influenced by individual perspectives and cultural norms, as seen with Gemini.
While Keles’s proposal for an ethical framework is compelling, its practical implementation presents challenges. It would require data scientists and developers to deeply internalize and actively apply these virtues throughout every step of AI development and deployment—a daunting task given the current landscape.
Additionally, the issue of unconscious bias adds another layer of complexity. Much of our bias operates automatically, without conscious awareness. Our brains naturally and rapidly form categories and stereotypes, often with little to no awareness on our part.
This is clearly evident in the analysis of how bias is introduced into the algorithm development pipeline by data scientists:
- Bias in AI often starts with the selection of training data. Although it’s typically illegal to weight protected characteristics like race or gender in algorithm design, bias still creeps in when data scientists unknowingly choose data that mirrors their own experiences or assumptions, rather than the full diversity of the population. This has been a problem in facial recognition, where people with darker skin tones are often misidentified (see Joy Buolamwini for how people are starting to address this).
- Another source of bias arises from the selection and weighting of data variables. Even without directly considering protected characteristics, bias can emerge when variables have a high correlation with protected characteristics, effectively becoming proxies. For instance, in predicting healthcare needs, Optum used healthcare spending as a key factor, which led to fewer Black patients being referred for further care, despite being just as sick as white patients.
- And of course, biased output also results from the enormous data sets that an algorithm works with once deployed. With generative models and large language models, this data is vast, reflecting every opinion uploaded online, biased or otherwise.To give you an idea of the size: ChatGPT3 worked with 500 billion words. That is equivalent to 45 million long novels. It is around 45 terabytes of data. To put that in perspective, counting to just 1 terabyte at the rate of one bit per second would take a single person 253,678 years.
The Limits of AI in Detecting Bias
While it is critical for tech developers to do their best to eliminate damaging bias in AI output (e.g., blocking discriminatory language and other clear examples of prejudice), it is arguable that they are unlikely to be able to adequately deal with the pervasive undercurrents of bias in society.
The subtle correlations between data features and protected characteristics are so widespread and nuanced that they’re incredibly hard to detect, making unconscious bias more likely to go unnoticed and uncorrected.
There are ongoing efforts to address bias in AI, ranging from ethical frameworks that promote careful consideration of data and variable selection, to rigorous testing and continuous monitoring of outputs. Developers are increasingly avoiding discriminatory language and other clear indicators of bias in algorithm design. Additionally, government policies, such as the recent EU AI Act, are driving industry changes by requiring transparency in “high-risk” AI systems, where businesses will be expected to be able to explain to consumers the basis for algorithmic decisions—something that is not easily done with black-box algorithms.
There is also a lot of talk about developing bias detection tools; however, the chances of AI being able to detect bias effectively are remote.
The Challenge of AI Bias Detection
At its core, algorithmic processes in machine learning involve statistical computations that learn patterns, relationships, and features within a training dataset, which can then be applied to new data for classifications and predictions once deployed.
While AI can effectively identify what human programmers define as bias—such as discriminatory language—it can only function within the parameters set by its human creators. AI excels at mathematical deduction, but it struggles with reasoning, understanding nuance, subtlety, or context. Its strength lies in following rules and learning patterns, not in “understanding” the material in a human sense.
We know already from the development of generative AI and other LLMs (large language models) that AI systems struggle with detecting what is racist, misogynistic, or unethical. Social media platforms have similarly struggled with controlling harmful content online. We know racism, for example, when we see it, but an AI doesn’t necessarily—especially in a novel situation. Therefore, an AI detection tool will likely encounter the same limitations as the models it seeks to monitor, constrained by the boundaries set by human programmers.
Where Do We Go from Here? Is There an Opportunity?
Addressing bias, whether in humans or AI systems, is a complex challenge. Bias originates within us and permeates the data we feed into our algorithms.
Tech developers appear to have a better grasp—or certainly more focus—on tackling blatant and harmful prejudice online. However, the subtle, pervasive biases embedded in society are equally damaging but more difficult to detect, making them much harder to address effectively.
The Limits and Possibilities of Bias Correction
It seems difficult for data scientists to eliminate subtle, pervasive algorithm bias due to the nature of unconscious bias. If the issue of unconscious bias could be avoided (say, with some clever AI model), the problem of the varied subjectivity of bias remains. Who decides which worldview is the one to correct for bias with? While it seems theoretically possible to settle on a range of universally acceptable virtues to mitigate this ethical relativism, practically, it seems exceedingly difficult to implement.
Importantly, the idea of ‘correcting’ bias encourages consideration of a descriptive vs. normative dilemma: Should we depict the world as it is, or as we want it to be? Choosing the latter may result in AI output no longer reflecting reality, distorting the truth, and depriving us of valuable insights into disparities of justice and equity.
Can We Apply AI to Understanding Bias?
We have seen that AI does not appear to be a solution in dealing with bias in the sense of detection. However, it is worth exploring whether we can harness the powerful capabilities of AI to dissect and understand bias at its core. This is more in line with depicting the world as it is.
The bias in AI output, especially unconscious bias, often arises from correlations and patterns detected by advanced statistical methods that go unnoticed by society. If we could harness this statistical power to explore underlying causes and drivers, we might drive meaningful change at the root level, addressing bias on the ground. This would eventually filter through to public data online.
Next Steps:
In the next phase of this discussion, we will explore the practical feasibility of this idea with data scientists and other experts. Stay tuned!
