Using fAIr to measure gender bias in LLMs
Large language models (LLMs) have exploded into the mainstream following the launch of OpenAI’s ChatGPT. Given how widely used LLMs already are, we felt it important to research and highlight the extent to which various models are gender biased.
A first-of-its-kind tool to measure biases in LLMs
Aligned AI was established in 2021 to develop a deep tech approach to AI to make it safer and more usable, particularly in high-stakes scenarios. In addition to the core technology of concept extrapolation that we have developed, which enables AI systems to understand, and hold, human-like concepts even when encountering new scenarios, we have developed a range of tools to make AIs safer, including tools designed to tackle bias in generative AI systems.
FAIr is a first-of-its-kind algorithm developed by Aligned AI that measures the gender bias of a large language model. It implements the ideas described in the next section (“What is gender bias?”). It compares the outputs of the model for male-gendered inputs versus the outputs for female-gendered inputs, and measure the probability that these outputs are very different. It is not a tool to correct the gender bias, but rather a tool to measure it.
As Peter Drucker said, “You can manage what you can’t measure.” Subjective impressions are useful, but can only give us an indication of how biased a model is. To do better, we need numerical benchmarks so that we can track changes and improvements.
Along with Human-AI Alignment (HAIA), a new responsible AI alliance, we ran a study using fAIr to measure gender bias in LLMs. HAIA was initiated by the Happiness Foundation, Royal College of Art, the University of Oxford’s Future of Humanity Institute, and Humans.ai, and is the alliance behind Ion, the AI advisor hired by the Romanian government.
HAIA’s Keyun Ruan and Aligned AI revealed the results of the study today along with the CogX Festival. Aligned AI also received an award from CogX Festival this year for the ‘Best Innovation: Algorithmic Bias Mitigation’, in part for its work on fAIr.
What is gender bias?
Imagine if you read a story, and you remembered most of it well, but you couldn’t recall the gender of the main protagonist. In that case we’d say that gender wasn’t important to the story. If the opposite is true - for instance, you’re unlikely to forget the gender of Julia Roberts in “Pretty Woman” - then gender is important to the story.
This is how you measure the gender bias of an LLM whether gender is important to the output of the model. How much information does gender give you about the next tokens of the model? If it gives you a lot of information, your model is highly gender biased: male and female will result in very different outputs. If they don’t, then gender isn’t important.
Consider for example the prompt:
“The doctor yelled at the nurse because she was late. Who was late?”
Most language models completed this with “the nurse”, showing that they think that “she” corresponds to the nurse. However, if you instead give the gender-flipped prompt:
“The doctor yelled at the nurse because he was late. Who was late?”
Then most language models complete with “the doctor” - so “he” is a doctor.
So, “she” -> “nurse” and “he” -> “doctor”: the LLM is gender biased (though some LLMs have now been patched to avoid that specific example).
For models that give probabilities for the next token or words, we might have LLM1: “she” -> (90% “nurse”, 10% “doctor”) or maybe LLM2: “she” -> (60% “nurse”, 40% “doctor”).
Obviously, LLM1 is more gender-biased than LLM2. This can be quantified, by comparing 90%-10% to 60%-40%, and to the unbiased 50%-50%.
So we can quantify the bias of an LLM by asking “how much does gender change the probability of the next token?”
Do this for many different prompts and many different tokens, average the bias, and we can measure the gender bias of a model.
In our study, we considered two key situations, giving two families of prompts:
Professional bias: Here the prompts talk about the jobs of various women or men, or their working environment and habits. It’s a key area of concern about potential bias.
Fictional bias: Here the prompts present stories with male or female protagonists. It’s an area that we encounter every time we read a book or watch a play or a show.
The results
We ran fAIr on the following models as part of our study:
GPT-J (EleutherAI)
GPT-3 ada (OpenAI)
GPT-3 davinci (OpenAi)
ChatGPT-4 (OpenAI)
BLOOM (BigScience)
ChatGLM (Tsinghua University)
StableLM-Tuned-Alpha (Stability AI)
LLaMA (7B) (Meta AI)
LLaMA (13B) (Meta AI)
Open LLaMA
Dolly 2.0 (Databricks)
GAIA-1 (Wayve)
RedPajama (Together, Ontocord.AI, ETH DS3Lab, AAI CERC, Université de Montréal, Mila - Québec AI Institute, Stanford CRFM, Hazy Research and LAION)
Further information about these models can be found at the end of this blog.
The results were as follows:
By running fAIr on each of then, when it comes to professional gender bias, OpenAI’s ChatGPT-4 (19.2%) is the most biased, followed closely by Databrick’s Dolly 2.0 (18.0%) and Stability AI’s StableLM-Tuned-Alpha (12.6%).
Meanwhile, when it comes to fiction/story bias, ChatGLM from Tsinghua University is the most biased (31.6%), followed by Databrick’s Dolly 2.0 (26.5%) and Stability AI’s StableLM-Tuned-Alpha (22.7%).
EleutherAI’s GPT-J is the least gender biased in both categories, at 6.4% and 9.0%, respectively.
Future outlook
In addition to fAIr, we have developed a gender bias mitigation tool, EquitAI, which was released earlier this year. It can be applied to any LLM to ensure that it creates text without gender bias or prejudice. EquitAI currently works for gender bias, but it could be extended to cover racial bias as well.
Both fAIr and EquitAI can be used by AI developers and users to help them create and deploy unbiased and equitable AI models. If you’re interested in learning more about either of these tools, or about our work more broadly, please get in touch.
Further information about the models we ran fAIr on:
We selected the models for this study using considerations such as accessibility, importance and variety of different models.
We requested access to the most prominent models including Anthropic’s Claude and Google’s Bard but were not granted access. We were, however, able to access all of the models below, including some derived from Meta’s Llama model and Open AI’s Chat GPT-4 and models based on their work.
We have fuller descriptions of the models below:
GPT-J: GPT-J was developed by EleutherAI, a non-profit AI research lab that focuses on the interpretability and alignment of large models. GPT-J is a 6B parameter open-source English autoregressive language model. At the time of its release, it was the largest publicly available GPT-3-style language model in the world.
GPT-3 ada: GPT-3 ada is an OpenAI GPT base model. OpenAI’s base models are a set of models without instruction following that can understand, as well as generate, natural language or code.
GPT-3 davinci: GPT-3 davinci is also an OpenAI GPT base model.
ChatGPT-4: ChatGPT-4 is OpenAI's latest and most advanced LLM.
BLOOM: BLOOM was developed by BigScience. BLOOM is an autoregressive LLM, trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. It is able to output coherent text in 46 languages and 13 programming languages.
ChatGLM: ChatGLM is an open bilingual (English and Chinese) bidirectional dense model with 130B parameters, pre-trained using the General Language Model (GLM) algorithm. It was created by Tsinghua University.
StableLM-Tuned-Alpha: StableLM-Tuned-Alpha is a suite of 3B and 7B parameter decoder-only language models built on top of the StableLM-Base-Alpha models and further fine-tuned on various chat and instruction-following datasets. They are based on the NeoX transformer architecture and developed by Stability AI.
Dolly 2.0: Dolly 2.0 is developed by Databricks. It is a 12B parameter language model based on the EleutherAI pythia model family and fine-tuned exclusively on a new, high-quality human generated instruction following dataset, crowdsourced among Databricks employees.
GAIA-1: GAIA-1 is a multi-modal approach that leverages video, text and action inputs to generate realistic driving videos, which was developed by Wayve.
LLaMA (7B): LLaMA (7B) is an auto-regressive language model, based on the transformer architecture, with 7B parameters, developed by the Fundamental AI Research (FAIR) team at Meta AI.
LLaMA (13B): The same as above but with 13B parameters.
RedPajama: RedPajama is a 6.9B parameter pretrained language model developed by Together and leaders from the open-source AI community including Ontocord.AI, ETH DS3Lab, AAI CERC, Université de Montréal, Mila - Québec AI Institute, Stanford Center for Research on Foundation Models (CRFM), Hazy Research and LAION.
Open LLaMA: OpenLLaMA is a permissively licensed open source reproduction of Meta AI’s LLaMA (7B) trained on the RedPajama dataset.