The Challenge

Machine learning algorithms can contain significant gender and ethnic bias. Where in the machine learning pipeline does bias reside: The input data? the algorithm itself? the types of deployment? More importantly, how can humans intervene in automated processes to enhance and, at least, not harm social equalities? And who should make these decisions?

Importantly, AI is creating the future (technology, i.e., our devices, programs, and processes shape human attitudes, behaviors, and culture). In other words, AI may unintentionally perpetuate past bias into the future, even when governments, universities, and companies such as Google and Facebook have implemented policies to foster equality. So, the big question is: how can we humans best ensure that AI supports social justice?

Gendered Innovation 1: Mapping Known Examples of Human Bias Amplified by Technology

Known examples of gender bias

• a. In Google Search, men are five times more likely than women to be offered ads for high-paying executive jobs (Datta, A. et al., 2015).
• b. Google translate has a masculine default. State-of-the-art translation systems like Google Translate massively overuse male pronouns (he, him) even where the text is referring to a woman. Google Translate defaults to the masculine pronoun because «he» is more commonly found on the web than «she» (at a ratio of 2:1). This unconscious gender bias amplifies gender inequality (Schiebinger et al., 2011-2018). When a translation program defaults to «he said» in reference to a woman, it again increases the relative frequency of the masculine pronoun on the web that may reverse hard-won advances toward gender-equal language in human society.
• c. Word embeddings capture associations between words that risk perpetuating harmful stereotypes, such as «man:computer programmer :: woman:homemaker» (Bolukbasi, T. et al., 2016). Word embedding is a popular machine learning method that maps each English word to a geometric vector such that the distance between the vectors captures semantic similarities between the corresponding words. The embedding successfully captures analogy relations such as man is to king as woman is to queen. However, the same embeddings also yield man is to doctor as woman is to nurse, and man is to computer programmer as woman is to homemaker. Taking no action means that we may relive the 1950s indefinitely.

Method: Analyzing Factors Intersecting with Sex and Gender

It is important to analyze sex and gender, but other important factors intersect with sex and gender. This is what scholars call «intersectionality.» These factors or variables can be biological, socio-cultural, or psychological, and may include: age, disabilities, ethnicity, nationality, religion, sexual orientation, etc.

View General Method

Known examples of ethnic bias

• a. Software used by US courts may be more likely to flag black (as opposed to white) defendants as being at a higher risk of committing future crimes—see discussion on defining fairness below (Angwin, J., & Larson, J. 2016). Solution offered by this paper (Kleinberg et al., 2017).
• b. Searches for black-identifying names result in ads using the word “arrest” more often than searches for white-identifying names, whether or not the person in question has an arrest record (Sweeney, 2013).
• c. Nikon’s camera software is designed not to take a photo if someone is blinking. It misreads images of Asian people as perpetually blinking.
• d. Researchers have deployed convolutional neural networks to identify skin cancer from photographs. They trained their model on a dataset of more than 1.28 million images, 60% of which were scrapped from Google Images (Esteva et al., 2017). But less than 5% of these images are of dark-skinned individuals and the algorithm was not tested on dark-skinned people. Thus the performance of the classifier could vary substantially across different populations.
• e. Growing applications of predictive models in genetic medicine also highlight the danger of data disparities. As of 2016, over 80% of all genetics data are collected from individuals of European ancestry. This means that disease risk predictors may fail dramatically for non-European populations. Indeed, patients of African ancestry are more likely than those of European ancestry to be wrongly diagnosed with increased genetic risk of developing hypertrophic cardiomyopathy—a life-threatening heart condition (Popejoy & Fullerton, 2016).

Known examples of gender intersecting with ethnicity

• a. Commercial facial recognition systems misclassify gender far more often when presented with darker-skinned women compared with lighter-skinned men, with an error rate of 35% versus 0.8%—see also solutions below (Buolamwini & Gebru, 2018).

Gendered Innovation 2: Mapping Solutions

Building training datasets to avoid bias

• a. Several research groups are actively designing “datasheets” containing metadata and “nutrition labels” for machine learning datasets (Gebru et al., 2018; MIT Media Lab, 2018). While the specific form of metadata will depend on the domain, each training set should be accompanied by information on how data were collected and annotated. Data containing human information should summarize statistics on the geography of the participants, their gender and ethnicity, etc. Data labelling done via crowdsourcing, such as MS Turk, should include information about crowd participants, along with the instructions given for labelling.
• b. Peer-reviewed journals and conference organizers of the International Conference on Machine Learning or Neural Information Processing Systems should require standardized metadata for submissions. AI competition platforms, such as Kaggle, and data repositories, such as OpenML, should do the same.
• c. In collecting more diverse data, it is also important to go beyond convenient classifications—woman/man, black/white, etc.—which fail to capture the complexities of (trans)gender and ethnic identities (Bivens, 2017).

Developing methods to algorithmically detect and remove bias
It is important to be able to detect when an algorithm is potentially biased. Several groups are developing tools for this purpose.

• a. In one case, researchers carefully curated a gender and skin phenotype-balanced image benchmark dataset and labelled images by skin type. They then used this dataset to evaluate the accuracy of commercial gender classifiers, with resulting discrepancies as discussed above (Buolamwini & Gebru, 2018).
• b. To identify the gender biases in word embeddings discussed above, researchers developed geometry-based techniques. They then developed an algorithm to de-bias such embeddings, e.g. such that babysitter would be as equally close to “grandfather” as to “grandmother” (Bolukbasi, T. et al., 2016). Other researchers applied the standard Implicit Association Test to word embeddings to demonstrate that they contain many human-like biases (Caliskan et al., 2017). Still, others have developed a gender-neutral variant of GloVe (Zhao, 2018). For images, researchers first showed that training algorithms can amplify biases present in the training data and then developed a technique to reduce this amplification (Zhao et al., 2017).
• c. Similar techniques can also be used to study historical stereotypes and biases in society at large. Researchers used geometry-based metrics for word embeddings trained on text data over 100 years to identify changes in historical gender and ethnic biases (Garg et al., 2018). Similar methods may be used to compare biases in texts today.

Such bias detection and reduction techniques are also necessary for other domains, such as criminal justice (e.g., estimating defendant risk and making bail decisions, or for sentencing), loans and other financial matters, and hiring. However, in such areas, careful attention must be paid to the exact definition of unlawful bias and of the consequences of various mitigation techniques. Such applications are part of a growing research area that requires interdisciplinary collaborations.

• a. One well-studied approach is to specify a fairness definition, e.g., that the model is equally accurate across groups, and then constrain the algorithm so that it follows the definition (Dwork et al., 2012; Hardt et al., 2016; Corbett-Davies et al., 2017). Similarly, one could post-process the results from an algorithm to be fair (Hébert-Johnson et al., 2017). Defining fairness in such domains, however, has proven to be a significant challenge (Kleinberg et al., 2017).
• b. One further challenge is identifying which groups should be protected and then developing an efficient algorithm that is fair to all such groups. Recent work on AI audit has made progress in automatically identifying subgroups where an algorithm might be biased, and to reduce such biases. The auditor, which itself is a machine learning algorithm, is able to identify bias even without explicit annotation of the sex/race of individuals (Kim et al., 2018).

Gendered Innovation 3: Systemic Solutions

As we strive to improve the fairness of data and AI, we need to think carefully about appropriate notions of fairness. Should data, for example, represent the world as it is, or represent a world we aspire to—i.e., a world that achieves social equality? Who should make these decisions? Computer scientists and engineers working on problems? Ethics teams within companies? Government oversight committees? If computer scientists, how should they be educated?

Creating an AI that results in both high-quality techniques and social justice requires a number of important steps. Here we highlight four:

• 1. Attend to infrastructure issues Attempts to create fairness are complicated by the need to understand how bias is embedded in institutional infrastructures and social power relations. Unless carefully studied, structural bias is often invisible to social actors—whether humans or algorithms. Wikipedia, for example, seems a good data source. Information rich, it is the fifth most used website in the world. But there are structural problems: entries on women constitute less than 30 percent of biographical coverage, link more often to articles on men than vice-versa, and include disproportionately high mention of romantic partners and family (Wagner et al., 2015; Ford & Wajcman, 2017). These asymmetries may derive from the fact that globally women account for less than 20 percent of Wikipedia’s editors (Wikimedia, 2018).
• 2. Rigorous social benefit review. The Asilomar Conference 2017 set out 23 principles for Beneficial AI (https://futureoflife.org/ai-principles/). We now need to develop review mechanisms designed to achieve these principles. One organization working toward this ends is Fairness, Accountability, and Transparency in Machine Learning (https://www.fatml.org). Several governmental reports have addressed the issue (Executive Office of the President, 2016; CNIL, 2017).
• 3. Create interdisciplinary and socially diverse teams. New initiatives, such as the Stanford Human-Centered AI Institute; the Fairness, Accountability, and Transparency in Machine Learning Organization; and others are bringing together interdisciplinary teams of computer scientists, lawyers, social scientists, humanists, medical, environmental, and gender experts to optimize fairness in AI. This type of cognitive and social diversity also leads to creativity, discovery, and innovation (Nielsen et al., 2017).
• 4. Integrate social issues into the core CS curriculum. Computer science students should graduate equipped with the basic conceptual tools for analyzing gender and ethnicity as well as the broader social impact of their work. We recommend that social issues be integrated into core CS courses —at both the undergraduate and graduate levels—alongside introductions to algorithms.

Conclusion

Artificial intelligence is poised to transform economies and societies, change the way we communicate and work, and reshape governance and politics. We humans have created long enduring social inequalities; equipped with new tools, we have the opportunity to engineer fairness across and within cultures and thereby enhance the quality of life for women, men, and gender-diverse people worldwide.Note: Some materials in this case study draw from Zou & Schiebinger (2018).