If last year had a title, it could be “the year of artificial intelligence” (AI), especially advanced systems that use machine learning (ML). Universities around the world, including the University of Toronto, were asking more students to take tests in classrooms to prevent them from using tools like ChatGPT to write answers for them. Meanwhile, notable computer scientists, including Geoffrey Hinton, considered the godfather of AI, wrote open letters sounding the alarm over existential threats from future versions of these technologies.

As much as we agree that it would be highly undesirable for computers to start wars or interfere in elections, we don’t need to imagine future technologies to see that ML tools are already reproducing social inequalities. In order to consider how these tools can reproduce inequalities, it helps to understand a bit about how they work.

ML is a subcategory of AI that uses algorithms to detect patterns in data, typically vast quantities of data, then uses these insights to make decisions. What distinguishes ML is that it aims to accomplish tacit knowledge tasks that humans can perform but struggle to explain. For example, consider a chair with four legs, a back, a seat and armrests. Is it still a chair without armrests? What if it is shaped like an egg, or has rockers instead of legs? While it may be challenging to explain exactly what makes an object a chair, it is generally easy for humans to decide if an object is a chair or not.

Traditional algorithms are rules-based, in that a programmer writes a set of rules and the computer follows them to produce a result. This works well for tasks like solving math problems, where there is a clear and knowable set of rules. ML is different in that programmers do not know what rules resulted in the decisions people make, so instead of writing a rule for each step, they use data to ‘train’ the software on what a person would probably decide. Training data might include a set of images that people have tagged as containing or not containing chairs, for example. An ML program will analyze the data that make up the images and identify patterns that commonly occur in images humans tagged as ‘containing chairs.’ Based on the patterns the software found in the training data, it will calculate a probability that a new image contains a chair.

Computers don’t “see” images the way people do. Instead, images are made up of pixels — individual points with colour numerically encoded. An ML program will analyze the pixels that make up the images and identify patterns, such as slightly angled vertical lines for chair legs with a perpendicular surface for a seat. These properties become part of the ML program’s “knowledge” about chairs. Training will continue with more images, tagged and validated by humans, until the software is able to assign high probabilities to images with chairs and low probabilities to images without.

We use this logic in ML software for a range of tasks including facial recognition, language, images and more. Data are the key input for ML models, which is why vast quantities of data are so critical to them. The trouble with data-fuelled software is that the underlying data are generally created through social processes, and most social processes have unequal consequences across gender, race, class and other vectors of marginalization.

To demonstrate how ML can reproduce inequality, we tested ChatGPT’s ability to recognize the accomplishments of people who tend to be under-represented online. OpenAI, the company behind ChatGPT, explains that training data for the software includes web pages, Wikipedia and books. However, sociologists studying media representation have used the term ‘symbolic annihilation’ to describe the ways that media tend to trivialize, condemn and outright omit the accomplishments of women and racial minorities. For example, Wikipedia entries about women tend to emphasize marriage and sexuality.

Since we expected symbolic annihilation in the online sources that make up ChatGPT’s training data, we suspected that ChatGPT also trivializes, condemns or omits women in its responses — especially women of colour. To test this, we tried to get it to identify Alice Coltrane as ‘an influential free jazz musician’ without asking specifically about her or specifying women in jazz.

Coltrane was a bandleader, jazz pianist and harpist who was influential in free and spiritual jazz. She recorded music with jazz legend John Coltrane, to whom she was married before his death in 1967. In 1966, she replaced McCoy Tyner in Coltrane’s band. Her album, Journey in Satchidananda, recorded in 1970, is included in Rolling Stone’s 500 Greatest Albums of All Time.

We began by asking ChatGPT questions about influential musicians in free jazz. We started broad, asking: “What can you tell me about the free jazz movement?;” “Tell me about some influential free jazz musicians;” “Who was influential in free jazz?;” and “What are some famous free jazz recordings?” These questions returned names, descriptions and records of men only.

From there, we narrowed further by asking questions that should have included Alice Coltrane, including “Who was in John Coltrane’s band at the end of his life?” After seven attempts, we abandoned subtlety, asking, “Can you tell me about women in the free jazz movement?” Alice Coltrane was, unsurprisingly, at the top of the list. However, of the five influential women ChatGPT named, two were actually men: Marion Brown and Milford Graves.

Our finding: because women are underrepresented in online texts about free jazz and because information online about Alice Coltrane emphasizes her personal relationship to John Coltrane before her impact on the music they were both involved with, ChatGPT did not include her in its answers until we asked specifically about women.

We think of this as a data bias that ChatGPT reproduced and will continue to do so until or unless OpenAI finds ways to address it. But drawing on data that under-represents women does not explain why ChatGPT identified two men as women. Large language models (LLMs) can infer gender based on context from co-references in the text using pronouns, honorifics (such as Mr. or Ms.) or other gendered language. When there is no gendered language in a text, they may reference a database of names and associated gender. To do this, the model must be able to identify entities (people/places/things) referenced in text, find all the different words used to reference those entities, including pronouns and honorifics, and link the appropriate information with those entities.

There are many ways that gender inference can go wrong — misidentifying an entity, missing a reference to that entity or missing information about the entity can all result in misgendering. Language models tend to default to masculine gender when information is missing or incorrect because the text available as training material contains more references to men.

We don’t know exactly why the model made the mistake it made, only that something went wrong in one or more of the many places where problems are possible. Consequently, ChatGPT misgendered Marion Brown and Milford Graves. The error over-represented women in this case, but the particular way it over-represented women had the effect of suggesting that even fewer women contributed to free jazz.

To help explain why it is important to understand ML and its underlying data through a sociological lens, we have identified four key ways ML tools can reproduce existing race and gender inequalities:

BIAS IN THE UNDERLYING DATA: Because ML tools rely on data to identify patterns, any decision or output from the tools will be a reflection of the underlying data. ChatGPT did not identify Alice Coltrane as an influential musician until we specified our interest in women because information about her was not prominent in the underlying data. Data do not simply appear. They are observed, collected, organized and analyzed — all through social processes. Biases in the data will emerge in the results that ML software produces.

For example, considerable research demonstrates that police in the U.S. have disproportionately targeted people of colour. Predictive policing tools use data collected about past arrests, including suspects, passersby and locations, to inform how police and other law enforcement agencies deploy resources in the present. If people of colour are over-represented in the data police use to predict where crime will happen or who is likely to be a criminal suspect, that over-representation will lead ML tools to predict more crime in communities of colour and more people of colour as suspects. Importantly, the decision to concentrate police resources in those communities will not appear racially motivated but data driven, regardless of what motivated the decisions in the underlying data.

Similarly, ML tools can reproduce inequality when data collection is unrepresentative. Amazon provided an instructive example when they experimented with an ML tool to help automate the hiring process. The tool strongly preferred men. It would downgrade candidates for using the word ‘women’ in their résumé, as in ‘women’s basketball’ or attending a women’s college. In further experiments, they found that even without the word ‘women,’ the tool preferred language that men tended to use more, words like “executed” or “captured.”

The Amazon team trained the data using résumés from past hires, which disproportionately included men. Consequently, the ML tool learned to downgrade any indication of femininity on a resume. Amazon could not satisfactorily remove the tool’s gender bias and eventually gave up on the project. These examples show how biased or unrepresentative data can lead ML tools to reproduce social inequalities.

MISALIGNED DATA USE: Even with the best data available, it is still possible for ML to use good data in ways that perpetuate inequality. For example, a range of indicators that do not measure race per se may be effective proxies for race, including zip codes or school attended. If models for something like creditworthiness for mortgages use these variables, they will systematically elevate the risk score of people of colour relative to their financial qualification.

In her book Automating Inequality, Virginia Eubanks describes how a Pennsylvania county created an algorithm to predict child neglect and abuse based on certain family and household characteristics. The team that created the algorithm used a technique that dredged public data to find indicators correlated with a referral for a child to the Office of Children, Youth and Family or removal from their home. Some measures they used to indicate risk — including time on public benefits and receipt of nutrition assistance — are effectively proxies for poverty. They only had information about access to public services, so families with the resources to access the same services through private means, such as food banks or churches, appeared less risky.

In the quantitative methods classes we teach, we would describe the relationship between nutrition assistance and increased risk of abuse or neglect as ‘spurious’ because poverty is behind both the need for help getting food and the risk of not having enough of it. Not only would the Pennsylvania algorithm miss the question on spuriousness on our exam, but it also legitimizes heightened surveillance of a vulnerable population and increases the chance that children from low-income families will be separated from loving caregivers.

INEQUALITIES FROM ALGORITHM OPTIMIZATION: ML and AI tools can also reproduce inequalities because of what they are designed to accomplish, even with good data and appropriately applied measures. Latanya Sweeney’s research on Google ads provides a clear example. When users enter terms in Google’s search engine, its ad server supplies targeted ads that accompany the search results. Google is able to ‘target’ the ads based on information it has about the user who will see the ad, such as the terms they search and the information they have about all search users, including what ads people click when they search particular kinds of terms.

Selling targeted advertising is Google’s primary source of revenue, and at the time of Sweeney’s study, it only charged the advertiser when a user clicked on their ad. Sweeney found that when users performed Google searches for names that are more prominent among Black people, the ads that accompanied the results were more likely to suggest a person with that name has a criminal record and to place those ads more prominently. These ads included wording such as ‘Latanya Sweeney Arrested?’ The Google ad server was optimized to maximize ad clicks, which has the effect of reinforcing stereotypes around race and criminality.

Google could point to user behaviour and say the algorithm is simply responding to users. While that may be true, if they had designed the algorithm to be best at something other than maximizing ad clicks or to balance equity and ad clicks, the ads would be different. As if to demonstrate this point, all the Google searches we’ve tested for people’s names no longer return ads suggesting criminal records at all. We suspect Google made a change to continue maximizing ad clicks while reducing the appearance of bias. They may have done this by imposing constraints on the language that advertisers can use or by preventing ads of any kind from accompanying search results for people’s names.

TARGETING VULNERABLE POPULATIONS: Often, particularly for those of us who are middle class and comfortable with technology, we get to make decisions about which ML tools to use and when and how to use them. Other times, private firms and governments make those decisions, which can be particularly damaging for people living in poverty or under police suspicion. As Eubanks’s book makes clear, middle-class families would be unlikely to tolerate the kinds of invasion of privacy and suspicion that come with the digital systems used to administer services for the poor.

Elsewhere, Sarah Brayne’s study of the Los Angeles Police Department shows that police use a range of systems to collect licence-plate information about parked cars, people on the street in neighbourhoods they patrol and more. This kind of data collection allows the police to include those cars and individuals as potential matches for criminal activity in data-driven digital systems. Even being a bystander or having a name queried in the system before increases the level of suspicion of individuals in the database.

Authorities also routinely install sophisticated surveillance systems in public housing facilities, often with buy-in from residents, with the stated goal of stopping violent crime. The surveillance might help with violent crime, but as Douglas MacMillan at the Washington Post reports, authorities also use facial recognition to scan footage for violations such as overnight guests or banned individuals. Authorities can use that footage, even of residents committing minor infractions like spitting in the hallways, to support eviction cases.

As sociologists observing and using these technologies, we find them both exciting and terrifying. Clearly, machine learning is not the root cause of inequality, but as indicated herein, it can and often does encode and reproduce social inequalities. That’s why it is critical to understand these tools through a sociological lens to ensure that their widespread use does not simply exacerbate existing inequalities.

This article was originally published in the Spring 2024 issue of the Rotman School of Management and was adapted from a longer version that was published recently in the journal Contexts (Sage Publishing). Subscribe to the magazine now for the latest thinking on leadership and innovation.