Today’s powerful AI systems make a lot of important decisions: in content moderation, sentencing recommendations, loan applications, and much more. Essentially, AI systems are increasingly called on to apply what we might call “normative” judgments to real-world facts on behalf of human decision-makers.

By “normative,” we mean a judgment about what should or shouldn’t be done. Machines of course already make factual decisions, but in order to be effective and fair decision-makers, they need to make normative decisions much like human beings do.

With so much at stake, it’s no surprise that there is a great deal of concern about calibrating machine behaviour to human norms. So, how can we train machines to make normative decisions in the same way that people do?

A new paper by U of T alumna Aparna Balagopalan (MIT) and co-authors David Madras (University of Toronto), David H. Yang (University of Toronto), Dylan Hadfield-Menell (MIT), and Marzyeh Ghassemi (MIT) offers evidence on the relationship between the methods used to label the data that trains a type of AI called machine learning (ML) models and the performance of those models when they are applying normative judgments.

Their paper, “Judging Facts, Judging Norms: Training Machine Learning Models to Judge Humans Requires a Modified Approach to Labeling Data,” challenge conventional wisdom on human-computer interaction and reducing bias in AI.

We know that an ML model can predict facts and apply logic to those facts to come up with a normative judgment. For example, if a person has not paid their debts in x amount of time, then they should not be given a loan. But ML models are often trained on data that is "labelled" (i.e. marked up) by human beings. So, the researchers set out to study a crucial new question: Can standard data labelling practices for machine learning with descriptive or factual questions be used to train automated decision systems to judge humans?

The team created a series of four experiments, with two types of labelling exercises: normative and factual. They enlisted 20 labellers per item, used 2,000 items per dataset, and worked with four different datasets.

For example, one group of participants was asked to label dogs that exhibited certain characteristics (namely: those that were “large sized … not well groomed … [or] aggressive.”) Meanwhile, another group of participants was instead asked whether or not the dogs shown to them were fit to be in an apartment, rather than assessing the presence or absence of specific features.

The first group was asked to make a factual assessment, and the second, a normative one.

The results?

“We were surprised,” says Hadfield. “When you ask people a normative question, they answer it differently than when you ask them a factual question.”

Human participants in the experiments were more likely to recognize (and label) a factual feature than the violation of an explicit rule predicated on the factual feature.

The researchers took some steps to verify their conclusions relative to potential confounding factors:

• They confirmed that the differences between “descriptive” (i.e. factual) and “normative” labels persisted for items subject to considerable disagreement (i.e. controversial edge cases). This shows that AI taught to reason about rule-adherence from factual data will deviate considerably from human expectations in those contentious contexts where we should most hope to rely on it.

• The researchers also assessed whether giving human participants additional context might lead the “descriptive” labellers to mark items more consistently with the “normative” labellers, but this was shown not to be the case.

Current thinking on this topic presumes that calibrating AI behaviour to human conventions requires value-neutral, observational data from which AI can best reason toward sound normative conclusions.

But this new research suggests that labelling data with labels that explicitly reflect value-judgments, rather than the facts used to reach those judgments, might yield ML models that assess rule-adherence and rule-violation in a manner that we humans would deem acceptable.

The results of these experiments showed that ML models trained on normative labels achieve higher accuracy in predicting human normative judgments. Essentially, they are better at predicting. Therefore, if we train automated judgment systems on factual labels — which is how several existing systems are being built — they are likely overpredicting rule violations.

The implications of this research are significant. Not only does it show that reasoning about norms is qualitatively different from reasoning about facts, it also has important real-world ramifications.

“People could say ‘I don’t want to be judged by a machine, I want to be judged by a human, given that we’ve got evidence to show that the machine will not judge them properly” says Hadfield. “Our research shows that this factor has a bigger effect [on an ML model’s performance] than things like model architecture, label noise, and subsampling—factors that are often looked to for errors in prediction."

While the researchers have shown that ML models trained on normative labels are better at making predictions, actually training ML models like this would significantly increase the cost of developing them.

“Getting normative labels means you have to actually pay a lot of attention to how you solicited those labels, who you solicited them from, etc.,” says Hadfield. “You want to be doing careful design of that. We need to train on and evaluate normative labels. We have to pay the money for normative labels, and probably for specific applications. We should be a lot better at documenting that labelling practice. Otherwise, it’s not a fair judgment system. There’s a ton more research we need to be doing on this."

This article was originally published by the Schwartz Reisman Institute for Technology and Society.