The importance of testing for Bias

In the context of AI-based systems, bias is a statistical measure of the distance between the outputs provided by the system and what are considered to be fair outputs — outputs that show no favouritism to a particular group. Inappropriate bias can be linked to attributes such as gender, race, ethnicity, sexual orientation, income level, and age.

Bias can be introduced into many types of AI systems. In expert systems, it is difficult to prevent the bias of the experts themselves from being embedded in the rules they define. But it is in machine learning systems — where algorithms are trained on collected data — that bias most commonly surfaces, and where the consequences are most widely discussed.

ML systems make decisions and predictions using algorithms trained on collected data. Both of these components can introduce bias into results

• Algorithmic bias occurs when the learning algorithm is incorrectly configured — for example, when it overvalues some data compared to others. This can be caused and managed through hyperparameter tuning of the ML algorithm.
• Sample bias occurs when the training data is not fully representative of the data space to which the ML model is applied. If a model is trained on data that does not reflect the real-world population it will serve, its outputs will reflect those gaps.

Inappropriate bias is most commonly caused by sample bias — but algorithmic bias can also be a contributing factor. A biased model is typically the product of data that is incomplete, unbalanced, unfair, lacking in diversity, or duplicated. As an example: if all medical data used to train a disease prediction model is gathered from subjects of one particular gender, the resulting model is likely to perform poorly for everyone else — unless it was only ever intended to serve that group.

Many AI systems are designed to assist human decision-making rather than replace it. But this creates a risk of its own: automation bias, also known as complacency bias. This is the tendency for humans to be too trusting of AI recommendations, and it takes two distinct forms.

• Accepting without questioning. The human accepts the system's recommendation and fails to consider inputs from other sources — including their own judgement. Research has shown that this form of automation bias typically reduces the quality of human decisions by around 5%, though the impact can be significantly greater depending on the system context. The automatic correction of typed text on mobile devices is a familiar everyday example — users frequently fail to notice when autocorrect has changed their meaning.
• Failing to monitor adequately. The human becomes so trusting of the system that they stop watching it carefully enough to catch failures. Semi-autonomous vehicles illustrate this well — as systems become more self-directing, the human occupant gradually pays less attention, potentially reaching a point where they cannot react appropriately when the system needs them to.

In both scenarios, testers need to understand how human decision-making may be compromised. Testing must cover not only the quality of the system's recommendations, but also the quality of the corresponding human input provided by representative users.

An ML system should be evaluated against the different forms of bias, with actions taken to remove any inappropriate bias identified. In some cases, this may involve deliberately introducing a positive bias to counteract an existing inappropriate one.

Testing with an independent dataset can often detect bias. However, identifying all the data that causes bias is challenging — ML algorithms can use combinations of seemingly unrelated features to produce unwanted bias that is not immediately obvious.

A structured approach to bias testing may include:

• Analysis during model training, evaluation, and tuning to identify whether algorithmic bias is present.
• Reviewing the source of training data and the processes used to acquire it, to identify the presence of sample bias.
• Reviewing data pre-processing steps within the ML workflow to identify whether the data has been affected in ways that could introduce sample bias.
• Measuring how changes in system inputs affect outputs across a large number of interactions, and examining results based on the groups of people or objects the system may be biased towards or against. This approach is similar to the LIME (Local Interpretable Model-Agnostic Explanations) method, and can be applied both in pre-release testing and in production environments.
• Obtaining additional information about input data attributes potentially related to bias — such as demographic data — and correlating this to system outputs. This is particularly relevant when testing for bias that affects groups of people, where group membership is relevant to assessing bias but is not itself an input to the model. Bias can be based on hidden variables that are not explicitly present in the input data but are inferred by the algorithm.

Where systems are likely to be affected by bias, testing should use an independent bias-free test suite, or be validated by expert reviewers who can assess outputs against a known fair baseline.

External data — such as census data — can be used to compare test results and check for unwanted bias on inferred variables. This is known as external validity testing, and it is one of the most reliable methods for surfacing bias that exists below the surface of a model's inputs.