lesson
50 min

How Bias Gets Into Models

Bias enters through who is in the data, what the labels mean, and which columns stand in for others. Learn to find it and measure it.

Objective

By the end of this lesson, you can name three ways bias enters a model and measure an outcome gap between groups.

Models learn the past

A model learns patterns from data about the past. If the past was unfair, the model learns that too, and then repeats it faster and at larger scale. Bias in AI is usually not a bug in the math. It is in the data and the choices around it.

Three ways bias gets in

  1. Who is in the data. If a group is missing or rare, the model learns little about them and makes more mistakes for them.
  2. What the label really measures. A label is the answer column. "Arrested" is not the same as "committed a crime." "Was hired" reflects past hiring decisions, not who could do the job.
  3. Proxies. A proxy is a column that stands in for something else. Zip code can stand in for race. Spending can stand in for need, unfairly, for people who were never able to spend.

A national example

Measure the gap

A simple first check is to compare outcome rates between groups. One rule of thumb from U.S. employment guidance is the four-fifths rule: if one group's selection rate is less than 80 percent of the highest group's rate, look closer.

Run it. These are made-up results from a screening model. Change Group B's approved number to 100 and run it again. Does the flag go away?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

A flag is a reason to investigate, not proof of unfairness. A ratio above 0.8 is not proof of fairness either. Also compare error rates by group (Module 11).

Removing the column is not enough

Deleting race or gender from the inputs does not remove bias if proxies remain. You often need those columns, kept out of the model's inputs, to measure whether results differ by group.

Quiz: Bias check

  1. 01In the health algorithm study, where was the main source of bias?

    Choose one.

  2. 02Why is deleting the race column not enough to remove bias?

    Choose one.

Go deeper: Consent as signal: the first-party data layer you own

Ask. Answer. Build together.