lesson
50 min

Cleaning Messy Data

Fix wrong types, handle missing values on purpose, and remove duplicates without hiding problems.

Objective

By the end of this lesson, you can convert text to numbers, decide how to handle missing values, and explain the choice you made.

Real data is messy

Public data has typos, blanks, numbers stored as text, and repeated rows. Data scientists often say most of the work is cleaning. Cleaning is not busywork. Every cleaning choice changes what your model learns.

Missing values: three choices

When a value is missing, you can:

ChoiceWhat it doesRisk
Drop the rowRemoves itYou may drop whole groups, like rural counties with fewer reports
Fill with a typical valueUses the median or meanMakes data look more certain than it is
Flag itAdds a column saying "this was missing"More columns, but the gap stays visible

The median is the middle value when sorted. It is often safer than the mean for filling, because one huge value does not pull it.

Run it. The code turns text into numbers, skips blanks, and fills them with the median. Then change one value to 9000 and see how the mean moves but the median barely does.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The same steps in pandas

Run this in Colab on your own CSV:

import pandas as pd

df = pd.read_csv("your_file.csv")

# 1. Fix a number column stored as text (bad values become NaN, meaning missing)
df["meals_served"] = pd.to_numeric(df["meals_served"], errors="coerce")

# 2. Flag missing values before you fill them
df["meals_missing"] = df["meals_served"].isna()

# 3. Fill with the median
df["meals_served"] = df["meals_served"].fillna(df["meals_served"].median())

# 4. Remove exact duplicate rows
before = len(df)
df = df.drop_duplicates()
print("Removed", before - len(df), "duplicates")

A national example

Judgment check

Write down every cleaning choice. Someone else should be able to repeat it and question it. If your cleaning drops a group of people or places, say so in your results.

Cleaning log

0 of 5 done

Ask. Answer. Build together.