Cleaning Messy Data
Fix wrong types, handle missing values on purpose, and remove duplicates without hiding problems.
Objective
By the end of this lesson, you can convert text to numbers, decide how to handle missing values, and explain the choice you made.
Real data is messy
Public data has typos, blanks, numbers stored as text, and repeated rows. Data scientists often say most of the work is cleaning. Cleaning is not busywork. Every cleaning choice changes what your model learns.
Missing values: three choices
When a value is missing, you can:
| Choice | What it does | Risk |
|---|---|---|
| Drop the row | Removes it | You may drop whole groups, like rural counties with fewer reports |
| Fill with a typical value | Uses the median or mean | Makes data look more certain than it is |
| Flag it | Adds a column saying "this was missing" | More columns, but the gap stays visible |
The median is the middle value when sorted. It is often safer than the mean for filling, because one huge value does not pull it.
The same steps in pandas
Run this in Colab on your own CSV:
import pandas as pd
df = pd.read_csv("your_file.csv")
# 1. Fix a number column stored as text (bad values become NaN, meaning missing)
df["meals_served"] = pd.to_numeric(df["meals_served"], errors="coerce")
# 2. Flag missing values before you fill them
df["meals_missing"] = df["meals_served"].isna()
# 3. Fill with the median
df["meals_served"] = df["meals_served"].fillna(df["meals_served"].median())
# 4. Remove exact duplicate rows
before = len(df)
df = df.drop_duplicates()
print("Removed", before - len(df), "duplicates")
A national example
Judgment check
Write down every cleaning choice. Someone else should be able to repeat it and question it. If your cleaning drops a group of people or places, say so in your results.
Cleaning log
0 of 5 doneDiscussion
Ask. Answer. Build together.
Loading…