lesson
45 min

Grouping and Summaries

Split a table into groups, summarize each group, and compare them fairly.

Objective

By the end of this lesson, you can group rows by a column, compute a summary for each group, and compare groups using rates instead of raw counts.

Split, apply, combine

Grouping means three steps:

  1. Split the rows into groups (for example, by state).
  2. Apply a summary to each group (sum, mean, count).
  3. Combine the answers into a new, smaller table.

Here is the idea in plain Python with a dictionary:

Run it. Each row is a made-up county with its state, population, and number of households without internet. The code totals each state. Then compute the rate (no_internet divided by households) for each state.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

Counts versus rates

State BB has more households without internet (11,000 versus 4,500). But BB is also much bigger. The rate tells a different story: AA is 4,500 of 25,000 (18 percent), and BB is 11,000 of 100,000 (11 percent).

Raw counts mostly measure size. To compare places of different sizes, use rates or per-person numbers.

The same thing in pandas

Run this in Google Colab:

import pandas as pd

df = pd.DataFrame({
    "state": ["AA", "AA", "BB", "BB"],
    "households": [20000, 5000, 90000, 10000],
    "no_internet": [3000, 1500, 9000, 2000],
})

by_state = df.groupby("state")[["households", "no_internet"]].sum()
by_state["rate"] = by_state["no_internet"] / by_state["households"]
print(by_state.sort_values("rate", ascending=False))

A national example

Quiz: Grouping check

  1. 01What are the three steps of grouping?

    Choose one.

  2. 02Why compare rates instead of raw counts across states?

    Choose one.

Ask. Answer. Build together.