lesson
35 min

The Human in the Loop

Design the review step: which rows go straight through, which a person checks, and which are skipped.

What you will be able to do

Design a three-lane routing rule (send, review, skip) and explain what the reviewer checks.

Where the person sits

"Human in the loop" means a person makes or checks a decision inside an automated process. It does not mean a person clicks "approve" on everything without reading. That is a rubber stamp, not a review.

A good loop puts the person where their judgment matters most and their time is used well.

Three lanes

Route every scored row into one of three lanes:

LaneRuleWhat happens
Review firstScore at or above your thresholdA person checks the facts and the draft before anything sends
HoldScore in a middle band, or a key fact is missingA person looks when there is time; nothing sends automatically
SkipScore below the band, or on the not-a-fit listNo message. Logged so you can audit it later

For a new machine, keep the person on every high-score row. Once the reviewer agrees with the AI almost every time for several weeks, you can let a sample pass straight through and keep checking a share by hand. Never remove the review entirely.

Route the rows

The code below sorts made-up scored rows into the three lanes. Change the threshold and run it again. Watch how many rows move.

If you are on phone data, note that the first run downloads the Python runtime (several MB). You can read the code here first:

threshold = 70
hold_floor = 50

rows = [
    {"org": "Lakeview School District", "score": 88, "missing_fact": False, "not_a_fit": False},
    {"org": "Northside Library", "score": 72, "missing_fact": True, "not_a_fit": False},
    {"org": "Hillcrest Youth Center", "score": 64, "missing_fact": False, "not_a_fit": False},
    {"org": "Summit Prep Academy", "score": 91, "missing_fact": False, "not_a_fit": True},
    {"org": "Eastgate Schools", "score": 41, "missing_fact": False, "not_a_fit": False},
]

def lane(row):
    if row["not_a_fit"]:
        return "skip"
    if row["missing_fact"]:
        return "hold"
    if row["score"] >= threshold:
        return "review first"
    if row["score"] >= hold_floor:
        return "hold"
    return "skip"

for row in rows:
    print(row["org"], "->", lane(row))
Run the routing rule. Then change threshold to 60 and run it again. How many rows moved lanes?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

What the reviewer checks

The reviewer reads the evidence, not just the number:

  • Is every fact true? Open the source link.
  • Does the score match the rubric?
  • Is anything about a private person or a minor? If so, remove it and flag the row.
  • Would I be comfortable if this organization saw exactly why we chose them?

Keep a disagreement log

When the reviewer disagrees with the AI, write it down: the row, the AI's score, the reviewer's score, and why. Read the log every week. Patterns show you where the rubric or the prompt is weak. This is how the machine gets better without getting riskier.

Reviewer gate

0 of 6 done

Live session

Go deeper: Staff Training: AI Automation for GTM - From Playbooks to Production in 8 Weeks

Ask. Answer. Build together.