lesson
25 min

Teaching Experiments

Teach a team to write a one-page test, run it fairly, and accept the result even when it surprises them.

Objective

By the end of this lesson, you can teach a team to write a one-page experiment and compare two results with simple math.

Why teams need experiments

An experiment is a planned test. You change one thing, keep everything else the same, and see what happens. It replaces "I think" with "we checked."

Without experiments, the loudest voice in the meeting wins. With them, the evidence wins.

A national example: the favorite lost

During the 2008 U.S. presidential campaign, the Obama campaign tested different versions of its sign-up page. They tried different buttons and images, and showed each version to a share of visitors. Staff had a favorite version, a video. It did not win. A simpler version got more sign-ups.

The lesson for trainers: your team's favorite idea is a guess until it is tested. Teach people to expect surprises.

The one-page test card

Teach every test on one page. If it does not fit, the test is too big.

PartExample
QuestionWill a shorter sign-up form get more families to finish?
ChangeForm A has 9 questions. Form B has 4.
MeasureShare of people who start the form and finish it
TimeTwo weeks, or until each form has at least 100 starts
Decision ruleIf B wins clearly, we keep B. If not, we keep A.
Who is affectedFamilies signing up. Check that B still collects what staff need.

The decision rule matters most. Write it before the test starts. Otherwise people move the goalposts after they see the result.

The math: conversion rate

A conversion rate is the share of people who did the thing you wanted. Divide finishes by starts.

Run the code below. Then change the numbers and run it again. The first run downloads a small Python program, which can take a minute on phone data.

starts_a = 200
finishes_a = 50
starts_b = 180
finishes_b = 63
rate_a = finishes_a / starts_a * 100
rate_b = finishes_b / starts_b * 100
print("Form A:", round(rate_a, 1), "%")
print("Form B:", round(rate_b, 1), "%")
Run the code to compare two sign-up forms. Then change the numbers to test your own idea.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

Judgment: when not to test

Testing is not always right. Teach your team to skip a test when:

  • The numbers are too small. If only 12 people a month see the page, a test will not tell you much. Ask people directly instead.
  • One version could harm people. Never test whether removing safety, privacy, or access information changes results.
  • The answer is already clear. If the form is broken on phones, fix it. Do not test it.
  • Youth are involved. Testing messages on minors needs extra care and a parent or guardian's knowledge. Keep tests on adults unless your organization has clear rules.

Quiz: Check your understanding

  1. 01When should a team write its decision rule?

    Choose one.

  2. 02Form A had 40 finishes from 160 starts. What is its conversion rate?

    Choose one.

Go deeper: Performance Marketing: Stop Guessing, Start Printing

Ask. Answer. Build together.