Teaching Experiments
Teach a team to write a one-page test, run it fairly, and accept the result even when it surprises them.
Objective
By the end of this lesson, you can teach a team to write a one-page experiment and compare two results with simple math.
Why teams need experiments
An experiment is a planned test. You change one thing, keep everything else the same, and see what happens. It replaces "I think" with "we checked."
Without experiments, the loudest voice in the meeting wins. With them, the evidence wins.
A national example: the favorite lost
During the 2008 U.S. presidential campaign, the Obama campaign tested different versions of its sign-up page. They tried different buttons and images, and showed each version to a share of visitors. Staff had a favorite version, a video. It did not win. A simpler version got more sign-ups.
The lesson for trainers: your team's favorite idea is a guess until it is tested. Teach people to expect surprises.
The one-page test card
Teach every test on one page. If it does not fit, the test is too big.
| Part | Example |
|---|---|
| Question | Will a shorter sign-up form get more families to finish? |
| Change | Form A has 9 questions. Form B has 4. |
| Measure | Share of people who start the form and finish it |
| Time | Two weeks, or until each form has at least 100 starts |
| Decision rule | If B wins clearly, we keep B. If not, we keep A. |
| Who is affected | Families signing up. Check that B still collects what staff need. |
The decision rule matters most. Write it before the test starts. Otherwise people move the goalposts after they see the result.
The math: conversion rate
A conversion rate is the share of people who did the thing you wanted. Divide finishes by starts.
Run the code below. Then change the numbers and run it again. The first run downloads a small Python program, which can take a minute on phone data.
starts_a = 200
finishes_a = 50
starts_b = 180
finishes_b = 63
rate_a = finishes_a / starts_a * 100
rate_b = finishes_b / starts_b * 100
print("Form A:", round(rate_a, 1), "%")
print("Form B:", round(rate_b, 1), "%")
Judgment: when not to test
Testing is not always right. Teach your team to skip a test when:
- The numbers are too small. If only 12 people a month see the page, a test will not tell you much. Ask people directly instead.
- One version could harm people. Never test whether removing safety, privacy, or access information changes results.
- The answer is already clear. If the form is broken on phones, fix it. Do not test it.
- Youth are involved. Testing messages on minors needs extra care and a parent or guardian's knowledge. Keep tests on adults unless your organization has clear rules.
Quiz: Check your understanding
Go deeper: Performance Marketing: Stop Guessing, Start Printing
Discussion
Ask. Answer. Build together.
Loading…