How long should a cohort test run?

2 to 4 weeks. Shorter than 2 weeks misses the back-half-of-cadence reply lag. Longer than 4 weeks introduces too much external-event noise.

What sample size do I need?

For positive reply rate at 6% baseline, detecting a 20% lift with 95% confidence requires ~1,000 prospects per cohort. Meeting rate tests need 3x to 4x more sample.

Can I test multiple variables at once?

No. Multi-variable tests produce results nobody can attribute. The discipline of one variable per test is structurally important.

What if the test produces a null result?

Null results are useful. A test that says the variable does not move the metric saves the team from rebuilding around something with no leverage.

How to Run Cohort Tests on Outreach (So You Stop Random Tweaks)

Cohort testing replaces opinion-based tweaks with structured 2-week experiments. Two matched cohorts, one primary metric, 20% minimum lift, 95% confidence.

Test Design Rules

One variable per test, random assignment within persona, equal cohort sizes, same time window, pre-registered hypothesis and threshold, 20% minimum lift.

Sample Size

Positive reply rate 6% with 20% lift = 1,000 per cohort. Meeting rate 2% with 20% lift = 3,000 per cohort. Underpowered tests produce false positives.

How A Test Works

Pick one variable, build matched cohorts, run in parallel for 2 to 4 weeks, hold variants stable, measure primary metric, declare winner or null result.

Common Mistakes

Multi-variable tests, mid-stream tweaks, underpowered samples, calling the winner early, optimizing on a secondary metric, generalizing too fast.