A/B Test Design Document

A statistically rigorous A/B test design with sample size math, guardrail metrics, and clear decision criteria.

Best For

•A CRO specialist calculating required sample size before launching a checkout page test, to avoid calling a winner too early
•A product manager writing the hypothesis and guardrail metrics for a pricing page experiment before it goes to engineering
•A growth team lead documenting rollout and decision criteria so a test isn't stopped the moment it looks statistically significant

Prompt Template

Design a rigorous A/B test for the following change: [CHANGE_DESCRIPTION] on [PRODUCT_OR_PAGE]. The test document must include: (1) Hypothesis, written in the format: "We believe that [CHANGE] will cause [METRIC] to [INCREASE/DECREASE] because [RATIONALE]," (2) Primary metric, the single metric that will determine the winner, and why it was chosen over alternatives, (3) Secondary and guardrail metrics, metrics to monitor for unintended effects, (4) Sample size calculation, using a baseline conversion rate of [BASELINE_RATE]%, minimum detectable effect of [MDE]%, confidence level of [CONFIDENCE, e.g., 95%], and power of [POWER, e.g., 80%]: calculate the required sample size per variant, (5) Test duration, based on current traffic of [DAILY_TRAFFIC] sessions/day, calculate the minimum test runtime and explain why stopping early is dangerous, (6) Segmentation, which user segments to include/exclude and why, (7) Rollout plan, traffic split ([CONTROL]% control vs. [VARIANT]% variant) and how it will be implemented, (8) Decision criteria, exact rules for calling a winner, a loser, or extending the test.

Pro Tip

Push for the guardrail metrics section even when the test seems low-risk, since a variant that lifts the primary metric while quietly increasing refund rate or support tickets is the most common reason tests get reversed post-launch.

Example Output

A sample of what this prompt produces once you fill in the placeholders.

Hypothesis: adding a one-click checkout button will increase completed-purchase rate because it removes a form-filling step. Baseline conversion: 3.2%, MDE: 15%, 95% confidence, 80% power, requiring approximately 18,400 sessions per variant. At 2,600 daily sessions, minimum runtime is 15 days across two full weekly cycles.