A/B Test Design Document

Design a rigorous A/B test for the following change: [CHANGE_DESCRIPTION] on [PRODUCT_OR_PAGE]. The test document must include: (1) Hypothesis — written in the format: "We believe that [CHANGE] will cause [METRIC] to [INCREASE/DECREASE] because [RATIONALE]," (2) Primary metric — the single metric that will determine the winner, and why it was chosen over alternatives, (3) Secondary and guardrail metrics — metrics to monitor for unintended effects, (4) Sample size calculation — using a baseline conversion rate of [BASELINE_RATE]%, minimum detectable effect of [MDE]%, confidence level of [CONFIDENCE — e.g., 95%], and power of [POWER — e.g., 80%]: calculate the required sample size per variant, (5) Test duration — based on current traffic of [DAILY_TRAFFIC] sessions/day, calculate the minimum test runtime and explain why stopping early is dangerous, (6) Segmentation — which user segments to include/exclude and why, (7) Rollout plan — traffic split ([CONTROL]% control vs. [VARIANT]% variant) and how it will be implemented, (8) Decision criteria — exact rules for calling a winner, a loser, or extending the test.