Experimentation

Calculate how many users you need in experiments to detect meaningful differences and avoid declaring winners prematurely based on insufficient data.

Interpret experiment results to understand the probability that observed differences occurred by chance rather than because your changes actually work.

Design experiments that answer specific questions with minimum time and resources to maximise learning velocity without over-investing in unproven ideas.

Structure experiments around clear predictions to focus efforts on learning rather than random changes and make results easier to interpret afterward.

Maintain an unchanged version in experiments to isolate the impact of your changes and prove causation rather than correlation with external factors.

Determine whether experiment results reflect real differences or random chance to avoid making expensive decisions based on noise instead of signal.

Compare two versions of a page, email, or feature to determine which performs better using statistical methods that isolate the impact of specific changes.

Deploy fast, low-cost experiments to discover scalable acquisition and retention tactics, learning through iteration rather than big bets.