Running experiments

Execute tests with proper controls. Avoid peeking at results early. Monitor for external factors that could invalidate your test.

Introduction

You can't run every experiment at once. Limited time, budget, and traffic force trade-offs. Prioritisation frameworks like ICE (Impact, Confidence, Ease) and PIE (Potential, Importance, Ease) score experiments objectively so you're not guessing. This removes politics and gut feel from the decision. The highest-scoring experiments get resourced first. This chapter shows you how to score experiments consistently and build a prioritised testing queue.

Setting up the test

Avoiding common mistakes

Monitoring during the test

Update priorities as you learn from tests

Conclusion

Related tools

VWO

Rating

Rating

Rating

Rating

Rating

From

393

per month

VWO

VWO provides A/B testing, personalisation, and behaviour analytics to optimise website conversion rates through data-driven experimentation.

Hotjar

Rating

Rating

Rating

Rating

Rating

From

39

per month

Hotjar

Hotjar captures user behaviour through heatmaps, session recordings, and feedback polls to understand how visitors use your website.

Microsoft Clarity

Rating

Rating

Rating

Rating

Rating

From

0

per month

Microsoft Clarity

Microsoft Clarity provides free session recordings, heatmaps, and user behaviour analytics without traffic limits or time restrictions.

Notion

Rating

Rating

Rating

Rating

Rating

From

12

per month

Notion

Flexible workspace for docs, wikis, and lightweight databases ideal when you need custom systems without heavy project management overhead.

Related wiki articles

A/B testing

Compare two versions of a page, email, or feature to determine which performs better using statistical methods that isolate the impact of specific changes.

Hypothesis testing

Structure experiments around clear predictions to focus efforts on learning rather than random changes and make results easier to interpret afterward.

Prioritisation

Systematically rank projects and opportunities using objective frameworks, ensuring scarce resources flow to highest-impact work.

Constraint

Identify and leverage limitations as forcing functions that drive creative problem-solving and strategic focus.

Further reading

Experimentation

Experimentation

Execute tests with proper controls. Avoid peeking at results early. Monitor for external factors that could invalidate your test.