Analysing and acting on results

The value of experimentation isn't just the winning variant. It's the learning you extract and apply systematically across your operations. Most companies run a test, implement the winner, forget about it. They don't document why it worked, don't apply the pattern elsewhere, don't build institutional knowledge.

This chapter shows you how to document experiment learnings in a way that builds knowledge over time, extract patterns from multiple experiments, update playbooks and templates with proven approaches, and build a culture where experimentation is continuous.

Create an experiment log documenting every test, whether it wins or loses. Structure: experiment ID, date, hypothesis (prediction and mechanism), test design (what you changed, control versus variant), results (primary metric, secondary metrics, segments), confidence level, decision (implement, partial implement, don't implement, test again), and learnings (why it worked or didn't, what this tells us about segment beliefs).

Example experiment log entry:

Experiment ID: LP-003
Date: January 2025
Hypothesis: Compliance-driven segment doubts that engaging training satisfies auditors. Adding testimonials from compliance officers will reduce doubt and improve lead conversion from 4% to 5% via social proof mechanism.
Test design: Landing page for compliance segment. Control headline + generic features. Variant headline + 3 testimonial quotes from CISOs at similar companies emphasising auditor acceptance.
Results: Overall conversion 4% → 4.6% (+15%, 97% confidence). Segment breakdown: compliance-driven 4% → 5.2% (+30%, 99% confidence). Proactive 4.5% → 4.4% (-2%, not significant). Exit survey: 45% of variant converters mentioned "others like us use it" versus 18% of control converters.
Decision: Implement for compliance-driven traffic only. Keep testing for proactive segment.
Learnings: Social proof works strongly for risk-averse compliance segment. Testimonials from peers reduce "will this satisfy our auditors?" concern. Mechanism validated. Pattern to apply: use peer testimonials for compliance segment, use data for proactive segment.

This structure captures not just results but understanding. Six months later, when designing campaigns for compliance-driven segment, you consult the log and know: peer testimonials work, emphasise auditor acceptance, use CISOs from similar companies.

After running 10-20 experiments, patterns emerge. Outcomes that seem random in individual tests reveal clear patterns across multiple tests.

Pattern example 1: Tests LP-003, LP-007, AD-012 all show compliance-driven segment responds to social proof (testimonials, customer counts, named companies) better than data (metrics, behaviour tracking, ROI stats). Tests LP-005, LP-009, AD-015 all show proactive segment responds to data better than social proof. Pattern: compliance needs peer validation, proactive needs quantitative proof. This becomes a design principle: when creating anything for compliance segment, lead with social proof. For proactive segment, lead with data.

Pattern example 2: Tests AD-004, AD-008, AD-013 all show outcome-focused headlines ("Train your team in 30 minutes") outperform pain-focused headlines ("Stop wasting time on ineffective training") for cold traffic, but pain-focused headlines outperform for warm traffic (people who've visited the site before). Pattern: cold traffic responds to positive framing (what they'll achieve), warm traffic responds to negative framing (what they'll avoid). Design principle: use outcome headlines for acquisition campaigns, pain headlines for remarketing campaigns.

Pattern example 3: Tests LP-006, LP-011, SP-003 (sales process test) all show that emphasising speed ("30-minute setup", "deployed today") improves conversion for SMB buyers but has no effect or slight negative effect for enterprise buyers. Pattern: SMB values speed and simplicity, enterprise values thoroughness and support. Design principle: emphasise speed for SMB campaigns, emphasise support and customisation for enterprise campaigns.

Build a pattern library: document these cross-experiment patterns with supporting evidence (which experiments proved it), confidence level (how certain are we), and application guidance (when to use this pattern). This library becomes your institutional knowledge.

Don't let learnings sit in a document. Update your operational playbooks and templates so everyone uses proven approaches by default.

Update ad creative templates: After learning outcome headlines outperform pain headlines for compliance segment, update your ad creative template for compliance campaigns. Template now includes: "Use outcome-focused headline emphasising speed (reference experiment AD-012, LP-003). Example: 'Complete SOC 2 training in 30 minutes'. Avoid pain-focused headlines for cold traffic (reference experiment AD-008)."

Now when anyone creates ads for compliance segment, they start with the proven pattern. They're not re-testing outcome versus pain headlines, they're applying the validated learning.

Update landing page playbooks: After learning peer testimonials work for compliance segment, update landing page playbook. Guidance: "Compliance-driven segment: Include 2-3 testimonials from CISOs at similar-size companies. Emphasise auditor acceptance in quotes. Placement: above the fold, near headline. Reference: experiment LP-003 showed +30% conversion lift."

Update sales playbooks: After experiment SP-003 shows offering free pilot improves SQL → opportunity conversion, update sales playbook. Guidance: "When SQL hesitates due to implementation concerns, offer free pilot: 3 users, 30 days, no IT involvement required. 67% of SQLs offered pilot become opportunities versus 33% without pilot (reference: experiment SP-003). Pilot offer script: [template language]"

The goal: proven approaches become default behaviour, not special knowledge held by one person. Anyone running campaigns for compliance segment knows to use outcome headlines and peer testimonials. Anyone doing sales calls knows to offer pilots when prospects hesitate about implementation.

Experimentation shouldn't be a one-off project ("we'll do a test quarter"). It should be continuous. Always have 1-2 tests running.

Quarterly experiment planning: At the start of each quarter, review your dashboard for current bottlenecks, consult customer research for new insights, build experiment backlog prioritised by impact. Plan 4-6 experiments for the quarter (1-2 at a time, running sequentially or in parallel if traffic permits).

Weekly experiment reviews: Every week, review active experiments for implementation issues (not results, just health checks). Every completed experiment triggers an experiment review meeting: review results, discuss learnings, decide on rollout, add learnings to experiment log, identify follow-up experiments.

Monthly pattern review: Once a month, review experiment log looking for patterns across tests. Update pattern library. Update playbooks with new learnings. Share insights across team (marketing, sales, product).

Build an experimentation backlog: Maintain a prioritised list of experiment ideas with impact scores. As you complete experiments, pull the next highest-priority idea. The backlog never empties (as you learn from experiments, new questions emerge).

Celebrate learning, not just winning: In team meetings, celebrate both winning and losing experiments. "This test failed but we learned compliance segment doesn't care about behaviour metrics" is valuable. Don't create a culture where only wins are shared (this leads to cherry-picking and hiding losses). Losing experiments that teach you something are valuable.

Compound learning: Each experiment informs the next. You learn compliance segment responds to social proof, so next experiment tests which type of social proof works best (testimonials versus customer counts versus named logos). You learn outcome headlines beat pain headlines for cold traffic, so next experiment tests which outcome promise is most compelling (speed versus cost savings versus risk reduction). Each test narrows the possibilities and builds more precise knowledge.

Document every experiment systematically in an experiment log: hypothesis, test design, results by segment, decision, and learnings. This builds institutional knowledge over time.

Extract patterns from multiple experiments. Individual tests show local learning. Patterns across 10-20 tests reveal design principles that apply broadly. Build a pattern library documenting cross-experiment insights.

Update playbooks and templates with proven approaches. Make validated learnings the default behaviour, not special knowledge. Anyone creating campaigns for a segment should automatically apply proven patterns.

Build experimentation culture where testing is continuous. Plan experiments quarterly, review weekly, extract patterns monthly. Maintain prioritised backlog. Celebrate learning from both wins and losses. Each experiment informs the next, building compound learning over time.

With experimentation systematic, you're ready to automate repetitive processes. That's the next playbook.