A Journey Experiment Improves One Group and Harms Another

What the journey problem looks like and why it affects the business

A journey experiment can improve the headline outcome while shifting effort, exclusion or harm to a group that is small, less visible or poorly represented in the primary metric. A statistically tidy result does not authorise scale when affected users cannot complete the task or recover from the change.

Bound the experiment by population, route and decision, then test inclusion, harm, support and rollback guardrails alongside the intended outcome.

This guide is for product leads, experimentation leads or accessibility leads. Budget approval sits with the product, experimentation or accessibility executive accountable for scale decisions, user trust and exposure created by uneven journey outcomes.

The named owner holds decision authority; specialists contribute evidence within their remit, but contribution does not replace accountable ownership.

Check one experiment and its affected users before deciding whether to scale. This is not experimentation-platform design, statistical certification or automatic scale approval.

What to expect in the working pack

Start the working pack with the charter. Use stable IDs to connect sources, tests, decisions and review records across its eight sheets. Entries need supporting evidence; the workbook alone cannot prove a conclusion:

• Start here: Read the bounded decision, evidence labels, sequence and stop conditions before changing the journey.
• Journey charter: Define the user outcome, start and end, routes, exclusions, consequence and authority.
• Journey map: Connect user tasks, context, channels, systems, handoffs, owners and unresolved questions.
• Evidence register: Separate observed facts and statements from interpretations, inferences, proposals and unknowns.
• Friction and failure log: Record symptoms, affected routes, severity, recurrence, consequences and competing causes.
• Intervention and acceptance: Link a proposed change to its owner, dependencies, acceptance, guardrails and rollback.
• Outcome and recovery: Observe progression, value, support, recovery, exceptions and harm across the agreed window.
• Review and change log: Record versions, evidence changes, approvals, review triggers, replacement and retirement.

Use the same IDs across the workbook so a reviewer can move from a claim to its source, consequence, proposed change, outcome and decision. The guide-specific seed rows are prompts, not local evidence; replace them with authorised organisation-specific material. Do not enter participant names, contact details, recordings, free-text case notes or unnecessary personal information. Store sensitive material in an approved controlled repository and record only a permitted reference. The workbook does not create consent, lawful authority or publication approval.

What evidence distinguishes the real cause

The visible symptom does not establish the cause. Before redrawing the journey or commissioning an intervention, build a dated evidence set that distinguishes observed behaviour from interpretation.

Label every material statement. Use VERIFIED only when a current attributable source supports the statement and permitted use. Use OBSERVED for an event or participant statement without turning it into a population claim. Use INFERRED where evidence supports but does not prove an explanation. Use PROPOSED for a future design or decision. Use UNKNOWN when evidence is missing. Use REQUIRES VALIDATION where testing is required before investment, public copy or scale.

Test three competing explanations: The primary metric hides a material negative effect in an under-represented route or group. The treatment changes support burden or recovery effort without appearing in the success metric. The experiment is scaled before a meaningful rollback and human review path exists. Record confirming and contrary cases with equal care. A completion event, dashboard, map or consensus can indicate where to investigate, but none proves why value is delayed, lost or uneven.

The research frame should cover the whole user problem and the routes people actually use. GOV.UK Service Standard: Make sure everyone can use the service and GOV.UK Service Manual: User research support end-to-end research and joined-up channel thinking; GOV.UK Service Standard: Solve a whole problem for users provides a user-needs frame where available. Preserve interfaces across teams, channels and dependencies instead of hiding them in a single stage. For timing and outcomes, define the observation window and reconcile timestamps with experienced effort.

Accessibility is part of the journey outcome and acceptance test, not a final compliance tick. The Web Content Accessibility Guidelines 2.2 include requirements relevant to complete processes, errors, interruptions, input assistance, redundant entry and accessible authentication. Test with disabled users and people who need assistance when this decision may affect them. Do not infer universal access from an automated scan alone.

Personal information must follow the organisation's approved privacy and research process. POPIA, the Protection of Personal Information Act (https://www.gov.za/documents/protection-personal-information-act), is the primary South African statutory source. This guide is not legal advice; the Information Officer or qualified adviser retains legal authority. Record purpose, permitted use, access, retention and deletion arrangements before collecting identifiable evidence. Minimise workbook content and use controlled references.

Stop and re-charter when the decision is unnamed, the population cannot be bounded, the outcome is only a proxy, evidence cannot be used lawfully, assisted or excluded users cannot be represented, the owner lacks authority, or the work crosses the scope boundary. Record UNKNOWN when evidence is missing.

How to fix it step by step

Step 1: Define the journey decision

Open Start here and Journey charter. Write the decision as a choice an authorised owner must make, not an ambition such as improve onboarding. Define the user outcome in observable terms: what can the user do, under which conditions, and what counts as value? Begin by testing exposure and population and do not substitute a familiar completion event for the outcome.

Make the completion boundary explicit in the charter. Name the event that currently looks like success, the user state it is assumed to represent, and the evidence needed to test that assumption. Ask what the user expected to happen next, what the organisation promised, and what a receiving team needs to do. Keep separate fields for action complete, ready to proceed, first successful use and meaningful value. If the source definition changes by route, preserve the variants and explain why they cannot yet be combined. A stable ID should link the definition to its evidence, owner, date and permitted use.

Checkpoint 1: Link E01 to exposure and population and label whether the completion boundary is VERIFIED, OBSERVED, INFERRED, PROPOSED, UNKNOWN or REQUIRES VALIDATION. Test this explanation: the primary metric hides a material negative effect in an under-represented route or group. Before continuing, check the acceptance condition: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 2: Collect and label the evidence

Create evidence IDs before interpretation. In Evidence register, record source, date, population, route, context, limitation, confidence and permitted use. Reconcile product, service, support, operations, research and commercial evidence without pasting personal data. Collect successful, assisted, interrupted, failed and contrary cases; a completed sample cannot prove that every user reaches value.

Triangulate evidence without pretending that sources have the same strength. A timestamp can establish sequence but not experienced effort; a support note can explain context but not prevalence; an interview can reveal a need but not a population rate. Record these limitations beside the claim. Compare people who reached value with those who did not, and include assisted, interrupted, delayed and returned cases. For every important conclusion, write the observation, interpretation, alternative explanation and next test. This makes uncertainty visible and prevents the workbook from becoming a polished record of the first team’s opinion.

Checkpoint 2: Check intended outcome. In the Evidence register, answer “What evidence establishes intended outcome?” and keep its limitation beside the claim. Challenge the hypothesis that the treatment changes support burden or recovery effort without appearing in the success metric. Do not advance until the acceptance gate can be checked from an attributable record: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 3: Set the population, routes and observation window

Use Journey charter to set the start condition, end condition, population, route variants, material exclusions, observation window and decision boundary. For this guide, make intended outcome visible rather than letting a channel or system status stand in for progress. Keep unknown prevalence and unobserved routes explicit.

Bound the decision with conditions rather than interface labels. “User has a configured workflow and can complete the first useful task” is testable; “user reaches the dashboard” is only a screen event. Include the time period, route, population, dependencies and material exclusion. Define what happens when a user pauses, changes channel, needs assistance, withdraws, or receives an incomplete response. Mark those routes as included, excluded or unknown. A narrow boundary is not a claim that other routes do not matter; it is a transparent statement of what this decision can safely govern.

Checkpoint 3: Check inclusion guardrail. Use E03 to test the boundaries of the population, route and observation window, then mark exclusions and unknowns. The relevant hypothesis is “The experiment is scaled before a meaningful rollback and human review path exists.” and the acceptance gate remains: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 4: Map tasks, handoffs and possible causes

In Journey map, map user tasks and context first, then add channels, systems, handoffs and owners. Connect each material transition to evidence IDs. Show where inclusion guardrail or a dependency changes the user's state. In Friction and failure log, describe symptoms separately from causes, note severity and recurrence, and record at least one competing explanation.

Use the map to show transitions and interfaces, not to decorate a workshop. For each transition, record entry condition, user task, context, channel, enabling system, receiving owner, evidence IDs and unresolved question. An internal status belongs in the process context unless evidence shows that it changes the user’s experienced state. Add a failure log row when a transition creates delay, re-entry, confusion, support demand, loss of progress or risk. Ask which other explanation would produce the same symptom, and state what observation would distinguish it before selecting an intervention.

Checkpoint 4: Check contrary effect. Link the mapped transition to “What evidence establishes contrary effect?” and distinguish what was observed from what the team infers. Look for evidence that could disprove “The primary metric hides a material negative effect in an under-represented route or group.” before designing a fix. Acceptance requires: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 5: Check the business consequences and competing causes

Connect the issue to value, harm, cost, delay, quality, trust, retention or service demand only where a defensible source exists. If the size is unknown, label it UNKNOWN or REQUIRES VALIDATION. Test whether contrary effect changes the decision. Do not claim causation from correlation, an aggregate rate or stakeholder agreement.

Translate the journey issue into a decision consequence carefully. A delayed first value may affect retention, revenue timing, service cost, trust, quality or staff capacity, but the size and direction must be evidenced. Use a typed date and source period; do not combine incomparable cohorts. Distinguish harm from inconvenience and report unknowns where no safe estimate exists. A trade-off is a decision object: name who gains, who bears additional effort, what guardrail protects them, and who can stop the change. Do not use a local conversion lift as proof of total business value.

Checkpoint 5: Check rollback evidence. Record the value, harm and trade-off consequence alongside “What evidence establishes rollback evidence?” Ask whether the evidence supports or contradicts “The treatment changes support burden or recovery effort without appearing in the success metric.” and leave the result UNKNOWN when it cannot be established. The decision cannot pass the acceptance gate unless: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 6: Name owners, escalation and recovery routes

In Intervention and acceptance, name one accountable owner, contributors, dependency owners, authority, exception route and recovery owner. Specify what the receiving team must accept. A role without access or power to pause, reject or reverse is not sufficient ownership. Keep proposed technology, automation or AI subordinate to the bounded user and business decision.

Ownership must include authority, access and receiving acceptance. Name the person who can choose, pause, reject or reverse the bounded change, then list contributors and dependencies separately. Ask whether the owner can access the evidence, obtain a decision from privacy or accessibility authorities, and reach the team that carries the downstream work. Define an escalation time and a human route for exceptions. If a vendor, policy, system or legal boundary controls the outcome, record that dependency and its owner. A RACI or ticket alone does not prove accountable recovery.

Checkpoint 6: Check exposure and population. Use “What evidence establishes exposure and population?” to name the accountable authority, receiving team and recovery route. Test the hypothesis that the experiment is scaled before a meaningful rollback and human review path exists against actual access and escalation constraints. Acceptance still means: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 7: Agree acceptance, guardrails and rollback tests

Write acceptance before implementation: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale. Add guardrails for accessibility, privacy, support demand, quality, excluded routes and harm. Use Outcome and recovery to record value, assistance, recovery, exception, negative-case and rollback tests over a stated observation window. The negative case is: A shorter digital path increases completion for unaided users but removes a needed explanation or recovery option, causing another group to fail or seek costly assistance.

Write tests that can fail. The value test should state the user outcome and observation window; the outcome test should state the business or service consequence; the harm test should cover affected routes and material groups; the support test should capture contacts and handling effort; the accessibility test should include keyboard, assistive technology and content needs where relevant; the privacy test should check purpose and minimisation; the trade-off test should state what burden may move; and the rollback test should identify the last accepted state. Keep acceptance and guardrail results separately labelled.

Checkpoint 7: Check intended outcome. Turn “What evidence establishes intended outcome?” into value, accessibility, privacy, harm, support, trade-off and rollback tests. Include the negative case: A shorter digital path increases completion for unaided users but removes a needed explanation or recovery option, causing another group to fail or seek costly assistance. The acceptance gate is only met when: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Step 8: Record the decision and review triggers

Record the human decision and version together. In Review and change log, state proceed, adapt, pause, stop or investigate, the approver, effective date, next review, evidence change and reopen trigger. Approval changes what may be done; it does not turn PROPOSED work into VERIFIED evidence. Observe the result and preserve dissent.

The closing decision should be reversible where feasible and explicit where it is not. Record the authorised approver, decision date, effective date, review date, evidence that supports the decision, dissent, dependencies, rollback and the trigger that would reopen it. During review, compare observed outcome with the pre-declared acceptance and guardrails, not with a post-hoc target. A failed test is a useful result: pause, adapt, stop or commission deeper work. Preserve the private working pack as a controlled decision record, and do not infer public publication or client proof from its completion.

Checkpoint 8: Check inclusion guardrail. Close the loop using “What evidence establishes inclusion guardrail?”, record the decision owner and preserve dissent or contrary evidence. Revisit the hypothesis that the treatment changes support burden or recovery effort without appearing in the success metric at the next review, especially if the negative case appears: A shorter digital path increases completion for unaided users but removes a needed explanation or recovery option, causing another group to fail or seek costly assistance. The acceptance decision remains bounded by: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale.

Worked example: A hypothetical application experiment removes a review screen and improves completion. Screen-reader users and people who need to correct a value now submit errors more often, while support contacts rise. The team pauses scale, restores the last accepted state and tests an accessible alternative. This is a hypothetical teaching example, not an Abrasim client result and not evidence of a guaranteed outcome. Its purpose is to show how a bounded decision, competing explanations, responsible ownership, recovery and acceptance tests fit together.

How to verify the journey change

Verification asks whether the decision changed the user outcome under stated conditions, not whether a map or workbook was completed. Review the evidence chain in this order:

1. Recheck population, routes, start, end, exclusions and observation window in Journey charter.
2. Sample IDs from Journey map back to the Evidence register and permitted sources.
3. Confirm every severe item in Friction and failure log has an owner, disposition and recovery route.
4. Execute the acceptance, accessibility, privacy, value, outcome, harm, support, negative-case and rollback tests in Intervention and acceptance and Outcome and recovery.
5. Observe completion, progression, first value, assistance, recovery, exceptions and adverse outcomes for the agreed window.
6. Reconcile route-level and contrary outcomes; do not use an aggregate to hide material variation.
7. Ask the authorised owner to decide proceed, adapt, pause, stop or investigate.
8. Record evidence changes, decision, approver, review date and reopen trigger in Review and change log.

Use evidence appropriate to the decision. Research and observation explain context; operational records show delay and recurrence; analytics show patterns within collection limits; support records show assistance and recovery; commercial records show qualified value. Reconcile definitions, periods and populations. Verification is incomplete when only the easiest route was tested, personal data was used beyond purpose, accessibility was checked only by automation, downstream ownership was absent, or the window ends before value or recovery can be observed.

Use this evidence with AI safely. AI may organise authorised, minimised evidence, surface contradictions and draft questions. It must not invent users, prevalence, outcomes or legal conclusions, infer sensitive attributes, make the final decision or receive personal or confidential data outside an approved environment. Remove identifiers and use controlled references. Use this prompt:

Role: Journey evidence reviewer. Decision: Use guardrails, inclusion checks and contrary outcomes before scaling a change. Scope: One journey experiment and affected populations; not experimentation-platform design, statistical certification or automatic scale approval. Evidence questions: E01: What evidence establishes exposure and population | E02: What evidence establishes intended outcome | E03: What evidence establishes inclusion guardrail | E04: What evidence establishes contrary effect | E05: What evidence establishes rollback evidence. Competing explanations: The primary metric hides a material negative effect in an under-represented route or group | The treatment changes support burden or recovery effort without appearing in the success metric | The experiment is scaled before a meaningful rollback and human review path exists. Acceptance and guardrails: The experiment owner can show who was exposed, what decision the result supports, which inclusion and harm guardrails were tested, what contrary evidence exists, and who may pause, rollback or refuse scale. Use only authorised, minimised summaries with evidence IDs and labels. Separate VERIFIED, OBSERVED, INFERRED, PROPOSED, UNKNOWN and REQUIRES VALIDATION; cite the supplied ID for every material statement. Identify contradictions, missing routes, unsupported causal claims, trade-offs and the smallest next test. Do not invent users, prevalence, outcomes or legal conclusions. Return an evidence table, risks, questions, decision options and human approvals required.

Review AI output against source records. If it cannot cite the supplied evidence ID, mark the statement REQUIRES VALIDATION. Human owners retain responsibility for research, privacy, accessibility, commercial, operational and recovery decisions.

How to keep the problem from returning

Use three standing controls:

• Define affected populations and guardrails before exposure.
• Review contrary and assisted outcomes alongside the primary result.
• Require human scale approval and a usable rollback route when a guardrail is breached.

Review after a material offer, channel, system, policy, dependency, operating-model or population change; after recurring exceptions; when evidence becomes stale; or when the acceptance result changes. Update evidence, decision and change log together. Assign a review owner even when several teams maintain the journey. That owner must convene contributors, preserve dissent, access permitted evidence and obtain an authorised decision.

Continue with the preceding guide when earlier progression or recovery evidence is unresolved, then use the next guide when that is the next question you need to answer. Use experiment results not repeatable transferable when the adjacent problem owns the wider diagnosis: The analytics guide owns whether experiment results can support later decisions; this guide owns journey harm and inclusion guardrails.

If the issue crosses teams, budgets, systems or evidence sources and cannot be resolved safely within the guide, Challenge the journey experiment before scale. An assessment should first agree the decision, evidence access and authority. Technology choices follow the findings.

Primary sources used

Source cutoff: 2 September 2026. Review by 2 March 2027, or sooner after a material change. These dates apply to the external references below; local organisational evidence still needs its own date and limitation.

• GOV.UK Service Standard: Make sure everyone can use the service
• GOV.UK Service Manual: User research
• GOV.UK Service Standard: Solve a whole problem for users
• W3C Web Content Accessibility Guidelines 2.2
• South African Government: Protection of Personal Information Act