Screen 4 of 5 ยท Learning, Adjusting and Solutions
Experiment and outcome ledger
One falsifiable hypothesis, a randomized control cohort, a pre-registered stop/adjust rule checked every week, and payments tied to measured outcomes rather than hours.
Synthetic pilot results
Hypothesis
Proactive nudges plus tier-0 self-service cut human-touch tickets per 100 new users by at least 30% against a control cohort, without lowering sentiment.
Questions this answers
- Can we maximize first-pass self-service and become proactive instead of counting tickets?
- Does reducing human touch cost us sentiment, or improve it?
- Which sites and segments need a different approach (adjust, not abandon)?
Design
- New users at each of 8 sites randomized 50/50 to treatment (nudges + tier-0 assistant) or control (current help).
- Primary measure: human-touch tickets per 100 new users. Rate ratio with a 95% log-scale Wald interval.
- Guardrail: sentiment proxy (1-5); treatment must not be worse by more than 0.10 (lower bound of the difference).
- Tickets per 100 new users, treatment
- 15.2
- 241 tickets, 1,582 new users
- Tickets per 100 new users, control
- 25.2
- 401 tickets, 1,592 new users
- Reduction vs control
- 40%
- 95% CI 29% to 48%; p < 0.001
- Sentiment difference
- +0.19
- 95% CI +0.10 to +0.27 on a 1-5 scale
Effect size with 95% confidence intervals
Week 6 (day 42). Dots are estimates; bars are 95% intervals. The dashed line is the 30% target.
Decision now
ContinueEstimate 40% (95% CI 29% to 48%); keep collecting.
30% threshold metSuccess-based payment test: point estimate at or above 30%, interval excludes no effect, sentiment non-inferior.
Pre-registered rule (checked weekly)
- Stop: success when the lower bound of the reduction is at least 30% and sentiment is non-inferior (from week 4).
- Adjust when the sentiment lower bound falls below -0.10, or the estimate is under 15% at week 3 or later.
- Stop: futility when even the upper bound is below 30% (from week 4).
- Continue otherwise.
Weekly looks
Cumulative results at each pre-registered look. This is the audit trail for the Learning criterion.
| Week | Treatment /100 | Control /100 | Reduction (95% CI) | Sentiment diff (95% CI) | Decision |
|---|---|---|---|---|---|
| 1 | 5.6 (n 342) | 6.0 (n 351) | 7% (95% CI -73% to 50%) | +0.05 (95% CI -0.16 to +0.27) | AdjustSentiment lower bound -0.16 is below the -0.1 margin. |
| 2 | 8.6 (n 845) | 13.9 (n 859) | 38% (95% CI 17% to 53%) | +0.11 (95% CI -0.01 to +0.24) | ContinueEstimate 38% (95% CI 17% to 53%); keep collecting. |
| 3 | 10.8 (n 1326) | 18.2 (n 1346) | 41% (95% CI 27% to 52%) | +0.16 (95% CI +0.06 to +0.26) | ContinueEstimate 41% (95% CI 27% to 52%); keep collecting. |
| 4 | 13.9 (n 1507) | 22.1 (n 1517) | 37% (95% CI 26% to 47%) | +0.16 (95% CI +0.07 to +0.26) | ContinueEstimate 37% (95% CI 26% to 47%); keep collecting. |
| 5 | 14.8 (n 1563) | 24.3 (n 1573) | 39% (95% CI 28% to 48%) | +0.18 (95% CI +0.09 to +0.27) | ContinueEstimate 39% (95% CI 28% to 48%); keep collecting. |
| 6 | 15.2 (n 1582) | 25.2 (n 1592) | 40% (95% CI 29% to 48%) | +0.19 (95% CI +0.10 to +0.27) | ContinueEstimate 40% (95% CI 29% to 48%); keep collecting. |
Arms at a glance
| Cohort | New users | Human-touch tickets | Per 100 new users | Self-service containment | Median time to first task | Sentiment (n) |
|---|---|---|---|---|---|---|
| Treatment: nudges + tier-0 | 1,582 | 241 | 15.2 | 64% | 8.9 h | 3.52 (771) |
| Control: current help | 1,592 | 401 | 25.2 | 34% | 14.0 h | 3.33 (737) |
Adjustment log
Adjusting criterion: the partner changes course in real time, at its own discretion and inside the fixed price. No added Government funding.
| Week | Site | Signal | Change made |
|---|---|---|---|
| 2 | Site D (Saltmarsh) | Containment 43% (< 55%) | Review unanswered questions; add 5 approved articles; move the walkthrough nudge from 48 h to 24 h |
| 3 | Site D (Saltmarsh) | Containment 51% (< 55%) | Keep the change; re-check next week |
| 4 | Site D (Saltmarsh) | Containment 51% (< 55%) | Keep the change; re-check next week |
| 5 | Site D (Saltmarsh) | Containment 52% (< 55%) | Keep the change; re-check next week |
| 6 | Site D (Saltmarsh) | Containment 52% (< 55%) | Keep the change; re-check next week |
Containment shown is cumulative since go-live, so it rises slowly after a fix. Sites monitored: 8. Synthetic.
Outcome-based milestones
Each payment is tied to a measurable outcome, not to hours or level of effort.
Illustrative, from the solution brief. Figures show the proposed payment structure; they are not a price quote, an award or a Government commitment.
| ID | Milestone | Measurable outcome (acceptance) | Evidence | Payment |
|---|---|---|---|---|
| M1 | Baseline instrumentation live | Go-live radar reports adoption, time to first success, containment, tickets per 100 new users and sentiment for every pilot site, with a two-week baseline. | Radar access for the Government; baseline report | $30,000 |
| M2 | Tier-0 assistant in a sandbox with an approved content set | Assistant answers only from a Government-approved content set, cites every answer, hands off below the confidence threshold with a context packet. | Sandbox demo; content set sign-off; hand-off test log | $40,000 |
| M3 | Pilot cohort results measured against control | Randomized treatment vs control results on tickets per 100 new users and sentiment, with 95% confidence intervals and the weekly decision log. | Experiment ledger export; analysis notebook | $45,000 |
| M4 | Insights report and go/no-go recommendation | Themes from unresolved questions mapped to backlog items, plus a go/no-go recommendation on scaling. | Insights report; backlog items filed with product teams | $20,000 |
| S | Success-based paymentMet in this synthetic run | Paid only if the 30% reduction threshold is met: point estimate at or above 30%, 95% interval excludes no effect, sentiment non-inferior (lower bound at or above -0.10). | M3 ledger result | $15,000 |
| Total (illustrative), of which $15,000 is at risk on outcome | $150,000 | |||