Methods and sources

Fundamental prototype questions it speaks to

• What tools can we bring to bear to improve instrumentation of go-lives instead of merely counting tickets? (Go-live radar)
• Can we reduce dependence on legacy ITSM ticket thinking and become more proactive? (Nudge rules engine)
• Can we maximize the use of first-pass self-service support? (Tier-0 assistant, containment)
• What digital technologies can reduce the need for boots on the ground? (Self-service plus context-packet hand-offs)
• Can we adopt payment models that pay for insights and successes rather than labor? (Outcome milestones)
• Is our customer definition right? (Segment-aware answers: TRICARE beneficiary, Veteran, clinician)

Rebuild locally with node scripts/build-corpus.mjs. It fetches each page, keeps the main content, splits it into passages of about 25-170 words under their headings, and writes data/corpus.json with the URL and retrieval date. Nothing runs in CI or on a schedule.

• Okapi BM25 (k1 = 1.2, b = 0.75) over passage headings and text, with light stemming, stopwords and a small everyday-words synonym list (for example “login” to “sign in”).
• Answers are extractive: 1-3 sentences quoted from one passage, with the passage ID, page title, publisher, link and retrieval date. Nothing is generated; there is no LLM and no API key.
• Confidence = 0.55 × word coverage + 0.45 × match strength, times a segment-fit factor (content written for another audience counts less). It answers at 0.55 or above with at least 50% of the user's words covered; otherwise it hands off with a context packet (question, segment, what was tried, suggested queue, transcript).
• Urgent wording (for example chest pain or suicide) skips retrieval and shows 911 and 988 guidance.

• Go-live telemetry: a fictional product release at 8 fictional sites over 42 days, generated by a seeded simulation (seed 20261104, mulberry32). Same numbers on every load.
• The treatment effect in the experiment ledger is built into the simulation; it shows how the ledger works, not a measured result.
• The 35 unresolved questions on the insights screen were written for this prototype.
• Milestone amounts are illustrative, from the solution brief.

• No PII or PHI. The prototype has no accounts, no server-side storage and no analytics. Questions and the event log stay in your browser (localStorage) and can be cleared from the assistant screen.
• The question box warns users not to type personal or health details. Context packets carry the question, segment and routing only.
• The design assumes, at award, a HIPAA Business Associate Agreement, hosting inside a DoD-authorized environment, and DoD identity for staff. None of that is claimed for this prototype.

• The content set is a snapshot of 18 public pages, not the MHS GENESIS help library; answers about DoD-specific portal tasks often hand off by design.
• Keyword retrieval misses paraphrases that share no words with the content. A production version could add a vetted embedding model inside the boundary, keeping the same extract-and-cite rule.
• The rate-ratio interval assumes Poisson counts; real ticket data are often over-dispersed, so production analysis would use a negative binomial or bootstrap interval and account for site clustering.
• Weekly looks inflate the false-positive rate if read naively; the success stop requires the lower bound, not the point estimate, to clear 30%, and a production plan would set an alpha-spending rule.
• Theme-to-backlog mapping uses a hand-written rule table; new themes fall back to “needs triage”.