ss-tools/specs/047-dashboard-scenario-analytics/spec.md

#region ScenarioAnalytics.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,scenario,triage,flakiness,analytics,health] @BRIEF Investigation Queue and agent-led case workspaces over deterministic scenario analytics, so failures become evidence-led remediation work rather than isolated forms. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0005] @RELATION DEPENDS_ON -> [ScenarioExecution.Spec] @RELATION DEPENDS_ON -> [ScenarioRegistry.Spec] @RELATION DEPENDS_ON -> [ScenarioRunMonitor.Spec] @RATIONALE 036 supplies durable agent work and 044 supplies immutable run evidence, but neither provides an analyst-controlled queue and long-lived investigation case. Health and failure identity remain deterministic inputs to that work. @REJECTED Ending at pass/fail/inconclusive — rejected because operational workflow begins there. @REJECTED Opening an agent chat for every failure — rejected because queue deduplication and analyst intent are needed to avoid noise.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, scenario, triage, flakiness, analytics, health, trend

Feature Branch: 047-dashboard-scenario-analytics Created: 2026-08-07 | Status: Draft Input: "Provide an Investigation Queue and agentic case workspace for failed/stale/load/automation evidence, backed by deterministic flakiness, health, trends and recurring-failure analytics."

User Scenarios

Story 1 — Open an Agent-Led Investigation (P1)

Why P1: A failed test must enter a deliberate evidence-led investigation, not a classification form.

Independent Test: Feed a failed run into the queue, open it explicitly, and verify the case creates an agent thread with immutable evidence and an audited disposition projection.

Acceptance:

1. Given a failed/inconclusive/blocked run When its deterministic evidence is produced Then an Investigation Queue item is created or updated; no agent chat or tool run starts automatically.
2. Given an analyst selects “Investigate with agent” When the item opens Then a durable case has chat, source/evidence snapshots, tool timeline and linked AgentRuns.
3. Given the case reaches a disposition When it is persisted Then a versioned, audited TriageRecord projection updates the run/step view without changing historical run truth.

---

Story 2 — Detect Flakiness and Compute Health (P1)

Why P1: Distinguish a broken dashboard from a broken test.

Independent Test: Feed a run history with repeated step failures and verify flakiness and contextual health are calculated deterministically before any agent decision.

Acceptance:

1. Given a step fails intermittently When analytics compute Then it is classified flaky with a flakiness ratio.
2. Given run history When health is derived Then 30d success rate, flaky ratio, infra-failure ratio, and most-unstable-step are shown.
3. Given a scenario health changes When thresholds are crossed Then the registry health badge updates (042), while an untriaged failure remains attention with unknown cause.

---

Story 3 — Analyze Trends and Recurring Failures (P2)

Why P2: Repeated failures and trends need surfacing.

Independent Test: Feed multiple runs and verify trends and recurring-failure grouping render.

Acceptance:

1. Given run history When trends are rendered Then success-rate trend and failure-classification distribution are shown.
2. Given the same step fails repeatedly When aggregated Then recurring failures are grouped with a count and first/last occurrence.
3. Given a failure recurs after its FailureEpisode was resolved When aggregated Then a new alertable episode opens; only duplicates inside an active episode are suppressed.

---

Edge & Failure Cases

#	Scenario	Expected Behavior	Recovery
E1	No run history	Health unknown, no flaky signal	—
E2	Case disposition concurrent edit	409 conflict panel	Reload
E3	Case/agent access denied	403 permission_denied	Contact admin
E4	Infra failure spike	Classified infra, not product regression	Investigate infra

Requirements

Functional

• SCAN-FR-001: Failed/inconclusive/blocked runs, staleness, baseline immutability, load and repeated automation findings MUST emit the canonical 036 InvestigationSignal; 047 MUST create/update a deduplicated Investigation Queue item from that idempotent envelope. The queue MUST NOT auto-start an agent chat, AgentRun or tool action.
• SCAN-FR-002: An analyst MUST be able to open a queue item into a durable InvestigationCase with chat, immutable evidence snapshot, hypotheses, AgentAction timeline, linked AgentRuns, approvals, verification and final disposition. The agent may execute only delegated/policy-authorized actions.
• SCAN-FR-003: Triage MUST be a CAS/audited compact projection of a case disposition; the historical RunResult stays immutable truth.
• SCAN-FR-004: The system MUST detect flakiness per step using same environment class, logical_step_id, compatibility_family, baseline family and comparable context. It requires a post-failure pass and two result transitions; infra/cancelled/inconclusive outcomes are excluded from the pass/fail denominator.
• SCAN-FR-005: Health MUST be contextual and computed separately for product, scenario-test, infrastructure and agent-evaluation health plus overall attention. Agent verdict variance MUST NOT be classified as deterministic flakiness.
• SCAN-FR-006: Health MUST feed the 042 registry health badge.
• SCAN-FR-007: The system MUST render trends and group recurring failures by immutable compatibility-scoped fingerprint, never including triage classification.
• SCAN-FR-008: A matching failure inside an active FailureEpisode is deduplicated; a matching failure after resolution MUST open a new alertable FailureEpisode and Queue item.
• SCAN-FR-009: Object-level result/evidence access and agent-action authority MUST enforce ACL separately from scenario-result:view and scenario-result:triage.
• SCAN-FR-010: Health/trends/recurring MUST aggregate scenario history, not a single run.
• SCAN-FR-011: Investigation, approval, conflict recovery and closure MUST use persistent case workspaces and inline cards; modal/dialog interaction MUST NOT be required.
• SCAN-FR-012: A case may become resolved only with verification evidence and completed reconciliation. accepted requires an analyst-recorded accepted-risk/won't-fix/duplicate rationale. A new matching active signal MUST reopen the work (or create a new case for a new FailureEpisode).
• SCAN-FR-013: Health and flakiness eligibility MUST use the exact 044 AnalyticsContextKey; grouping by environment or revision alone is forbidden.
• SCAN-FR-014: Analytics MUST distinguish deterministic failure, agent-evaluation disagreement, low confidence, model instability, infrastructure failure, scenario-definition defect and product regression. Recurring fingerprints remain immutable raw evidence fingerprints and MUST NOT include a human/agent classification.

Key Entities

• InvestigationQueueItem: Deduplicated attention item with source/evidence context; opening it is explicit.
• InvestigationCase: Agent-led durable workstream with chat, tools, approvals, verification and closure.
• TriageRecord: Compact audited projection of case disposition for a run/step.
• FlakinessSignal: Per-step intermittent-failure metric (ratio, window).
• ScenarioHealth: 30d success rate, flaky ratio, infra ratio, most unstable step.
• RecurringFailureGroup: Grouped identical failures with count, first/last occurrence, current triage.

Success Criteria

• SC-001: 100% of qualifying fixture events create/update a deduplicated queue item without starting agent work; an opened case has immutable evidence and an audited disposition projection.
• SC-002: Flaky steps are detected and health derived from run history with deterministic output.
• SC-003: Health badge in 042 updates when thresholds cross.
• SC-004: Trends and recurring failures render; recurrence after a resolved episode creates a new alertable episode.
• SC-005: RBAC enforces view vs triage.

Clarifications

Session 2026-08-07

• Q: New data? → A: Reuses 044/037/040/041 evidence; adds Queue, Case, AgentAction projection and deterministic analytics; feeds 042 health.
• Q: Does a case change graph/result? → A: No. A case may create a separately immutable revision or policy-bound action, but it never rewrites historical run truth.
• Q: Does every failure open chat? → A: No. It enters a deduplicated Investigation Queue; the analyst explicitly opens the case.

Implementation Status & MVP Debt (audit 2026-08-07)

Facts (code check):

• 🔴 No triage/flakiness/health/trend functionality exists for scenario runs.
• 🟡 042 derives a basic health badge; 047 extends it with flakiness + trends.
• 🟡 Reusable analytics/aggregation patterns exist in 040 (consistency detection) and 037 (comparison).

#endregion ScenarioAnalytics.Spec