7.1 KiB
#region ScenarioAnalytics.DataModel [C:5] [TYPE ADR] [SEMANTICS data-model,scenario,investigation,queue,agent,analytics,health] @BRIEF Investigation Queue/Case, compact triage projection, and deterministic analytics models for 047. @RELATION DEPENDS_ON -> [AgentInvestigation.Cases] @RATIONALE Failed runs need an evidence-led agentic workstream, while historical execution truth and calculated analytics must remain independently reproducible. @REJECTED Treating triage as a standalone form — rejected because analysts investigate through an agent thread with evidence and tools, not isolated classification fields. @REJECTED Letting agent judgment rewrite RunResult, health calculations, baseline truth or recurring identity — rejected because those are deterministic historical facts.
InvestigationQueueItem — attention, not automatic chat
Queue entries are generated only by consuming the canonical 036 InvestigationSignal from failed/inconclusive/blocked ScenarioRuns, staleness signals, baseline immutability violations, load circuit-breaker/consistency findings and repeated automation failures. Fields: id, source_type, source_id, scenario_id?, run_id?, logical_step_id?, severity, fingerprint?, active_episode_id?, evidence_summary, target_snapshot, execution_principal_fingerprint?, suggested_next_action, state, count, first_seen_at, last_seen_at, case_id?. The signal idempotency identity is preserved, so a producer cannot manufacture duplicate Queue work by retry.
State: new | acknowledged | case_opened | suppressed | resolved. A matching occurrence updates one item only inside an active FailureEpisode; a matching occurrence after its resolution creates a new item. Queue creation does not start AgentRun, tool calls or a chat.
InvestigationCase and AgentAction — agentic workstream
An analyst explicitly opens a queue item into InvestigationCase { id, queue_item_id, status, source_snapshot, evidence_snapshot, owner_actor_id, agent_thread_id, opened_at, resolved_at?, final_disposition?, resolution_summary? }.
Status: open | investigating | awaiting_approval | awaiting_external_change | verifying | resolved | accepted | reopened.
The case owns chat, hypotheses, tool timeline and linked AgentRuns. Every tool call is the shared 036 AgentAction record: canonical inputs, risk/policy decision, target/key scope, side-effect identity, pre/postcondition evidence, cleanup/reconciliation, approvals and actor/agent/tool provenance. The agent may autonomously read, diagnose, run policy-permitted diagnostics, mutate authorized fixture data, and save validated scenario revisions. It never bypasses ACL, deterministic validation, ActionRegistry mutation contracts, capacity, immutable revision creation, or a required ActionApprovalGate. Failed cleanup prevents case resolution.
Closure policy: resolved requires non-empty verification evidence proving the stated acceptance condition and no unresolved cleanup/reconciliation action. accepted requires an analyst-confirmed accepted-risk/won't-fix/duplicate rationale; it is not a claim that the product passed. awaiting_external_change is used while an external remediation or cleanup is pending. A new matching active signal after a terminal disposition reopens the case (or opens a new case if the episode is new); it never silently remains resolved.
TriageRecord — compact audited case projection
RunResult is immutable historical truth. TriageRecord projects the current case disposition onto run_id + logical_step_id? for Registry, Run Monitor and analytics: investigation_status (new|investigating|resolved), classification (product_regression|data_regression|baseline_stale|scenario_bug|infrastructure_failure|not_confirmed), resolution (fixed|accepted_risk|duplicate|wont_fix|false_positive), comment, case_id, decision_version, actor_id, timestamps.
The projection is versioned/CAS and append-only audited. It is never a free-standing modal form, never changes a RunResult, scenario graph or baseline, and is derived/updated only by an authenticated case decision.
FlakinessSignal — deterministic rules
Window: last N eligible runs (default 30). Eligibility per deterministic step: same environment class, same logical_step_id, same compatibility_family, same baseline family, and comparable target/principal context. Infrastructure outcomes, cancelled and inconclusive runs are excluded from the deterministic pass/fail denominator. AgentEvaluation verdict variation is never counted as deterministic flakiness. It is separately aggregated as evaluation disagreement, low-confidence rate and model instability keyed by AgentEvaluationSpec/model/prompt version. A deterministic step is flaky only when pass and fail are both observed, a pass occurs after the first failure, at least two pass/fail state transitions occur, and failure ratio is within (X,Y) (default 5%–50%). A one-way PASS→FAIL change is a regression signal, never flaky.
Fields: scenario_id, logical_step_id, compatibility_family, context_key, window, eligible_runs, total_runs, failures, flaky_runs, ratio, is_flaky, excluded_outcomes.
ScenarioHealth — deterministic, contextual
AnalyticsContextKey is exactly the server-derived 044 SHA-256 over environment_class + compatibility_family + baseline_family + dashboard_release_id + dashboard_fingerprint + dataset_lineage_fingerprint + execution_principal_fingerprint. Fields: scenario_id, environment_class, context_key (=AnalyticsContextKey), window, product_health, scenario_test_health, infrastructure_health, agent_evaluation_health, overall_attention, success_rate, flaky_ratio, infra_failure_ratio, inconclusive_ratio, agent_disagreement_ratio, low_confidence_ratio, model_instability_ratio, most_unstable_step, generated_at, confidence.
Product health consumes product/data regressions; scenario-test health consumes scenario bugs, stale baselines and deterministic flaky signals; infrastructure health consumes typed infrastructure outcomes; agent-evaluation health consumes disagreement/low-confidence/model-instability. Untriaged failures raise overall_attention=attention with cause unknown, but do not fabricate a classification. 042 displays only overall_attention, or unknown if analytics is unavailable.
RecurringFailureGroup and FailureEpisode — immutable identity
Group identity is (scenario_id, compatibility_family, logical_step_id, error_code, normalized_error_signature, assertion_kind, affected_ref). Triage data is excluded. RecurringFailureGroup has counts and first/last occurrence; FailureEpisode { id, group_id, opened_at, resolved_at?, resolution? } is active iff resolved_at is null. A matching occurrence in an active episode increments it and suppresses a duplicate queue item; a matching occurrence after resolution opens a new alertable episode and a new queue item.
Boundary
047 owns queue, case, triage projection and analytics. It consumes 044 execution, 042 staleness, 037 baseline and 040 load evidence; it may request actions through 036 but never replaces their deterministic execution or ownership.
#endregion ScenarioAnalytics.DataModel