ss-tools/specs/044-dashboard-scenario-execution/spec.md

#region ScenarioExecution.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,scenario,execution,run,step,engine] @BRIEF First-class ScenarioRun/ScenarioStepRun execution engine that walks a validated DashboardTestScenario DAG and dispatches each step to a typed executor, with run lifecycle, cancellation, retry, timeout, and human-checkpoint pause/resume. The most critical missing piece — a scenario can be created but not executed. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0003] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec] @RELATION DEPENDS_ON -> [ScenarioRegistry.Spec] @RELATION DEPENDS_ON -> [DashboardScenarioModel.Spec] @RATIONALE 038 generates an immutable Verification Program IR. A ScenarioRun is a first-class entity independent of AgentRun and VerificationRun. Orchestration and deterministic program executors remain deterministic; a revision may additionally declare bounded AgentEvaluation steps whose typed outputs are resolved by DecisionPolicy, never by free-form orchestration. @REJECTED Agent-orchestrated step execution — rejected because each step would be an LLM call and could rewrite program flow. Explicit versioned AgentEvaluationSpec inside a deterministic step boundary is allowed. @REJECTED Reusing AgentRun as the execution run — rejected because AgentRun is the creation-process run; ScenarioRun is the created-test execution. Reusing VerificationRun — rejected because it is release-pipeline category verification, not arbitrary-DAG execution. @REJECTED human as a dispatched executor — rejected; it is a runner-lifecycle suspend/resume control primitive, not a side-effect executor.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, scenario, execution, run, step, runner, engine, resume, human

Feature Branch: 044-dashboard-scenario-execution Created: 2026-08-07 | Status: Draft Input: "Provide a Scenario Execution Engine: a deterministic runner that walks the validated DashboardTestScenario graph, dispatches each step by tool to a typed executor (browser, superset_api, xlsx, assertion, screenshot, report, artifact), and manages the run lifecycle (queued, running, waiting_human, blocked, cancel, retry, timeout, resume, passed, failed, inconclusive) with immutable execution snapshots pinned to a scenario revision."

User Scenarios

Story 1 — Start a Scenario Run (P1)

Why P1: Users must be able to run a saved scenario against an environment with pinned revision and parameters.

Independent Test: Start a run from a registry scenario + params and verify a ScenarioRun is created with the exact revision, environment, and parameter snapshot.

Acceptance:

1. Given a saved scenario with params When "Run" is submitted Then a ScenarioRun is created pinned to scenario_id + revision_id + content_hash, environment, and parameter snapshot.
2. Given the runner starts When execution begins Then the run derives RunnerPlan, preflights, and transitions through queued → running.
3. Given a PROD environment When start is requested Then the service creates ScenarioRun(pending_approval) plus an ActionApprovalGate before any step executes; approval queues it and denial/expiry blocks it.

---

Story 2 — Execute Steps Deterministically (P1)

Why P1: The core value — running the checks.

Independent Test: Execute a fixture scenario with browser/superset/xlsx/assertion/screenshot steps and verify each step dispatches to the correct executor and records a ScenarioStepRun.

Acceptance:

1. Given the runner walks the DAG in topological order When each step is reached Then ScenarioStep.Dispatch routes by the version-pinned 038 ActionRegistry to a typed executor (browser→BrowserExecutor/Playwright, superset_api→037, sql_evidence→Superset SQL Lab adapter, transform/assertion→bounded 038 program engines, agent_evaluation→bounded evaluation adapter, xlsx→parser+norm, screenshot→ScreenshotService).
2. Given a step produces an output ref When a dependent step consumes it Then the ref is bound from the producer's outputs.
3. Given a step fails When execution continues Then dependents are blocked per failure policy and the run records the failure.

---

Story 3 — Pause at Human Checkpoint and Resume (P1)

Why P1: Human checkpoints must suspend the run without losing state.

Independent Test: Reach a human step and verify the run pauses (waiting_human), persists step state, and resumes from the correct point after disposition.

Acceptance:

1. Given a human step is reached When the runner executes Then the run sets status waiting_human, persists the ScenarioStepRun as paused, creates a HumanCheckpoint (confirm/false_positive/inconclusive — NOT a 036 ApprovalGate) with evidence context, and stops advancing the DAG.
2. Given a human decision (confirm/false-positive/inconclusive) When submitted with its checkpoint version Then an atomic CAS consumes the checkpoint, records the outcome, and internally resumes dependents.
3. Given a page reload mid-wait When reopened Then the run status, waiting step, and gate are recoverable by scenario_run_id.

---

Story 4 — Cancel, Retry, Timeout (P2)

Why P2: Operational control over runs.

Independent Test: Cancel a running scenario, retry a failed step, and simulate a timeout; verify terminal states and step attempts.

Acceptance:

1. Given a user requests cancel When submitted Then the run enters cancel_requested, drains in-flight steps within a bounded window, and terminates cancelled.
2. Given a step fails When retry is requested Then a new attempt runs with bounded attempts and updated attempt count.
3. Given a step exceeds its timeout When triggered Then the step is marked failed/inconclusive with a timeout reason, and the run continues per policy.

---

Story 5 — Immutable Execution Snapshot (P2)

Why P2: Results must be reproducible regardless of later edits.

Independent Test: Run a scenario, then edit it, and verify the earlier run's results reference the original revision.

Acceptance:

1. Given a run started When the scenario is later edited Then the run's revision_id + content_hash snapshot is unchanged.
2. Given results are stored When queried Then they include provenance (revision, runner_version, template_version, baseline_revision, environment, parameter snapshot).

---

Edge & Failure Cases

#	Scenario	Category	Expected Behavior	Recovery
E1	PROD run without approval	auth/gate	Durable pending_approval run + gate, no dispatch	Approve via ActionApprovalGate
E2	Step timeout	execution	Step failed/inconclusive with reason	Retry / continue
E3	Retry exhausted	execution	Step failed; dependents blocked	Triage (047)
E4	Superset 5xx/403/422	integration	Typed error taxonomy preserved	Retry / continue
E5	Runner crash mid-run	resilience	API/XLSX steps resume only if retry-safe; browser state replays from last browser-safe checkpoint	Recover / replay
E6	Cancel mid-step	concurrency	In-flight completes or times out in drain window	—
E7	Duplicate output ref	data-integrity	038 validator rejects before run	Fix graph
E8	Stale scenario run (registry)	data-quality	Warning-gated or blocked per policy	Revalidate

Requirements

Functional

• SCEX-FR-001: A ScenarioRun MUST be a first-class entity pinned to scenario_id (UUID) + revision_id (UUID) + content_hash, environment, parameter snapshot, and optional agent_run_id/verification_run_id provenance.
• SCEX-FR-002: The runner MUST walk the DAG in topological order and dispatch each step only through the version-pinned 038 ActionRegistry. Browser, Superset API/SQL Lab, XLSX, bounded transform/assertion, screenshot, report and artifact executors are typed; unknown {tool,action} is rejected.
• SCEX-FR-003: Scenario orchestration MUST be deterministic and MUST NOT generate/rewrite SQL, DSL, assertions, graph or executor order at runtime. A declared AgentEvaluationSpec MAY run inside its bounded step contract; it is not agent-per-step orchestration.
• SCEX-FR-004: A human step MUST suspend the run (status waiting_human), persist full HumanCheckpoint state (eligibility, evidence, expiry, CAS decision version), and internally resume dependents after its one-time disposition. HumanCheckpoint is distinct from ActionApprovalGate.
• SCEX-FR-004a: A revision containing a human step MUST be manual_run_only. Scheduled, trigger, deploy, ETL and API execution are rejected before run creation; a HumanCheckpoint is never skipped to obtain an automatic PASS.
• SCEX-FR-005: The lifecycle MUST include pending_approval, queued, running, waiting_human, blocked, cancel_requested, cancelled, passed, failed, inconclusive; cancel drains in-flight within a bounded window.
• SCEX-FR-006: Steps MUST support attempt counts, retry policy, per-step timeout, input/output ref binding, artifact_refs, and error_code.
• SCEX-FR-007: A ScenarioRun MUST be recoverable by scenario_run_id; browser recovery MUST reconstruct state from a declared browser-safe checkpoint, not continue a dead Playwright context. Results MUST carry target and execution-principal provenance.
• SCEX-FR-008: PROD-classified environments MUST require an ActionApprovalGate before execution.
• SCEX-FR-009: Executors MUST reuse 037 (metric_executor/comparison), 038 (capture), 036 (evidence/artifacts/HITL), and existing browser/xlsx infra; a second Playwright/LLM/SQL stack is forbidden.
• SCEX-FR-010: human is a runner-lifecycle control, not a dispatched executor; the executor registry covers the other seven tools.
• SCEX-FR-011: Failed, inconclusive and blocked runs MUST emit idempotent 036 InvestigationSignals carrying immutable run/evidence provenance; 047 creates/updates the Queue/Episode. They MUST NOT automatically start a chat, an AgentRun, or a remediation action.
• SCEX-FR-012: An opened InvestigationCase MAY use the agent to construct diagnostic runs and controlled experiments under delegated policy. Agent work cannot bypass executor contracts, runner lifecycle, capacity, mutation policy or a required ActionApprovalGate; declared AgentEvaluationSpec is the only permitted runtime reasoning boundary.
• SCEX-FR-013: A HumanCheckpoint remains a manual-run-only analyst decision. The agent may summarize evidence but MUST NOT consume the checkpoint, choose its disposition, or turn it into scheduled automation.
• SCEX-FR-014: SqlEvidenceExecutor MUST execute exactly the immutable 038 SqlEvidenceSpec through Superset backend/SQL Lab with pinned database identity, ExecutionPrincipal and RLS/security fingerprint. Runtime only supplies typed ParameterBindings and may not alter SQL, relation, projection, JOIN or WHERE.
• SCEX-FR-015: AgentEvaluation MUST be a separate immutable runtime record and DecisionPolicy MUST deterministically map it plus deterministic evidence to StepOutcome. A bare model verdict never directly sets ScenarioResult.
• SCEX-FR-016: Browser actions and mutation safety MUST use the same versioned 038 ActionRegistry/mutation contract. Mutating browser steps in PROD are prohibited; test-data mutation needs fixture scope, record keys, side-effect/retry and cleanup policy independent of PROD approval.

Key Entities

• ScenarioRun: Recoverable execution instance of a pinned scenario revision; owns status, phase, params, provenance, steps.
• ScenarioStepRun: One step execution attempt/record: status, attempt, timing, inputs/outputs, artifact_refs, error_code, progress.
• RunnerPlan: Deterministically derived at run start from ScenarioRevision + ParameterBinding + TargetSnapshot + baseline/runtime policy. runner.plan.json is diagnostic/reference materialization only.
• ScenarioExecutorRegistry: Mapping of step.tool → typed executor; human excluded (lifecycle control).
• ScenarioExecutionResult: Aggregated run result with pass/fail/inconclusive per step and provenance.

Success Criteria

• SC-001: A fixture scenario runs end-to-end, dispatching every step to the correct executor, in deterministic order.
• SC-002: Human checkpoint suspends the run and resumes from the exact next step without rerunning completed steps.
• SC-003: 100% of cancel requests terminate with cancelled and bounded drain.
• SC-004: Runs are recoverable by scenario_run_id after disconnect.
• SC-005: Results reference an immutable revision snapshot + full provenance; editing the scenario after a run never changes that run's results.
• SC-006: human is not invoked as an executor; PROD runs are gated.

Clarifications

Session 2026-08-07

• Q: Is the agent the runner? → A: No. The backend runner orchestrates deterministically. The agent authors before/after runs and may reason only inside a declared bounded AgentEvaluationSpec.
• Q: Is human an executor? → A: No. It is a runner-lifecycle suspend/resume control; the executor registry covers browser/superset_api/xlsx/assertion/screenshot/report/artifact.
• Q: How does this differ from VerificationRun/AgentRun? → A: ScenarioRun executes the user-created DashboardTestScenario DAG; AgentRun is the creation run; VerificationRun is release-pipeline category verification. Three distinct run concepts.

Implementation Status & MVP Debt (audit 2026-08-07)

Facts (code check):

• 🔴 No ScenarioRun/ScenarioStepRun model or runner exists; runner.plan.json is a stub {scenario_id, revision_id, step_count} (and is a reference artifact only — the RunnerPlan is derived at run start).
• 🔴 No executor registry or dispatch; execute_step/ScenarioStep.*execute grep = empty.
• ✅ Executors to reuse already exist: 037 metric_executor_async/comparison.py, 038 capture.py, 036 evidence/artifacts/HITL, 040 RunnerPool pattern.

#endregion ScenarioExecution.Spec