Implementation Plan: Scenario Execution Engine

Branch: 044-dashboard-scenario-execution | Date: 2026-08-07 | Spec: spec.md | Status: Draft

Summary

Implement deterministic ScenarioRun/ScenarioStepRun orchestration for the validated immutable Verification Program, dispatching each registered {tool, action} to a typed executor (reusing 037/038/036/040 services) and handling lifecycle. RunnerPlan is derived at launch; runner.plan.json is diagnostic/reference only. Explicit AgentEvaluationSpec is bounded inside a typed step, never orchestration.

Technical Context

Language/Version: Python 3.13+ (backend), TypeScript DTOs (frontend consumed by 045) Primary Dependencies: FastAPI, SQLAlchemy, Pydantic 2; reuse 037 metric_executor/comparison, 038 capture, 036 evidence/HITL, 040 RunnerPool pattern, browser/xlsx infra Storage: new tables scenario_runs, scenario_step_runs Testing: pytest (unit/contract/integration for executor dispatch, resume, cancel, timeout) Frontend Architecture: DTOs only (run monitor is 045) Performance Goals: step dispatch < 100ms overhead; deterministic topological order; bounded drain on cancel Constraints: deterministic (no LLM per step); human not an executor; PROD gated; immutable revision snapshot Scale: up to 100-step graphs, concurrent scenario runs per environment bounded

Constitution Check

Principle	Result
I. Semantic Contract First	PASS — run/step/runner contracted C4-C5
II. Decision Memory	PASS — research R1-R5
III. External Orchestrator	PASS — executor reuse
IV. Module Discipline	PASS — runner/registry/executors separated
V. RBAC Enforcement	PASS — scenario:run, scenario:run:prod
VII. Test-Driven C3+	PASS — dispatch/resume/cancel tests first
VIII. Attention-Optimized	PASS

Project Structure

specs/044-dashboard-scenario-execution/
├── spec.md / data-model.md / research.md / plan.md / tasks.md / traceability.md / quickstart.md / ux_reference.md
├── checklists/requirements.md
├── contracts/modules.md, contracts/openapi.yaml, contracts/ux/
└── prototype/index.html + manifest.md

backend/src/models/scenario_run.py          # ScenarioRun, ScenarioStepRun
backend/src/services/dashboard_testing/execution/
│   ├── runner.py  runner_plan.py  dispatch.py  executor_registry.py
│   ├── executors/ (browser.py superset.py xlsx.py assertion.py screenshot.py report.py artifact.py)
│   ├── human.py  resume.py  cancel.py  retry.py
└── api/routes/dashboard_testing/scenario_runs.py  # POST/start/cancel/resume/human/GET/events

Delivery Phases

ScenarioRun/ScenarioStepRun models + migration + fixtures.
RunnerPlan Derive/Validate from revision + bindings + target.
Executor registry + per-tool executors (reuse 037/038/036).
Deterministic DAG dispatch + ref binding + failure propagation.
Human suspend/resume (036 gate) + recoverable run.
Cancel/retry/timeout + lifecycle.
Immutable snapshot + provenance; REST routes; frontend DTOs.
PROD gate + RBAC + regression gates.

API and Schema

contracts/openapi.yaml — POST /scenario-runs, cancel, resume, human/decision, GET detail, GET events (SSE).

Traceability

traceability.md maps Story → model → operationId → contract → task → test.

Cross-Spec Boundary

Consumes persisted scenario + revision from 042.
Reuses executors from 037 (metrics/comparison), 038 (capture), 036 (evidence/HITL), 040 (RunnerPool).
Feeds run results to 045 (monitor) and 047 (triage); triggered by 046 (automation).
Distinct from VerificationRun (037) and AgentRun (036).

Complexity Tracking

No exception planned. Runner/dispatch are C5 but decomposed (runner, dispatch, executors, human, cancel). Do not collapse into one oversized orchestrator.

3.8 KiB Raw Blame History