4.0 KiB
Scenario Execution Engine — Phase 0/1 Research (044)
Branch: 044-dashboard-scenario-execution | Date: 2026-08-07 | Spec: spec.md
R1. Execution model — deterministic backend runner
Decision: Deterministic ScenarioRunner orchestration walks the immutable Verification Program in topological order and dispatches registered executors. A declared bounded AgentEvaluationSpec may reason inside one typed step, but cannot own orchestration or rewrite program content.
Rationale: Matches 038 determinism (direct LLM-to-code rejected) and 040's non-agent RunnerPool pattern. Execution must be reproducible; an LLM per step breaks that.
Alternatives: agent-orchestrated LangGraph nodes per step (rejected: nondeterminism/drift); hybrid (deferred, not MVP).
Impact: ScenarioRunner, ScenarioExecutorRegistry, topological iteration.
R2. Run state — new ScenarioRun/ScenarioStepRun tables
Decision: Persist run + per-step state in DB. Runs recoverable by scenario_run_id.
Rationale: Reload during a test, human-checkpoint resume, and reproducibility require persisted state.
Alternatives: in-memory only (rejected: no recovery); reuse AgentRun (rejected: different concept); reuse VerificationRun (rejected: release-pipeline, not DAG execution).
Impact: new SQLAlchemy models + alembic migration.
R3. Human as lifecycle control, not executor
Decision: human is a runner-lifecycle suspend/resume primitive (waiting_human + HumanCheckpoint, confirm/false_positive/inconclusive), excluded from the executor registry. PROD authorization uses a distinct ActionApprovalGate (036 mechanism, generalized owner). Executor registry covers browser/superset_api/xlsx/assertion/screenshot/report/artifact.
Rationale: A human checkpoint is a gate, not a side-effect check. Treating it as an executor would couple pause/resume into the executor contract.
Impact: ScenarioRunner.SuspendForHuman, resume_token, 036 gate reuse.
R4. RunnerPlan ownership
Decision: Derive RunnerPlan at run start from ScenarioRevision + ParameterBinding + TargetSnapshot + baselines + runtime policy. runner.plan.json is a diagnostic/reference materialization only.
Rationale: Currently an execution plan is generated with no engine. Owning the contract closes that gap.
Alternatives: consuming a saved runner.plan.json (rejected: it can diverge from revision/bindings/target); a raw unbound graph (rejected: no preflight binding).
Impact: RunnerPlan.Derive/Validate; reference artifact may be regenerated but is never execution truth.
R5. Executor reuse
Decision: Reuse 037 (metric_executor/comparison), 038 (capture), 036 (evidence/artifacts/HITL), existing browser/xlsx infra, 040 RunnerPool patterns. No second Playwright/LLM/SQL stack.
Rationale: Consistency with 038/040 rejected-path guards.
Impact: executor registry binds to existing services.
Data Model
See data-model.md: ScenarioRun, ScenarioStepRun, RunnerPlan, ScenarioExecutorRegistry, lifecycle.
Contracts & API
contracts/modules.md—Execution.Start,Execution.Dispatch,Execution.SuspendForHuman,Execution.Resume,Execution.Cancel,Execution.RetryStep,Execution.RunnerPlan.- OpenAPI:
POST /scenario-runs,POST /scenario-runs/{id}/cancel,POST /scenario-runs/{id}/resume,POST /scenario-runs/{id}/human/decision,GET /scenario-runs/{id},GET /scenario-runs/{id}/events(SSE).
Constitution Check
| Principle | Result |
|---|---|
| I. Semantic Contract First | PASS — run/step/runner contracted C4-C5 |
| II. Decision Memory | PASS — R1-R5 |
| III. External Orchestrator | PASS — executor reuse, no new Superset coupling |
| IV. Module Discipline | PASS — runner/registry/executors separated |
| V. RBAC Enforcement | PASS — scenario:run, scenario:run:prod |
| VII. Test-Driven C3+ | PASS — executor dispatch/resume/cancel tests first |
| VIII. Attention-Optimized | PASS — [SEMANTICS scenario,execution,...] |