ss-tools/specs/038-dashboard-scenario-model/spec.md

#region DashboardScenarioModel.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,scenario,dashboard-testing] @BRIEF Define the validated immutable Verification Program IR for dashboard test flows authored by agents and executed by 044. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0002] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec] @RATIONALE Unique dashboard tests need a stable intermediate model between agent reasoning and generated artifacts; direct LLM-to-code generation is not reviewable or safely composable. @REJECTED Asking users to choose low-level outputs such as Playwright vs SQL vs XLSX — rejected because each dashboard scenario is goal-oriented; the agent proposes inspectable compiled evidence/transform/assertion steps. @REJECTED Direct generation of executable scripts without a validated scenario graph — rejected because it hides missing selectors, baseline refs, and unsafe steps until runtime.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, scenario, graph, dashboard-testing, checklist, validation, artifacts

Feature Branch: 038-dashboard-scenario-model Created: 2026-07-07 | Reworked: 2026-07-31 | Status: Reworked per new speckit flow Input: "Define the dashboard test scenario model used by agents to represent unique dashboard test flows as a validated ScenarioGraph. The model must express ordered and dependent steps across browser automation, Superset query execution, XLSX parsing, assertions, screenshots, reports, human checkpoints, baseline references, parameters, warnings, and missing context markers without exposing users to low-level tool selection."

Applicability

• Feature type: Fullstack (backend compiler/validator/resolver core + thin UI preview surface consumed by 039 + agent tools).
• UI surface: Yes — scenario graph preview, coverage, resolution, and pack states (rendered by 039; 038 supplies DTO contracts).
• API surface: Yes — compile, validate, resolve, draft-pack, capture, VLM, disposition endpoints.
• Prototype: Applicable — scenario preview is a real UI surface (see prototype/index.html).
• OpenAPI: Applicable — REST surface is a first-class deliverable (see contracts/openapi.yaml).

User Scenarios

Story 1 — Build Scenario Graph From Dashboard Goal (P1)

Why P1: The agent must express a dashboard testing goal as a reviewable graph of steps before generating any executable artifacts.

Independent Test: Provide dashboard query model and checklist fixture input and verify a deterministic DashboardTestScenario graph is produced.

Acceptance:

1. Given a dashboard context, ChangeRequestContext, query model, and testing objective When the agent builds a scenario Then the output contains an inspectable Verification Program, parameters, steps, dependencies, evidence sources, warnings, and risk summary (038 does not emit persistence ids).
2. Given a scenario needs browser, Superset API, XLSX parsing, screenshots, assertions, and report steps When graph is rendered Then every step declares its tool and input/output refs.
3. Given required metadata is missing When graph is generated Then missing data is represented as NEEDS_CONTEXT or NEEDS_SELECTOR, not invented.

---

Story 2 — Validate Scenario Graph Safety and Completeness (P1)

Why P1: Scenario artifacts must be generated only from a graph whose refs, baselines, tools, and risks are internally consistent.

Independent Test: Run valid and invalid scenario fixtures through the validator and verify precise errors for broken refs, missing baseline refs, and unsupported steps.

Acceptance:

1. Given a step consumes an output ref When validation runs Then the referenced producer must exist and precede or depend correctly.
2. Given an assertion compares a metric to expected truth When validation runs Then it must reference an approved baseline or a draft baseline candidate with explicit approval requirement.
3. Given a step uses an unsupported tool or unsafe action When validation runs Then the graph is rejected or marked blocked with a user-visible reason.

---

Story 3 — Map Checklist Cases to Scenario Capabilities (P2)

Why P2: The PDF checklist should guide scenario coverage without forcing all dashboards into identical scripts.

Independent Test: Normalize the research checklist into capability-tagged cases and verify generated scenarios include applicable cases and mark non-applicable ones.

Acceptance:

1. Given the normalized checklist template contains basic, complex, and technical cases When a dashboard capability model is supplied Then applicable cases are mapped to scenario steps or human checkpoints.
2. Given a checklist case requires capabilities absent from the dashboard When mapping runs Then the case is marked unsupported or manual-only with rationale.
3. Given multiple tool choices could verify a case When mapping runs Then the scenario selects inspectable navigation/evidence/transform/assertion steps; source-mart evidence may be a validated SqlEvidenceSpec.

---

Story 4 — Represent Parameters and Human Checkpoints (P2)

Why P2: Unique dashboard scenarios require business inputs and sometimes cannot be fully automated.

Independent Test: Generate a scenario that requires test date, counterparty, and baseline selection; verify parameter prompts and human checkpoint steps are represented structurally.

Acceptance:

1. Given a scenario needs test data When graph is generated Then required parameters include name, type, validation rule, default/source, and affected steps.
2. Given an action cannot be automated reliably When graph is generated Then a human checkpoint step describes the manual action and expected evidence.
3. Given a reusable scenario has a required ParameterDefinition without a default When it is saved Then it remains save-eligible; a later 044 launch binds the value without changing the graph or content_hash.

---

Story 5 — Capture, VLM Analysis, and Human Disposition (P2)

Why P2: Screenshot evidence and visual verification require typed, auditable capture/VLM/disposition semantics (AGSCN-FR-010..012).

Independent Test: Generate a scenario with screenshot capture, VLM analysis, and human disposition; verify typed findings and auditable dispositions.

Acceptance:

1. Given a screenshot step When capture executes Then a reproducible ScreenshotCaptureSpec (target, viewport, readiness, masking, max wait) is honored and artifacts are registered.
2. Given a masked screenshot When VLM analysis runs Then typed VlmFinding[] (severity, region, confidence, model/prompt provenance) are returned; raw prose is never treated as step state.
3. Given a human checkpoint references VLM finding ids When the user disposes Then confirm/false_positive/inconclusive is typed and auditable, and disposition never mutates the graph structure.

Edge & Failure Cases

#	Scenario	Category	Expected Behavior	Recovery / Test Ownership
E1	Scenario has a cycle in dependencies	data-integrity	Validator rejects with cycle path	User fixes graph; L1 validator test
E2	Two steps produce the same output ref	data-integrity	Validator rejects ambiguous ref	User fixes ref; L1 test
E3	Baseline is stale	data-quality	Assertion stays present but blocked/warning-gated	037 baseline discovery or mark pending; L1 test
E4	XLSX export unavailable	integration	XLSX-dependent checklist cases become unsupported or manual checkpoints	Rationale shown in coverage; L1 mapping test
E5	UI selector unknown	integration	Browser step uses NEEDS_SELECTOR and blocks executable generation	User provides selector hint / converts to checkpoint; L1 test
E6	409 stale base revision on resolve	concurrency	New revision rejected with 409; snapshot/recompile guidance; never silent merge	User recompiles; L1 API test
E7	422 invalid resolution/parameter type	validation	Field/step-mapped validation error	User corrects input; L1 API test
E8	403 forbidden role on scenario operations	auth	Permission denial rendered without approval gate	User contacts admin / RBAC test
E9	429 rate limit on compile/validate	throttling	Retry-After honored; UI countdown	User waits; L2 UX test
E10	5xx backend failure on compile	server-error	Error section + retry; partial graph not persisted	User retries; L2 UX test
E11	Malformed VLM response / empty findings	integration	Findings array empty; step inconclusive with reason; stale prompt blocked (422 STALE_PROMPT)	Re-run analysis; L1 VLM test
E12	Missing runtime parameter at launch	data-quality	Saved pack remains save_eligible; 044 RunPreflight rejects only that launch	Supply a typed binding; L1 preflight test
E13	Unsafe path / executable code / SQL injection into pack	security	SQL compilation gate blocks unsafe SQL; arbitrary code/path validation blocks before draft registration	L1 security test; injected-code fixture
E14	Duplicate submit of draft-pack	idempotency	Idempotency key / revision hash prevents double registration	L1 API test; 409 on changed revision

Requirements

Functional

• AGSCN-FR-001: The system MUST define a DashboardTestScenario model with dashboard context, objective, parameters, steps, dependencies, outputs, artifacts, risks, and warnings.
• AGSCN-FR-002: Every scenario step MUST declare tool category, action, inputs, outputs, expected result, dependencies, and automation status.
• AGSCN-FR-003: Supported tool categories MUST include browser automation, Superset API execution, XLSX parsing, assertion, screenshot/evidence, report generation, artifact generation, and human checkpoint.
• AGSCN-FR-003a: DashboardTestScenario MUST contain a first-class, content-hashed Verification Program with navigation, evidence, transformation, assertion and semantic-evaluation programs. It is immutable runtime input, not a runtime planning hint.
• AGSCN-FR-004: Assertions against reference values MUST use baseline references or baseline candidate references; raw expected numbers MUST NOT be embedded directly in executable steps.
• AGSCN-FR-005: Authoring validation MUST detect missing refs, cycles, duplicate outputs, missing/stale baselines, unknown selectors, unsupported tools, and invalid ParameterDefinitions. Required runtime bindings are enforced only by 044 RunPreflight.
• AGSCN-FR-006: Checklist mapping MUST use normalized checklist cases derived from the research PDF and capability tags, not hardcoded one-size-fits-all scripts. Mapping accepts the target release_version for baseline lookup.
• AGSCN-FR-007: The model MUST allow manual/human checkpoint steps where automation is unsafe, unavailable, or underspecified.
• AGSCN-FR-008: Scenario output MUST be deterministic for the same dashboard query model, checklist template, baseline catalog, and user parameters.
• AGSCN-FR-009: The scenario model MUST remain implementation-neutral and must not require the user to choose low-level artifacts such as Playwright, XLSX, or API output upfront.
• AGSCN-FR-010: Screenshot steps MUST carry a capture SPECIFICATION: target (tab/viewport), viewport dimensions, readiness strategy, masking selectors, and max wait. Execution of capture is owned by 044 (ScenarioExecution CaptureService delegating to Plugin.Service.ScreenshotService), with artifacts owner_type=scenario_run; 038 defines the spec, not the runtime path.
• AGSCN-FR-011: Visual-analysis steps MUST carry a typed VlmAnalysisSpec (profile/provider/model/prompt template/hash/confidence). Runtime VLM submission and VlmFinding production are owned by 044, reusing Plugin.Service.LLMClient resolved through Services.LlmProvider.LLMProviderService (multimodal-required, encrypted-key handling, JSON mode) and redacting raw responses via Plugin.Service.RedactionService. A stub/default submit returning empty findings without a real provider call is incomplete.
• AGSCN-FR-011a: Read-only SqlEvidenceSpec MAY be authored during creation/edit/revalidation or investigation proposal only. Save MUST require AST/policy/schema/preview validation. A ScenarioRun executes exactly the saved SQL template via the Superset SQL Lab adapter with typed bindings; runtime LLM SQL rewrite, relation/projection/join/filter mutation and credential handoff are forbidden.
• AGSCN-FR-011b: TransformSpec MUST use only the bounded versioned DSL and ComparisonSpec/AssertionSpec MUST compare declared evidence refs. Arbitrary Python/code is forbidden.
• AGSCN-FR-011c: AgentEvaluationSpec MAY cover only declared semantic/visual/ambiguous checks. It MUST pin model/prompt/evidence/input/tool access/output schema and DecisionPolicy; it cannot alter graph, SQL/DSL, orchestration, lifecycle or mutations.
• AGSCN-FR-011d: Authoring MUST accept first-class ChangeRequestContext; the compiler must mark missing needed context as needs_context, never guess it.
• AGSCN-FR-012: Human checkpoint steps MAY reference specific VLM finding ids. Resolution options (confirm, false_positive, inconclusive) MUST be typed and auditable — this is a 044 HumanCheckpoint, distinct from the 036 authorization ActionApprovalGate. Disposition changes finding status, not graph structure.
• AGSCN-FR-013: The agent MAY plan scenario coverage, resolve ambiguity, and create a validated WorkingDraft or executable revision when delegated policy permits. Every resulting graph MUST pass the deterministic 038 validator and retain canonical immutable provenance.
• AGSCN-FR-014: Graph authoring and revision actions MUST be available in the persistent scenario workspace or agent thread; modal/dialog interaction MUST NOT be required for authoring, review, conflict recovery, or approval.

Key Entities

• DashboardTestScenario: Reviewable graph representing a dashboard-specific testing objective and all steps required to validate it.
• ScenarioStep: One executable, generated, assertion, evidence, or human checkpoint node in the graph.
• ParameterDefinition: Immutable name/type/default/validation/affected-step declaration; launch values are 044 ParameterBindings and never affect content_hash.
• VerificationProgram: Content-hashed navigation/evidence/transform/assertion/semantic-evaluation IR.
• SqlEvidenceSpec: Validated immutable read-only source-mart evidence query.
• ScenarioRef: Named output produced by one step and consumed by later steps.
• ChecklistCase: Normalized item from the research checklist with capability tags and expected verification semantics.
• CapabilityMapping: Decision record mapping dashboard capabilities to applicable checklist cases and step templates.
• ScenarioValidationResult: Structured validator output with errors, warnings, blockers, and graph coverage.
• ScreenshotCaptureSpec: Capture CONFIGURATION for a screenshot step — viewport, target tab, readiness strategy, masking selectors, and timeouts. (Execution owned by 044.)
• VlmAnalysisSpec: Typed SPEC of visual analysis — provider/model/prompt/hash/confidence. (Runtime VlmFinding owned by 044.)
• HumanDisposition: Typed human decision on a VLM finding — confirm, false_positive, or inconclusive (044 HumanCheckpoint).

Success Criteria

• SC-001: Scenario validator catches 100% of invalid fixture cases for missing refs, cycles, duplicate outputs, and missing baseline refs.
• SC-002: At least 80% of normalized PDF checklist cases are classifiable as automated, human checkpoint, unsupported, or needs-context for fixture dashboards.
• SC-003: Same inputs produce byte-stable scenario JSON/YAML in deterministic snapshot tests.
• SC-004: No generated scenario fixture embeds raw baseline numbers directly in executable steps.
• SC-005: Scenario graph preview can display phase order, tools per step, parameters, warnings, and blockers without reading generated code.
• SC-006: VLM findings are advisory only; disposition is auditable; no finding alters metric baseline truth.

Clarifications

Session 2026-07-31

• Q: Is the scenario model a backend-only library or does it expose a UI/API surface? → A: Fullstack — backend compiler/validator core plus thin UI preview (rendered by 039) plus REST/agent-tool API surface.
• Q: How are missing selectors/context handled? → A: Represented structurally as NEEDS_SELECTOR/NEEDS_CONTEXT, never invented; blocks executable generation for that step.
• Q: What is the determinism contract? → A: Byte-stable output for identical canonical inputs + compiler/template versions; stable derived ids, no random UUIDs in canonical graph; temperature=0 alone is rejected as a determinism mechanism.
• Q: What is the VLM safety boundary? → A: VLM output is typed, advisory findings for human review; raw prose is never step state and findings never alter baseline truth; stale prompts block analysis.
• Q: Is the draft-pack compiled through templates or direct code generation? → A: Versioned repository-owned templates only; LLM text may populate bounded descriptions but never executable code bodies, paths, imports, or shell commands.

Implementation Status & Cross-Spec Reconciliation (audit 2026-08-07)

Facts (code check):

• ✅ Ядро работает: deterministic compiler/validator/resolver/serializer/pack, checklist-каталог (19 cases), REST surface (compile/validate/resolve/draft-pack).
• 🔴 Runtime capture/VLM/disposition теперь принадлежат 044, НЕ 038. Прежние задачи T057–T059 (real VLM submit, real capture, e2e evidence) переносятся в 044 ScenarioExecution; 038 владеет только VlmAnalysisSpec/ScreenshotCaptureSpec и компилятором.
• 🔴 Identity reconciliation: 038 эмитит scenario_key + content_hash; scenario_id (UUID) и revision_id (UUID) назначает 042 при Save (CreateScenario). Компилятор не выдаёт identity до persistence.

Закрытие: cross-spec pass привёл 038 к роли чистого IR/compiler слоя; все runtime-контуры (execution, evidence owner_type=scenario_run, HumanCheckpoint) закрываются 044. Отдельный 038/validation.md PASS аннулирован как self-contradictory — см. validation.md.

#endregion DashboardScenarioModel.Spec