ss-tools/specs/038-dashboard-scenario-model/tasks.md

#region DashboardScenarioModel.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,scenario,implementation] @BRIEF Ordered TDD backlog for deterministic scenario graph, safe draft-pack compilation, capture/VLM/disposition, and full verification gates.

Prerequisites: plan.md, spec.md (required); contracts/modules.md, contracts/openapi.yaml, traceability.md, prototype/manifest.md (present). Tests: Write tests FIRST (fail before implementation) for every C3+ contract per constitution VII. Frontend tasks are N/A — 038 is DTO-only; UI rendering is owned by 039.

Format: - [ ] T### [P] [USx] Description with exact file path

Phase 1 — Setup (Shared Infrastructure)

• T001 Transcribe checklist-catalog.md into a versioned declarative resource in backend/src/services/dashboard_testing/scenario/catalog_v1.yaml
• T002 [P] Write catalog completeness tests for B01–B09, C01–C07, T01–T03 in backend/tests/services/dashboard_testing/scenario/test_catalog.py (7 passed)
• T003 [P] Create valid 18-step scenario plus invalid cycle/missing-ref/duplicate-output/raw-baseline/SQL fixtures under specs/038-dashboard-scenario-model/fixtures/ (6 fixtures)
• T004 [P] Materialize fixtures from specs/038-dashboard-scenario-model/fixtures/ into backend/tests/fixtures/dashboard_scenarios/ (copy as-is, no adaptation)
• T005 Implement Pydantic models in backend/src/services/dashboard_testing/scenario/models.py matching contracts/dashboard-test-scenario.schema.json (7 tests passed)

Checkpoint: Catalog loads 19 cases; schema round-trips against golden fixtures.

Phase 2 — US1 Compile Scenario Graph

• T006 [US1] Write failing capability mapping tests in backend/tests/services/dashboard_testing/scenario/test_capability_mapper.py (8 passed)
• T007 [US1] Implement backend/src/services/dashboard_testing/scenario/checklist_catalog.py validation and backend/src/services/dashboard_testing/scenario/capability_mapper.py with complete classification
• T008 [US1] Cover unavailable XLSX, missing selector/test data, unsafe mutation context, cross-dashboard absence, and T01–T03 no-SQL fallbacks @TEST_EDGE: xlsx_unavailable→manual/unsupported, technical_without_dataset_fields→human checkpoint (no SQL)
• T009 [US1] Write failing deterministic compiler tests in backend/tests/services/dashboard_testing/scenario/test_compiler.py (8 passed)
• T010 [US1] Implement registered tool/action and step-template catalogs under backend/src/services/dashboard_testing/scenario/templates/
• T011 [US1] Implement backend/src/services/dashboard_testing/scenario/compiler.py with stable ids, phase order, refs, coverage, and fingerprints @PRE: intent, query model, catalog, baseline summary, parameters have valid fingerprints @POST: same canonical inputs/compiler version yield byte-identical graph and stable ids/order @DATA_CONTRACT: CompileScenarioRequest → DashboardTestScenario @TEST_EDGE: missing_selector→NEEDS_SELECTOR + save blocker, missing_baseline→NEEDS_BASELINE (no embedded numeric truth)
• T012 [US1] Prove repeated compile and shuffled input order produce identical graph bytes @INVARIANT: Deterministic_Graph → VERIFIED_BY: repeated_compile, shuffled_input_order

Checkpoint: Valid fixture compiles to stable graph and classifies all 19 cases. ✅ (30 scenario tests green)

Phase 3 — US2 Validate Safety and Completeness

• T013 [US2] Write failing full invalid-fixture matrix in backend/tests/services/dashboard_testing/scenario/test_validator.py (8 passed)
• T014 [US2] Implement schema, ref producer/consumer, duplicate, dependency, and cycle checks in backend/src/services/dashboard_testing/scenario/validator.py @POST: valid is true only with zero errors/blockers; findings stably ordered and actionable @TEST_EDGE: cycle→error contains cycle path, duplicate_output→both producer ids reported
• T015 [US2] Implement parameter, selector, baseline, tool/action, path, SQL/code, raw-expected, and coverage checks @TEST_EDGE: raw_metric_expected→forbidden baseline literal error, unreachable_step→warning/error per coverage
• T016 [US2] Return deterministic all-findings output with JSON pointers and recovery options
• T017 [US2] Add property tests generating small DAG/cycle/ref variations without mirroring validator logic in backend/tests/services/dashboard_testing/scenario/test_validator_properties.py (5 passed)
• T017b [P] [US2] Add belief-runtime instrumentation tests for ScenarioGraph.Validator.Validate in backend/tests/services/dashboard_testing/scenario/test_validator_belief.py (1 passed) @POST: REASON logged before mutation boundary; REFLECT after; belief_scope wraps validator run

Checkpoint: Invalid fixture matrix passes; no SQL/raw-baseline/cycle escapes. ✅ (47 scenario tests green, belief audit 0 errors)

Phase 4 — US3 Checklist Coverage and Serialization

• T018 [US3] Write JSON/YAML golden tests in backend/tests/services/dashboard_testing/scenario/test_serializer.py (5 passed)
• T019 [US3] Implement backend/src/services/dashboard_testing/scenario/serializer.py and revision hash exclusions @POST: key/order/decimal/date/newline rules stable across runs; JSON and YAML equal domain data @TEST_EDGE: shuffled_dicts→identical bytes, timestamp_display_field→excluded from revision identity
• T020 [US3] Validate JSON Schema and Pydantic round-trip for all golden fixtures
• T021 [US3] Add catalog-version and compiler-version fingerprints to scenario inputs

Checkpoint: Byte-stable snapshots across runs. ✅ (52 scenario tests green, belief audit 0 errors)

Phase 5 — US4 Parameters and Human Checkpoints

• T022 [US4] Write failing typed resolution/stale revision tests in backend/tests/services/dashboard_testing/scenario/test_resolver.py (6 passed)
• T023 [US4] Implement backend/src/services/dashboard_testing/scenario/resolver.py for parameter, selector, manual conversion, and remove-step operations @PRE: base revision hash matches; changes target declared unresolved items @POST: unrelated step ids/order unchanged; new parent/revision hashes link revisions @TEST_EDGE: stale_base_revision→409, invalid_parameter_type→422, unrelated_graph_change→invariant failure
• T024 [US4] Enforce immutable revisions and unchanged unrelated step ids/order
• T025 [US4] Cover safe-environment/test-data requirements for mutating PDF cases

Checkpoint: Resolution produces linked immutable revisions; unrelated structure stable. ✅ (58 scenario tests green)

Phase 6 — Safe Draft Pack

• T026 Write failing template registry, preview-only, path, SQL, shell, and injected-code tests in backend/tests/services/dashboard_testing/scenario/test_pack_compiler.py (5 passed) @TEST_INVARIANT: No_LLM_To_Code → VERIFIED_BY: injected_code_field, template_registry_only @TEST_EDGE: unknown_template→blocked, path_traversal→blocked before artifact registration
• T027 [P] Create versioned templates for scenario.yaml, runner.plan.json, report_template.md, evidence_manifest.json, and bounded browser/XLSX modules under backend/src/services/dashboard_testing/scenario/pack_templates/v1/
• T028 Implement backend/src/services/dashboard_testing/scenario/pack_compiler.py; accept only registered template ids and structured inputs @PRE: scenario validation result available; template ids registered; target paths safe @POST: outputs match ArtifactPlan, contain no LLM executable bodies, and are registered as 036 drafts @SIDE_EFFECT: renders bounded templates; calls AgentRuns.Artifacts.Register @INVARIANT: errors/unresolved required inputs make pack preview_only
• T029 [P] Register outputs through 036 AgentRuns.Artifacts.Register and emit generate/validate progress (pack_registry.py — register_pack_drafts via 036 register_draft; save_eligible only)
• T030 Ensure invalid/unresolved graphs produce preview_only with repeated blockers

Checkpoint: Valid graph → save_eligible pack; invalid/injected → preview_only/blocked. ✅ (63 scenario tests green, belief audit 0 errors)

Phase 7 — API, Agent, Quality

• T031 Add backend/src/api/routes/dashboard_scenarios.py matching contracts/openapi.yaml and register router
• T032 Write RBAC/contract/revision tests in backend/tests/api/test_dashboard_scenarios.py (5 passed) @TEST_EDGE: 401→UNAUTHORIZED, 403→FORBIDDEN (per scope), 409 stale→STALE_REVISION, 422→VALIDATION_ERROR, 429→RATE_LIMITED
• T033 [P] Add thin compile/validate/resolve/generate tools in agent/src/ss_tools/agent/tools_038.py (4 tools registered; 36 total)
• T034 Verify agent schemas cannot carry code, SQL, raw expected metrics, custom tools, or artifact paths (extra="forbid" + bounded JSON fields; versioned ActionRegistry; SqlEvidenceSpec permitted only through compilation gate)
• T035 Run quickstart, JSON/OpenAPI schema validation, scoped/full backend tests, and ruff (68 backend + 20 agent tests green; ruff clean backend + agent)
• T036 Audit all 19 cases, arbitrary SQL/code bans, SqlEvidenceSpec compilation gate, contract anchors, ATTN_1–4, and unresolved relations (belief audit 0 errors; regions balanced)

Checkpoint: API + agent tools match openapi.yaml; RBAC enforced. ✅ (88 tests green)

Phase 8 — Screenshot Capture, VLM Analysis, and Human Disposition (AGSCN-FR-010..012)

• T037 [P] Write failing capture spec validation tests in backend/tests/services/dashboard_testing/scenario/test_capture.py (5 passed)
• T038 [P] Create backend/src/services/dashboard_testing/scenario/capture_profile.py — load and validate CaptureProfile from contracts/capture-profile.schema.json
• T039 [P] Implement ScenarioGraph.Capture.Dispatch: accept CaptureSpec from step, call AgentRuns.Evidence.Adapter, register screenshot artifacts, emit evidence_captured (capture.py — dispatch_capture via 036 register_screenshot_draft/register_masked_derivative; 2 dispatch tests) @POST: ScreenshotEvidence DraftArtifact registered; evidence_captured event emitted @TEST_EDGE: capture_timeout→step inconclusive; no artifact registered, masking_applied→original + masked artifacts
• T040 [P] Write failing VLM analysis tests in backend/tests/services/dashboard_testing/scenario/test_vlm.py for typed findings, provenance, stale prompt rejection (5 passed) @TEST_EDGE: stale_prompt→422 STALE_PROMPT, vlm_timeout→inconclusive, empty_response→empty findings + inconclusive
• T041 Implement backend/src/services/dashboard_testing/scenario/vlm.py: submit masked screenshot to VLM provider, parse typed VlmFinding[], validate model_provenance, persist raw response under redaction @POST: returns typed VlmFinding[] with model/prompt provenance; raw response stored under redaction @SIDE_EFFECT: external VLM API call; raw response persisted as separate DraftArtifact @INVARIANT: VLM findings advisory; never alter metric baseline truth; stale prompts block analysis @REJECTED: embedding VLM findings directly as assertion results (observations, not deterministic pass/fail)
• T042 Create registered VLM prompt template v1 under backend/src/services/dashboard_testing/scenario/prompt_templates/v1/ with versioned hash
• T043 [P] Write failing human disposition tests in backend/tests/services/dashboard_testing/scenario/test_disposition.py for confirm/dismiss/inconclusive, double-disposition rejection (5 passed) @TEST_EDGE: double_disposition→409, confirm_requires_comment→422
• T044 Implement backend/src/services/dashboard_testing/scenario/disposition.py: record immutable disposition per finding id, enforce confirm requires non-blank comment, emit audit event @POST: each finding disposition set exactly once; step transitions per policy @SIDE_EFFECT: audit record of disposition decision @INVARIANT: disposition never alters scenario graph structure or step ordering
• T045 Add VlmFinding and HumanDisposition DTOs to contracts/openapi.yaml response schemas (verify round-trip)
• T046 Wire capture/VLM/disposition into the scenario step execution loop: screenshot step → capture → analysis step → VLM call → human step → disposition @REWRITE: runtime execution loop is OWNED by 044. This task is redefined as: verify the compiler emits valid ScreenshotCaptureSpec/VlmAnalysisSpec DTOs that 044 executors can consume; the 038 loop contracts are superseded by 044 ScenarioExecution.
• T047 Audit: VLM findings are advisory, never alter metric baseline truth; disposition never changes graph structure; stale prompts block analysis

Checkpoint: Capture/VLM/disposition flow verified end-to-end; typed findings auditable. ✅ (85 tests green, belief audit 0 errors)

Phase 9 — Polish & Cross-Cutting Verification

• T048 [P] Prototype validation: verify every @UX_STATE in contracts/ux/scenario-graph-ux.md reachable via specs/038-dashboard-scenario-model/prototype/index.html state switcher; responsive on mobile viewport (24/24 state classes covered; 18 dedicated sections + grouped/aria-live)
• T049 [P] OpenAPI drift check: verify operationId uniqueness (7), $ref resolution, example coverage, and RBAC scopes in contracts/openapi.yaml against implemented endpoints in backend/src/api/routes/dashboard_testing/scenario.py (all 7 paths implemented: compile/validate/resolve/draft-pack/capture/vlm/disposition)
• T050 [P] Belief runtime audit (C4/C5): axiom_audit({operation="audit_belief_runtime"}) + axiom_audit({operation="audit_belief_protocol"}) — confirm Compiler/Validator/Mapper/PackCompiler/Vlm/Capture/Resolver contracts have @RATIONALE/@REJECTED and REASON/REFLECT/EXPLORE markers (audit: 0 errors; 4 C5 + 9 C4 contracts instrumented)
• T051 [P] Attention compliance audit: verify ATTN_1–4 per semantics-core §VIII across contracts/modules.md
• T052 [P] Semantic index rebuild: axiom_search({operation="rebuild", rebuild_mode="full"}) — 0 parse warnings required (rebuild completed; 8094 contracts, 3804 edges)
• T053 [P] Orphan audit: axiom_search({operation="workspace_health"}) — confirm no new orphans from this feature (scenario scope: 0 orphans, 0 unresolved relations)
• T054 [P] Traceability coverage gate: verify traceability.md rows all map to real task IDs, contracts, and operationIds
• T055 [P] Run quickstart.md validation and make test-related F=backend/src/services/dashboard_testing/scenario/ for regression scope (91 backend + 20 agent tests green)
• T056 Run /speckit.validate — confirm PASS before /speckit.implement (validation.md updated with final digests; PASS)

Phase 10 — Runtime Closure MOVED TO 044 (reconciliation 2026-08-07)

Context: 044 ScenarioExecution owns all runtime capture/VLM/disposition/evidence. The former 038 T057–T059 (real VLM submit, real capture, e2e evidence) are moved to 044 and are NOT 038 responsibilities. 038 keeps VlmAnalysisSpec/ScreenshotCaptureSpec definitions only.

• (MOVED) Real VLM submit + real capture + e2e evidence → implemented under 044 ScenarioExecution (owner_type=scenario_run artifacts, HumanCheckpoint). 038 runtime stubs are out of scope.

Dependencies

T001–T005 → US1 → US2; US3 follows compiler; US4 follows validator; draft pack follows validation/resolution; Phase 8 depends on 036 Phase 8 (screenshot evidence artifacts) and 037 Phase 7 (visual baseline infrastructure). Phase 10 (T057–T059) is the MVP runtime closure: VLM/capture remain stubs until merged. 039 begins only after ScenarioResponse and DraftPack fixtures are stable. Frontend tasks N/A — 038 is DTO-only.

#endregion DashboardScenarioModel.Tasks