ss-tools/specs/037-superset-baseline-engine/spec.md

#region SupersetBaselineEngine.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,superset,baseline,dashboard-testing] @BRIEF Superset-native query and baseline engine for dashboard testing without direct SQL execution. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0003] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0005] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RATIONALE Dashboard test assertions must validate the same Superset-side chart or dataset execution path that powers dashboards, not a parallel SQL path that can diverge from Superset filter/query semantics. @REJECTED Direct SQL execution for chart/baseline truth — rejected because this feature requires stable Superset dataset/chart execution and filter fidelity. This does not prohibit 038 validated SqlEvidenceSpec for an independent source-mart oracle.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, superset, baseline, chart-data, dataset, filters, normalization, dashboard-testing

Feature Branch: 037-superset-baseline-engine
Created: 2026-07-07 | Status: Ready for Implementation Input: "Create a Superset-native query and baseline engine for dashboard testing. The system must inspect dashboard query models, map dashboard native filters into Superset chart or dataset query context, execute Superset-side chart or dataset queries without direct SQL, normalize returned metric and table values, compare them with approved baseline catalog entries, and create draft baseline candidates requiring human approval."

User Scenarios

Story 1 — Inspect Dashboard Query Model (P1)

Why P1: Scenario generation must know charts, datasets, metrics, filters, and filter scopes before it can build reliable tests.

Independent Test: Provide a dashboard fixture and verify the engine returns structured query model data for charts, datasets, metrics, native filters, and capabilities.

Acceptance:

1. Given a dashboard id and environment When the query model is inspected Then the result lists dashboard title, charts, datasets, metric labels/keys, native filters, filter targets, and export capabilities.
2. Given a native filter applies only to selected charts When inspection runs Then filter scope is represented so unrelated charts are not queried with invalid filters.
3. Given a chart or dataset is inaccessible When inspection runs Then the engine reports a structured warning or permission error without inventing missing metadata.

---

Story 2 — Execute Superset Query Context (P1)

Why P1: Baseline validation depends on executing Superset chart/dataset queries through Superset-native APIs, never through direct SQL.

Independent Test: Execute a fixture chart query with normalized dashboard filters and verify returned data is traceable to chart id, dataset id, metric, filters hash, and environment.

Acceptance:

1. Given a chart metric and normalized filter context When execution is requested Then the engine calls the Superset-native chart/dataset execution path and returns structured values.
2. Given dashboard filters are applied When query context is built Then Superset receives semantically equivalent filters to the UI native filter state.
3. Given a request would require arbitrary SQL text from the agent When execution is attempted Then the request is rejected as unsupported.

---

Story 3 — Normalize and Compare Results (P1)

Why P1: UI, Superset API, and XLSX values must be comparable despite formatting differences.

Independent Test: Feed formatted decimal, date, percent, big-number, and table-row fixtures and verify normalized values compare consistently with tolerance rules.

Acceptance:

1. Given Superset returns a metric value When normalization runs Then the value is represented with type, raw value, normalized value, label, and source metadata.
2. Given an approved baseline exists When an actual value is compared Then exact, absolute tolerance, relative tolerance, range, or row-set comparison rules are applied according to baseline policy.
3. Given actual data cannot be normalized When comparison is requested Then the result is inconclusive with an explanation, not a false pass.

---

Story 4 — Baseline Candidate Lifecycle (P2)

Why P2: The agent may discover candidate reference values but must not silently define the truth.

Independent Test: Run discovery for a metric without baseline and verify a draft candidate is produced; approving it follows the bound inline ActionApprovalGate policy from feature 036.

Acceptance:

1. Given no approved baseline exists for metric+filters When discovery runs Then a draft baseline candidate is created with provenance and source values.
2. Given a dashboard/chart/dataset/filter fingerprint changes When existing baselines are loaded Then affected baselines are marked or reported as stale.
3. Given UI/API/XLSX sources disagree When a candidate is created Then candidate approval is blocked or warning-gated until a user reviews the discrepancy.

---

Edge Cases

• Superset returns empty result → compare result distinguishes expected empty, unexpected empty, and inconclusive.
• Filter value is not available in the target environment → query execution fails with recoverable validation error.
• Dashboard or chart metadata changes after baseline approval → fingerprint mismatch surfaces stale baseline warning.
• Superset API returns 403/404/422/5xx/timeout → taxonomy is preserved in comparison/report output.
• Date and number formats differ between locales → normalization uses canonical decimal/date representation.

Requirements

Functional

• AGBASE-FR-001: The engine MUST inspect dashboard query models into structured charts, datasets, metrics, native filters, filter scopes, and capabilities.
• AGBASE-FR-002: The engine MUST execute Superset-native chart/dataset query contexts without accepting arbitrary SQL text from the agent.
• AGBASE-FR-003: Dashboard native filters MUST be normalized into query filters with target columns, operators, values, scopes, and a deterministic filter hash.
• AGBASE-FR-004: Superset result normalization MUST support scalar metrics, big-number charts, table rows, dates, decimals, percentages, empty values, and raw/source metadata.
• AGBASE-FR-005: Baseline catalog entries MUST be pinned to a specific DashboardRelease via release_version and release_commit_hash. The release identity replaces per-field fingerprint tracking. Baseline without release pinning is invalid.
• AGBASE-FR-006: Each baseline entry MUST record a source_response_hash (SHA-256 of the Superset API response at the time the expected value was captured) and captured_at (ISO-8601 timestamp). These enable immutability violation detection independent of metric value comparison.
• AGBASE-FR-007: For closed-period entries (immutability.enabled=true), the system MUST detect immutability violations: when source_response_hash changes for the same filters, the status MUST be immutability_violation (CRITICAL severity), NOT a stale warning. Automated baseline updates for closed-period entries are forbidden.
• AGBASE-FR-008: Comparison output MUST include source, actual value, expected value, diff, status, tolerance rule, and warnings. Status enum MUST include immutability_violation as a distinct, critical category separate from stale_baseline and stale_visual_baseline.
• AGBASE-FR-009: Non-pass comparison outcomes MUST emit an idempotent 036 InvestigationSignal with immutable evidence provenance; 047 owns Queue creation/update. They MUST NOT automatically begin an agent conversation, alter a baseline, or reclassify comparison truth.
• AGBASE-FR-010: An opened InvestigationCase MAY use baseline-engine tools for read-only inspection, diagnostic comparison and candidate creation. Any baseline publication remains governed by deterministic catalog validation and its ActionApprovalGate policy.
• AGBASE-FR-009: Direct SQL execution, generated SQL assertions and SQL-based provenance are out of scope for chart/baseline truth. 038 validated immutable SqlEvidenceSpec remains the independent source-mart oracle contract and does not alter this feature's Superset-native baseline semantics.
• AGBASE-FR-010: The baseline catalog MUST support visual baselines for screenshot comparison, with the same release-pinning and immutability rules as metric baselines.
• AGBASE-FR-011: The system MUST compute a StructureDiff between two releases' DashboardQueryModel snapshots. The diff MUST classify changes by target (chart, filter, column), kind (scope_change, column_reorder, chart_removed, etc.), severity (critical, warning, info), and affected artifacts. StructureDiff is orthogonal to metric comparison.
• AGBASE-FR-012: Baseline entries for closed periods MUST carry an immutability block: enabled flag, period identifier, frozen_at timestamp, and policy (alert, block_publish, require_investigation). Immutability violations MUST be surfaced at the PROD publish gate and scheduled checks, not silently ignored.
• AGBASE-FR-013: Baseline inheritance between releases: when creating a new release, metric entries whose chart content_hash has NOT changed since the previous release inherit their expected value and source_response_hash. Entries whose chart content_hash HAS changed require re-extraction from the PREPROD environment. The analyst reviews the inheritance diff before confirming.

Key Entities

• DashboardQueryModel: Structured representation of charts, datasets, metrics, native filters, scopes, and execution capabilities for a dashboard.
• NormalizedFilterContext: Canonical filter state shared by UI, Superset query execution, scenario graph, XLSX comparison, and baseline lookup.
• SupersetQueryContext: Executable Superset-native query request derived from chart/dataset metadata plus normalized filters.
• NormalizedSupersetResult: Canonical metric/table result with type metadata and provenance.
• BaselineEntry: Release-pinned expected value for a metric/table/visual result. Pinned to release_version + release_commit_hash. Carries source_response_hash for immutability detection. Stored in baseline.yaml in the dashboard's git repository.
• StructureDiff: Structural comparison of DashboardQueryModel between two releases. Classifies changes: filter scope, column order, chart add/remove, viz_type change, group_by change. Severity: critical, warning, info. Computed without metric execution.
• ImmutabilityViolation: Critical status raised when source_response_hash for a closed-period baseline entry diverges from the recorded value. Indicates data was modified retroactively. Blocks release publication and triggers investigation.
• VisualBaseline: Approved expected visual state of a dashboard tab or region — pinned to release, with same immutability rules as metric baselines.
• ComparisonResult: Result of comparing actual normalized values to baseline expectations. Status includes immutability_violation (critical data integrity breach), stale_visual_baseline (layout/selectors diverged), and standard pass/fail/inconclusive.
• VerificationRun: Record linking an AgentRun (036) to a DashboardRelease, capturing which verification categories were executed, their outcomes, and the trigger (deploy_to_preprod, release_create, scheduled, etl_completed).

Success Criteria

• SC-001: Dashboard query model inspection returns complete chart/filter/dataset mappings for fixture dashboards with 100% deterministic JSON snapshots.
• SC-002: Superset-native execution validates at least scalar metric and table chart fixtures without direct SQL.
• SC-003: Decimal/date/percent normalization compares equivalent API/UI/XLSX formatted values without false diffs in fixture tests.
• SC-004: Baseline entries without release_version + release_commit_hash are rejected at catalog load.
• SC-005: StructureDiff correctly classifies filter scope change, column reorder, chart add/remove, and group_by change for fixture releases with 100% deterministic JSON output.
• SC-006: Immutability violation is raised within 60 seconds of detecting a changed source_response_hash for a closed-period entry in scheduled-check fixtures.

Implementation Status & Confirmed Reuse (audit 2026-08-07)

Facts (code check, not tasks.md):

• ✅ Работает: SupersetClient.ChartData.Execute (backend/src/core/superset_client/_chart_data.py) выполняет реальный async POST /api/v1/chart/data через httpx; QueryModel.Inspect, Filters.Normalize, Result.Normalize, Comparison, Candidates/Approvals, StructureDiff, Immutability — реализованы и покрыты тестами.
• ✅ Visual-слой переиспользует BaselineEngine.Visual.SSIM (pure NumPy) и BaselineEngine.Verification.ExecutorVisual.Async (authoritative fingerprints, независимость evidence-раннов). Relations зафиксированы в contracts/modules.md.
• ✅ Дискретные инструменты оценки метрик реализованы и реальны: comparison.py (exact / absolute / relative / range / row-set + immutability precedence), normalization.py (locale decimal/date/percent/big-number/table), metric_executor_async.py (полный Superset-поток: entry validation → get_superset_client → inspect_dashboard_query_model → query envelope → source_response_hash → _compare_with_baseline). API: GET /query-model, POST /filters/normalize, POST /queries/execute, POST /comparisons, GET /baselines.
• 🔴 Gap A — VerificationRun не создаётся автоматически pipeline-хуками: create_verification_run_async вызывается только из POST-роута и verification_scheduler (scheduled daily). Deploy-хук deploy_to_preprod / release_create / etl_completed НЕ создают VerificationRun (037 AGBASE-FR-012-контекст, 039 AGUI-FR-014-016 обещают это) — в git_deployment_recorder.py и deploy-маршрутах отсутствует вызов verification. Scheduled-проверка работает (core/scheduler.py → execute_scheduled_verification_check).
• 🔴 Gap B — GET-эндпоинты verification отсутствуют: frontend getVerificationHistory()/getVerificationRun() (AGUI-FR-015/016) зовут GET /dashboard-testing/verification/history и GET /dashboard-testing/verification/{runId}, но backend имеет только POST /verification-runs. Pipeline views 039 не могут загрузить историю/детали.

Закрытие: задачи T080–T081 в tasks.md Phase 10 (deploy-hook trigger + GET-эндпоинты) — дискретный контур оценки метрик работает «напрямую», но пайплайн-автоматизация и read-API не дописаны.

#endregion SupersetBaselineEngine.Spec