ss-tools/specs/027-dataset-llm-orchestration/tests/reports/2026-03-17-report.md

Test Report: Superset Preview Pipeline Validation

Date: 2026-03-17
Executed by: Tester Agent (Kilo Code)

Scope

Validated the Superset preview pipeline fix set around:

• preview compilation using the real /chart/data path
• prevention of false dashboard-not-found semantics for non-dashboard preview 404 responses
• preservation of the current API response contract in backend/src/api/routes/__tests__/test_dataset_review_api.py

Files Changed

• backend/src/api/routes/__tests__/test_dataset_review_api.py
• backend/src/core/__tests__/test_superset_preview_pipeline.py

Coverage Added

API-level coverage

In backend/src/api/routes/__tests__/test_dataset_review_api.py:

• verifies preview endpoint still returns the existing preview DTO contract when preview compilation fails
• verifies failed preview payload remains generic for non-dashboard 404 behavior
• verifies error details mention the upstream resource path and do not regress into false dashboard-not-found wording

Core preview/network coverage

In backend/src/core/__tests__/test_superset_preview_pipeline.py:

• verifies SupersetClient.compile_dataset_preview calls the explicit chart-data endpoint
• verifies compiled SQL is extracted from result[].query
• verifies sync APIClient keeps non-dashboard 404s as generic SupersetAPIError
• verifies sync APIClient still translates dashboard 404s into DashboardNotFoundError
• verifies async AsyncAPIClient keeps non-dashboard 404s as generic SupersetAPIError
• verifies async AsyncAPIClient still translates dashboard 404s into DashboardNotFoundError

Semantic Audit

Ran workspace semantic audit through axiom-core.

Result

• Workspace warnings: 1341
• Audit status for this task: pre-existing global semantic debt present across workspace
• Targeted change set proceeded because no blocking semantic failure was isolated to the modified test scope

Test Commands Executed

1. Initial direct targeted run:

   • cd backend && .venv/bin/python3 -m pytest src/api/routes/__tests__/test_dataset_review_api.py src/core/__tests__/test_superset_preview_pipeline.py -q
   • Failed during collection
   • Cause: app import triggered DB init against local PostgreSQL

2. Isolated targeted run with SQLite-backed env overrides:

   • cd backend && DATABASE_URL=sqlite:///./test_app.db TASKS_DATABASE_URL=sqlite:///./test_tasks.db AUTH_DATABASE_URL=sqlite:///./test_auth.db .venv/bin/python3 -m pytest src/api/routes/__tests__/test_dataset_review_api.py src/core/__tests__/test_superset_preview_pipeline.py -q
   • Failed
   • Cause: two then one mismatched assertion(s) against actual DashboardNotFoundError message format

3. Final rerun with same SQLite overrides:

   • cd backend && DATABASE_URL=sqlite:///./test_app.db TASKS_DATABASE_URL=sqlite:///./test_tasks.db AUTH_DATABASE_URL=sqlite:///./test_auth.db .venv/bin/python3 -m pytest src/api/routes/__tests__/test_dataset_review_api.py src/core/__tests__/test_superset_preview_pipeline.py -q
   • Passed

Final Test Result

• Total targeted tests passed: 20
• Failed: 0
• Status: PASS

Notes / Remaining Limitations

• The targeted API tests require temporary DB URL overrides because importing src.app initializes the database eagerly through init_db().
• The semantic audit still reports substantial unrelated pre-existing workspace warnings.
• This change set does not add direct unit tests for SupersetCompilationAdapter.compile_preview(); coverage is achieved indirectly through API failure-contract assertions plus direct client/network tests.