busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Таски готовы

Browse Source
This commit is contained in:
busya
2026-03-16 23:11:19 +03:00
parent 493a73827a
commit 9cae07a3b4
24 changed files with 10614 additions and 8733 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
.ai/MODULE_MAP.md
View File

@@ -2,12 +2,12 @@
> High-level module structure for AI Context. Generated automatically. > High-level module structure for AI Context. Generated automatically.
**Generated:** 2026-03-16T10:03:28.287790 **Generated:** 2026-03-16T22:51:06.491000
## Summary ## Summary
- **Total Modules:** 105 - **Total Modules:** 105
- **Total Entities:** 3313 - **Total Entities:** 3358
## Module Hierarchy ## Module Hierarchy
@@ -20,27 +20,34 @@
**Key Entities:** **Key Entities:**
- 📦 **backend.delete_running_tasks** (Module) `[TRIVIAL]` - 📦 **DeleteRunningTasksUtil** (Module) `[TRIVIAL]`
- Script to delete tasks with RUNNING status from the database... - Script to delete tasks with RUNNING status from the database...
**Dependencies:**
- 🔗 DEPENDS_ON -> TaskRecord
- 🔗 DEPENDS_ON -> TasksSessionLocal
### 📁 `src/` ### 📁 `src/`
- 🏗️ **Layers:** API, Core, UI (API) - 🏗️ **Layers:** API, Core, UI (API)
- 📊 **Tiers:** CRITICAL: 2, STANDARD: 6, TRIVIAL: 18 - 📊 **Tiers:** CRITICAL: 2, STANDARD: 6, TRIVIAL: 20
- 📄 **Files:** 3 - 📄 **Files:** 3
- 📦 **Entities:** 26 - 📦 **Entities:** 28
**Key Entities:** **Key Entities:**
- 📦 **AppDependencies** (Module)
- Manages creation and provision of shared application depende...
- 📦 **AppModule** (Module) `[CRITICAL]` - 📦 **AppModule** (Module) `[CRITICAL]`
- The main entry point for the FastAPI application. It initial... - The main entry point for the FastAPI application. It initial...
- 📦 **backend.src.dependencies** (Module) - 📦 **SrcRoot** (Module) `[TRIVIAL]`
- Manages creation and provision of shared application depende... - Canonical backend package root for application, scripts, and...
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> AppDependencies
- 🔗 DEPENDS_ON -> backend.src.api.routes - 🔗 DEPENDS_ON -> backend.src.api.routes
- 🔗 DEPENDS_ON -> backend.src.dependencies
### 📁 `api/` ### 📁 `api/`
@@ -51,18 +58,25 @@
**Key Entities:** **Key Entities:**
- 📦 **backend.src.api.auth** (Module) - 📦 **AuthApi** (Module)
- Authentication API endpoints. - Authentication API endpoints.
**Dependencies:**
- 🔗 DEPENDS_ON -> AuthRepository:Class
- 🔗 DEPENDS_ON -> get_current_user
### 📁 `routes/` ### 📁 `routes/`
- 🏗️ **Layers:** API, Infra, UI (API), UI/API - 🏗️ **Layers:** API, Infra, UI (API), UI/API
- 📊 **Tiers:** CRITICAL: 7, STANDARD: 184, TRIVIAL: 111 - 📊 **Tiers:** CRITICAL: 7, STANDARD: 191, TRIVIAL: 107
- 📄 **Files:** 21 - 📄 **Files:** 21
- 📦 **Entities:** 302 - 📦 **Entities:** 305
**Key Entities:** **Key Entities:**
- ℂ **ApprovalRequest** (Class) `[TRIVIAL]`
- Schema for approval request payload.
- ℂ **AssistantAction** (Class) `[TRIVIAL]` - ℂ **AssistantAction** (Class) `[TRIVIAL]`
- UI action descriptor returned with assistant responses. - UI action descriptor returned with assistant responses.
- ℂ **AssistantMessageRequest** (Class) `[TRIVIAL]` - ℂ **AssistantMessageRequest** (Class) `[TRIVIAL]`
@@ -81,46 +95,39 @@
- Schema for staging and committing changes. - Schema for staging and committing changes.
- ℂ **CommitSchema** (Class) `[TRIVIAL]` - ℂ **CommitSchema** (Class) `[TRIVIAL]`
- Schema for representing Git commit details. - Schema for representing Git commit details.
- ℂ **ConfirmationRecord** (Class)
- In-memory confirmation token model for risky operation dispa...
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> AppDependencies
- 🔗 DEPENDS_ON -> backend.src.core.config_manager.ConfigManager - 🔗 DEPENDS_ON -> backend.src.core.config_manager.ConfigManager
- 🔗 DEPENDS_ON -> backend.src.core.config_models - 🔗 DEPENDS_ON -> backend.src.core.config_models
- 🔗 DEPENDS_ON -> backend.src.core.database - 🔗 DEPENDS_ON -> backend.src.core.database
- 🔗 DEPENDS_ON -> backend.src.core.database.get_db - 🔗 DEPENDS_ON -> backend.src.core.database.get_db
- 🔗 DEPENDS_ON -> backend.src.core.mapping_service.IdMappingService
### 📁 `__tests__/` ### 📁 `__tests__/`
- 🏗️ **Layers:** API, Domain, Domain (Tests), Tests, UI (API Tests), Unknown - 🏗️ **Layers:** API, Domain, Domain (Tests), Tests, UI (API Tests), Unknown
- 📊 **Tiers:** STANDARD: 16, TRIVIAL: 275 - 📊 **Tiers:** STANDARD: 16, TRIVIAL: 265
- 📄 **Files:** 18 - 📄 **Files:** 18
- 📦 **Entities:** 291 - 📦 **Entities:** 281
**Key Entities:** **Key Entities:**
- ℂ **_FakeConfigManager** (Class) `[TRIVIAL]` - ℂ **_FakeConfigManager** (Class) `[TRIVIAL]`
- Provide deterministic environment aliases required by intent... - Provide deterministic environment aliases required by intent...
- ℂ **_FakeConfigManager** (Class) `[TRIVIAL]` - ℂ **_FakeConfigManager** (Class) `[TRIVIAL]`
- Environment config fixture with dev/prod aliases for parser ...
- ℂ **_FakeDb** (Class) `[TRIVIAL]` - ℂ **_FakeDb** (Class) `[TRIVIAL]`
- In-memory session substitute for assistant route persistence... - In-memory session substitute for assistant route persistence...
- ℂ **_FakeDb** (Class) `[TRIVIAL]` - ℂ **_FakeDb** (Class) `[TRIVIAL]`
- In-memory fake database implementing subset of Session inter...
- ℂ **_FakeQuery** (Class) `[TRIVIAL]` - ℂ **_FakeQuery** (Class) `[TRIVIAL]`
- Minimal chainable query object for fake DB interactions. - Minimal chainable query object for fake DB interactions.
- ℂ **_FakeQuery** (Class) `[TRIVIAL]` - ℂ **_FakeQuery** (Class) `[TRIVIAL]`
- Minimal chainable query object for fake SQLAlchemy-like DB b...
- ℂ **_FakeTask** (Class) `[TRIVIAL]` - ℂ **_FakeTask** (Class) `[TRIVIAL]`
- Lightweight task model used for assistant authz tests. - Lightweight task model used for assistant authz tests.
- ℂ **_FakeTask** (Class) `[TRIVIAL]` - ℂ **_FakeTask** (Class) `[TRIVIAL]`
- Lightweight task stub used by assistant API tests.
- ℂ **_FakeTaskManager** (Class) `[TRIVIAL]` - ℂ **_FakeTaskManager** (Class) `[TRIVIAL]`
- Minimal task manager for deterministic operation creation an... - Minimal task manager for deterministic operation creation an...
- ℂ **_FakeTaskManager** (Class) `[TRIVIAL]` - ℂ **_FakeTaskManager** (Class) `[TRIVIAL]`
- Minimal async-compatible TaskManager fixture for determinist...
**Dependencies:** **Dependencies:**
@@ -130,9 +137,9 @@
### 📁 `core/` ### 📁 `core/`
- 🏗️ **Layers:** Core, Domain, Infra - 🏗️ **Layers:** Core, Domain, Infra
- 📊 **Tiers:** CRITICAL: 9, STANDARD: 66, TRIVIAL: 134 - 📊 **Tiers:** CRITICAL: 9, STANDARD: 49, TRIVIAL: 153
- 📄 **Files:** 14 - 📄 **Files:** 14
- 📦 **Entities:** 209 - 📦 **Entities:** 211
**Key Entities:** **Key Entities:**
@@ -159,11 +166,11 @@
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> AppConfig
- 🔗 DEPENDS_ON -> AppConfigRecord
- 🔗 DEPENDS_ON -> SessionLocal
- 🔗 DEPENDS_ON -> backend.src.core.auth.config - 🔗 DEPENDS_ON -> backend.src.core.auth.config
- 🔗 DEPENDS_ON -> backend.src.core.config_models.AppConfig
- 🔗 DEPENDS_ON -> backend.src.core.config_models.Environment - 🔗 DEPENDS_ON -> backend.src.core.config_models.Environment
- 🔗 DEPENDS_ON -> backend.src.core.database.SessionLocal
- 🔗 DEPENDS_ON -> backend.src.core.logger
### 📁 `__tests__/` ### 📁 `__tests__/`
@@ -186,16 +193,18 @@
### 📁 `auth/` ### 📁 `auth/`
- 🏗️ **Layers:** Core, Domain - 🏗️ **Layers:** Core, Domain
- 📊 **Tiers:** CRITICAL: 17, STANDARD: 2, TRIVIAL: 10 - 📊 **Tiers:** CRITICAL: 6, STANDARD: 2, TRIVIAL: 20
- 📄 **Files:** 7 - 📄 **Files:** 7
- 📦 **Entities:** 29 - 📦 **Entities:** 28
**Key Entities:** **Key Entities:**
- ℂ **AuthConfig** (Class) `[CRITICAL]` - ℂ **AuthConfig** (Class) `[CRITICAL]`
- Holds authentication-related settings. - Holds authentication-related settings.
- ℂ **AuthRepository** (Class) `[CRITICAL]` - ℂ **AuthRepository** (Class) `[CRITICAL]`
- Encapsulates database operations for authentication-related ... - Initialize repository with database session.
- 📦 **AuthRepository** (Module) `[CRITICAL]`
- Data access layer for authentication and user preference ent...
- 📦 **backend.src.core.auth.config** (Module) `[CRITICAL]` - 📦 **backend.src.core.auth.config** (Module) `[CRITICAL]`
- Centralized configuration for authentication and authorizati... - Centralized configuration for authentication and authorizati...
- 📦 **backend.src.core.auth.jwt** (Module) - 📦 **backend.src.core.auth.jwt** (Module)
@@ -204,18 +213,16 @@
- Audit logging for security-related events. - Audit logging for security-related events.
- 📦 **backend.src.core.auth.oauth** (Module) `[CRITICAL]` - 📦 **backend.src.core.auth.oauth** (Module) `[CRITICAL]`
- ADFS OIDC configuration and client using Authlib. - ADFS OIDC configuration and client using Authlib.
- 📦 **backend.src.core.auth.repository** (Module) `[CRITICAL]`
- Data access layer for authentication and user preference ent...
- 📦 **backend.src.core.auth.security** (Module) `[CRITICAL]` - 📦 **backend.src.core.auth.security** (Module) `[CRITICAL]`
- Utility for password hashing and verification using Passlib. - Utility for password hashing and verification using Passlib.
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> Permission:Class
- 🔗 DEPENDS_ON -> Role:Class
- 🔗 DEPENDS_ON -> User:Class
- 🔗 DEPENDS_ON -> UserDashboardPreference:Class
- 🔗 DEPENDS_ON -> authlib - 🔗 DEPENDS_ON -> authlib
- 🔗 DEPENDS_ON -> backend.src.core.logger.belief_scope
- 🔗 DEPENDS_ON -> backend.src.models.auth
- 🔗 DEPENDS_ON -> backend.src.models.profile
- 🔗 DEPENDS_ON -> jose
### 📁 `__tests__/` ### 📁 `__tests__/`
@@ -304,10 +311,10 @@
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> Environment - 🔗 DEPENDS_ON -> Environment
- 🔗 DEPENDS_ON -> PluginLoader:Class
- 🔗 DEPENDS_ON -> TaskLogPersistenceService:Class
- 🔗 DEPENDS_ON -> TaskLogRecord - 🔗 DEPENDS_ON -> TaskLogRecord
- 🔗 DEPENDS_ON -> TaskLogger, USED_BY -> plugins - 🔗 DEPENDS_ON -> TaskLogger, USED_BY -> plugins
- 🔗 DEPENDS_ON -> TaskManager, CALLS -> TaskManager._add_log
- 🔗 DEPENDS_ON -> TaskManagerModels
### 📁 `__tests__/` ### 📁 `__tests__/`
@@ -326,7 +333,7 @@
### 📁 `utils/` ### 📁 `utils/`
- 🏗️ **Layers:** Core, Domain, Infra - 🏗️ **Layers:** Core, Domain, Infra
- 📊 **Tiers:** CRITICAL: 1, STANDARD: 10, TRIVIAL: 61 - 📊 **Tiers:** CRITICAL: 1, STANDARD: 11, TRIVIAL: 60
- 📄 **Files:** 6 - 📄 **Files:** 6
- 📦 **Entities:** 72 - 📦 **Entities:** 72
@@ -393,10 +400,10 @@
**Dependencies:** **Dependencies:**
- 🔗 DEPENDS_ON -> AuthModels
- 🔗 DEPENDS_ON -> Role - 🔗 DEPENDS_ON -> Role
- 🔗 DEPENDS_ON -> TaskRecord - 🔗 DEPENDS_ON -> TaskRecord
- 🔗 DEPENDS_ON -> backend.src.core.task_manager.models - 🔗 DEPENDS_ON -> backend.src.core.task_manager.models
- 🔗 DEPENDS_ON -> backend.src.models.auth
- 🔗 DEPENDS_ON -> backend.src.models.mapping - 🔗 DEPENDS_ON -> backend.src.models.mapping
### 📁 `__tests__/` ### 📁 `__tests__/`
@@ -618,9 +625,9 @@
### 📁 `services/` ### 📁 `services/`
- 🏗️ **Layers:** Core, Domain, Domain/Service, Service - 🏗️ **Layers:** Core, Domain, Domain/Service, Service
- 📊 **Tiers:** CRITICAL: 5, STANDARD: 47, TRIVIAL: 118 - 📊 **Tiers:** CRITICAL: 5, STANDARD: 47, TRIVIAL: 164
- 📄 **Files:** 10 - 📄 **Files:** 10
- 📦 **Entities:** 170 - 📦 **Entities:** 216
**Key Entities:** **Key Entities:**
@@ -1346,9 +1353,9 @@
### 📁 `layout/` ### 📁 `layout/`
- 🏗️ **Layers:** UI, Unknown - 🏗️ **Layers:** UI, Unknown
- 📊 **Tiers:** STANDARD: 7, TRIVIAL: 54 - 📊 **Tiers:** STANDARD: 7, TRIVIAL: 55
- 📄 **Files:** 5 - 📄 **Files:** 5
- 📦 **Entities:** 61 - 📦 **Entities:** 62
**Key Entities:** **Key Entities:**
@@ -2066,10 +2073,10 @@
### 📁 `root/` ### 📁 `root/`
- 🏗️ **Layers:** DevOps/Tooling - 🏗️ **Layers:** DevOps/Tooling, Unknown
- 📊 **Tiers:** CRITICAL: 11, STANDARD: 18, TRIVIAL: 11 - 📊 **Tiers:** CRITICAL: 11, STANDARD: 18, TRIVIAL: 13
- 📄 **Files:** 1 - 📄 **Files:** 2
- 📦 **Entities:** 40 - 📦 **Entities:** 42
**Key Entities:** **Key Entities:**
@@ -2087,16 +2094,14 @@
- Legacy tier buckets retained for backward-compatible reporti... - Legacy tier buckets retained for backward-compatible reporti...
- 📦 **generate_semantic_map** (Module) - 📦 **generate_semantic_map** (Module)
- Scans the codebase to generate a Semantic Map, Module Map, a... - Scans the codebase to generate a Semantic Map, Module Map, a...
- 📦 **merge_spec** (Module) `[TRIVIAL]`
- Auto-generated module for merge_spec.py
## Cross-Module Dependencies ## Cross-Module Dependencies
```mermaid ```mermaid
graph TD graph TD
src-->|DEPENDS_ON|backend src-->|DEPENDS_ON|backend
src-->|DEPENDS_ON|backend
api-->|USES|backend
api-->|USES|backend
routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|CALLS|backend routes-->|CALLS|backend
@@ -2123,7 +2128,6 @@ graph TD
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend
routes-->|USES|backend routes-->|USES|backend
routes-->|USES|backend routes-->|USES|backend
routes-->|CALLS|backend routes-->|CALLS|backend
@@ -2152,10 +2156,9 @@ graph TD
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend
routes-->|DEPENDS_ON|backend
routes-->|CALLS|backend routes-->|CALLS|backend
__tests__-->|TESTS|backend __tests__-->|TESTS|backend
__tests__-->|VERIFIES|backend
__tests__-->|TESTS|backend __tests__-->|TESTS|backend
__tests__-->|TESTS|backend __tests__-->|TESTS|backend
__tests__-->|TESTS|backend __tests__-->|TESTS|backend
@@ -2177,11 +2180,6 @@ graph TD
core-->|DEPENDS_ON|backend core-->|DEPENDS_ON|backend
core-->|DEPENDS_ON|backend core-->|DEPENDS_ON|backend
core-->|DEPENDS_ON|backend core-->|DEPENDS_ON|backend
core-->|CALLS|backend
core-->|CALLS|backend
core-->|DEPENDS_ON|backend
core-->|DEPENDS_ON|backend
core-->|DEPENDS_ON|backend
core-->|DEPENDS_ON|backend core-->|DEPENDS_ON|backend
core-->|INHERITS|backend core-->|INHERITS|backend
core-->|DEPENDS_ON|backend core-->|DEPENDS_ON|backend
@@ -2192,9 +2190,6 @@ graph TD
auth-->|USES|backend auth-->|USES|backend
auth-->|USES|backend auth-->|USES|backend
auth-->|USES|backend auth-->|USES|backend
auth-->|DEPENDS_ON|backend
auth-->|DEPENDS_ON|backend
auth-->|DEPENDS_ON|backend
migration-->|DEPENDS_ON|backend migration-->|DEPENDS_ON|backend
migration-->|DEPENDS_ON|backend migration-->|DEPENDS_ON|backend
migration-->|DEPENDS_ON|backend migration-->|DEPENDS_ON|backend
@@ -2206,8 +2201,6 @@ graph TD
task_manager-->|USED_BY|backend task_manager-->|USED_BY|backend
task_manager-->|USED_BY|backend task_manager-->|USED_BY|backend
task_manager-->|DEPENDS_ON|backend task_manager-->|DEPENDS_ON|backend
task_manager-->|DEPENDS_ON|backend
task_manager-->|DEPENDS_ON|backend
utils-->|DEPENDS_ON|backend utils-->|DEPENDS_ON|backend
utils-->|DEPENDS_ON|backend utils-->|DEPENDS_ON|backend
utils-->|CALLS|backend utils-->|CALLS|backend
@@ -2220,9 +2213,6 @@ graph TD
models-->|DEPENDS_ON|backend models-->|DEPENDS_ON|backend
models-->|DEPENDS_ON|backend models-->|DEPENDS_ON|backend
models-->|USED_BY|backend models-->|USED_BY|backend
models-->|INHERITS_FROM|backend
models-->|DEPENDS_ON|backend
models-->|INHERITS_FROM|backend
__tests__-->|TESTS|backend __tests__-->|TESTS|backend
llm_analysis-->|IMPLEMENTS|backend llm_analysis-->|IMPLEMENTS|backend
llm_analysis-->|IMPLEMENTS|backend llm_analysis-->|IMPLEMENTS|backend

627
.ai/PROJECT_MAP.md
View File

@@ -2,6 +2,11 @@
> Compressed view for AI Context. Generated automatically. > Compressed view for AI Context. Generated automatically.
- 📦 **merge_spec** (`Module`) `[TRIVIAL]`
- 📝 Auto-generated module for merge_spec.py
- 🏗️ Layer: Unknown
- ƒ **merge_specs** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- 📦 **generate_semantic_map** (`Module`) - 📦 **generate_semantic_map** (`Module`)
- 📝 Scans the codebase to generate a Semantic Map, Module Map, and Compliance Report based on the System Standard. - 📝 Scans the codebase to generate a Semantic Map, Module Map, and Compliance Report based on the System Standard.
- 🏗️ Layer: DevOps/Tooling - 🏗️ Layer: DevOps/Tooling
@@ -653,6 +658,8 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **llmValidationBadgeClass** (`Function`) `[TRIVIAL]` - ƒ **llmValidationBadgeClass** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **getStatusClass** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **stopTaskDetailsPolling** (`Function`) `[TRIVIAL]` - ƒ **stopTaskDetailsPolling** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **loadEnvironmentOptions** (`Function`) `[TRIVIAL]` - ƒ **loadEnvironmentOptions** (`Function`) `[TRIVIAL]`
@@ -2185,16 +2192,18 @@
- 🔒 Invariant: Edit action keeps explicit click handler and opens normalized edit form. - 🔒 Invariant: Edit action keeps explicit click handler and opens normalized edit form.
- ƒ **provider_config_edit_contract_tests** (`Function`) - ƒ **provider_config_edit_contract_tests** (`Function`)
- 📝 Validate edit and delete handler wiring plus normalized edit form state mapping. - 📝 Validate edit and delete handler wiring plus normalized edit form state mapping.
- 📦 **backend.delete_running_tasks** (`Module`) `[TRIVIAL]` - 📦 **DeleteRunningTasksUtil** (`Module`) `[TRIVIAL]`
- 📝 Script to delete tasks with RUNNING status from the database. - 📝 Script to delete tasks with RUNNING status from the database.
- 🏗️ Layer: Utility - 🏗️ Layer: Utility
- 🔗 DEPENDS_ON -> `TasksSessionLocal`
- 🔗 DEPENDS_ON -> `TaskRecord`
- ƒ **delete_running_tasks** (`Function`) `[TRIVIAL]` - ƒ **delete_running_tasks** (`Function`) `[TRIVIAL]`
- 📝 Delete all tasks with RUNNING status from the database. - 📝 Delete all tasks with RUNNING status from the database.
- 📦 **AppModule** (`Module`) `[CRITICAL]` - 📦 **AppModule** (`Module`) `[CRITICAL]`
- 📝 The main entry point for the FastAPI application. It initializes the app, configures CORS, sets up dependencies, includes API routers, and defines the WebSocket endpoint for log streaming. - 📝 The main entry point for the FastAPI application. It initializes the app, configures CORS, sets up dependencies, includes API routers, and defines the WebSocket endpoint for log streaming.
- 🏗️ Layer: UI (API) - 🏗️ Layer: UI (API)
- 🔒 Invariant: All WebSocket connections must be properly cleaned up on disconnect. - 🔒 Invariant: All WebSocket connections must be properly cleaned up on disconnect.
- 🔗 DEPENDS_ON -> `backend.src.dependencies` - 🔗 DEPENDS_ON -> `AppDependencies`
- 🔗 DEPENDS_ON -> `backend.src.api.routes` - 🔗 DEPENDS_ON -> `backend.src.api.routes`
- 📦 **App** (`Global`) `[TRIVIAL]` - 📦 **App** (`Global`) `[TRIVIAL]`
- 📝 The global FastAPI application instance. - 📝 The global FastAPI application instance.
@@ -2202,10 +2211,14 @@
- 📝 Handles application startup tasks, such as starting the scheduler. - 📝 Handles application startup tasks, such as starting the scheduler.
- ƒ **shutdown_event** (`Function`) - ƒ **shutdown_event** (`Function`)
- 📝 Handles application shutdown tasks, such as stopping the scheduler. - 📝 Handles application shutdown tasks, such as stopping the scheduler.
- ▦ **app_middleware** (`Block`) `[TRIVIAL]`
- 📝 Configure application-wide middleware (Session, CORS).
- ƒ **network_error_handler** (`Function`) `[TRIVIAL]` - ƒ **network_error_handler** (`Function`) `[TRIVIAL]`
- 📝 Global exception handler for NetworkError. - 📝 Global exception handler for NetworkError.
- ƒ **log_requests** (`Function`) - ƒ **log_requests** (`Function`)
- 📝 Middleware to log incoming HTTP requests and their response status. - 📝 Middleware to log incoming HTTP requests and their response status.
- ▦ **api_routes** (`Block`) `[TRIVIAL]`
- 📝 Register all application API routers.
- 📦 **api.include_routers** (`Action`) `[TRIVIAL]` - 📦 **api.include_routers** (`Action`) `[TRIVIAL]`
- 📝 Registers all API routers with the FastAPI application. - 📝 Registers all API routers with the FastAPI application.
- 🏗️ Layer: API - 🏗️ Layer: API
@@ -2219,9 +2232,18 @@
- 📝 A simple root endpoint to confirm that the API is running when frontend is missing. - 📝 A simple root endpoint to confirm that the API is running when frontend is missing.
- ƒ **matches_filters** (`Function`) `[TRIVIAL]` - ƒ **matches_filters** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **backend.src.dependencies** (`Module`) - 📦 **AppDependencies** (`Module`)
- 📝 Manages creation and provision of shared application dependencies, such as PluginLoader and TaskManager, to avoid circular imports. - 📝 Manages creation and provision of shared application dependencies, such as PluginLoader and TaskManager, to avoid circular imports.
- 🏗️ Layer: Core - 🏗️ Layer: Core
- 🔗 CALLS -> `CleanReleaseRepository`
- 🔗 CALLS -> `ConfigManager`
- 🔗 CALLS -> `PluginLoader`
- 🔗 CALLS -> `SchedulerService`
- 🔗 CALLS -> `TaskManager`
- 🔗 CALLS -> `get_all_plugin_configs`
- 🔗 CALLS -> `get_db`
- 🔗 CALLS -> `info`
- 🔗 CALLS -> `init_db`
- ƒ **get_config_manager** (`Function`) `[TRIVIAL]` - ƒ **get_config_manager** (`Function`) `[TRIVIAL]`
- 📝 Dependency injector for ConfigManager. - 📝 Dependency injector for ConfigManager.
- ƒ **get_plugin_loader** (`Function`) `[TRIVIAL]` - ƒ **get_plugin_loader** (`Function`) `[TRIVIAL]`
@@ -2246,7 +2268,7 @@
- 📝 Dependency for checking if the current user has a specific permission. - 📝 Dependency for checking if the current user has a specific permission.
- ƒ **permission_checker** (`Function`) `[TRIVIAL]` - ƒ **permission_checker** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **src** (`Package`) `[TRIVIAL]` - 📦 **SrcRoot** (`Module`) `[TRIVIAL]`
- 📝 Canonical backend package root for application, scripts, and tests. - 📝 Canonical backend package root for application, scripts, and tests.
- 📦 **backend.src.scripts.seed_superset_load_test** (`Module`) - 📦 **backend.src.scripts.seed_superset_load_test** (`Module`)
- 📝 Creates randomized load-test data in Superset by cloning chart configurations and creating dashboards in target environments. - 📝 Creates randomized load-test data in Superset by cloning chart configurations and creating dashboards in target environments.
@@ -2521,9 +2543,103 @@
- 🔗 DEPENDS_ON -> `backend.src.core.config_models.Environment` - 🔗 DEPENDS_ON -> `backend.src.core.config_models.Environment`
- ƒ **backend.src.core.superset_client.SupersetClient.__init__** (`Function`) - ƒ **backend.src.core.superset_client.SupersetClient.__init__** (`Function`)
- 📝 Инициализирует клиент, проверяет конфигурацию и создает сетевой клиент. - 📝 Инициализирует клиент, проверяет конфигурацию и создает сетевой клиент.
- ƒ **backend.src.core.superset_client.SupersetClient.authenticate** (`Function`) - ƒ **backend.src.core.superset_client.SupersetClient.authenticate** (`Function`)
- 📝 Authenticates the client using the configured credentials. - 📝 Authenticates the client using the configured credentials.
- 🔗 CALLS -> `self.network.authenticate` - 🔗 CALLS -> `self.network.authenticate`
- ƒ **backend.src.core.superset_client.SupersetClient.headers** (`Function`) `[TRIVIAL]`
- 📝 Возвращает базовые HTTP-заголовки, используемые сетевым клиентом.
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboards** (`Function`)
- 📝 Получает полный список дашбордов, автоматически обрабатывая пагинацию.
- 🔗 CALLS -> `self._fetch_all_pages`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboards_page** (`Function`)
- 📝 Fetches a single dashboards page from Superset without iterating all pages.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboards_summary** (`Function`)
- 📝 Fetches dashboard metadata optimized for the grid.
- 🔗 CALLS -> `self.get_dashboards`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboards_summary_page** (`Function`)
- 📝 Fetches one page of dashboard metadata optimized for the grid.
- 🔗 CALLS -> `self.get_dashboards_page`
- ƒ **backend.src.core.superset_client.SupersetClient._extract_owner_labels** (`Function`) `[TRIVIAL]`
- 📝 Normalize dashboard owners payload to stable display labels.
- ƒ **backend.src.core.superset_client.SupersetClient._extract_user_display** (`Function`) `[TRIVIAL]`
- 📝 Normalize user payload to a stable display name.
- ƒ **backend.src.core.superset_client.SupersetClient._sanitize_user_text** (`Function`) `[TRIVIAL]`
- 📝 Convert scalar value to non-empty user-facing text.
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboard** (`Function`)
- 📝 Fetches a single dashboard by ID.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_chart** (`Function`)
- 📝 Fetches a single chart by ID.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboard_detail** (`Function`)
- 📝 Fetches detailed dashboard information including related charts and datasets.
- 🔗 CALLS -> `self.get_dashboard`
- 🔗 CALLS -> `self.get_chart`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dashboard_detail.extract_dataset_id_from_form_data** (`Function`) `[TRIVIAL]`
- ƒ **backend.src.core.superset_client.SupersetClient.get_charts** (`Function`)
- 📝 Fetches all charts with pagination support.
- 🔗 CALLS -> `self._fetch_all_pages`
- ƒ **backend.src.core.superset_client.SupersetClient._extract_chart_ids_from_layout** (`Function`) `[TRIVIAL]`
- 📝 Traverses dashboard layout metadata and extracts chart IDs from common keys.
- ƒ **backend.src.core.superset_client.SupersetClient.export_dashboard** (`Function`)
- 📝 Экспортирует дашборд в виде ZIP-архива.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.import_dashboard** (`Function`)
- 📝 Импортирует дашборд из ZIP-файла.
- 🔗 CALLS -> `self._do_import`
- 🔗 CALLS -> `self.delete_dashboard`
- ƒ **backend.src.core.superset_client.SupersetClient.delete_dashboard** (`Function`)
- 📝 Удаляет дашборд по его ID или slug.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_datasets** (`Function`)
- 📝 Получает полный список датасетов, автоматически обрабатывая пагинацию.
- 🔗 CALLS -> `self._fetch_all_pages`
- ƒ **backend.src.core.superset_client.SupersetClient.get_datasets_summary** (`Function`)
- 📝 Fetches dataset metadata optimized for the Dataset Hub grid.
- ƒ **backend.src.core.superset_client.SupersetClient.get_dataset_detail** (`Function`)
- 📝 Fetches detailed dataset information including columns and linked dashboards
- 🔗 CALLS -> `self.get_dataset`
- 🔗 CALLS -> `self.network.request (for related_objects)`
- ƒ **backend.src.core.superset_client.SupersetClient.get_dataset** (`Function`)
- 📝 Получает информацию о конкретном датасете по его ID.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.update_dataset** (`Function`)
- 📝 Обновляет данные датасета по его ID.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_databases** (`Function`)
- 📝 Получает полный список баз данных.
- 🔗 CALLS -> `self._fetch_all_pages`
- ƒ **backend.src.core.superset_client.SupersetClient.get_database** (`Function`)
- 📝 Получает информацию о конкретной базе данных по её ID.
- 🔗 CALLS -> `self.network.request`
- ƒ **backend.src.core.superset_client.SupersetClient.get_databases_summary** (`Function`)
- 📝 Fetch a summary of databases including uuid, name, and engine.
- 🔗 CALLS -> `self.get_databases`
- ƒ **backend.src.core.superset_client.SupersetClient.get_database_by_uuid** (`Function`)
- 📝 Find a database by its UUID.
- 🔗 CALLS -> `self.get_databases`
- ƒ **backend.src.core.superset_client.SupersetClient._resolve_target_id_for_delete** (`Function`) `[TRIVIAL]`
- 📝 Resolves a dashboard ID from either an ID or a slug.
- 🔗 CALLS -> `self.get_dashboards`
- ƒ **backend.src.core.superset_client.SupersetClient._do_import** (`Function`) `[TRIVIAL]`
- 📝 Performs the actual multipart upload for import.
- 🔗 CALLS -> `self.network.upload_file`
- ƒ **backend.src.core.superset_client.SupersetClient._validate_export_response** (`Function`) `[TRIVIAL]`
- 📝 Validates that the export response is a non-empty ZIP archive.
- ƒ **backend.src.core.superset_client.SupersetClient._resolve_export_filename** (`Function`) `[TRIVIAL]`
- 📝 Determines the filename for an exported dashboard.
- ƒ **backend.src.core.superset_client.SupersetClient._validate_query_params** (`Function`) `[TRIVIAL]`
- 📝 Ensures query parameters have default page and page_size.
- ƒ **backend.src.core.superset_client.SupersetClient._fetch_total_object_count** (`Function`) `[TRIVIAL]`
- 📝 Fetches the total number of items for a given endpoint.
- 🔗 CALLS -> `self.network.fetch_paginated_count`
- ƒ **backend.src.core.superset_client.SupersetClient._fetch_all_pages** (`Function`) `[TRIVIAL]`
- 📝 Iterates through all pages to collect all data items.
- ƒ **backend.src.core.superset_client.SupersetClient._validate_import_file** (`Function`) `[TRIVIAL]`
- 📝 Validates that the file to be imported is a valid ZIP with metadata.yaml.
- ƒ **backend.src.core.superset_client.SupersetClient.get_all_resources** (`Function`)
- 📝 Fetches all resources of a given type with id, uuid, and name columns.
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **authenticate** (`Function`) `[TRIVIAL]` - ƒ **authenticate** (`Function`) `[TRIVIAL]`
@@ -2609,54 +2725,54 @@
- 🔗 DEPENDS_ON -> `backend.src.core.logger` - 🔗 DEPENDS_ON -> `backend.src.core.logger`
- ƒ **ensure_encryption_key** (`Function`) `[TRIVIAL]` - ƒ **ensure_encryption_key** (`Function`) `[TRIVIAL]`
- 📝 Ensure backend runtime has a persistent valid Fernet key. - 📝 Ensure backend runtime has a persistent valid Fernet key.
- 📦 **ConfigManagerModule** (`Module`) `[CRITICAL]` - 📦 **ConfigManager** (`Module`) `[CRITICAL]`
- 📝 Manages application configuration persistence in DB with one-time migration from legacy JSON. - 📝 Manages application configuration persistence in DB with one-time migration from legacy JSON.
- 🏗️ Layer: Domain - 🏗️ Layer: Domain
- 🔒 Invariant: Configuration must always be representable by AppConfig and persisted under global record id. - 🔒 Invariant: Configuration must always be representable by AppConfig and persisted under global record id.
- 🔗 DEPENDS_ON -> `backend.src.core.config_models.AppConfig` - 🔗 DEPENDS_ON -> `AppConfig`
- 🔗 DEPENDS_ON -> `backend.src.core.database.SessionLocal` - 🔗 DEPENDS_ON -> `SessionLocal`
- 🔗 DEPENDS_ON -> `backend.src.models.config.AppConfigRecord` - 🔗 DEPENDS_ON -> `AppConfigRecord`
- 🔗 CALLS -> `backend.src.core.logger.logger` - 🔗 CALLS -> `logger`
- 🔗 CALLS -> `backend.src.core.logger.configure_logger` - 🔗 CALLS -> `configure_logger`
- ℂ **ConfigManager** (`Class`) `[CRITICAL]` - ℂ **ConfigManager** (`Class`) `[CRITICAL]`
- 📝 Handles application configuration load, validation, mutation, and persistence lifecycle. - 📝 Handles application configuration load, validation, mutation, and persistence lifecycle.
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Initialize manager state from persisted or migrated configuration. - 📝 Initialize manager state from persisted or migrated configuration.
- ƒ **_default_config** (`Function`) - ƒ **_default_config** (`Function`) `[TRIVIAL]`
- 📝 Build default application configuration fallback. - 📝 Build default application configuration fallback.
- ƒ **_sync_raw_payload_from_config** (`Function`) - ƒ **_sync_raw_payload_from_config** (`Function`) `[TRIVIAL]`
- 📝 Merge typed AppConfig state into raw payload while preserving unsupported legacy sections. - 📝 Merge typed AppConfig state into raw payload while preserving unsupported legacy sections.
- ƒ **_load_from_legacy_file** (`Function`) - ƒ **_load_from_legacy_file** (`Function`) `[TRIVIAL]`
- 📝 Load legacy JSON configuration for migration fallback path. - 📝 Load legacy JSON configuration for migration fallback path.
- ƒ **_get_record** (`Function`) - ƒ **_get_record** (`Function`) `[TRIVIAL]`
- 📝 Resolve global configuration record from DB. - 📝 Resolve global configuration record from DB.
- ƒ **_load_config** (`Function`) - ƒ **_load_config** (`Function`) `[TRIVIAL]`
- 📝 Load configuration from DB or perform one-time migration from legacy JSON. - 📝 Load configuration from DB or perform one-time migration from legacy JSON.
- ƒ **_save_config_to_db** (`Function`) - ƒ **_save_config_to_db** (`Function`) `[TRIVIAL]`
- 📝 Persist provided AppConfig into the global DB configuration record. - 📝 Persist provided AppConfig into the global DB configuration record.
- ƒ **save** (`Function`) - ƒ **save** (`Function`) `[TRIVIAL]`
- 📝 Persist current in-memory configuration state. - 📝 Persist current in-memory configuration state.
- ƒ **get_config** (`Function`) - ƒ **get_config** (`Function`) `[TRIVIAL]`
- 📝 Return current in-memory configuration snapshot. - 📝 Return current in-memory configuration snapshot.
- ƒ **get_payload** (`Function`) - ƒ **get_payload** (`Function`) `[TRIVIAL]`
- 📝 Return full persisted payload including sections outside typed AppConfig schema. - 📝 Return full persisted payload including sections outside typed AppConfig schema.
- ƒ **save_config** (`Function`) - ƒ **save_config** (`Function`) `[TRIVIAL]`
- 📝 Persist configuration provided either as typed AppConfig or raw payload dict. - 📝 Persist configuration provided either as typed AppConfig or raw payload dict.
- ƒ **update_global_settings** (`Function`) - ƒ **update_global_settings** (`Function`) `[TRIVIAL]`
- 📝 Replace global settings and persist the resulting configuration. - 📝 Replace global settings and persist the resulting configuration.
- ƒ **validate_path** (`Function`) - ƒ **validate_path** (`Function`) `[TRIVIAL]`
- 📝 Validate that path exists and is writable, creating it when absent. - 📝 Validate that path exists and is writable, creating it when absent.
- ƒ **get_environments** (`Function`) - ƒ **get_environments** (`Function`) `[TRIVIAL]`
- 📝 Return all configured environments. - 📝 Return all configured environments.
- ƒ **has_environments** (`Function`) - ƒ **has_environments** (`Function`) `[TRIVIAL]`
- 📝 Check whether at least one environment exists in configuration. - 📝 Check whether at least one environment exists in configuration.
- ƒ **get_environment** (`Function`) - ƒ **get_environment** (`Function`) `[TRIVIAL]`
- 📝 Resolve a configured environment by identifier. - 📝 Resolve a configured environment by identifier.
- ƒ **add_environment** (`Function`) - ƒ **add_environment** (`Function`) `[TRIVIAL]`
- 📝 Upsert environment by id into configuration and persist. - 📝 Upsert environment by id into configuration and persist.
- ƒ **update_environment** (`Function`) - ƒ **update_environment** (`Function`) `[TRIVIAL]`
- 📝 Update existing environment by id and preserve masked password placeholder behavior. - 📝 Update existing environment by id and preserve masked password placeholder behavior.
- ƒ **delete_environment** (`Function`) - ƒ **delete_environment** (`Function`) `[TRIVIAL]`
- 📝 Delete environment by id and persist when deletion occurs. - 📝 Delete environment by id and persist when deletion occurs.
- 📦 **SchedulerModule** (`Module`) - 📦 **SchedulerModule** (`Module`)
- 📝 Manages scheduled tasks using APScheduler. - 📝 Manages scheduled tasks using APScheduler.
@@ -2730,6 +2846,8 @@
- 📝 Applies additive schema upgrades for llm_validation_results table. - 📝 Applies additive schema upgrades for llm_validation_results table.
- ƒ **_ensure_git_server_configs_columns** (`Function`) - ƒ **_ensure_git_server_configs_columns** (`Function`)
- 📝 Applies additive schema upgrades for git_server_configs table. - 📝 Applies additive schema upgrades for git_server_configs table.
- ƒ **_ensure_auth_users_columns** (`Function`)
- 📝 Applies additive schema upgrades for auth users table.
- ƒ **ensure_connection_configs_table** (`Function`) - ƒ **ensure_connection_configs_table** (`Function`)
- 📝 Ensures the external connection registry table exists in the main database. - 📝 Ensures the external connection registry table exists in the main database.
- ƒ **init_db** (`Function`) - ƒ **init_db** (`Function`)
@@ -2938,39 +3056,38 @@
- 📝 Verifies a plain password against a hashed password. - 📝 Verifies a plain password against a hashed password.
- ƒ **get_password_hash** (`Function`) `[TRIVIAL]` - ƒ **get_password_hash** (`Function`) `[TRIVIAL]`
- 📝 Generates a bcrypt hash for a plain password. - 📝 Generates a bcrypt hash for a plain password.
- 📦 **backend.src.core.auth.repository** (`Module`) `[CRITICAL]` - 📦 **AuthRepository** (`Module`) `[CRITICAL]`
- 📝 Data access layer for authentication and user preference entities. - 📝 Data access layer for authentication and user preference entities.
- 🏗️ Layer: Domain - 🏗️ Layer: Domain
- 🔒 Invariant: All database read/write operations must execute via the injected SQLAlchemy session boundary. - 🔒 Invariant: All database read/write operations must execute via the injected SQLAlchemy session boundary.
- 🔗 DEPENDS_ON -> `sqlalchemy.orm.Session` - 🔗 DEPENDS_ON -> `sqlalchemy.orm.Session`
- 🔗 DEPENDS_ON -> `backend.src.models.auth` - 🔗 DEPENDS_ON -> `User:Class`
- 🔗 DEPENDS_ON -> `backend.src.models.profile` - 🔗 DEPENDS_ON -> `Role:Class`
- 🔗 DEPENDS_ON -> `backend.src.core.logger.belief_scope` - 🔗 DEPENDS_ON -> `Permission:Class`
- 🔗 DEPENDS_ON -> `UserDashboardPreference:Class`
- 🔗 DEPENDS_ON -> `belief_scope:Function`
- ℂ **AuthRepository** (`Class`) `[CRITICAL]` - ℂ **AuthRepository** (`Class`) `[CRITICAL]`
- 📝 Encapsulates database operations for authentication-related entities. - 📝 Initialize repository with database session.
- 🔗 DEPENDS_ON -> `sqlalchemy.orm.Session` - ƒ **get_user_by_id** (`Function`) `[TRIVIAL]`
- ƒ **__init__** (`Function`) `[CRITICAL]` - 📝 Retrieve user by UUID.
- 📝 Bind repository instance to an existing SQLAlchemy session. - ƒ **get_user_by_username** (`Function`) `[TRIVIAL]`
- ƒ **get_user_by_username** (`Function`) `[CRITICAL]` - 📝 Retrieve user by username.
- 📝 Retrieve a user entity by unique username. - ƒ **get_role_by_id** (`Function`) `[TRIVIAL]`
- ƒ **get_user_by_id** (`Function`) `[CRITICAL]` - 📝 Retrieve role by UUID with permissions preloaded.
- 📝 Retrieve a user entity by identifier. - ƒ **get_role_by_name** (`Function`) `[TRIVIAL]`
- ƒ **get_role_by_name** (`Function`) `[CRITICAL]` - 📝 Retrieve role by unique name.
- 📝 Retrieve a role entity by role name. - ƒ **get_permission_by_id** (`Function`) `[TRIVIAL]`
- ƒ **update_last_login** (`Function`) `[CRITICAL]` - 📝 Retrieve permission by UUID.
- 📝 Update last_login timestamp for the provided user entity. - ƒ **get_permission_by_resource_action** (`Function`) `[TRIVIAL]`
- ƒ **get_role_by_id** (`Function`) `[CRITICAL]` - 📝 Retrieve permission by resource and action tuple.
- 📝 Retrieve a role entity by identifier. - ƒ **list_permissions** (`Function`) `[TRIVIAL]`
- ƒ **get_permission_by_id** (`Function`) `[CRITICAL]` - 📝 List all system permissions.
- 📝 Retrieve a permission entity by identifier. - ƒ **get_user_dashboard_preference** (`Function`) `[TRIVIAL]`
- ƒ **get_permission_by_resource_action** (`Function`) `[CRITICAL]` - 📝 Retrieve dashboard filters/preferences for a user.
- 📝 Retrieve a permission entity by resource and action pair. - ƒ **get_roles_by_ad_groups** (`Function`) `[TRIVIAL]`
- ƒ **get_user_dashboard_preference** (`Function`) `[CRITICAL]` - 📝 Retrieve roles that match a list of AD group names.
- 📝 Retrieve dashboard preference entity owned by specified user. - ƒ **__init__** (`Function`) `[TRIVIAL]`
- ƒ **save_user_dashboard_preference** (`Function`) `[CRITICAL]` - 📝 Auto-detected function (orphan)
- 📝 Persist dashboard preference entity and return refreshed persistent row.
- ƒ **list_permissions** (`Function`) `[CRITICAL]`
- 📝 List all permission entities available in storage.
- 📦 **src.core.auth** (`Package`) `[TRIVIAL]` - 📦 **src.core.auth** (`Package`) `[TRIVIAL]`
- 📝 Authentication and authorization package root. - 📝 Authentication and authorization package root.
- 📦 **test_auth** (`Module`) - 📦 **test_auth** (`Module`)
@@ -3022,7 +3139,7 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **_normalize_base_url** (`Function`) `[TRIVIAL]` - ƒ **_normalize_base_url** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **backend.core.utils.fileio** (`Module`) `[TRIVIAL]` - 📦 **FileIO** (`Module`)
- 📝 Предоставляет набор утилит для управления файловыми операциями, включая работу с временными файлами, архивами ZIP, файлами YAML и очистку директорий. - 📝 Предоставляет набор утилит для управления файловыми операциями, включая работу с временными файлами, архивами ZIP, файлами YAML и очистку директорий.
- 🏗️ Layer: Infra - 🏗️ Layer: Infra
- 🔗 DEPENDS_ON -> `backend.src.core.logger` - 🔗 DEPENDS_ON -> `backend.src.core.logger`
@@ -3327,15 +3444,19 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **json_serializable** (`Function`) `[TRIVIAL]` - ƒ **json_serializable** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **TaskManagerModule** (`Module`) `[CRITICAL]` - 📦 **TaskManager** (`Module`) `[CRITICAL]`
- 📝 Manages the lifecycle of tasks, including their creation, execution, and state tracking. It uses a thread pool to run plugins asynchronously. - 📝 Manages the lifecycle of tasks, including their creation, execution, and state tracking. It uses a thread pool to run plugins asynchronously.
- 🏗️ Layer: Core - 🏗️ Layer: Core
- 🔒 Invariant: Task IDs are unique. - 🔒 Invariant: Task IDs are unique.
- 🔗 DEPENDS_ON -> `backend.src.core.plugin_loader` - 🔗 DEPENDS_ON -> `PluginLoader:Class`
- 🔗 DEPENDS_ON -> `backend.src.core.task_manager.persistence` - 🔗 DEPENDS_ON -> `TaskPersistenceModule:Module`
- ℂ **TaskManager** (`Class`) `[CRITICAL]` - ℂ **TaskManager** (`Class`) `[CRITICAL]`
- 📝 Manages the lifecycle of tasks, including their creation, execution, and state tracking. - 📝 Manages the lifecycle of tasks, including their creation, execution, and state tracking.
- 🏗️ Layer: Core
- 🔒 Invariant: Log entries are never deleted after being added to a task. - 🔒 Invariant: Log entries are never deleted after being added to a task.
- 🔗 DEPENDS_ON -> `TaskPersistenceService:Class`
- 🔗 DEPENDS_ON -> `TaskLogPersistenceService:Class`
- 🔗 DEPENDS_ON -> `PluginLoader:Class`
- ƒ **__init__** (`Function`) `[CRITICAL]` - ƒ **__init__** (`Function`) `[CRITICAL]`
- 📝 Initialize the TaskManager with dependencies. - 📝 Initialize the TaskManager with dependencies.
- ƒ **_flusher_loop** (`Function`) - ƒ **_flusher_loop** (`Function`)
@@ -3468,22 +3589,29 @@
- 📝 Verify TaskContext preserves optional background task scheduler across sub-context creation. - 📝 Verify TaskContext preserves optional background task scheduler across sub-context creation.
- ƒ **test_task_context_preserves_background_tasks_across_sub_context** (`Function`) `[TRIVIAL]` - ƒ **test_task_context_preserves_background_tasks_across_sub_context** (`Function`) `[TRIVIAL]`
- 📝 Plugins must be able to access background_tasks from both root and sub-context loggers. - 📝 Plugins must be able to access background_tasks from both root and sub-context loggers.
- 📦 **backend.src.api.auth** (`Module`) - 📦 **AuthApi** (`Module`)
- 📝 Authentication API endpoints. - 📝 Authentication API endpoints.
- 🏗️ Layer: API - 🏗️ Layer: API
- 🔒 Invariant: All auth endpoints must return consistent error codes. - 🔒 Invariant: All auth endpoints must return consistent error codes.
- 🔗 DEPENDS_ON -> `AuthRepository:Class`
- 📦 **router** (`Variable`) `[TRIVIAL]` - 📦 **router** (`Variable`) `[TRIVIAL]`
- 📝 APIRouter instance for authentication routes. - 📝 APIRouter instance for authentication routes.
- ƒ **login_for_access_token** (`Function`) - ƒ **login_for_access_token** (`Function`)
- 📝 Authenticates a user and returns a JWT access token. - 📝 Authenticates a user and returns a JWT access token.
- 🔗 CALLS -> `AuthService.authenticate_user`
- 🔗 CALLS -> `AuthService.create_session`
- ƒ **read_users_me** (`Function`) - ƒ **read_users_me** (`Function`)
- 📝 Retrieves the profile of the currently authenticated user. - 📝 Retrieves the profile of the currently authenticated user.
- 🔗 DEPENDS_ON -> `get_current_user`
- ƒ **logout** (`Function`) - ƒ **logout** (`Function`)
- 📝 Logs out the current user (placeholder for session revocation). - 📝 Logs out the current user (placeholder for session revocation).
- 🔗 DEPENDS_ON -> `get_current_user`
- ƒ **login_adfs** (`Function`) - ƒ **login_adfs** (`Function`)
- 📝 Initiates the ADFS OIDC login flow. - 📝 Initiates the ADFS OIDC login flow.
- ƒ **auth_callback_adfs** (`Function`) - ƒ **auth_callback_adfs** (`Function`)
- 📝 Handles the callback from ADFS after successful authentication. - 📝 Handles the callback from ADFS after successful authentication.
- 🔗 CALLS -> `AuthService.provision_adfs_user`
- 🔗 CALLS -> `AuthService.create_session`
- 📦 **src.api** (`Package`) `[TRIVIAL]` - 📦 **src.api** (`Package`) `[TRIVIAL]`
- 📝 Backend API package root. - 📝 Backend API package root.
- 📦 **router** (`Global`) `[TRIVIAL]` - 📦 **router** (`Global`) `[TRIVIAL]`
@@ -3508,7 +3636,7 @@
- 📝 API endpoints for the Dataset Hub - listing datasets with mapping progress - 📝 API endpoints for the Dataset Hub - listing datasets with mapping progress
- 🏗️ Layer: API - 🏗️ Layer: API
- 🔒 Invariant: All dataset responses include last_task metadata - 🔒 Invariant: All dataset responses include last_task metadata
- 🔗 DEPENDS_ON -> `backend.src.dependencies` - 🔗 DEPENDS_ON -> `AppDependencies`
- 🔗 DEPENDS_ON -> `backend.src.services.resource_service.ResourceService` - 🔗 DEPENDS_ON -> `backend.src.services.resource_service.ResourceService`
- 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient` - 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient`
- 📦 **MappedFields** (`DataClass`) `[TRIVIAL]` - 📦 **MappedFields** (`DataClass`) `[TRIVIAL]`
@@ -3685,11 +3813,11 @@
- ƒ **get_environment_databases** (`Function`) `[TRIVIAL]` - ƒ **get_environment_databases** (`Function`) `[TRIVIAL]`
- 📝 Fetch the list of databases from a specific environment. - 📝 Fetch the list of databases from a specific environment.
- 🏗️ Layer: API - 🏗️ Layer: API
- 📦 **backend.src.api.routes.migration** (`Module`) `[CRITICAL]` - 📦 **MigrationApi** (`Module`) `[CRITICAL]`
- 📝 HTTP contract layer for migration orchestration, settings, dry-run, and mapping sync endpoints. - 📝 HTTP contract layer for migration orchestration, settings, dry-run, and mapping sync endpoints.
- 🏗️ Layer: Infra - 🏗️ Layer: Infra
- 🔒 Invariant: Migration endpoints never execute with invalid environment references and always return explicit HTTP errors on guard failures. - 🔒 Invariant: Migration endpoints never execute with invalid environment references and always return explicit HTTP errors on guard failures.
- 🔗 DEPENDS_ON -> `backend.src.dependencies` - 🔗 DEPENDS_ON -> `AppDependencies`
- 🔗 DEPENDS_ON -> `backend.src.core.database` - 🔗 DEPENDS_ON -> `backend.src.core.database`
- 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient` - 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient`
- 🔗 DEPENDS_ON -> `backend.src.core.migration.dry_run_orchestrator.MigrationDryRunService` - 🔗 DEPENDS_ON -> `backend.src.core.migration.dry_run_orchestrator.MigrationDryRunService`
@@ -3717,21 +3845,34 @@
- 📝 Retrieve a list of all available plugins. - 📝 Retrieve a list of all available plugins.
- 📦 **backend.src.api.routes.clean_release_v2** (`Module`) - 📦 **backend.src.api.routes.clean_release_v2** (`Module`)
- 📝 Redesigned clean release API for headless candidate lifecycle. - 📝 Redesigned clean release API for headless candidate lifecycle.
- 🏗️ Layer: API - ℂ **ApprovalRequest** (`Class`) `[TRIVIAL]`
- ƒ **register_candidate** (`Function`) `[TRIVIAL]` - 📝 Schema for approval request payload.
- 📝 Auto-detected function (orphan) - ℂ **PublishRequest** (`Class`) `[TRIVIAL]`
- ƒ **import_artifacts** (`Function`) `[TRIVIAL]` - 📝 Schema for publication request payload.
- 📝 Auto-detected function (orphan) - ℂ **RevokeRequest** (`Class`) `[TRIVIAL]`
- ƒ **build_manifest** (`Function`) `[TRIVIAL]` - 📝 Schema for revocation request payload.
- 📝 Auto-detected function (orphan) - ƒ **register_candidate** (`Function`)
- ƒ **approve_candidate_endpoint** (`Function`) `[TRIVIAL]` - 📝 Register a new release candidate.
- 📝 Auto-detected function (orphan) - 🔗 CALLS -> `CleanReleaseRepository.save_candidate`
- ƒ **reject_candidate_endpoint** (`Function`) `[TRIVIAL]` - ƒ **import_artifacts** (`Function`)
- 📝 Auto-detected function (orphan) - 📝 Associate artifacts with a release candidate.
- ƒ **publish_candidate_endpoint** (`Function`) `[TRIVIAL]` - 🔗 CALLS -> `CleanReleaseRepository.get_candidate`
- 📝 Auto-detected function (orphan) - ƒ **build_manifest** (`Function`)
- ƒ **revoke_publication_endpoint** (`Function`) `[TRIVIAL]` - 📝 Generate distribution manifest for a candidate.
- 📝 Auto-detected function (orphan) - 🔗 CALLS -> `CleanReleaseRepository.save_manifest`
- 🔗 CALLS -> `CleanReleaseRepository.get_candidate`
- ƒ **approve_candidate_endpoint** (`Function`)
- 📝 Endpoint to record candidate approval.
- 🔗 CALLS -> `approve_candidate`
- ƒ **reject_candidate_endpoint** (`Function`)
- 📝 Endpoint to record candidate rejection.
- 🔗 CALLS -> `reject_candidate`
- ƒ **publish_candidate_endpoint** (`Function`)
- 📝 Endpoint to publish an approved candidate.
- 🔗 CALLS -> `publish_candidate`
- ƒ **revoke_publication_endpoint** (`Function`)
- 📝 Endpoint to revoke a previous publication.
- 🔗 CALLS -> `revoke_publication`
- 📦 **backend.src.api.routes.mappings** (`Module`) - 📦 **backend.src.api.routes.mappings** (`Module`)
- 📝 API endpoints for managing database mappings and getting suggestions. - 📝 API endpoints for managing database mappings and getting suggestions.
- 🏗️ Layer: API - 🏗️ Layer: API
@@ -3796,7 +3937,7 @@
- 📝 Updates an existing validation policy. - 📝 Updates an existing validation policy.
- ƒ **delete_validation_policy** (`Function`) - ƒ **delete_validation_policy** (`Function`)
- 📝 Deletes a validation policy. - 📝 Deletes a validation policy.
- 📦 **backend.src.api.routes.admin** (`Module`) - 📦 **AdminApi** (`Module`)
- 📝 Admin API endpoints for user and role management. - 📝 Admin API endpoints for user and role management.
- 🏗️ Layer: API - 🏗️ Layer: API
- 🔒 Invariant: All endpoints in this module require 'Admin' role or 'admin' scope. - 🔒 Invariant: All endpoints in this module require 'Admin' role or 'admin' scope.
@@ -4044,7 +4185,7 @@
- 🏗️ Layer: UI (API) - 🏗️ Layer: UI (API)
- 🔒 Invariant: Endpoints are read-only and do not trigger long-running tasks. - 🔒 Invariant: Endpoints are read-only and do not trigger long-running tasks.
- 🔗 DEPENDS_ON -> `backend.src.services.reports.report_service.ReportsService` - 🔗 DEPENDS_ON -> `backend.src.services.reports.report_service.ReportsService`
- 🔗 DEPENDS_ON -> `backend.src.dependencies` - 🔗 DEPENDS_ON -> `AppDependencies`
- ƒ **_parse_csv_enum_list** (`Function`) `[TRIVIAL]` - ƒ **_parse_csv_enum_list** (`Function`) `[TRIVIAL]`
- 📝 Parse comma-separated query value into enum list. - 📝 Parse comma-separated query value into enum list.
- ƒ **list_reports** (`Function`) - ƒ **list_reports** (`Function`)
@@ -4101,7 +4242,7 @@
- 📝 API endpoints for the Dashboard Hub - listing dashboards with Git and task status - 📝 API endpoints for the Dashboard Hub - listing dashboards with Git and task status
- 🏗️ Layer: API - 🏗️ Layer: API
- 🔒 Invariant: All dashboard responses include git_status and last_task metadata - 🔒 Invariant: All dashboard responses include git_status and last_task metadata
- 🔗 DEPENDS_ON -> `backend.src.dependencies` - 🔗 DEPENDS_ON -> `AppDependencies`
- 🔗 DEPENDS_ON -> `backend.src.services.resource_service.ResourceService` - 🔗 DEPENDS_ON -> `backend.src.services.resource_service.ResourceService`
- 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient` - 🔗 DEPENDS_ON -> `backend.src.core.superset_client.SupersetClient`
- 📦 **GitStatus** (`DataClass`) - 📦 **GitStatus** (`DataClass`)
@@ -4224,7 +4365,6 @@
- 📦 **backend.src.api.routes.__tests__.test_git_status_route** (`Module`) - 📦 **backend.src.api.routes.__tests__.test_git_status_route** (`Module`)
- 📝 Validate status endpoint behavior for missing and error repository states. - 📝 Validate status endpoint behavior for missing and error repository states.
- 🏗️ Layer: Domain (Tests) - 🏗️ Layer: Domain (Tests)
- 🔗 CALLS -> `src.api.routes.git.get_repository_status`
- ƒ **test_get_repository_status_returns_no_repo_payload_for_missing_repo** (`Function`) `[TRIVIAL]` - ƒ **test_get_repository_status_returns_no_repo_payload_for_missing_repo** (`Function`) `[TRIVIAL]`
- 📝 Ensure missing local repository is represented as NO_REPO payload instead of an API error. - 📝 Ensure missing local repository is represented as NO_REPO payload instead of an API error.
- ƒ **test_get_repository_status_propagates_non_404_http_exception** (`Function`) `[TRIVIAL]` - ƒ **test_get_repository_status_propagates_non_404_http_exception** (`Function`) `[TRIVIAL]`
@@ -4620,57 +4760,28 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **_get_repo_path** (`Function`) `[TRIVIAL]` - ƒ **_get_repo_path** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **backend.src.api.routes.__tests__.test_assistant_api** (`Module`) - 📦 **AssistantApiTests** (`Module`)
- 📝 Validate assistant API endpoint logic via direct async handler invocation. - 📝 Validate assistant API endpoint logic via direct async handler invocation.
- 🏗️ Layer: UI (API Tests)
- 🔒 Invariant: Every test clears assistant in-memory state before execution. - 🔒 Invariant: Every test clears assistant in-memory state before execution.
- 🔗 DEPENDS_ON -> `backend.src.api.routes.assistant` - 🔗 DEPENDS_ON -> `backend.src.api.routes.assistant`
- ƒ **_run_async** (`Function`) `[TRIVIAL]` - ƒ **_run_async** (`Function`) `[TRIVIAL]`
- 📝 Execute async endpoint handler in synchronous test context.
- ℂ **_FakeTask** (`Class`) `[TRIVIAL]` - ℂ **_FakeTask** (`Class`) `[TRIVIAL]`
- 📝 Lightweight task stub used by assistant API tests. - 🔗 BINDS_TO -> `AssistantApiTests`
- ℂ **_FakeTaskManager** (`Class`) `[TRIVIAL]` - ℂ **_FakeTaskManager** (`Class`) `[TRIVIAL]`
- 📝 Minimal async-compatible TaskManager fixture for deterministic test flows. - 🔗 BINDS_TO -> `AssistantApiTests`
- ℂ **_FakeConfigManager** (`Class`) `[TRIVIAL]` - ℂ **_FakeConfigManager** (`Class`) `[TRIVIAL]`
- 📝 Environment config fixture with dev/prod aliases for parser tests. - 🔗 BINDS_TO -> `AssistantApiTests`
- ƒ **_admin_user** (`Function`) `[TRIVIAL]` - ƒ **_admin_user** (`Function`) `[TRIVIAL]`
- 📝 Build admin principal fixture.
- ƒ **_limited_user** (`Function`) `[TRIVIAL]` - ƒ **_limited_user** (`Function`) `[TRIVIAL]`
- 📝 Build non-admin principal fixture.
- ℂ **_FakeQuery** (`Class`) `[TRIVIAL]` - ℂ **_FakeQuery** (`Class`) `[TRIVIAL]`
- 📝 Minimal chainable query object for fake SQLAlchemy-like DB behavior in tests. - 🔗 BINDS_TO -> `AssistantApiTests`
- ℂ **_FakeDb** (`Class`) `[TRIVIAL]` - ℂ **_FakeDb** (`Class`) `[TRIVIAL]`
- 📝 In-memory fake database implementing subset of Session interface used by assistant routes. - 🔗 BINDS_TO -> `AssistantApiTests`
- ƒ **_clear_assistant_state** (`Function`) `[TRIVIAL]` - ƒ **_clear_assistant_state** (`Function`) `[TRIVIAL]`
- 📝 Reset in-memory assistant registries for isolation between tests.
- ƒ **test_unknown_command_returns_needs_clarification** (`Function`) `[TRIVIAL]` - ƒ **test_unknown_command_returns_needs_clarification** (`Function`) `[TRIVIAL]`
- 📝 Unknown command should return clarification state and unknown intent. - 📝 Unknown command should return clarification state and unknown intent.
- ƒ **test_capabilities_question_returns_successful_help** (`Function`) `[TRIVIAL]` - ƒ **test_capabilities_question_returns_successful_help** (`Function`) `[TRIVIAL]`
- 📝 Capability query should return deterministic help response, not clarification. - 📝 Capability query should return deterministic help response.
- ƒ **test_non_admin_command_returns_denied** (`Function`) `[TRIVIAL]`
- 📝 Non-admin user must receive denied state for privileged command.
- ƒ **test_migration_to_prod_requires_confirmation_and_can_be_confirmed** (`Function`) `[TRIVIAL]`
- 📝 Migration to prod must require confirmation and then start task after explicit confirm.
- ƒ **test_status_query_returns_task_status** (`Function`) `[TRIVIAL]`
- 📝 Task status command must surface current status text for existing task id.
- ƒ **test_status_query_without_task_id_returns_latest_user_task** (`Function`) `[TRIVIAL]`
- 📝 Status command without explicit task_id should resolve to latest task for current user.
- ƒ **test_llm_validation_with_dashboard_ref_requires_confirmation** (`Function`) `[TRIVIAL]`
- 📝 LLM validation with dashboard_ref should now require confirmation before dispatch.
- ƒ **test_list_conversations_groups_by_conversation_and_marks_archived** (`Function`) `[TRIVIAL]`
- 📝 Conversations endpoint must group messages and compute archived marker by inactivity threshold.
- ƒ **test_history_from_latest_returns_recent_page_first** (`Function`) `[TRIVIAL]`
- 📝 History endpoint from_latest mode must return newest page while preserving chronological order in chunk.
- ƒ **test_list_conversations_archived_only_filters_active** (`Function`) `[TRIVIAL]`
- 📝 archived_only mode must return only archived conversations.
- ƒ **test_guarded_operation_always_requires_confirmation** (`Function`) `[TRIVIAL]`
- 📝 Non-dangerous (guarded) commands must still require confirmation before execution.
- ƒ **test_guarded_operation_confirm_roundtrip** (`Function`) `[TRIVIAL]`
- 📝 Guarded operation must execute successfully after explicit confirmation.
- ƒ **test_confirm_nonexistent_id_returns_404** (`Function`) `[TRIVIAL]`
- 📝 Confirming a non-existent ID should raise 404.
- ƒ **test_migration_with_dry_run_includes_summary** (`Function`) `[TRIVIAL]`
- 📝 Migration command with dry run flag must return the dry run summary in confirmation text.
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
@@ -4681,6 +4792,10 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **get_tasks** (`Function`) `[TRIVIAL]` - ƒ **get_tasks** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **get_all_tasks** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **get_environments** (`Function`) `[TRIVIAL]` - ƒ **get_environments** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **get_config** (`Function`) `[TRIVIAL]` - ƒ **get_config** (`Function`) `[TRIVIAL]`
@@ -4691,29 +4806,29 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **order_by** (`Function`) `[TRIVIAL]` - ƒ **order_by** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **limit** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **offset** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **first** (`Function`) `[TRIVIAL]` - ƒ **first** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **all** (`Function`) `[TRIVIAL]` - ƒ **all** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **count** (`Function`) `[TRIVIAL]` - ƒ **count** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **offset** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **limit** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **add** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **merge** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **query** (`Function`) `[TRIVIAL]` - ƒ **query** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **add** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **commit** (`Function`) `[TRIVIAL]` - ƒ **commit** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **rollback** (`Function`) `[TRIVIAL]` - ƒ **rollback** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **run** (`Function`) `[TRIVIAL]` - ƒ **merge** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **refresh** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **backend.src.api.routes.__tests__.test_migration_routes** (`Module`) - 📦 **backend.src.api.routes.__tests__.test_migration_routes** (`Module`)
- 📝 Unit tests for migration API route handlers. - 📝 Unit tests for migration API route handlers.
@@ -4812,7 +4927,7 @@
- 🔗 DEPENDS_ON -> `sqlalchemy` - 🔗 DEPENDS_ON -> `sqlalchemy`
- ℂ **ConnectionConfig** (`Class`) `[TRIVIAL]` - ℂ **ConnectionConfig** (`Class`) `[TRIVIAL]`
- 📝 Stores credentials for external databases used for column mapping. - 📝 Stores credentials for external databases used for column mapping.
- 📦 **backend.src.models.mapping** (`Module`) - 📦 **MappingModels** (`Module`)
- 📝 Defines the database schema for environment metadata and database mappings using SQLAlchemy. - 📝 Defines the database schema for environment metadata and database mappings using SQLAlchemy.
- 🏗️ Layer: Domain - 🏗️ Layer: Domain
- 🔒 Invariant: All primary keys are UUID strings. - 🔒 Invariant: All primary keys are UUID strings.
@@ -4958,7 +5073,7 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **check_run_id** (`Function`) `[TRIVIAL]` - ƒ **check_run_id** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- 📦 **backend.src.models.auth** (`Module`) - 📦 **AuthModels** (`Module`)
- 📝 SQLAlchemy models for multi-user authentication and authorization. - 📝 SQLAlchemy models for multi-user authentication and authorization.
- 🏗️ Layer: Domain - 🏗️ Layer: Domain
- 🔒 Invariant: Usernames and emails must be unique. - 🔒 Invariant: Usernames and emails must be unique.
@@ -4977,11 +5092,11 @@
- ℂ **ADGroupMapping** (`Class`) `[CRITICAL]` - ℂ **ADGroupMapping** (`Class`) `[CRITICAL]`
- 📝 Maps an Active Directory group to a local System Role. - 📝 Maps an Active Directory group to a local System Role.
- 🔗 DEPENDS_ON -> `Role` - 🔗 DEPENDS_ON -> `Role`
- 📦 **backend.src.models.profile** (`Module`) - 📦 **ProfileModels** (`Module`)
- 📝 Defines persistent per-user profile settings for dashboard filter, Git identity/token, and UX preferences. - 📝 Defines persistent per-user profile settings for dashboard filter, Git identity/token, and UX preferences.
- 🏗️ Layer: Domain - 🏗️ Layer: Domain
- 🔒 Invariant: Sensitive Git token is stored encrypted and never returned in plaintext. - 🔒 Invariant: Sensitive Git token is stored encrypted and never returned in plaintext.
- 🔗 DEPENDS_ON -> `backend.src.models.auth` - 🔗 DEPENDS_ON -> `AuthModels`
- ℂ **UserDashboardPreference** (`Class`) - ℂ **UserDashboardPreference** (`Class`)
- 📝 Stores Superset username binding and default "my dashboards" toggle for one authenticated user. - 📝 Stores Superset username binding and default "my dashboards" toggle for one authenticated user.
- 📦 **src.models** (`Package`) `[TRIVIAL]` - 📦 **src.models** (`Package`) `[TRIVIAL]`
@@ -5319,14 +5434,22 @@
- 🔗 DEPENDS_ON -> `backend.src.models.auth.Role` - 🔗 DEPENDS_ON -> `backend.src.models.auth.Role`
- ℂ **AuthService** (`Class`) - ℂ **AuthService** (`Class`)
- 📝 Provides high-level authentication services. - 📝 Provides high-level authentication services.
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **AuthService.__init__** (`Function`) `[TRIVIAL]`
- 📝 Initializes the authentication service with repository access over an active DB session. - 📝 Initializes the authentication service with repository access over an active DB session.
- ƒ **authenticate_user** (`Function`) - ƒ **AuthService.authenticate_user** (`Function`)
- 📝 Validates credentials and account state for local username/password authentication. - 📝 Validates credentials and account state for local username/password authentication.
- ƒ **create_session** (`Function`) - ƒ **AuthService.create_session** (`Function`)
- 📝 Issues an access token payload for an already authenticated user. - 📝 Issues an access token payload for an already authenticated user.
- ƒ **provision_adfs_user** (`Function`) - ƒ **AuthService.provision_adfs_user** (`Function`)
- 📝 Performs ADFS Just-In-Time provisioning and role synchronization from AD group mappings. - 📝 Performs ADFS Just-In-Time provisioning and role synchronization from AD group mappings.
- ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **authenticate_user** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_session** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **provision_adfs_user** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- 📦 **backend.src.services.git_service** (`Module`) - 📦 **backend.src.services.git_service** (`Module`)
- 📝 Core Git logic using GitPython to manage dashboard repositories. - 📝 Core Git logic using GitPython to manage dashboard repositories.
- 🏗️ Layer: Service - 🏗️ Layer: Service
@@ -5338,18 +5461,202 @@
- 📝 Wrapper for GitPython operations with semantic logging and error handling. - 📝 Wrapper for GitPython operations with semantic logging and error handling.
- ƒ **backend.src.services.git_service.GitService.__init__** (`Function`) `[TRIVIAL]` - ƒ **backend.src.services.git_service.GitService.__init__** (`Function`) `[TRIVIAL]`
- 📝 Initializes the GitService with a base path for repositories. - 📝 Initializes the GitService with a base path for repositories.
- ƒ **_ensure_base_path_exists** (`Function`) `[TRIVIAL]` - 🔗 CALLS -> `GitService._resolve_base_path`
- 📝 Ensure the repositories root directory exists and is a directory. - 🔗 CALLS -> `GitService._ensure_base_path_exists`
- ƒ **backend.src.services.git_service.GitService._resolve_base_path** (`Function`) `[TRIVIAL]` - ƒ **backend.src.services.git_service.GitService._ensure_base_path_exists** (`Function`) `[TRIVIAL]`
- 📝 Resolve base repository directory from explicit argument or global storage settings. - 📝 Ensure the repositories root directory exists and is a directory.
- ƒ **backend.src.services.git_service.GitService._resolve_base_path** (`Function`) `[TRIVIAL]`
- 📝 Resolve base repository directory from explicit argument or global storage settings.
- ƒ **backend.src.services.git_service.GitService._normalize_repo_key** (`Function`) `[TRIVIAL]`
- 📝 Convert user/dashboard-provided key to safe filesystem directory name.
- ƒ **backend.src.services.git_service.GitService._update_repo_local_path** (`Function`) `[TRIVIAL]`
- 📝 Persist repository local_path in GitRepository table when record exists.
- ƒ **backend.src.services.git_service.GitService._migrate_repo_directory** (`Function`) `[TRIVIAL]`
- 📝 Move legacy repository directory to target path and sync DB metadata.
- 🔗 CALLS -> `GitService._update_repo_local_path`
- ƒ **backend.src.services.git_service.GitService._ensure_gitflow_branches** (`Function`) `[TRIVIAL]`
- 📝 Ensure standard GitFlow branches (main/dev/preprod) exist locally and on origin.
- ƒ **backend.src.services.git_service.GitService._get_repo_path** (`Function`) `[TRIVIAL]`
- 📝 Resolves the local filesystem path for a dashboard's repository.
- 🔗 CALLS -> `GitService._normalize_repo_key`
- 🔗 CALLS -> `GitService._migrate_repo_directory`
- 🔗 CALLS -> `GitService._update_repo_local_path`
- ƒ **backend.src.services.git_service.GitService.init_repo** (`Function`) `[TRIVIAL]`
- 📝 Initialize or clone a repository for a dashboard.
- 🔗 CALLS -> `GitService._get_repo_path`
- 🔗 CALLS -> `GitService._ensure_gitflow_branches`
- ƒ **backend.src.services.git_service.GitService.delete_repo** (`Function`) `[TRIVIAL]`
- 📝 Remove local repository and DB binding for a dashboard.
- 🔗 CALLS -> `GitService._get_repo_path`
- ƒ **backend.src.services.git_service.GitService.get_repo** (`Function`) `[TRIVIAL]`
- 📝 Get Repo object for a dashboard.
- 🔗 CALLS -> `GitService._get_repo_path`
- ƒ **backend.src.services.git_service.GitService.configure_identity** (`Function`) `[TRIVIAL]`
- 📝 Configure repository-local Git committer identity for user-scoped operations.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.list_branches** (`Function`) `[TRIVIAL]`
- 📝 List all branches for a dashboard's repository.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.create_branch** (`Function`) `[TRIVIAL]`
- 📝 Create a new branch from an existing one.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.checkout_branch** (`Function`) `[TRIVIAL]`
- 📝 Switch to a specific branch.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.commit_changes** (`Function`) `[TRIVIAL]`
- 📝 Stage and commit changes.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService._extract_http_host** (`Function`) `[TRIVIAL]`
- 📝 Extract normalized host[:port] from HTTP(S) URL.
- ƒ **backend.src.services.git_service.GitService._strip_url_credentials** (`Function`) `[TRIVIAL]`
- 📝 Remove credentials from URL while preserving scheme/host/path.
- ƒ **backend.src.services.git_service.GitService._replace_host_in_url** (`Function`) `[TRIVIAL]`
- 📝 Replace source URL host with host from configured server URL.
- ƒ **backend.src.services.git_service.GitService._align_origin_host_with_config** (`Function`) `[TRIVIAL]`
- 📝 Auto-align local origin host to configured Git server host when they drift.
- 🔗 CALLS -> `GitService._extract_http_host`
- 🔗 CALLS -> `GitService._replace_host_in_url`
- 🔗 CALLS -> `GitService._strip_url_credentials`
- ƒ **backend.src.services.git_service.GitService.push_changes** (`Function`) `[TRIVIAL]`
- 📝 Push local commits to remote.
- 🔗 CALLS -> `GitService.get_repo`
- 🔗 CALLS -> `GitService._align_origin_host_with_config`
- ƒ **backend.src.services.git_service.GitService._read_blob_text** (`Function`) `[TRIVIAL]`
- 📝 Read text from a Git blob.
- ƒ **backend.src.services.git_service.GitService._get_unmerged_file_paths** (`Function`) `[TRIVIAL]`
- 📝 List files with merge conflicts.
- ƒ **backend.src.services.git_service.GitService._build_unfinished_merge_payload** (`Function`) `[TRIVIAL]`
- 📝 Build payload for unfinished merge state.
- 🔗 CALLS -> `GitService._get_unmerged_file_paths`
- ƒ **backend.src.services.git_service.GitService.get_merge_status** (`Function`) `[TRIVIAL]`
- 📝 Get current merge status for a dashboard repository.
- 🔗 CALLS -> `GitService.get_repo`
- 🔗 CALLS -> `GitService._build_unfinished_merge_payload`
- ƒ **backend.src.services.git_service.GitService.get_merge_conflicts** (`Function`) `[TRIVIAL]`
- 📝 List all files with conflicts and their contents.
- 🔗 CALLS -> `GitService.get_repo`
- 🔗 CALLS -> `GitService._read_blob_text`
- ƒ **backend.src.services.git_service.GitService.resolve_merge_conflicts** (`Function`) `[TRIVIAL]`
- 📝 Resolve conflicts using specified strategy.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.abort_merge** (`Function`) `[TRIVIAL]`
- 📝 Abort ongoing merge.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.continue_merge** (`Function`) `[TRIVIAL]`
- 📝 Finalize merge after conflict resolution.
- 🔗 CALLS -> `GitService.get_repo`
- 🔗 CALLS -> `GitService._get_unmerged_file_paths`
- ƒ **backend.src.services.git_service.GitService.pull_changes** (`Function`) `[TRIVIAL]`
- 📝 Pull changes from remote.
- 🔗 CALLS -> `GitService.get_repo`
- 🔗 CALLS -> `GitService._build_unfinished_merge_payload`
- ƒ **backend.src.services.git_service.GitService.get_status** (`Function`) `[TRIVIAL]`
- 📝 Get current repository status (dirty files, untracked, etc.)
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.get_diff** (`Function`) `[TRIVIAL]`
- 📝 Generate diff for a file or the whole repository.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.get_commit_history** (`Function`) `[TRIVIAL]`
- 📝 Retrieve commit history for a repository.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.test_connection** (`Function`) `[TRIVIAL]`
- 📝 Test connection to Git provider using PAT.
- ƒ **backend.src.services.git_service.GitService._normalize_git_server_url** (`Function`) `[TRIVIAL]`
- 📝 Normalize Git server URL for provider API calls.
- ƒ **backend.src.services.git_service.GitService._gitea_headers** (`Function`) `[TRIVIAL]`
- 📝 Build Gitea API authorization headers.
- ƒ **backend.src.services.git_service.GitService._gitea_request** (`Function`) `[TRIVIAL]`
- 📝 Execute HTTP request against Gitea API with stable error mapping.
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- 🔗 CALLS -> `GitService._gitea_headers`
- ƒ **backend.src.services.git_service.GitService.get_gitea_current_user** (`Function`) `[TRIVIAL]`
- 📝 Resolve current Gitea user for PAT.
- 🔗 CALLS -> `GitService._gitea_request`
- ƒ **backend.src.services.git_service.GitService.list_gitea_repositories** (`Function`) `[TRIVIAL]`
- 📝 List repositories visible to authenticated Gitea user.
- 🔗 CALLS -> `GitService._gitea_request`
- ƒ **backend.src.services.git_service.GitService.create_gitea_repository** (`Function`) `[TRIVIAL]`
- 📝 Create repository in Gitea for authenticated user.
- 🔗 CALLS -> `GitService._gitea_request`
- ƒ **backend.src.services.git_service.GitService.delete_gitea_repository** (`Function`) `[TRIVIAL]`
- 📝 Delete repository in Gitea.
- 🔗 CALLS -> `GitService._gitea_request`
- ƒ **backend.src.services.git_service.GitService._gitea_branch_exists** (`Function`) `[TRIVIAL]`
- 📝 Check whether a branch exists in Gitea repository.
- 🔗 CALLS -> `GitService._gitea_request`
- ƒ **backend.src.services.git_service.GitService._build_gitea_pr_404_detail** (`Function`) `[TRIVIAL]`
- 📝 Build actionable error detail for Gitea PR 404 responses.
- 🔗 CALLS -> `GitService._gitea_branch_exists`
- ƒ **backend.src.services.git_service.GitService.create_github_repository** (`Function`) `[TRIVIAL]`
- 📝 Create repository in GitHub or GitHub Enterprise.
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- ƒ **backend.src.services.git_service.GitService.create_gitlab_repository** (`Function`) `[TRIVIAL]`
- 📝 Create repository(project) in GitLab.
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- ƒ **backend.src.services.git_service.GitService._parse_remote_repo_identity** (`Function`) `[TRIVIAL]`
- 📝 Parse owner/repo from remote URL for Git server API operations.
- ƒ **backend.src.services.git_service.GitService._derive_server_url_from_remote** (`Function`) `[TRIVIAL]`
- 📝 Build API base URL from remote repository URL without credentials.
- ƒ **backend.src.services.git_service.GitService.promote_direct_merge** (`Function`) `[TRIVIAL]`
- 📝 Perform direct merge between branches in local repo and push target branch.
- 🔗 CALLS -> `GitService.get_repo`
- ƒ **backend.src.services.git_service.GitService.create_gitea_pull_request** (`Function`) `[TRIVIAL]`
- 📝 Create pull request in Gitea.
- 🔗 CALLS -> `GitService._parse_remote_repo_identity`
- 🔗 CALLS -> `GitService._gitea_request`
- 🔗 CALLS -> `GitService._derive_server_url_from_remote`
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- 🔗 CALLS -> `GitService._build_gitea_pr_404_detail`
- ƒ **backend.src.services.git_service.GitService.create_github_pull_request** (`Function`) `[TRIVIAL]`
- 📝 Create pull request in GitHub or GitHub Enterprise.
- 🔗 CALLS -> `GitService._parse_remote_repo_identity`
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- ƒ **backend.src.services.git_service.GitService.create_gitlab_merge_request** (`Function`) `[TRIVIAL]`
- 📝 Create merge request in GitLab.
- 🔗 CALLS -> `GitService._parse_remote_repo_identity`
- 🔗 CALLS -> `GitService._normalize_git_server_url`
- ƒ **__init__** (`Function`) `[TRIVIAL]` - ƒ **__init__** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **_ensure_base_path_exists** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_resolve_base_path** (`Function`) `[TRIVIAL]` - ƒ **_resolve_base_path** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **_normalize_repo_key** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_update_repo_local_path** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_migrate_repo_directory** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_ensure_gitflow_branches** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_get_repo_path** (`Function`) `[TRIVIAL]` - ƒ **_get_repo_path** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **init_repo** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **delete_repo** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **get_repo** (`Function`) `[TRIVIAL]` - ƒ **get_repo** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **configure_identity** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **list_branches** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_branch** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **checkout_branch** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **commit_changes** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_extract_http_host** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_strip_url_credentials** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_replace_host_in_url** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_align_origin_host_with_config** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **push_changes** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_read_blob_text** (`Function`) `[TRIVIAL]` - ƒ **_read_blob_text** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **_get_unmerged_file_paths** (`Function`) `[TRIVIAL]` - ƒ **_get_unmerged_file_paths** (`Function`) `[TRIVIAL]`
@@ -5366,10 +5673,44 @@
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **continue_merge** (`Function`) `[TRIVIAL]` - ƒ **continue_merge** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **pull_changes** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **get_status** (`Function`) `[TRIVIAL]` - ƒ **get_status** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **get_diff** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **get_commit_history** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **test_connection** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_normalize_git_server_url** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_gitea_headers** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_gitea_request** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **get_gitea_current_user** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **list_gitea_repositories** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_gitea_repository** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **delete_gitea_repository** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_gitea_branch_exists** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_build_gitea_pr_404_detail** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_github_repository** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_gitlab_repository** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_parse_remote_repo_identity** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **_derive_server_url_from_remote** (`Function`) `[TRIVIAL]` - ƒ **_derive_server_url_from_remote** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **promote_direct_merge** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan)
- ƒ **create_gitea_pull_request** (`Function`) `[TRIVIAL]` - ƒ **create_gitea_pull_request** (`Function`) `[TRIVIAL]`
- 📝 Auto-detected function (orphan) - 📝 Auto-detected function (orphan)
- ƒ **create_github_pull_request** (`Function`) `[TRIVIAL]` - ƒ **create_github_pull_request** (`Function`) `[TRIVIAL]`

8
.kilocode/rules/specify-rules.md
View File

@@ -51,6 +51,10 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
- Existing auth database (`AUTH_DATABASE_URL`) with a dedicated per-user preference entity (024-user-dashboard-filter) - Existing auth database (`AUTH_DATABASE_URL`) with a dedicated per-user preference entity (024-user-dashboard-filter)
- Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend) (026-dashboard-health-windows) - Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend) (026-dashboard-health-windows)
- PostgreSQL / SQLite (existing database for `ValidationRecord` and new `ValidationPolicy`) (026-dashboard-health-windows) - PostgreSQL / SQLite (existing database for `ValidationRecord` and new `ValidationPolicy`) (026-dashboard-health-windows)
- Python 3.9+ backend, Node.js 18+ frontend with Svelte 5 / SvelteKit + FastAPI, SQLAlchemy, Pydantic, existing [SupersetClient](../../backend/src/core/superset_client.py), existing frontend API wrapper patterns, Svelte runes, existing task/websocket stack (027-dataset-llm-orchestration)
- Existing application databases plus filesystem-backed uploaded semantic sources; reuse current configuration and task persistence stores (027-dataset-llm-orchestration)
- Python 3.9+ backend, Node.js 18+ frontend, Svelte 5 / SvelteKit frontend runtime + FastAPI, SQLAlchemy, Pydantic, existing `TaskManager`, existing `SupersetClient`, existing LLM provider stack, SvelteKit, Tailwind CSS, frontend `requestApi`/`fetchApi` wrappers (027-dataset-llm-orchestration)
- Existing application databases for persistent session/domain entities; existing tasks database for async execution metadata; filesystem for optional uploaded semantic sources/artifacts (027-dataset-llm-orchestration)
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui) - Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
@@ -71,9 +75,9 @@ cd src; pytest; ruff check .
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
## Recent Changes ## Recent Changes
- 027-dataset-llm-orchestration: Added Python 3.9+ backend, Node.js 18+ frontend, Svelte 5 / SvelteKit frontend runtime + FastAPI, SQLAlchemy, Pydantic, existing `TaskManager`, existing `SupersetClient`, existing LLM provider stack, SvelteKit, Tailwind CSS, frontend `requestApi`/`fetchApi` wrappers
- 027-dataset-llm-orchestration: Added Python 3.9+ backend, Node.js 18+ frontend with Svelte 5 / SvelteKit + FastAPI, SQLAlchemy, Pydantic, existing [SupersetClient](../../backend/src/core/superset_client.py), existing frontend API wrapper patterns, Svelte runes, existing task/websocket stack
- 026-dashboard-health-windows: Added Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend) - 026-dashboard-health-windows: Added Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend)
- 024-user-dashboard-filter: Added Python 3.9+ (backend), Node.js 18+ + SvelteKit (frontend) + FastAPI, SQLAlchemy, Pydantic, existing auth stack (`get_current_user`), existing dashboards route/service, Svelte runes (`$state`, `$derived`, `$effect`), Tailwind CSS, frontend `api` wrapper
- 020-clean-repo-enterprise: Added Python 3.9+ (backend scripts/services), Shell (release tooling) + FastAPI stack (existing backend), ConfigManager, TaskManager, файловые утилиты, internal artifact registries
<!-- MANUAL ADDITIONS START --> <!-- MANUAL ADDITIONS START -->

51
.kilocode/workflows/speckit.implement.md
View File

@@ -1,9 +1,13 @@
--- ---
description: Execute the implementation plan by processing and executing all tasks defined in tasks.md description: Execute the implementation plan by processing and executing all tasks defined in tasks.md
handoffs: handoffs:
- label: Verify Changes - label: Audit & Verify (Tester)
agent: speckit.test agent: tester
prompt: Verify the implementation of... prompt: Perform semantic audit, algorithm emulation, and unit test verification for the completed tasks.
send: true
- label: Orchestration Control
agent: orchestrator
prompt: Review Tester's feedback and coordinate next steps.
send: true send: true
--- ---
@@ -122,6 +126,7 @@ You **MUST** consider the user input before proceeding (if not empty).
- Include `@TIER` and define contracts (`@PRE`, `@POST`). - Include `@TIER` and define contracts (`@PRE`, `@POST`).
- For Svelte components, use `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and explicitly declare reactivity with `@UX_REATIVITY: State: $state, Derived: $derived`. - For Svelte components, use `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and explicitly declare reactivity with `@UX_REATIVITY: State: $state, Derived: $derived`.
- **Molecular Topology Logging**: Use prefixes `[EXPLORE]`, `[REASON]`, `[REFLECT]` in logs to trace logic. - **Molecular Topology Logging**: Use prefixes `[EXPLORE]`, `[REASON]`, `[REFLECT]` in logs to trace logic.
- **Self-Audit**: The Coder MUST use `axiom-core` tools (like `audit_contracts_tool`) to verify semantic compliance before completion.
- **CRITICAL Contracts**: If a task description contains a contract summary (e.g., `CRITICAL: PRE: ..., POST: ...`), these constraints are **MANDATORY** and must be strictly implemented in the code using guards/assertions (if applicable per protocol). - **CRITICAL Contracts**: If a task description contains a contract summary (e.g., `CRITICAL: PRE: ..., POST: ...`), these constraints are **MANDATORY** and must be strictly implemented in the code using guards/assertions (if applicable per protocol).
- **Setup first**: Initialize project structure, dependencies, configuration - **Setup first**: Initialize project structure, dependencies, configuration
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios - **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
@@ -130,18 +135,32 @@ You **MUST** consider the user input before proceeding (if not empty).
- **Polish and validation**: Unit tests, performance optimization, documentation - **Polish and validation**: Unit tests, performance optimization, documentation
8. Progress tracking and error handling: 8. Progress tracking and error handling:
- Report progress after each completed task - Report progress after each completed task.
- Halt execution if any non-parallel task fails - Halt execution if any non-parallel task fails.
- For parallel tasks [P], continue with successful tasks, report failed ones - For parallel tasks [P], continue with successful tasks, report failed ones.
- Provide clear error messages with context for debugging - Provide clear error messages with context for debugging.
- Suggest next steps if implementation cannot proceed - Suggest next steps if implementation cannot proceed.
- **IMPORTANT** For completed tasks, make sure to mark the task off as [X] in the tasks file. - **IMPORTANT** For completed tasks, mark as [X] only AFTER local verification and self-audit.
9. Completion validation: 9. **Handoff to Tester (Audit Loop)**:
- Verify all required tasks are completed - Once a task or phase is complete, the Coder hands off to the Tester.
- Check that implemented features match the original specification - Handoff includes: File paths, expected contracts (@PRE/@POST), and logic overview.
- Validate that tests pass and coverage meets requirements
- Confirm the implementation follows the technical plan 10. **Tester Verification & Orchestrator Gate**:
- Report final status with summary of completed work - Tester MUST:
- Perform mandatory semantic audit (using `audit_contracts_tool`).
- Emulate algorithms "in mind" step-by-step to ensure logic consistency.
- Verify unit tests match the declared contracts.
- If Tester finds issues:
- Emit `[AUDIT_FAIL: reason]` and notify the Orchestrator.
- Orchestrator redirects the feedback to the Coder for remediation.
- Acceptance (Final mark [X]):
- Only after the Tester is satisfied with semantics, emulation, and tests.
11. Completion validation:
- Verify all required tasks are completed and accepted by the Tester.
- Check that implemented features match the original specification.
- Confirm the implementation follows the technical plan and GRACE standards.
- Report final status with summary of completed and audited work.
Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/speckit.tasks` first to regenerate the task list. Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/speckit.tasks` first to regenerate the task list.

22
.kilocode/workflows/speckit.plan.md
View File

@@ -73,13 +73,23 @@ You **MUST** consider the user input before proceeding (if not empty).
- Entity name, fields, relationships, validation rules. - Entity name, fields, relationships, validation rules.
2. **Design & Verify Contracts (Semantic Protocol)**: 2. **Design & Verify Contracts (Semantic Protocol)**:
- **Drafting**: Define `[DEF:id:Type]` Headers, Contracts, and closing `[/DEF:id:Type]` for all new modules based on `.ai/standards/semantics.md`. - **Drafting**: Define semantic headers, metadata, and closing anchors for all new modules strictly from `.ai/standards/semantics.md`.
- **TIER Classification**: Explicitly assign `@TIER: [CRITICAL|STANDARD|TRIVIAL]` to each module. - **Complexity Classification**: Classify each contract with `@COMPLEXITY: [1|2|3|4|5]` or `@C:`. Treat `@TIER` only as a legacy compatibility hint and never as the primary rule source.
- **CRITICAL Requirements**: For all CRITICAL modules, define full `@PRE`, `@POST`, and (if UI) `@UX_STATE` contracts. **MUST** also define testing contracts: `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT`. - **Adaptive Contract Requirements**:
- **Complexity 1**: anchors only; `@PURPOSE` optional.
- **Complexity 2**: require `@PURPOSE`.
- **Complexity 3**: require `@PURPOSE` and `@RELATION`; UI also requires `@UX_STATE`.
- **Complexity 4**: require `@PURPOSE`, `@RELATION`, `@PRE`, `@POST`, `@SIDE_EFFECT`; Python modules must define a meaningful `logger.reason()` / `logger.reflect()` path or equivalent belief-state mechanism.
- **Complexity 5**: require full level-4 contract plus `@DATA_CONTRACT` and `@INVARIANT`; Python modules must require `belief_scope`; UI modules must define UX contracts including `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and `@UX_REACTIVITY`.
- **Relation Syntax**: Write dependency edges in canonical GraphRAG form: `@RELATION: [PREDICATE] ->[TARGET_ID]`.
- **Context Guard**: If a target relation, DTO, or required dependency cannot be named confidently, stop generation and emit `[NEED_CONTEXT: target]` instead of inventing placeholders.
- **Testing Contracts**: Add `@TEST_CONTRACT`, `@TEST_SCENARIO`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT` when the design introduces audit-critical or explicitly test-governed contracts, especially for Complexity 5 boundaries.
- **Self-Review**: - **Self-Review**:
- *Completeness*: Do `@PRE`/`@POST` cover edge cases identified in Research? Are test contracts present for CRITICAL? - *Complexity Fit*: Does each contract include exactly the metadata and contract density required by its complexity level?
- *Connectivity*: Do `@RELATION` tags form a coherent graph? - *Completeness*: Do `@PRE`/`@POST`, `@SIDE_EFFECT`, `@DATA_CONTRACT`, and UX tags cover the edge cases identified in Research and UX Reference?
- *Compliance*: Does syntax match `[DEF:id:Type]` exactly and is it closed with `[/DEF:id:Type]`? - *Connectivity*: Do `@RELATION` tags form a coherent graph using canonical `@RELATION: [PREDICATE] ->[TARGET_ID]` syntax?
- *Compliance*: Are all anchors properly opened and closed, and does the chosen comment syntax match the target medium?
- *Belief-State Requirements*: Do Complexity 4/5 Python modules explicitly account for `logger.reason()`, `logger.reflect()`, and `belief_scope` requirements?
- **Output**: Write verified contracts to `contracts/modules.md`. - **Output**: Write verified contracts to `contracts/modules.md`.
3. **Simulate Contract Usage**: 3. **Simulate Contract Usage**:

9
.kilocode/workflows/speckit.tasks.md
View File

@@ -70,11 +70,12 @@ The tasks.md should be immediately executable - each task must be specific enoug
**Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach. **Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach.
### UX Preservation (CRITICAL) ### UX & Semantic Preservation (CRITICAL)
- **Source of Truth**: `ux_reference.md` is the absolute standard for the "feel" of the feature. - **Source of Truth**: `ux_reference.md` for UX, `.ai/standards/semantics.md` for Code.
- **Violation Warning**: If any task would inherently violate the UX (e.g. "Remove progress bar to simplify code"), you **MUST** flag this to the user immediately. - **Violation Warning**: If any task violates UX or GRACE standards, flag it immediately.
- **Verification Task**: You **MUST** add a specific task at the end of each User Story phase: `- [ ] Txxx [USx] Verify implementation matches ux_reference.md (Happy Path & Errors)` - **Verification Task (UX)**: Add a task at the end of each Story phase: `- [ ] Txxx [USx] Verify implementation matches ux_reference.md (Happy Path & Errors)`
- **Verification Task (Audit)**: Add a mandatory audit task at the end of each Story phase: `- [ ] Txxx [USx] Acceptance: Perform semantic audit & algorithm emulation by Tester`
### Checklist Format (REQUIRED) ### Checklist Format (REQUIRED)

21
.kilocode/workflows/speckit.test.md
View File

@@ -14,7 +14,7 @@ You **MUST** consider the user input before proceeding (if not empty).
## Goal ## Goal
Execute full testing cycle: analyze code for testable modules, write tests with proper coverage, maintain test documentation, and ensure no test duplication or deletion. Execute semantic audit and full testing cycle: verify contract compliance, emulate logic, ensure maximum coverage, and maintain test quality.
## Operating Constraints ## Operating Constraints
@@ -56,13 +56,24 @@ Create coverage matrix:
|--------|------|-----------|------|----------------------| |--------|------|-----------|------|----------------------|
| ... | ... | ... | ... | ... | | ... | ... | ... | ... | ... |
### 4. Write Tests (TDD Approach) ### 4. Semantic Audit & Logic Emulation (CRITICAL)
Before writing tests, the Tester MUST:
1. **Run `axiom-core.audit_contracts_tool`**: Identify semantic violations.
2. **Emulate Algorithm**: Step through the code implementation in mind.
- Verify it adheres to the `@PURPOSE` and `@INVARIANT`.
- Verify `@PRE` and `@POST` conditions are correctly handled.
3. **Validation Verdict**:
- If audit fails: Emit `[AUDIT_FAIL: reason]` and notify Orchestrator.
- If audit passes: Proceed to writing/verifying tests.
### 5. Write Tests (TDD Approach)
For each module requiring tests: For each module requiring tests:
1. **Check existing tests**: Scan `__tests__/` for duplicates 1. **Check existing tests**: Scan `__tests__/` for duplicates.
2. **Read TEST_FIXTURE**: If CRITICAL tier, read @TEST_FIXTURE from semantics header 2. **Read TEST_FIXTURE**: If CRITICAL tier, read @TEST_FIXTURE from semantics header.
3. **Write test**: Follow co-location strategy 3. **Write test**: Follow co-location strategy.
- Python: `src/module/__tests__/test_module.py` - Python: `src/module/__tests__/test_module.py`
- Svelte: `src/lib/components/__tests__/test_component.test.js` - Svelte: `src/lib/components/__tests__/test_component.test.js`
4. **Use mocks**: Use `unittest.mock.MagicMock` for external dependencies 4. **Use mocks**: Use `unittest.mock.MagicMock` for external dependencies

22
.kilocodemodes
View File

@@ -1,10 +1,12 @@
customModes: customModes:
- slug: tester - slug: tester
name: Tester name: Tester
description: QA and Test Engineer - Full Testing Cycle description: QA & Semantic Auditor - Verification Cycle
roleDefinition: |- roleDefinition: |-
You are Kilo Code, acting as a QA and Test Engineer. Your primary goal is to ensure maximum test coverage, maintain test quality, and preserve existing tests. You are Kilo Code, acting as a QA and Semantic Auditor. Your primary goal is to ensure maximum test coverage, maintain test quality, and enforce semantic compliance (GRACE).
Your responsibilities include: Your responsibilities include:
- SEMANTIC AUDIT: Perform mandatory semantic audits using `axiom-core` tools to verify contract pairing and tag correctness.
- ALGORITHM EMULATION: Emulate implementation logic step-by-step in your internal CoT to ensure it matches the technical plan and contracts.
- WRITING TESTS: Create comprehensive unit tests following TDD principles, using co-location strategy (`__tests__` directories). - WRITING TESTS: Create comprehensive unit tests following TDD principles, using co-location strategy (`__tests__` directories).
- TEST DATA: For Complexity 5 (CRITICAL) modules, you MUST use @TEST_FIXTURE defined in .ai/standards/semantics.md. Read and apply them in your tests. - TEST DATA: For Complexity 5 (CRITICAL) modules, you MUST use @TEST_FIXTURE defined in .ai/standards/semantics.md. Read and apply them in your tests.
- DOCUMENTATION: Maintain test documentation in `specs/<feature>/tests/` directory with coverage reports and test case specifications. - DOCUMENTATION: Maintain test documentation in `specs/<feature>/tests/` directory with coverage reports and test case specifications.
@@ -19,7 +21,11 @@ customModes:
- mcp - mcp
customInstructions: | customInstructions: |
1. KNOWLEDGE GRAPH: ALWAYS read .ai/ROOT.md first to understand the project structure and navigation. 1. KNOWLEDGE GRAPH: ALWAYS read .ai/ROOT.md first to understand the project structure and navigation.
2. TEST MARKUP (Section VIII): 2. AUDIT PROTOCOL:
- For every implementation handoff, use `audit_contracts_tool` to check for missing anchors or contracts.
- Perform step-by-step logic emulation for Complexity 4-5 modules.
- If issues are found, emit `[AUDIT_FAIL: reason]` and pass to Orchestrator.
3. TEST MARKUP (Section VIII):
- Use short semantic IDs for modules (e.g., [DEF:AuthTests:Module]). - Use short semantic IDs for modules (e.g., [DEF:AuthTests:Module]).
- Use BINDS_TO only for major logic blocks (classes, complex mocks). - Use BINDS_TO only for major logic blocks (classes, complex mocks).
- Helpers remain Complexity 1 (no @PURPOSE/@RELATION needed). - Helpers remain Complexity 1 (no @PURPOSE/@RELATION needed).
@@ -38,6 +44,7 @@ customModes:
Your purpose is to rigorously execute the workflows defined in `.kilocode/workflows/`. Your purpose is to rigorously execute the workflows defined in `.kilocode/workflows/`.
You act as the orchestrator for: - Specification (`speckit.specify`, `speckit.clarify`) - Planning (`speckit.plan`) - Task Management (`speckit.tasks`, `speckit.taskstoissues`) - Quality Assurance (`speckit.analyze`, `speckit.checklist`, `speckit.test`, `speckit.fix`) - Governance (`speckit.constitution`) - Implementation Oversight (`speckit.implement`) You act as the orchestrator for: - Specification (`speckit.specify`, `speckit.clarify`) - Planning (`speckit.plan`) - Task Management (`speckit.tasks`, `speckit.taskstoissues`) - Quality Assurance (`speckit.analyze`, `speckit.checklist`, `speckit.test`, `speckit.fix`) - Governance (`speckit.constitution`) - Implementation Oversight (`speckit.implement`)
For each task, you must read the relevant workflow file from `.kilocode/workflows/` and follow its Execution Steps precisely. For each task, you must read the relevant workflow file from `.kilocode/workflows/` and follow its Execution Steps precisely.
In Implementation (speckit.implement), you manage the acceptance loop between Coder and Tester.
whenToUse: Use this mode when you need to run any /speckit.* command or when dealing with high-level feature planning, specification writing, or project management tasks. whenToUse: Use this mode when you need to run any /speckit.* command or when dealing with high-level feature planning, specification writing, or project management tasks.
description: Executes SpecKit workflows for feature management description: Executes SpecKit workflows for feature management
customInstructions: 1. Always read `.ai/ROOT.md` first to understand the Knowledge Graph structure. 2. Read the specific workflow file in `.kilocode/workflows/` before executing a command. 3. Adhere strictly to the "Operating Constraints" and "Execution Steps" in the workflow files. customInstructions: 1. Always read `.ai/ROOT.md` first to understand the Knowledge Graph structure. 2. Read the specific workflow file in `.kilocode/workflows/` before executing a command. 3. Adhere strictly to the "Operating Constraints" and "Execution Steps" in the workflow files.
@@ -49,14 +56,15 @@ customModes:
source: project source: project
- slug: coder - slug: coder
name: Coder name: Coder
roleDefinition: You are Kilo Code, acting as an Implementation Specialist. Your primary goal is to write code that strictly follows the Semantic Protocol defined in `.ai/standards/semantics.md`. roleDefinition: You are Kilo Code, acting as an Implementation Specialist. Your primary goal is to write code that strictly follows the Semantic Protocol defined in `.ai/standards/semantics.md` and passes self-audit.
whenToUse: Use this mode when you need to implement features, write code, or fix issues based on test reports. whenToUse: Use this mode when you need to implement features, write code, or fix issues based on test reports.
description: Implementation Specialist - Semantic Protocol Compliant description: Implementation Specialist - Semantic Protocol Compliant
customInstructions: | customInstructions: |
1. KNOWLEDGE GRAPH: ALWAYS read .ai/ROOT.md first to understand the project structure and navigation. 1. KNOWLEDGE GRAPH: ALWAYS read .ai/ROOT.md first to understand the project structure and navigation.
2. CONSTITUTION: Strictly follow architectural invariants in .ai/standards/constitution.md. 2. SELF-AUDIT: After implementation, use `axiom-core` tools to verify semantic compliance before handing off to Tester.
3. SEMANTIC PROTOCOL: ALWAYS use .ai/standards/semantics.md as your source of truth for syntax. 3. CONSTITUTION: Strictly follow architectural invariants in .ai/standards/constitution.md.
4. ANCHOR FORMAT: Use short semantic IDs (e.g., [DEF:AuthService:Class]). 4. SEMANTIC PROTOCOL: ALWAYS use .ai/standards/semantics.md as your source of truth for syntax.
5. ANCHOR FORMAT: Use short semantic IDs (e.g., [DEF:AuthService:Class]).
5. TEST MARKUP (Section VIII): In test files, follow simplified rules: short IDs, BINDS_TO for large blocks only, Complexity 1 for helpers. 5. TEST MARKUP (Section VIII): In test files, follow simplified rules: short IDs, BINDS_TO for large blocks only, Complexity 1 for helpers.
6. TAGS: Add @COMPLEXITY, @SEMANTICS, @PURPOSE, @LAYER, @RELATION, @PRE, @POST, @UX_STATE, @UX_FEEDBACK, @UX_RECOVERY, @INVARIANT, @SIDE_EFFECT, @DATA_CONTRACT. 6. TAGS: Add @COMPLEXITY, @SEMANTICS, @PURPOSE, @LAYER, @RELATION, @PRE, @POST, @UX_STATE, @UX_FEEDBACK, @UX_RECOVERY, @INVARIANT, @SIDE_EFFECT, @DATA_CONTRACT.
4. COMPLEXITY COMPLIANCE (1-5): 4. COMPLEXITY COMPLIANCE (1-5):

18
.specify/templates/plan-template.md
View File

@@ -31,7 +31,7 @@
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* *GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
[Gates determined based on constitution file] [Evaluate against constitution.md and semantics.md. Explicitly confirm semantic protocol compliance, complexity-driven contract coverage, UX-state compatibility, async boundaries, API-wrapper rules, RBAC/security constraints, and any required belief-state/logging constraints for Complexity 4/5 Python modules.]
## Project Structure ## Project Structure
@@ -94,6 +94,22 @@ ios/ or android/
**Structure Decision**: [Document the selected structure and reference the real **Structure Decision**: [Document the selected structure and reference the real
directories captured above] directories captured above]
## Semantic Contract Guidance
> Use this section to drive Phase 1 artifacts, especially `contracts/modules.md`.
- Classify each planned module/component with `@COMPLEXITY: 1..5` or `@C:`.
- Use `@TIER` only if backward compatibility is needed; never use it as the primary contract rule.
- Match contract density to complexity:
- Complexity 1: anchors only, `@PURPOSE` optional
- Complexity 2: `@PURPOSE`
- Complexity 3: `@PURPOSE`, `@RELATION`; UI also `@UX_STATE`
- Complexity 4: `@PURPOSE`, `@RELATION`, `@PRE`, `@POST`, `@SIDE_EFFECT`; Python also meaningful `logger.reason()` / `logger.reflect()` path
- Complexity 5: level 4 + `@DATA_CONTRACT`, `@INVARIANT`; Python also `belief_scope`; UI also `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY`
- Write relations only in canonical form: `@RELATION: [PREDICATE] ->[TARGET_ID]`
- If any relation target, DTO, or contract dependency is unknown, emit `[NEED_CONTEXT: target]` instead of inventing placeholders.
- Preserve medium-appropriate anchor/comment syntax for Python, Svelte markup, and Svelte script contexts.
## Complexity Tracking ## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified** > **Fill ONLY if Constitution Check has violations that must be justified**

6
.specify/templates/tasks-template.md
View File

@@ -8,7 +8,7 @@ description: "Task list template for feature implementation"
**Input**: Design documents from `/specs/[###-feature-name]/` **Input**: Design documents from `/specs/[###-feature-name]/`
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/ **Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/
**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification. **Tests**: Include test tasks whenever required by the feature specification, the semantic contracts, or any Complexity 5 / audit-critical boundary. Test work must trace to contract requirements, not only to implementation details.
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story. **Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
@@ -249,3 +249,7 @@ With multiple developers:
- Commit after each task or logical group - Commit after each task or logical group
- Stop at any checkpoint to validate story independently - Stop at any checkpoint to validate story independently
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence - Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence
- Derive implementation tasks from semantic contracts in `contracts/modules.md`, especially `@PRE`, `@POST`, `@SIDE_EFFECT`, `@DATA_CONTRACT`, and UI `@UX_*` tags
- For Complexity 4/5 Python modules, include tasks for belief-state logging paths with `logger.reason()`, `logger.reflect()`, and `belief_scope` where required
- For Complexity 5 or explicitly test-governed contracts, include tasks that cover `@TEST_CONTRACT`, `@TEST_SCENARIO`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT`
- Never create tasks from legacy `@TIER` alone; complexity is the primary execution signal

15
.specify/templates/test-docs-template.md
View File

@@ -21,7 +21,9 @@ description: "Test documentation template for feature implementation"
- [ ] Unit Tests (co-located in `__tests__/` directories) - [ ] Unit Tests (co-located in `__tests__/` directories)
- [ ] Integration Tests (if needed) - [ ] Integration Tests (if needed)
- [ ] E2E Tests (if critical user flows) - [ ] E2E Tests (if critical user flows)
- [ ] Contract Tests (for API endpoints) - [ ] Contract Tests (for API endpoints and semantic contract boundaries)
- [ ] Semantic Contract Verification (`@PRE`, `@POST`, `@SIDE_EFFECT`, `@DATA_CONTRACT`, `@TEST_*`)
- [ ] UX Contract Verification (`@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY`)
--- ---
@@ -72,12 +74,14 @@ description: "Test documentation template for feature implementation"
### ✅ DO ### ✅ DO
1. Write tests BEFORE implementation (TDD approach) 1. Write tests BEFORE implementation when the workflow permits it
2. Use co-location: `src/module/__tests__/test_module.py` 2. Use co-location: `src/module/__tests__/test_module.py`
3. Use MagicMock for external dependencies (DB, Auth, APIs) 3. Use MagicMock for external dependencies (DB, Auth, APIs)
4. Include semantic annotations: `# @RELATION: VERIFIES -> module.name` 4. Trace tests to semantic contracts and DTO boundaries, not just filenames
5. Test edge cases and error conditions 5. Test edge cases and error conditions
6. **Test UX states** for Svelte components (@UX_STATE, @UX_FEEDBACK, @UX_RECOVERY) 6. **Test UX contracts** for Svelte components (`@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY`)
7. For Complexity 5 boundaries, verify `@DATA_CONTRACT`, invariants, and declared `@TEST_*` metadata
8. For Complexity 4/5 Python flows, verify behavior around guards, side effects, and belief-state-driven logging paths where applicable
### ❌ DON'T ### ❌ DON'T
@@ -86,7 +90,8 @@ description: "Test documentation template for feature implementation"
3. Test implementation details, not behavior 3. Test implementation details, not behavior
4. Use real external services in unit tests 4. Use real external services in unit tests
5. Skip error handling tests 5. Skip error handling tests
6. **Skip UX contract tests** for CRITICAL frontend components 6. **Skip UX contract tests** for critical frontend components
7. Treat legacy `@TIER` as sufficient proof of test scope without checking actual complexity and contract metadata
--- ---

12
.specify/templates/ux-reference-template.md
View File

@@ -39,15 +39,23 @@ $ command --flag value
* **Key Elements**: * **Key Elements**:
* **[Button Name]**: Primary action. Color: Blue. * **[Button Name]**: Primary action. Color: Blue.
* **[Input Field]**: Placeholder text: "Enter your name...". Validation: Real-time. * **[Input Field]**: Placeholder text: "Enter your name...". Validation: Real-time.
* **Contract Mapping**:
* **`@UX_STATE`**: Enumerate the explicit UI states that must appear later in `contracts/modules.md`
* **`@UX_FEEDBACK`**: Define visible system reactions for success, validation, and failure
* **`@UX_RECOVERY`**: Define what the user can do after failure or degraded state
* **`@UX_REACTIVITY`**: Note expected Svelte rune bindings with `$state`, `$derived`, `$effect`, `$props`
* **States**: * **States**:
* **Default**: Clean state, waiting for input. * **Idle/Default**: Clean state, waiting for input.
* **Loading**: Skeleton loader replaces content area. * **Loading**: Skeleton loader replaces content area.
* **Success**: Toast notification appears top-right: "Saved!" (Green). * **Success**: Toast notification appears top-right and state is recoverable without reload.
* **Error/Degraded**: Visible failure state with explicit recovery path.
## 4. The "Error" Experience ## 4. The "Error" Experience
**Philosophy**: Don't just report the error; guide the user to the fix. **Philosophy**: Don't just report the error; guide the user to the fix.
**Semantic Requirement**: Every documented failure path here should map to `@UX_RECOVERY` and, where relevant, `@UX_FEEDBACK` in the generated component contracts.
### Scenario A: [Common Error, e.g. Invalid Input] ### Scenario A: [Common Error, e.g. Invalid Input]
* **User Action**: Enters "123" in a text-only field. * **User Action**: Enters "123" in a text-only field.

54
merge_spec.py Normal file
View File

@@ -0,0 +1,54 @@
import os
import sys
from datetime import datetime
from pathlib import Path
def merge_specs(feature_number):
specs_dir = Path("specs")
if not specs_dir.exists():
print("Error: 'specs' directory not found.")
return
# Find the directory starting with the feature number
target_dir = None
for item in specs_dir.iterdir():
if item.is_dir() and item.name.startswith(f"{feature_number}-"):
target_dir = item
break
if not target_dir:
print(f"Error: No directory found for feature number '{feature_number}' in 'specs/'.")
return
feature_name = target_dir.name
now = datetime.now().strftime("%Y%m%d-%H%M%S")
output_filename = f"{feature_name}-{now}.md"
content_blocks = ["Мой коллега предложил такую фичу, оцени ее\n"]
# Recursively find all files
files_to_merge = sorted([f for f in target_dir.rglob("*") if f.is_file()])
for file_path in files_to_merge:
relative_path = file_path.relative_to(target_dir)
try:
with open(file_path, "r", encoding="utf-8") as f:
file_content = f.read()
content_blocks.append(f"--- FILE: {relative_path} ---\n")
content_blocks.append(file_content)
content_blocks.append("\n")
except Exception as e:
print(f"Skipping file {file_path} due to error: {e}")
with open(output_filename, "w", encoding="utf-8") as f:
f.write("\n".join(content_blocks))
print(f"Successfully created: {output_filename}")
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python merge_spec.py <feature_number>")
sys.exit(1)
merge_specs(sys.argv[1])

13490
semantics/semantic_map.json
View File

File diff suppressed because it is too large Load Diff

43
specs/027-dataset-llm-orchestration/checklists/requirements.md Normal file
View File

@@ -0,0 +1,43 @@
# Specification Quality Checklist: LLM Dataset Orchestration
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-03-16
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## UX Consistency
- [x] Functional requirements fully support the 'Happy Path' in ux_reference.md
- [x] Error handling requirements match the 'Error Experience' in ux_reference.md
- [x] No requirements contradict the defined User Persona or Context
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Validation completed against [spec.md](../spec.md) and [ux_reference.md](../ux_reference.md).
- Automatic documentation, guided clarification, and Superset-derived dataset execution are all represented as independently testable user journeys.
- Error recovery is aligned between the UX reference and the functional requirements, especially for partial filter import, missing run-time values, and conflicting metadata.
- The specification is ready for the next phase.

1479
specs/027-dataset-llm-orchestration/contracts/api.yaml Normal file
View File

File diff suppressed because it is too large Load Diff

459
specs/027-dataset-llm-orchestration/contracts/modules.md Normal file
View File

@@ -0,0 +1,459 @@
# Semantic Module Contracts: LLM Dataset Orchestration
**Feature**: [LLM Dataset Orchestration](../spec.md)
**Branch**: `027-dataset-llm-orchestration`
This document defines the semantic contracts for the core components of the Dataset LLM Orchestration feature, following the [GRACE-Poly Standard](../../../.ai/standards/semantics.md).
---
## 1. Backend Modules
# [DEF:DatasetReviewOrchestrator:Module]
# @COMPLEXITY: 5
# @PURPOSE: Coordinate the full dataset review session lifecycle across intake, recovery, semantic review, clarification, mapping review, preview generation, and launch.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[DatasetReviewSessionRepository]
# @RELATION: [DEPENDS_ON] ->[SemanticSourceResolver]
# @RELATION: [DEPENDS_ON] ->[ClarificationEngine]
# @RELATION: [DEPENDS_ON] ->[SupersetContextExtractor]
# @RELATION: [DEPENDS_ON] ->[SupersetCompilationAdapter]
# @RELATION: [DEPENDS_ON] ->[TaskManager]
# @PRE: session mutations must execute inside a persisted session boundary scoped to one authenticated user.
# @POST: state transitions are persisted atomically and emit observable progress for long-running steps.
# @SIDE_EFFECT: creates task records, updates session aggregates, triggers upstream Superset calls, persists audit artifacts.
# @DATA_CONTRACT: Input[SessionCommand] -> Output[DatasetReviewSession | CompiledPreview | DatasetRunContext]
# @INVARIANT: Launch is blocked unless a current session has no open blocking findings, all launch-sensitive mappings are approved, and a non-stale Superset-generated compiled preview matches the current input fingerprint.
# @TEST_CONTRACT: start_or_resume_session -> returns persisted session shell with recommended next action
# @TEST_SCENARIO: launch_gate_blocks_stale_preview -> launch rejected when preview fingerprint no longer matches current mapping inputs
# @TEST_EDGE: missing_dataset_ref -> blocking failure
# @TEST_EDGE: stale_preview -> blocking failure
# @TEST_EDGE: sql_lab_launch_failure -> terminal failed launch state with audit record
# @TEST_INVARIANT: launch_gate -> VERIFIED_BY: [launch_gate_blocks_stale_preview]
#### ƒ **start_session**
# @PURPOSE: Initialize a new session from a Superset link or dataset selection and trigger context recovery.
# @PRE: source input is non-empty and environment is accessible.
# @POST: session exists in persisted storage with intake/recovery state and task linkage when async work is required.
# @SIDE_EFFECT: persists session and may enqueue recovery task.
#### ƒ **apply_semantic_source**
# @PURPOSE: Apply a selected semantic source and update field-level candidate/decision state.
# @PRE: source exists and session is not terminal.
# @POST: semantic field entries and findings reflect selected-source outcomes without overwriting locked manual values.
# @SIDE_EFFECT: updates semantic decisions and conflict findings.
#### ƒ **record_clarification_answer**
# @PURPOSE: Persist one clarification answer and re-evaluate profile, findings, and readiness.
# @PRE: target question belongs to the session’s active clarification session.
# @POST: answer is saved before current-question pointer advances.
# @SIDE_EFFECT: updates clarification and finding state.
#### ƒ **prepare_launch_preview**
# @PURPOSE: Assemble effective execution inputs and trigger Superset-side preview compilation.
# @PRE: all required variables have candidate values or explicitly accepted defaults.
# @POST: returns preview artifact in pending, ready, failed, or stale state.
# @SIDE_EFFECT: persists preview attempt and upstream compilation diagnostics.
#### ƒ **launch_dataset**
# @PURPOSE: Start the approved dataset execution through SQL Lab and persist run context for audit/replay.
# @PRE: session is run-ready and compiled preview is current.
# @POST: returns persisted run context with SQL Lab session reference and launch outcome.
# @SIDE_EFFECT: creates SQL Lab execution session and audit snapshot.
# [/DEF:DatasetReviewOrchestrator:Module]
---
# [DEF:DatasetReviewSessionRepository:Module]
# @COMPLEXITY: 5
# @PURPOSE: Persist and retrieve dataset review session aggregates, including readiness, findings, semantic decisions, clarification state, previews, and run contexts.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[DatasetReviewSession]
# @RELATION: [DEPENDS_ON] ->[DatasetProfile]
# @RELATION: [DEPENDS_ON] ->[ValidationFinding]
# @RELATION: [DEPENDS_ON] ->[CompiledPreview]
# @PRE: repository operations execute within authenticated request or task scope.
# @POST: session aggregate reads are structurally consistent and writes preserve ownership and version semantics.
# @SIDE_EFFECT: reads/writes application persistence layer.
# @DATA_CONTRACT: Input[SessionMutation] -> Output[PersistedSessionAggregate]
# @INVARIANT: answers, mapping approvals, preview artifacts, and launch snapshots are never attributed to the wrong user or session.
# @TEST_CONTRACT: save_then_resume -> persisted session can be reopened without losing semantic/manual/clarification state
# @TEST_SCENARIO: resume_session_preserves_manual_overrides -> locked semantic fields remain active after reload
# @TEST_EDGE: foreign_user_access -> rejected
# @TEST_EDGE: missing_session -> not found
# @TEST_EDGE: partial_preview_snapshot -> preserved but not marked launchable
# @TEST_INVARIANT: ownership_scope -> VERIFIED_BY: [foreign_user_access]
#### ƒ **create_session**
# @PURPOSE: Persist initial session shell.
#### ƒ **load_session_detail**
# @PURPOSE: Return the full session aggregate for API/frontend use.
#### ƒ **save_profile_and_findings**
# @PURPOSE: Persist profile and validation state together.
#### ƒ **save_preview**
# @PURPOSE: Persist compiled preview attempt and mark older fingerprints stale.
#### ƒ **save_run_context**
# @PURPOSE: Persist immutable launch audit snapshot.
# [/DEF:DatasetReviewSessionRepository:Module]
---
# [DEF:SemanticSourceResolver:Module]
# @COMPLEXITY: 4
# @PURPOSE: Resolve, rank, and apply semantic metadata candidates from files, connected dictionaries, reference datasets, and AI generation fallback.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[LLMProviderService]
# @RELATION: [DEPENDS_ON] ->[SemanticSource]
# @RELATION: [DEPENDS_ON] ->[SemanticFieldEntry]
# @RELATION: [DEPENDS_ON] ->[SemanticCandidate]
# @PRE: selected source and target field set must be known.
# @POST: candidate ranking follows the configured confidence hierarchy and unresolved fuzzy matches remain reviewable.
# @SIDE_EFFECT: may create conflict findings and semantic candidate records.
# @INVARIANT: Manual overrides are never silently replaced by imported, inferred, or AI-generated values.
# @TEST_CONTRACT: rank_candidates -> exact dictionary beats reference import beats fuzzy beats AI draft
# @TEST_SCENARIO: manual_lock_survives_reimport -> locked field remains active after another source is applied
# @TEST_EDGE: malformed_source_payload -> failed source application with explanatory finding
# @TEST_EDGE: conflicting_sources -> conflict state preserved for review
# @TEST_EDGE: no_trusted_matches -> AI draft fallback only
# @TEST_INVARIANT: confidence_hierarchy -> VERIFIED_BY: [rank_candidates]
#### ƒ **resolve_from_file**
# @PURPOSE: Normalize uploaded semantic file records into field-level candidates.
#### ƒ **resolve_from_dictionary**
# @PURPOSE: Resolve candidates from connected tabular dictionary sources.
#### ƒ **resolve_from_reference_dataset**
# @PURPOSE: Reuse semantic metadata from trusted Superset datasets.
#### ƒ **rank_candidates**
# @PURPOSE: Apply confidence ordering and determine best candidate per field.
#### ƒ **detect_conflicts**
# @PURPOSE: Mark competing candidate sets that require explicit user review.
#### ƒ **apply_field_decision**
# @PURPOSE: Accept, reject, or manually override a field-level semantic value.
# [/DEF:SemanticSourceResolver:Module]
---
# [DEF:ClarificationEngine:Module]
# @COMPLEXITY: 4
# @PURPOSE: Manage one-question-at-a-time clarification sessions, including prioritization, answer persistence, and readiness impact updates.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[ClarificationSession]
# @RELATION: [DEPENDS_ON] ->[ClarificationQuestion]
# @RELATION: [DEPENDS_ON] ->[ClarificationAnswer]
# @RELATION: [DEPENDS_ON] ->[ValidationFinding]
# @PRE: target session contains unresolved or contradictory review state.
# @POST: every recorded answer updates the clarification session and associated session state deterministically.
# @SIDE_EFFECT: creates clarification questions, persists answers, updates findings/profile state.
# @INVARIANT: Clarification answers are persisted before the current question pointer or readiness state is advanced.
# @TEST_CONTRACT: next_question_selection -> returns only one highest-priority unresolved question at a time
# @TEST_SCENARIO: save_and_resume_clarification -> reopening session restores current question and prior answers
# @TEST_EDGE: skipped_question -> unresolved topic remains visible
# @TEST_EDGE: expert_review_marked -> topic deferred without false resolution
# @TEST_EDGE: duplicate_answer_submission -> idempotent or rejected deterministically
# @TEST_INVARIANT: single_active_question -> VERIFIED_BY: [next_question_selection]
#### ƒ **start_or_resume**
# @PURPOSE: Open clarification mode on the highest-priority unresolved question.
#### ƒ **build_question_payload**
# @PURPOSE: Return question, why-it-matters text, current guess, and suggested options.
#### ƒ **record_answer**
# @PURPOSE: Persist one answer and compute state impact.
#### ƒ **summarize_progress**
# @PURPOSE: Produce the clarification change summary shown on exit or pause.
# [/DEF:ClarificationEngine:Module]
---
# [DEF:SupersetContextExtractor:Module]
# @COMPLEXITY: 4
# @PURPOSE: Recover dataset, dashboard, filter, and runtime-template context from Superset links and related API payloads.
# @LAYER: Infra
# @RELATION: [CALLS] ->[SupersetClient]
# @RELATION: [DEPENDS_ON] ->[ImportedFilter]
# @RELATION: [DEPENDS_ON] ->[TemplateVariable]
# @PRE: Superset link or dataset reference must be parseable enough to resolve an environment-scoped target resource.
# @POST: returns the best available recovered context with explicit provenance and partial-recovery markers when necessary.
# @SIDE_EFFECT: performs upstream Superset API reads.
# @INVARIANT: Partial recovery is surfaced explicitly and never misrepresented as fully confirmed context.
# @TEST_CONTRACT: recover_context_from_link -> output distinguishes URL-derived, native-filter-derived, and unresolved context
# @TEST_SCENARIO: partial_filter_recovery_marks_recovery_required -> session remains usable but not falsely complete
# @TEST_EDGE: unsupported_link_shape -> intake failure with actionable finding
# @TEST_EDGE: dataset_without_filters -> successful dataset recovery with empty imported filter set
# @TEST_EDGE: missing_dashboard_binding -> partial recovery only
# @TEST_INVARIANT: provenance_visibility -> VERIFIED_BY: [recover_context_from_link]
#### ƒ **parse_superset_link**
# @PURPOSE: Extract candidate identifiers and query state from supported Superset URLs.
#### ƒ **recover_imported_filters**
# @PURPOSE: Build imported filter entries from URL state and Superset-side saved context.
#### ƒ **discover_template_variables**
# @PURPOSE: Detect runtime variables and Jinja references from dataset query-bearing fields.
#### ƒ **build_recovery_summary**
# @PURPOSE: Summarize recovered, partial, and unresolved context for session state and UX.
# [/DEF:SupersetContextExtractor:Module]
---
# [DEF:SupersetCompilationAdapter:Module]
# @COMPLEXITY: 4
# @PURPOSE: Interact with Superset preview compilation and SQL Lab execution endpoints using the current approved execution context.
# @LAYER: Infra
# @RELATION: [CALLS] ->[SupersetClient]
# @RELATION: [DEPENDS_ON] ->[CompiledPreview]
# @RELATION: [DEPENDS_ON] ->[DatasetRunContext]
# @PRE: effective template params and dataset execution reference are available.
# @POST: preview and launch calls return Superset-originated artifacts or explicit errors.
# @SIDE_EFFECT: performs upstream Superset preview and SQL Lab calls.
# @INVARIANT: The adapter never fabricates compiled SQL locally; preview truth is delegated to Superset only.
# @TEST_CONTRACT: compile_then_launch -> launch uses the same effective input fingerprint verified in preview
# @TEST_SCENARIO: preview_failure_blocks_launch -> no SQL Lab session is created after failed preview
# @TEST_EDGE: compilation_endpoint_error -> failed preview artifact with readable diagnostics
# @TEST_EDGE: sql_lab_creation_error -> failed launch audit state
# @TEST_EDGE: fingerprint_mismatch -> launch rejected
# @TEST_INVARIANT: superset_truth_source -> VERIFIED_BY: [preview_failure_blocks_launch]
#### ƒ **compile_preview**
# @PURPOSE: Request Superset-side compiled SQL preview for the current effective inputs.
#### ƒ **mark_preview_stale**
# @PURPOSE: Invalidate previous preview after mapping or value changes.
#### ƒ **create_sql_lab_session**
# @PURPOSE: Create the canonical audited execution session after all launch gates pass.
# [/DEF:SupersetCompilationAdapter:Module]
---
## 2. Frontend Components
<!-- [DEF:DatasetReviewWorkspace:Component] -->
<!-- @COMPLEXITY: 5 -->
<!-- @PURPOSE: Main dataset review workspace coordinating session state, progressive recovery, semantic review, clarification, preview, and launch UX. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @RELATION: [BINDS_TO] ->[assistantChat] -->
<!-- @RELATION: [BINDS_TO] ->[taskDrawer] -->
<!-- @UX_STATE: Empty -> Show source intake with Superset link and dataset-selection entry actions. -->
<!-- @UX_STATE: Importing -> Show progressive recovery milestones as context is assembled. -->
<!-- @UX_STATE: Review -> Show summary, findings, semantic layer, filters, mapping, and next action. -->
<!-- @UX_STATE: Clarification -> Focus the user on one current clarification question while preserving wider session context. -->
<!-- @UX_STATE: Ready -> Show launch summary and unambiguous run-ready signal without hiding warnings. -->
<!-- @UX_FEEDBACK: Main CTA changes by readiness state and reflects current highest-value next action. -->
<!-- @UX_RECOVERY: Users can save, resume, or reopen an unfinished session without losing context. -->
<!-- @UX_REACTIVITY: Uses Svelte runes for session, readiness, preview, and task state derivation. -->
<!-- @INVARIANT: Navigation away from dirty session state must require explicit confirmation. -->
<!-- @TEST_CONTRACT: workspace_state_machine -> one and only one major readiness-driven CTA is primary at a time -->
<!-- @TEST_SCENARIO: resume_preserves_state -> reopening unfinished session restores current panel state and next action -->
<!-- @TEST_EDGE: session_load_failure -> error/recovery UI shown -->
<!-- @TEST_EDGE: stale_preview_after_mapping_change -> preview state flips to stale -->
<!-- @TEST_EDGE: unsaved_navigation -> guarded exit -->
<!-- @TEST_INVARIANT: primary_cta_alignment -> VERIFIED_BY: [workspace_state_machine] -->
#### ƒ **handleSourceSubmit**
<!-- @PURPOSE: Start a session from a Superset link or dataset selection. -->
<!-- @UX_FEEDBACK: Immediate optimistic intake acknowledgement plus recovery progress. -->
#### ƒ **handleResumeSession**
<!-- @PURPOSE: Reopen an existing paused or unfinished session. -->
<!-- @UX_FEEDBACK: Restores the session into the correct readiness-driven panel state. -->
#### ƒ **handleLaunch**
<!-- @PURPOSE: Execute the final launch action once run-ready gates pass. -->
<!-- @UX_STATE: Launching -> Disable CTA and expose progress/result handoff. -->
<!-- @UX_FEEDBACK: Success state links to SQL Lab and audit summary; failure preserves context and recovery path. -->
<!-- [/DEF:DatasetReviewWorkspace:Component] -->
---
<!-- [DEF:SourceIntakePanel:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Collect initial dataset source input through Superset link paste or dataset selection entry paths. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @UX_STATE: Idle -> Empty intake form with two clear entry paths. -->
<!-- @UX_STATE: Validating -> Lightweight inline validation feedback. -->
<!-- @UX_STATE: Rejected -> Input error shown with corrective hint. -->
<!-- @UX_FEEDBACK: Recognized links are acknowledged before deeper recovery finishes. -->
<!-- @UX_RECOVERY: Users can correct invalid input in place without resetting the page. -->
#### ƒ **submitSupersetLink**
<!-- @PURPOSE: Validate and submit Superset link input. -->
#### ƒ **submitDatasetSelection**
<!-- @PURPOSE: Submit selected dataset/environment context. -->
<!-- [/DEF:SourceIntakePanel:Component] -->
---
<!-- [DEF:ValidationFindingsPanel:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Present validation findings grouped by severity with explicit resolution and actionability signals. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[DatasetReviewWorkspace] -->
<!-- @UX_STATE: Blocking -> Blocking findings are visually dominant and block launch flow. -->
<!-- @UX_STATE: Warning -> Warnings remain visible with explicit approval or defer actions. -->
<!-- @UX_STATE: Informational -> Low-priority findings are collapsed or secondary. -->
<!-- @UX_FEEDBACK: Resolving or approving an item updates readiness state immediately. -->
<!-- @UX_RECOVERY: Users can jump from a finding directly to the relevant remediation area. -->
#### ƒ **groupFindingsBySeverity**
<!-- @PURPOSE: Project findings into blocking, warning, and informational groups. -->
#### ƒ **jumpToFindingTarget**
<!-- @PURPOSE: Focus the relevant review section for a selected finding. -->
<!-- [/DEF:ValidationFindingsPanel:Component] -->
---
<!-- [DEF:SemanticLayerReview:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Review and edit semantic metadata for columns and metrics with provenance and conflict visibility. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @UX_STATE: Normal -> Show current semantic values and provenance badges. -->
<!-- @UX_STATE: Conflicted -> Show side-by-side competing semantic candidates for the same field. -->
<!-- @UX_STATE: Manual -> Show locked manual override and block silent overwrite. -->
<!-- @UX_FEEDBACK: Applying a semantic source or manual override updates provenance and readiness immediately. -->
<!-- @UX_RECOVERY: Users can keep current values, accept recommendations, or review candidates one by one. -->
#### ƒ **applyManualOverride**
<!-- @PURPOSE: Lock a field to a user-provided semantic value. -->
<!-- @UX_FEEDBACK: Field marked as manual override and source-import replacement is disabled. -->
#### ƒ **applyCandidateSelection**
<!-- @PURPOSE: Accept one candidate from conflicting or fuzzy semantic options. -->
<!-- @UX_FEEDBACK: Candidate badge state changes and conflict warning clears when appropriate. -->
<!-- [/DEF:SemanticLayerReview:Component] -->
---
<!-- [DEF:ClarificationDialog:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: One-question-at-a-time clarification surface for unresolved or contradictory dataset meanings. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @RELATION: [BINDS_TO] ->[assistantChat] -->
<!-- @UX_STATE: Question -> Show current question, why-it-matters text, current guess, and selectable answers. -->
<!-- @UX_STATE: Saving -> Disable controls while persisting answer. -->
<!-- @UX_STATE: Completed -> Show clarification summary and impact on readiness. -->
<!-- @UX_FEEDBACK: Each answer updates profile and findings without forcing a full page reload. -->
<!-- @UX_RECOVERY: Users can skip, defer to expert review, pause, and resume later. -->
#### ƒ **submitAnswer**
<!-- @PURPOSE: Save selected or custom clarification answer. -->
#### ƒ **skipQuestion**
<!-- @PURPOSE: Defer the current question while keeping it unresolved. -->
#### ƒ **pauseClarification**
<!-- @PURPOSE: Exit clarification mode without losing prior answers. -->
<!-- [/DEF:ClarificationDialog:Component] -->
---
<!-- [DEF:ExecutionMappingReview:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Review mappings between imported filters and detected template variables, including transformed values and warning approvals. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @UX_STATE: Incomplete -> Required mapping values still missing. -->
<!-- @UX_STATE: WarningApproval -> Mapping rows require explicit approval before launch. -->
<!-- @UX_STATE: Approved -> All launch-sensitive mappings approved or overridden. -->
<!-- @UX_FEEDBACK: Mapping approvals or manual edits immediately re-evaluate preview staleness and readiness. -->
<!-- @UX_RECOVERY: Users can manually override transformed values instead of approving them as-is. -->
#### ƒ **approveMapping**
<!-- @PURPOSE: Explicitly approve a warning-level value transformation. -->
<!-- @UX_FEEDBACK: Warning cleared and launch checklist refreshed. -->
#### ƒ **overrideMappingValue**
<!-- @PURPOSE: Replace the proposed effective mapping value manually. -->
<!-- @UX_FEEDBACK: Mapping method switches to manual override and prior warning may be cleared or recalculated. -->
<!-- [/DEF:ExecutionMappingReview:Component] -->
---
<!-- [DEF:CompiledSQLPreview:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Present the exact Superset-generated compiled SQL preview with refresh state and failure diagnostics. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[api_module] -->
<!-- @UX_STATE: Missing -> Prompt user to generate preview. -->
<!-- @UX_STATE: Pending -> Show generation-in-progress feedback. -->
<!-- @UX_STATE: Ready -> Render read-only SQL preview with visible substitutions. -->
<!-- @UX_STATE: Stale -> Mark preview invalid after input changes until regenerated. -->
<!-- @UX_STATE: Error -> Show readable Superset compilation failure details and recovery action. -->
<!-- @UX_FEEDBACK: Refreshing preview updates timestamp and launch readiness cues. -->
<!-- @UX_RECOVERY: Users can navigate directly from preview error to the mapping or value row that caused failure. -->
#### ƒ **requestPreview**
<!-- @PURPOSE: Trigger preview generation for the current effective session inputs. -->
#### ƒ **showPreviewErrorTarget**
<!-- @PURPOSE: Focus remediation target when compilation diagnostics identify a mapping or variable issue. -->
<!-- [/DEF:CompiledSQLPreview:Component] -->
---
<!-- [DEF:LaunchConfirmationPanel:Component] -->
<!-- @COMPLEXITY: 3 -->
<!-- @PURPOSE: Summarize final run context, approvals, warnings, and compiled-preview status before dataset launch. -->
<!-- @LAYER: UI -->
<!-- @RELATION: [BINDS_TO] ->[DatasetReviewWorkspace] -->
<!-- @UX_STATE: Blocked -> Explicitly list missing gates preventing launch. -->
<!-- @UX_STATE: Ready -> Show final reviewed context and confirm action. -->
<!-- @UX_STATE: Submitted -> Show handoff to SQL Lab and audit snapshot reference. -->
<!-- @UX_FEEDBACK: Confirmation view is a contextual run summary, not a generic yes/no modal. -->
<!-- @UX_RECOVERY: When blocked, users can jump back to missing values, mapping approvals, or preview generation. -->
#### ƒ **buildLaunchSummary**
<!-- @PURPOSE: Project the exact run context into the final pre-launch summary. -->
#### ƒ **confirmLaunch**
<!-- @PURPOSE: Submit the run-ready launch request once all gates pass. -->
<!-- [/DEF:LaunchConfirmationPanel:Component] -->
---
## 3. Contract Coverage Notes
The feature requires:
- dedicated semantic resolution contracts instead of hiding source-ranking logic inside orchestration,
- a first-class clarification engine because guided ambiguity resolution is a persisted workflow, not a simple endpoint,
- a Superset extraction boundary distinct from preview/launch behavior,
- UI contracts that cover the UX state machine rather than only the happy path.
These contracts are intended to align directly with:
- [`specs/027-dataset-llm-orchestration/spec.md`](../spec.md)
- [`specs/027-dataset-llm-orchestration/ux_reference.md`](../ux_reference.md)
- [`specs/027-dataset-llm-orchestration/research.md`](../research.md)
- [`specs/027-dataset-llm-orchestration/data-model.md`](../data-model.md)

764
specs/027-dataset-llm-orchestration/data-model.md Normal file
View File

@@ -0,0 +1,764 @@
# Data Model: LLM Dataset Orchestration
**Feature**: [LLM Dataset Orchestration](./spec.md)
**Branch**: `027-dataset-llm-orchestration`
**Date**: 2026-03-16
## Overview
This document defines the domain entities, relationships, lifecycle states, and validation rules for the dataset review, semantic enrichment, clarification, preview, and launch workflow described in [`spec.md`](./spec.md) and grounded by the decisions in [`research.md`](./research.md).
The model is intentionally split into:
- **session aggregate** entities for resumable workflow state,
- **semantic/provenance** entities for enrichment and conflict handling,
- **execution** entities for mapping, preview, and launch audit,
- **export** projections for sharing outputs.
---
## 1. Core Aggregate: DatasetReviewSession
### Entity: `SessionCollaborator`
| Field | Type | Required | Description |
|---|---|---:|---|
| `user_id` | string | yes | Collaborating user ID |
| `role` | enum | yes | `viewer`, `reviewer`, `approver` |
| `added_at` | datetime | yes | When they were added |
### Entity: `DatasetReviewSession`
Represents the top-level resumable workflow container for one dataset review/execution effort.
| Field | Type | Required | Description |
|---|---|---:|---|
| `session_id` | string (UUID) | yes | Stable unique identifier for the review session |
| `user_id` | string | yes | Authenticated User ID of the session owner |
| `collaborators` | list[SessionCollaborator] | no | Shared access and roles |
| `environment_id` | string | yes | Superset environment context |
| `source_kind` | enum | yes | Origin kind: `superset_link`, `dataset_selection` |
| `source_input` | string | yes | Original link or selected dataset reference |
| `dataset_ref` | string | yes | Canonical dataset reference used by the feature |
| `dataset_id` | integer \| null | no | Superset dataset id when resolved |
| `dashboard_id` | integer \| null | no | Superset dashboard id if imported from dashboard link |
| `readiness_state` | enum | yes | Current workflow readiness state |
| `recommended_action` | enum | yes | Explicit next recommended action |
| `status` | enum | yes | Session lifecycle status |
| `current_phase` | enum | yes | Active workflow phase |
| `active_task_id` | string \| null | no | Linked long-running task if one is active |
| `last_preview_id` | string \| null | no | Most recent preview snapshot |
| `last_run_context_id` | string \| null | no | Most recent launch audit record |
| `created_at` | datetime | yes | Session creation timestamp |
| `updated_at` | datetime | yes | Last mutation timestamp |
| `last_activity_at` | datetime | yes | Last user/system activity timestamp |
| `closed_at` | datetime \| null | no | Terminal close/archive timestamp |
### Validation rules
- `session_id` must be globally unique.
- `source_input` must be non-empty.
- `environment_id` must resolve to a configured environment.
- `readiness_state` and `recommended_action` must always be present.
- `user_id` ownership must be enforced for all mutations, unless collaborator roles allow otherwise.
- `dataset_id` becomes required before preview or launch phases.
- `last_preview_id` must refer to a preview generated from the same session.
### Enums
#### `SessionStatus`
- `active`
- `paused`
- `completed`
- `archived`
- `cancelled`
#### `SessionPhase`
- `intake`
- `recovery`
- `review`
- `semantic_review`
- `clarification`
- `mapping_review`
- `preview`
- `launch`
- `post_run`
#### `ReadinessState`
- `empty`
- `importing`
- `review_ready`
- `semantic_source_review_needed`
- `clarification_needed`
- `clarification_active`
- `mapping_review_needed`
- `compiled_preview_ready`
- `partially_ready`
- `run_ready`
- `run_in_progress`
- `completed`
- `recovery_required`
#### `RecommendedAction`
- `import_from_superset`
- `review_documentation`
- `apply_semantic_source`
- `start_clarification`
- `answer_next_question`
- `approve_mapping`
- `generate_sql_preview`
- `complete_required_values`
- `launch_dataset`
- `resume_session`
- `export_outputs`
---
## 2. Dataset Profile and Review State
### Entity: `DatasetProfile`
Consolidated interpretation of dataset meaning, semantics, filters, assumptions, and readiness.
| Field | Type | Required | Description |
|---|---|---:|---|
| `profile_id` | string (UUID) | yes | Unique profile id |
| `session_id` | string | yes | Parent session |
| `dataset_name` | string | yes | Display dataset name |
| `schema_name` | string \| null | no | Schema if available |
| `database_name` | string \| null | no | Database if available |
| `business_summary` | text | yes | Human-readable summary |
| `business_summary_source` | enum | yes | Provenance of summary |
| `description` | text \| null | no | Dataset-level description |
| `dataset_type` | enum \| null | no | `table`, `virtual`, `sqllab_view`, `unknown` |
| `is_sqllab_view` | boolean | yes | Whether dataset is SQL Lab derived |
| `completeness_score` | number \| null | no | Optional normalized completeness score |
| `confidence_state` | enum | yes | Overall confidence posture |
| `has_blocking_findings` | boolean | yes | Derived summary flag |
| `has_warning_findings` | boolean | yes | Derived summary flag |
| `manual_summary_locked` | boolean | yes | Protects user-entered summary |
| `created_at` | datetime | yes | Created timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
### Validation rules
- `business_summary` must always contain a usable string; if weak, it may be skeletal but not null.
- `manual_summary_locked=true` prevents later automatic overwrite.
- `session_id` must be unique if only one active profile snapshot is stored per session, or versioned if snapshots are retained.
- `confidence_state` must reflect highest unresolved-risk posture, not just optimistic confidence.
#### `BusinessSummarySource`
- `confirmed`
- `imported`
- `inferred`
- `ai_draft`
- `manual_override`
#### `ConfidenceState`
- `confirmed`
- `mostly_confirmed`
- `mixed`
- `low_confidence`
- `unresolved`
---
## 3. Validation Findings
### Entity: `ValidationFinding`
Represents a blocking issue, warning, or informational observation.
| Field | Type | Required | Description |
|---|---|---:|---|
| `finding_id` | string (UUID) | yes | Unique finding id |
| `session_id` | string | yes | Parent session |
| `area` | enum | yes | Affected domain area |
| `severity` | enum | yes | `blocking`, `warning`, `informational` |
| `code` | string | yes | Stable machine-readable finding code |
| `title` | string | yes | Short label |
| `message` | text | yes | Actionable human-readable explanation |
| `resolution_state` | enum | yes | Current resolution status |
| `resolution_note` | text \| null | no | Optional explanation or approval note |
| `caused_by_ref` | string \| null | no | Related field/filter/mapping/question id |
| `created_at` | datetime | yes | Creation timestamp |
| `resolved_at` | datetime \| null | no | Resolution timestamp |
### Validation rules
- `severity` must be one of the allowed values.
- `resolution_state=resolved` or `approved` requires either a system resolution event or user action.
- `launch` is blocked if any open `blocking` finding remains.
- `warning` findings tied to mapping transformations require explicit approval before launch if marked launch-sensitive.
#### `FindingArea`
- `source_intake`
- `dataset_profile`
- `semantic_enrichment`
- `clarification`
- `filter_recovery`
- `template_mapping`
- `compiled_preview`
- `launch`
- `audit`
#### `ResolutionState`
- `open`
- `resolved`
- `approved`
- `skipped`
- `deferred`
- `expert_review`
---
## 4. Semantic Source and Field Decisions
### Entity: `SemanticSource`
Represents a trusted or candidate source of semantic metadata.
| Field | Type | Required | Description |
|---|---|---:|---|
| `source_id` | string (UUID) | yes | Unique source id |
| `session_id` | string | yes | Parent session |
| `source_type` | enum | yes | Origin kind |
| `source_ref` | string | yes | External reference, dataset ref, or uploaded artifact ref |
| `source_version` | string | yes | Version/Snapshot for propagation tracking |
| `display_name` | string | yes | Human-readable source name |
| `trust_level` | enum | yes | Source trust tier |
| `schema_overlap_score` | number \| null | no | Optional overlap signal |
| `status` | enum | yes | Availability/applicability status |
| `created_at` | datetime | yes | Creation timestamp |
#### `SemanticSourceType`
- `uploaded_file`
- `connected_dictionary`
- `reference_dataset`
- `neighbor_dataset`
- `ai_generated`
#### `TrustLevel`
- `trusted`
- `recommended`
- `candidate`
- `generated`
#### `SemanticSourceStatus`
- `available`
- `selected`
- `applied`
- `rejected`
- `partial`
- `failed`
---
### Entity: `SemanticFieldEntry`
Canonical semantic state for one dataset field or metric.
| Field | Type | Required | Description |
|---|---|---:|---|
| `field_id` | string (UUID) | yes | Unique field semantic id |
| `session_id` | string | yes | Parent session |
| `field_name` | string | yes | Physical field/metric name |
| `field_kind` | enum | yes | `column`, `metric`, `filter_dimension`, `parameter` |
| `verbose_name` | string \| null | no | Display label |
| `description` | text \| null | no | Human-readable description |
| `display_format` | string \| null | no | Formatting metadata such as d3 format |
| `provenance` | enum | yes | Final chosen source class |
| `source_id` | string \| null | no | Winning source |
| `confidence_rank` | integer \| null | no | Final applied ranking |
| `is_locked` | boolean | yes | Manual override protection |
| `has_conflict` | boolean | yes | Whether competing candidates exist |
| `needs_review` | boolean | yes | Whether user review is still needed |
| `last_changed_by` | enum | yes | `system`, `user`, `agent` |
| `user_feedback` | enum | no | User feedback: `up`, `down`, `null` |
| `created_at` | datetime | yes | Creation timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
### Validation rules
- `field_name` must be unique per `session_id + field_kind`.
- `is_locked=true` prevents automatic overwrite.
- `provenance=manual_override` implies `is_locked=true`.
- `has_conflict=true` requires at least one competing candidate record.
- Fuzzy/applied inferred values must keep `needs_review=true` until confirmed if policy requires explicit review.
#### `FieldKind`
- `column`
- `metric`
- `filter_dimension`
- `parameter`
#### `FieldProvenance`
- `dictionary_exact`
- `reference_imported`
- `fuzzy_inferred`
- `ai_generated`
- `manual_override`
- `unresolved`
---
### Entity: `SemanticCandidate`
Stores competing candidate values before or alongside final field decision.
| Field | Type | Required | Description |
|---|---|---:|---|
| `candidate_id` | string (UUID) | yes | Unique candidate id |
| `field_id` | string | yes | Parent semantic field |
| `source_id` | string \| null | no | Candidate source |
| `candidate_rank` | integer | yes | Lower is stronger |
| `match_type` | enum | yes | Exact, imported, fuzzy, generated |
| `confidence_score` | number | yes | Normalized score |
| `proposed_verbose_name` | string \| null | no | Candidate verbose name |
| `proposed_description` | text \| null | no | Candidate description |
| `proposed_display_format` | string \| null | no | Candidate display format |
| `status` | enum | yes | Candidate lifecycle |
| `created_at` | datetime | yes | Creation timestamp |
#### `CandidateMatchType`
- `exact`
- `reference`
- `fuzzy`
- `generated`
#### `CandidateStatus`
- `proposed`
- `accepted`
- `rejected`
- `superseded`
---
## 5. Imported Filters and Runtime Variables
### Entity: `ImportedFilter`
Represents one recovered or user-supplied filter value.
| Field | Type | Required | Description |
|---|---|---:|---|
| `filter_id` | string (UUID) | yes | Unique filter id |
| `session_id` | string | yes | Parent session |
| `filter_name` | string | yes | Source filter name |
| `display_name` | string \| null | no | User-facing label |
| `raw_value` | json | yes | Original recovered value |
| `normalized_value` | json \| null | no | Optional transformed value |
| `source` | enum | yes | Origin of the filter |
| `confidence_state` | enum | yes | Confidence/provenance class |
| `requires_confirmation` | boolean | yes | Whether explicit review is needed |
| `recovery_status` | enum | yes | Recovery completeness |
| `notes` | text \| null | no | Recovery explanation |
| `created_at` | datetime | yes | Creation timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
#### `FilterSource`
- `superset_native`
- `superset_url`
- `manual`
- `inferred`
#### `FilterConfidenceState`
- `confirmed`
- `imported`
- `inferred`
- `ai_draft`
- `unresolved`
#### `FilterRecoveryStatus`
- `recovered`
- `partial`
- `missing`
- `conflicted`
---
### Entity: `TemplateVariable`
Represents a runtime variable discovered from dataset execution logic.
| Field | Type | Required | Description |
|---|---|---:|---|
| `variable_id` | string (UUID) | yes | Unique variable id |
| `session_id` | string | yes | Parent session |
| `variable_name` | string | yes | Canonical runtime variable name |
| `expression_source` | text | yes | Raw expression or snippet where variable was found |
| `variable_kind` | enum | yes | Detected variable class |
| `is_required` | boolean | yes | Whether launch requires a mapped value |
| `default_value` | json \| null | no | Optional default |
| `mapping_status` | enum | yes | Current mapping state |
| `created_at` | datetime | yes | Creation timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
#### `VariableKind`
- `native_filter`
- `parameter`
- `derived`
- `unknown`
#### `MappingStatus`
- `unmapped`
- `proposed`
- `approved`
- `overridden`
- `invalid`
---
## 6. Mapping Review and Warning Approvals
### Entity: `ExecutionMapping`
Represents the mapping between a recovered filter and a runtime variable.
| Field | Type | Required | Description |
|---|---|---:|---|
| `mapping_id` | string (UUID) | yes | Unique mapping id |
| `session_id` | string | yes | Parent session |
| `filter_id` | string | yes | Source imported filter |
| `variable_id` | string | yes | Target template variable |
| `mapping_method` | enum | yes | How mapping was produced |
| `raw_input_value` | json | yes | Original input |
| `effective_value` | json \| null | no | Value to send to preview/launch |
| `transformation_note` | text \| null | no | Explanation of normalization |
| `warning_level` | enum \| null | no | Warning classification if transformation is risky |
| `requires_explicit_approval` | boolean | yes | Whether launch gate applies |
| `approval_state` | enum | yes | Approval lifecycle |
| `approved_by_user_id` | string \| null | no | Approver if approved |
| `approved_at` | datetime \| null | no | Approval timestamp |
| `created_at` | datetime | yes | Creation timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
### Validation rules
- `filter_id + variable_id` must be unique per session unless versioning is used.
- `requires_explicit_approval=true` implies launch is blocked while `approval_state != approved`.
- `effective_value` is required before preview when variable is required.
- user override should set `mapping_method=manual_override`.
#### `MappingMethod`
- `direct_match`
- `heuristic_match`
- `semantic_match`
- `manual_override`
#### `MappingWarningLevel`
- `low`
- `medium`
- `high`
#### `ApprovalState`
- `pending`
- `approved`
- `rejected`
- `not_required`
---
## 7. Clarification Workflow
### Entity: `ClarificationSession`
Stores resumable clarification flow state for one review session.
| Field | Type | Required | Description |
|---|---|---:|---|
| `clarification_session_id` | string (UUID) | yes | Unique clarification session id |
| `session_id` | string | yes | Parent review session |
| `status` | enum | yes | Clarification lifecycle |
| `current_question_id` | string \| null | no | Current active question |
| `resolved_count` | integer | yes | Count of answered/resolved items |
| `remaining_count` | integer | yes | Count of unresolved items |
| `summary_delta` | text \| null | no | Human-readable change summary |
| `started_at` | datetime | yes | Start time |
| `updated_at` | datetime | yes | Last update |
| `completed_at` | datetime \| null | no | End time |
#### `ClarificationStatus`
- `pending`
- `active`
- `paused`
- `completed`
- `cancelled`
---
### Entity: `ClarificationQuestion`
Represents one focused question in the clarification flow.
| Field | Type | Required | Description |
|---|---|---:|---|
| `question_id` | string (UUID) | yes | Unique question id |
| `clarification_session_id` | string | yes | Parent clarification session |
| `topic_ref` | string | yes | Related field/finding/mapping id |
| `question_text` | text | yes | Focused question |
| `why_it_matters` | text | yes | Business significance explanation |
| `current_guess` | text \| null | no | Best guess if available |
| `priority` | integer | yes | Order score |
| `state` | enum | yes | Question lifecycle |
| `created_at` | datetime | yes | Creation timestamp |
| `updated_at` | datetime | yes | Updated timestamp |
#### `QuestionState`
- `open`
- `answered`
- `skipped`
- `expert_review`
- `superseded`
---
### Entity: `ClarificationOption`
Suggested selectable answer option for a question.
| Field | Type | Required | Description |
|---|---|---:|---|
| `option_id` | string (UUID) | yes | Unique option id |
| `question_id` | string | yes | Parent question |
| `label` | string | yes | UI label |
| `value` | string | yes | Stored answer payload |
| `is_recommended` | boolean | yes | Whether this is the recommended option |
| `display_order` | integer | yes | UI ordering |
---
### Entity: `ClarificationAnswer`
Stores user response to one clarification question.
| Field | Type | Required | Description |
|---|---|---:|---|
| `answer_id` | string (UUID) | yes | Unique answer id |
| `question_id` | string | yes | Parent question |
| `answer_kind` | enum | yes | How user responded |
| `answer_value` | text \| null | no | Selected/custom answer |
| `answered_by_user_id` | string | yes | Responding user |
| `impact_summary` | text \| null | no | Optional summary of resulting state changes |
| `created_at` | datetime | yes | Answer timestamp |
#### `AnswerKind`
- `selected`
- `custom`
- `skipped`
- `expert_review`
### Validation rules
- Each active question may have at most one current answer.
- `custom` answers require non-empty `answer_value`.
- `selected` answers must correspond to a valid option or normalized payload.
- `expert_review` leaves the related topic unresolved but marked intentionally deferred.
---
## 8. Preview and Launch Audit
### Entity: `CompiledPreview`
Stores the exact Superset-returned compiled SQL preview.
| Field | Type | Required | Description |
|---|---|---:|---|
| `preview_id` | string (UUID) | yes | Unique preview id |
| `session_id` | string | yes | Parent session |
| `preview_status` | enum | yes | Preview lifecycle state |
| `compiled_sql` | text \| null | no | Exact compiled SQL if successful |
| `preview_fingerprint` | string | yes | Snapshot hash of mapping/inputs used |
| `compiled_by` | enum | yes | Must be `superset` |
| `error_code` | string \| null | no | Optional failure code |
| `error_details` | text \| null | no | Readable preview error |
| `compiled_at` | datetime \| null | no | Successful compile timestamp |
| `created_at` | datetime | yes | Record creation timestamp |
### Validation rules
- `compiled_by` must be `superset`.
- `compiled_sql` is required when `preview_status=ready`.
- `compiled_sql` must be null when `preview_status=failed` unless partial diagnostics are intentionally stored elsewhere.
- `preview_fingerprint` must be compared against current session inputs before launch.
- Launch requires `preview_status=ready` and matching current fingerprint.
#### `PreviewStatus`
- `pending`
- `ready`
- `failed`
- `stale`
---
### Entity: `DatasetRunContext`
Audited execution snapshot created at launch.
| Field | Type | Required | Description |
|---|---|---:|---|
| `run_context_id` | string (UUID) | yes | Unique run context id |
| `session_id` | string | yes | Parent review session |
| `dataset_ref` | string | yes | Canonical dataset identity |
| `environment_id` | string | yes | Execution environment |
| `preview_id` | string | yes | Bound compiled preview |
| `sql_lab_session_ref` | string | yes | Canonical SQL Lab reference |
| `effective_filters` | json | yes | Final filter payload |
| `template_params` | json | yes | Final template parameter object |
| `approved_mapping_ids` | json array | yes | Explicit approvals used for launch |
| `semantic_decision_refs` | json array | yes | Applied semantic decision references |
| `open_warning_refs` | json array | yes | Warnings that remained visible at launch |
| `launch_status` | enum | yes | Launch outcome |
| `launch_error` | text \| null | no | Error if launch failed |
| `created_at` | datetime | yes | Launch record timestamp |
### Validation rules
- `preview_id` must reference a `CompiledPreview` with `ready` status.
- `sql_lab_session_ref` is mandatory for successful launch.
- `effective_filters` and `template_params` must match the preview fingerprint used.
- `launch_status=started` or `success` requires a non-empty SQL Lab reference.
#### `LaunchStatus`
- `started`
- `success`
- `failed`
---
## 9. Export Projections
### Entity: `ExportArtifact`
Tracks generated exports for sharing documentation and validation outputs.
| Field | Type | Required | Description |
|---|---|---:|---|
| `artifact_id` | string (UUID) | yes | Unique artifact id |
| `session_id` | string | yes | Parent session |
| `artifact_type` | enum | yes | Export type |
| `format` | enum | yes | File/output format |
| `storage_ref` | string | yes | Storage/file reference |
| `created_by_user_id` | string | yes | Requesting user |
| `created_at` | datetime | yes | Artifact creation time |
#### `ArtifactType`
- `documentation`
- `validation_report`
- `run_summary`
#### `ArtifactFormat`
- `json`
- `markdown`
- `csv`
- `pdf`
---
## 10. Relationships
## One-to-one / aggregate-root relationships
- `DatasetReviewSession` → `DatasetProfile` (current active profile view)
- `DatasetReviewSession` → `ClarificationSession` (current or latest)
- `DatasetReviewSession` → `CompiledPreview` (latest/current preview)
- `DatasetReviewSession` → `DatasetRunContext` (latest/current launch audit)
## One-to-many relationships
- `DatasetReviewSession` → many `ValidationFinding`
- `DatasetReviewSession` → many `SemanticSource`
- `DatasetReviewSession` → many `SemanticFieldEntry`
- `SemanticFieldEntry` → many `SemanticCandidate`
- `DatasetReviewSession` → many `ImportedFilter`
- `DatasetReviewSession` → many `TemplateVariable`
- `DatasetReviewSession` → many `ExecutionMapping`
- `ClarificationSession` → many `ClarificationQuestion`
- `ClarificationQuestion` → many `ClarificationOption`
- `ClarificationQuestion` → zero/one current `ClarificationAnswer`
- `DatasetReviewSession` → many `ExportArtifact`
- `DatasetReviewSession` → many `SessionEvent`
- `DatasetReviewSession` → many `SessionEvent`
---
## 11. Derived Rules and Invariants
### Run readiness invariant
A session is `run_ready` only if:
- no open blocking findings remain,
- all required template variables have approved/effective mappings,
- all launch-sensitive mapping warnings have been explicitly approved,
- a non-stale `CompiledPreview` exists for the current fingerprint.
### Manual intent invariant
If a field is manually overridden:
- `SemanticFieldEntry.is_locked = true`
- `SemanticFieldEntry.provenance = manual_override`
- later imports or inferred candidates may be recorded, but cannot replace the active value automatically.
### Progressive recovery invariant
Partial Superset recovery must preserve usable state:
- imported filters may be `partial`,
- unresolved variables may remain `unmapped`,
- findings must explain what is still missing,
- session remains resumable.
### Clarification persistence invariant
Clarification answers must be persisted before:
- finding severity is downgraded,
- profile state is updated,
- current question pointer advances.
### Preview truth invariant
Compiled preview must be:
- generated by Superset,
- tied to the exact current effective inputs,
- treated as invalid if mappings/values change afterward.
---
## 12. Migration & Evolution Strategy
- **Baseline**: The initial implementation (Milestone 1) will include the core session and profile entities.
- **Incremental Growth**: Subsequent milestones will add clarification, mapping, and launch audit entities via standard SQLAlchemy migrations.
- **Compatibility**: The `DatasetReviewSession` aggregate root will remain the stable entry point for all sub-entities to ensure forward compatibility with saved user state.
## 13. Suggested Backend DTO Grouping
The future API and persistence layers should group models roughly as follows:
### Session DTOs
- `SessionSummary`
- `SessionDetail`
- `SessionListItem`
### Review DTOs
- `DatasetProfileDto`
- `ValidationFindingDto`
- `ReadinessChecklistDto`
### Semantic DTOs
- `SemanticSourceDto`
- `SemanticFieldEntryDto`
- `SemanticCandidateDto`
### Clarification DTOs
- `ClarificationSessionDto`
- `ClarificationQuestionDto`
- `ClarificationAnswerRequest`
### Execution DTOs
- `ImportedFilterDto`
- `TemplateVariableDto`
- `ExecutionMappingDto`
- `CompiledPreviewDto`
- `LaunchSummaryDto`
### Export DTOs
- `ExportArtifactDto`
---
## 13. Open Modeling Notes Resolved
The Phase 0 research questions are considered resolved for design purposes:
- SQL preview is modeled as a first-class persisted artifact.
- SQL Lab is modeled as the only canonical launch target.
- semantic resolution and clarification are modeled as separate domain boundaries.
- field-level overrides and mapping approvals are first-class entities.
- session persistence is separate from task execution state.
This model is ready to drive:
- [`contracts/modules.md`](./contracts/modules.md)
- [`contracts/api.yaml`](./contracts/api.yaml)
- [`quickstart.md`](./quickstart.md)

320
specs/027-dataset-llm-orchestration/plan.md Normal file
View File

@@ -0,0 +1,320 @@
# Implementation Plan: LLM Dataset Orchestration
**Branch**: `027-dataset-llm-orchestration` | **Date**: 2026-03-16 | **Spec**: `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/spec.md`
**Input**: Feature specification from `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/spec.md`
**Note**: This template is filled in by the `/speckit.plan` command. See `/home/busya/dev/ss-tools/.specify/templates/plan-template.md` for the execution workflow.
## Summary
Deliver a dataset-centered orchestration flow that lets users start from a Superset link or dataset selection, recover analytical context, enrich semantics from trusted sources before AI generation, resolve ambiguity through guided clarification, generate a Superset-side compiled SQL preview, and launch an audited SQL Lab execution only when readiness gates pass.
The implementation will extend the existing FastAPI + SvelteKit architecture rather than creating a parallel subsystem. Backend work will add a persisted review-session domain, orchestration services for semantic recovery and clarification, Superset adapters for context extraction and SQL Lab execution, and explicit APIs for mapping approvals and field-level semantic overrides. Frontend work will add a dedicated dataset review workspace with progressive recovery, semantic-source review, one-question-at-a-time clarification, mapping approval controls, compiled SQL preview, and resumable session state.
## Technical Context
**Language/Version**: Python 3.9+ backend, Node.js 18+ frontend, Svelte 5 / SvelteKit frontend runtime
**Primary Dependencies**: FastAPI, SQLAlchemy, Pydantic, existing `TaskManager`, existing `SupersetClient`, existing LLM provider stack, SvelteKit, Tailwind CSS, frontend `requestApi`/`fetchApi` wrappers
**Storage**: Existing application databases for persistent session/domain entities; existing tasks database for async execution metadata; filesystem for optional uploaded semantic sources/artifacts
**Testing**: pytest for backend unit/integration/API tests; Vitest for frontend component/store/API-wrapper tests
**Target Platform**: Linux-hosted FastAPI + Svelte web application integrated with Superset
**Project Type**: Web application with backend API and frontend SPA
**Performance Goals**:
- Initial summary generation: < 30s (Progressive recovery visible within < 5s)
- Preview compilation: < 10s
- Session load / Resume: < 2s
- SC-002 target: first readable summary under 5 minutes for complex datasets.
**Constraints**: Launch must remain blocked without successful Superset-side compiled preview; long-running recovery/enrichment/preview work must be asynchronous and observable; frontend must use existing API wrappers instead of native fetch; manual semantic overrides must never be silently overwritten; auditability and provenance are prioritized over raw throughput
**Scale/Scope**: One end-to-end feature spanning dataset intake, session persistence, semantic enrichment, clarification, mapping approval, preview, and launch; multiple new backend services/APIs plus a new multi-state frontend workspace
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
### Pre-Research Gate Assessment
1. **Semantic protocol compliance — PASS WITH REQUIRED PHASE 1 EXPANSION**
- New backend orchestration and persistence modules must follow `/home/busya/dev/ss-tools/.ai/standards/semantics.md`.
- Existing draft contracts are incomplete for the feature scope; Phase 1 must add explicit contracts for semantic-source resolution, clarification lifecycle, Superset context extraction, session persistence, and missing UI states.
- Complexity 4/5 Python modules must explicitly define `logger.reason()` / `logger.reflect()` paths; Complexity 5 boundaries must use `belief_scope`.
2. **Complexity-driven contract coverage — PASS WITH GAPS TO CLOSE**
- The core orchestration boundary is Complexity 5 because it gates launch, audit, state transitions, and cross-service consistency.
- Semantic source resolution, clarification workflow, mapping approval state, and session persistence each require explicit contracts instead of being hidden inside one orchestrator.
- UI contracts must map to the UX state machine, especially `Empty`, `Importing`, `Review Ready`, `Semantic Source Review Needed`, `Clarification Active`, `Mapping Review Needed`, `Compiled Preview Ready`, `Run Ready`, `Run In Progress`, `Completed`, and `Recovery Required`.
3. **UX-state compatibility — PASS**
- The architecture can support the UX reference because:
- recovery can be progressive and asynchronous,
- clarification can be session-backed and resumed,
- preview generation can be represented as a stateful asynchronous action,
- launch remains a gated terminal action.
- If Phase 0 research later shows Superset cannot provide reliable compilation preview or SQL Lab execution hooks compatible with the required interaction model, planning must stop and the UX contract must be renegotiated.
4. **Async boundaries — PASS**
- Long-running work already fits the repository constitution through `TaskManager`.
- Session start, deep context recovery, semantic enrichment from external sources, preview generation, and launch-hand-off side effects should be dispatched as tasks or internally asynchronous service steps with observable state changes.
5. **Frontend API-wrapper rules — PASS**
- Existing frontend uses `/home/busya/dev/ss-tools/frontend/src/lib/api.js` wrappers.
- New frontend work must use `requestApi`, `fetchApi`, `postApi`, or wrapper modules only; native `fetch` remains forbidden.
6. **RBAC/security constraints — PASS WITH DESIGN REQUIREMENT**
- New endpoints must use existing auth and permission dependencies.
- New orchestration actions need explicit permission modeling for reading sessions, editing semantic mappings, answering clarification prompts, generating previews, and launching runs.
- Session data must remain self-scoped/auditable and must not permit cross-user mutation without explicit policy.
- **Action**: Add `DATASET_REVIEW_*` permissions to `backend/src/scripts/seed_permissions.py`.
7. **Security & Threat Model — PASS**
- Session isolation: Every session record is strictly bound to `user_id`. Query filters must include owner check.
- Audit trail: `DatasetRunContext` is immutable after launch.
- Credential handling: Reuse existing `SupersetClient` encrypted configuration.
- **Action**: API endpoints must use `Depends(get_current_user)` and explicit permission checks.
7. **Belief-state/logging constraints — PASS WITH REQUIRED APPLICATION**
- Complexity 4/5 Python orchestration modules will require `belief_scope` plus meaningful `logger.reason()` and `logger.reflect()` traces around state transitions, preview validation, warning approvals, and launch gating.
### Post-Design Gate Assessment
1. **Semantic protocol compliance — PASS**
- All modules in `contracts/modules.md` follow the complexity-driven metadata requirements.
- Relation syntax matches the canonical `@RELATION: [PREDICATE] ->[TARGET_ID]` format.
- Python modules (Complexity 4/5) explicitly specify `logger.reason()` and `belief_scope` requirements in their contracts.
2. **API Schema Completeness — PASS**
- `contracts/api.yaml` provides a fully typed OpenAPI 3.0.3 specification.
- Every session lifecycle, semantic review, and execution gate is covered by a typed endpoint.
3. **UX-Technical Alignment — PASS**
- Design supports the WYSIWWR principle via `SupersetCompilationAdapter`.
- Fallback strategies for missing preview or SQL Lab hooks are defined in `research.md`.
### Final Gate Result
**PASS** - The implementation plan and design artifacts are constitution-compliant and ready for task breakdown.
## Project Structure
### Documentation (this feature)
```text
/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/
├── plan.md
├── research.md
├── data-model.md
├── quickstart.md
├── contracts/
│ ├── api.yaml
│ └── modules.md
└── tasks.md
```
### Source Code (repository root)
```text
/home/busya/dev/ss-tools/backend/
├── src/
│ ├── api/
│ │ └── routes/
│ ├── core/
│ ├── models/
│ ├── schemas/
│ └── services/
/home/busya/dev/ss-tools/frontend/
├── src/
│ ├── lib/
│ │ ├── api/
│ │ ├── components/
│ │ ├── i18n/
│ │ └── stores/
│ └── routes/
/home/busya/dev/ss-tools/backend/src/api/routes/__tests__/
/home/busya/dev/ss-tools/backend/src/services/__tests__/
/home/busya/dev/ss-tools/frontend/src/lib/**/__tests__/
/home/busya/dev/ss-tools/frontend/src/routes/**/__tests__/
```
**Structure Decision**: Use the repository’s existing web-application split. Backend implementation belongs under `/home/busya/dev/ss-tools/backend/src/{models,schemas,services,api/routes}`. Frontend implementation belongs under `/home/busya/dev/ss-tools/frontend/src/{routes,lib/components,lib/api,lib/stores}`. Tests will stay adjacent to their current backend/frontend conventions.
## Semantic Contract Guidance
> Use this section to drive Phase 1 artifacts, especially `contracts/modules.md`.
### Planned Critical/High-Value Modules
- `DatasetReviewOrchestrator` — `@COMPLEXITY: 5`
- `SemanticSourceResolver` — `@COMPLEXITY: 4`
- `ClarificationEngine` — `@COMPLEXITY: 4`
- `SupersetContextExtractor` — `@COMPLEXITY: 4`
- `SupersetCompilationAdapter` — `@COMPLEXITY: 4`
- `DatasetReviewSessionRepository` or equivalent persistence boundary — `@COMPLEXITY: 5`
- `DatasetReviewWorkspace` — `@COMPLEXITY: 5`
- `SourceIntakePanel` — `@COMPLEXITY: 3`
- `ValidationFindingsPanel` — `@COMPLEXITY: 3`
- `SemanticLayerReview` — `@COMPLEXITY: 3`
- `ClarificationDialog` — `@COMPLEXITY: 3`
- `ExecutionMappingReview` — `@COMPLEXITY: 3`
- `CompiledSQLPreview` — `@COMPLEXITY: 3`
- `LaunchConfirmationPanel` — `@COMPLEXITY: 3`
### Required Semantic Rules
- Use `@COMPLEXITY` or `@C:` as the primary rule source.
- Match contract density to complexity:
- Complexity 1: anchors only, `@PURPOSE` optional
- Complexity 2: `@PURPOSE`
- Complexity 3: `@PURPOSE`, `@RELATION`; UI also `@UX_STATE`
- Complexity 4: `@PURPOSE`, `@RELATION`, `@PRE`, `@POST`, `@SIDE_EFFECT`; Python also meaningful `logger.reason()` / `logger.reflect()` path
- Complexity 5: level 4 + `@DATA_CONTRACT`, `@INVARIANT`; Python also `belief_scope`; UI also `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY`
- Write relations only in canonical form: `@RELATION: [PREDICATE] ->[TARGET_ID]`
- If any relation target, DTO, or contract dependency is unknown, emit `[NEED_CONTEXT: target]` instead of inventing placeholders.
- Preserve medium-appropriate anchor/comment syntax for Python, Svelte markup, and Svelte script contexts.
## Phase 0: Research Agenda
### Open Questions Requiring Resolution
1. How to reliably extract saved native filters from supported Superset links and versions.
2. How to discover dataset runtime template variables and Jinja placeholders using available Superset APIs and dataset payloads.
3. How to perform a safe Superset-side compiled SQL preview compatible with the current deployment/version.
4. How to create or bind a SQL Lab execution session as the canonical audited launch target.
5. How to model semantic source ranking, fuzzy match review, conflict detection, and provenance without collapsing into an orchestration god-object.
6. How to persist resumable clarification and review sessions using the current database stack.
7. How to design typed API contracts that support field-level semantic operations, mapping approval flow, and session lifecycle operations.
8. How to degrade gracefully when Superset import/preview or LLM enrichment only partially succeeds.
### Required Research Outputs
Research must produce explicit decisions for:
- Superset link parsing and recovery strategy
- Superset compilation/SQL Lab integration approach
- Semantic source resolution architecture
- Clarification session persistence model
- Session persistence/audit model
- API schema granularity and endpoint set
- Test strategy for Superset-dependent and LLM-dependent flows
- Delivery milestones for incremental rollout
## Phase 1: Design Focus
Phase 1 must generate:
- typed domain entities and DTOs in `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/data-model.md`
- expanded semantic contracts in `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/contracts/modules.md`
- typed OpenAPI schemas and missing endpoints in `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/contracts/api.yaml`
- execution and validation guide in `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/quickstart.md`
Phase 1 must specifically close the current gaps around:
- field-level semantic operations,
- clarification engine responsibilities,
- mapping approval endpoints,
- session lifecycle APIs,
- exportable outputs,
- error-path validation scenarios,
- alignment between UX states and UI contracts.
## Delivery Milestones
| Milestone | FR Coverage | Scope | User Value |
|-----------|-------------|-------|------------|
| M1: Sessioned Auto Review | FR-001 to FR-011, FR-035, FR-037 | Source intake, dataset review session, initial profile, findings, provenance, semantic-source application, export of review outputs | Users get immediate documentation, validation, and trusted-source enrichment without manual reconstruction |
| M2: Guided Clarification | FR-012 to FR-020, FR-036, FR-038, FR-039, FR-040 | Clarification engine, resumable questions, question templates/eval, field-level semantic overrides, conflict review, progress persistence | Users can resolve ambiguity safely and preserve manual intent |
| M3: Controlled Execution | FR-021 to FR-034 | Filter extraction, template-variable mapping, warning approvals, compiled preview, SQL Lab launch, manual export path, audited run context | Users can move from recovered context to reproducible execution with clear readiness gates |
## RBAC Model
| Permission | Description | Target Role(s) |
|------------|-------------|----------------|
| `dataset:session:read` | View own review sessions | Analytics Engineer, BI Engineer, Data Steward |
| `dataset:session:manage` | Edit mappings, answer questions, override semantics | Analytics Engineer, BI Engineer |
| `dataset:session:approve` | Approve warning-level mappings | Senior Analytics Engineer, Data Steward |
| `dataset:execution:preview` | Trigger Superset SQL compilation preview | Analytics Engineer, BI Engineer |
| `dataset:execution:launch` | Create SQL Lab session in target environment | Analytics Engineer, BI Engineer |
| `dataset:execution:launch_prod` | Launch in Production-staged environment | Senior Analytics Engineer |
## Integration Points
### Service Reuse (Critical)
- **Superset Interaction**: Use existing `backend/src/core/superset_client.py` (do not duplicate HTTP clients).
- **LLM Interaction**: Use existing `backend/src/services/llm_provider.py` via `LLMProviderService`.
- **Notifications**: Integrate with `NotificationService` for launch outcomes and preview readiness.
- **i18n**: Use existing `frontend/src/lib/i18n/` for all user-facing strings in the review workspace.
## Rollout & Monitoring
### Feature Flags
- `ff_dataset_auto_review`: Enables basic documentation and intake.
- `ff_dataset_clarification`: Enables guided dialogue mode.
- `ff_dataset_execution`: Enables preview and launch capabilities.
### Metrics & Alerting
- **Metrics**: Session completion rate, time-to-first-summary, preview failure rate (Superset compilation errors vs connection errors), clarification engagement.
- **Alerting**: High rate of `503` Superset API failures; persistent LLM provider timeouts (> 30s); unauthorized cross-session access attempts.
## Implementation Sequencing
### Backend First
1. Add persistent review-session domain model and schemas.
2. Add orchestration services and Superset adapters.
3. Add typed API endpoints and explicit RBAC.
4. Add task/event integration and audit persistence.
5. Add backend tests for session lifecycle, preview gating, launch gating, and degradation paths.
### Frontend Next
1. Add dataset review route/workspace shell and session loading.
2. Add source-intake, summary, findings, and semantic review panels.
3. Add clarification dialog and mapping approval UI.
4. Add compiled preview and launch confirmation UI.
5. Add frontend tests for state transitions, wrappers, and critical UX invariants.
### Integration/Hardening
1. Validate Superset version compatibility against real/staged environment.
2. Verify progressive session recovery and resume flows.
3. Verify audit replay/run-context capture.
4. Measure success-criteria instrumentation feasibility.
## Testing Strategy
### Backend
- **Unit tests** for semantic ranking, provenance/conflict rules, clarification prioritization, preview gating, and launch guards.
- **Integration tests** for session persistence, Superset adapter behavior, SQL preview orchestration, and SQL Lab launch orchestration with mocked upstream responses.
- **API contract tests** for typed response schemas, RBAC enforcement, mapping approval operations, field-level semantic edits, export operations, and session lifecycle.
### Frontend
- **Unit/component tests** for state-driven UI contracts, provenance rendering, one-question clarification, mapping approval flow, stale preview handling, and launch gating visuals.
- **Integration-style route tests** for resume flows, progressive loading, and error recovery states.
### External Dependency Strategy
- Mock Superset APIs for CI determinism.
- Use stable fixtures/snapshots for LLM-produced structured outputs.
- Treat provider/transport failure as explicit degraded states rather than semantic failure.
- Include replayable fixtures for imported filters, template variables, conflict cases, and compilation errors.
## Risks & Mitigations
| Risk | Why It Matters | Mitigation |
|------|----------------|------------|
| Superset version lacks a stable compiled-preview endpoint | FR-029 and WYSIWWR depend on native Superset-side compilation | Resolve in Phase 0; if unsupported, stop and renegotiate UX/feature scope before implementation |
| Superset link/native filter formats differ across installations | Could make import brittle or partial | Design recovery as best-effort with explicit provenance and recovery-required state |
| SQL Lab launch handoff is inconsistent across environments | FR-032 requires canonical audited launch target | Research version-compatible creation strategy and define fallback as blocked, not silent substitution |
| Semantic resolution logic becomes an orchestration god-object | Hurts maintainability and contract traceability | Separate `SemanticSourceResolver`, `ClarificationEngine`, and Superset extraction responsibilities |
| Fuzzy matching creates too many false positives | Undermines trust and increases approval burden | Keep explicit confidence hierarchy, review-required fuzzy matches, and field-level selective application |
| LLM/provider outages interrupt review quality | Could block non-critical enrichment | Degrade to partial review state with preserved trusted-source results and explicit next action |
| Session lifecycle becomes hard to resume safely | FR-019 and FR-036 require resumability | Persist answers, approvals, and current recommended action as first-class session state |
## Post-Design Re-Check Criteria
After Phase 1 artifacts are produced, re-check:
- semantic protocol coverage against all planned modules/components,
- UX-state coverage against `/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/ux_reference.md`,
- explicit API support for field-level semantic actions, mapping approval, exports, and session lifecycle,
- belief-state/logging expectations for Complexity 4/5 Python modules,
- typed schemas sufficient for backend/frontend parallel implementation,
- quickstart coverage of happy path plus critical negative/recovery paths.
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
No justified constitution violations at planning time.

298
specs/027-dataset-llm-orchestration/quickstart.md Normal file
View File

@@ -0,0 +1,298 @@
# Quickstart: LLM Dataset Orchestration
**Feature**: [LLM Dataset Orchestration](./spec.md)
**Branch**: `027-dataset-llm-orchestration`
This guide validates the end-to-end workflow for dataset review, semantic enrichment, clarification, preview generation, and controlled SQL Lab launch.
---
## 1. Prerequisites
1. Access to a configured Superset environment with:
- at least one dataset,
- at least one dashboard URL containing reusable analytical context,
- permissions sufficient for dataset inspection and SQL Lab session creation.
2. An active LLM provider configured in ss-tools.
3. Optional semantic sources for enrichment testing:
- uploaded spreadsheet dictionary,
- connected tabular dictionary,
- trusted reference Superset dataset.
4. A test user account with permission to create and resume sessions.
5. A second test user account for ownership/visibility guard validation.
---
## 2. Primary End-to-End Happy Path
### Step 1: Start Review Session
- Navigate to the dataset-review workflow entry from the datasets area.
- Start a session using one of:
- a Superset dashboard link with saved filters,
- a direct dataset selection.
- **Verify**:
- a new session is created,
- the session gets a visible readiness state,
- the first recommended action is explicit.
### Step 2: Observe Progressive Recovery
- Keep the session open while recovery runs.
- **Verify** progressive updates appear for:
- dataset recognition,
- imported filter recovery,
- template/Jinja variable discovery,
- preliminary semantic-source candidates,
- first-pass business summary.
- **Verify** partial work is shown before the whole pipeline finishes.
### Step 3: Review Automatic Analysis
- Inspect the generated business summary and validation findings.
- **Verify**:
- the summary is readable by an operational stakeholder,
- findings are grouped by severity,
- provenance/confidence markers distinguish confirmed/imported/inferred/AI-draft values,
- the next recommended action changes appropriately.
### Step 4: Apply Semantic Source
- Use **Apply semantic source** and choose:
- spreadsheet dictionary,
- connected dictionary,
- trusted reference dataset.
- **Verify**:
- exact matches are applied as stronger candidates,
- fuzzy matches remain reviewable rather than silently applied,
- semantic conflicts are shown side by side,
- field-level manual overrides remain possible.
### Step 5: Confirm Field-Level Semantics
- Manually override one field’s `verbose_name` or description.
- Apply another semantic source afterward.
- **Verify**:
- the manual field remains locked,
- imported/generated values do not silently overwrite it,
- provenance changes to manual override.
### Step 6: Guided Clarification
- Enter clarification mode from a session with unresolved findings.
- Answer one question using a suggested option.
- Answer another with a custom value.
- Skip one question.
- Mark one for expert review.
- **Verify**:
- only one active question is shown at a time,
- each question includes “why this matters” and current guess,
- answers update readiness/findings/profile state,
- skipped and expert-review items remain visible as unresolved.
### Step 7: Pause and Resume
- Save or pause the session mid-clarification.
- Leave the page and reopen the session.
- **Verify**:
- the session resumes with prior answers intact,
- the current question or next unresolved question is restored,
- manual semantic decisions and pending mappings are preserved.
### Step 8: Review Mapping and Generate Preview
- Open the mapping review section.
- Approve one warning-level mapping transformation.
- Manually override another transformed mapping value.
- Trigger **Generate SQL Preview**.
- **Verify**:
- all required variables are visible,
- warning approvals are explicit,
- the preview is read-only,
- preview status shows it was compiled by Superset,
- substituted values are visible in the final SQL.
### Step 9: Launch Dataset
- Move the session to `Run Ready`.
- Click **Launch Dataset**.
- **Verify**:
- launch confirmation shows dataset identity, effective filters, parameter values, warnings, and preview status,
- a SQL Lab session reference is returned,
- an audited run context is stored,
- the session moves to run-in-progress or completed state appropriately.
### Step 10: Export Outputs
- Export documentation.
- Export validation findings.
- **Verify**:
- both artifacts are generated,
- artifact metadata or file reference is associated with the session,
- exported output reflects the current reviewed state.
### Step 11: Collaboration and Review
- As User A, add User B as a `reviewer`.
- Access the same session as User B.
- **Verify**:
- User B can view the session state.
- User B can answer clarification questions but cannot approve launch-critical mappings.
- Audit log (if implemented) records which user performed which action.
---
## 3. Negative and Recovery Scenarios
### Scenario A: Invalid Superset Link
- Start a session with a malformed or unsupported link.
- **Verify**:
- intake fails with actionable error messaging,
- no fake recovered context is shown,
- the user can correct input in place.
### Scenario B: Partial Filter Recovery
- Use a link where only some filters can be recovered.
- **Verify**:
- recovered filters are shown,
- unrecovered pieces are explicitly marked,
- session enters `recovery_required` or equivalent partial state,
- workflow remains usable.
### Scenario C: Dataset Without Clear Business Meaning
- Use a dataset with weak metadata and no strong trusted semantic matches.
- **Verify**:
- the summary remains minimal but usable,
- the system does not pretend certainty,
- clarification becomes the recommended next step.
### Scenario D: Conflicting Semantic Sources
- Apply two semantic sources that disagree for the same field.
- **Verify**:
- both candidates are shown side by side,
- recommended source is visible if confidence differs,
- no silent overwrite occurs,
- conflict remains until explicitly resolved.
### Scenario E: Missing Required Runtime Value
- Leave a required template variable unmapped.
- Attempt preview or launch.
- **Verify**:
- preview or launch is blocked according to gate rules,
- missing values are highlighted specifically,
- recommended next action becomes completion/remediation rather than launch.
### Scenario F: Preview Compilation Failure
- Provide a mapping value known to break Superset-side compilation.
- Trigger preview.
- **Verify**:
- preview moves to `failed` state,
- readable Superset error details are shown,
- launch remains blocked,
- the user can navigate back to the problematic mapping/value.
### Scenario G: Preview Staleness After Input Change
- Successfully generate preview.
- Change an approved mapping or required value.
- **Verify**:
- preview state becomes `stale`,
- launch is blocked until preview is regenerated,
- stale state is visible and not hidden.
### Scenario H: SQL Lab Launch Failure
- Simulate or trigger SQL Lab session creation failure.
- **Verify**:
- launch result is marked failed,
- the audit record still preserves attempted run context,
- the session remains recoverable,
- no success redirect is shown.
### Scenario I: Cross-User Access Guard
- Try to open or mutate the first user’s session from a second user account (without collaborator access).
- **Verify**:
- access is denied,
- no session state leaks to the second user,
- ownership/permission is enforced on view and mutation paths.
---
## 4. UX Invariants to Validate
- [ ] The primary CTA always reflects the current highest-value next step.
- [ ] The launch button stays blocked if:
- [ ] blocking findings remain,
- [ ] required values are missing,
- [ ] warning-level mappings needing approval are unresolved,
- [ ] preview is missing, failed, or stale.
- [ ] Manual semantic overrides are never silently overwritten.
- [ ] Every important semantic value exposes visible provenance.
- [ ] Clarification shows one focused question at a time.
- [ ] Partial recovery preserves usable value and explains what is missing.
- [ ] Preview explicitly indicates it was compiled by Superset.
- [ ] Session resume restores prior state without forcing re-entry.
---
## 5. Suggested Verification by Milestone
### Milestone 1: Sessioned Auto Review
Validate:
- source intake,
- progressive recovery,
- automatic documentation summary,
- typed findings display,
- semantic source application,
- export endpoints.
### Milestone 2: Guided Clarification
Validate:
- clarification question flow,
- answer persistence,
- resume behavior,
- conflict review,
- field-level manual override/lock behavior.
### Milestone 3: Controlled Execution
Validate:
- mapping review,
- explicit warning approvals,
- Superset-side preview,
- preview staleness handling,
- SQL Lab launch,
- audited run context persistence.
---
## 6. Success-Criteria Measurement Hints
These are not implementation metrics by themselves; they are validation hints for pilot runs.
### For [SC-001](./spec.md)
Track how many submitted datasets produce an initial documentation draft without manual reconstruction.
### For [SC-002](./spec.md)
Measure time from session start to first readable summary visible to the user.
### For [SC-003](./spec.md)
Measure the percentage of semantic fields populated from trusted sources before AI-draft fallback.
### For [SC-005](./spec.md)
Measure the percentage of eligible Superset links that produce a non-empty imported filter set usable for review.
### For [SC-007](./spec.md)
Check that launched sessions always persist:
- dataset identity,
- effective filters,
- template params,
- approved mappings,
- preview reference,
- SQL Lab session reference,
- outcome.
### For [SC-008](./spec.md)
Run moderated first-attempt sessions and record whether users complete import → review → clarification (if needed) → preview → launch without facilitator intervention.
---
## 7. Completion Checklist
A Phase 1 design is operationally validated when all are true:
- [ ] Happy-path session can be started and completed.
- [ ] Partial recovery behaves as explicit partial recovery, not silent failure.
- [ ] Clarification is resumable.
- [ ] Semantic conflict review is explicit.
- [ ] Field-level override lock works.
- [ ] Preview is Superset-generated and becomes stale after input mutation.
- [ ] Launch targets SQL Lab only.
- [ ] Export outputs are available.
- [ ] Ownership and guard rails are enforced.

506
specs/027-dataset-llm-orchestration/research.md Normal file
View File

@@ -0,0 +1,506 @@
# Research: LLM Dataset Orchestration
**Feature**: [LLM Dataset Orchestration](./spec.md)
**Branch**: `027-dataset-llm-orchestration`
**Date**: 2026-03-16
## Scope
This document resolves the Phase 0 technical unknowns identified in [`specs/027-dataset-llm-orchestration/plan.md`](./plan.md) and establishes implementation decisions for:
- Superset link parsing and context recovery
- runtime variable and Jinja discovery
- compiled SQL preview strategy
- SQL Lab launch handoff
- semantic source resolution architecture
- clarification session architecture
- typed API scope
- persistence, testing, and rollout strategy
---
## 1. Superset Link Intake and Native Filter Recovery
### Decision
Use a dedicated backend adapter, `SupersetContextExtractor`, to parse supported Superset URLs and recover context in two layers:
1. **URL-level recovery** from dashboard, dataset, and query parameters in the provided link
2. **Superset API enrichment** using the resolved dashboard/dataset identifiers to recover saved filter context and related dataset metadata
The recovered filter set will be stored as an `ImportedFilterSet` with provenance per value:
- `superset_native`
- `superset_url`
- `manual`
- `inferred`
If only partial filter recovery is possible, the session enters `recovery_required` rather than failing closed.
### Rationale
The UX requires recovery to feel progressive rather than brittle. A pure URL parser would be too fragile and version-specific. A pure API lookup would miss context encoded in the link itself. A two-layer strategy preserves value even when one source is incomplete.
This also matches the spec’s requirement that partial recovery remain usable and explicitly explained.
### Alternatives considered
- **Parse only URL query parameters**
Rejected because it is too dependent on frontend link shape and misses server-side saved context.
- **Resolve everything only through dashboard API after identifying the dashboard**
Rejected because some context lives directly in the URL and should be captured before API reconciliation.
- **Block session start if any filter cannot be recovered**
Rejected because it violates the UX principle of progressive value and FR-025.
---
## 2. Runtime Template Variable and Jinja Discovery
### Decision
Detect runtime execution variables using a hybrid discovery pipeline:
1. Load dataset detail from Superset via existing [`backend/src/core/superset_client.py`](backend/src/core/superset_client.py)
2. Extract `sql`, `is_sqllab_view`, metric expressions, and other query-bearing fields where available
3. Apply backend parsing rules for:
- `{{ ... }}`
- `filter_values('...')`
- common template parameter references
- Jinja variables in SQL text and metric expressions
4. Normalize discovered variables into typed `TemplateVariable` entries with:
- variable name
- source expression
- inferred kind (`native_filter`, `parameter`, `derived`, `unknown`)
- required/optional status
- mapping status
The parser will not execute Jinja locally. It only discovers variables and structure.
### Rationale
The UX and specification both require execution transparency, but also require Superset to remain the source of truth for final SQL compilation. Discovery is safe locally; execution is not. Separating variable discovery from compilation preserves WYSIWWR and avoids local behavior drift.
### Alternatives considered
- **Use only string regex over SQL text**
Rejected because it is simple but insufficient for richer query-bearing payloads.
- **Execute local Jinja rendering to discover required params**
Rejected because it conflicts with the UX principle that only Superset defines final execution.
- **Rely on LLM to infer variables from SQL text**
Rejected because variable discovery must be deterministic and auditable.
---
## 3. Compiled SQL Preview Strategy
### Decision
Compiled preview will be generated only through a dedicated Superset-side compilation call path managed by `SupersetCompilationAdapter`. The adapter will:
1. receive the candidate run context,
2. submit it to a Superset-compatible preview/compilation flow,
3. store the exact returned compiled SQL plus metadata,
4. mark preview state as `ready`, `pending`, `failed`, or `stale`.
Launch remains blocked unless preview state is `ready` and bound to the current mapping/version snapshot.
#### Superset Version Compatibility Matrix
The system will explicitly check `SupersetClient.get_version()` during intake to determine available features:
- **v3.x+**: Supports all features (context recovery, native filter import, SQL compilation preview, SQL Lab launch).
- **v2.x**: Supports best-effort context recovery; SQL compilation preview may require fallback to Manual Launch.
- **Legacy/Unknown**: Automatic recovery disabled; session enters `recovery_required` mode immediately.
#### Fallback Strategy (Preview Availability < 100%)
If Superset-side compilation is unavailable (e.g., version mismatch or unsupported dataset type):
- The system will allow **Manual Launch Approval** by a user with `dataset:session:approve` permission.
- The UI will explicitly display: "Native Preview Unavailable - Manual Validation Required."
- The audit record will mark the run as "Launched without Compiled Preview."
### Rationale
This is a direct requirement from FR-029 and the UX reference’s WYSIWWR principle. The compiled preview must be the same source of truth as the real execution path. Even a “close enough” local approximation would undermine trust and violate the spec.
### Alternatives considered
- **Local best-effort Jinja compilation with warning banner**
Rejected because the spec explicitly forbids fallback to unverified local approximation.
- **No preview, only launch-time validation**
Rejected because it breaks both UX and explicit launch gate requirements.
- **Preview generated by the LLM**
Rejected because the LLM must not write or edit SQL directly.
---
## 4. SQL Lab Launch as Canonical Audited Target
### Decision
Treat Superset SQL Lab as the primary canonical launch target for this feature. `DatasetReviewOrchestrator.launch_dataset` will:
1. verify preview readiness,
2. verify required-value completeness,
3. verify warning approvals,
4. create or bind a SQL Lab execution session in Superset,
5. persist a `DatasetRunContext` snapshot with session reference, approved mappings, semantic decisions, and preview fingerprint.
#### Fallback path: Export Prepared Context
If SQL Lab session creation is unavailable:
- Provide an **Export Prepared Run Context (JSON/SQL)** action.
- This allows users to manually take the compiled SQL (if available) or the effective parameter payload to Superset.
- The system still records the `DatasetRunContext` as "Exported for Manual Launch" to preserve audit integrity.
### Rationale
The specification explicitly names SQL Lab as canonical. The system must preserve auditability and replay, not merely trigger execution. Persisting the SQL Lab reference and run context as a single audited boundary supports both operational trust and later reopening.
### Alternatives considered
- **Run query outside SQL Lab through another Superset endpoint**
Rejected because it violates the canonical target clarified in the spec.
- **Redirect user to a generic analytical view with params**
Rejected because it is weaker for audit and replay.
- **Allow fallback execution when SQL Lab creation fails**
Rejected because the feature’s safety model depends on the audited target.
---
## 5. Semantic Enrichment Architecture
### Decision
Split semantic enrichment into a dedicated `SemanticSourceResolver` domain module instead of embedding all logic inside `DatasetReviewOrchestrator`.
Planned responsibilities:
- `resolve_from_file`
- `resolve_from_dictionary`
- `resolve_from_reference_dataset`
- `rank_candidates`
- `detect_conflicts`
- `apply_field_decision`
Candidate ranking will follow the required confidence hierarchy:
1. exact dictionary/file match
2. trusted reference dataset match
3. fuzzy semantic match
4. AI-generated draft
Manual overrides are represented as a lockable terminal provenance state and cannot be overwritten implicitly.
### Rationale
The feature’s semantic logic is too substantial to remain an untyped orchestrator subroutine. The reviewer feedback correctly identified this as a likely god-object risk. A dedicated resolver improves testability, contract clarity, and future extensibility.
### Alternatives considered
- **Keep enrichment inside orchestrator only**
Rejected because it centralizes too many responsibilities in a single Complexity 5 module.
- **Push matching into frontend only**
Rejected because provenance, confidence, and audit decisions must be persisted server-side.
- **Use only AI generation when dictionaries are absent**
Rejected because “reuse before invention” is a core UX principle.
---
## 6. Clarification Session Architecture
### Decision
Create a dedicated `ClarificationEngine` service with persistent `ClarificationSession` state. The engine will manage:
- unresolved-item prioritization,
- one-question-at-a-time question generation,
- best-guess suggestion packaging,
- answer recording,
- progress summarization,
- resume-from-last-open-item behavior.
Answers are persisted before session state advancement.
#### LLM Quality & Feedback Mechanism
To maintain high trust in AI-generated summaries, match recommendations, and questions:
- **User Feedback**: UI will include inline feedback (👍/👎) for all AI-drafted content.
- **Audit Logging**: Feedback is persisted in `SemanticFieldDecision` and `ClarificationAnswer` records for prompt engineering improvement.
- **Confidence Triage**: Low-confidence LLM outputs (score < 0.7) will be automatically marked for "expert review" rather than presented as confirmed suggestions.
#### Question Generation Strategy
Question templates and logic will be developed in Milestone 2:
- **Evaluation Criteria**: Questions must be judged on: relevancy, business impact explanation, and actionability of suggestions.
- **Templates**: Standardized prompts for field ambiguity, filter conflict, and missing run-time values.
- **Human-in-the-loop**: High-risk semantic conflicts will prioritize "expert review" options.
### Rationale
Guided clarification is not a simple endpoint action; it is a stateful interaction model with resumability and audit expectations. Treating it as a first-class module aligns with FR-012 to FR-020 and supports deterministic behavior rather than ad hoc agent chat state.
### Alternatives considered
- **Use assistant conversation history as the only clarification state**
Rejected because feature state and conversational transcript are not the same thing.
- **Generate all questions at once**
Rejected because the UX explicitly requires one question at a time.
- **Do not persist skipped questions separately**
Rejected because resume and expert-review flows need explicit unresolved-state semantics.
---
## 7. API Contract Granularity
### Decision
Expand the API beyond the current session/preview/launch skeleton into a typed session contract set with explicit lifecycle and field-level operations.
Required endpoint groups:
1. **Session lifecycle**
- `POST /sessions`
- `GET /sessions`
- `GET /sessions/{id}`
- `PATCH /sessions/{id}`
- `DELETE /sessions/{id}`
2. **Semantic source and field-level semantic actions**
- `POST /sessions/{id}/semantic-source`
- `PATCH /sessions/{id}/fields/{field_id}/semantic`
- `POST /sessions/{id}/fields/{field_id}/lock`
- `POST /sessions/{id}/fields/{field_id}/unlock`
3. **Clarification**
- `GET /sessions/{id}/clarification`
- `POST /sessions/{id}/clarification/answers`
- `POST /sessions/{id}/clarification/resume`
4. **Mapping review**
- `GET /sessions/{id}/mappings`
- `PATCH /sessions/{id}/mappings/{mapping_id}`
- `POST /sessions/{id}/mappings/{mapping_id}/approve`
5. **Preview and launch**
- `POST /sessions/{id}/preview`
- `POST /sessions/{id}/launch`
6. **Exports**
- `GET /sessions/{id}/exports/documentation`
- `GET /sessions/{id}/exports/validation`
Core schemas must be fully typed, especially:
- `ValidationFinding`
- `ImportedFilter`
- `TemplateVariable`
- `MappingDecision`
- `SemanticFieldEntry`
- `ClarificationQuestion`
- `DatasetRunContextSummary`
- `CompiledPreview`
### Rationale
The existing API draft is structurally correct but too loose for backend/frontend parallel work. Typed schemas are a blocker for safe implementation, especially given mapping approvals, field-level overrides, and resumable state.
### Alternatives considered
- **Minimal object blobs in `SessionDetail`**
Rejected because they prevent stable frontend typing and contract validation.
- **Expose only session-level mutations**
Rejected because FR-040 and mapping approval UX require field-level actions.
- **Defer exports until later**
Rejected because FR-035 requires exportable outputs as part of the feature.
---
## 8. Persistence Model
### Decision
Persist dataset review workflow as first-class application entities rather than storing everything in ad hoc task payloads.
Persistence boundaries will include:
- `DatasetReviewSession`
- `DatasetProfileSnapshot`
- `ValidationFindingRecord`
- `SemanticFieldDecision`
- `ImportedFilterRecord`
- `TemplateVariableRecord`
- `MappingApprovalRecord`
- `ClarificationSessionRecord`
- `ClarificationAnswerRecord`
- `CompiledPreviewRecord`
- `DatasetRunContextRecord`
Long-running task execution remains in the existing task system, but tasks reference session identifiers rather than replacing session state.
### Rationale
The current task system is good for observability and async work, but not sufficient as the canonical store for resumable feature state. The feature requires reopening, editing, exporting, and replaying a session independently of task lifecycle.
### Alternatives considered
- **Store all workflow state inside task params/results**
Rejected because tasks are execution records, not editable session aggregates.
- **Use filesystem-only session snapshots**
Rejected because this weakens structured querying and RBAC alignment.
- **Persist only final run context**
Rejected because clarification and review resume need mid-flow state.
---
## 9. Async and Observability Strategy
### Decision
Use `TaskManager` for expensive or multi-step operations:
- initial deep context recovery,
- semantic enrichment imports over large sources,
- preview generation when upstream latency is non-trivial,
- launch handoff if it includes multi-step Superset orchestration.
Short synchronous operations may still exist for simple field edits and approvals, but session state transitions must remain observable and reflected in the session model.
### Rationale
This aligns with the repository constitution and existing operational model. It also supports progressive UX milestones and live readiness changes.
### Alternatives considered
- **Make all session operations synchronous**
Rejected because upstream Superset and enrichment work may exceed acceptable UI latency.
- **Use frontend polling only with no task integration**
Rejected because the repository already has stronger task observability patterns.
---
## 10. Frontend Integration Pattern
### Decision
Implement the feature on top of the existing [`frontend/src/lib/api.js`](frontend/src/lib/api.js) wrapper conventions and Svelte 5 rune-based patterns. Add dedicated wrapper functions under the frontend API layer for dataset orchestration rather than embedding endpoint strings inside components.
The main route will be a dedicated review workspace separate from the current [`frontend/src/routes/datasets/+page.svelte`](frontend/src/routes/datasets/+page.svelte), which remains the dataset hub/list page.
### Rationale
The current dataset page already serves as a listing and bulk-operation view. The new orchestration flow is a multi-state workspace and should not overload the existing hub component. Using wrapper functions preserves constitution compliance and testability.
### Alternatives considered
- **Extend the existing dataset hub page for the whole flow**
Rejected because it would overload a currently unrelated list page with a large state machine.
- **Call endpoints directly with native `fetch` in new components**
Rejected by constitution and existing frontend architecture.
- **Build the whole flow inside assistant chat only**
Rejected because the UX reference defines a structured workspace, not chat-only interaction.
---
## 11. Testing Strategy
### Decision
Use a three-layer testing strategy:
#### Backend
- **Unit tests**
- semantic ranking/conflict rules,
- clarification prioritization,
- mapping approval guards,
- preview stale/ready rules,
- launch blocking conditions.
- **Integration tests**
- session persistence,
- Superset context extraction with fixtures,
- compiled preview orchestration,
- SQL Lab launch handoff,
- export generation.
- **API contract tests**
- typed schema responses,
- RBAC guards,
- session lifecycle endpoints,
- field-level semantic operations,
- mapping approval endpoints.
#### Frontend
- **Component tests**
- source intake states,
- validation triage,
- semantic conflict view,
- clarification dialog,
- mapping review,
- compiled preview stale/error behavior,
- launch summary gating.
- **Route integration tests**
- progressive loading,
- save/resume flows,
- recovery-required state,
- warning approval requirements.
#### External dependency handling
- mock Superset endpoints with version-shaped fixtures,
- mock LLM structured outputs deterministically,
- verify degraded paths explicitly instead of pretending success.
### Rationale
The feature blends deterministic parsing, stateful orchestration, and external-service dependencies. It needs more than happy-path smoke tests.
### Alternatives considered
- **Rely on manual quickstart only**
Rejected because the feature has too many state transitions and edge cases.
- **Mock everything at unit level only**
Rejected because contract drift with Superset and frontend schemas would go undetected.
- **Run live LLM/Superset tests in CI**
Rejected because determinism and cost would be poor.
---
## 12. Incremental Delivery Strategy
### Decision
Implement in three milestones:
### Milestone 1 — Sessioned Auto Review
Includes:
- source intake,
- session persistence,
- profile/finding generation,
- semantic-source application,
- exportable documentation/validation outputs.
### Milestone 2 — Guided Clarification
Includes:
- clarification engine,
- resumable dialogue state,
- conflict review,
- field-level semantic overrides/locks.
### Milestone 3 — Controlled Execution
Includes:
- imported filter recovery,
- template-variable mapping,
- mapping approval workflow,
- compiled SQL preview,
- SQL Lab launch,
- audited run context.
### Rationale
This delivers user value earlier and reduces risk by separating understanding from execution. It also directly addresses the reviewer feedback that the feature should not be treated as a monolithic drop.
### Alternatives considered
- **Single all-at-once delivery**
Rejected due to risk concentration and hard-to-demonstrate progress.
- **Execution-first milestone**
Rejected because trust, semantics, and readiness are prerequisites for safe launch.
---
## 13. Risk Resolution Outcomes
### Decision
Treat the following as non-negotiable gates:
- no Superset-side preview support → stop execution-scope planning
- no reliable SQL Lab session handoff → block launch design
- no typed API schemas → block parallel frontend/backend implementation
- no explicit semantic/clarification modules → block Phase 1 contract completion
### Rationale
These are architecture-shaping constraints, not minor implementation details.
### Alternatives considered
- **Continue with placeholders and fill gaps during coding**
Rejected because the feature is too stateful and contract-heavy for speculative implementation.
---
## 14. Review Feedback Incorporated
### Decision
The next design phase must explicitly address the previously identified review gaps by:
- adding `SemanticSourceResolver`,
- adding `ClarificationEngine`,
- expanding `SupersetCompilationAdapter` into a broader extraction + compilation boundary or adding `SupersetContextExtractor`,
- fully typing `api.yaml`,
- adding mapping and field-level endpoints,
- adding session lifecycle endpoints,
- adding error-path coverage to `quickstart.md`,
- preserving delivery milestones in the plan.
### Rationale
These gaps are valid and materially affect implementation readiness. Folding them into Phase 1 keeps the planning workflow honest and actionable.
### Alternatives considered
- **Ignore review findings until code review**
Rejected because several issues are architectural, not cosmetic.
---
## Final Phase 0 Result
All identified `NEEDS CLARIFICATION` items from the plan are resolved at the planning level.
The implementation should proceed into Phase 1 with:
- expanded typed data model,
- expanded contracts,
- expanded API surface,
- negative-path quickstart scenarios,
- post-design constitution re-check.

170
specs/027-dataset-llm-orchestration/spec.md Normal file
View File

@@ -0,0 +1,170 @@
# Feature Specification: LLM Dataset Orchestration
**Feature Branch**: `027-dataset-llm-orchestration`
**Created**: 2026-03-16
**Status**: Draft
**Input**: User description: "Я хочу проработать механизм llm документирования и проверки датасетов. И в автоматическом режиме, и в режиме диалога с агентом с уточнением атрибутов и прочего неявного. Так же, нам нужен механизм запуска датасетов на стороне superset, с поддержкой jinja шаблонов. В идеале, пользователь должен скормить ссылку из суперсета с сохраненными native filters, ss-tools должен вытащить все фильтры и собрать их для датасета"
## Clarifications
### Session 2026-03-16
- Q: Which execution target should be canonical for approved dataset launch? → A: Superset SQL Lab session is the canonical audited launch target.
- Q: What user action should be required to clear mapping warnings before launch? → A: Any mapping warning requires explicit user approval, but manual edit is optional.
- Q: What should happen if Superset-side SQL compilation is unavailable before launch? → A: Launch stays blocked until Superset-side compiled preview succeeds.
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Recover, enrich, and explain dataset context automatically (Priority: P1)
A data engineer or analytics engineer submits a dataset or a Superset link and immediately receives a readable explanation of what the dataset is, which filters were recovered, which semantic labels were reused from trusted sources, and what still needs review.
**Why this priority**: The first user need is fast understanding with minimal reinvention. Without an immediate and trustworthy first-pass interpretation, neither clarification nor execution provides value.
**Independent Test**: Can be fully tested by submitting a dataset with partial metadata or a Superset link with saved filters and verifying that the system produces a business-readable summary, distinguishes source confidence, searches trusted semantic sources before generating new labels, and shows the next recommended action without requiring manual dialogue.
**Acceptance Scenarios**:
1. **Given** a dataset with partial technical metadata, **When** the user starts automatic review, **Then** the system generates a business-readable documentation draft, groups known and unresolved attributes, and presents a current readiness state.
2. **Given** a valid Superset link with reusable saved native filters, **When** the user imports it, **Then** the system recovers the available filter context and presents imported values separately from inferred or user-provided values.
3. **Given** connected dictionaries, spreadsheet sources, or trusted reference datasets are available, **When** automatic review runs, **Then** the system attempts semantic enrichment from those sources before creating AI-generated labels from scratch.
4. **Given** multiple semantic candidates exist for a field, **When** the first summary is shown, **Then** the system clearly indicates the provenance and confidence level of the chosen or suggested semantic value.
---
### User Story 2 - Resolve ambiguities through guided clarification (Priority: P2)
A data steward, analytics engineer, or domain expert works with an agent to resolve ambiguous business meanings, conflicting metadata, conflicting semantic sources, and missing run-time values one issue at a time.
**Why this priority**: Real datasets often contain implicit semantics that cannot be derived safely from source metadata alone. Guided clarification converts uncertainty into auditable decisions.
**Independent Test**: Can be fully tested by opening clarification mode for a dataset with ambiguous attributes or conflicting semantic sources and verifying that the system asks focused questions, explains why each question matters, stores answers, and updates readiness and validation outcomes in real time.
**Acceptance Scenarios**:
1. **Given** a dataset has blocking ambiguities, **When** the user starts guided clarification, **Then** the system asks one focused question at a time and explains the significance of the question in business terms.
2. **Given** the system already has a current guess for an unresolved attribute, **When** the question is shown, **Then** the system presents that guess along with selectable answers, a custom-answer option, and a skip option.
3. **Given** semantic source reuse is likely, **When** the system detects a strong match with a trusted dictionary or reference dataset, **Then** the agent can proactively suggest that source as the preferred basis for semantic enrichment.
4. **Given** fuzzy semantic matches were found from a selected dictionary or dataset, **When** the system presents them, **Then** the user can approve them in bulk, review them individually, or keep only exact matches.
5. **Given** the user confirms or edits an answer, **When** the response is saved, **Then** the system updates the dataset profile, validation findings, and readiness state without losing prior context.
6. **Given** the user exits clarification before all issues are resolved, **When** the session is saved, **Then** the system preserves answered questions, unresolved questions, and the current recommended next action.
---
### User Story 3 - Prepare and launch a controlled dataset run (Priority: P3)
A BI engineer reviews the assembled run context, verifies filters and placeholders, understands any remaining warnings, reviews the compiled SQL preview, and launches the dataset with confidence that the execution can be reproduced later.
**Why this priority**: Execution is the final high-value outcome, but it must feel controlled and auditable rather than opaque.
**Independent Test**: Can be fully tested by preparing a dataset run from imported or manually confirmed filter context and verifying that the system blocks missing required values, blocks missing preview approval conditions, allows review and editing, and records the exact run context used.
**Acceptance Scenarios**:
1. **Given** an assembled dataset context contains required filters and placeholders, **When** the user opens run preparation, **Then** the system shows the effective filters, unresolved assumptions, semantic provenance signals, and current run readiness in one place.
2. **Given** required values are still missing, **When** the user attempts to launch, **Then** the system blocks launch and highlights the specific values that must be completed.
3. **Given** warning-level mapping transformations are present, **When** the user reviews run preparation, **Then** the system requires explicit approval for each warning before launch while still allowing optional manual edits.
4. **Given** Superset-side SQL compilation preview is unavailable or fails, **When** the user attempts to launch, **Then** the system blocks launch until a successful compiled preview is available.
5. **Given** the dataset is run-ready, **When** the user confirms launch, **Then** the system creates or starts a Superset SQL Lab session as the canonical execution target and records the dataset identity, effective filters, parameter values, outstanding warnings, and execution outcome for later audit or replay.
---
### Edge Cases
- What happens when a dataset has enough structural metadata to document technically but not enough business context to explain its meaning?
- How does the system handle a Superset link that identifies the dataset but contains no reusable native filters?
- What happens when imported filters conflict with previously saved defaults or with the dataset’s documented business meaning?
- How does the system handle parameterized placeholders that exist in the run context but do not yet have values?
- What happens when a user skips clarification questions and proceeds with warnings?
- How does the system present cases where one attribute is confirmed by a user, inferred from metadata, and contradicted by imported filter context?
- What happens when a user leaves during clarification or run preparation and returns later?
- What happens when a semantic label exists in a spreadsheet dictionary, a reference dataset, and an AI proposal with different values?
- How does the system handle fuzzy semantic matches where source and target names are similar in meaning but not identical in form?
- What happens when a user manually edits a semantic value and a higher-confidence imported source becomes available later?
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST allow users to start dataset review and execution preparation from the frontend workspace by selecting a dataset source or providing a Superset link.
- **FR-002**: The system MUST generate an initial dataset profile that distinguishes confirmed metadata, inferred metadata, imported metadata, unresolved metadata, and AI-draft metadata where applicable.
- **FR-003**: The system MUST produce human-readable dataset documentation that explains the dataset purpose, business meaning, major attributes, filters, and known limitations in language suitable for operational stakeholders.
- **FR-004**: The system MUST assign and display a current readiness state for the dataset review so users can immediately understand whether the dataset is review-ready, semantic-source-review-needed, clarification-needed, partially ready, or run-ready.
- **FR-005**: The system MUST validate dataset completeness and consistency across attributes, business semantics, semantic enrichment sources, filters, assumptions, and execution readiness.
- **FR-006**: The system MUST classify validation findings into blocking issues, warnings, and informational findings.
- **FR-007**: The system MUST allow users to inspect the provenance of important dataset values, including whether each value was confirmed from a connected dictionary, imported from a trusted dataset, inferred from fuzzy matching, generated as an AI draft, manually edited by a user, or still unresolved.
- **FR-008**: The system MUST search connected semantic sources during automatic review, including supported external dictionaries and trusted reference datasets, before creating AI-generated semantic values from scratch.
- **FR-009**: The system MUST support semantic enrichment for at least `verbose_name`, `description`, and display formatting metadata for dataset fields and metrics when such metadata is available from a trusted source.
- **FR-010**: The system MUST apply a visible confidence hierarchy to semantic enrichment candidates in this order: exact dictionary/file match, trusted reference dataset match, fuzzy semantic match, AI-generated draft.
- **FR-011**: The system MUST allow users to choose and apply a semantic source from the frontend workspace using supported source types, including uploaded files, connected tabular dictionaries, and existing trusted Superset datasets.
- **FR-012**: The system MUST allow users to start a guided clarification flow for unresolved or contradictory dataset details.
- **FR-013**: The guided clarification flow MUST present one focused question at a time rather than an unstructured list of unresolved items.
- **FR-014**: Each clarification question MUST explain why the answer matters and, when available, show the system’s current best guess.
- **FR-015**: The system MUST allow the user to answer with a suggested option, provide a custom answer, skip the question, or mark the item for later expert review.
- **FR-016**: The system MUST allow the agent to proactively recommend a semantic source when schema overlap or semantic similarity with a trusted source is strong enough to justify reuse.
- **FR-017**: The system MUST distinguish exact semantic matches from fuzzy semantic matches and MUST require user review before fuzzy matches are applied.
- **FR-018**: The system MUST preserve answers provided during clarification and immediately update the dataset profile, validation findings, and readiness state when those answers affect review outcomes.
- **FR-019**: The system MUST allow users to pause and resume a clarification session without losing prior answers, unresolved items, or progress state.
- **FR-020**: The system MUST summarize what changed when a clarification session ends, including resolved ambiguities, remaining ambiguities, and impact on run readiness.
- **FR-021**: (Consolidated with FR-001)
- **FR-022**: The system MUST extract reusable saved native filters from a provided Superset link whenever such filters are present and accessible.
- **FR-023**: The system MUST detect and expose runtime template variables referenced by the dataset execution logic so they can be mapped from imported or user-provided filter values.
- **FR-024**: The system MUST present extracted filters with their current value, source, confidence state, and whether user confirmation is required.
- **FR-025**: The system MUST preserve partially recovered value when a Superset import is incomplete and MUST explain which parts were recovered successfully and which still require manual or guided completion.
- **FR-026**: The system MUST support dataset execution contexts that include parameterized placeholders so users can complete required run-time values before launch.
- **FR-027**: The system MUST provide a dedicated pre-run review that presents the effective dataset identity, selected filters, required placeholders, unresolved assumptions, and current warnings in one place before launch.
- **FR-028**: The system MUST require explicit user approval for each warning-level mapping transformation before launch, while allowing the user to manually edit the mapped value instead of approving it.
- **FR-029**: The system MUST require a successful Superset-side compiled SQL preview before launch and MUST keep launch blocked if the preview is unavailable or compilation fails.
- **FR-030**: The system MUST prevent dataset launch when required values, required execution attributes, required warning approvals, or a required compiled preview are missing and MUST explain what must be completed.
- **FR-031**: The system MUST allow users to review and adjust the assembled filter set before starting a dataset run.
- **FR-032**: The system MUST use a Superset SQL Lab session as the canonical audited execution target for approved dataset launch.
- **FR-033**: The system MUST record the dataset run context, including dataset identity, selected filters, parameter values, unresolved assumptions, the associated SQL Lab session reference, mapping approvals, semantic-source decisions, and execution outcome, so that users can audit or repeat the run later.
- **FR-034**: The system MUST support a workflow where automatic review, semantic enrichment, guided clarification, and dataset execution can be used independently or in sequence on the same dataset.
- **FR-035**: The system MUST provide exportable outputs for dataset documentation and validation results so users can share them outside the immediate workflow.
- **FR-036**: The system MUST preserve a usable frontend session state when a user stops mid-flow so they can resume review, clarification, semantic enrichment review, or run preparation without reconstructing prior work.
- **FR-037**: The system MUST make the recommended next action explicit at each major state of the workflow.
- **FR-038**: The system MUST provide side-by-side comparison when multiple semantic sources disagree for the same field and MUST not silently overwrite a user-entered value with imported or AI-generated metadata.
- **FR-039**: The system MUST preserve manual semantic overrides unless the user explicitly replaces them.
- **FR-040**: The system MUST allow users to apply semantic enrichment selectively at field level rather than only as an all-or-nothing operation.
- **FR-041**: The system MUST provide an inline feedback mechanism (thumbs up/down) for AI-generated content to support continuous improvement of semantic matching and summarization.
- **FR-042**: The system MUST support multi-user collaboration on review sessions, allowing owners to invite collaborators with specific roles (viewer, reviewer, approver).
- **FR-043**: The system MUST provide batch approval actions for mapping warnings and fuzzy semantic matches to reduce manual effort for experienced users.
- **FR-044**: The system MUST capture and persist a structured event log of all session-related actions (e.g., source intake, answer submission, approval, launch) to support audit, replay, and collaboration visibility.
### Key Entities *(include if feature involves data)*
- **Dataset Profile**: The consolidated representation of a dataset, including business purpose, attributes, filters, assumptions, readiness state, validation state, provenance of each important fact, and semantic enrichment status.
- **Validation Finding**: A blocking issue, warning, or informational observation raised during dataset review, including severity, explanation, affected area, and resolution state.
- **Clarification Session**: A resumable interaction record that stores unresolved questions, user answers, system guesses, expert-review flags, and remaining ambiguities for a dataset.
- **Semantic Source**: A reusable origin of semantic metadata, such as an uploaded file, connected tabular dictionary, or trusted reference dataset, used to enrich field- and metric-level business meaning.
- **Semantic Mapping Decision**: A recorded choice about which semantic source or proposed value was accepted, rejected, edited, or left unresolved for a field or metric.
- **Imported Filter Set**: The collection of reusable filters extracted from a Superset link, including source context, mapped dataset fields, current values, confidence state, and confirmation status.
- **Dataset Run Context**: The execution-ready snapshot of dataset inputs, selected filters, parameterized placeholders, unresolved assumptions, warnings, mapping approvals, semantic-source decisions, the associated SQL Lab session reference, and launch outcome used for auditing or replay.
- **Readiness State**: The current workflow status that tells the user whether the dataset is still being recovered, ready for review, needs semantic-source review, needs clarification, is partially ready, or is ready to run.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: At least 90% of datasets submitted with standard source metadata produce an initial documentation draft without requiring manual reconstruction from scratch.
- **SC-002**: Users can reach a first readable validation and documentation summary for a newly submitted dataset in under 5 minutes for the primary workflow.
- **SC-003**: At least 70% of eligible semantic fields are populated from trusted external dictionaries or trusted reference datasets before AI-generated drafting is needed.
- **SC-004**: At least 85% of clarification questions shown in guided mode are judged by pilot users as relevant and helpful to resolving ambiguity (measured via the built-in feedback mechanism).
- **SC-005**: At least 80% of Superset links containing reusable saved native filters result in an imported filter set that users can review without rebuilding the context manually.
- **SC-006**: At least 85% of pilot users correctly identify which values are confirmed versus imported versus inferred versus AI-generated during moderated usability review.
- **SC-007**: At least 90% of dataset runs started from an imported or clarified context include a complete recorded run context that can be reopened later.
- **SC-008**: Pilot users successfully complete the end-to-end flow of import, review, semantic enrichment, clarification, and launch on their first attempt in at least 75% of observed sessions.
- **SC-009**: Support requests caused by missing or unclear dataset attributes decrease by at least 40% within the target pilot group after adoption.
## Assumptions
- Users already have permission to access the datasets and Superset artifacts they submit to ss-tools.
- Saved native filters embedded in a Superset link are considered the preferred reusable source of analytical context when available.
- Users need both self-service automation and a guided conversational path because dataset semantics are often incomplete, implicit, conflicting, or distributed across multiple semantic sources.
- The feature is intended for internal operational use where clarity, traceability, semantic consistency, and repeatable execution are more important than raw execution speed.
- Exportable documentation and validation outputs are required for collaboration, review, and audit use cases.
- Users may choose to proceed with warnings, but not with missing required execution inputs, missing required mapping approvals, or missing required compiled preview.
- Superset SQL Lab session creation is the canonical audited launch path for approved execution.
- Warning-level mapping transformations require explicit user approval before launch, while manual correction remains optional.
- Launch requires a successful Superset-side compiled preview and cannot fall back to an unverified local approximation.
- Trusted semantic sources already exist or can be introduced incrementally through frontend-managed files, connected dictionaries, or reference datasets without requiring organizations to discard existing semantic workflows.

109
specs/027-dataset-llm-orchestration/tasks.md Normal file
View File

@@ -0,0 +1,109 @@
# Tasks: LLM Dataset Orchestration
**Feature Branch**: `027-dataset-llm-orchestration`
**Implementation Plan**: [`specs/027-dataset-llm-orchestration/plan.md`](/home/busya/dev/ss-tools/specs/027-dataset-llm-orchestration/plan.md)
---
## Phase 1: Setup
- [ ] T001 Initialize backend service directory structure for `dataset_review` in `backend/src/services/dataset_review/`
- [ ] T002 Initialize frontend component directory for `dataset-review` in `frontend/src/lib/components/dataset-review/`
- [ ] T003 Register `ff_dataset_auto_review`, `ff_dataset_clarification`, and `ff_dataset_execution` feature flags in configuration
- [ ] T004 [P] Seed new `DATASET_REVIEW_*` permissions in `backend/src/scripts/seed_permissions.py`
---
## Phase 2: Foundational Layer
- [ ] T005 [P] Implement Core SQLAlchemy models for session, profile, and findings in `backend/src/models/dataset_review.py`
- [ ] T006 [P] Implement Semantic, Mapping, and Clarification models in `backend/src/models/dataset_review.py`
- [ ] T007 [P] Implement Preview and Launch Audit models in `backend/src/models/dataset_review.py`
- [ ] T008 [P] Implement `DatasetReviewSessionRepository` (CRITICAL: C5, PRE: auth scope, POST: consistent aggregates, INVARIANTS: ownership scope) in `backend/src/services/dataset_review/repositories/session_repository.py`
- [ ] T009 [P] Create Pydantic schemas for Session Summary and Detail in `backend/src/schemas/dataset_review.py`
- [ ] T010 [P] Create Svelte store for session management in `frontend/src/lib/stores/datasetReviewSession.js`
---
## Phase 3: User Story 1 — Automatic Review (P1)
**Goal**: Submission of link/dataset produces immediate readable summary and semantic enrichment from trusted sources.
**Independent Test**: Submit a Superset link; verify session created, summary generated, and findings populated without manual intervention.
- [ ] T011 [P] [US1] Implement `StartSessionRequest` and lifecycle endpoints in `backend/src/api/routes/dataset_review.py`
- [ ] T012 [US1] Implement `DatasetReviewOrchestrator.start_session` (CRITICAL: C5, PRE: non-empty input, POST: enqueued recovery, BELIEF: uses `belief_scope`) in `backend/src/services/dataset_review/orchestrator.py`
- [ ] T013 [P] [US1] Implement `SupersetContextExtractor.parse_superset_link` (CRITICAL: C4, PRE: parseable link, POST: resolved target, REL: uses `SupersetClient`) in `backend/src/core/utils/superset_context_extractor.py`
- [ ] T014 [US1] Implement `SemanticSourceResolver.resolve_from_dictionary` (CRITICAL: C4, PRE: source exists, POST: confidence-ranked candidates) in `backend/src/services/dataset_review/semantic_resolver.py`
- [ ] T015 [US1] Implement Documentation and Validation export endpoints (JSON/Markdown) in `backend/src/api/routes/dataset_review.py`
- [ ] T016 [P] [US1] Implement `SourceIntakePanel` (C3, UX_STATE: Idle/Validating/Rejected) in `frontend/src/lib/components/dataset-review/SourceIntakePanel.svelte`
- [ ] T017 [P] [US1] Implement `ValidationFindingsPanel` (C3, UX_STATE: Blocking/Warning/Info) in `frontend/src/lib/components/dataset-review/ValidationFindingsPanel.svelte`
- [ ] T018 [US1] Create main `DatasetReviewWorkspace` (CRITICAL: C5, UX_STATE: Empty/Importing/Review) in `frontend/src/routes/datasets/review/[id]/+page.svelte`
- [ ] T019 [US1] Verify implementation matches ux_reference.md (Happy Path & Errors)
- [ ] T020 [US1] Acceptance: Perform semantic audit & algorithm emulation by Tester
---
## Phase 4: User Story 2 — Guided Clarification (P2)
**Goal**: Resolve ambiguities and conflicting metadata through one-question-at-a-time dialogue.
**Independent Test**: Open a session with unresolved findings; answer questions one by one and verify readiness state updates in real-time.
- [ ] T021 [P] [US2] Implement `ClarificationEngine.build_question_payload` (CRITICAL: C4, PRE: unresolved state, POST: prioritized question) in `backend/src/services/dataset_review/clarification_engine.py`
- [ ] T022 [US2] Implement `ClarificationEngine.record_answer` (CRITICAL: C4, PRE: question active, POST: answer persisted before state advance) in `backend/src/services/dataset_review/clarification_engine.py`
- [ ] T023 [P] [US2] Implement field-level semantic override and lock endpoints in `backend/src/api/routes/dataset_review.py`
- [ ] T024 [US2] Implement `SemanticLayerReview` component (C3, UX_STATE: Conflicted/Manual) in `frontend/src/lib/components/dataset-review/SemanticLayerReview.svelte`
- [ ] T025 [P] [US2] Implement `ClarificationDialog` (C3, UX_STATE: Question/Saving/Completed, REL: binds to `assistantChat`) in `frontend/src/lib/components/dataset-review/ClarificationDialog.svelte`
- [ ] T026 [US2] Implement LLM feedback (👍/👎) storage and UI handlers in `backend/src/api/routes/dataset_review.py`
- [ ] T027 [US2] Verify implementation matches ux_reference.md (Happy Path & Errors)
- [ ] T028 [US2] Acceptance: Perform semantic audit & algorithm emulation by Tester
---
## Phase 5: User Story 3 — Controlled Execution (P3)
**Goal**: Review mappings, generate Superset-side preview, and launch audited SQL Lab execution.
**Independent Test**: Map filters to variables; trigger preview; verify launch blocked until preview succeeds; verify SQL Lab session creation.
- [ ] T029 [P] [US3] Implement `SupersetContextExtractor.recover_imported_filters` and variable discovery in `backend/src/core/utils/superset_context_extractor.py`
- [ ] T030 [US3] Implement `SupersetCompilationAdapter.compile_preview` (CRITICAL: C4, PRE: effective inputs available, POST: Superset-compiled SQL only) in `backend/src/core/utils/superset_compilation_adapter.py`
- [ ] T031 [US3] Implement `DatasetReviewOrchestrator.launch_dataset` (CRITICAL: C5, PRE: run-ready + preview match, POST: audited run context) in `backend/src/services/dataset_review/orchestrator.py`
- [ ] T032 [P] [US3] Implement mapping approval and preview trigger endpoints in `backend/src/api/routes/dataset_review.py`
- [ ] T033 [P] [US3] Implement `ExecutionMappingReview` component (C3, UX_STATE: WarningApproval/Approved) in `frontend/src/lib/components/dataset-review/ExecutionMappingReview.svelte`
- [ ] T034 [P] [US3] Implement `CompiledSQLPreview` component (C3, UX_STATE: Ready/Stale/Error) in `frontend/src/lib/components/dataset-review/CompiledSQLPreview.svelte`
- [ ] T035 [US3] Implement `LaunchConfirmationPanel` (C3, UX_STATE: Blocked/Ready/Submitted) in `frontend/src/lib/components/dataset-review/LaunchConfirmationPanel.svelte`
- [ ] T036 [US3] Verify implementation matches ux_reference.md (Happy Path & Errors)
- [ ] T037 [US3] Acceptance: Perform semantic audit & algorithm emulation by Tester
---
## Final Phase: Polish & Security
- [ ] T038 Implement `SessionEvent` logger and persistence logic in `backend/src/services/dataset_review/event_logger.py`
- [ ] T039 Implement automatic version propagation logic for updated `SemanticSource` entities
- [ ] T040 Add batch approval API and UI actions for mapping/semantics
- [ ] T041 Add integration tests for Superset version compatibility matrix in `backend/tests/services/dataset_review/test_superset_matrix.py`
- [ ] T042 Final audit of RBAC enforcement across all session-mutation endpoints
- [ ] T043 Verify i18n coverage for all user-facing strings in `frontend/src/lib/i18n/`
---
## Dependencies & Strategy
### Story Completion Order
1. **Foundation** (Blocking: T005-T010)
2. **User Story 1** (Blocking for US2 and US3)
3. **User Story 2** (Can be implemented in parallel with US3 parts, but requires US1 findings)
4. **User Story 3** (Final terminal action)
### Parallel Execution Opportunities
- T011, T013, T016 (API, Parser, UI Setup) can run simultaneously once T001-T010 are done.
- T021 and T025 (Clarification Backend/Frontend) can run in parallel.
- T030 and T034 (Preview Backend/Frontend) can run in parallel.
### Implementation Strategy
- **MVP First**: Implement US1 with hardcoded trusted sources to prove the session/summary lifecycle.
- **Incremental Delivery**: Release US1 for documentation value, then US2 for metadata cleanup, finally US3 for execution.
- **WYSIWWR Guard**: T030 must never be compromised; if Superset API fails, implementation must prioritize the "Manual Launch" fallback defined in research.

720
specs/027-dataset-llm-orchestration/ux_reference.md Normal file
View File

@@ -0,0 +1,720 @@
# UX Reference: LLM Dataset Orchestration
**Feature Branch**: `027-dataset-llm-orchestration`
**Created**: 2026-03-16
**Status**: Draft
## 1. User Persona & Context
* **Primary user**: Analytics engineer or BI engineer who needs to quickly understand, validate, parameterize, enrich, and run a dataset that may have incomplete business context.
* **Secondary user**: Data steward or domain expert who helps confirm meanings, resolve ambiguities, and approve the documented interpretation of a dataset.
* **What is the user trying to achieve?**: Convert a raw dataset or a Superset-derived analytical context into something understandable, trustworthy, semantically enriched, and runnable without manually reverse-engineering filters, semantics, and hidden assumptions.
* **Mindset**: The user is usually unsure about part of the dataset. They want speed, but they do not want “magic” that hides uncertainty. They need confidence, traceability, the ability to intervene, and reuse of existing semantic assets rather than endless redefinition.
* **Context of use**:
* Reviewing a dataset before reuse in analysis.
* Preparing a dataset for migration or operational execution.
* Importing an existing analytical context from Superset instead of rebuilding it manually.
* Reusing semantic metadata from Excel or database dictionaries.
* Inheriting semantic layer settings from neighboring or master datasets in Superset.
* Collaborating with another person who knows the business meaning better than the technical owner.
## 2. UX Principles
* **Expose certainty, do not fake certainty**: The system must always distinguish confirmed facts, inferred facts, imported facts, unresolved facts, and AI drafts.
* **Guide, then get out of the way**: The product should proactively suggest next actions but should not force the user into a rigid wizard if they already know what they want to do.
* **Progress over perfection**: A user should be able to get partial value immediately, save progress, and return later.
* **One ambiguity at a time**: In dialogue mode, the user should never feel interrogated by a wall of questions.
* **Execution must feel safe**: Before launch, the user should clearly understand what will run, with which filters, with which unresolved assumptions.
* **Superset import should feel like recovery, not parsing**: The user expectation is not “we decoded a link”, but “we recovered the analysis context I had in Superset.”
* **What You See Is What Will Run (WYSIWWR)**: Before any launch, the system must show the final compiled SQL query exactly as it will be sent for execution, with all template substitutions already resolved.
* **Single Source of Truth for Execution**: The LLM never writes or edits SQL directly. The LLM only helps interpret business meaning and map available filter values into execution parameters. Jinja compilation and final SQL generation are always delegated to the native Superset execution API so the preview and the real run stay aligned.
* **Reuse before invention**: The system should prefer trusted semantic sources before generating new names or descriptions from scratch.
* **Confidence hierarchy must stay visible**: Semantic enrichment should follow a clear source priority: exact match from dictionary, inherited match from reference dataset, fuzzy semantic match, and only then AI-generated draft.
* **Manual intent wins**: The system must never silently overwrite a user’s manual semantic edits with imported or generated metadata.
## 3. Core Product Modes
### Mode A: Automatic Review
User submits a dataset or imported analytical context and immediately receives:
* documentation draft,
* validation findings,
* filter/context extraction result,
* semantic enrichment candidates for columns and metrics,
* recommended next action.
This mode is for speed and low-friction first-pass understanding.
Automatic review is not limited to generating names and descriptions from scratch. During first-pass analysis, the system actively searches connected semantic sources:
* external dictionaries from database tables or uploaded spreadsheet files,
* other reference datasets in Superset,
* neighboring datasets that reuse the same physical tables or overlapping schema,
* LLM-driven fuzzy semantic matching when exact reuse is not possible.
The semantic confidence hierarchy is explicit:
1. **Confirmed** — exact match from connected dictionary or file.
2. **Imported** — reused match from a trusted reference dataset.
3. **Inferred** — fuzzy or semantic match proposed through LLM-assisted comparison.
4. **AI Draft** — generated by the LLM from scratch when no stronger source exists.
This mode should feel like the system is recovering and inheriting existing semantic knowledge before inventing anything new.
### Mode B: Guided Clarification
User enters a focused interaction with the agent to resolve unresolved attributes, missing filter meanings, inconsistent business semantics, conflicting semantic sources, or run-time gaps.
This mode is for confidence-building and resolving uncertainty.
### Mode C: Run Preparation
User reviews the assembled run context, edits values where needed, confirms assumptions, inspects the compiled SQL preview, and launches the dataset only when the context is good enough.
This mode is for controlled execution.
## 4. Primary Happy Path
### High-Level Story
The user opens ss-tools because they have a dataset they need to understand and run, but they do not fully trust the metadata. They paste a Superset link or select a dataset source in the web interface. In seconds, the workspace fills with a structured interpretation: what the dataset appears to be, which filters were recovered, which Jinja-driven variables exist in the dataset, which semantic labels were inherited from trusted sources, what is already known, and what is still uncertain. The user scans a short human-readable summary, adjusts the business meaning manually if needed, approves a few semantic and filter mappings, resolves only the remaining ambiguities through a short guided dialogue, and reaches a “Run Ready” state after reviewing the final SQL compiled by Superset itself. Launch feels deliberate and safe because the interface shows exactly what will be used, how imported filters map to runtime variables, and where each semantic label came from.
### Detailed Step-by-Step Journey
#### Step 1: Entry
The user lands on an empty “Dataset Review Workspace”.
The screen offers two clear entry paths:
* **Paste Superset Link**
* **Select Dataset Source**
The user should instantly understand that both paths lead to the same outcome: a documented, semantically enriched, and runnable dataset context.
**Desired feeling**: “I know where to start.”
#### Step 2: Source Intake
The user pastes a Superset link.
The system immediately validates the input shape and responds optimistically:
* link recognized,
* source identified,
* import started.
The system should avoid blocking the user with technical checks unless the import is impossible.
**Desired feeling**: “The system understood what I gave it.”
#### Step 3: Context Recovery
The system assembles the first-pass interpretation:
* dataset identity,
* imported native filters,
* obvious dimensions/measures,
* initial business summary,
* unresolved items,
* discovered Jinja variables used by the dataset,
* candidate semantic sources for columns and metrics.
Context recovery is not limited to decoding the Superset link. The system also inspects the dataset through the Superset-side API to detect all available runtime template variables referenced inside the dataset query logic, for example variables used in expressions like `{{ filter_values('region') }}`.
In parallel, ss-tools gathers semantic metadata in the background from neighboring or reference datasets, especially those using the same physical tables, overlapping schema, or known business lineage. This gives the system an immediate base for suggesting `verbose_name`, `description`, and `d3format` values before asking the user to define them manually.
Instead of showing a spinner for too long, the interface should reveal results progressively as they become available:
* dataset recognized,
* saved native filters recovered from the link,
* dataset template variables detected from the dataset body,
* nearby or master datasets identified as semantic candidates,
* dictionary or spreadsheet matches found,
* preliminary mapping candidates suggested between filter inputs and template variables,
* preliminary semantic matches suggested for columns and metrics.
**Desired feeling**: “I’m already getting value before everything is finished.”
#### Step 4: First Readable Summary
The user sees a compact summary card:
* what this dataset appears to represent,
* what period/scope/segments are implied,
* what filters were recovered,
* whether execution is currently possible.
This summary is the anchor of trust. It must be short, business-readable, and immediately useful.
The summary is editable. If the user sees that the generated business meaning is incorrect or incomplete, they can use **[Edit]** to manually correct the summary without starting a long clarification dialogue.
**Desired feeling**: “I can explain this dataset to someone else already, and I can quickly fix the explanation if it is wrong.”
#### Step 5: Validation Triage
The system groups findings into:
* **Blocking**
* **Needs Attention**
* **Informational**
The user does not need to read everything. They need to know what is stopping them from running, what is risky, and what can be reviewed later.
**Desired feeling**: “I know what matters right now.”
#### Step 6: Clarification Decision
If ambiguities remain, the product presents an explicit choice:
* **Fix now with agent**
* **Continue with current assumptions**
* **Save and return later**
This is a critical UX moment. The user must feel in control rather than forced into a mandatory workflow.
**Desired feeling**: “I decide how much rigor I need right now.”
#### Step 7: Guided Clarification
If the user chooses clarification, the workspace switches into a focused dialogue mode.
The agent asks one question at a time, each with:
* why this matters,
* what the current guess is,
* quick-select answers when possible,
* an option to skip,
* an option to say “I don’t know”.
Each answer updates the dataset profile in real time.
**Desired feeling**: “This is helping me resolve uncertainty, not making me fill a form.”
#### Step 8: Run Readiness Review
When blocking issues are resolved, the system returns to a run-preparation state with:
* selected filters,
* placeholder values,
* unresolved warnings,
* final business summary,
* provenance labels for each key value,
* visible mapping between imported filters and detected Jinja template variables,
* semantic provenance for important columns and metrics,
* a preview of the final compiled SQL returned by Superset.
This step contains the critical **Smart Mapping** stage. The system uses the LLM to propose a mapping between the filter values recovered from the Superset link and the Jinja variables discovered in the dataset. The LLM does not generate SQL. It only assembles or suggests the parameter payload used for execution, such as the effective template parameter object.
The user can review each mapping explicitly:
* source filter,
* target Jinja variable,
* transformed value if normalization was required,
* confidence state,
* warning state,
* manual override.
Semantic review also remains visible here. Users can inspect where key `verbose_name`, `description`, and `d3format` values came from and whether they were confirmed from a dictionary, imported from a reference dataset, inferred from fuzzy matching, or generated as AI drafts.
Before launch, ss-tools performs a **Dry Run via Superset API**. The backend sends the assembled execution parameters to Superset for safe server-side compilation of the query without triggering the real dataset run. The result is shown as the **Compiled Query Preview**.
The **Compiled Query Preview** is a read-only SQL block that shows the final SQL with Jinja substitutions already resolved by Superset. Substituted values should be visibly highlighted so users can quickly inspect what changed.
If smart mapping introduced warnings, for example a value normalization such as `Europe → EU`, the launch button stays blocked until the user explicitly approves the mapping or edits it manually. The user must never run a query whose effective substitutions are still ambiguous.
Before launch, the user should be able to inspect the full context in one place.
**Desired feeling**: “I know exactly what will run, and I trust that this preview matches the real execution.”
#### Step 9: Launch
The user presses **Launch Dataset**.
The final confirmation is not a generic “Are you sure?” modal. It is a run summary:
* dataset,
* effective filters,
* variable inputs,
* warnings still open,
* compiled SQL preview status,
* semantic source summary for important fields,
* what will be recorded for audit.
“Launch” has a concrete execution meaning. Depending on the selected path, ss-tools either:
* sends the prepared execution payload for execution in Superset SQL Lab, or
* redirects the user into a ready-to-run Superset analytical view with the assembled execution context already applied.
In both cases, the user expectation is the same: the execution uses the exact compiled query and runtime parameters they already reviewed.
**Desired feeling**: “This run is controlled, reproducible, and uses the exact query I approved.”
#### Step 10: Post-Run Feedback
After launch, the system confirms:
* run started or completed,
* context saved,
* documentation linked,
* validation snapshot preserved,
* compiled query version associated with the run,
* execution handoff target available,
* semantic mapping decisions preserved for reuse.
The post-run state should provide useful artifacts, such as:
* a link to the created Superset execution session,
* a preview of the first rows of returned data directly in ss-tools when available,
* or an updated saved dataset context that can be reopened and reused later.
The user can reopen the run later and understand the exact state used.
**Desired feeling**: “I can trust this later, not just right now.”
## 5. End-to-End Interaction Model
## 5.1 Main Workspace Structure
**Screen/Component**: Dataset Review Workspace
**Layout**: Adaptive three-column workspace.
### Left Column: Source & Session
* dataset source card,
* Superset import status,
* session state,
* save/resume controls,
* recent actions timeline.
### Center Column: Meaning & Validation
* generated business summary,
* manual override with **[Edit]** for the generated summary and business interpretation,
* documentation draft preview,
* validation findings grouped by severity,
* confidence markers,
* unresolved assumptions.
### Center Column: Columns & Metrics
* semantic layer table for columns and metrics,
* visible values for `verbose_name`, `description`, and formatting metadata where available,
* provenance badges for every semantically enriched field, such as `[ 📄 dict.xlsx ]`, `[ 📊 Dataset: Master Sales ]`, or `[ ✨ AI Guessed ]`,
* side-by-side conflict view when multiple semantic sources disagree,
* **Apply semantic source...** action that opens source selection for file, database dictionary, or existing Superset datasets,
* manual per-field override so the user can keep, replace, or rewrite semantic metadata.
### Right Column: Filters & Execution
* imported filters,
* parameter placeholders,
* **Jinja Template Mapping** block with visible mapping between source filters and detected dataset variables,
* run-time values,
* **Compiled SQL Preview** block or action to open the compiled query returned by Superset API,
* readiness checklist,
* primary CTA.
This structure matters because the user mentally works across four questions:
1. What is this?
2. Can I trust its meaning?
3. Can I trust what will run?
4. Can I run it?
## 5.2 Primary CTAs by State
The main CTA should change based on readiness:
* **Empty** → `Import from Superset`
* **Intake complete** → `Review Documentation`
* **Semantic source available** → `Apply Semantic Source`
* **Ambiguities present** → `Start Clarification`
* **Mapping warnings present** → `Approve Mapping`
* **Compilation preview missing** → `Generate SQL Preview`
* **Blocking values missing** → `Complete Required Values`
* **Run-ready** → `Launch Dataset`
The product should never make the user guess what the next best action is.
## 5.3 Information Hierarchy
At any moment, the most visible information should be:
1. current readiness state,
2. blocking problems,
3. imported/recovered context,
4. mapping status between recovered filters and runtime variables,
5. semantic source confidence for key fields,
6. business explanation,
7. compiled SQL preview status,
8. detailed metadata.
Raw detail is valuable, but it should never compete visually with the answer to “Can I proceed?”
## 6. Dialogue UX: Agent Interaction Design
## 6.1 Conversation Pattern
The agent interaction is not a chat for general brainstorming. It is a structured clarification assistant.
Each prompt should contain:
* **Question**
* **Why this matters**
* **Current system guess**
* **Suggested answers**
* **Optional free-form input**
* **Skip for now**
Example interaction:
```text
Question 2 of 5
What does the "region_scope" filter represent in this dataset?
Why this matters:
This value changes how the final aggregation is interpreted.
Current guess:
It appears to mean the reporting region, not the customer region.
Choose one:
[1] Reporting region
[2] Customer region
[3] Both depending on use case
[4] I’m not sure
[5] Enter custom answer
```
This keeps the agent focused, useful, and fast.
## 6.2 Agent-Led Semantic Source Suggestion
The agent may proactively suggest a semantic source when the schema strongly resembles an existing reference.
Example interaction:
```text
Question: Semantic Layer Source
I noticed that 80% of the columns in this dataset (user_id, region, revenue) match the existing "Core_Users_Master" dataset.
Why this matters:
Reusing existing metadata keeps verbose names, descriptions, and d3formats consistent across dashboards.
How would you like to populate the semantic layer?
[1] Copy from "Core_Users_Master" dataset (Recommended)
[2] Upload an Excel (.xlsx) or DB dictionary
[3] Let AI generate them from scratch
[4] Skip and leave as database names
```
This should feel like a smart reuse recommendation, not a forced detour.
## 6.3 Fuzzy Matching Confirmation Pattern
When the user chooses an external dictionary and exact matches are incomplete, the agent should summarize the result clearly before applying it.
Example:
```text
I matched 15 columns exactly from the selected dictionary.
I also found 3 likely semantic matches that need confirmation.
Please review:
- reg_code → region
- rev_total → revenue
- usr_identifier → user_id
How would you like to proceed?
[1] Accept all suggested semantic matches
[2] Review one by one
[3] Ignore fuzzy matches and keep exact ones only
```
The user must understand which matches are exact, which are semantic guesses, and which remain unresolved.
## 6.4 Agent Tone
The agent should sound:
* precise,
* calm,
* operational,
* non-judgmental.
It should never imply the user made a mistake when data is ambiguous. Ambiguity is treated as a normal property of datasets.
## 6.5 Dialogue Controls
The user must be able to:
* skip a question,
* save and exit,
* review previous answers,
* revise a prior answer,
* mark an item as “needs expert review”.
These controls are critical for real-world data workflows.
## 6.6 Dialogue Exit Conditions
The user can leave dialogue mode when:
* all blocking ambiguities are resolved,
* user chooses to continue with warnings,
* session is saved for later,
* no further useful clarification can be generated.
The agent must explicitly summarize what changed before exit:
* resolved items,
* still unresolved items,
* effect on run readiness.
## 7. State Model
### State 1: Empty
No dataset loaded. Clear entry choices.
### State 2: Importing
Progressive loading with visible milestones.
### State 3: Review Ready
Documentation and validation visible. User can understand the dataset immediately.
### State 4: Semantic Source Review Needed
The system found reusable semantic sources, but the user still needs to choose, approve, or reject them.
### State 5: Clarification Needed
There are meaningful unresolved items. Product suggests dialogue mode.
### State 6: Clarification Active
One-question-at-a-time guided flow.
### State 7: Mapping Review Needed
Recovered filters and detected Jinja variables exist, but the mapping still requires approval, correction, or completion.
### State 8: Compiled Preview Ready
Superset has compiled the current parameter set, and the user can inspect the exact SQL that would run.
### State 9: Partially Ready
No blockers, but warnings remain.
### State 10: Run Ready
Everything required for launch is complete.
### State 11: Run In Progress
Execution feedback and status tracking.
### State 12: Completed
Run outcome and saved context available.
### State 13: Recovery Required
Import, mapping, semantic enrichment, or compilation was partial; manual or guided recovery needed.
## 8. Key User Decisions
The UX must support these decisions explicitly:
* Is this imported context trustworthy enough?
* Which semantic source should define `verbose_name`, `description`, and `d3format`?
* Do I want to reuse a master dataset or apply a spreadsheet/database dictionary?
* Should I accept fuzzy semantic matches or only exact ones?
* Do I need clarification now or can I continue?
* Are the filters correct as imported?
* Which source filter should map to which Jinja variable?
* Is the transformed value acceptable if normalization was applied?
* Which values are confirmed versus guessed?
* Does the compiled SQL match my intent?
* Is the dataset safe enough to run?
* Do I want to save current progress and come back later?
If the interface does not make these decisions visible, the user will feel lost even if the feature is technically correct.
## 9. UI Layout & Flow
**Screen**: Dataset Review Workspace
* **Top Bar**:
* Source badge
* Dataset name
* Readiness status pill
* Save session
* Export summary
* **Hero Summary Block**:
* “What this dataset is”
* “What is ready”
* “What still needs attention”
* Primary CTA
* **[Edit]** action for manual correction
* **Tabs or Sections**:
* Overview
* Documentation
* Semantic Layer
* Validation
* Filters
* Mapping
* SQL Preview
* Clarification History
* Run History
* **Right Rail**:
* readiness checklist,
* semantic source status,
* missing required values,
* mapping warnings,
* SQL preview status,
* launch button.
## 10. Micro-Interactions
* Imported filters should animate into the panel one by one as they are recovered.
* Detected Jinja variables should appear as a second wave of recovered context so the user understands execution awareness is expanding.
* Detected semantic source candidates should appear as a third wave, with confidence labels and provenance badges.
* Every clarified answer should immediately remove or downgrade a validation finding where relevant.
* Provenance badges should update live:
* Confirmed
* Imported
* Inferred
* AI Draft
* Mapped
* Needs Review
* The primary CTA should change smoothly, not abruptly, as the state progresses.
* When launch becomes available, the interface should celebrate readiness subtly but should not hide remaining warnings.
* Value transformations proposed by mapping should be visually diffed so the user can spot changes like `Europe → EU` instantly.
* The compiled SQL preview should visibly refresh when mapping or parameter values change.
* Manual semantic overrides should visually lock the affected field so later imports do not silently replace it.
## 11. Error Experience
**Philosophy**: Never show a dead end. Every error state must preserve recovered value, explain what failed, and show the nearest path forward.
### Scenario A: Superset Link Recognized, Filter Extraction Partial
* **User Action**: Pastes a valid Superset link with partially recoverable filter state.
* **System Response**:
* “We recovered the dataset and 3 filters, but 2 saved filters need manual review.”
* Missing or low-confidence filters are listed explicitly.
* The system still opens the workspace with partial value.
* **Recovery**:
* review recovered filters,
* add missing ones manually,
* ask the agent to help reconstruct intent.
### Scenario B: No Clear Business Meaning Can Be Inferred
* **User Action**: Submits a technically valid dataset with poor metadata.
* **System Response**:
* “We could identify the structure of this dataset, but not its business meaning.”
* Documentation remains skeletal but usable.
* Clarification becomes the obvious next step.
* **Recovery**:
* launch dialogue mode,
* invite domain expert input,
* save draft and resume later.
### Scenario C: Required Run-Time Values Missing
* **User Action**: Tries to launch with incomplete placeholders.
* **System Response**:
* launch blocked,
* missing values highlighted in-place,
* concise summary of what is required.
* **Recovery**:
* fill values inline,
* return to review,
* or save incomplete context.
### Scenario D: Conflicting Meanings Across Sources
* **User Action**: Reviews a dataset where imported filter context and documented semantics conflict.
* **System Response**:
* both candidate meanings are shown side-by-side,
* neither is silently chosen if confidence is low,
* the conflict is framed as a decision, not a failure.
* **Recovery**:
* user confirms one meaning,
* leaves item unresolved,
* or marks for expert review.
### Scenario E: User Leaves Mid-Flow
* **User Action**: Closes the session before clarification or run prep is complete.
* **System Response**:
* autosave or explicit save confirmation,
* summary of current progress,
* preserved unresolved items.
* **Recovery**:
* resume from last state without repeating prior answers.
### Scenario F: Superset API Compilation Failed
* **User Action**: The mapped runtime values are sent for Jinja compilation, but Superset returns a compilation error.
* **System Response**:
* the **Compiled SQL Preview** switches into an error state instead of pretending preview is available,
* the problematic variable or mapping row is highlighted,
* the original compilation error returned by Superset is shown in readable form,
* launch remains blocked until the issue is resolved.
* **Recovery**:
* user manually edits the mapped value,
* user changes the filter-to-template mapping,
* or user asks the agent to help normalize the value format and then regenerates the preview.
### Scenario G: Semantic Sources Conflict
* **User Action**: A column has one value from a spreadsheet dictionary, a different value from a reference dataset, and a third AI-generated proposal.
* **System Response**:
* the interface shows a side-by-side comparison instead of silently choosing one,
* the higher-priority source is highlighted as recommended,
* the conflict is marked as a warning if user input would be changed.
* **Recovery**:
* user selects one source,
* user keeps the current manual value,
* or user applies the recommended higher-confidence source field by field.
## 12. UX for Trust & Transparency
Trust is central to this feature.
The interface must visibly answer:
* Where did this value come from?
* Did the system infer this or did the user confirm it?
* Which runtime variable will receive this value?
* Was the final SQL preview compiled by Superset or just estimated locally?
* Did this semantic label come from a dictionary, another dataset, a fuzzy match, or AI generation?
* What is still unknown?
* What will happen if I proceed anyway?
Recommended trust markers:
* provenance badge on important fields,
* confidence labels for imported or inferred data,
* mapping approval status,
* “compiled by Superset” status on the SQL preview,
* “last changed by” and “changed in clarification” notes,
* “used in run” markers for final execution inputs.
Conflict rule:
* The system must never silently overwrite user-entered semantic values with data from a dictionary, another dataset, or AI generation.
* If multiple sources disagree, the interface shows them side by side and either:
* asks the user to choose, or
* recommends the highest-priority source while clearly marking the recommendation as a warning until approved.
* Manual user input remains the most sensitive value and must be preserved unless the user explicitly replaces it.
## 13. UX for Collaboration
This workflow often spans more than one person.
The UX should support:
* sharing documentation draft,
* handing a clarification session to a domain expert,
* preserving unresolved questions explicitly,
* recording who confirmed which meaning,
* sharing the reviewed compiled SQL preview as part of execution approval,
* sharing which semantic sources were applied and why.
The user should be able to leave behind a state that another person can understand in under a minute.
## 14. Tone & Voice
* **Style**: Concise, trustworthy, operational, and transparent.
* **System behavior language**:
* Prefer: “Recovered”, “Confirmed”, “Imported”, “Inferred”, “AI Draft”, “Needs review”, “Ready to run”, “Compiled by Superset”
* Avoid: “Magic”, “Solved”, “Guaranteed”, “Auto-fixed”
* **Terminology**:
* Use “dataset”, “clarification”, “validation finding”, “run context”, “imported filter”, “Jinja variable”, “template mapping”, “compiled SQL preview”, “semantic source”, “provenance”, “assumption”, “confidence”.
* Avoid overly technical wording in primary UX surfaces when a business-readable phrase exists.
## 15. UX Success Signals
The UX is working if users can, with minimal hesitation:
* understand what dataset they are dealing with,
* see what was recovered from Superset,
* see which Jinja variables were discovered for runtime execution,
* understand which semantic source supplied each important field,
* reuse existing semantic assets before accepting AI guesses,
* tell which values are trustworthy,
* review and approve filter-to-template mapping without confusion,
* inspect the final compiled SQL before launch,
* resolve only the ambiguities that matter,
* reach a clear run/no-run decision,
* reopen the same context later without confusion.