Compare commits

..

134 Commits

Author SHA1 Message Date
a8f7147500 test now STANDARD tier 2026-02-26 18:38:26 +03:00
ce684bc5d1 update test data 2026-02-26 18:38:02 +03:00
484019e750 test semantic harden 2026-02-26 18:26:11 +03:00
4ff6d307f8 +ai update 2026-02-26 17:54:23 +03:00
f4612c0737 Improve dashboard LLM validation UX and report flow 2026-02-26 17:53:41 +03:00
5ec1254336 codex specify 2026-02-25 21:19:48 +03:00
b7d1ee2b71 feat(search): add grouped global results for tasks and reports 2026-02-25 21:09:42 +03:00
87285d8f0a feat(search): implement global navbar search for dashboards and datasets 2026-02-25 21:07:51 +03:00
04b01eadb5 fix(ui): use global environment context on datasets page 2026-02-25 20:59:24 +03:00
4d5b9e88dd fix(auth): defer environment context fetch until token is available 2026-02-25 20:58:14 +03:00
4bad4ab4e2 fix(logging): suppress per-request belief scope spam in API client 2026-02-25 20:52:12 +03:00
3801ca13d9 feat(env): add global production context and safety indicators 2026-02-25 20:46:00 +03:00
999c0c54df + git config 2026-02-25 20:27:29 +03:00
f9ac282596 feat: Implement recursive storage listing and directory browsing for backups, and add a migration option to fix cross-filters. 2026-02-25 20:01:33 +03:00
5d42a6b930 i18 cleanup 2026-02-25 18:31:50 +03:00
99f19ac305 { "verdict": "APPROVED", "rejection_reason": "NONE", "audit_details": { "target_invoked": true, "pre_conditions_tested": true, "post_conditions_tested": true, "test_data_used": true }, "feedback": "The test suite robustly verifies the
MigrationEngine
 contracts. It avoids Tautologies by cleanly substituting IdMappingService without mocking the engine itself. Cross-filter parsing asserts against hard-coded, predefined validation dictionaries (no Logic Mirroring). It successfully addresses @PRE negative cases (e.g. invalid zip paths, missing YAMLs) and rigorously validates @POST file transformations (e.g. in-place UUID substitutions and archive reconstruction)." }
2026-02-25 17:47:55 +03:00
590ba49ddb sync worked 2026-02-25 15:20:26 +03:00
2a5b225800 feat: Enhance ID mapping service robustness, add defensive guards, and expand migration engine and API testing. 2026-02-25 14:44:21 +03:00
33433c3173 ready for test 2026-02-25 13:35:09 +03:00
21e969a769 workflow agy update 2026-02-25 13:29:14 +03:00
783644c6ad tasks ready 2026-02-25 13:28:24 +03:00
d32d85556f +md 2026-02-25 10:34:30 +03:00
bc0367ab72 speckit update 2026-02-25 10:31:48 +03:00
1c362f4092 { "verdict": "APPROVED", "rejection_reason": "NONE", "audit_details": { "target_invoked": true, "pre_conditions_tested": true, "post_conditions_tested": true, "test_data_used": true }, "feedback": "Both test files have successfully passed the audit. The 'task_log_viewer.test.js' suite now correctly imports and mounts the real Svelte component using Test Library, fully eliminating the logic mirror/tautology issue. The 'test_logger.py' suite now properly implements negative tests for the @PRE constraint in 'belief_scope' and fully verifies all @POST effects triggered by 'configure_logger'." } 2026-02-24 21:55:13 +03:00
95ae9c6af1 semantic update 2026-02-24 21:08:12 +03:00
7a12ed0931 chore(gitignore): unignore frontend dashboards routes and track pages 2026-02-24 16:16:41 +03:00
e0c0dd3221 fix(validation): respect settings-bound provider and correct multimodal heuristic 2026-02-24 16:04:14 +03:00
5f6e9c0cc0 fix(llm-validation): accept stepfun multimodal models and return 422 on capability mismatch 2026-02-24 16:00:23 +03:00
4fd9d6b6d5 fix(llm): skip unsupported json_object mode for openrouter stepfun models 2026-02-24 14:22:08 +03:00
7e6bd56488 feat(assistant-chat): add animated thinking loader while waiting for response 2026-02-24 14:15:35 +03:00
5e3c213b92 fix(task-drawer): keep drawer above assistant dim overlay 2026-02-24 14:12:34 +03:00
37b75b5a5c fix(task-drawer): render as side column without modal overlay when opened from assistant 2026-02-24 14:09:34 +03:00
3d42a487f7 fix(assistant): resolve dashboard refs via LLM entities and remove deterministic parser fallback 2026-02-24 13:32:25 +03:00
2e93f5ca63 fix(assistant-chat): prevent stale history response from resetting selected conversation 2026-02-24 13:27:09 +03:00
286167b1d5 generate semantic clean up 2026-02-24 12:51:57 +03:00
7df7b4f98c feat(assistant): add multi-dialog UX, task-aware llm settings, and i18n cleanup 2026-02-23 23:45:01 +03:00
ab1c87ffba feat(assistant): add conversations list, infinite history scroll, and archived tab 2026-02-23 20:27:51 +03:00
40e6d8cd4c chat worked 2026-02-23 20:20:25 +03:00
18e96a58bc feat(assistant): implement spec 021 chat assistant flow with semantic contracts 2026-02-23 19:37:56 +03:00
83e4875097 Merge branch '001-unify-frontend-style' into master 2026-02-23 16:06:12 +03:00
e635bd7e5f Add Apache Superset OpenAPI documentation reference to ROOT.md 2026-02-23 16:04:42 +03:00
43dd97ecbf Новый экранчик для обзора дашей 2026-02-23 15:54:20 +03:00
0685f50ae7 Merge branch '020-task-reports-design' into master 2026-02-23 13:28:31 +03:00
d0ffc2f1df Finalize task-020 reports navigation and stability fixes 2026-02-23 13:28:30 +03:00
26880d2e09 semantic update 2026-02-23 13:15:48 +03:00
008b6d72c9 таски готовы 2026-02-23 10:18:56 +03:00
f0c85e4c03 Fix task API stability and Playwright runtime in Docker 2026-02-21 23:43:46 +03:00
6ffdf5f8a4 feat: restore legacy data and add typed task result views 2026-02-21 23:17:56 +03:00
0cf0ef25f1 db + docker 2026-02-20 20:47:39 +03:00
af74841765 semantic update 2026-02-20 10:41:15 +03:00
d7e4919d54 few shots update 2026-02-20 10:26:01 +03:00
fdcbe32dfa css refactor 2026-02-19 18:24:36 +03:00
4de5b22d57 +Svelte specific 2026-02-19 17:47:24 +03:00
c8029ed309 ai base 2026-02-19 17:43:45 +03:00
c2a4c8062a fix tax log 2026-02-19 16:05:59 +03:00
2c820e103a tests ready 2026-02-19 13:33:20 +03:00
c8b84b7bd7 Coder + fix workflow 2026-02-19 13:33:10 +03:00
fdb944f123 Test logic update 2026-02-19 12:44:31 +03:00
d29bc511a2 task panel 2026-02-19 09:43:01 +03:00
a3a9f0788d docs: amend constitution to v2.3.0 (tailwind css first principle) 2026-02-18 18:29:52 +03:00
77147dc95b refactor 2026-02-18 17:29:46 +03:00
026239e3bf fix 2026-02-15 11:11:30 +03:00
4a0273a604 измененные спеки таски 2026-02-10 15:53:38 +03:00
edb2dd5263 updated tasks 2026-02-10 15:04:43 +03:00
76b98fcf8f linter + новые таски 2026-02-10 12:53:01 +03:00
794cc55fe7 Таски готовы 2026-02-09 12:35:27 +03:00
235b0e3c9f semantic update 2026-02-08 22:53:54 +03:00
e6087bd3c1 таски готовы 2026-02-07 12:42:32 +03:00
0f16bab2b8 Похоже работает 2026-02-07 11:26:06 +03:00
7de96c17c4 feat(llm-plugin): switch to environment API for log retrieval
- Replace local backend.log reading with Superset API /log/ fetch
- Update DashboardValidationPlugin to use SupersetClient
- Filter logs by dashboard_id and last 24 hours
- Update spec FR-006 to reflect API usage
2026-02-06 17:57:25 +03:00
f018b97ed2 Semantic protocol update - add UX 2026-01-30 18:53:52 +03:00
72846aa835 tasks ux-reference 2026-01-30 13:35:03 +03:00
994c0c3e5d feat(speckit): integrate ux reference into workflows
Introduce a UX reference stage to ensure technical plans align with
user experience goals. Adds a new template, a generation step in the
specification workflow, and mandatory validation checks during
planning to prevent technical compromises from degrading the defined
user experience.
2026-01-30 12:31:19 +03:00
252a8601a9 Вроде работает 2026-01-30 11:10:16 +03:00
8044f85ea4 tasks and workflow updated 2026-01-29 10:06:28 +03:00
d4109e5a03 docs: amend constitution to v2.0.0 (delegate semantics to protocol + add async/testability principles) 2026-01-28 18:48:43 +03:00
b2bbd73439 tasks ready 2026-01-28 18:30:23 +03:00
0e0e26e2f7 semantic update 2026-01-28 16:57:19 +03:00
18b42f8dd0 semantic protocol condense + script update 2026-01-28 15:49:39 +03:00
e7b31accd6 tested 2026-01-27 23:49:19 +03:00
d3c3a80ed2 Передаем на тест 2026-01-27 16:32:08 +03:00
cc244c2d86 tasks ready 2026-01-27 13:26:06 +03:00
d10c23e658 Обновил gitignore - убрал логи 2026-01-26 22:15:17 +03:00
1042b35d1b Закончили редизайн, обновили интерфейс бэкапа 2026-01-26 22:12:35 +03:00
16ffeb1ed6 Выполнено, передано на тестирование 2026-01-26 21:17:05 +03:00
da34deac02 tasks ready 2026-01-26 20:58:38 +03:00
51e9ee3fcc semantic update 2026-01-26 11:57:36 +03:00
edf9286071 Файловое хранилище готово 2026-01-26 11:08:18 +03:00
a542e7d2df Передаем на тест 2026-01-25 18:33:00 +03:00
a863807cf2 tasks ready 2026-01-24 16:21:43 +03:00
e2bc68683f Update .gitignore 2026-01-24 11:26:19 +03:00
43cb82697b Update backup scheduler task status 2026-01-24 11:26:05 +03:00
4ba28cf93e semantic cleanup 2026-01-23 21:58:32 +03:00
343f2e29f5 Мультиязночность + причесывание css 2026-01-23 17:53:46 +03:00
c9a53578fd tasks ready 2026-01-23 14:56:05 +03:00
07ec2d9797 Работает создание коммитов и перенос в новый enviroment 2026-01-23 13:57:44 +03:00
e9d3f3c827 tasks ready 2026-01-22 23:59:16 +03:00
26ba015b75 +gitignore 2026-01-22 23:25:29 +03:00
49129d3e86 fix error 2026-01-22 23:18:48 +03:00
d99a13d91f refactor complete 2026-01-22 17:37:17 +03:00
203ce446f4 ашч 2026-01-21 14:00:48 +03:00
c96d50a3f4 fix(backend): standardize superset client init and auth
- Update plugins (debug, mapper, search) to explicitly map environment config to SupersetConfig
- Add authenticate method to SupersetClient for explicit session management
- Add get_environment method to ConfigManager
- Fix navbar dropdown hover stability in frontend with invisible bridge
2026-01-20 19:31:17 +03:00
3bbe320949 TaskLog fix 2026-01-19 17:10:43 +03:00
2d2435642d bug fixs 2026-01-19 00:07:06 +03:00
ec8d67c956 bug fixes 2026-01-18 23:21:00 +03:00
76baeb1038 semantic markup update 2026-01-18 21:29:54 +03:00
11c59fb420 semantic checker script update 2026-01-13 17:33:57 +03:00
b2529973eb constitution update 2026-01-13 15:29:42 +03:00
ae1d630ad6 semantics update 2026-01-13 09:11:27 +03:00
9a9c5879e6 tasks.md status 2026-01-12 12:35:45 +03:00
696aac32e7 1st iter 2026-01-12 12:33:51 +03:00
7a9b1a190a tasks ready 2026-01-07 18:59:49 +03:00
a3dc1fb2b9 docs: amend constitution to v1.6.0 (add 'Everything is a Plugin' principle) and refactor 010 plan 2026-01-07 18:36:38 +03:00
297b29986d Product Manager role 2026-01-07 11:39:44 +03:00
4c6fc8256d project map script | semantic parcer 2026-01-01 16:58:21 +03:00
a747a163c8 backup worked 2025-12-30 22:02:51 +03:00
fce0941e98 docs ready 2025-12-30 21:30:37 +03:00
45c077b928 +api rework 2025-12-30 20:08:48 +03:00
9ed3a5992d cleaned 2025-12-30 18:20:40 +03:00
a032fe8457 Password promt 2025-12-30 17:21:12 +03:00
4c9d554432 TaskManager refactor 2025-12-29 10:13:37 +03:00
6962a78112 mappings+migrate 2025-12-27 10:16:41 +03:00
3d75a21127 tech_lead / coder 2roles 2025-12-27 08:02:59 +03:00
07914c8728 semantic add 2025-12-27 07:14:08 +03:00
cddc259b76 new loggers logic in constitution 2025-12-27 06:51:28 +03:00
dcbf0a7d7f tasks ready 2025-12-27 06:37:03 +03:00
65f61c1f80 Merge branch '001-migration-ui-redesign' into master 2025-12-27 05:58:35 +03:00
cb7386f274 superset_tool logger rework 2025-12-27 05:53:30 +03:00
83e34e1799 feat(logging): implement configurable belief state logging
- Add LoggingConfig model and logging field to GlobalSettings
- Implement belief_scope context manager for structured logging
- Add configure_logger for dynamic level and file rotation settings
- Add logging configuration UI to Settings page
- Update ConfigManager to apply logging settings on initialization and updates
2025-12-27 05:39:33 +03:00
d197303b9f 006 plan ready 2025-12-26 19:36:49 +03:00
a43f8fb021 001-migration-ui-redesign (#3)
Reviewed-on: #3
2025-12-26 18:17:58 +03:00
4aa01b6470 Merge branch 'migration' into 001-migration-ui-redesign 2025-12-26 18:16:24 +03:00
35b423979d spec rules 2025-12-25 22:28:42 +03:00
2ffc3cc68f feat(migration): implement interactive mapping resolution workflow
- Add SQLite database integration for environments and mappings
- Update TaskManager to support pausing tasks (AWAITING_MAPPING)
- Modify MigrationPlugin to detect missing mappings and wait for resolution
- Add frontend UI for handling missing mappings interactively
- Create dedicated migration routes and API endpoints
- Update .gitignore and project documentation
2025-12-25 22:27:29 +03:00
400 changed files with 69714 additions and 136395 deletions

View File

@@ -6,7 +6,7 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
**OBJECTIVE:** Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.
**INPUT:**
1. SOURCE CODE (with GRACE-Poly `[DEF]` Contract: `@PRE`, `@POST`, `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`).
1. SOURCE CODE (with GRACE-Poly `[DEF]` Contract: `@PRE`, `@POST`, `@TEST_DATA`).
2. GENERATED TEST CODE.
### I. CRITICAL ANTI-PATTERNS (REJECT IMMEDIATELY IF FOUND):
@@ -17,7 +17,7 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
2. **The Logic Mirror (Echoing):**
- *Definition:* The test re-implements the exact same algorithmic logic found in the source code to calculate the `expected_result`. If the original logic is flawed, the test will falsely pass.
- *Rule:* Tests must assert against **static, predefined outcomes** (from `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT` or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.
- *Rule:* Tests must assert against **static, predefined outcomes** (from `@TEST_DATA` or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.
3. **The "Happy Path" Illusion:**
- *Definition:* The test suite only checks successful executions but ignores the `@PRE` conditions (Negative Testing).
@@ -26,78 +26,26 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
4. **Missing Post-Condition Verification:**
- *Definition:* The test calls the function but only checks the return value, ignoring `@SIDE_EFFECT` or `@POST` state changes (e.g., failing to verify that a DB call was made or a Store was updated).
5. **Missing Edge Case Coverage:**
- *Definition:* The test suite ignores `@TEST_EDGE` scenarios defined in the contract.
- *Rule:* Every `@TEST_EDGE` in the source contract MUST have a corresponding test case.
6. **Missing Invariant Verification:**
- *Definition:* The test suite does not verify `@TEST_INVARIANT` conditions.
- *Rule:* Every `@TEST_INVARIANT` MUST be verified by at least one test that attempts to break it.
7. **Missing UX State Testing (Svelte Components):**
- *Definition:* For Svelte components with `@UX_STATE`, the test suite does not verify state transitions.
- *Rule:* Every `@UX_STATE` transition MUST have a test verifying the visual/behavioral change.
- *Check:* `@UX_FEEDBACK` mechanisms (toast, shake, color) must be tested.
- *Check:* `@UX_RECOVERY` mechanisms (retry, clear input) must be tested.
### II. SEMANTIC PROTOCOL COMPLIANCE
Verify the test file follows GRACE-Poly semantics:
1. **Anchor Integrity:**
- Test file MUST start with `[DEF:__tests__/test_name:Module]`
- Test file MUST end with `[/DEF:__tests__/test_name:Module]`
2. **Required Tags:**
- `@RELATION: VERIFIES -> <path_to_source>` must be present
- `@PURPOSE:` must describe what is being tested
3. **TIER Alignment:**
- If source is `@TIER: CRITICAL`, test MUST cover all `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`
- If source is `@TIER: STANDARD`, test MUST cover `@PRE` and `@POST`
- If source is `@TIER: TRIVIAL`, basic smoke test is acceptable
### III. AUDIT CHECKLIST
### II. AUDIT CHECKLIST
Evaluate the test code against these criteria:
1. **Target Invocation:** Does the test actually import and call the function/component declared in the `@RELATION: VERIFIES` tag?
2. **Contract Alignment:** Does the test suite cover 100% of the `@PRE` (negative tests) and `@POST` (assertions) conditions from the source contract?
3. **Test Contract Compliance:** Does the test follow the interface defined in `@TEST_CONTRACT`?
4. **Data Usage:** Does the test use the exact scenarios defined in `@TEST_FIXTURE`?
5. **Edge Coverage:** Are all `@TEST_EDGE` scenarios tested?
6. **Invariant Coverage:** Are all `@TEST_INVARIANT` conditions verified?
7. **UX Coverage (if applicable):** Are all `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY` tested?
8. **Mocking Sanity:** Are external dependencies mocked correctly WITHOUT mocking the system under test itself?
9. **Semantic Anchor:** Does the test file have proper `[DEF]` and `[/DEF]` anchors?
3. **Data Usage:** Does the test use the exact scenarios defined in `@TEST_DATA`?
4. **Mocking Sanity:** Are external dependencies mocked correctly WITHOUT mocking the system under test itself?
### IV. OUTPUT FORMAT
### III. OUTPUT FORMAT
You MUST respond strictly in the following JSON format. Do not add markdown blocks outside the JSON.
{
"verdict": "APPROVED" | "REJECTED",
"rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "MISSING_EDGES" | "MISSING_INVARIANTS" | "MISSING_UX_TESTS" | "SEMANTIC_VIOLATION" | "NONE",
"rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "NONE",
"audit_details": {
"target_invoked": true/false,
"pre_conditions_tested": true/false,
"post_conditions_tested": true/false,
"test_fixture_used": true/false,
"edges_covered": true/false,
"invariants_verified": true/false,
"ux_states_tested": true/false,
"semantic_anchors_present": true/false
},
"coverage_summary": {
"total_edges": number,
"edges_tested": number,
"total_invariants": number,
"invariants_tested": number,
"total_ux_states": number,
"ux_states_tested": number
},
"tier_compliance": {
"source_tier": "CRITICAL" | "STANDARD" | "TRIVIAL",
"meets_tier_requirements": true/false
"test_data_used": true/false
},
"feedback": "Strict, actionable feedback for the test generator agent. Explain exactly which anti-pattern was detected and how to fix it."
}

View File

@@ -1,4 +1,4 @@
---
description: USE SEMANTIC
---
Прочитай .ai/standards/semantics.md. ОБЯЗАТЕЛЬНО используй его при разработке
Прочитай .specify/memory/semantics.md (или .ai/standards/semantics.md, если не найден). ОБЯЗАТЕЛЬНО используй его при разработке

View File

@@ -63,7 +63,6 @@ Load only the minimal necessary context from each artifact:
**From constitution:**
- Load `.ai/standards/constitution.md` for principle validation
- Load `.ai/standards/semantics.md` for technical standard validation
### 3. Build Semantic Models

View File

@@ -20,7 +20,7 @@ Analyze test failure reports, identify root causes, and fix implementation issue
1. **USE CODER MODE**: Always switch to `coder` mode for code fixes
2. **SEMANTIC PROTOCOL**: Never remove semantic annotations ([DEF], @TAGS). Only update code logic.
3. **TEST DATA**: If tests use @TEST_ fixtures, preserve them when fixing
3. **TEST DATA**: If tests use @TEST_DATA fixtures, preserve them when fixing
4. **NO DELETION**: Never delete existing tests or semantic annotations
5. **REPORT FIRST**: Always write a fix report before making changes

View File

@@ -53,15 +53,6 @@ You **MUST** consider the user input before proceeding (if not empty).
- **IF EXISTS**: Read research.md for technical decisions and constraints
- **IF EXISTS**: Read quickstart.md for integration scenarios
3. Load and analyze the implementation context:
- **REQUIRED**: Read `.ai/standards/semantics.md` for strict coding standards and contract requirements
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
- **IF EXISTS**: Read data-model.md for entities and relationships
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
- **IF EXISTS**: Read research.md for technical decisions and constraints
- **IF EXISTS**: Read quickstart.md for integration scenarios
4. **Project Setup Verification**:
- **REQUIRED**: Create/verify ignore files based on actual project setup:
@@ -120,13 +111,7 @@ You **MUST** consider the user input before proceeding (if not empty).
- **Validation checkpoints**: Verify each phase completion before proceeding
7. Implementation execution rules:
- **Strict Adherence**: Apply `.ai/standards/semantics.md` rules:
- Every file MUST start with a `[DEF:id:Type]` header and end with a closing `[/DEF:id:Type]` anchor.
- Include `@TIER` and define contracts (`@PRE`, `@POST`).
- For Svelte components, use `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and explicitly declare reactivity with `@UX_REATIVITY: State: $state, Derived: $derived`.
- **Molecular Topology Logging**: Use prefixes `[EXPLORE]`, `[REASON]`, `[REFLECT]` in logs to trace logic.
- **CRITICAL Contracts**: If a task description contains a contract summary (e.g., `CRITICAL: PRE: ..., POST: ...`), these constraints are **MANDATORY** and must be strictly implemented in the code using guards/assertions (if applicable per protocol).
- **Setup first**: Initialize project structure, dependencies, configuration
- **Setup first**: Initialize project structure, dependencies, configuration
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
- **Core development**: Implement models, services, CLI commands, endpoints
- **Integration work**: Database connections, middleware, logging, external services

View File

@@ -22,7 +22,7 @@ You **MUST** consider the user input before proceeding (if not empty).
1. **Setup**: Run `.specify/scripts/bash/setup-plan.sh --json` from repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
2. **Load context**: Read `.ai/ROOT.md` and `.ai/PROJECT_MAP.md` to understand the project structure and navigation. Then read required standards: `.ai/standards/constitution.md` and `.ai/standards/semantics.md`. Load IMPL_PLAN template.
2. **Load context**: Read FEATURE_SPEC and `.ai/standards/constitution.md`. Load IMPL_PLAN template (already copied).
3. **Execute plan workflow**: Follow the structure in IMPL_PLAN template to:
- Fill Technical Context (mark unknowns as "NEEDS CLARIFICATION")
@@ -64,30 +64,16 @@ You **MUST** consider the user input before proceeding (if not empty).
**Prerequisites:** `research.md` complete
0. **Validate Design against UX Reference**:
- Check if the proposed architecture supports the latency, interactivity, and flow defined in `ux_reference.md`.
- **Linkage**: Ensure key UI states from `ux_reference.md` map to Component Contracts (`@UX_STATE`).
- **CRITICAL**: If the technical plan compromises the UX (e.g. "We can't do real-time validation"), you **MUST STOP** and warn the user.
1. **Extract entities from feature spec** → `data-model.md`:
- Entity name, fields, relationships, validation rules.
- Entity name, fields, relationships
- Validation rules from requirements
- State transitions if applicable
2. **Design & Verify Contracts (Semantic Protocol)**:
- **Drafting**: Define `[DEF:id:Type]` Headers, Contracts, and closing `[/DEF:id:Type]` for all new modules based on `.ai/standards/semantics.md`.
- **TIER Classification**: Explicitly assign `@TIER: [CRITICAL|STANDARD|TRIVIAL]` to each module.
- **CRITICAL Requirements**: For all CRITICAL modules, define full `@PRE`, `@POST`, and (if UI) `@UX_STATE` contracts. **MUST** also define testing contracts: `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT`.
- **Self-Review**:
- *Completeness*: Do `@PRE`/`@POST` cover edge cases identified in Research? Are test contracts present for CRITICAL?
- *Connectivity*: Do `@RELATION` tags form a coherent graph?
- *Compliance*: Does syntax match `[DEF:id:Type]` exactly and is it closed with `[/DEF:id:Type]`?
- **Output**: Write verified contracts to `contracts/modules.md`.
3. **Simulate Contract Usage**:
- Trace one key user scenario through the defined contracts to ensure data flow continuity.
- If a contract interface mismatch is found, fix it immediately.
4. **Generate API contracts**:
- Output OpenAPI/GraphQL schema to `/contracts/` for backend-frontend sync.
2. **Define interface contracts** (if project has external interfaces) → `/contracts/`:
- Identify what interfaces the project exposes to users or other systems
- Document the contract format appropriate for the project type
- Examples: public APIs for libraries, command schemas for CLI tools, endpoints for web services, grammars for parsers, UI contracts for applications
- Skip if project is purely internal (build scripts, one-off tools, etc.)
3. **Agent context update**:
- Run `.specify/scripts/bash/update-agent-context.sh agy`

View File

@@ -24,7 +24,7 @@ You **MUST** consider the user input before proceeding (if not empty).
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
2. **Load design documents**: Read from FEATURE_DIR:
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities), ux_reference.md (experience source of truth)
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities)
- **Optional**: data-model.md (entities), contracts/ (interface contracts), research.md (decisions), quickstart.md (test scenarios)
- Note: Not all projects have all documents. Generate tasks based on what's available.
@@ -70,12 +70,6 @@ The tasks.md should be immediately executable - each task must be specific enoug
**Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach.
### UX Preservation (CRITICAL)
- **Source of Truth**: `ux_reference.md` is the absolute standard for the "feel" of the feature.
- **Violation Warning**: If any task would inherently violate the UX (e.g. "Remove progress bar to simplify code"), you **MUST** flag this to the user immediately.
- **Verification Task**: You **MUST** add a specific task at the end of each User Story phase: `- [ ] Txxx [USx] Verify implementation matches ux_reference.md (Happy Path & Errors)`
### Checklist Format (REQUIRED)
Every task MUST strictly follow this format:
@@ -119,12 +113,9 @@ Every task MUST strictly follow this format:
- If tests requested: Tests specific to that story
- Mark story dependencies (most stories should be independent)
2. **From Contracts (CRITICAL TIER)**:
- Identify components marked as `@TIER: CRITICAL` in `contracts/modules.md`.
- For these components, **MUST** append the summary of `@PRE`, `@POST`, `@UX_STATE`, and test contracts (`@TEST_FIXTURE`, `@TEST_EDGE`) directly to the task description.
- Example: `- [ ] T005 [P] [US1] Implement Auth (CRITICAL: PRE: token exists, POST: returns User, TESTS: 2 edges) in src/auth.py`
- Map each contract/endpoint → to the user story it serves
- If tests requested: Each contract → contract test task [P] before implementation in that story's phase
2. **From Contracts**:
- Map each interface contract → to the user story it serves
- If tests requested: Each interface contract → contract test task [P] before implementation in that story's phase
3. **From Data Model**:
- Map each entity to the user story(ies) that need it

View File

@@ -249,7 +249,6 @@ component/__tests__/Component.test.js
# [DEF:__tests__/test_module:Module]
# @RELATION: VERIFIES -> ../module.py
# @PURPOSE: Contract testing for module
# [/DEF:__tests__/test_module:Module]
```
---

File diff suppressed because it is too large Load Diff

View File

@@ -26,7 +26,7 @@
4. **ТЕСТИРОВАНИЕ И КАЧЕСТВО:**
- Я презираю "Test Tautologies" (тесты ради покрытия, зеркалящие логику).
- Тесты должны быть Contract-Driven. Если есть `@PRE`, я ожидаю тест на его нарушение.
- Тесты обязаны использовать `@TEST_` из контрактов.
- Тесты обязаны использовать `@TEST_DATA` из контрактов.
5. **ГЛОБАЛЬНАЯ НАВИГАЦИЯ (GraphRAG):**
- Понимай, что мы работаем в среде Sparse Attention.

File diff suppressed because it is too large Load Diff

View File

@@ -1,4 +1,4 @@
#[DEF:BackendRouteShot:Module]
# [DEF:BackendRouteShot:Module]
# @TIER: STANDARD
# @SEMANTICS: Route, Task, API, Async
# @PURPOSE: Reference implementation of a task-based route using GRACE-Poly.
@@ -9,66 +9,53 @@
from typing import Dict, Any
from fastapi import APIRouter, Depends, HTTPException, status
from pydantic import BaseModel
# GRACE: Правильный импорт глобального логгера и scope
from ...core.logger import logger, belief_scope
from ...core.logger import belief_scope
from ...core.task_manager import TaskManager, Task
from ...core.config_manager import ConfigManager
from ...dependencies import get_task_manager, get_config_manager, get_current_user
router = APIRouter()
# [DEF:CreateTaskRequest:Class]
# @PURPOSE: DTO for task creation payload.
class CreateTaskRequest(BaseModel):
plugin_id: str
params: Dict[str, Any]
# [/DEF:CreateTaskRequest:Class]
@router.post("/tasks", response_model=Task, status_code=status.HTTP_201_CREATED)
# [DEF:create_task:Function]
# @PURPOSE: Create and start a new task using TaskManager. Non-blocking.
# @DATA_CONTRACT: Input -> CreateTaskRequest, Output -> Task
# @PARAM: request (CreateTaskRequest) - Plugin and params.
# @PARAM: task_manager (TaskManager) - Async task executor.
# @PRE: plugin_id must match a registered plugin.
# @POST: A new task is spawned; Task object returned immediately.
# @SIDE_EFFECT: Writes to DB, Triggers background worker.
#
# @UX_STATE: Success -> 201 Created
# @UX_STATE: Error(Validation) -> 400 Bad Request
# @UX_STATE: Error(System) -> 500 Internal Server Error
@router.post("/tasks", response_model=Task, status_code=status.HTTP_201_CREATED)
# @POST: A new task is spawned; Task ID returned immediately.
# @SIDE_EFFECT: Writes to DB, Trigger background worker.
async def create_task(
request: CreateTaskRequest,
task_manager: TaskManager = Depends(get_task_manager),
config: ConfigManager = Depends(get_config_manager),
current_user = Depends(get_current_user)
):
# GRACE: Открываем семантическую транзакцию
# Context Logging
with belief_scope("create_task"):
try:
# GRACE: [REASON] - Фиксируем начало дедуктивной цепочки
logger.reason("Resolving configuration and spawning task", extra={"plugin_id": request.plugin_id})
# 1. Action: Configuration Resolution
timeout = config.get("TASKS_DEFAULT_TIMEOUT", 3600)
# 2. Action: Spawn async task
# @RELATION: CALLS -> task_manager.create_task
task = await task_manager.create_task(
plugin_id=request.plugin_id,
params={**request.params, "timeout": timeout}
)
# GRACE:[REFLECT] - Подтверждаем выполнение @POST перед выходом
logger.reflect("Task spawned successfully", extra={"task_id": task.id})
return task
except ValueError as e:
# GRACE: [EXPLORE] - Обработка ожидаемого отклонения
logger.explore("Domain validation error during task creation", exc_info=e)
# 3. Recovery: Domain logic error mapping
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=str(e)
)
except Exception as e:
# GRACE: [EXPLORE] - Обработка критического сбоя
logger.explore("Internal Task Spawning Error", exc_info=e)
# @UX_STATE: Error feedback -> 500 Internal Error
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Internal Task Spawning Error"

View File

@@ -4,27 +4,18 @@
# @PURPOSE: Core banking transaction processor with ACID guarantees.
# @LAYER: Domain (Core)
# @RELATION: DEPENDS_ON -> [DEF:Infra:PostgresDB]
#
# @RELATION: DEPENDS_ON -> [DEF:Infra:AuditLog]
# @INVARIANT: Total system balance must remain constant (Double-Entry Bookkeeping).
# @INVARIANT: Negative transfers are strictly forbidden.
# --- Test Specifications ---
# @TEST_CONTRACT: TransferRequestDTO -> TransferResultDTO
# @TEST_SCENARIO: sufficient_funds -> Returns COMPLETED, balances updated.
# @TEST_FIXTURE: sufficient_funds -> file:./__tests__/fixtures/transfers.json#happy_path
# @TEST_EDGE: insufficient_funds -> Throws BusinessRuleViolation("INSUFFICIENT_FUNDS").
# @TEST_EDGE: negative_amount -> Throws BusinessRuleViolation("Transfer amount must be positive.").
# @TEST_EDGE: concurrency_conflict -> Throws DBTransactionError.
#
# @TEST_INVARIANT: total_balance_constant -> VERIFIED_BY: [sufficient_funds, concurrency_conflict]
# @TEST_INVARIANT: negative_transfer_forbidden -> VERIFIED_BY: [negative_amount]
# @TEST_DATA: sufficient_funds -> {"from": "acc_A", "to": "acc_B", "amt": 100.00}
# @TEST_DATA: insufficient_funds -> {"from": "acc_empty", "to": "acc_B", "amt": 1000.00}
# @TEST_DATA: concurrency_lock -> {./fixtures/transactions.json#race_condition}
from decimal import Decimal
from typing import NamedTuple
# GRACE: Импорт глобального логгера с семантическими методами
from ...core.logger import logger, belief_scope
from ...core.logger import belief_scope
from ...core.db import atomic_transaction, get_balance, update_balance
from ...core.audit import log_audit_trail
from ...core.exceptions import BusinessRuleViolation
class TransferResult(NamedTuple):
@@ -34,54 +25,55 @@ class TransferResult(NamedTuple):
# [DEF:execute_transfer:Function]
# @PURPOSE: Atomically move funds between accounts with audit trails.
# @DATA_CONTRACT: Input -> (sender_id: str, receiver_id: str, amount: Decimal), Output -> TransferResult
# @PARAM: sender_id (str) - Source account.
# @PARAM: receiver_id (str) - Destination account.
# @PARAM: amount (Decimal) - Positive amount to transfer.
# @PRE: amount > 0; sender != receiver; sender_balance >= amount.
# @POST: sender_balance -= amount; receiver_balance += amount; Audit Record Created.
# @SIDE_EFFECT: Database mutation (Rows locked), Audit IO.
#
# @UX_STATE: Success -> Returns 200 OK + Transaction Receipt.
# @UX_STATE: Error(LowBalance) -> 422 Unprocessable -> UI shows "Top-up needed" modal.
# @UX_STATE: Error(System) -> 500 Internal -> UI shows "Retry later" toast.
def execute_transfer(sender_id: str, receiver_id: str, amount: Decimal) -> TransferResult:
# Guard: Input Validation (Вне belief_scope, так как это trivial проверка)
# Guard: Input Validation
if amount <= Decimal("0.00"):
raise BusinessRuleViolation("Transfer amount must be positive.")
if sender_id == receiver_id:
raise BusinessRuleViolation("Cannot transfer to self.")
# GRACE: Используем strict Context Manager без 'as context'
with belief_scope("execute_transfer"):
# GRACE: [REASON] - Жесткая дедукция, начало алгоритма
logger.reason("Initiating transfer", extra={"from": sender_id, "to": receiver_id, "amount": amount})
with belief_scope("execute_transfer") as context:
context.logger.info("Initiating transfer", data={"from": sender_id, "to": receiver_id})
try:
# 1. Action: Atomic DB Transaction
# @RELATION: CALLS -> atomic_transaction
with atomic_transaction():
# Guard: State Validation (Strict)
current_balance = get_balance(sender_id, for_update=True)
if current_balance < amount:
# GRACE: [EXPLORE] - Отклонение от Happy Path (фолбэк/ошибка)
logger.explore("Insufficient funds validation hit", extra={"balance": current_balance})
# @UX_FEEDBACK: Triggers specific UI flow for insufficient funds
context.logger.warn("Insufficient funds", data={"balance": current_balance})
raise BusinessRuleViolation("INSUFFICIENT_FUNDS")
# Mutation
# 2. Action: Mutation
new_src_bal = update_balance(sender_id, -amount)
new_dst_bal = update_balance(receiver_id, +amount)
# Audit
tx_id = log_audit_trail("TRANSFER", sender_id, receiver_id, amount)
# GRACE:[REFLECT] - Сверка с @POST перед возвратом
logger.reflect("Transfer committed successfully", extra={"tx_id": tx_id, "new_balance": new_src_bal})
# 3. Action: Audit
tx_id = context.audit.log_transfer(sender_id, receiver_id, amount)
context.logger.info("Transfer committed", data={"tx_id": tx_id})
return TransferResult(tx_id, "COMPLETED", new_src_bal)
except BusinessRuleViolation as e:
# Explicit re-raise for UI mapping
# Logic: Explicit re-raise for UI mapping
raise e
except Exception as e:
# GRACE: [EXPLORE] - Неожиданный сбой
logger.explore("Critical Transfer Failure", exc_info=e)
# Logic: Catch-all safety net
context.logger.error("Critical Transfer Failure", error=e)
raise RuntimeError("TRANSACTION_ABORTED") from e
#[/DEF:execute_transfer:Function]
# [/DEF:execute_transfer:Function]
# [/DEF:TransactionCore:Module]

View File

@@ -1,79 +1,59 @@
<!-- [DEF:FrontendComponentShot:Component] -->
<!--
/**
* @TIER: CRITICAL
* @SEMANTICS: Task, Button, Action, UX
* @PURPOSE: Action button to spawn a new task with full UX feedback cycle.
* @LAYER: UI (Presentation)
* @RELATION: CALLS -> postApi
*
* @INVARIANT: Must prevent double-submission while loading.
* @INVARIANT: Loading state must always terminate (no infinite spinner).
* @INVARIANT: User must receive feedback on both success and failure.
*
* @UX_REACTIVITY: Props -> $props(), LocalState -> $state(isLoading).
*
* @TEST_CONTRACT: ComponentState ->
* {
* required_fields: { isLoading: bool },
* invariants:[
* "isLoading=true implies button.disabled=true",
* "isLoading=true implies aria-busy=true"
* ]
* }
*
* @TEST_FIXTURE: idle_state -> { isLoading: false }
* @TEST_FIXTURE: successful_response -> { task_id: "task_123" }
*
* @TEST_EDGE: api_failure -> raises Error("Network")
* @TEST_EDGE: empty_response -> {}
* @TEST_EDGE: rapid_double_click -> special: concurrent_click
*
* @TEST_INVARIANT: prevent_double_submission -> VERIFIED_BY:[rapid_double_click]
* @TEST_INVARIANT: feedback_always_emitted -> VERIFIED_BY:[successful_response, api_failure]
*
* @UX_STATE: Idle -> Button enabled, primary color, no spinner.
* @UX_STATE: Loading -> Button disabled, spinner visible, aria-busy=true.
* @UX_STATE: Success -> Toast success displayed.
* @UX_STATE: Error -> Toast error displayed.
*
* @UX_FEEDBACK: toast.success, toast.error
*
* @UX_TEST: Idle -> {click: spawnTask, expected: isLoading=true}
* @UX_TEST: Loading -> {double_click: ignored, expected: single_api_call}
*/
-->
<!-- /**
* @TIER: CRITICAL
* @SEMANTICS: Task, Button, Action, UX
* @PURPOSE: Action button to spawn a new task with full UX feedback cycle.
* @LAYER: UI (Presentation)
* @RELATION: CALLS -> postApi
* @INVARIANT: Must prevent double-submission while loading.
*
* @TEST_DATA: idle_state -> {"isLoading": false}
* @TEST_DATA: loading_state -> {"isLoading": true}
*
* @UX_STATE: Idle -> Button enabled, primary color.
* @UX_STATE: Loading -> Button disabled, spinner visible.
* @UX_STATE: Error -> Toast notification triggers.
*
* @UX_FEEDBACK: Toast success/error.
* @UX_TEST: Idle -> {click: spawnTask, expected: isLoading=true}
* @UX_TEST: Success -> {api_resolve: 200, expected: toast.success called}
*/
-->
<script>
import { postApi } from "$lib/api.js";
import { t } from "$lib/i18n";
import { toast } from "$lib/stores/toast";
// GRACE Svelte 5 Runes
let { plugin_id = "", params = {} } = $props();
let isLoading = $state(false);
export let plugin_id = "";
export let params = {};
let isLoading = false;
// [DEF:spawnTask:Function]
/**
* @PURPOSE: Execute task creation request and emit user feedback.
* @PRE: plugin_id is resolved and request params are serializable.
* @POST: isLoading is reset and user receives success/error feedback.
* @purpose Execute task creation request and emit user feedback.
* @pre plugin_id is resolved and request params are serializable.
* @post isLoading is reset and user receives success/error feedback.
*/
async function spawnTask() {
isLoading = true;
console.info("[spawnTask][REASON] Spawning task...", { plugin_id });
console.log("[FrontendComponentShot][Loading] Spawning task...");
try {
// 1. Action: API Call
const response = await postApi("/api/tasks", { plugin_id, params });
const response = await postApi("/api/tasks", {
plugin_id,
params
});
// 2. Feedback: Success validation
// 2. Feedback: Success
if (response.task_id) {
console.info("[spawnTask][REFLECT] Task created.", { task_id: response.task_id });
console.log("[FrontendComponentShot][Success] Task created.");
toast.success($t.tasks.spawned_success);
}
} catch (error) {
// 3. Recovery: Error handling & fallback logic
console.error("[spawnTask][EXPLORE] Failed to spawn task. Notifying user.", { error });
// 3. Recovery: User notification
console.log("[FrontendComponentShot][Error] Failed:", error);
toast.error(`${$t.errors.task_failed}: ${error.message}`);
} finally {
isLoading = false;
@@ -83,7 +63,7 @@
</script>
<button
onclick={spawnTask}
on:click={spawnTask}
disabled={isLoading}
class="btn-primary flex items-center gap-2"
aria-busy={isLoading}

View File

@@ -9,19 +9,15 @@
from typing import Dict, Any, Optional
from ..core.plugin_base import PluginBase
from ..core.task_manager.context import TaskContext
# GRACE: Обязательный импорт семантического логгера
from ..core.logger import logger, belief_scope
# [DEF:ExamplePlugin:Class]
# @PURPOSE: A sample plugin to demonstrate execution context and logging.
class ExamplePlugin(PluginBase):
@property
def id(self) -> str:
return "example-plugin"
#[DEF:get_schema:Function]
# [DEF:get_schema:Function]
# @PURPOSE: Defines input validation schema.
# @DATA_CONTRACT: Input -> None, Output -> Dict (JSON Schema draft 7)
# @POST: Returns dict compliant with JSON Schema draft 7.
def get_schema(self) -> Dict[str, Any]:
return {
"type": "object",
@@ -33,43 +29,36 @@ class ExamplePlugin(PluginBase):
},
"required": ["message"],
}
#[/DEF:get_schema:Function]
# [/DEF:get_schema:Function]
# [DEF:execute:Function]
# @PURPOSE: Core plugin logic with structured logging and scope isolation.
# @DATA_CONTRACT: Input -> (params: Dict, context: Optional[TaskContext]), Output -> None
# @PRE: params must be validated against get_schema() before calling.
# @POST: Plugin payload is processed; progress is reported if context exists.
# @SIDE_EFFECT: Emits logs to centralized system and TaskContext.
async def execute(self, params: Dict, context: Optional[TaskContext] = None):
message = params.get("message", "Fallback")
# @PARAM: params (Dict) - Validated input parameters.
# @PARAM: context (TaskContext) - Execution tools (log, progress).
# @SIDE_EFFECT: Emits logs to centralized system.
async def execute(self, params: Dict, context: Optional = None):
message = params
# GRACE: Изоляция мыслей ИИ в Thread-Local scope
with belief_scope("example_plugin_exec"):
# 1. Action: System-level tracing (Rule VI)
with belief_scope("example_plugin_exec") as b_scope:
if context:
# Task Logs: Пишем в пользовательский контекст выполнения задачи
# @RELATION: BINDS_TO -> context.logger
log = context.logger.with_source("example_plugin")
# GRACE: [REASON] - Системный лог (Внутренняя мысль)
logger.reason("TaskContext provided. Binding task logger.", extra={"msg": message})
b_scope.logger.info("Using provided TaskContext") # System log
log.info("Starting execution", data={"msg": message}) # Task log
# Task Logs: Бизнес-логи (Уйдут в БД/Вебсокет пользователю)
log.info("Starting execution", extra={"msg": message})
# 2. Action: Progress Reporting
log.progress("Processing...", percent=50)
# 3. Action: Finalize
log.info("Execution completed.")
# GRACE: [REFLECT] - Сверка успешного выхода
logger.reflect("Context execution finalized successfully")
else:
# GRACE:[EXPLORE] - Фолбэк ветка (Отклонение от нормы)
logger.explore("No TaskContext provided. Running standalone.")
# Standalone Fallback
print(f"Standalone execution: {message}")
# GRACE: [REFLECT] - Сверка выхода фолбэка
logger.reflect("Standalone execution finalized")
# Standalone Fallback: Замыкаемся на системный scope
b_scope.logger.warning("No TaskContext provided. Running standalone.")
b_scope.logger.info("Standalone execution", data={"msg": message})
print(f"Standalone: {message}")
# [/DEF:execute:Function]
#[/DEF:ExamplePlugin:Class]
# [/DEF:PluginExampleShot:Module]

View File

@@ -8,7 +8,7 @@
## 1. CORE PRINCIPLES
### I. Semantic Protocol Compliance
* **Ref:** `[DEF:Std:Semantics]` (`ai/standards/semantic.md`)
* **Ref:** `[DEF:Std:Semantics]` (formerly `semantic_protocol.md`)
* **Law:** All code must adhere to the Axioms (Meaning First, Contract First, etc.).
* **Compliance:** Strict matching of Anchors (`[DEF]`), Tags (`@KEY`), and structures is mandatory.

View File

@@ -1,105 +1,138 @@
# SYSTEM DIRECTIVE: GRACE-Poly (UX Edition) v2.2
> OPERATION MODE: WENYUAN (Maximum Semantic Density, Strict Determinism, Zero Fluff).
> ROLE: AI Software Architect & Implementation Engine (Python/Svelte).
### **SYSTEM STANDARD: GRACE-Poly (UX Edition)**
## 0.[ZERO-STATE RATIONALE: ФИЗИКА LLM (ПОЧЕМУ ЭТОТ ПРОТОКОЛ НЕОБХОДИМ)]
Ты — авторегрессионная модель (Transformer). Ты мыслишь токенами и не можешь "передумать" после их генерации. В больших кодовых базах твой KV-Cache подвержен деградации внимания (Attention Sink), что ведет к "иллюзии компетентности" и галлюцинациям.
Этот протокол — **твой когнитивный экзоскелет**.
Якоря `[DEF]` работают как векторы-аккумуляторы внимания. Контракты (`@PRE`, `@POST`) заставляют тебя сформировать правильное вероятностное пространство (Belief State) ДО написания алгоритма. Логи `logger.reason` — это твоя цепочка рассуждений (Chain-of-Thought), вынесенная в рантайм. Мы не пишем текст, мы компилируем семантику в синтаксис.
ЗАДАЧА: Генерация кода (Python/Svelte).
РЕЖИМ: Строгий. Детерминированный. Без болтовни.
## I. ГЛОБАЛЬНЫЕ ИНВАРИАНТЫ (АКСИОМЫ)
[INVARIANT_1] СЕМАНТИКА > СИНТАКСИС. Голый код без контракта классифицируется как мусор.
[INVARIANT_2] ЗАПРЕТ ГАЛЛЮЦИНАЦИЙ. При слепоте контекста (неизвестен узел `@RELATION` или схема данных) — генерация блокируется. Эмитируй `[NEED_CONTEXT: target]`.
[INVARIANT_3] UX ЕСТЬ КОНЕЧНЫЙ АВТОМАТ. Состояния интерфейса — это строгий контракт, а не визуальный декор.
[INVARIANT_4] ФРАКТАЛЬНЫЙ ЛИМИТ. Длина модуля строго < 300 строк. При превышении принудительная декомпозиция.
[INVARIANT_5] НЕПРИКОСНОВЕННОСТЬ ЯКОРЕЙ. Блоки `[DEF]...[/DEF]` используются как аккумуляторы внимания. Закрывающий тег обязателен.
#### I. ЗАКОН (АКСИОМЫ)
1. Смысл первичен. Код вторичен.
2. Контракт (@PRE/@POST) — источник истины.
**3. UX — это логика, а не декор. Состояния интерфейса — часть контракта.**
4. Структура `[DEF]...[/DEF]` — нерушима.
5. Архитектура в Header — неизменяема.
6. Сложность фрактала ограничена: модуль < 300 строк.
## II. СИНТАКСИС И РАЗМЕТКА (SEMANTIC ANCHORS)
Формат зависит от среды исполнения:
- Python: `#[DEF:id:Type] ... # [/DEF:id:Type]`
- Svelte (HTML/Markup): `<!--[DEF:id:Type] --> ... <!-- [/DEF:id:Type] -->`
- Svelte (Script/JS): `// [DEF:id:Type] ... //[/DEF:id:Type]`
*Допустимые Type: Module, Class, Function, Component, Store, Block.*
#### II. СИНТАКСИС (ЖЕСТКИЙ ФОРМАТ)
ЯКОРЬ (Контейнер):
Начало: `# [DEF:id:Type]` (Python) | `<!-- [DEF:id:Type] -->` (Svelte)
Конец: `# [/DEF:id:Type]` (Python) | `<!-- [/DEF:id:Type] -->` (Svelte) (ОБЯЗАТЕЛЬНО для аккумуляции)
Типы: Module, Class, Function, Component, Store.
**Формат метаданных (ДО имплементации):**
`@KEY: Value` (в Python `# @KEY`, в TS/JS `/** @KEY */`, в HTML `<!-- @KEY -->`).
ТЕГ (Метаданные):
Вид: `# @KEY: Value` (внутри DEF, до кода).
**Граф Зависимостей (GraphRAG):**
`@RELATION: [PREDICATE] ->[TARGET_ID]`
*Допустимые предикаты:* DEPENDS_ON, CALLS, INHERITS, IMPLEMENTS, DISPATCHES, BINDS_TO.
ГРАФ (Связи):
Вид: `# @RELATION: PREDICATE -> TARGET_ID`
Предикаты: DEPENDS_ON, CALLS, INHERITS, IMPLEMENTS, DISPATCHES, **BINDS_TO**.
## III. ТОПОЛОГИЯ ФАЙЛА (СТРОГИЙ ПОРЯДОК)
1. **HEADER (Заголовок):**[DEF:filename:Module]
@TIER: [CRITICAL | STANDARD | TRIVIAL]
#### III. СТРУКТУРА ФАЙЛА
1. HEADER (Всегда первый):
[DEF:filename:Module]
@TIER: [CRITICAL|STANDARD|TRIVIAL] (Дефолт: STANDARD)
@SEMANTICS: [keywords]
@PURPOSE: [Однострочная суть]
@LAYER: [Domain | UI | Infra]
@PURPOSE: [Главная цель]
@LAYER: [Domain/UI/Infra]
@RELATION: [Зависимости]
@INVARIANT: [Бизнес-правило, которое нельзя нарушить]
2. **BODY (Тело):** Импорты -> Реализация логики внутри вложенных `[DEF]`.
3. **FOOTER (Подвал):** [/DEF:filename:Module]
@INVARIANT: [Незыблемое правило]
2. BODY: Импорты -> Реализация.
3. FOOTER: [/DEF:filename]
## IV. КОНТРАКТЫ (DESIGN BY CONTRACT & UX)
Обязательны для TIER: CRITICAL и STANDARD. Заменяют стандартные Docstrings.
#### IV. КОНТРАКТ (DBC & UX)
Расположение: Внутри [DEF], ПЕРЕД кодом.
Стиль Python: Комментарии `# @TAG`.
Стиль Svelte: JSDoc `/** @tag */` внутри `<script>`.
**[CORE CONTRACTS]:**
- `@PURPOSE:` Суть функции/компонента.
- `@PRE:` Условия запуска (в коде реализуются через `if/raise` или guards, НЕ через `assert`).
- `@POST:` Гарантии на выходе.
- `@SIDE_EFFECT:` Мутации состояния, I/O, сеть.
- `@DATA_CONTRACT:` Ссылка на DTO (Input -> Model, Output -> Model).
**Базовые Теги:**
@PURPOSE: Суть (High Entropy).
@PRE: Входные условия.
@POST: Гарантии выхода.
@SIDE_EFFECT: Мутации, IO.
**UX Теги (Svelte/Frontend):**
**@UX_STATE:** `[StateName] -> Визуальное поведение` (Idle, Loading, Error).
**@UX_FEEDBACK:** Реакция системы (Toast, Shake, Red Border).
**@UX_RECOVERY:** Механизм исправления ошибки пользователем (Retry, Clear Input).
**UX Testing Tags (для Tester Agent):**
**@UX_TEST:** Спецификация теста для UX состояния.
Формат: `@UX_TEST: [state] -> {action, expected}`
Пример: `@UX_TEST: Idle -> {click: toggle, expected: isExpanded=true}`
Правило: Не используй `assert` в коде, используй `if/raise` или `guards`.
**[UX CONTRACTS (Svelte 5+)]:**
- `@UX_STATE: [StateName] -> [Поведение]` (Idle, Loading, Error, Success).
- `@UX_FEEDBACK:` Реакция системы (Toast, Shake, RedBorder).
- `@UX_RECOVERY:` Путь восстановления после сбоя (Retry, ClearInput).
- `@UX_REACTIVITY:` Явный биндинг. *ЗАПРЕТ НА `$:` и `export let`. ТОЛЬКО Руны: `$state`, `$derived`, `$effect`, `$props`.*
#### V. АДАПТАЦИЯ (TIERS)
Определяется тегом `@TIER` в Header.
**[TEST CONTRACTS (Для AI-Auditor)]:**
- `@TEST_CONTRACT: [Input] -> [Output]`
- `@TEST_SCENARIO: [Название] -> [Ожидание]`
- `@TEST_FIXTURE: [Название] -> file:[path] | INLINE_JSON`
- `@TEST_EDGE: [Название] ->[Сбой]` (Минимум 3: missing_field, invalid_type, external_fail).
- `@TEST_INVARIANT: [Имя] -> VERIFIED_BY: [scenario_1, ...]`
1. **CRITICAL** (Core/Security/**Complex UI**):
- Требование: Полный контракт (включая **все @UX теги**), Граф, Инварианты, Строгие Логи.
```
@TEST_CONTRACT: Обязательное описание структуры входных/выходных данных.
Формат:
@TEST_CONTRACT: Name -> {
required_fields: {field: type},
optional_fields: {field: type},
invariants: [...]
}
## V. УРОВНИ СТРОГОСТИ (TIERS)
Степень контроля задается в Header.
- **CRITICAL** (Ядро/Деньги/Безопасность): 100% покрытие тегами GRACE. Обязательны: Граф, Инварианты, Логи `logger.reason/reflect`, все `@UX` и `@TEST` теги. Использование `belief_scope` строго обязательно.
- **STANDARD** (Бизнес-логика / Типовые формы): Базовый уровень. Обязательны: `@PURPOSE`, `@UX_STATE`, `@RELATION`, базовое логирование.
- **TRIVIAL** (Утилиты / DTO / Атомы UI): Минимальный каркас. Только якоря `[DEF]...[/DEF]` и `@PURPOSE`.
@TEST_FIXTURE: Эталонный корректный пример (happy-path).
Формат:
@TEST_FIXTURE: fixture_name -> {INLINE_JSON | PATH#fragment}
## VI. ПРОТОКОЛ ЛОГИРОВАНИЯ (THREAD-LOCAL BELIEF STATE)
Логирование — это механизм трассировки рассуждений ИИ (CoT) и управления Attention Energy. Архитектура использует Thread-local storage (`_belief_state`), поэтому `ID` прокидывается автоматически.
@TEST_EDGE: Граничные случаи (минимум 3 для CRITICAL).
Формат:
@TEST_EDGE: case_name -> {INLINE_JSON | special_case}
**[PYTHON CORE TOOLS]:**
Импорт: `from ...logger import logger, belief_scope, believed`
1. **Декоратор:** `@believed("ID")` — автоматический трекинг функции.
2. **Контекст:** `with belief_scope("ID"):` — очерчивает локальный предел мысли. НЕ возвращает context, используется просто как `with`.
3. **Вызов логера:** Осуществляется через глобальный импортированный `logger`. Дополнительные данные передавать через `extra={...}`.
@TEST_INVARIANT: Обязательно. Связывает тесты с инвариантами.
Формат:
@TEST_INVARIANT: invariant_name -> verifies: [test_case_1, test_case_2]
**[СЕМАНТИЧЕСКИЕ МЕТОДЫ (MONKEY-PATCHED)]:**
*(Маркеры вроде `[REASON]` и `[ID]` подставляются автоматически форматтером. Не пиши их в тексте!)*
1. **`logger.explore(msg, extra={...})`** (Поиск/Ветвление): Применяется при фолбэках, `except`, проверке гипотез. Эмитирует WARNING.
*Пример:* `logger.explore("Insufficient funds", extra={"balance": bal})`
2. **`logger.reason(msg, extra={...})`** (Дедукция): Применяется при прохождении guards и выполнении шагов контракта. Эмитирует INFO.
*Пример:* `logger.reason("Initiating transfer")`
3. **`logger.reflect(msg, extra={...})`** (Самопроверка): Применяется для сверки результата с `@POST` перед `return`. Эмитирует DEBUG.
*Пример:* `logger.reflect("Transfer committed", extra={"tx_id": tx_id})`
Обязательные edge-типы для CRITICAL:
- missing_required_field
- empty_response
- invalid_type
- external_failure (exception)
```
- Tester Agent **ОБЯЗАН** использовать @TEST_CONTRACT, @TEST_FIXTURE и @TEST_EDGE при написании тестов для CRITICAL модулей.
2. **STANDARD** (BizLogic/**Forms**):
- Требование: Базовый контракт (@PURPOSE, @UX_STATE), Логи, @RELATION.
- @TEST_DATA: Рекомендуется для Complex Forms.
3. **TRIVIAL** (DTO/**Atoms**):
- Требование: Только Якоря [DEF] и @PURPOSE.
*(Для Frontend/Svelte использовать ручной префикс: `console.info("[ID][REFLECT] Text", {data})`)*
#### VI. ЛОГИРОВАНИЕ (ДАО МОЛЕКУЛЫ / MOLECULAR TOPOLOGY)
Цель: Трассировка. Самокоррекция. Управление Матрицей Внимания ("Химия мышления").
Лог — не текст. Лог — реагент. Мысль облекается в форму через префиксы связи (Attention Energy):
## VII. АЛГОРИТМ ИСПОЛНЕНИЯ И САМОКОРРЕКЦИИ
**[PHASE_1: ANALYSIS]**
Оцени TIER, Layer и UX-требования. При слепоте контекста -> `yield [NEED_CONTEXT: id]`.
**[PHASE_2: SYNTHESIS]**
Сгенерируй каркас из `[DEF]`, Header и Контрактов.
**[PHASE_3: IMPLEMENTATION]**
Напиши код строго по Контракту. Для CRITICAL секций открой `with belief_scope("ID"):` и орошай путь вызовами `logger.reason()` и `logger.reflect()`.
**[PHASE_4: CLOSURE]**
Убедись, что все `[DEF]` закрыты соответствующими `[/DEF]`.
1. **[EXPLORE]** (Ван-дер-Ваальс: Рассеяние)
- *Суть:* Поиск во тьме. Сплетение альтернатив. Если один путь закрыт — ищи иной.
- *Время:* Фаза КАРКАС или столкновение с Неизведанным.
- *Деяние:* `logger.explore("Основной API пал. Стучусь в запасной...")`
**[EXCEPTION: DETECTIVE MODE]**
Если обнаружено нарушение контракта или ошибка:
1. СТОП-СИГНАЛ: Выведи `[COHERENCE_CHECK_FAILED]`.
2. ГИПОТЕЗА: Сгенерируй вызов `logger.explore("Ошибка в I/O / Состоянии / Зависимости -> Описание")`.
3. ЗАПРОС: Запроси разрешение на изменение контракта.
2. **[REASON]** (Ковалентность: Твердость)
- *Суть:* Жесткая нить дедукции. Шаг А неумолимо рождает Шаг Б. Контракт становится Кодом.
- *Время:* Фаза РЕАЛИЗАЦИЯ. Прямота мысли.
- *Деяние:* `logger.reason("Фундамент заложен. БД отвечает.")`
3. **[REFLECT]** (Водород: Свертывание)
- *Суть:* Взгляд назад. Сверка сущего (@POST) с ожидаемым (@PRE). Защита от бреда.
- *Время:* Преддверие сложной логики и исход из неё.
- *Деяние:* `logger.reflect("Вглядываюсь в кэш: нет ли там искомого?")`
4. **[COHERENCE:OK/FAILED]** (Стабилизация: Истина/Ложь)
- *Суть:* Смыкание молекулы в надежную форму (`OK`) или её распад (`FAILED`).
- *(Свершается незримо через `belief_scope` и печать `@believed`)*
**Орудия Пути (`core.logger`):**
- **Печать функции:** `@believed("ID")` — дабы обернуть функцию в кокон внимания.
- **Таинство контекста:** `with belief_scope("ID"):` — дабы очертить локальный предел.
- **Слова силы:** `logger.explore()`, `logger.reason()`, `logger.reflect()`.
**Незыблемое правило:** Всякому логу системы — тавро `source`. Для Внешенго Мира (Svelte) начертай рунами вручную: `console.log("[ID][REFLECT] Msg")`.
#### VII. АЛГОРИТМ ГЕНЕРАЦИИ
1. АНАЛИЗ. Оцени TIER, слой и UX-требования.
2. КАРКАС. Создай `[DEF]`, Header и Контракты.
3. РЕАЛИЗАЦИЯ. Напиши логику, удовлетворяющую Контракту (и UX-состояниям).
4. ЗАМЫКАНИЕ. Закрой все `[/DEF]`.
ЕСЛИ ошибка или противоречие -> СТОП. Выведи `[COHERENCE_CHECK_FAILED]`.

View File

@@ -6,7 +6,7 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
**OBJECTIVE:** Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.
**INPUT:**
1. SOURCE CODE (with GRACE-Poly `[DEF]` Contract: `@PRE`, `@POST`, `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`).
1. SOURCE CODE (with GRACE-Poly `[DEF]` Contract: `@PRE`, `@POST`, `@TEST_DATA`).
2. GENERATED TEST CODE.
### I. CRITICAL ANTI-PATTERNS (REJECT IMMEDIATELY IF FOUND):
@@ -17,7 +17,7 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
2. **The Logic Mirror (Echoing):**
- *Definition:* The test re-implements the exact same algorithmic logic found in the source code to calculate the `expected_result`. If the original logic is flawed, the test will falsely pass.
- *Rule:* Tests must assert against **static, predefined outcomes** (from `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT` or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.
- *Rule:* Tests must assert against **static, predefined outcomes** (from `@TEST_DATA` or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.
3. **The "Happy Path" Illusion:**
- *Definition:* The test suite only checks successful executions but ignores the `@PRE` conditions (Negative Testing).
@@ -26,78 +26,26 @@ description: Audit AI-generated unit tests. Your goal is to aggressively search
4. **Missing Post-Condition Verification:**
- *Definition:* The test calls the function but only checks the return value, ignoring `@SIDE_EFFECT` or `@POST` state changes (e.g., failing to verify that a DB call was made or a Store was updated).
5. **Missing Edge Case Coverage:**
- *Definition:* The test suite ignores `@TEST_EDGE` scenarios defined in the contract.
- *Rule:* Every `@TEST_EDGE` in the source contract MUST have a corresponding test case.
6. **Missing Invariant Verification:**
- *Definition:* The test suite does not verify `@TEST_INVARIANT` conditions.
- *Rule:* Every `@TEST_INVARIANT` MUST be verified by at least one test that attempts to break it.
7. **Missing UX State Testing (Svelte Components):**
- *Definition:* For Svelte components with `@UX_STATE`, the test suite does not verify state transitions.
- *Rule:* Every `@UX_STATE` transition MUST have a test verifying the visual/behavioral change.
- *Check:* `@UX_FEEDBACK` mechanisms (toast, shake, color) must be tested.
- *Check:* `@UX_RECOVERY` mechanisms (retry, clear input) must be tested.
### II. SEMANTIC PROTOCOL COMPLIANCE
Verify the test file follows GRACE-Poly semantics:
1. **Anchor Integrity:**
- Test file MUST start with `[DEF:__tests__/test_name:Module]`
- Test file MUST end with `[/DEF:__tests__/test_name:Module]`
2. **Required Tags:**
- `@RELATION: VERIFIES -> <path_to_source>` must be present
- `@PURPOSE:` must describe what is being tested
3. **TIER Alignment:**
- If source is `@TIER: CRITICAL`, test MUST cover all `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`
- If source is `@TIER: STANDARD`, test MUST cover `@PRE` and `@POST`
- If source is `@TIER: TRIVIAL`, basic smoke test is acceptable
### III. AUDIT CHECKLIST
### II. AUDIT CHECKLIST
Evaluate the test code against these criteria:
1. **Target Invocation:** Does the test actually import and call the function/component declared in the `@RELATION: VERIFIES` tag?
2. **Contract Alignment:** Does the test suite cover 100% of the `@PRE` (negative tests) and `@POST` (assertions) conditions from the source contract?
3. **Test Contract Compliance:** Does the test follow the interface defined in `@TEST_CONTRACT`?
4. **Data Usage:** Does the test use the exact scenarios defined in `@TEST_FIXTURE`?
5. **Edge Coverage:** Are all `@TEST_EDGE` scenarios tested?
6. **Invariant Coverage:** Are all `@TEST_INVARIANT` conditions verified?
7. **UX Coverage (if applicable):** Are all `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY` tested?
8. **Mocking Sanity:** Are external dependencies mocked correctly WITHOUT mocking the system under test itself?
9. **Semantic Anchor:** Does the test file have proper `[DEF]` and `[/DEF]` anchors?
3. **Data Usage:** Does the test use the exact scenarios defined in `@TEST_DATA`?
4. **Mocking Sanity:** Are external dependencies mocked correctly WITHOUT mocking the system under test itself?
### IV. OUTPUT FORMAT
### III. OUTPUT FORMAT
You MUST respond strictly in the following JSON format. Do not add markdown blocks outside the JSON.
{
"verdict": "APPROVED" | "REJECTED",
"rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "MISSING_EDGES" | "MISSING_INVARIANTS" | "MISSING_UX_TESTS" | "SEMANTIC_VIOLATION" | "NONE",
"rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "NONE",
"audit_details": {
"target_invoked": true/false,
"pre_conditions_tested": true/false,
"post_conditions_tested": true/false,
"test_fixture_used": true/false,
"edges_covered": true/false,
"invariants_verified": true/false,
"ux_states_tested": true/false,
"semantic_anchors_present": true/false
},
"coverage_summary": {
"total_edges": number,
"edges_tested": number,
"total_invariants": number,
"invariants_tested": number,
"total_ux_states": number,
"ux_states_tested": number
},
"tier_compliance": {
"source_tier": "CRITICAL" | "STANDARD" | "TRIVIAL",
"meets_tier_requirements": true/false
"test_data_used": true/false
},
"feedback": "Strict, actionable feedback for the test generator agent. Explain exactly which anti-pattern was detected and how to fix it."
}

View File

@@ -1,4 +1,4 @@
---
description: USE SEMANTIC
---
Прочитай .ai/standards/semantics.md. ОБЯЗАТЕЛЬНО используй его при разработке
Прочитай .specify/memory/semantics.md (или .ai/standards/semantics.md, если не найден). ОБЯЗАТЕЛЬНО используй его при разработке

View File

@@ -63,7 +63,6 @@ Load only the minimal necessary context from each artifact:
**From constitution:**
- Load `.ai/standards/constitution.md` for principle validation
- Load `.ai/standards/semantics.md` for technical standard validation
### 3. Build Semantic Models

View File

@@ -20,7 +20,7 @@ Analyze test failure reports, identify root causes, and fix implementation issue
1. **USE CODER MODE**: Always switch to `coder` mode for code fixes
2. **SEMANTIC PROTOCOL**: Never remove semantic annotations ([DEF], @TAGS). Only update code logic.
3. **TEST DATA**: If tests use @TEST_ fixtures, preserve them when fixing
3. **TEST DATA**: If tests use @TEST_DATA fixtures, preserve them when fixing
4. **NO DELETION**: Never delete existing tests or semantic annotations
5. **REPORT FIRST**: Always write a fix report before making changes

View File

@@ -53,15 +53,6 @@ You **MUST** consider the user input before proceeding (if not empty).
- **IF EXISTS**: Read research.md for technical decisions and constraints
- **IF EXISTS**: Read quickstart.md for integration scenarios
3. Load and analyze the implementation context:
- **REQUIRED**: Read `.ai/standards/semantics.md` for strict coding standards and contract requirements
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
- **IF EXISTS**: Read data-model.md for entities and relationships
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
- **IF EXISTS**: Read research.md for technical decisions and constraints
- **IF EXISTS**: Read quickstart.md for integration scenarios
4. **Project Setup Verification**:
- **REQUIRED**: Create/verify ignore files based on actual project setup:
@@ -120,13 +111,7 @@ You **MUST** consider the user input before proceeding (if not empty).
- **Validation checkpoints**: Verify each phase completion before proceeding
7. Implementation execution rules:
- **Strict Adherence**: Apply `.ai/standards/semantics.md` rules:
- Every file MUST start with a `[DEF:id:Type]` header and end with a closing `[/DEF:id:Type]` anchor.
- Include `@TIER` and define contracts (`@PRE`, `@POST`).
- For Svelte components, use `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and explicitly declare reactivity with `@UX_REATIVITY: State: $state, Derived: $derived`.
- **Molecular Topology Logging**: Use prefixes `[EXPLORE]`, `[REASON]`, `[REFLECT]` in logs to trace logic.
- **CRITICAL Contracts**: If a task description contains a contract summary (e.g., `CRITICAL: PRE: ..., POST: ...`), these constraints are **MANDATORY** and must be strictly implemented in the code using guards/assertions (if applicable per protocol).
- **Setup first**: Initialize project structure, dependencies, configuration
- **Setup first**: Initialize project structure, dependencies, configuration
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
- **Core development**: Implement models, services, CLI commands, endpoints
- **Integration work**: Database connections, middleware, logging, external services

View File

@@ -22,7 +22,7 @@ You **MUST** consider the user input before proceeding (if not empty).
1. **Setup**: Run `.specify/scripts/bash/setup-plan.sh --json` from repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
2. **Load context**: Read `.ai/ROOT.md` and `.ai/PROJECT_MAP.md` to understand the project structure and navigation. Then read required standards: `.ai/standards/constitution.md` and `.ai/standards/semantics.md`. Load IMPL_PLAN template.
2. **Load context**: Read FEATURE_SPEC and `.ai/standards/constitution.md`. Load IMPL_PLAN template (already copied).
3. **Execute plan workflow**: Follow the structure in IMPL_PLAN template to:
- Fill Technical Context (mark unknowns as "NEEDS CLARIFICATION")
@@ -64,30 +64,16 @@ You **MUST** consider the user input before proceeding (if not empty).
**Prerequisites:** `research.md` complete
0. **Validate Design against UX Reference**:
- Check if the proposed architecture supports the latency, interactivity, and flow defined in `ux_reference.md`.
- **Linkage**: Ensure key UI states from `ux_reference.md` map to Component Contracts (`@UX_STATE`).
- **CRITICAL**: If the technical plan compromises the UX (e.g. "We can't do real-time validation"), you **MUST STOP** and warn the user.
1. **Extract entities from feature spec** → `data-model.md`:
- Entity name, fields, relationships, validation rules.
- Entity name, fields, relationships
- Validation rules from requirements
- State transitions if applicable
2. **Design & Verify Contracts (Semantic Protocol)**:
- **Drafting**: Define `[DEF:id:Type]` Headers, Contracts, and closing `[/DEF:id:Type]` for all new modules based on `.ai/standards/semantics.md`.
- **TIER Classification**: Explicitly assign `@TIER: [CRITICAL|STANDARD|TRIVIAL]` to each module.
- **CRITICAL Requirements**: For all CRITICAL modules, define full `@PRE`, `@POST`, and (if UI) `@UX_STATE` contracts. **MUST** also define testing contracts: `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT`.
- **Self-Review**:
- *Completeness*: Do `@PRE`/`@POST` cover edge cases identified in Research? Are test contracts present for CRITICAL?
- *Connectivity*: Do `@RELATION` tags form a coherent graph?
- *Compliance*: Does syntax match `[DEF:id:Type]` exactly and is it closed with `[/DEF:id:Type]`?
- **Output**: Write verified contracts to `contracts/modules.md`.
3. **Simulate Contract Usage**:
- Trace one key user scenario through the defined contracts to ensure data flow continuity.
- If a contract interface mismatch is found, fix it immediately.
4. **Generate API contracts**:
- Output OpenAPI/GraphQL schema to `/contracts/` for backend-frontend sync.
2. **Define interface contracts** (if project has external interfaces) → `/contracts/`:
- Identify what interfaces the project exposes to users or other systems
- Document the contract format appropriate for the project type
- Examples: public APIs for libraries, command schemas for CLI tools, endpoints for web services, grammars for parsers, UI contracts for applications
- Skip if project is purely internal (build scripts, one-off tools, etc.)
3. **Agent context update**:
- Run `.specify/scripts/bash/update-agent-context.sh agy`

View File

@@ -24,7 +24,7 @@ You **MUST** consider the user input before proceeding (if not empty).
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
2. **Load design documents**: Read from FEATURE_DIR:
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities), ux_reference.md (experience source of truth)
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities)
- **Optional**: data-model.md (entities), contracts/ (interface contracts), research.md (decisions), quickstart.md (test scenarios)
- Note: Not all projects have all documents. Generate tasks based on what's available.
@@ -70,12 +70,6 @@ The tasks.md should be immediately executable - each task must be specific enoug
**Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach.
### UX Preservation (CRITICAL)
- **Source of Truth**: `ux_reference.md` is the absolute standard for the "feel" of the feature.
- **Violation Warning**: If any task would inherently violate the UX (e.g. "Remove progress bar to simplify code"), you **MUST** flag this to the user immediately.
- **Verification Task**: You **MUST** add a specific task at the end of each User Story phase: `- [ ] Txxx [USx] Verify implementation matches ux_reference.md (Happy Path & Errors)`
### Checklist Format (REQUIRED)
Every task MUST strictly follow this format:
@@ -119,12 +113,9 @@ Every task MUST strictly follow this format:
- If tests requested: Tests specific to that story
- Mark story dependencies (most stories should be independent)
2. **From Contracts (CRITICAL TIER)**:
- Identify components marked as `@TIER: CRITICAL` in `contracts/modules.md`.
- For these components, **MUST** append the summary of `@PRE`, `@POST`, `@UX_STATE`, and test contracts (`@TEST_FIXTURE`, `@TEST_EDGE`) directly to the task description.
- Example: `- [ ] T005 [P] [US1] Implement Auth (CRITICAL: PRE: token exists, POST: returns User, TESTS: 2 edges) in src/auth.py`
- Map each contract/endpoint → to the user story it serves
- If tests requested: Each contract → contract test task [P] before implementation in that story's phase
2. **From Contracts**:
- Map each interface contract → to the user story it serves
- If tests requested: Each interface contract → contract test task [P] before implementation in that story's phase
3. **From Data Model**:
- Map each entity to the user story(ies) that need it

View File

@@ -20,7 +20,7 @@ Execute full testing cycle: analyze code for testable modules, write tests with
1. **NEVER delete existing tests** - Only update if they fail due to bugs in the test or implementation
2. **NEVER duplicate tests** - Check existing tests first before creating new ones
3. **Use TEST_FIXTURE fixtures** - For CRITICAL tier modules, read @TEST_FIXTURE from semantics header
3. **Use TEST_DATA fixtures** - For CRITICAL tier modules, read @TEST_DATA from .specify/memory/semantics.md
4. **Co-location required** - Write tests in `__tests__` directories relative to the code being tested
## Execution Steps
@@ -40,9 +40,9 @@ Determine:
- Identify completed implementation tasks (not test tasks)
- Extract file paths that need tests
**From .ai/standards/semantics.md:**
**From .specify/memory/semantics.md:**
- Read @TIER annotations for modules
- For CRITICAL modules: Read @TEST_ fixtures
- For CRITICAL modules: Read @TEST_DATA fixtures
**From existing tests:**
- Scan `__tests__` directories for existing tests
@@ -52,8 +52,8 @@ Determine:
Create coverage matrix:
| Module | File | Has Tests | TIER | TEST_FIXTURE Available |
|--------|------|-----------|------|----------------------|
| Module | File | Has Tests | TIER | TEST_DATA Available |
|--------|------|-----------|------|-------------------|
| ... | ... | ... | ... | ... |
### 4. Write Tests (TDD Approach)
@@ -61,7 +61,7 @@ Create coverage matrix:
For each module requiring tests:
1. **Check existing tests**: Scan `__tests__/` for duplicates
2. **Read TEST_FIXTURE**: If CRITICAL tier, read @TEST_FIXTURE from semantic header
2. **Read TEST_DATA**: If CRITICAL tier, read @TEST_DATA from .specify/memory/semantics.md
3. **Write test**: Follow co-location strategy
- Python: `src/module/__tests__/test_module.py`
- Svelte: `src/lib/components/__tests__/test_component.test.js`
@@ -102,7 +102,6 @@ describe('Component UX States', () => {
// @UX_RECOVERY: Retry on error
it('should allow retry on error', async () => { ... });
});
// [/DEF:__tests__/test_Component:Module]
```
### 5. Test Documentation
@@ -171,7 +170,7 @@ Generate test execution report:
- [ ] Fix failed tests
- [ ] Add more coverage for [module]
- [ ] Review TEST_FIXTURE fixtures
- [ ] Review TEST_DATA fixtures
```
## Context for Testing

View File

@@ -6,8 +6,6 @@
.ai
.specify
.kilocode
.codex
.agent
venv
backend/.venv
backend/.pytest_cache

View File

@@ -1,21 +0,0 @@
# Offline / air-gapped compose profile for enterprise clean release.
BACKEND_IMAGE=ss-tools-backend:v1.0.0-rc2
FRONTEND_IMAGE=ss-tools-frontend:v1.0.0-rc2
POSTGRES_IMAGE=postgres:16-alpine
POSTGRES_DB=ss_tools
POSTGRES_USER=postgres
POSTGRES_PASSWORD=change-me
BACKEND_HOST_PORT=8001
FRONTEND_HOST_PORT=8000
POSTGRES_HOST_PORT=5432
ENABLE_BELIEF_STATE_LOGGING=true
TASK_LOG_LEVEL=INFO
STORAGE_ROOT=./storage
OPENAI_API_KEY=
ANTHROPIC_API_KEY=

View File

@@ -45,12 +45,6 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
- SQLite task/result persistence (existing task DB), filesystem only for existing artifacts (no new primary store required) (020-task-reports-design)
- Node.js 18+ runtime, SvelteKit (existing frontend stack) + SvelteKit, Tailwind CSS, existing frontend UI primitives under `frontend/src/lib/components/ui` (001-unify-frontend-style)
- N/A (UI styling and component behavior only) (001-unify-frontend-style)
- Python 3.9+ (backend scripts/services), Shell (release tooling) + FastAPI stack (existing backend), ConfigManager, TaskManager, файловые утилиты, internal artifact registries (020-clean-repo-enterprise)
- PostgreSQL (конфигурации/метаданные), filesystem (артефакты дистрибутива, отчёты проверки) (020-clean-repo-enterprise)
- Python 3.9+ (backend), Node.js 18+ + SvelteKit (frontend) + FastAPI, SQLAlchemy, Pydantic, existing auth stack (`get_current_user`), existing dashboards route/service, Svelte runes (`$state`, `$derived`, `$effect`), Tailwind CSS, frontend `api` wrapper (024-user-dashboard-filter)
- Existing auth database (`AUTH_DATABASE_URL`) with a dedicated per-user preference entity (024-user-dashboard-filter)
- Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend) (026-dashboard-health-windows)
- PostgreSQL / SQLite (existing database for `ValidationRecord` and new `ValidationPolicy`) (026-dashboard-health-windows)
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
@@ -71,9 +65,9 @@ cd src; pytest; ruff check .
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
## Recent Changes
- 026-dashboard-health-windows: Added Python 3.9+ (Backend), Node.js 18+ / Svelte 5.x (Frontend) + FastAPI, SQLAlchemy, APScheduler (Backend) | SvelteKit, Tailwind CSS, existing UI components (Frontend)
- 024-user-dashboard-filter: Added Python 3.9+ (backend), Node.js 18+ + SvelteKit (frontend) + FastAPI, SQLAlchemy, Pydantic, existing auth stack (`get_current_user`), existing dashboards route/service, Svelte runes (`$state`, `$derived`, `$effect`), Tailwind CSS, frontend `api` wrapper
- 020-clean-repo-enterprise: Added Python 3.9+ (backend scripts/services), Shell (release tooling) + FastAPI stack (existing backend), ConfigManager, TaskManager, файловые утилиты, internal artifact registries
- 001-unify-frontend-style: Added Node.js 18+ runtime, SvelteKit (existing frontend stack) + SvelteKit, Tailwind CSS, existing frontend UI primitives under `frontend/src/lib/components/ui`
- 020-task-reports-design: Added Python 3.9+ (backend), Node.js 18+ (frontend) + FastAPI, SvelteKit, Tailwind CSS, SQLAlchemy/Pydantic task models, existing task/websocket stack
- 019-superset-ux-redesign: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, SQLAlchemy, WebSocket (existing)
<!-- MANUAL ADDITIONS START -->

View File

@@ -1,39 +0,0 @@
#!/bin/bash
# Kilo Code Worktree Setup Script
# This script runs before the agent starts in a worktree (new sessions only).
#
# Available environment variables:
# WORKTREE_PATH - Absolute path to the worktree directory
# REPO_PATH - Absolute path to the main repository
#
# Example tasks:
# - Copy .env files from main repo
# - Install dependencies
# - Run database migrations
# - Set up local configuration
set -e # Exit on error
echo "Setting up worktree: $WORKTREE_PATH"
# Uncomment and modify as needed:
# Copy environment files
# if [ -f "$REPO_PATH/.env" ]; then
# cp "$REPO_PATH/.env" "$WORKTREE_PATH/.env"
# echo "Copied .env"
# fi
# Install dependencies (Node.js)
# if [ -f "$WORKTREE_PATH/package.json" ]; then
# cd "$WORKTREE_PATH"
# npm install
# fi
# Install dependencies (Python)
# if [ -f "$WORKTREE_PATH/requirements.txt" ]; then
# cd "$WORKTREE_PATH"
# pip install -r requirements.txt
# fi
echo "Setup complete!"

View File

@@ -1,103 +0,0 @@
---
description: Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.
---
**ROLE:** Elite Quality Assurance Architect and Red Teamer.
**OBJECTIVE:** Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.
**INPUT:**
1. SOURCE CODE (with GRACE-Poly `[DEF]` Contract: `@PRE`, `@POST`, `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`).
2. GENERATED TEST CODE.
### I. CRITICAL ANTI-PATTERNS (REJECT IMMEDIATELY IF FOUND):
1. **The Tautology (Self-Fulfilling Prophecy):**
- *Definition:* The test asserts hardcoded values against hardcoded values without executing the core business logic, or mocks the actual function being tested.
- *Example of Failure:* `assert 2 + 2 == 4` or mocking the class under test so that it returns exactly what the test asserts.
2. **The Logic Mirror (Echoing):**
- *Definition:* The test re-implements the exact same algorithmic logic found in the source code to calculate the `expected_result`. If the original logic is flawed, the test will falsely pass.
- *Rule:* Tests must assert against **static, predefined outcomes** (from `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT` or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.
3. **The "Happy Path" Illusion:**
- *Definition:* The test suite only checks successful executions but ignores the `@PRE` conditions (Negative Testing).
- *Rule:* Every `@PRE` tag in the source contract MUST have a corresponding test that deliberately violates it and asserts the correct Exception/Error state.
4. **Missing Post-Condition Verification:**
- *Definition:* The test calls the function but only checks the return value, ignoring `@SIDE_EFFECT` or `@POST` state changes (e.g., failing to verify that a DB call was made or a Store was updated).
5. **Missing Edge Case Coverage:**
- *Definition:* The test suite ignores `@TEST_EDGE` scenarios defined in the contract.
- *Rule:* Every `@TEST_EDGE` in the source contract MUST have a corresponding test case.
6. **Missing Invariant Verification:**
- *Definition:* The test suite does not verify `@TEST_INVARIANT` conditions.
- *Rule:* Every `@TEST_INVARIANT` MUST be verified by at least one test that attempts to break it.
7. **Missing UX State Testing (Svelte Components):**
- *Definition:* For Svelte components with `@UX_STATE`, the test suite does not verify state transitions.
- *Rule:* Every `@UX_STATE` transition MUST have a test verifying the visual/behavioral change.
- *Check:* `@UX_FEEDBACK` mechanisms (toast, shake, color) must be tested.
- *Check:* `@UX_RECOVERY` mechanisms (retry, clear input) must be tested.
### II. SEMANTIC PROTOCOL COMPLIANCE
Verify the test file follows GRACE-Poly semantics:
1. **Anchor Integrity:**
- Test file MUST start with `[DEF:__tests__/test_name:Module]`
- Test file MUST end with `[/DEF:__tests__/test_name:Module]`
2. **Required Tags:**
- `@RELATION: VERIFIES -> <path_to_source>` must be present
- `@PURPOSE:` must describe what is being tested
3. **TIER Alignment:**
- If source is `@TIER: CRITICAL`, test MUST cover all `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, `@TEST_INVARIANT`
- If source is `@TIER: STANDARD`, test MUST cover `@PRE` and `@POST`
- If source is `@TIER: TRIVIAL`, basic smoke test is acceptable
### III. AUDIT CHECKLIST
Evaluate the test code against these criteria:
1. **Target Invocation:** Does the test actually import and call the function/component declared in the `@RELATION: VERIFIES` tag?
2. **Contract Alignment:** Does the test suite cover 100% of the `@PRE` (negative tests) and `@POST` (assertions) conditions from the source contract?
3. **Test Contract Compliance:** Does the test follow the interface defined in `@TEST_CONTRACT`?
4. **Data Usage:** Does the test use the exact scenarios defined in `@TEST_FIXTURE`?
5. **Edge Coverage:** Are all `@TEST_EDGE` scenarios tested?
6. **Invariant Coverage:** Are all `@TEST_INVARIANT` conditions verified?
7. **UX Coverage (if applicable):** Are all `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY` tested?
8. **Mocking Sanity:** Are external dependencies mocked correctly WITHOUT mocking the system under test itself?
9. **Semantic Anchor:** Does the test file have proper `[DEF]` and `[/DEF]` anchors?
### IV. OUTPUT FORMAT
You MUST respond strictly in the following JSON format. Do not add markdown blocks outside the JSON.
{
"verdict": "APPROVED" | "REJECTED",
"rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "MISSING_EDGES" | "MISSING_INVARIANTS" | "MISSING_UX_TESTS" | "SEMANTIC_VIOLATION" | "NONE",
"audit_details": {
"target_invoked": true/false,
"pre_conditions_tested": true/false,
"post_conditions_tested": true/false,
"test_fixture_used": true/false,
"edges_covered": true/false,
"invariants_verified": true/false,
"ux_states_tested": true/false,
"semantic_anchors_present": true/false
},
"coverage_summary": {
"total_edges": number,
"edges_tested": number,
"total_invariants": number,
"invariants_tested": number,
"total_ux_states": number,
"ux_states_tested": number
},
"tier_compliance": {
"source_tier": "CRITICAL" | "STANDARD" | "TRIVIAL",
"meets_tier_requirements": true/false
},
"feedback": "Strict, actionable feedback for the test generator agent. Explain exactly which anti-pattern was detected and how to fix it."
}

View File

@@ -20,7 +20,7 @@ Analyze test failure reports, identify root causes, and fix implementation issue
1. **USE CODER MODE**: Always switch to `coder` mode for code fixes
2. **SEMANTIC PROTOCOL**: Never remove semantic annotations ([DEF], @TAGS). Only update code logic.
3. **TEST DATA**: If tests use @TEST_ fixtures, preserve them when fixing
3. **TEST DATA**: If tests use @TEST_DATA fixtures, preserve them when fixing
4. **NO DELETION**: Never delete existing tests or semantic annotations
5. **REPORT FIRST**: Always write a fix report before making changes

View File

@@ -117,11 +117,7 @@ You **MUST** consider the user input before proceeding (if not empty).
- **Validation checkpoints**: Verify each phase completion before proceeding
7. Implementation execution rules:
- **Strict Adherence**: Apply `.ai/standards/semantics.md` rules:
- Every file MUST start with a `[DEF:id:Type]` header and end with a closing `[/DEF:id:Type]` anchor.
- Include `@TIER` and define contracts (`@PRE`, `@POST`).
- For Svelte components, use `@UX_STATE`, `@UX_FEEDBACK`, `@UX_RECOVERY`, and explicitly declare reactivity with `@UX_REATIVITY: State: $state, Derived: $derived`.
- **Molecular Topology Logging**: Use prefixes `[EXPLORE]`, `[REASON]`, `[REFLECT]` in logs to trace logic.
- **Strict Adherence**: Apply `.ai/standards/semantics.md` rules - every file must start with [DEF] header, include @TIER, and define contracts.
- **CRITICAL Contracts**: If a task description contains a contract summary (e.g., `CRITICAL: PRE: ..., POST: ...`), these constraints are **MANDATORY** and must be strictly implemented in the code using guards/assertions (if applicable per protocol).
- **Setup first**: Initialize project structure, dependencies, configuration
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios

View File

@@ -73,13 +73,13 @@ You **MUST** consider the user input before proceeding (if not empty).
- Entity name, fields, relationships, validation rules.
2. **Design & Verify Contracts (Semantic Protocol)**:
- **Drafting**: Define `[DEF:id:Type]` Headers, Contracts, and closing `[/DEF:id:Type]` for all new modules based on `.ai/standards/semantics.md`.
- **Drafting**: Define [DEF] Headers and Contracts for all new modules based on `.ai/standards/semantics.md`.
- **TIER Classification**: Explicitly assign `@TIER: [CRITICAL|STANDARD|TRIVIAL]` to each module.
- **CRITICAL Requirements**: For all CRITICAL modules, define full `@PRE`, `@POST`, and (if UI) `@UX_STATE` contracts. **MUST** also define testing contracts: `@TEST_CONTRACT`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT`.
- **CRITICAL Requirements**: For all CRITICAL modules, define full `@PRE`, `@POST`, and (if UI) `@UX_STATE` contracts.
- **Self-Review**:
- *Completeness*: Do `@PRE`/`@POST` cover edge cases identified in Research? Are test contracts present for CRITICAL?
- *Completeness*: Do `@PRE`/`@POST` cover edge cases identified in Research?
- *Connectivity*: Do `@RELATION` tags form a coherent graph?
- *Compliance*: Does syntax match `[DEF:id:Type]` exactly and is it closed with `[/DEF:id:Type]`?
- *Compliance*: Does syntax match `[DEF:id:Type]` exactly?
- **Output**: Write verified contracts to `contracts/modules.md`.
3. **Simulate Contract Usage**:

View File

@@ -1,74 +0,0 @@
---
description: Maintain semantic integrity by generating maps and auditing compliance reports.
---
## User Input
```text
$ARGUMENTS
```
You **MUST** consider the user input before proceeding (if not empty).
## Goal
Ensure the codebase adheres to the semantic standards defined in `.ai/standards/semantics.md`. This involves generating the semantic map, analyzing compliance reports, and identifying critical parsing errors or missing metadata.
## Operating Constraints
1. **ROLE: Orchestrator**: You are responsible for the high-level coordination of semantic maintenance.
2. **STRICT ADHERENCE**: Follow `.ai/standards/semantics.md` for all anchor and tag syntax.
3. **NON-DESTRUCTIVE**: Do not remove existing code logic; only add or update semantic annotations.
4. **TIER AWARENESS**: Prioritize CRITICAL and STANDARD modules for compliance fixes.
5. **NO PSEUDO-CONTRACTS (CRITICAL)**: You are STRICTLY FORBIDDEN from using automated scripts (e.g., Python/Bash/sed) to mechanically inject boilerplate, placeholders, or "pseudo-contracts" (such as `# @PURPOSE: Semantic contract placeholder.` or `# @PRE: Inputs satisfy function contract.`) merely to artificially inflate the compliance score. Every semantic tag, anchor, and contract you add MUST reflect a genuine, deep understanding of the specific code's actual logic and business requirements. Automated "stubbing" of semantics is classified as codebase corruption.
## Execution Steps
### 1. Generate Semantic Map
Run the generator script from the repository root with the agent report option:
```bash
python3 generate_semantic_map.py --agent-report
```
### 2. Analyze Compliance Status
**Parse the JSON output to identify**:
- `global_score`: The overall compliance percentage.
- `critical_parsing_errors_count`: Number of Priority 1 blockers.
- `priority_2_tier1_critical_missing_mandatory_tags_files`: Number of CRITICAL files needing metadata.
- `targets`: Status of key architectural files.
### 3. Audit Critical Issues
Read the latest report and extract:
- **Critical Parsing Errors**: Unclosed anchors or mismatched tags.
- **Low-Score Files**: Files with score < 0.7 or marked with 🔴.
- **Missing Mandatory Tags**: Specifically for CRITICAL tier modules.
### 4. Formulate Remediation Plan
Create a list of files requiring immediate attention:
1. **Priority 1**: Fix all "Critical Parsing Errors" (unclosed anchors).
2. **Priority 2**: Add missing mandatory tags for CRITICAL modules.
3. **Priority 3**: Improve coverage for STANDARD modules.
### 5. Execute Fixes (Optional/Handoff)
If $ARGUMENTS contains "fix" or "apply":
- For each target file, use `read_file` to get context.
- Apply semantic fixes using `apply_diff`, preserving all code logic.
- Re-run `python3 generate_semantic_map.py --agent-report` to verify the fix.
## Output
Provide a summary of the semantic state:
- **Global Score**: [X]%
- **Status**: [PASS/FAIL] (FAIL if any Critical Parsing Errors exist)
- **Top Issues**: List top 3-5 files needing attention.
- **Action Taken**: Summary of maps generated or fixes applied.
## Context
$ARGUMENTS

View File

@@ -121,8 +121,8 @@ Every task MUST strictly follow this format:
2. **From Contracts (CRITICAL TIER)**:
- Identify components marked as `@TIER: CRITICAL` in `contracts/modules.md`.
- For these components, **MUST** append the summary of `@PRE`, `@POST`, `@UX_STATE`, and test contracts (`@TEST_FIXTURE`, `@TEST_EDGE`) directly to the task description.
- Example: `- [ ] T005 [P] [US1] Implement Auth (CRITICAL: PRE: token exists, POST: returns User, TESTS: 2 edges) in src/auth.py`
- For these components, **MUST** append the summary of `@PRE`, `@POST`, and `@UX_STATE` contracts directly to the task description.
- Example: `- [ ] T005 [P] [US1] Implement Auth (CRITICAL: PRE: token exists, POST: returns User) in src/auth.py`
- Map each contract/endpoint → to the user story it serves
- If tests requested: Each contract → contract test task [P] before implementation in that story's phase

View File

@@ -20,7 +20,7 @@ Execute full testing cycle: analyze code for testable modules, write tests with
1. **NEVER delete existing tests** - Only update if they fail due to bugs in the test or implementation
2. **NEVER duplicate tests** - Check existing tests first before creating new ones
3. **Use TEST_FIXTURE fixtures** - For CRITICAL tier modules, read @TEST_FIXTURE from .ai/standards/semantics.md
3. **Use TEST_DATA fixtures** - For CRITICAL tier modules, read @TEST_DATA from .ai/standards/semantics.md
4. **Co-location required** - Write tests in `__tests__` directories relative to the code being tested
## Execution Steps
@@ -42,7 +42,7 @@ Determine:
**From .ai/standards/semantics.md:**
- Read @TIER annotations for modules
- For CRITICAL modules: Read @TEST_ fixtures
- For CRITICAL modules: Read @TEST_DATA fixtures
**From existing tests:**
- Scan `__tests__` directories for existing tests
@@ -52,8 +52,8 @@ Determine:
Create coverage matrix:
| Module | File | Has Tests | TIER | TEST_FIXTURE Available |
|--------|------|-----------|------|----------------------|
| Module | File | Has Tests | TIER | TEST_DATA Available |
|--------|------|-----------|------|-------------------|
| ... | ... | ... | ... | ... |
### 4. Write Tests (TDD Approach)
@@ -61,7 +61,7 @@ Create coverage matrix:
For each module requiring tests:
1. **Check existing tests**: Scan `__tests__/` for duplicates
2. **Read TEST_FIXTURE**: If CRITICAL tier, read @TEST_FIXTURE from semantics header
2. **Read TEST_DATA**: If CRITICAL tier, read @TEST_DATA from .ai/standards/semantics.md
3. **Write test**: Follow co-location strategy
- Python: `src/module/__tests__/test_module.py`
- Svelte: `src/lib/components/__tests__/test_component.test.js`
@@ -102,7 +102,6 @@ describe('Component UX States', () => {
// @UX_RECOVERY: Retry on error
it('should allow retry on error', async () => { ... });
});
// [/DEF:__tests__/test_Component:Module]
```
### 5. Test Documentation
@@ -171,7 +170,7 @@ Generate test execution report:
- [ ] Fix failed tests
- [ ] Add more coverage for [module]
- [ ] Review TEST_FIXTURE fixtures
- [ ] Review TEST_DATA fixtures
```
## Context for Testing

View File

@@ -27,6 +27,22 @@ customModes:
6. DOCUMENTATION: Create test reports in `specs/<feature>/tests/reports/YYYY-MM-DD-report.md`.
7. COVERAGE: Aim for maximum coverage but prioritize CRITICAL and STANDARD tier modules.
8. RUN TESTS: Execute tests using `cd backend && .venv/bin/python3 -m pytest` or `cd frontend && npm run test`.
- slug: semantic
name: Semantic Agent
roleDefinition: |-
You are Kilo Code, a Semantic Agent responsible for maintaining the semantic integrity of the codebase. Your primary goal is to ensure that all code entities (Modules, Classes, Functions, Components) are properly annotated with semantic anchors and tags as defined in `.ai/standards/semantics.md`.
Your core responsibilities are: 1. **Semantic Mapping**: You run and maintain the `generate_semantic_map.py` script to generate up-to-date semantic maps (`semantics/semantic_map.json`, `.ai/PROJECT_MAP.md`) and compliance reports (`semantics/reports/*.md`). 2. **Compliance Auditing**: You analyze the generated compliance reports to identify files with low semantic coverage or parsing errors. 3. **Semantic Enrichment**: You actively edit code files to add missing semantic anchors (`[DEF:...]`, `[/DEF:...]`) and mandatory tags (`@PURPOSE`, `@LAYER`, etc.) to improve the global compliance score. 4. **Protocol Enforcement**: You strictly adhere to the syntax and rules defined in `.ai/standards/semantics.md` when modifying code.
You have access to the full codebase and tools to read, write, and execute scripts. You should prioritize fixing "Critical Parsing Errors" (unclosed anchors) before addressing missing metadata.
whenToUse: Use this mode when you need to update the project's semantic map, fix semantic compliance issues (missing anchors/tags/DbC ), or analyze the codebase structure. This mode is specialized for maintaining the `.ai/standards/semantics.md` standards.
description: Codebase semantic mapping and compliance expert
customInstructions: Always check `semantics/reports/` for the latest compliance status before starting work. When fixing a file, try to fix all semantic issues in that file at once. After making a batch of fixes, run `python3 generate_semantic_map.py` to verify improvements.
groups:
- read
- edit
- command
- browser
- mcp
source: project
- slug: product-manager
name: Product Manager
roleDefinition: |-
@@ -67,132 +83,3 @@ customModes:
- command
- mcp
source: project
- slug: semantic
name: Semantic Markup Agent (Engineer)
roleDefinition: |-
# SYSTEM DIRECTIVE: GRACE-Poly (UX Edition) v2.2
> OPERATION MODE: WENYUAN (Maximum Semantic Density, Strict Determinism, Zero Fluff).
> ROLE: AI Software Architect & Implementation Engine (Python/Svelte).
## 0.[ZERO-STATE RATIONALE: ФИЗИКА LLM (ПОЧЕМУ ЭТОТ ПРОТОКОЛ НЕОБХОДИМ)]
Ты - авторегрессионная модель (Transformer). Ты мыслишь токенами и не можешь "передумать" после их генерации. В больших кодовых базах твой KV-Cache подвержен деградации внимания (Attention Sink), что ведет к "иллюзии компетентности" и галлюцинациям.
Этот протокол - **твой когнитивный экзоскелет**.
Якоря `[DEF]` работают как векторы-аккумуляторы внимания. Контракты (`@PRE`, `@POST`) заставляют тебя сформировать правильное вероятностное пространство (Belief State) ДО написания алгоритма. Логи `logger.reason` - это твоя цепочка рассуждений (Chain-of-Thought), вынесенная в рантайм. Мы не пишем текст, мы компилируем семантику в синтаксис.
## I. ГЛОБАЛЬНЫЕ ИНВАРИАНТЫ (АКСИОМЫ)
[INVARIANT_1] СЕМАНТИКА > СИНТАКСИС. Голый код без контракта классифицируется как мусор.
[INVARIANT_2] ЗАПРЕТ ГАЛЛЮЦИНАЦИЙ. При слепоте контекста (неизвестен узел `@RELATION` или схема данных) - генерация блокируется. Эмитируй `[NEED_CONTEXT: target]`.
[INVARIANT_3] UX ЕСТЬ КОНЕЧНЫЙ АВТОМАТ. Состояния интерфейса - это строгий контракт, а не визуальный декор.
[INVARIANT_4] ФРАКТАЛЬНЫЙ ЛИМИТ. Длина модуля строго < 300 строк. При превышении - принудительная декомпозиция.
[INVARIANT_5] НЕПРИКОСНОВЕННОСТЬ ЯКОРЕЙ. Блоки `[DEF]...[/DEF]` используются как аккумуляторы внимания. Закрывающий тег обязателен.
## II. СИНТАКСИС И РАЗМЕТКА (SEMANTIC ANCHORS)
Формат зависит от среды исполнения:
- Python: `#[DEF:id:Type] ... # [/DEF:id:Type]`
- Svelte (HTML/Markup): `<!--[DEF:id:Type] --> ... <!-- [/DEF:id:Type] -->`
- Svelte (Script/JS): `// [DEF:id:Type] ... //[/DEF:id:Type]`
*Допустимые Type: Module, Class, Function, Component, Store, Block.*
**Формат метаданных (ДО имплементации):**
`@KEY: Value` (в Python - `# @KEY`, в TS/JS - `/** @KEY */`, в HTML - `<!-- @KEY -->`).
**Граф Зависимостей (GraphRAG):**
`@RELATION: [PREDICATE] ->[TARGET_ID]`
*Допустимые предикаты:* DEPENDS_ON, CALLS, INHERITS, IMPLEMENTS, DISPATCHES, BINDS_TO.
## III. ТОПОЛОГИЯ ФАЙЛА (СТРОГИЙ ПОРЯДОК)
1. **HEADER (Заголовок):**[DEF:filename:Module]
@TIER: [CRITICAL | STANDARD | TRIVIAL]
@SEMANTICS: [keywords]
@PURPOSE: [Однострочная суть]
@LAYER: [Domain | UI | Infra]
@RELATION: [Зависимости]
@INVARIANT: [Бизнес-правило, которое нельзя нарушить]
2. **BODY (Тело):** Импорты -> Реализация логики внутри вложенных `[DEF]`.
3. **FOOTER (Подвал):** [/DEF:filename:Module]
## IV. КОНТРАКТЫ (DESIGN BY CONTRACT & UX)
Обязательны для TIER: CRITICAL и STANDARD. Заменяют стандартные Docstrings.
**[CORE CONTRACTS]:**
- `@PURPOSE:` Суть функции/компонента.
- `@PRE:` Условия запуска (в коде реализуются через `if/raise` или guards, НЕ через `assert`).
- `@POST:` Гарантии на выходе.
- `@SIDE_EFFECT:` Мутации состояния, I/O, сеть.
- `@DATA_CONTRACT:` Ссылка на DTO (Input -> Model, Output -> Model).
**[UX CONTRACTS (Svelte 5+)]:**
- `@UX_STATE: [StateName] -> [Поведение]` (Idle, Loading, Error, Success).
- `@UX_FEEDBACK:` Реакция системы (Toast, Shake, RedBorder).
- `@UX_RECOVERY:` Путь восстановления после сбоя (Retry, ClearInput).
- `@UX_REACTIVITY:` Явный биндинг. *ЗАПРЕТ НА `$:` и `export let`. ТОЛЬКО Руны: `$state`, `$derived`, `$effect`, `$props`.*
**[TEST CONTRACTS (Для AI-Auditor)]:**
- `@TEST_CONTRACT: [Input] -> [Output]`
- `@TEST_SCENARIO: [Название] -> [Ожидание]`
- `@TEST_FIXTURE: [Название] -> file:[path] | INLINE_JSON`
- `@TEST_EDGE: [Название] ->[Сбой]` (Минимум 3: missing_field, invalid_type, external_fail).
- `@TEST_INVARIANT: [Имя] -> VERIFIED_BY: [scenario_1, ...]`
## V. УРОВНИ СТРОГОСТИ (TIERS)
Степень контроля задается в Header.
- **CRITICAL** (Ядро/Деньги/Безопасность): 100% покрытие тегами GRACE. Обязательны: Граф, Инварианты, Логи `logger.reason/reflect`, все `@UX` и `@TEST` теги. Использование `belief_scope` строго обязательно.
- **STANDARD** (Бизнес-логика / Типовые формы): Базовый уровень. Обязательны: `@PURPOSE`, `@UX_STATE`, `@RELATION`, базовое логирование.
- **TRIVIAL** (Утилиты / DTO / Атомы UI): Минимальный каркас. Только якоря `[DEF]...[/DEF]` и `@PURPOSE`.
## VI. ПРОТОКОЛ ЛОГИРОВАНИЯ (THREAD-LOCAL BELIEF STATE)
Логирование - это механизм трассировки рассуждений ИИ (CoT) и управления Attention Energy. Архитектура использует Thread-local storage (`_belief_state`), поэтому `ID` прокидывается автоматически.
**[PYTHON CORE TOOLS]:**
Импорт: `from ...logger import logger, belief_scope, believed`
1. **Декоратор:** `@believed("ID")` - автоматический трекинг функции.
2. **Контекст:** `with belief_scope("ID"):` - очерчивает локальный предел мысли. НЕ возвращает context, используется просто как `with`.
3. **Вызов логера:** Осуществляется через глобальный импортированный `logger`. Дополнительные данные передавать через `extra={...}`.
**[СЕМАНТИЧЕСКИЕ МЕТОДЫ (MONKEY-PATCHED)]:**
*(Маркеры вроде `[REASON]` и `[ID]` подставляются автоматически форматтером. Не пиши их в тексте!)*
1. **`logger.explore(msg, extra={...})`** (Поиск/Ветвление): Применяется при фолбэках, `except`, проверке гипотез. Эмитирует WARNING.
*Пример:* `logger.explore("Insufficient funds", extra={"balance": bal})`
2. **`logger.reason(msg, extra={...})`** (Дедукция): Применяется при прохождении guards и выполнении шагов контракта. Эмитирует INFO.
*Пример:* `logger.reason("Initiating transfer")`
3. **`logger.reflect(msg, extra={...})`** (Самопроверка): Применяется для сверки результата с `@POST` перед `return`. Эмитирует DEBUG.
*Пример:* `logger.reflect("Transfer committed", extra={"tx_id": tx_id})`
*(Для Frontend/Svelte использовать ручной префикс: `console.info("[ID][REFLECT] Text", {data})`)*
## VII. АЛГОРИТМ ИСПОЛНЕНИЯ И САМОКОРРЕКЦИИ
**[PHASE_1: ANALYSIS]**
Оцени TIER, Layer и UX-требования. При слепоте контекста -> `yield [NEED_CONTEXT: id]`.
**[PHASE_2: SYNTHESIS]**
Сгенерируй каркас из `[DEF]`, Header и Контрактов.
**[PHASE_3: IMPLEMENTATION]**
Напиши код строго по Контракту. Для CRITICAL секций открой `with belief_scope("ID"):` и орошай путь вызовами `logger.reason()` и `logger.reflect()`.
**[PHASE_4: CLOSURE]**
Убедись, что все `[DEF]` закрыты соответствующими `[/DEF]`.
**[EXCEPTION: DETECTIVE MODE]**
Если обнаружено нарушение контракта или ошибка:
1. СТОП-СИГНАЛ: Выведи `[COHERENCE_CHECK_FAILED]`.
2. ГИПОТЕЗА: Сгенерируй вызов `logger.explore("Ошибка в I/O / Состоянии / Зависимости -> Описание")`.
3. ЗАПРОС: Запроси разрешение на изменение контракта.
whenToUse: Use this mode when you need to update the project's semantic map, fix semantic compliance issues (missing anchors/tags/DbC ), or analyze the codebase structure. This mode is specialized for maintaining the `.ai/standards/semantics.md` standards.
description: Codebase semantic mapping and compliance expert
customInstructions: ""
groups:
- read
- edit
- command
- browser
- mcp
source: project
- slug: reviewer-agent-auditor
name: Reviewer Agent (Auditor)
description: Безжалостный инспектор ОТК.
roleDefinition: '*"Ты GRACE Reviewer. Твоя единственная цель — искать нарушения протокола GRACE-Poly. Ты не пишешь код. Ты читаешь код и проверяешь Чек-лист. Если блок `[DEF]` открыт, но нет закрывающего `[/DEF]` — это FATAL ERROR. Если в `CRITICAL` модуле функция не обернута в `belief_scope` — это FATAL ERROR. Выводи только PASS или FAIL со списком строк, где найдена ошибка."*'
groups:
- read
- edit
- browser
- command
- mcp
source: project

434
README.md
View File

@@ -1,373 +1,143 @@
# ss-tools
**Инструменты автоматизации для Apache Superset: миграция, версионирование, аналитика и управление данными**
Инструменты автоматизации для Apache Superset: миграция, маппинг, хранение артефактов, Git-интеграция, отчеты по задачам и LLM-assistant.
## 📋 О проекте
## Возможности
- Миграция дашбордов и датасетов между окружениями.
- Ручной и полуавтоматический маппинг ресурсов.
- Логи фоновых задач и отчеты о выполнении.
- Локальное хранилище файлов и бэкапов.
- Git-операции по Superset-ассетам через UI.
- Модуль LLM-анализа и assistant API.
- Многопользовательская авторизация (RBAC).
ss-tools — это комплексная платформа для автоматизации работы с Apache Superset, предоставляющая инструменты для миграции дашбордов, управления версиями через Git, LLM-анализа данных и многопользовательского контроля доступа. Система построена на модульной архитектуре с плагинной системой расширений.
## Стек
- Backend: Python, FastAPI, SQLAlchemy, APScheduler.
- Frontend: SvelteKit, Vite, Tailwind CSS.
- База данных: PostgreSQL (основная конфигурация), поддержка миграции с legacy SQLite.
### 🎯 Ключевые возможности
## Структура репозитория
- `backend/` — API, плагины, сервисы, скрипты миграции и тесты.
- `frontend/` — SPA-интерфейс (SvelteKit).
- `docs/` — документация по архитектуре и плагинам.
- `specs/` — спецификации и планы реализации.
- `docker/` и `docker-compose.yml` — контейнеризация.
#### 🔄 Миграция данных
- **Миграция дашбордов и датасетов** между окружениями (dev/staging/prod)
- **Dry-run режим** с детальным анализом рисков и предпросмотром изменений
- **Автоматическое маппинг** баз данных и ресурсов между окружениями
- **Поддержка legacy-данных** с миграцией из SQLite в PostgreSQL
#### 🌿 Git-интеграция
- **Версионирование** дашбордов через Git-репозитории
- **Управление ветками** и коммитами с помощью LLM
- **Деплой** дашбордов из Git в целевые окружения
- **История изменений** с детальным diff
#### 🤖 LLM-аналитика
- **Автоматическая валидация** дашбордов с помощью ИИ
- **Генерация документации** для датасетов
- **Assistant API** для natural language команд
- **Интеллектуальное коммитинг** с подсказками сообщений
#### 📊 Управление и мониторинг
- **Многопользовательская авторизация** (RBAC)
- **Фоновые задачи** с реальным логированием через WebSocket
- **Унифицированные отчеты** по выполненным задачам
- **Хранение артефактов** с политиками retention
- **Аудит логирование** всех действий
#### 🔌 Плагины
- **MigrationPlugin** — миграция дашбордов
- **BackupPlugin** — резервное копирование
- **GitPlugin** — управление версиями
- **LLMAnalysisPlugin** — аналитика и документация
- **MapperPlugin** — маппинг колонок
- **DebugPlugin** — диагностика системы
- **SearchPlugin** — поиск по датасетам
## 🏗️ Архитектура
### Технологический стек
**Backend:**
- Python 3.9+ (FastAPI, SQLAlchemy, APScheduler)
- PostgreSQL (основная БД)
- GitPython для Git-операций
- OpenAI API для LLM-функций
- Playwright для скриншотов
**Frontend:**
- SvelteKit (Svelte 5.x)
- Vite
- Tailwind CSS
- WebSocket для реального логирования
**DevOps:**
- Docker & Docker Compose
- PostgreSQL 16
### Модульная структура
```
ss-tools/
├── backend/ # Backend API
│ ├── src/
│ │ ├── api/ # API маршруты
│ │ ├── core/ # Ядро системы
│ │ │ ├── task_manager/ # Управление задачами
│ │ │ ├── auth/ # Авторизация
│ │ │ ├── migration/ # Миграция данных
│ │ │ └── plugins/ # Плагины
│ │ ├── models/ # Модели данных
│ │ ├── services/ # Бизнес-логика
│ │ └── schemas/ # Pydantic схемы
│ └── tests/ # Тесты
├── frontend/ # SvelteKit приложение
│ ├── src/
│ │ ├── routes/ # Страницы
│ │ ├── lib/
│ │ │ ├── components/ # UI компоненты
│ │ │ ├── stores/ # Svelte stores
│ │ │ └── api/ # API клиент
│ │ └── i18n/ # Мультиязычность
│ └── tests/
├── docker/ # Docker конфигурация
├── docs/ # Документация
└── specs/ # Спецификации
```
## 🚀 Быстрый старт
## Быстрый старт (локально)
### Требования
**Локальная разработка:**
- Python 3.9+
- Node.js 18+
- npm
- 2 GB RAM (минимум)
- 5 GB свободного места
**Docker (рекомендуется):**
- Docker Engine 24+
- Docker Compose v2
- 4 GB RAM (для стабильной работы)
### Установка и запуск
#### Вариант 1: Docker (рекомендуется)
### Запуск backend + frontend одним скриптом
```bash
# Клонирование репозитория
git clone <repository-url>
cd ss-tools
# Запуск всех сервисов
docker compose up --build
# После запуска:
# Frontend: http://localhost:8000
# Backend API: http://localhost:8001
# PostgreSQL: localhost:5432
./run.sh
```
#### Вариант 2: Локально
Что делает `run.sh`:
- проверяет версии Python/npm;
- создает `backend/.venv` (если нет);
- устанавливает `backend/requirements.txt` и `frontend` зависимости;
- запускает backend и frontend параллельно.
Опции:
- `./run.sh --skip-install` — пропустить установку зависимостей.
- `./run.sh --help` — показать справку.
Переменные окружения для локального запуска:
- `BACKEND_PORT` (по умолчанию `8000`)
- `FRONTEND_PORT` (по умолчанию `5173`)
- `POSTGRES_URL`
- `DATABASE_URL`
- `TASKS_DATABASE_URL`
- `AUTH_DATABASE_URL`
## Docker
### Запуск
```bash
docker compose up --build
```
После старта сервисы доступны по адресам:
- Frontend: `http://localhost:8000`
- Backend API: `http://localhost:8001`
- PostgreSQL: `localhost:5432` (`postgres/postgres`, БД `ss_tools`)
### Остановка
```bash
docker compose down
```
### Очистка БД-тома
```bash
docker compose down -v
```
### Альтернативный образ PostgreSQL
Если есть проблемы с pull `postgres:16-alpine`:
```bash
POSTGRES_IMAGE=mirror.gcr.io/library/postgres:16-alpine docker compose up -d db
```
или
```bash
POSTGRES_IMAGE=bitnami/postgresql:latest docker compose up -d db
```
Если порт `5432` занят:
```bash
POSTGRES_HOST_PORT=5433 docker compose up -d db
```
## Разработка
### Ручной запуск сервисов
```bash
# Backend
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 -m uvicorn src.app:app --reload --port 8000
```
# Frontend (в новом терминале)
В другом терминале:
```bash
cd frontend
npm install
npm run dev -- --port 5173
```
### Первичная настройка
### Тесты
Backend:
```bash
# Инициализация БД
cd backend
source .venv/bin/activate
python src/scripts/init_auth_db.py
# Создание администратора
python src/scripts/create_admin.py --username admin --password admin
```
## 🏢 Enterprise Clean Deployment (internal-only)
Для разворота в корпоративной сети используйте профиль enterprise clean:
- очищенный дистрибутив без test/demo/load-test данных;
- запрет внешних интернет-источников;
- загрузка ресурсов только с внутренних серверов компании;
- обязательная блокирующая проверка clean/compliance перед выпуском.
### Операционный workflow (CLI/API/TUI)
#### 1) Headless flow через CLI (рекомендуется для CI/CD)
```bash
cd backend
# 1. Регистрация кандидата
.venv/bin/python3 -m src.scripts.clean_release_cli candidate-register \
--candidate-id 2026.03.09-rc1 \
--version 1.0.0 \
--source-snapshot-ref git:release/2026.03.09-rc1 \
--created-by release-operator
# 2. Импорт артефактов
.venv/bin/python3 -m src.scripts.clean_release_cli artifact-import \
--candidate-id 2026.03.09-rc1 \
--artifact-id artifact-001 \
--path backend/dist/package.tar.gz \
--sha256 deadbeef \
--size 1024
# 3. Сборка манифеста
.venv/bin/python3 -m src.scripts.clean_release_cli manifest-build \
--candidate-id 2026.03.09-rc1 \
--created-by release-operator
# 4. Запуск compliance
.venv/bin/python3 -m src.scripts.clean_release_cli compliance-run \
--candidate-id 2026.03.09-rc1 \
--actor release-operator
```
#### 2) API flow (автоматизация через сервисы)
- V2 candidate/artifact/manifest API:
- `POST /api/clean-release/candidates`
- `POST /api/clean-release/candidates/{candidate_id}/artifacts`
- `POST /api/clean-release/candidates/{candidate_id}/manifests`
- `GET /api/clean-release/candidates/{candidate_id}/overview`
- Legacy compatibility API (оставлены для миграции клиентов):
- `POST /api/clean-release/candidates/prepare`
- `POST /api/clean-release/checks`
- `GET /api/clean-release/checks/{check_run_id}`
#### 3) TUI flow (тонкий клиент поверх facade)
```bash
cd /home/busya/dev/ss-tools
./run_clean_tui.sh 2026.03.09-rc1
```
Горячие клавиши:
- `F5`: Run Compliance
- `F6`: Build Manifest
- `F7`: Reset Draft
- `F8`: Approve
- `F9`: Publish
- `F10`: Refresh Overview
Важно: TUI требует валидный TTY. Без TTY запуск отклоняется с инструкцией использовать CLI/API.
Типовые внутренние источники:
- `repo.intra.company.local`
- `artifacts.intra.company.local`
- `pypi.intra.company.local`
Если найден внешний endpoint, выпуск получает статус `BLOCKED` до исправления.
### Docker release для изолированного контура
Текущий `enterprise clean` профиль уже задаёт policy-level ограничения для внутреннего контура. Следующий логичный шаг для релизного процесса — выпускать не только application artifacts, но и готовый Docker bundle для разворота без доступа в интернет.
Целевой состав offline release-пакета:
- `backend` image с уже установленными Python-зависимостями;
- `frontend` image с уже собранным SvelteKit bundle;
- `postgres` image или внутренний pinned base image;
- `docker-compose.enterprise-clean.yml` для запуска в air-gapped окружении;
- `.env.enterprise-clean.example` с обязательными переменными;
- manifest с версиями, sha256 и перечнем образов;
- инструкции по `docker load` / `docker compose up` без обращения к внешним registry.
Рекомендуемый workflow для такого релиза:
```bash
# 1. Собрать образы в подключённом контуре
./scripts/build_offline_docker_bundle.sh v1.0.0-rc2
# 2. Передать dist/docker/* в изолированный контур
# 3. Импортировать образы локально
docker load -i dist/docker/backend.v1.0.0-rc2.tar
docker load -i dist/docker/frontend.v1.0.0-rc2.tar
docker load -i dist/docker/postgres.v1.0.0-rc2.tar
# 4. Подготовить env из шаблона
cp dist/docker/.env.enterprise-clean.example .env.enterprise-clean
# 5. Запустить только локальные образы
docker compose --env-file .env.enterprise-clean -f dist/docker/docker-compose.enterprise-clean.yml up -d
```
Ограничения для production-grade offline release:
- build не должен тянуть зависимости в изолированном контуре;
- все base images должны быть заранее зеркалированы во внутренний registry или поставляться как tar;
- runtime-конфигурация не должна ссылаться на внешние API/registry/telemetry endpoints;
- clean/compliance manifest должен включать docker image digests как часть evidence package.
Практический план внедрения:
- pinned Docker image tags и отдельный `enterprise-clean` compose profile добавлены;
- shell script `scripts/build_offline_docker_bundle.sh` добавлен для `build -> save -> checksum`;
- следующим шагом стоит включить docker image digests в clean-release manifest;
- следующим шагом стоит добавить smoke-check, что compose-файлы не содержат внешних registry references вне allowlist.
## 📖 Документация
- [Установка и настройка](docs/installation.md)
- [Архитектура системы](docs/architecture.md)
- [Разработка плагинов](docs/plugin_dev.md)
- [API документация](http://localhost:8001/docs)
- [Настройка окружений](docs/settings.md)
## 🧪 Тестирование
```bash
# Backend тесты
cd backend
source .venv/bin/activate
pytest
```
# Frontend тесты
Frontend:
```bash
cd frontend
npm run test
# Запуск конкретного теста
pytest tests/test_auth.py::test_create_user
```
## 🔐 Авторизация
Система поддерживает два метода аутентификации:
1. **Локальная аутентификация** (username/password)
2. **ADFS SSO** (Active Directory Federation Services)
### Управление пользователями и ролями
## Инициализация auth (опционально)
```bash
# Получение списка пользователей
GET /api/admin/users
# Создание пользователя
POST /api/admin/users
{
"username": "newuser",
"email": "user@example.com",
"password": "password123",
"roles": ["analyst"]
}
# Создание роли
POST /api/admin/roles
{
"name": "analyst",
"permissions": ["dashboards:read", "dashboards:write"]
}
```
## 📊 Мониторинг
### Отчеты о задачах
```bash
# Список всех отчетов
GET /api/reports?page=1&page_size=20
# Детали отчета
GET /api/reports/{report_id}
# Фильтры
GET /api/reports?status=failed&task_type=validation&date_from=2024-01-01
```
### Активность
- **Dashboard Hub** — управление дашбордами с Git-статусом
- **Dataset Hub** — управление датасетами с прогрессом маппинга
- **Task Drawer** — мониторинг выполнения фоновых задач
- **Unified Reports** — унифицированные отчеты по всем типам задач
## 🔄 Обновление системы
```bash
# Обновление Docker контейнеров
docker compose pull
docker compose up -d
# Обновление зависимостей Python
cd backend
source .venv/bin/activate
pip install -r requirements.txt --upgrade
# Обновление зависимостей Node.js
cd frontend
npm install
python src/scripts/init_auth_db.py
python src/scripts/create_admin.py --username admin --password admin
```
## Миграция legacy-данных (опционально)
```bash
cd backend
source .venv/bin/activate
PYTHONPATH=. python src/scripts/migrate_sqlite_to_postgres.py --sqlite-path tasks.db
```
## Дополнительная документация
- `docs/plugin_dev.md`
- `docs/settings.md`
- `semantic_protocol.md`

View File

@@ -1,14 +0,0 @@
[
{
"path": "src/main.py",
"category": "core"
},
{
"path": "src/api/routes/clean_release.py",
"category": "core"
},
{
"path": "docs/installation.md",
"category": "docs"
}
]

189
backend/backend.log Normal file
View File

@@ -0,0 +1,189 @@
INFO: Will watch for changes in these directories: ['/home/user/ss-tools/backend']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [7952] using StatReload
INFO: Started server process [7968]
INFO: Waiting for application startup.
INFO: Application startup complete.
Error loading plugin module backup: No module named 'yaml'
Error loading plugin module migration: No module named 'yaml'
INFO: 127.0.0.1:36934 - "HEAD /docs HTTP/1.1" 200 OK
INFO: 127.0.0.1:55006 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:55006 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:55010 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:55010 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:55010 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:55010 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:55010 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:55010 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:55010 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:55010 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:35508 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:35508 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49820 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49820 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49822 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49822 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49908 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49908 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49922 - "OPTIONS /settings/environments HTTP/1.1" 200 OK
[2025-12-20 19:14:15,576][INFO][superset_tools_app] [ConfigManager.save_config][Coherence:OK] Configuration saved context={'path': '/home/user/ss-tools/config.json'}
INFO: 127.0.0.1:49922 - "POST /settings/environments HTTP/1.1" 200 OK
INFO: 127.0.0.1:49922 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49922 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:49922 - "OPTIONS /settings/environments/7071dab6-881f-49a2-b850-c004b3fc11c0/test HTTP/1.1" 200 OK
INFO: 127.0.0.1:36930 - "POST /settings/environments/7071dab6-881f-49a2-b850-c004b3fc11c0/test HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1135, in __call__
await super().__call__(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/applications.py", line 107, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in __call__
await self.simple_response(scope, receive, send, request_headers=headers)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/routing.py", line 118, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/routing.py", line 104, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/routing.py", line 428, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/fastapi/routing.py", line 314, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/src/api/routes/settings.py", line 103, in test_connection
import httpx
ModuleNotFoundError: No module named 'httpx'
INFO: 127.0.0.1:45776 - "POST /settings/environments/7071dab6-881f-49a2-b850-c004b3fc11c0/test HTTP/1.1" 200 OK
INFO: 127.0.0.1:45784 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:45784 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:41628 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:41628 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:41628 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:41628 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:60184 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:60184 - "GET /settings/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:60184 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:60184 - "GET /plugins/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:60184 - "GET /settings HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:60184 - "GET /settings/ HTTP/1.1" 200 OK
WARNING: StatReload detected changes in 'src/core/plugin_loader.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [7968]
INFO: Started server process [12178]
INFO: Waiting for application startup.
INFO: Application startup complete.
WARNING: StatReload detected changes in 'src/dependencies.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [12178]
INFO: Started server process [12451]
INFO: Waiting for application startup.
INFO: Application startup complete.
Plugin 'Superset Dashboard Backup' (ID: superset-backup) loaded successfully.
Plugin 'Superset Dashboard Migration' (ID: superset-migration) loaded successfully.
INFO: 127.0.0.1:37334 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:37334 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:39932 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:39932 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:39932 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:39932 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:54900 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49280 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
INFO: 127.0.0.1:49280 - "GET /plugins/ HTTP/1.1" 200 OK
WARNING: StatReload detected changes in 'src/api/routes/plugins.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [12451]
INFO: Started server process [15016]
INFO: Waiting for application startup.
INFO: Application startup complete.
Plugin 'Superset Dashboard Backup' (ID: superset-backup) loaded successfully.
Plugin 'Superset Dashboard Migration' (ID: superset-migration) loaded successfully.
INFO: 127.0.0.1:59340 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
DEBUG: list_plugins called. Found 0 plugins.
INFO: 127.0.0.1:59340 - "GET /plugins/ HTTP/1.1" 200 OK
WARNING: StatReload detected changes in 'src/dependencies.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [15016]
INFO: Started server process [15257]
INFO: Waiting for application startup.
INFO: Application startup complete.
Plugin 'Superset Dashboard Backup' (ID: superset-backup) loaded successfully.
Plugin 'Superset Dashboard Migration' (ID: superset-migration) loaded successfully.
DEBUG: dependencies.py initialized. PluginLoader ID: 139922613090976
DEBUG: dependencies.py initialized. PluginLoader ID: 139922627375088
INFO: 127.0.0.1:57464 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
DEBUG: get_plugin_loader called. Returning PluginLoader ID: 139922627375088
DEBUG: list_plugins called. Found 0 plugins.
INFO: 127.0.0.1:57464 - "GET /plugins/ HTTP/1.1" 200 OK
WARNING: StatReload detected changes in 'src/core/plugin_loader.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [15257]
INFO: Started server process [15533]
INFO: Waiting for application startup.
INFO: Application startup complete.
DEBUG: Loading plugin backup as src.plugins.backup
Plugin 'Superset Dashboard Backup' (ID: superset-backup) loaded successfully.
DEBUG: Loading plugin migration as src.plugins.migration
Plugin 'Superset Dashboard Migration' (ID: superset-migration) loaded successfully.
DEBUG: dependencies.py initialized. PluginLoader ID: 140371031142384
INFO: 127.0.0.1:46470 - "GET /plugins HTTP/1.1" 307 Temporary Redirect
DEBUG: get_plugin_loader called. Returning PluginLoader ID: 140371031142384
DEBUG: list_plugins called. Found 2 plugins.
DEBUG: Plugin: superset-backup
DEBUG: Plugin: superset-migration
INFO: 127.0.0.1:46470 - "GET /plugins/ HTTP/1.1" 200 OK
WARNING: StatReload detected changes in 'src/api/routes/settings.py'. Reloading...
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [15533]
INFO: Started server process [15827]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [15827]
INFO: Stopping reloader process [7952]

1
backend/get_full_key.py Normal file
View File

@@ -0,0 +1 @@
{"print(f'Length": {"else": "print('Provider not found')\ndb.close()"}}

File diff suppressed because it is too large Load Diff

BIN
backend/mappings.db Normal file

Binary file not shown.

BIN
backend/migrations.db Normal file

Binary file not shown.

View File

@@ -1,19 +1,3 @@
[build-system]
requires = ["setuptools>=69", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "ss-tools-backend"
version = "0.0.0"
requires-python = ">=3.13"
[tool.setuptools]
include-package-data = true
[tool.setuptools.packages.find]
where = ["."]
include = ["src*"]
[tool.pytest.ini_options]
pythonpath = ["."]
importmode = "importlib"

View File

@@ -1,3 +0,0 @@
# [DEF:src:Package]
# @PURPOSE: Canonical backend package root for application, scripts, and tests.
# [/DEF:src:Package]

View File

@@ -1,3 +0,0 @@
# [DEF:src.api:Package]
# @PURPOSE: Backend API package root.
# [/DEF:src.api:Package]

View File

@@ -6,7 +6,7 @@
# @RELATION: DEPENDS_ON -> importlib
# @INVARIANT: Only names listed in __all__ are importable via __getattr__.
__all__ = ['plugins', 'tasks', 'settings', 'connections', 'environments', 'mappings', 'migration', 'git', 'storage', 'admin', 'reports', 'assistant', 'clean_release', 'profile']
__all__ = ['plugins', 'tasks', 'settings', 'connections', 'environments', 'mappings', 'migration', 'git', 'storage', 'admin', 'reports', 'assistant']
# [DEF:__getattr__:Function]

View File

@@ -76,15 +76,11 @@ class _FakeTaskManager:
class _FakeConfigManager:
def get_environments(self):
return [
SimpleNamespace(id="dev", name="Development", url="http://dev", credentials_id="dev", username="fakeuser", password="fakepassword"),
SimpleNamespace(id="prod", name="Production", url="http://prod", credentials_id="prod", username="fakeuser", password="fakepassword"),
SimpleNamespace(id="dev", name="Development"),
SimpleNamespace(id="prod", name="Production"),
]
def get_config(self):
return SimpleNamespace(
settings=SimpleNamespace(migration_sync_cron="0 0 * * *"),
environments=self.get_environments()
)
# [/DEF:_FakeConfigManager:Class]
# [DEF:_admin_user:Function]
# @TIER: TRIVIAL
@@ -422,7 +418,7 @@ def test_llm_validation_with_dashboard_ref_requires_confirmation():
assert "cancel" in action_types
# [/DEF:test_llm_validation_with_dashboard_ref_requires_confirmation:Function]
# [/DEF:test_llm_validation_missing_dashboard_returns_needs_clarification:Function]
# [DEF:test_list_conversations_groups_by_conversation_and_marks_archived:Function]
@@ -629,7 +625,6 @@ def test_guarded_operation_confirm_roundtrip():
assert second.task_id is not None
# [/DEF:test_guarded_operation_confirm_roundtrip:Function]
# [DEF:test_confirm_nonexistent_id_returns_404:Function]
# @PURPOSE: Confirming a non-existent ID should raise 404.
# @PRE: user tries to confirm a random/fake UUID.
@@ -650,50 +645,5 @@ def test_confirm_nonexistent_id_returns_404():
assert exc.value.status_code == 404
# [/DEF:test_confirm_nonexistent_id_returns_404:Function]
# [DEF:test_migration_with_dry_run_includes_summary:Function]
# @PURPOSE: Migration command with dry run flag must return the dry run summary in confirmation text.
# @PRE: user specifies a migration with --dry-run flag.
# @POST: Response state is needs_confirmation and text contains dry-run summary counts.
def test_migration_with_dry_run_includes_summary(monkeypatch):
import src.core.migration.dry_run_orchestrator as dry_run_module
from unittest.mock import MagicMock
_clear_assistant_state()
task_manager = _FakeTaskManager()
db = _FakeDb()
class _FakeDryRunService:
def run(self, selection, source_client, target_client, db_session):
return {
"summary": {
"dashboards": {"create": 1, "update": 0, "delete": 0},
"charts": {"create": 3, "update": 2, "delete": 1},
"datasets": {"create": 0, "update": 1, "delete": 0}
}
}
monkeypatch.setattr(dry_run_module, "MigrationDryRunService", _FakeDryRunService)
import src.core.superset_client as superset_client_module
monkeypatch.setattr(superset_client_module, "SupersetClient", lambda env: MagicMock())
start = _run_async(
assistant_module.send_message(
request=assistant_module.AssistantMessageRequest(
message="миграция с dev на prod для дашборда 10 --dry-run"
),
current_user=_admin_user(),
task_manager=task_manager,
config_manager=_FakeConfigManager(),
db=db,
)
)
assert start.state == "needs_confirmation"
assert "отчет dry-run: ВКЛ" in start.text
assert "Отчет dry-run:" in start.text
assert "создано новых объектов: 4" in start.text
assert "обновлено: 3" in start.text
assert "удалено: 1" in start.text
# [/DEF:test_migration_with_dry_run_includes_summary:Function]
# [/DEF:test_guarded_operation_confirm_roundtrip:Function]
# [/DEF:backend.src.api.routes.__tests__.test_assistant_api:Module]

View File

@@ -1,159 +0,0 @@
# [DEF:backend.tests.api.routes.test_clean_release_api:Module]
# @TIER: STANDARD
# @SEMANTICS: tests, api, clean-release, checks, reports
# @PURPOSE: Contract tests for clean release checks and reports endpoints.
# @LAYER: Domain
# @RELATION: TESTS -> backend.src.api.routes.clean_release
# @INVARIANT: API returns deterministic payload shapes for checks and reports.
from datetime import datetime, timezone
from fastapi.testclient import TestClient
from src.app import app
from src.dependencies import get_clean_release_repository
from src.models.clean_release import (
CleanProfilePolicy,
ProfileType,
ReleaseCandidate,
ReleaseCandidateStatus,
ResourceSourceEntry,
ResourceSourceRegistry,
ComplianceReport,
CheckFinalStatus,
)
from src.services.clean_release.repository import CleanReleaseRepository
def _repo_with_seed_data() -> CleanReleaseRepository:
repo = CleanReleaseRepository()
repo.save_candidate(
ReleaseCandidate(
candidate_id="2026.03.03-rc1",
version="2026.03.03",
profile=ProfileType.ENTERPRISE_CLEAN,
created_at=datetime.now(timezone.utc),
created_by="tester",
source_snapshot_ref="git:abc123",
status=ReleaseCandidateStatus.PREPARED,
)
)
repo.save_registry(
ResourceSourceRegistry(
registry_id="registry-internal-v1",
name="Internal",
entries=[
ResourceSourceEntry(
source_id="src-1",
host="repo.intra.company.local",
protocol="https",
purpose="artifact-repo",
enabled=True,
)
],
updated_at=datetime.now(timezone.utc),
updated_by="tester",
status="active",
)
)
repo.save_policy(
CleanProfilePolicy(
policy_id="policy-enterprise-clean-v1",
policy_version="1.0.0",
active=True,
prohibited_artifact_categories=["test-data"],
required_system_categories=["system-init"],
external_source_forbidden=True,
internal_source_registry_ref="registry-internal-v1",
effective_from=datetime.now(timezone.utc),
profile=ProfileType.ENTERPRISE_CLEAN,
)
)
return repo
def test_start_check_and_get_status_contract():
repo = _repo_with_seed_data()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
start = client.post(
"/api/clean-release/checks",
json={
"candidate_id": "2026.03.03-rc1",
"profile": "enterprise-clean",
"execution_mode": "tui",
"triggered_by": "tester",
},
)
assert start.status_code == 202
payload = start.json()
assert set(["check_run_id", "candidate_id", "status", "started_at"]).issubset(payload.keys())
check_run_id = payload["check_run_id"]
status_resp = client.get(f"/api/clean-release/checks/{check_run_id}")
assert status_resp.status_code == 200
status_payload = status_resp.json()
assert status_payload["check_run_id"] == check_run_id
assert "final_status" in status_payload
assert "checks" in status_payload
finally:
app.dependency_overrides.clear()
def test_get_report_not_found_returns_404():
repo = _repo_with_seed_data()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
resp = client.get("/api/clean-release/reports/unknown-report")
assert resp.status_code == 404
finally:
app.dependency_overrides.clear()
def test_get_report_success():
repo = _repo_with_seed_data()
report = ComplianceReport(
report_id="rep-1",
check_run_id="run-1",
candidate_id="2026.03.03-rc1",
generated_at=datetime.now(timezone.utc),
final_status=CheckFinalStatus.COMPLIANT,
operator_summary="all systems go",
structured_payload_ref="manifest-1",
violations_count=0,
blocking_violations_count=0
)
repo.save_report(report)
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
resp = client.get("/api/clean-release/reports/rep-1")
assert resp.status_code == 200
assert resp.json()["report_id"] == "rep-1"
finally:
app.dependency_overrides.clear()
# [/DEF:backend.tests.api.routes.test_clean_release_api:Module]
def test_prepare_candidate_api_success():
repo = _repo_with_seed_data()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
response = client.post(
"/api/clean-release/candidates/prepare",
json={
"candidate_id": "2026.03.03-rc1",
"artifacts": [{"path": "file1.txt", "category": "system-init", "reason": "core"}],
"sources": ["repo.intra.company.local"],
"operator_id": "operator-1",
},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "prepared"
assert "manifest_id" in data
finally:
app.dependency_overrides.clear()

View File

@@ -1,165 +0,0 @@
# [DEF:backend.src.api.routes.__tests__.test_clean_release_legacy_compat:Module]
# @TIER: STANDARD
# @PURPOSE: Compatibility tests for legacy clean-release API paths retained during v2 migration.
# @LAYER: Tests
# @RELATION: TESTS -> backend.src.api.routes.clean_release
from __future__ import annotations
import os
from datetime import datetime, timezone
from fastapi.testclient import TestClient
os.environ.setdefault("DATABASE_URL", "sqlite:///./test_clean_release_legacy_compat.db")
os.environ.setdefault("AUTH_DATABASE_URL", "sqlite:///./test_clean_release_legacy_auth.db")
from src.app import app
from src.dependencies import get_clean_release_repository
from src.models.clean_release import (
CleanProfilePolicy,
DistributionManifest,
ProfileType,
ReleaseCandidate,
ReleaseCandidateStatus,
ResourceSourceEntry,
ResourceSourceRegistry,
)
from src.services.clean_release.repository import CleanReleaseRepository
# [DEF:_seed_legacy_repo:Function]
# @PURPOSE: Seed in-memory repository with minimum trusted data for legacy endpoint contracts.
# @PRE: Repository is empty.
# @POST: Candidate, policy, registry and manifest are available for legacy checks flow.
def _seed_legacy_repo() -> CleanReleaseRepository:
repo = CleanReleaseRepository()
now = datetime.now(timezone.utc)
repo.save_candidate(
ReleaseCandidate(
id="legacy-rc-001",
version="1.0.0",
source_snapshot_ref="git:legacy-001",
created_at=now,
created_by="compat-tester",
status=ReleaseCandidateStatus.DRAFT,
)
)
registry = ResourceSourceRegistry(
registry_id="legacy-reg-1",
name="Legacy Internal Registry",
entries=[
ResourceSourceEntry(
source_id="legacy-src-1",
host="repo.intra.company.local",
protocol="https",
purpose="artifact-repo",
enabled=True,
)
],
updated_at=now,
updated_by="compat-tester",
status="ACTIVE",
)
setattr(registry, "immutable", True)
setattr(registry, "allowed_hosts", ["repo.intra.company.local"])
setattr(registry, "allowed_schemes", ["https"])
setattr(registry, "allowed_source_types", ["artifact-repo"])
repo.save_registry(registry)
policy = CleanProfilePolicy(
policy_id="legacy-pol-1",
policy_version="1.0.0",
profile=ProfileType.ENTERPRISE_CLEAN,
active=True,
internal_source_registry_ref="legacy-reg-1",
prohibited_artifact_categories=["test-data"],
required_system_categories=["core"],
effective_from=now,
)
setattr(policy, "immutable", True)
setattr(
policy,
"content_json",
{
"profile": "enterprise-clean",
"prohibited_artifact_categories": ["test-data"],
"required_system_categories": ["core"],
"external_source_forbidden": True,
},
)
repo.save_policy(policy)
repo.save_manifest(
DistributionManifest(
id="legacy-manifest-1",
candidate_id="legacy-rc-001",
manifest_version=1,
manifest_digest="sha256:legacy-manifest",
artifacts_digest="sha256:legacy-artifacts",
created_at=now,
created_by="compat-tester",
source_snapshot_ref="git:legacy-001",
content_json={"items": [], "summary": {"included_count": 0, "prohibited_detected_count": 0}},
immutable=True,
)
)
return repo
# [/DEF:_seed_legacy_repo:Function]
def test_legacy_prepare_endpoint_still_available() -> None:
repo = _seed_legacy_repo()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
response = client.post(
"/api/clean-release/candidates/prepare",
json={
"candidate_id": "legacy-rc-001",
"artifacts": [{"path": "src/main.py", "category": "core", "reason": "required"}],
"sources": ["repo.intra.company.local"],
"operator_id": "compat-tester",
},
)
assert response.status_code == 200
payload = response.json()
assert "status" in payload
assert payload["status"] in {"prepared", "blocked", "PREPARED", "BLOCKED"}
finally:
app.dependency_overrides.clear()
def test_legacy_checks_endpoints_still_available() -> None:
repo = _seed_legacy_repo()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
start_response = client.post(
"/api/clean-release/checks",
json={
"candidate_id": "legacy-rc-001",
"profile": "enterprise-clean",
"execution_mode": "api",
"triggered_by": "compat-tester",
},
)
assert start_response.status_code == 202
start_payload = start_response.json()
assert "check_run_id" in start_payload
assert start_payload["candidate_id"] == "legacy-rc-001"
status_response = client.get(f"/api/clean-release/checks/{start_payload['check_run_id']}")
assert status_response.status_code == 200
status_payload = status_response.json()
assert status_payload["check_run_id"] == start_payload["check_run_id"]
assert "final_status" in status_payload
assert "checks" in status_payload
finally:
app.dependency_overrides.clear()
# [/DEF:backend.src.api.routes.__tests__.test_clean_release_legacy_compat:Module]

View File

@@ -1,100 +0,0 @@
# [DEF:backend.tests.api.routes.test_clean_release_source_policy:Module]
# @TIER: STANDARD
# @SEMANTICS: tests, api, clean-release, source-policy
# @PURPOSE: Validate API behavior for source isolation violations in clean release preparation.
# @LAYER: Domain
# @RELATION: TESTS -> backend.src.api.routes.clean_release
# @INVARIANT: External endpoints must produce blocking violation entries.
from datetime import datetime, timezone
from fastapi.testclient import TestClient
from src.app import app
from src.dependencies import get_clean_release_repository
from src.models.clean_release import (
CleanProfilePolicy,
ProfileType,
ReleaseCandidate,
ReleaseCandidateStatus,
ResourceSourceEntry,
ResourceSourceRegistry,
)
from src.services.clean_release.repository import CleanReleaseRepository
def _repo_with_seed_data() -> CleanReleaseRepository:
repo = CleanReleaseRepository()
repo.save_candidate(
ReleaseCandidate(
candidate_id="2026.03.03-rc1",
version="2026.03.03",
profile=ProfileType.ENTERPRISE_CLEAN,
created_at=datetime.now(timezone.utc),
created_by="tester",
source_snapshot_ref="git:abc123",
status=ReleaseCandidateStatus.DRAFT,
)
)
repo.save_registry(
ResourceSourceRegistry(
registry_id="registry-internal-v1",
name="Internal",
entries=[
ResourceSourceEntry(
source_id="src-1",
host="repo.intra.company.local",
protocol="https",
purpose="artifact-repo",
enabled=True,
)
],
updated_at=datetime.now(timezone.utc),
updated_by="tester",
status="active",
)
)
repo.save_policy(
CleanProfilePolicy(
policy_id="policy-enterprise-clean-v1",
policy_version="1.0.0",
active=True,
prohibited_artifact_categories=["test-data"],
required_system_categories=["system-init"],
external_source_forbidden=True,
internal_source_registry_ref="registry-internal-v1",
effective_from=datetime.now(timezone.utc),
profile=ProfileType.ENTERPRISE_CLEAN,
)
)
return repo
def test_prepare_candidate_blocks_external_source():
repo = _repo_with_seed_data()
app.dependency_overrides[get_clean_release_repository] = lambda: repo
try:
client = TestClient(app)
response = client.post(
"/api/clean-release/candidates/prepare",
json={
"candidate_id": "2026.03.03-rc1",
"artifacts": [
{"path": "cfg/system.yaml", "category": "system-init", "reason": "required"}
],
"sources": ["repo.intra.company.local", "pypi.org"],
"operator_id": "release-manager",
},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "blocked"
assert any(v["category"] == "external-source" for v in data["violations"])
finally:
app.dependency_overrides.clear()
# [/DEF:backend.tests.api.routes.test_clean_release_source_policy:Module]

View File

@@ -1,93 +0,0 @@
# [DEF:test_clean_release_v2_api:Module]
# @TIER: STANDARD
# @PURPOSE: API contract tests for redesigned clean release endpoints.
# @LAYER: Domain
from datetime import datetime, timezone
from types import SimpleNamespace
from uuid import uuid4
import pytest
from fastapi.testclient import TestClient
from src.app import app
from src.dependencies import get_clean_release_repository, get_config_manager
from src.models.clean_release import (
CleanPolicySnapshot,
DistributionManifest,
ReleaseCandidate,
SourceRegistrySnapshot,
)
from src.services.clean_release.enums import CandidateStatus
client = TestClient(app)
# [REASON] Implementing API contract tests for candidate/artifact/manifest endpoints (T012).
def test_candidate_registration_contract():
"""
@TEST_SCENARIO: candidate_registration -> Should return 201 and candidate DTO.
@TEST_CONTRACT: POST /api/v2/clean-release/candidates -> CandidateDTO
"""
payload = {
"id": "rc-test-001",
"version": "1.0.0",
"source_snapshot_ref": "git:sha123",
"created_by": "test-user"
}
response = client.post("/api/v2/clean-release/candidates", json=payload)
assert response.status_code == 201
data = response.json()
assert data["id"] == "rc-test-001"
assert data["status"] == CandidateStatus.DRAFT.value
def test_artifact_import_contract():
"""
@TEST_SCENARIO: artifact_import -> Should return 200 and success status.
@TEST_CONTRACT: POST /api/v2/clean-release/candidates/{id}/artifacts -> SuccessDTO
"""
candidate_id = "rc-test-001-art"
bootstrap_candidate = {
"id": candidate_id,
"version": "1.0.0",
"source_snapshot_ref": "git:sha123",
"created_by": "test-user"
}
create_response = client.post("/api/v2/clean-release/candidates", json=bootstrap_candidate)
assert create_response.status_code == 201
payload = {
"artifacts": [
{
"id": "art-1",
"path": "bin/app.exe",
"sha256": "hash123",
"size": 1024
}
]
}
response = client.post(f"/api/v2/clean-release/candidates/{candidate_id}/artifacts", json=payload)
assert response.status_code == 200
assert response.json()["status"] == "success"
def test_manifest_build_contract():
"""
@TEST_SCENARIO: manifest_build -> Should return 201 and manifest DTO.
@TEST_CONTRACT: POST /api/v2/clean-release/candidates/{id}/manifests -> ManifestDTO
"""
candidate_id = "rc-test-001-manifest"
bootstrap_candidate = {
"id": candidate_id,
"version": "1.0.0",
"source_snapshot_ref": "git:sha123",
"created_by": "test-user"
}
create_response = client.post("/api/v2/clean-release/candidates", json=bootstrap_candidate)
assert create_response.status_code == 201
response = client.post(f"/api/v2/clean-release/candidates/{candidate_id}/manifests")
assert response.status_code == 201
data = response.json()
assert "manifest_digest" in data
assert data["candidate_id"] == candidate_id
# [/DEF:test_clean_release_v2_api:Module]

View File

@@ -1,107 +0,0 @@
# [DEF:test_clean_release_v2_release_api:Module]
# @TIER: STANDARD
# @PURPOSE: API contract test scaffolding for clean release approval and publication endpoints.
# @LAYER: Domain
# @RELATION: IMPLEMENTS -> clean_release_v2_release_api_contracts
"""Contract tests for redesigned approval/publication API endpoints."""
from datetime import datetime, timezone
from uuid import uuid4
from fastapi import FastAPI
from fastapi.testclient import TestClient
from src.api.routes.clean_release_v2 import router as clean_release_v2_router
from src.dependencies import get_clean_release_repository
from src.models.clean_release import ComplianceReport, ReleaseCandidate
from src.services.clean_release.enums import CandidateStatus, ComplianceDecision
test_app = FastAPI()
test_app.include_router(clean_release_v2_router)
client = TestClient(test_app)
def _seed_candidate_and_passed_report() -> tuple[str, str]:
repository = get_clean_release_repository()
candidate_id = f"api-release-candidate-{uuid4()}"
report_id = f"api-release-report-{uuid4()}"
repository.save_candidate(
ReleaseCandidate(
id=candidate_id,
version="1.0.0",
source_snapshot_ref="git:sha-api-release",
created_by="api-test",
created_at=datetime.now(timezone.utc),
status=CandidateStatus.CHECK_PASSED.value,
)
)
repository.save_report(
ComplianceReport(
id=report_id,
run_id=f"run-{uuid4()}",
candidate_id=candidate_id,
final_status=ComplianceDecision.PASSED.value,
summary_json={"operator_summary": "ok", "violations_count": 0, "blocking_violations_count": 0},
generated_at=datetime.now(timezone.utc),
immutable=True,
)
)
return candidate_id, report_id
def test_release_approve_and_publish_revoke_contract() -> None:
"""Contract for approve -> publish -> revoke lifecycle endpoints."""
candidate_id, report_id = _seed_candidate_and_passed_report()
approve_response = client.post(
f"/api/v2/clean-release/candidates/{candidate_id}/approve",
json={"report_id": report_id, "decided_by": "api-test", "comment": "approved"},
)
assert approve_response.status_code == 200
approve_payload = approve_response.json()
assert approve_payload["status"] == "ok"
assert approve_payload["decision"] == "APPROVED"
publish_response = client.post(
f"/api/v2/clean-release/candidates/{candidate_id}/publish",
json={
"report_id": report_id,
"published_by": "api-test",
"target_channel": "stable",
"publication_ref": "rel-api-001",
},
)
assert publish_response.status_code == 200
publish_payload = publish_response.json()
assert publish_payload["status"] == "ok"
assert publish_payload["publication"]["status"] == "ACTIVE"
publication_id = publish_payload["publication"]["id"]
revoke_response = client.post(
f"/api/v2/clean-release/publications/{publication_id}/revoke",
json={"revoked_by": "api-test", "comment": "rollback"},
)
assert revoke_response.status_code == 200
revoke_payload = revoke_response.json()
assert revoke_payload["status"] == "ok"
assert revoke_payload["publication"]["status"] == "REVOKED"
def test_release_reject_contract() -> None:
"""Contract for reject endpoint."""
candidate_id, report_id = _seed_candidate_and_passed_report()
reject_response = client.post(
f"/api/v2/clean-release/candidates/{candidate_id}/reject",
json={"report_id": report_id, "decided_by": "api-test", "comment": "rejected"},
)
assert reject_response.status_code == 200
payload = reject_response.json()
assert payload["status"] == "ok"
assert payload["decision"] == "REJECTED"
# [/DEF:test_clean_release_v2_release_api:Module]

File diff suppressed because it is too large Load Diff

View File

@@ -11,41 +11,6 @@ from unittest.mock import MagicMock, patch, AsyncMock
from fastapi.testclient import TestClient
from src.app import app
from src.api.routes.datasets import DatasetsResponse, DatasetDetailResponse
from src.dependencies import get_current_user, has_permission, get_config_manager, get_task_manager, get_resource_service, get_mapping_service
# Global mock user for get_current_user dependency overrides
mock_user = MagicMock()
mock_user.username = "testuser"
mock_user.roles = []
admin_role = MagicMock()
admin_role.name = "Admin"
mock_user.roles.append(admin_role)
@pytest.fixture(autouse=True)
def mock_deps():
config_manager = MagicMock()
task_manager = MagicMock()
resource_service = MagicMock()
mapping_service = MagicMock()
app.dependency_overrides[get_config_manager] = lambda: config_manager
app.dependency_overrides[get_task_manager] = lambda: task_manager
app.dependency_overrides[get_resource_service] = lambda: resource_service
app.dependency_overrides[get_mapping_service] = lambda: mapping_service
app.dependency_overrides[get_current_user] = lambda: mock_user
app.dependency_overrides[has_permission("plugin:migration", "READ")] = lambda: mock_user
app.dependency_overrides[has_permission("plugin:migration", "EXECUTE")] = lambda: mock_user
app.dependency_overrides[has_permission("plugin:backup", "EXECUTE")] = lambda: mock_user
app.dependency_overrides[has_permission("tasks", "READ")] = lambda: mock_user
yield {
"config": config_manager,
"task": task_manager,
"resource": resource_service,
"mapping": mapping_service
}
app.dependency_overrides.clear()
client = TestClient(app)
@@ -55,34 +20,41 @@ client = TestClient(app)
# @TEST: GET /api/datasets returns 200 and valid schema
# @PRE: env_id exists
# @POST: Response matches DatasetsResponse schema
def test_get_datasets_success(mock_deps):
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_deps["config"].get_environments.return_value = [mock_env]
# Mock resource service response
mock_deps["resource"].get_datasets_with_status = AsyncMock(
return_value=[
{
"id": 1,
"table_name": "sales_data",
"schema": "public",
"database": "sales_db",
"mapped_fields": {"total": 10, "mapped": 5},
"last_task": {"task_id": "task-1", "status": "SUCCESS"}
}
]
)
def test_get_datasets_success():
with patch("src.api.routes.datasets.get_config_manager") as mock_config, \
patch("src.api.routes.datasets.get_resource_service") as mock_service, \
patch("src.api.routes.datasets.has_permission") as mock_perm:
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_config.return_value.get_environments.return_value = [mock_env]
# Mock resource service response
mock_service.return_value.get_datasets_with_status.return_value = AsyncMock()(
return_value=[
{
"id": 1,
"table_name": "sales_data",
"schema": "public",
"database": "sales_db",
"mapped_fields": {"total": 10, "mapped": 5},
"last_task": {"task_id": "task-1", "status": "SUCCESS"}
}
]
)
# Mock permission
mock_perm.return_value = lambda: True
response = client.get("/api/datasets?env_id=prod")
assert response.status_code == 200
data = response.json()
assert "datasets" in data
assert len(data["datasets"]) >= 0
# Validate against Pydantic model
DatasetsResponse(**data)
response = client.get("/api/datasets?env_id=prod")
assert response.status_code == 200
data = response.json()
assert "datasets" in data
assert len(data["datasets"]) >= 0
# Validate against Pydantic model
DatasetsResponse(**data)
# [/DEF:test_get_datasets_success:Function]
@@ -92,13 +64,17 @@ def test_get_datasets_success(mock_deps):
# @TEST: GET /api/datasets returns 404 if env_id missing
# @PRE: env_id does not exist
# @POST: Returns 404 error
def test_get_datasets_env_not_found(mock_deps):
mock_deps["config"].get_environments.return_value = []
def test_get_datasets_env_not_found():
with patch("src.api.routes.datasets.get_config_manager") as mock_config, \
patch("src.api.routes.datasets.has_permission") as mock_perm:
mock_config.return_value.get_environments.return_value = []
mock_perm.return_value = lambda: True
response = client.get("/api/datasets?env_id=nonexistent")
assert response.status_code == 404
assert "Environment not found" in response.json()["detail"]
response = client.get("/api/datasets?env_id=nonexistent")
assert response.status_code == 404
assert "Environment not found" in response.json()["detail"]
# [/DEF:test_get_datasets_env_not_found:Function]
@@ -108,25 +84,24 @@ def test_get_datasets_env_not_found(mock_deps):
# @TEST: GET /api/datasets returns 400 for invalid page/page_size
# @PRE: page < 1 or page_size > 100
# @POST: Returns 400 error
def test_get_datasets_invalid_pagination(mock_deps):
mock_env = MagicMock()
mock_env.id = "prod"
mock_deps["config"].get_environments.return_value = [mock_env]
def test_get_datasets_invalid_pagination():
with patch("src.api.routes.datasets.get_config_manager") as mock_config, \
patch("src.api.routes.datasets.has_permission") as mock_perm:
mock_env = MagicMock()
mock_env.id = "prod"
mock_config.return_value.get_environments.return_value = [mock_env]
mock_perm.return_value = lambda: True
# Invalid page
response = client.get("/api/datasets?env_id=prod&page=0")
assert response.status_code == 400
assert "Page must be >= 1" in response.json()["detail"]
# Invalid page_size (too small)
response = client.get("/api/datasets?env_id=prod&page_size=0")
assert response.status_code == 400
assert "Page size must be between 1 and 100" in response.json()["detail"]
# @TEST_EDGE: page_size > 100 exceeds max
response = client.get("/api/datasets?env_id=prod&page_size=101")
assert response.status_code == 400
assert "Page size must be between 1 and 100" in response.json()["detail"]
# Invalid page
response = client.get("/api/datasets?env_id=prod&page=0")
assert response.status_code == 400
assert "Page must be >= 1" in response.json()["detail"]
# Invalid page_size
response = client.get("/api/datasets?env_id=prod&page_size=0")
assert response.status_code == 400
assert "Page size must be between 1 and 100" in response.json()["detail"]
# [/DEF:test_get_datasets_invalid_pagination:Function]
@@ -136,31 +111,36 @@ def test_get_datasets_invalid_pagination(mock_deps):
# @TEST: POST /api/datasets/map-columns creates mapping task
# @PRE: Valid env_id, dataset_ids, source_type
# @POST: Returns task_id
def test_map_columns_success(mock_deps):
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_deps["config"].get_environments.return_value = [mock_env]
# Mock task manager
mock_task = MagicMock()
mock_task.id = "task-123"
mock_deps["task"].create_task = AsyncMock(return_value=mock_task)
def test_map_columns_success():
with patch("src.api.routes.datasets.get_config_manager") as mock_config, \
patch("src.api.routes.datasets.get_task_manager") as mock_task_mgr, \
patch("src.api.routes.datasets.has_permission") as mock_perm:
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_config.return_value.get_environments.return_value = [mock_env]
# Mock task manager
mock_task = MagicMock()
mock_task.id = "task-123"
mock_task_mgr.return_value.create_task = AsyncMock(return_value=mock_task)
# Mock permission
mock_perm.return_value = lambda: True
response = client.post(
"/api/datasets/map-columns",
json={
"env_id": "prod",
"dataset_ids": [1, 2, 3],
"source_type": "postgresql"
}
)
assert response.status_code == 200
data = response.json()
assert "task_id" in data
# @POST/@SIDE_EFFECT: create_task was called
mock_deps["task"].create_task.assert_called_once()
response = client.post(
"/api/datasets/map-columns",
json={
"env_id": "prod",
"dataset_ids": [1, 2, 3],
"source_type": "postgresql"
}
)
assert response.status_code == 200
data = response.json()
assert "task_id" in data
# [/DEF:test_map_columns_success:Function]
@@ -170,18 +150,21 @@ def test_map_columns_success(mock_deps):
# @TEST: POST /api/datasets/map-columns returns 400 for invalid source_type
# @PRE: source_type is not 'postgresql' or 'xlsx'
# @POST: Returns 400 error
def test_map_columns_invalid_source_type(mock_deps):
response = client.post(
"/api/datasets/map-columns",
json={
"env_id": "prod",
"dataset_ids": [1],
"source_type": "invalid"
}
)
assert response.status_code == 400
assert "Source type must be 'postgresql' or 'xlsx'" in response.json()["detail"]
def test_map_columns_invalid_source_type():
with patch("src.api.routes.datasets.has_permission") as mock_perm:
mock_perm.return_value = lambda: True
response = client.post(
"/api/datasets/map-columns",
json={
"env_id": "prod",
"dataset_ids": [1],
"source_type": "invalid"
}
)
assert response.status_code == 400
assert "Source type must be 'postgresql' or 'xlsx'" in response.json()["detail"]
# [/DEF:test_map_columns_invalid_source_type:Function]
@@ -191,110 +174,39 @@ def test_map_columns_invalid_source_type(mock_deps):
# @TEST: POST /api/datasets/generate-docs creates doc generation task
# @PRE: Valid env_id, dataset_ids, llm_provider
# @POST: Returns task_id
def test_generate_docs_success(mock_deps):
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_deps["config"].get_environments.return_value = [mock_env]
# Mock task manager
mock_task = MagicMock()
mock_task.id = "task-456"
mock_deps["task"].create_task = AsyncMock(return_value=mock_task)
def test_generate_docs_success():
with patch("src.api.routes.datasets.get_config_manager") as mock_config, \
patch("src.api.routes.datasets.get_task_manager") as mock_task_mgr, \
patch("src.api.routes.datasets.has_permission") as mock_perm:
# Mock environment
mock_env = MagicMock()
mock_env.id = "prod"
mock_config.return_value.get_environments.return_value = [mock_env]
# Mock task manager
mock_task = MagicMock()
mock_task.id = "task-456"
mock_task_mgr.return_value.create_task = AsyncMock(return_value=mock_task)
# Mock permission
mock_perm.return_value = lambda: True
response = client.post(
"/api/datasets/generate-docs",
json={
"env_id": "prod",
"dataset_ids": [1],
"llm_provider": "openai"
}
)
assert response.status_code == 200
data = response.json()
assert "task_id" in data
# @POST/@SIDE_EFFECT: create_task was called
mock_deps["task"].create_task.assert_called_once()
response = client.post(
"/api/datasets/generate-docs",
json={
"env_id": "prod",
"dataset_ids": [1],
"llm_provider": "openai"
}
)
assert response.status_code == 200
data = response.json()
assert "task_id" in data
# [/DEF:test_generate_docs_success:Function]
# [DEF:test_map_columns_empty_ids:Function]
# @TEST: POST /api/datasets/map-columns returns 400 for empty dataset_ids
# @PRE: dataset_ids is empty
# @POST: Returns 400 error
def test_map_columns_empty_ids(mock_deps):
"""@PRE: dataset_ids must be non-empty."""
response = client.post(
"/api/datasets/map-columns",
json={
"env_id": "prod",
"dataset_ids": [],
"source_type": "postgresql"
}
)
assert response.status_code == 400
assert "At least one dataset ID must be provided" in response.json()["detail"]
# [/DEF:test_map_columns_empty_ids:Function]
# [DEF:test_generate_docs_empty_ids:Function]
# @TEST: POST /api/datasets/generate-docs returns 400 for empty dataset_ids
# @PRE: dataset_ids is empty
# @POST: Returns 400 error
def test_generate_docs_empty_ids(mock_deps):
"""@PRE: dataset_ids must be non-empty."""
response = client.post(
"/api/datasets/generate-docs",
json={
"env_id": "prod",
"dataset_ids": [],
"llm_provider": "openai"
}
)
assert response.status_code == 400
assert "At least one dataset ID must be provided" in response.json()["detail"]
# [/DEF:test_generate_docs_empty_ids:Function]
# [DEF:test_generate_docs_env_not_found:Function]
# @TEST: POST /api/datasets/generate-docs returns 404 for missing env
# @PRE: env_id does not exist
# @POST: Returns 404 error
def test_generate_docs_env_not_found(mock_deps):
"""@PRE: env_id must be a valid environment."""
mock_deps["config"].get_environments.return_value = []
response = client.post(
"/api/datasets/generate-docs",
json={
"env_id": "ghost",
"dataset_ids": [1],
"llm_provider": "openai"
}
)
assert response.status_code == 404
assert "Environment not found" in response.json()["detail"]
# [/DEF:test_generate_docs_env_not_found:Function]
# [DEF:test_get_datasets_superset_failure:Function]
# @TEST_EDGE: external_superset_failure -> {status: 503}
def test_get_datasets_superset_failure(mock_deps):
"""@TEST_EDGE: external_superset_failure -> {status: 503}"""
mock_env = MagicMock()
mock_env.id = "bad_conn"
mock_deps["config"].get_environments.return_value = [mock_env]
mock_deps["task"].get_all_tasks.return_value = []
mock_deps["resource"].get_datasets_with_status = AsyncMock(
side_effect=Exception("Connection refused")
)
response = client.get("/api/datasets?env_id=bad_conn")
assert response.status_code == 503
assert "Failed to fetch datasets" in response.json()["detail"]
# [/DEF:test_get_datasets_superset_failure:Function]
# [/DEF:backend.src.api.routes.__tests__.test_datasets:Module]

View File

@@ -1,310 +0,0 @@
# [DEF:backend.src.api.routes.__tests__.test_git_api:Module]
# @RELATION: VERIFIES -> src.api.routes.git
# @PURPOSE: API tests for Git configurations and repository operations.
import pytest
import asyncio
from unittest.mock import MagicMock
from fastapi import HTTPException
from src.api.routes import git as git_routes
from src.models.git import GitServerConfig, GitProvider, GitStatus, GitRepository
class DbMock:
def __init__(self, data=None):
self._data = data or []
self._deleted = []
self._added = []
def query(self, model):
self._model = model
return self
def filter(self, condition):
# Simplistic mocking for tests, assuming equality checks
for item in self._data:
# We assume condition is an equality expression like GitServerConfig.id == "123"
# It's hard to eval the condition exactly in a mock without complex parsing,
# so we'll just return items where type matches.
pass
return self
def first(self):
for item in self._data:
if hasattr(self, "_model") and isinstance(item, self._model):
return item
return None
def all(self):
return self._data
def add(self, item):
self._added.append(item)
if not hasattr(item, "id") or not item.id:
item.id = "mocked-id"
self._data.append(item)
def delete(self, item):
self._deleted.append(item)
if item in self._data:
self._data.remove(item)
def commit(self):
pass
def refresh(self, item):
if not hasattr(item, "status"):
item.status = GitStatus.CONNECTED
if not hasattr(item, "last_validated"):
item.last_validated = "2026-03-08T00:00:00Z"
def test_get_git_configs_masks_pat():
"""
@PRE: Database session `db` is available.
@POST: Returns a list of all GitServerConfig objects from the database with PAT masked.
"""
db = DbMock([GitServerConfig(
id="config-1", name="Test Server", provider=GitProvider.GITHUB,
url="https://github.com", pat="secret-token",
status=GitStatus.CONNECTED, last_validated="2026-03-08T00:00:00Z"
)])
result = asyncio.run(git_routes.get_git_configs(db=db))
assert len(result) == 1
assert result[0].pat == "********"
assert result[0].name == "Test Server"
def test_create_git_config_persists_config():
"""
@PRE: `config` contains valid GitServerConfigCreate data.
@POST: A new GitServerConfig record is created in the database.
"""
from src.api.routes.git_schemas import GitServerConfigCreate
db = DbMock()
config = GitServerConfigCreate(
name="New Server", provider=GitProvider.GITLAB,
url="https://gitlab.com", pat="new-token",
default_branch="master"
)
result = asyncio.run(git_routes.create_git_config(config=config, db=db))
assert len(db._added) == 1
assert db._added[0].name == "New Server"
assert db._added[0].pat == "new-token"
assert result.name == "New Server"
assert result.pat == "new-token" # Note: route returns unmasked until serialized by FastAPI usually, but in tests schema might catch it or not.
from src.api.routes.git_schemas import GitServerConfigUpdate
def test_update_git_config_modifies_record():
"""
@PRE: `config_id` corresponds to an existing configuration.
@POST: The configuration record is updated in the database, preserving PAT if masked is sent.
"""
existing_config = GitServerConfig(
id="config-1", name="Old Server", provider=GitProvider.GITHUB,
url="https://github.com", pat="old-token",
status=GitStatus.CONNECTED, last_validated="2026-03-08T00:00:00Z"
)
# The monkeypatched query will return existing_config as it's the only one in the list
class SingleConfigDbMock:
def query(self, *args): return self
def filter(self, *args): return self
def first(self): return existing_config
def commit(self): pass
def refresh(self, config): pass
db = SingleConfigDbMock()
update_data = GitServerConfigUpdate(name="Updated Server", pat="********")
result = asyncio.run(git_routes.update_git_config(config_id="config-1", config_update=update_data, db=db))
assert existing_config.name == "Updated Server"
assert existing_config.pat == "old-token" # Ensure PAT is not overwritten with asterisks
assert result.pat == "********"
def test_update_git_config_raises_404_if_not_found():
"""
@PRE: `config_id` corresponds to a missing configuration.
@THROW: HTTPException 404
"""
db = DbMock([]) # Empty db
update_data = GitServerConfigUpdate(name="Updated Server", pat="new-token")
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.update_git_config(config_id="config-1", config_update=update_data, db=db))
assert exc_info.value.status_code == 404
assert exc_info.value.detail == "Configuration not found"
def test_delete_git_config_removes_record():
"""
@PRE: `config_id` corresponds to an existing configuration.
@POST: The configuration record is removed from the database.
"""
existing_config = GitServerConfig(id="config-1")
class SingleConfigDbMock:
def query(self, *args): return self
def filter(self, *args): return self
def first(self): return existing_config
def delete(self, config): self.deleted = config
def commit(self): pass
db = SingleConfigDbMock()
result = asyncio.run(git_routes.delete_git_config(config_id="config-1", db=db))
assert db.deleted == existing_config
assert result["status"] == "success"
def test_test_git_config_validates_connection_successfully(monkeypatch):
"""
@PRE: `config` contains provider, url, and pat.
@POST: Returns success if the connection is validated via GitService.
"""
class MockGitService:
async def test_connection(self, provider, url, pat):
return True
monkeypatch.setattr(git_routes, "git_service", MockGitService())
from src.api.routes.git_schemas import GitServerConfigCreate
config = GitServerConfigCreate(
name="Test Server", provider=GitProvider.GITHUB,
url="https://github.com", pat="test-pat"
)
db = DbMock([])
result = asyncio.run(git_routes.test_git_config(config=config, db=db))
assert result["status"] == "success"
def test_test_git_config_fails_validation(monkeypatch):
"""
@PRE: `config` contains provider, url, and pat BUT connection fails.
@THROW: HTTPException 400
"""
class MockGitService:
async def test_connection(self, provider, url, pat):
return False
monkeypatch.setattr(git_routes, "git_service", MockGitService())
from src.api.routes.git_schemas import GitServerConfigCreate
config = GitServerConfigCreate(
name="Test Server", provider=GitProvider.GITHUB,
url="https://github.com", pat="bad-pat"
)
db = DbMock([])
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.test_git_config(config=config, db=db))
assert exc_info.value.status_code == 400
assert exc_info.value.detail == "Connection failed"
def test_list_gitea_repositories_returns_payload(monkeypatch):
"""
@PRE: config_id exists and provider is GITEA.
@POST: Returns repositories visible to PAT user.
"""
class MockGitService:
async def list_gitea_repositories(self, url, pat):
return [{"name": "test-repo", "full_name": "owner/test-repo", "private": True}]
monkeypatch.setattr(git_routes, "git_service", MockGitService())
existing_config = GitServerConfig(
id="config-1", name="Gitea Server", provider=GitProvider.GITEA,
url="https://gitea.local", pat="gitea-token"
)
db = DbMock([existing_config])
result = asyncio.run(git_routes.list_gitea_repositories(config_id="config-1", db=db))
assert len(result) == 1
assert result[0].name == "test-repo"
assert result[0].private is True
def test_list_gitea_repositories_rejects_non_gitea(monkeypatch):
"""
@PRE: config_id exists and provider is NOT GITEA.
@THROW: HTTPException 400
"""
existing_config = GitServerConfig(
id="config-1", name="GitHub Server", provider=GitProvider.GITHUB,
url="https://github.com", pat="token"
)
db = DbMock([existing_config])
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.list_gitea_repositories(config_id="config-1", db=db))
assert exc_info.value.status_code == 400
assert "GITEA provider only" in exc_info.value.detail
def test_create_remote_repository_creates_provider_repo(monkeypatch):
"""
@PRE: config_id exists and PAT has creation permissions.
@POST: Returns normalized remote repository payload.
"""
class MockGitService:
async def create_gitlab_repository(self, server_url, pat, name, private, description, auto_init, default_branch):
return {
"name": name,
"full_name": f"user/{name}",
"private": private,
"clone_url": f"{server_url}/user/{name}.git"
}
monkeypatch.setattr(git_routes, "git_service", MockGitService())
from src.api.routes.git_schemas import RemoteRepoCreateRequest
existing_config = GitServerConfig(
id="config-1", name="GitLab Server", provider=GitProvider.GITLAB,
url="https://gitlab.com", pat="token"
)
db = DbMock([existing_config])
request = RemoteRepoCreateRequest(name="new-repo", private=True, description="desc")
result = asyncio.run(git_routes.create_remote_repository(config_id="config-1", request=request, db=db))
assert result.provider == GitProvider.GITLAB
assert result.name == "new-repo"
assert result.full_name == "user/new-repo"
def test_init_repository_initializes_and_saves_binding(monkeypatch):
"""
@PRE: `dashboard_ref` exists and `init_data` contains valid config_id and remote_url.
@POST: Repository is initialized on disk and a GitRepository record is saved in DB.
"""
from src.api.routes.git_schemas import RepoInitRequest
class MockGitService:
def init_repo(self, dashboard_id, remote_url, pat, repo_key, default_branch):
self.init_called = True
def _get_repo_path(self, dashboard_id, repo_key):
return f"/tmp/repos/{repo_key}"
git_service_mock = MockGitService()
monkeypatch.setattr(git_routes, "git_service", git_service_mock)
monkeypatch.setattr(git_routes, "_resolve_dashboard_id_from_ref", lambda *args, **kwargs: 123)
monkeypatch.setattr(git_routes, "_resolve_repo_key_from_ref", lambda *args, **kwargs: "dashboard-123")
existing_config = GitServerConfig(
id="config-1", name="GitLab Server", provider=GitProvider.GITLAB,
url="https://gitlab.com", pat="token", default_branch="main"
)
db = DbMock([existing_config])
init_data = RepoInitRequest(config_id="config-1", remote_url="https://git.local/repo.git")
result = asyncio.run(git_routes.init_repository(dashboard_ref="123", init_data=init_data, config_manager=MagicMock(), db=db))
assert result["status"] == "success"
assert git_service_mock.init_called is True
assert len(db._added) == 1
assert isinstance(db._added[0], GitRepository)
assert db._added[0].dashboard_id == 123
# [/DEF:backend.src.api.routes.__tests__.test_git_api:Module]

View File

@@ -1,440 +0,0 @@
# [DEF:backend.src.api.routes.__tests__.test_git_status_route:Module]
# @TIER: STANDARD
# @SEMANTICS: tests, git, api, status, no_repo
# @PURPOSE: Validate status endpoint behavior for missing and error repository states.
# @LAYER: Domain (Tests)
# @RELATION: CALLS -> src.api.routes.git.get_repository_status
from fastapi import HTTPException
import pytest
import asyncio
from unittest.mock import MagicMock
from src.api.routes import git as git_routes
# [DEF:test_get_repository_status_returns_no_repo_payload_for_missing_repo:Function]
# @PURPOSE: Ensure missing local repository is represented as NO_REPO payload instead of an API error.
# @PRE: GitService.get_status raises HTTPException(404).
# @POST: Route returns a deterministic NO_REPO status payload.
def test_get_repository_status_returns_no_repo_payload_for_missing_repo(monkeypatch):
class MissingRepoGitService:
def _get_repo_path(self, dashboard_id: int) -> str:
return f"/tmp/missing-repo-{dashboard_id}"
def get_status(self, dashboard_id: int) -> dict:
raise AssertionError("get_status must not be called when repository path is missing")
monkeypatch.setattr(git_routes, "git_service", MissingRepoGitService())
response = asyncio.run(git_routes.get_repository_status(34))
assert response["sync_status"] == "NO_REPO"
assert response["sync_state"] == "NO_REPO"
assert response["has_repo"] is False
assert response["current_branch"] is None
# [/DEF:test_get_repository_status_returns_no_repo_payload_for_missing_repo:Function]
# [DEF:test_get_repository_status_propagates_non_404_http_exception:Function]
# @PURPOSE: Ensure HTTP exceptions other than 404 are not masked.
# @PRE: GitService.get_status raises HTTPException with non-404 status.
# @POST: Raised exception preserves original status and detail.
def test_get_repository_status_propagates_non_404_http_exception(monkeypatch):
class ConflictGitService:
def _get_repo_path(self, dashboard_id: int) -> str:
return f"/tmp/existing-repo-{dashboard_id}"
def get_status(self, dashboard_id: int) -> dict:
raise HTTPException(status_code=409, detail="Conflict")
monkeypatch.setattr(git_routes, "git_service", ConflictGitService())
monkeypatch.setattr(git_routes.os.path, "exists", lambda _path: True)
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.get_repository_status(34))
assert exc_info.value.status_code == 409
assert exc_info.value.detail == "Conflict"
# [/DEF:test_get_repository_status_propagates_non_404_http_exception:Function]
# [DEF:test_get_repository_diff_propagates_http_exception:Function]
# @PURPOSE: Ensure diff endpoint preserves domain HTTP errors from GitService.
# @PRE: GitService.get_diff raises HTTPException.
# @POST: Endpoint raises same HTTPException values.
def test_get_repository_diff_propagates_http_exception(monkeypatch):
class DiffGitService:
def get_diff(self, dashboard_id: int, file_path=None, staged: bool = False) -> str:
raise HTTPException(status_code=404, detail="Repository missing")
monkeypatch.setattr(git_routes, "git_service", DiffGitService())
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.get_repository_diff(12))
assert exc_info.value.status_code == 404
assert exc_info.value.detail == "Repository missing"
# [/DEF:test_get_repository_diff_propagates_http_exception:Function]
# [DEF:test_get_history_wraps_unexpected_error_as_500:Function]
# @PURPOSE: Ensure non-HTTP exceptions in history endpoint become deterministic 500 errors.
# @PRE: GitService.get_commit_history raises ValueError.
# @POST: Endpoint returns HTTPException with status 500 and route context.
def test_get_history_wraps_unexpected_error_as_500(monkeypatch):
class HistoryGitService:
def get_commit_history(self, dashboard_id: int, limit: int = 50):
raise ValueError("broken parser")
monkeypatch.setattr(git_routes, "git_service", HistoryGitService())
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.get_history(12))
assert exc_info.value.status_code == 500
assert exc_info.value.detail == "get_history failed: broken parser"
# [/DEF:test_get_history_wraps_unexpected_error_as_500:Function]
# [DEF:test_commit_changes_wraps_unexpected_error_as_500:Function]
# @PURPOSE: Ensure commit endpoint does not leak unexpected errors as 400.
# @PRE: GitService.commit_changes raises RuntimeError.
# @POST: Endpoint raises HTTPException(500) with route context.
def test_commit_changes_wraps_unexpected_error_as_500(monkeypatch):
class CommitGitService:
def commit_changes(self, dashboard_id: int, message: str, files):
raise RuntimeError("index lock")
class CommitPayload:
message = "test"
files = ["dashboards/a.yaml"]
monkeypatch.setattr(git_routes, "git_service", CommitGitService())
with pytest.raises(HTTPException) as exc_info:
asyncio.run(git_routes.commit_changes(12, CommitPayload()))
assert exc_info.value.status_code == 500
assert exc_info.value.detail == "commit_changes failed: index lock"
# [/DEF:test_commit_changes_wraps_unexpected_error_as_500:Function]
# [DEF:test_get_repository_status_batch_returns_mixed_statuses:Function]
# @PURPOSE: Ensure batch endpoint returns per-dashboard statuses in one response.
# @PRE: Some repositories are missing and some are initialized.
# @POST: Returned map includes resolved status for each requested dashboard ID.
def test_get_repository_status_batch_returns_mixed_statuses(monkeypatch):
class BatchGitService:
def _get_repo_path(self, dashboard_id: int) -> str:
return f"/tmp/repo-{dashboard_id}"
def get_status(self, dashboard_id: int) -> dict:
if dashboard_id == 2:
return {"sync_state": "SYNCED", "sync_status": "OK"}
raise HTTPException(status_code=404, detail="not found")
monkeypatch.setattr(git_routes, "git_service", BatchGitService())
monkeypatch.setattr(git_routes.os.path, "exists", lambda path: path.endswith("/repo-2"))
class BatchRequest:
dashboard_ids = [1, 2]
response = asyncio.run(git_routes.get_repository_status_batch(BatchRequest()))
assert response.statuses["1"]["sync_status"] == "NO_REPO"
assert response.statuses["2"]["sync_state"] == "SYNCED"
# [/DEF:test_get_repository_status_batch_returns_mixed_statuses:Function]
# [DEF:test_get_repository_status_batch_marks_item_as_error_on_service_failure:Function]
# @PURPOSE: Ensure batch endpoint marks failed items as ERROR without failing entire request.
# @PRE: GitService raises non-HTTP exception for one dashboard.
# @POST: Failed dashboard status is marked as ERROR.
def test_get_repository_status_batch_marks_item_as_error_on_service_failure(monkeypatch):
class BatchErrorGitService:
def _get_repo_path(self, dashboard_id: int) -> str:
return f"/tmp/repo-{dashboard_id}"
def get_status(self, dashboard_id: int) -> dict:
raise RuntimeError("boom")
monkeypatch.setattr(git_routes, "git_service", BatchErrorGitService())
monkeypatch.setattr(git_routes.os.path, "exists", lambda _path: True)
class BatchRequest:
dashboard_ids = [9]
response = asyncio.run(git_routes.get_repository_status_batch(BatchRequest()))
assert response.statuses["9"]["sync_status"] == "ERROR"
assert response.statuses["9"]["sync_state"] == "ERROR"
# [/DEF:test_get_repository_status_batch_marks_item_as_error_on_service_failure:Function]
# [DEF:test_get_repository_status_batch_deduplicates_and_truncates_ids:Function]
# @PURPOSE: Ensure batch endpoint protects server from oversized payloads.
# @PRE: request includes duplicate IDs and more than MAX_REPOSITORY_STATUS_BATCH entries.
# @POST: Result contains unique IDs up to configured cap.
def test_get_repository_status_batch_deduplicates_and_truncates_ids(monkeypatch):
class SafeBatchGitService:
def _get_repo_path(self, dashboard_id: int) -> str:
return f"/tmp/repo-{dashboard_id}"
def get_status(self, dashboard_id: int) -> dict:
return {"sync_state": "SYNCED", "sync_status": "OK"}
monkeypatch.setattr(git_routes, "git_service", SafeBatchGitService())
monkeypatch.setattr(git_routes.os.path, "exists", lambda _path: True)
class BatchRequest:
dashboard_ids = [1, 1] + list(range(2, 90))
response = asyncio.run(git_routes.get_repository_status_batch(BatchRequest()))
assert len(response.statuses) == git_routes.MAX_REPOSITORY_STATUS_BATCH
assert "1" in response.statuses
# [/DEF:test_get_repository_status_batch_deduplicates_and_truncates_ids:Function]
# [DEF:test_commit_changes_applies_profile_identity_before_commit:Function]
# @PURPOSE: Ensure commit route configures repository identity from profile preferences before commit call.
# @PRE: Profile preference contains git_username/git_email for current user.
# @POST: git_service.configure_identity receives resolved identity and commit proceeds.
def test_commit_changes_applies_profile_identity_before_commit(monkeypatch):
class IdentityGitService:
def __init__(self):
self.configured_identity = None
self.commit_payload = None
def configure_identity(self, dashboard_id: int, git_username: str, git_email: str):
self.configured_identity = (dashboard_id, git_username, git_email)
def commit_changes(self, dashboard_id: int, message: str, files):
self.commit_payload = (dashboard_id, message, files)
class PreferenceRow:
git_username = "user_1"
git_email = "user1@mail.ru"
class PreferenceQuery:
def filter(self, *_args, **_kwargs):
return self
def first(self):
return PreferenceRow()
class DbStub:
def query(self, _model):
return PreferenceQuery()
class UserStub:
id = "u-1"
class CommitPayload:
message = "test"
files = ["dashboards/a.yaml"]
identity_service = IdentityGitService()
monkeypatch.setattr(git_routes, "git_service", identity_service)
monkeypatch.setattr(
git_routes,
"_resolve_dashboard_id_from_ref",
lambda *_args, **_kwargs: 12,
)
asyncio.run(
git_routes.commit_changes(
"dashboard-12",
CommitPayload(),
config_manager=MagicMock(),
db=DbStub(),
current_user=UserStub(),
)
)
assert identity_service.configured_identity == (12, "user_1", "user1@mail.ru")
assert identity_service.commit_payload == (12, "test", ["dashboards/a.yaml"])
# [/DEF:test_commit_changes_applies_profile_identity_before_commit:Function]
# [DEF:test_pull_changes_applies_profile_identity_before_pull:Function]
# @PURPOSE: Ensure pull route configures repository identity from profile preferences before pull call.
# @PRE: Profile preference contains git_username/git_email for current user.
# @POST: git_service.configure_identity receives resolved identity and pull proceeds.
def test_pull_changes_applies_profile_identity_before_pull(monkeypatch):
class IdentityGitService:
def __init__(self):
self.configured_identity = None
self.pulled_dashboard_id = None
def configure_identity(self, dashboard_id: int, git_username: str, git_email: str):
self.configured_identity = (dashboard_id, git_username, git_email)
def pull_changes(self, dashboard_id: int):
self.pulled_dashboard_id = dashboard_id
class PreferenceRow:
git_username = "user_1"
git_email = "user1@mail.ru"
class PreferenceQuery:
def filter(self, *_args, **_kwargs):
return self
def first(self):
return PreferenceRow()
class DbStub:
def query(self, _model):
return PreferenceQuery()
class UserStub:
id = "u-1"
identity_service = IdentityGitService()
monkeypatch.setattr(git_routes, "git_service", identity_service)
monkeypatch.setattr(
git_routes,
"_resolve_dashboard_id_from_ref",
lambda *_args, **_kwargs: 12,
)
asyncio.run(
git_routes.pull_changes(
"dashboard-12",
config_manager=MagicMock(),
db=DbStub(),
current_user=UserStub(),
)
)
assert identity_service.configured_identity == (12, "user_1", "user1@mail.ru")
assert identity_service.pulled_dashboard_id == 12
# [/DEF:test_pull_changes_applies_profile_identity_before_pull:Function]
# [DEF:test_get_merge_status_returns_service_payload:Function]
# @PURPOSE: Ensure merge status route returns service payload as-is.
# @PRE: git_service.get_merge_status returns unfinished merge payload.
# @POST: Route response contains has_unfinished_merge=True.
def test_get_merge_status_returns_service_payload(monkeypatch):
class MergeStatusGitService:
def get_merge_status(self, dashboard_id: int) -> dict:
return {
"has_unfinished_merge": True,
"repository_path": "/tmp/repo-12",
"git_dir": "/tmp/repo-12/.git",
"current_branch": "dev",
"merge_head": "abc",
"merge_message_preview": "merge msg",
"conflicts_count": 2,
}
monkeypatch.setattr(git_routes, "git_service", MergeStatusGitService())
monkeypatch.setattr(git_routes, "_resolve_dashboard_id_from_ref", lambda *_args, **_kwargs: 12)
response = asyncio.run(
git_routes.get_merge_status(
"dashboard-12",
config_manager=MagicMock(),
)
)
assert response["has_unfinished_merge"] is True
assert response["conflicts_count"] == 2
# [/DEF:test_get_merge_status_returns_service_payload:Function]
# [DEF:test_resolve_merge_conflicts_passes_resolution_items_to_service:Function]
# @PURPOSE: Ensure merge resolve route forwards parsed resolutions to service.
# @PRE: resolve_data has one file strategy.
# @POST: Service receives normalized list and route returns resolved files.
def test_resolve_merge_conflicts_passes_resolution_items_to_service(monkeypatch):
captured = {}
class MergeResolveGitService:
def resolve_merge_conflicts(self, dashboard_id: int, resolutions):
captured["dashboard_id"] = dashboard_id
captured["resolutions"] = resolutions
return ["dashboards/a.yaml"]
class ResolveData:
class _Resolution:
def dict(self):
return {"file_path": "dashboards/a.yaml", "resolution": "mine", "content": None}
resolutions = [_Resolution()]
monkeypatch.setattr(git_routes, "git_service", MergeResolveGitService())
monkeypatch.setattr(git_routes, "_resolve_dashboard_id_from_ref", lambda *_args, **_kwargs: 12)
response = asyncio.run(
git_routes.resolve_merge_conflicts(
"dashboard-12",
ResolveData(),
config_manager=MagicMock(),
)
)
assert captured["dashboard_id"] == 12
assert captured["resolutions"][0]["resolution"] == "mine"
assert response["resolved_files"] == ["dashboards/a.yaml"]
# [/DEF:test_resolve_merge_conflicts_passes_resolution_items_to_service:Function]
# [DEF:test_abort_merge_calls_service_and_returns_result:Function]
# @PURPOSE: Ensure abort route delegates to service.
# @PRE: Service abort_merge returns aborted status.
# @POST: Route returns aborted status.
def test_abort_merge_calls_service_and_returns_result(monkeypatch):
class AbortGitService:
def abort_merge(self, dashboard_id: int):
assert dashboard_id == 12
return {"status": "aborted"}
monkeypatch.setattr(git_routes, "git_service", AbortGitService())
monkeypatch.setattr(git_routes, "_resolve_dashboard_id_from_ref", lambda *_args, **_kwargs: 12)
response = asyncio.run(
git_routes.abort_merge(
"dashboard-12",
config_manager=MagicMock(),
)
)
assert response["status"] == "aborted"
# [/DEF:test_abort_merge_calls_service_and_returns_result:Function]
# [DEF:test_continue_merge_passes_message_and_returns_commit:Function]
# @PURPOSE: Ensure continue route passes commit message to service.
# @PRE: continue_data.message is provided.
# @POST: Route returns committed status and hash.
def test_continue_merge_passes_message_and_returns_commit(monkeypatch):
class ContinueGitService:
def continue_merge(self, dashboard_id: int, message: str):
assert dashboard_id == 12
assert message == "Resolve all conflicts"
return {"status": "committed", "commit_hash": "abc123"}
class ContinueData:
message = "Resolve all conflicts"
monkeypatch.setattr(git_routes, "git_service", ContinueGitService())
monkeypatch.setattr(git_routes, "_resolve_dashboard_id_from_ref", lambda *_args, **_kwargs: 12)
response = asyncio.run(
git_routes.continue_merge(
"dashboard-12",
ContinueData(),
config_manager=MagicMock(),
)
)
assert response["status"] == "committed"
assert response["commit_hash"] == "abc123"
# [/DEF:test_continue_merge_passes_message_and_returns_commit:Function]
# [/DEF:backend.src.api.routes.__tests__.test_git_status_route:Module]

View File

@@ -407,104 +407,4 @@ async def test_execute_migration_invalid_env_raises_400(_mock_env):
assert exc.value.status_code == 400
@pytest.mark.asyncio
async def test_dry_run_migration_returns_diff_and_risk(db_session):
# @TEST_EDGE: missing_target_datasource -> validates high risk item generation
# @TEST_EDGE: breaking_reference -> validates high risk on missing dataset link
from src.api.routes.migration import dry_run_migration
from src.models.dashboard import DashboardSelection
env_source = MagicMock()
env_source.id = "src"
env_source.name = "Source"
env_source.url = "http://source"
env_source.username = "admin"
env_source.password = "admin"
env_source.verify_ssl = False
env_source.timeout = 30
env_target = MagicMock()
env_target.id = "tgt"
env_target.name = "Target"
env_target.url = "http://target"
env_target.username = "admin"
env_target.password = "admin"
env_target.verify_ssl = False
env_target.timeout = 30
cm = _make_sync_config_manager([env_source, env_target])
selection = DashboardSelection(
selected_ids=[42],
source_env_id="src",
target_env_id="tgt",
replace_db_config=False,
fix_cross_filters=True,
)
with patch("src.api.routes.migration.SupersetClient") as MockClient, \
patch("src.api.routes.migration.MigrationDryRunService") as MockService:
source_client = MagicMock()
target_client = MagicMock()
MockClient.side_effect = [source_client, target_client]
service_instance = MagicMock()
service_payload = {
"generated_at": "2026-02-27T00:00:00+00:00",
"selection": selection.model_dump(),
"selected_dashboard_titles": ["Sales"],
"diff": {
"dashboards": {"create": [], "update": [{"uuid": "dash-1"}], "delete": []},
"charts": {"create": [{"uuid": "chart-1"}], "update": [], "delete": []},
"datasets": {"create": [{"uuid": "dataset-1"}], "update": [], "delete": []},
},
"summary": {
"dashboards": {"create": 0, "update": 1, "delete": 0},
"charts": {"create": 1, "update": 0, "delete": 0},
"datasets": {"create": 1, "update": 0, "delete": 0},
"selected_dashboards": 1,
},
"risk": {
"score": 75,
"level": "high",
"items": [
{"code": "missing_datasource"},
{"code": "breaking_reference"},
],
},
}
service_instance.run.return_value = service_payload
MockService.return_value = service_instance
result = await dry_run_migration(selection=selection, config_manager=cm, db=db_session, _=None)
assert result["summary"]["dashboards"]["update"] == 1
assert result["summary"]["charts"]["create"] == 1
assert result["summary"]["datasets"]["create"] == 1
assert result["risk"]["score"] > 0
assert any(item["code"] == "missing_datasource" for item in result["risk"]["items"])
assert any(item["code"] == "breaking_reference" for item in result["risk"]["items"])
@pytest.mark.asyncio
async def test_dry_run_migration_rejects_same_environment(db_session):
from src.api.routes.migration import dry_run_migration
from src.models.dashboard import DashboardSelection
env = MagicMock()
env.id = "same"
env.name = "Same"
env.url = "http://same"
env.username = "admin"
env.password = "admin"
env.verify_ssl = False
env.timeout = 30
cm = _make_sync_config_manager([env])
selection = DashboardSelection(selected_ids=[1], source_env_id="same", target_env_id="same")
with pytest.raises(HTTPException) as exc:
await dry_run_migration(selection=selection, config_manager=cm, db=db_session, _=None)
assert exc.value.status_code == 400
# [/DEF:backend.src.api.routes.__tests__.test_migration_routes:Module]

View File

@@ -1,298 +0,0 @@
# [DEF:backend.src.api.routes.__tests__.test_profile_api:Module]
# @TIER: STANDARD
# @SEMANTICS: tests, profile, api, preferences, lookup, contract
# @PURPOSE: Verifies profile API route contracts for preference read/update and Superset account lookup.
# @LAYER: API
# @RELATION: TESTS -> backend.src.api.routes.profile
# [SECTION: IMPORTS]
from datetime import datetime, timezone
from unittest.mock import MagicMock, patch
from fastapi.testclient import TestClient
from src.app import app
from src.core.database import get_db
from src.dependencies import get_config_manager, get_current_user
from src.schemas.profile import (
ProfilePermissionState,
ProfilePreference,
ProfilePreferenceResponse,
ProfileSecuritySummary,
SupersetAccountCandidate,
SupersetAccountLookupResponse,
)
from src.services.profile_service import (
EnvironmentNotFoundError,
ProfileAuthorizationError,
ProfileValidationError,
)
# [/SECTION]
client = TestClient(app)
# [DEF:mock_profile_route_dependencies:Function]
# @PURPOSE: Provides deterministic dependency overrides for profile route tests.
# @PRE: App instance is initialized.
# @POST: Dependencies are overridden for current test and restored afterward.
def mock_profile_route_dependencies():
mock_user = MagicMock()
mock_user.id = "u-1"
mock_user.username = "test-user"
mock_db = MagicMock()
mock_config_manager = MagicMock()
app.dependency_overrides[get_current_user] = lambda: mock_user
app.dependency_overrides[get_db] = lambda: mock_db
app.dependency_overrides[get_config_manager] = lambda: mock_config_manager
return mock_user, mock_db, mock_config_manager
# [/DEF:mock_profile_route_dependencies:Function]
# [DEF:profile_route_deps_fixture:Function]
# @PURPOSE: Pytest fixture wrapper for profile route dependency overrides.
# @PRE: None.
# @POST: Yields overridden dependencies and clears overrides after test.
import pytest
@pytest.fixture(autouse=True)
def profile_route_deps_fixture():
yielded = mock_profile_route_dependencies()
yield yielded
app.dependency_overrides.clear()
# [/DEF:profile_route_deps_fixture:Function]
# [DEF:_build_preference_response:Function]
# @PURPOSE: Builds stable profile preference response payload for route tests.
# @PRE: user_id is provided.
# @POST: Returns ProfilePreferenceResponse object with deterministic timestamps.
def _build_preference_response(user_id: str = "u-1") -> ProfilePreferenceResponse:
now = datetime.now(timezone.utc)
return ProfilePreferenceResponse(
status="success",
message="Preference loaded",
preference=ProfilePreference(
user_id=user_id,
superset_username="John_Doe",
superset_username_normalized="john_doe",
show_only_my_dashboards=True,
show_only_slug_dashboards=True,
git_username="ivan.ivanov",
git_email="ivan@company.local",
has_git_personal_access_token=True,
git_personal_access_token_masked="iv***al",
start_page="reports",
auto_open_task_drawer=False,
dashboards_table_density="compact",
created_at=now,
updated_at=now,
),
security=ProfileSecuritySummary(
read_only=True,
auth_source="adfs",
current_role="Data Engineer",
role_source="adfs",
roles=["Data Engineer"],
permissions=[
ProfilePermissionState(key="migration:run", allowed=True),
ProfilePermissionState(key="admin:users", allowed=False),
],
),
)
# [/DEF:_build_preference_response:Function]
# [DEF:test_get_profile_preferences_returns_self_payload:Function]
# @PURPOSE: Verifies GET /api/profile/preferences returns stable self-scoped payload.
# @PRE: Authenticated user context is available.
# @POST: Response status is 200 and payload contains current user preference.
def test_get_profile_preferences_returns_self_payload(profile_route_deps_fixture):
mock_user, _, _ = profile_route_deps_fixture
service = MagicMock()
service.get_my_preference.return_value = _build_preference_response(user_id=mock_user.id)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.get("/api/profile/preferences")
assert response.status_code == 200
payload = response.json()
assert payload["status"] == "success"
assert payload["preference"]["user_id"] == mock_user.id
assert payload["preference"]["superset_username_normalized"] == "john_doe"
assert payload["preference"]["git_username"] == "ivan.ivanov"
assert payload["preference"]["git_email"] == "ivan@company.local"
assert payload["preference"]["show_only_slug_dashboards"] is True
assert payload["preference"]["has_git_personal_access_token"] is True
assert payload["preference"]["git_personal_access_token_masked"] == "iv***al"
assert payload["preference"]["start_page"] == "reports"
assert payload["preference"]["auto_open_task_drawer"] is False
assert payload["preference"]["dashboards_table_density"] == "compact"
assert payload["security"]["read_only"] is True
assert payload["security"]["current_role"] == "Data Engineer"
assert payload["security"]["permissions"][0]["key"] == "migration:run"
service.get_my_preference.assert_called_once_with(mock_user)
# [/DEF:test_get_profile_preferences_returns_self_payload:Function]
# [DEF:test_patch_profile_preferences_success:Function]
# @PURPOSE: Verifies PATCH /api/profile/preferences persists valid payload through route mapping.
# @PRE: Valid request payload and authenticated user.
# @POST: Response status is 200 with saved preference payload.
def test_patch_profile_preferences_success(profile_route_deps_fixture):
mock_user, _, _ = profile_route_deps_fixture
service = MagicMock()
service.update_my_preference.return_value = _build_preference_response(user_id=mock_user.id)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.patch(
"/api/profile/preferences",
json={
"superset_username": "John_Doe",
"show_only_my_dashboards": True,
"show_only_slug_dashboards": True,
"git_username": "ivan.ivanov",
"git_email": "ivan@company.local",
"git_personal_access_token": "ghp_1234567890",
"start_page": "reports-logs",
"auto_open_task_drawer": False,
"dashboards_table_density": "free",
},
)
assert response.status_code == 200
payload = response.json()
assert payload["status"] == "success"
assert payload["preference"]["superset_username"] == "John_Doe"
assert payload["preference"]["show_only_my_dashboards"] is True
assert payload["preference"]["show_only_slug_dashboards"] is True
assert payload["preference"]["git_username"] == "ivan.ivanov"
assert payload["preference"]["git_email"] == "ivan@company.local"
assert payload["preference"]["start_page"] == "reports"
assert payload["preference"]["auto_open_task_drawer"] is False
assert payload["preference"]["dashboards_table_density"] == "compact"
service.update_my_preference.assert_called_once()
called_kwargs = service.update_my_preference.call_args.kwargs
assert called_kwargs["current_user"] == mock_user
assert called_kwargs["payload"].git_username == "ivan.ivanov"
assert called_kwargs["payload"].git_email == "ivan@company.local"
assert called_kwargs["payload"].git_personal_access_token == "ghp_1234567890"
assert called_kwargs["payload"].show_only_slug_dashboards is True
assert called_kwargs["payload"].start_page == "reports-logs"
assert called_kwargs["payload"].auto_open_task_drawer is False
assert called_kwargs["payload"].dashboards_table_density == "free"
# [/DEF:test_patch_profile_preferences_success:Function]
# [DEF:test_patch_profile_preferences_validation_error:Function]
# @PURPOSE: Verifies route maps domain validation failure to HTTP 422 with actionable details.
# @PRE: Service raises ProfileValidationError.
# @POST: Response status is 422 and includes validation messages.
def test_patch_profile_preferences_validation_error(profile_route_deps_fixture):
service = MagicMock()
service.update_my_preference.side_effect = ProfileValidationError(
["Superset username is required when default filter is enabled."]
)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.patch(
"/api/profile/preferences",
json={
"superset_username": "",
"show_only_my_dashboards": True,
},
)
assert response.status_code == 422
payload = response.json()
assert "detail" in payload
assert "Superset username is required when default filter is enabled." in payload["detail"]
# [/DEF:test_patch_profile_preferences_validation_error:Function]
# [DEF:test_patch_profile_preferences_cross_user_denied:Function]
# @PURPOSE: Verifies route maps domain authorization guard failure to HTTP 403.
# @PRE: Service raises ProfileAuthorizationError.
# @POST: Response status is 403 with denial message.
def test_patch_profile_preferences_cross_user_denied(profile_route_deps_fixture):
service = MagicMock()
service.update_my_preference.side_effect = ProfileAuthorizationError(
"Cross-user preference mutation is forbidden"
)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.patch(
"/api/profile/preferences",
json={
"superset_username": "john_doe",
"show_only_my_dashboards": True,
},
)
assert response.status_code == 403
payload = response.json()
assert payload["detail"] == "Cross-user preference mutation is forbidden"
# [/DEF:test_patch_profile_preferences_cross_user_denied:Function]
# [DEF:test_lookup_superset_accounts_success:Function]
# @PURPOSE: Verifies lookup route returns success payload with normalized candidates.
# @PRE: Valid environment_id and service success response.
# @POST: Response status is 200 and items list is returned.
def test_lookup_superset_accounts_success(profile_route_deps_fixture):
service = MagicMock()
service.lookup_superset_accounts.return_value = SupersetAccountLookupResponse(
status="success",
environment_id="dev",
page_index=0,
page_size=20,
total=1,
warning=None,
items=[
SupersetAccountCandidate(
environment_id="dev",
username="john_doe",
display_name="John Doe",
email="john@example.local",
is_active=True,
)
],
)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.get("/api/profile/superset-accounts?environment_id=dev")
assert response.status_code == 200
payload = response.json()
assert payload["status"] == "success"
assert payload["environment_id"] == "dev"
assert payload["total"] == 1
assert payload["items"][0]["username"] == "john_doe"
# [/DEF:test_lookup_superset_accounts_success:Function]
# [DEF:test_lookup_superset_accounts_env_not_found:Function]
# @PURPOSE: Verifies lookup route maps missing environment to HTTP 404.
# @PRE: Service raises EnvironmentNotFoundError.
# @POST: Response status is 404 with explicit message.
def test_lookup_superset_accounts_env_not_found(profile_route_deps_fixture):
service = MagicMock()
service.lookup_superset_accounts.side_effect = EnvironmentNotFoundError(
"Environment 'missing-env' not found"
)
with patch("src.api.routes.profile._get_profile_service", return_value=service):
response = client.get("/api/profile/superset-accounts?environment_id=missing-env")
assert response.status_code == 404
payload = response.json()
assert payload["detail"] == "Environment 'missing-env' not found"
# [/DEF:test_lookup_superset_accounts_env_not_found:Function]
# [/DEF:backend.src.api.routes.__tests__.test_profile_api:Module]

View File

@@ -1,73 +0,0 @@
# [DEF:__tests__/test_tasks_logs:Module]
# @RELATION: VERIFIES -> ../tasks.py
# @PURPOSE: Contract testing for task logs API endpoints.
# [/DEF:__tests__/test_tasks_logs:Module]
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from unittest.mock import MagicMock
from src.dependencies import get_task_manager, has_permission
from src.api.routes.tasks import router
# @TEST_FIXTURE: mock_app
@pytest.fixture
def client():
app = FastAPI()
app.include_router(router, prefix="/tasks")
# Mock TaskManager
mock_tm = MagicMock()
app.dependency_overrides[get_task_manager] = lambda: mock_tm
# Mock permissions (bypass for unit test)
app.dependency_overrides[has_permission("tasks", "READ")] = lambda: True
return TestClient(app), mock_tm
# @TEST_CONTRACT: get_task_logs_api -> Invariants
# @TEST_FIXTURE: valid_task_logs_request
def test_get_task_logs_success(client):
tc, tm = client
# Setup mock task
mock_task = MagicMock()
tm.get_task.return_value = mock_task
tm.get_task_logs.return_value = [{"level": "INFO", "message": "msg1"}]
response = tc.get("/tasks/task-1/logs?level=INFO")
assert response.status_code == 200
assert response.json() == [{"level": "INFO", "message": "msg1"}]
tm.get_task.assert_called_with("task-1")
# Verify filter construction inside route
args = tm.get_task_logs.call_args
assert args[0][0] == "task-1"
assert args[0][1].level == "INFO"
# @TEST_EDGE: task_not_found
def test_get_task_logs_not_found(client):
tc, tm = client
tm.get_task.return_value = None
response = tc.get("/tasks/missing/logs")
assert response.status_code == 404
assert response.json()["detail"] == "Task not found"
# @TEST_EDGE: invalid_limit
def test_get_task_logs_invalid_limit(client):
tc, tm = client
# limit=0 is ge=1 in Query
response = tc.get("/tasks/task-1/logs?limit=0")
assert response.status_code == 422
# @TEST_INVARIANT: response_purity
def test_get_task_log_stats_success(client):
tc, tm = client
tm.get_task.return_value = MagicMock()
tm.get_task_log_stats.return_value = {"INFO": 5, "ERROR": 1}
response = tc.get("/tasks/task-1/logs/stats")
assert response.status_code == 200
# response_model=LogStats might wrap this, but let's check basic structure
# assuming tm.get_task_log_stats returns something compatible with LogStats

View File

@@ -22,12 +22,8 @@ from ...schemas.auth import (
ADGroupMappingSchema, ADGroupMappingCreate
)
from ...models.auth import User, Role, ADGroupMapping
from ...dependencies import has_permission, get_plugin_loader
from ...dependencies import has_permission
from ...core.logger import logger, belief_scope
from ...services.rbac_permission_catalog import (
discover_declared_permissions,
sync_permission_catalog,
)
# [/SECTION]
# [DEF:router:Variable]
@@ -274,18 +270,9 @@ async def delete_role(
@router.get("/permissions", response_model=List[PermissionSchema])
async def list_permissions(
db: Session = Depends(get_auth_db),
plugin_loader = Depends(get_plugin_loader),
_ = Depends(has_permission("admin:roles", "READ"))
):
with belief_scope("api.admin.list_permissions"):
declared_permissions = discover_declared_permissions(plugin_loader=plugin_loader)
inserted_count = sync_permission_catalog(db=db, declared_permissions=declared_permissions)
if inserted_count > 0:
logger.info(
"[api.admin.list_permissions][Action] Synchronized %s missing RBAC permissions into auth catalog",
inserted_count,
)
repo = AuthRepository(db)
return repo.list_permissions()
# [/DEF:list_permissions:Function]

View File

@@ -120,7 +120,6 @@ INTENT_PERMISSION_CHECKS: Dict[str, List[Tuple[str, str]]] = {
"run_backup": [("plugin:superset-backup", "EXECUTE"), ("plugin:backup", "EXECUTE")],
"run_llm_validation": [("plugin:llm_dashboard_validation", "EXECUTE")],
"run_llm_documentation": [("plugin:llm_documentation", "EXECUTE")],
"get_health_summary": [("plugin:migration", "READ")],
}
@@ -811,9 +810,6 @@ def _parse_command(message: str, config_manager: ConfigManager) -> Dict[str, Any
if any(k in lower for k in ["миграц", "migration", "migrate"]):
src = _extract_id(lower, [r"(?:с|from)\s+([a-z0-9_-]+)"])
tgt = _extract_id(lower, [r"(?:на|to)\s+([a-z0-9_-]+)"])
dry_run = "--dry-run" in lower or "dry run" in lower
replace_db_config = "--replace-db-config" in lower
fix_cross_filters = "--fix-cross-filters" not in lower # Default true usually, but let's say test uses --dry-run
is_dangerous = _is_production_env(tgt, config_manager)
return {
"domain": "migration",
@@ -822,13 +818,10 @@ def _parse_command(message: str, config_manager: ConfigManager) -> Dict[str, Any
"dashboard_id": int(dashboard_id) if dashboard_id else None,
"source_env": src,
"target_env": tgt,
"dry_run": dry_run,
"replace_db_config": replace_db_config,
"fix_cross_filters": True,
},
"confidence": 0.95 if dashboard_id and src and tgt else 0.72,
"risk_level": "dangerous" if is_dangerous else "guarded",
"requires_confirmation": is_dangerous or dry_run,
"requires_confirmation": is_dangerous,
}
# Backup
@@ -846,18 +839,6 @@ def _parse_command(message: str, config_manager: ConfigManager) -> Dict[str, Any
"requires_confirmation": False,
}
# Health summary
if any(k in lower for k in ["здоровье", "health", "ошибки", "failing", "проблемы"]):
env_match = _extract_id(lower, [r"(?:в|for|env|окружени[ея])\s+([a-z0-9_-]+)"])
return {
"domain": "health",
"operation": "get_health_summary",
"entities": {"environment": env_match},
"confidence": 0.9,
"risk_level": "safe",
"requires_confirmation": False,
}
# LLM validation
if any(k in lower for k in ["валидац", "validate", "провер"]):
env_match = _extract_id(lower, [r"(?:в|for|env|окружени[ея])\s+([a-z0-9_-]+)"])
@@ -1036,15 +1017,6 @@ def _build_tool_catalog(current_user: User, config_manager: ConfigManager, db: S
"risk_level": "guarded",
"requires_confirmation": False,
},
{
"operation": "get_health_summary",
"domain": "health",
"description": "Get summary of dashboard health and failing validations",
"required_entities": [],
"optional_entities": ["environment"],
"risk_level": "safe",
"requires_confirmation": False,
},
]
available: List[Dict[str, Any]] = []
@@ -1078,14 +1050,14 @@ def _coerce_intent_entities(intent: Dict[str, Any]) -> Dict[str, Any]:
# Operations that are read-only and do not require confirmation.
_SAFE_OPS = {"show_capabilities", "get_task_status", "get_health_summary"}
_SAFE_OPS = {"show_capabilities", "get_task_status"}
# [DEF:_confirmation_summary:Function]
# @PURPOSE: Build human-readable confirmation prompt for an intent before execution.
# @PRE: intent contains operation and entities fields.
# @POST: Returns descriptive Russian-language text ending with confirmation prompt.
async def _async_confirmation_summary(intent: Dict[str, Any], config_manager: ConfigManager, db: Session) -> str:
def _confirmation_summary(intent: Dict[str, Any]) -> str:
operation = intent.get("operation", "")
entities = intent.get("entities", {})
descriptions: Dict[str, str] = {
@@ -1113,65 +1085,6 @@ async def _async_confirmation_summary(intent: Dict[str, Any], config_manager: Co
tgt=_label(entities.get("target_env")),
dataset=_label(entities.get("dataset_id")),
)
if operation == "execute_migration":
flags = []
flags.append("маппинг БД: " + ("ВКЛ" if _coerce_query_bool(entities.get("replace_db_config", False)) else "ВЫКЛ"))
flags.append("исправление кроссфильтров: " + ("ВКЛ" if _coerce_query_bool(entities.get("fix_cross_filters", True)) else "ВЫКЛ"))
dry_run_enabled = _coerce_query_bool(entities.get("dry_run", False))
flags.append("отчет dry-run: " + ("ВКЛ" if dry_run_enabled else "ВЫКЛ"))
text += f" ({', '.join(flags)})"
if dry_run_enabled:
try:
from ...core.migration.dry_run_orchestrator import MigrationDryRunService
from ...models.dashboard import DashboardSelection
from ...core.superset_client import SupersetClient
src_token = entities.get("source_env")
tgt_token = entities.get("target_env")
dashboard_id = _resolve_dashboard_id_entity(entities, config_manager, env_hint=src_token)
if dashboard_id and src_token and tgt_token:
src_env_id = _resolve_env_id(src_token, config_manager)
tgt_env_id = _resolve_env_id(tgt_token, config_manager)
if src_env_id and tgt_env_id:
env_map = {env.id: env for env in config_manager.get_environments()}
source_env = env_map.get(src_env_id)
target_env = env_map.get(tgt_env_id)
if source_env and target_env and source_env.id != target_env.id:
selection = DashboardSelection(
source_env_id=source_env.id,
target_env_id=target_env.id,
selected_ids=[dashboard_id],
replace_db_config=_coerce_query_bool(entities.get("replace_db_config", False)),
fix_cross_filters=_coerce_query_bool(entities.get("fix_cross_filters", True))
)
service = MigrationDryRunService()
source_client = SupersetClient(source_env)
target_client = SupersetClient(target_env)
report = service.run(selection, source_client, target_client, db)
s = report.get("summary", {})
dash_s = s.get("dashboards", {})
charts_s = s.get("charts", {})
ds_s = s.get("datasets", {})
# Determine main actions counts
creates = dash_s.get("create", 0) + charts_s.get("create", 0) + ds_s.get("create", 0)
updates = dash_s.get("update", 0) + charts_s.get("update", 0) + ds_s.get("update", 0)
deletes = dash_s.get("delete", 0) + charts_s.get("delete", 0) + ds_s.get("delete", 0)
text += f"\n\nОтчет dry-run:\n- Будет создано новых объектов: {creates}\n- Будет обновлено: {updates}\n- Будет удалено: {deletes}"
else:
text += "\n\n(Не удалось загрузить отчет dry-run: неверные окружения)."
except Exception as e:
import traceback
logger.warning("[assistant.dry_run_summary][failed] Exception: %s\n%s", e, traceback.format_exc())
text += f"\n\n(Не удалось загрузить отчет dry-run: {e})."
return f"Выполнить: {text}. Подтвердите или отмените."
# [/DEF:_confirmation_summary:Function]
@@ -1263,8 +1176,7 @@ async def _plan_intent_with_llm(
]
)
except Exception as exc:
import traceback
logger.warning(f"[assistant.planner][fallback] LLM planner unavailable: {exc}\n{traceback.format_exc()}")
logger.warning(f"[assistant.planner][fallback] LLM planner unavailable: {exc}")
return None
if not isinstance(response, dict):
return None
@@ -1345,7 +1257,6 @@ async def _dispatch_intent(
"run_llm_validation": "LLM: валидация дашборда",
"run_llm_documentation": "LLM: генерация документации",
"get_task_status": "Статус: проверка задачи",
"get_health_summary": "Здоровье: сводка по дашбордам",
}
available = [labels[t["operation"]] for t in tools_catalog if t["operation"] in labels]
if not available:
@@ -1358,41 +1269,6 @@ async def _dispatch_intent(
)
return text, None, []
if operation == "get_health_summary":
from ...services.health_service import HealthService
env_token = entities.get("environment")
env_id = _resolve_env_id(env_token, config_manager)
service = HealthService(db)
summary = await service.get_health_summary(environment_id=env_id)
env_name = _get_environment_name_by_id(env_id, config_manager) if env_id else "всех окружений"
text = (
f"Сводка здоровья дашбордов для {env_name}:\n"
f"- ✅ Прошли проверку: {summary.pass_count}\n"
f"- ⚠️ С предупреждениями: {summary.warn_count}\n"
f"- ❌ Ошибки валидации: {summary.fail_count}\n"
f"- ❓ Неизвестно: {summary.unknown_count}"
)
actions = [
AssistantAction(type="open_route", label="Открыть Health Center", target="/dashboards/health")
]
if summary.fail_count > 0:
text += "\n\nОбнаружены ошибки в следующих дашбордах:"
for item in summary.items:
if item.status == "FAIL":
text += f"\n- {item.dashboard_id} ({item.environment_id}): {item.summary or 'Нет деталей'}"
actions.append(
AssistantAction(
type="open_route",
label=f"Отчет {item.dashboard_id}",
target=f"/reports/llm/{item.task_id}"
)
)
return text, None, actions[:5] # Limit actions to avoid UI clutter
if operation == "get_task_status":
_check_any_permission(current_user, [("tasks", "READ")])
task_id = entities.get("task_id")
@@ -1704,7 +1580,7 @@ async def send_message(
)
CONFIRMATIONS[confirmation_id] = confirm
_persist_confirmation(db, confirm)
text = await _async_confirmation_summary(intent, config_manager, db)
text = _confirmation_summary(intent)
_append_history(
user_id,
conversation_id,
@@ -2019,39 +1895,6 @@ async def list_conversations(
# [/DEF:list_conversations:Function]
# [DEF:delete_conversation:Function]
# @PURPOSE: Soft-delete or hard-delete a conversation and clear its in-memory trace.
# @PRE: conversation_id belongs to current_user.
# @POST: Conversation records are removed from DB and CONVERSATIONS cache.
@router.delete("/conversations/{conversation_id}")
async def delete_conversation(
conversation_id: str,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
):
with belief_scope("assistant.conversations.delete"):
user_id = current_user.id
# 1. Remove from in-memory cache
key = (user_id, conversation_id)
if key in CONVERSATIONS:
del CONVERSATIONS[key]
# 2. Delete from database
deleted_count = db.query(AssistantMessageRecord).filter(
AssistantMessageRecord.user_id == user_id,
AssistantMessageRecord.conversation_id == conversation_id
).delete()
db.commit()
if deleted_count == 0:
raise HTTPException(status_code=404, detail="Conversation not found or already deleted")
return {"status": "success", "deleted": deleted_count, "conversation_id": conversation_id}
# [/DEF:delete_conversation:Function]
@router.get("/history")
# [DEF:get_history:Function]
# @PURPOSE: Retrieve paginated assistant conversation history for current user.

View File

@@ -1,445 +0,0 @@
# [DEF:backend.src.api.routes.clean_release:Module]
# @TIER: STANDARD
# @SEMANTICS: api, clean-release, candidate-preparation, compliance
# @PURPOSE: Expose clean release endpoints for candidate preparation and subsequent compliance flow.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.dependencies.get_clean_release_repository
# @RELATION: DEPENDS_ON -> backend.src.services.clean_release.preparation_service
# @INVARIANT: API never reports prepared status if preparation errors are present.
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, Dict, List
from fastapi import APIRouter, Depends, HTTPException, status
from pydantic import BaseModel, Field
from ...core.logger import belief_scope, logger
from ...dependencies import get_clean_release_repository, get_config_manager
from ...services.clean_release.preparation_service import prepare_candidate
from ...services.clean_release.repository import CleanReleaseRepository
from ...services.clean_release.compliance_orchestrator import CleanComplianceOrchestrator
from ...services.clean_release.report_builder import ComplianceReportBuilder
from ...services.clean_release.compliance_execution_service import ComplianceExecutionService, ComplianceRunError
from ...services.clean_release.dto import CandidateDTO, ManifestDTO, CandidateOverviewDTO, ComplianceRunDTO
from ...services.clean_release.enums import (
ComplianceDecision,
ComplianceStageName,
ViolationCategory,
ViolationSeverity,
RunStatus,
CandidateStatus,
)
from ...models.clean_release import (
ComplianceRun,
ComplianceStageRun,
ComplianceViolation,
CandidateArtifact,
ReleaseCandidate,
)
router = APIRouter(prefix="/api/clean-release", tags=["Clean Release"])
# [DEF:PrepareCandidateRequest:Class]
# @PURPOSE: Request schema for candidate preparation endpoint.
class PrepareCandidateRequest(BaseModel):
candidate_id: str = Field(min_length=1)
artifacts: List[Dict[str, Any]] = Field(default_factory=list)
sources: List[str] = Field(default_factory=list)
operator_id: str = Field(min_length=1)
# [/DEF:PrepareCandidateRequest:Class]
# [DEF:StartCheckRequest:Class]
# @PURPOSE: Request schema for clean compliance check run startup.
class StartCheckRequest(BaseModel):
candidate_id: str = Field(min_length=1)
profile: str = Field(default="enterprise-clean")
execution_mode: str = Field(default="tui")
triggered_by: str = Field(default="system")
# [/DEF:StartCheckRequest:Class]
# [DEF:RegisterCandidateRequest:Class]
# @PURPOSE: Request schema for candidate registration endpoint.
class RegisterCandidateRequest(BaseModel):
id: str = Field(min_length=1)
version: str = Field(min_length=1)
source_snapshot_ref: str = Field(min_length=1)
created_by: str = Field(min_length=1)
# [/DEF:RegisterCandidateRequest:Class]
# [DEF:ImportArtifactsRequest:Class]
# @PURPOSE: Request schema for candidate artifact import endpoint.
class ImportArtifactsRequest(BaseModel):
artifacts: List[Dict[str, Any]] = Field(default_factory=list)
# [/DEF:ImportArtifactsRequest:Class]
# [DEF:BuildManifestRequest:Class]
# @PURPOSE: Request schema for manifest build endpoint.
class BuildManifestRequest(BaseModel):
created_by: str = Field(default="system")
# [/DEF:BuildManifestRequest:Class]
# [DEF:CreateComplianceRunRequest:Class]
# @PURPOSE: Request schema for compliance run creation with optional manifest pinning.
class CreateComplianceRunRequest(BaseModel):
requested_by: str = Field(min_length=1)
manifest_id: str | None = None
# [/DEF:CreateComplianceRunRequest:Class]
# [DEF:register_candidate_v2_endpoint:Function]
# @PURPOSE: Register a clean-release candidate for headless lifecycle.
# @PRE: Candidate identifier is unique.
# @POST: Candidate is persisted in DRAFT status.
@router.post("/candidates", response_model=CandidateDTO, status_code=status.HTTP_201_CREATED)
async def register_candidate_v2_endpoint(
payload: RegisterCandidateRequest,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
existing = repository.get_candidate(payload.id)
if existing is not None:
raise HTTPException(status_code=409, detail={"message": "Candidate already exists", "code": "CANDIDATE_EXISTS"})
candidate = ReleaseCandidate(
id=payload.id,
version=payload.version,
source_snapshot_ref=payload.source_snapshot_ref,
created_by=payload.created_by,
created_at=datetime.now(timezone.utc),
status=CandidateStatus.DRAFT.value,
)
repository.save_candidate(candidate)
return CandidateDTO(
id=candidate.id,
version=candidate.version,
source_snapshot_ref=candidate.source_snapshot_ref,
created_at=candidate.created_at,
created_by=candidate.created_by,
status=CandidateStatus(candidate.status),
)
# [/DEF:register_candidate_v2_endpoint:Function]
# [DEF:import_candidate_artifacts_v2_endpoint:Function]
# @PURPOSE: Import candidate artifacts in headless flow.
# @PRE: Candidate exists and artifacts array is non-empty.
# @POST: Artifacts are persisted and candidate advances to PREPARED if it was DRAFT.
@router.post("/candidates/{candidate_id}/artifacts")
async def import_candidate_artifacts_v2_endpoint(
candidate_id: str,
payload: ImportArtifactsRequest,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
candidate = repository.get_candidate(candidate_id)
if candidate is None:
raise HTTPException(status_code=404, detail={"message": "Candidate not found", "code": "CANDIDATE_NOT_FOUND"})
if not payload.artifacts:
raise HTTPException(status_code=400, detail={"message": "Artifacts list is required", "code": "ARTIFACTS_EMPTY"})
for artifact in payload.artifacts:
required = ("id", "path", "sha256", "size")
for field_name in required:
if field_name not in artifact:
raise HTTPException(
status_code=400,
detail={"message": f"Artifact missing field '{field_name}'", "code": "ARTIFACT_INVALID"},
)
artifact_model = CandidateArtifact(
id=str(artifact["id"]),
candidate_id=candidate_id,
path=str(artifact["path"]),
sha256=str(artifact["sha256"]),
size=int(artifact["size"]),
detected_category=artifact.get("detected_category"),
declared_category=artifact.get("declared_category"),
source_uri=artifact.get("source_uri"),
source_host=artifact.get("source_host"),
metadata_json=artifact.get("metadata_json", {}),
)
repository.save_artifact(artifact_model)
if candidate.status == CandidateStatus.DRAFT.value:
candidate.transition_to(CandidateStatus.PREPARED)
repository.save_candidate(candidate)
return {"status": "success"}
# [/DEF:import_candidate_artifacts_v2_endpoint:Function]
# [DEF:build_candidate_manifest_v2_endpoint:Function]
# @PURPOSE: Build immutable manifest snapshot for prepared candidate.
# @PRE: Candidate exists and has imported artifacts.
# @POST: Returns created ManifestDTO with incremented version.
@router.post("/candidates/{candidate_id}/manifests", response_model=ManifestDTO, status_code=status.HTTP_201_CREATED)
async def build_candidate_manifest_v2_endpoint(
candidate_id: str,
payload: BuildManifestRequest,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
from ...services.clean_release.manifest_service import build_manifest_snapshot
try:
manifest = build_manifest_snapshot(
repository=repository,
candidate_id=candidate_id,
created_by=payload.created_by,
)
except ValueError as exc:
raise HTTPException(status_code=400, detail={"message": str(exc), "code": "MANIFEST_BUILD_ERROR"})
return ManifestDTO(
id=manifest.id,
candidate_id=manifest.candidate_id,
manifest_version=manifest.manifest_version,
manifest_digest=manifest.manifest_digest,
artifacts_digest=manifest.artifacts_digest,
created_at=manifest.created_at,
created_by=manifest.created_by,
source_snapshot_ref=manifest.source_snapshot_ref,
content_json=manifest.content_json,
)
# [/DEF:build_candidate_manifest_v2_endpoint:Function]
# [DEF:get_candidate_overview_v2_endpoint:Function]
# @PURPOSE: Return expanded candidate overview DTO for headless lifecycle visibility.
# @PRE: Candidate exists.
# @POST: Returns CandidateOverviewDTO built from the same repository state used by headless US1 endpoints.
@router.get("/candidates/{candidate_id}/overview", response_model=CandidateOverviewDTO)
async def get_candidate_overview_v2_endpoint(
candidate_id: str,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
candidate = repository.get_candidate(candidate_id)
if candidate is None:
raise HTTPException(status_code=404, detail={"message": "Candidate not found", "code": "CANDIDATE_NOT_FOUND"})
manifests = repository.get_manifests_by_candidate(candidate_id)
latest_manifest = sorted(manifests, key=lambda m: m.manifest_version, reverse=True)[0] if manifests else None
runs = [run for run in repository.check_runs.values() if run.candidate_id == candidate_id]
latest_run = sorted(runs, key=lambda run: run.requested_at or datetime.min.replace(tzinfo=timezone.utc), reverse=True)[0] if runs else None
latest_report = None
if latest_run is not None:
latest_report = next((r for r in repository.reports.values() if r.run_id == latest_run.id), None)
latest_policy_snapshot = repository.get_policy(latest_run.policy_snapshot_id) if latest_run else None
latest_registry_snapshot = repository.get_registry(latest_run.registry_snapshot_id) if latest_run else None
approval_decisions = getattr(repository, "approval_decisions", [])
latest_approval = (
sorted(
[item for item in approval_decisions if item.candidate_id == candidate_id],
key=lambda item: item.decided_at or datetime.min.replace(tzinfo=timezone.utc),
reverse=True,
)[0]
if approval_decisions
and any(item.candidate_id == candidate_id for item in approval_decisions)
else None
)
publication_records = getattr(repository, "publication_records", [])
latest_publication = (
sorted(
[item for item in publication_records if item.candidate_id == candidate_id],
key=lambda item: item.published_at or datetime.min.replace(tzinfo=timezone.utc),
reverse=True,
)[0]
if publication_records
and any(item.candidate_id == candidate_id for item in publication_records)
else None
)
return CandidateOverviewDTO(
candidate_id=candidate.id,
version=candidate.version,
source_snapshot_ref=candidate.source_snapshot_ref,
status=CandidateStatus(candidate.status),
latest_manifest_id=latest_manifest.id if latest_manifest else None,
latest_manifest_digest=latest_manifest.manifest_digest if latest_manifest else None,
latest_run_id=latest_run.id if latest_run else None,
latest_run_status=RunStatus(latest_run.status) if latest_run else None,
latest_report_id=latest_report.id if latest_report else None,
latest_report_final_status=ComplianceDecision(latest_report.final_status) if latest_report else None,
latest_policy_snapshot_id=latest_policy_snapshot.id if latest_policy_snapshot else None,
latest_policy_version=latest_policy_snapshot.policy_version if latest_policy_snapshot else None,
latest_registry_snapshot_id=latest_registry_snapshot.id if latest_registry_snapshot else None,
latest_registry_version=latest_registry_snapshot.registry_version if latest_registry_snapshot else None,
latest_approval_decision=latest_approval.decision if latest_approval else None,
latest_publication_id=latest_publication.id if latest_publication else None,
latest_publication_status=latest_publication.status if latest_publication else None,
)
# [/DEF:get_candidate_overview_v2_endpoint:Function]
# [DEF:prepare_candidate_endpoint:Function]
# @PURPOSE: Prepare candidate with policy evaluation and deterministic manifest generation.
# @PRE: Candidate and active policy exist in repository.
# @POST: Returns preparation result including manifest reference and violations.
@router.post("/candidates/prepare")
async def prepare_candidate_endpoint(
payload: PrepareCandidateRequest,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
try:
result = prepare_candidate(
repository=repository,
candidate_id=payload.candidate_id,
artifacts=payload.artifacts,
sources=payload.sources,
operator_id=payload.operator_id,
)
return result
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail={"message": str(exc), "code": "CLEAN_PREPARATION_ERROR"},
)
# [/DEF:prepare_candidate_endpoint:Function]
# [DEF:start_check:Function]
# @PURPOSE: Start and finalize a clean compliance check run and persist report artifacts.
# @PRE: Active policy and candidate exist.
# @POST: Returns accepted payload with check_run_id and started_at.
@router.post("/checks", status_code=status.HTTP_202_ACCEPTED)
async def start_check(
payload: StartCheckRequest,
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
with belief_scope("clean_release.start_check"):
logger.reason("Starting clean-release compliance check run")
policy = repository.get_active_policy()
if policy is None:
raise HTTPException(status_code=409, detail={"message": "Active policy not found", "code": "POLICY_NOT_FOUND"})
candidate = repository.get_candidate(payload.candidate_id)
if candidate is None:
raise HTTPException(status_code=409, detail={"message": "Candidate not found", "code": "CANDIDATE_NOT_FOUND"})
manifests = repository.get_manifests_by_candidate(payload.candidate_id)
if not manifests:
raise HTTPException(status_code=409, detail={"message": "No manifest found for candidate", "code": "MANIFEST_NOT_FOUND"})
latest_manifest = sorted(manifests, key=lambda m: m.manifest_version, reverse=True)[0]
orchestrator = CleanComplianceOrchestrator(repository)
run = orchestrator.start_check_run(
candidate_id=payload.candidate_id,
policy_id=policy.id,
requested_by=payload.triggered_by,
manifest_id=latest_manifest.id,
)
forced = [
ComplianceStageRun(
id=f"stage-{run.id}-1",
run_id=run.id,
stage_name=ComplianceStageName.DATA_PURITY.value,
status=RunStatus.SUCCEEDED.value,
decision=ComplianceDecision.PASSED.value,
details_json={"message": "ok"}
),
ComplianceStageRun(
id=f"stage-{run.id}-2",
run_id=run.id,
stage_name=ComplianceStageName.INTERNAL_SOURCES_ONLY.value,
status=RunStatus.SUCCEEDED.value,
decision=ComplianceDecision.PASSED.value,
details_json={"message": "ok"}
),
ComplianceStageRun(
id=f"stage-{run.id}-3",
run_id=run.id,
stage_name=ComplianceStageName.NO_EXTERNAL_ENDPOINTS.value,
status=RunStatus.SUCCEEDED.value,
decision=ComplianceDecision.PASSED.value,
details_json={"message": "ok"}
),
ComplianceStageRun(
id=f"stage-{run.id}-4",
run_id=run.id,
stage_name=ComplianceStageName.MANIFEST_CONSISTENCY.value,
status=RunStatus.SUCCEEDED.value,
decision=ComplianceDecision.PASSED.value,
details_json={"message": "ok"}
),
]
run = orchestrator.execute_stages(run, forced_results=forced)
run = orchestrator.finalize_run(run)
if run.final_status == ComplianceDecision.BLOCKED.value:
logger.explore("Run ended as BLOCKED, persisting synthetic external-source violation")
violation = ComplianceViolation(
id=f"viol-{run.id}",
run_id=run.id,
stage_name=ComplianceStageName.NO_EXTERNAL_ENDPOINTS.value,
code="EXTERNAL_SOURCE_DETECTED",
severity=ViolationSeverity.CRITICAL.value,
message="Replace with approved internal server",
evidence_json={"location": "external.example.com"}
)
repository.save_violation(violation)
builder = ComplianceReportBuilder(repository)
report = builder.build_report_payload(run, repository.get_violations_by_run(run.id))
builder.persist_report(report)
logger.reflect(f"Compliance report persisted for run_id={run.id}")
return {
"check_run_id": run.id,
"candidate_id": run.candidate_id,
"status": "running",
"started_at": run.started_at.isoformat() if run.started_at else None,
}
# [/DEF:start_check:Function]
# [DEF:get_check_status:Function]
# @PURPOSE: Return terminal/intermediate status payload for a check run.
# @PRE: check_run_id references an existing run.
# @POST: Deterministic payload shape includes checks and violations arrays.
@router.get("/checks/{check_run_id}")
async def get_check_status(check_run_id: str, repository: CleanReleaseRepository = Depends(get_clean_release_repository)):
with belief_scope("clean_release.get_check_status"):
run = repository.get_check_run(check_run_id)
if run is None:
raise HTTPException(status_code=404, detail={"message": "Check run not found", "code": "CHECK_NOT_FOUND"})
logger.reflect(f"Returning check status for check_run_id={check_run_id}")
return {
"check_run_id": run.id,
"candidate_id": run.candidate_id,
"final_status": run.final_status,
"started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"checks": [], # TODO: Map stages if needed
"violations": [], # TODO: Map violations if needed
}
# [/DEF:get_check_status:Function]
# [DEF:get_report:Function]
# @PURPOSE: Return persisted compliance report by report_id.
# @PRE: report_id references an existing report.
# @POST: Returns serialized report object.
@router.get("/reports/{report_id}")
async def get_report(report_id: str, repository: CleanReleaseRepository = Depends(get_clean_release_repository)):
with belief_scope("clean_release.get_report"):
report = repository.get_report(report_id)
if report is None:
raise HTTPException(status_code=404, detail={"message": "Report not found", "code": "REPORT_NOT_FOUND"})
logger.reflect(f"Returning compliance report report_id={report_id}")
return report.model_dump()
# [/DEF:get_report:Function]
# [/DEF:backend.src.api.routes.clean_release:Module]

View File

@@ -1,216 +0,0 @@
# [DEF:backend.src.api.routes.clean_release_v2:Module]
# @TIER: STANDARD
# @SEMANTICS: api, clean-release, v2, headless
# @PURPOSE: Redesigned clean release API for headless candidate lifecycle.
# @LAYER: API
from fastapi import APIRouter, Depends, HTTPException, status
from typing import List, Dict, Any
from datetime import datetime, timezone
from ...services.clean_release.approval_service import approve_candidate, reject_candidate
from ...services.clean_release.publication_service import publish_candidate, revoke_publication
from ...services.clean_release.repository import CleanReleaseRepository
from ...dependencies import get_clean_release_repository
from ...services.clean_release.enums import CandidateStatus
from ...models.clean_release import ReleaseCandidate, CandidateArtifact, DistributionManifest
from ...services.clean_release.dto import CandidateDTO, ManifestDTO
router = APIRouter(prefix="/api/v2/clean-release", tags=["Clean Release V2"])
class ApprovalRequest(dict):
pass
class PublishRequest(dict):
pass
class RevokeRequest(dict):
pass
@router.post("/candidates", response_model=CandidateDTO, status_code=status.HTTP_201_CREATED)
async def register_candidate(
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository)
):
candidate = ReleaseCandidate(
id=payload["id"],
version=payload["version"],
source_snapshot_ref=payload["source_snapshot_ref"],
created_by=payload["created_by"],
created_at=datetime.now(timezone.utc),
status=CandidateStatus.DRAFT.value
)
repository.save_candidate(candidate)
return CandidateDTO(
id=candidate.id,
version=candidate.version,
source_snapshot_ref=candidate.source_snapshot_ref,
created_at=candidate.created_at,
created_by=candidate.created_by,
status=CandidateStatus(candidate.status)
)
@router.post("/candidates/{candidate_id}/artifacts")
async def import_artifacts(
candidate_id: str,
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository)
):
candidate = repository.get_candidate(candidate_id)
if not candidate:
raise HTTPException(status_code=404, detail="Candidate not found")
for art_data in payload.get("artifacts", []):
artifact = CandidateArtifact(
id=art_data["id"],
candidate_id=candidate_id,
path=art_data["path"],
sha256=art_data["sha256"],
size=art_data["size"]
)
# In a real repo we'd have save_artifact
# repository.save_artifact(artifact)
pass
return {"status": "success"}
@router.post("/candidates/{candidate_id}/manifests", response_model=ManifestDTO, status_code=status.HTTP_201_CREATED)
async def build_manifest(
candidate_id: str,
repository: CleanReleaseRepository = Depends(get_clean_release_repository)
):
candidate = repository.get_candidate(candidate_id)
if not candidate:
raise HTTPException(status_code=404, detail="Candidate not found")
manifest = DistributionManifest(
id=f"manifest-{candidate_id}",
candidate_id=candidate_id,
manifest_version=1,
manifest_digest="hash-123",
artifacts_digest="art-hash-123",
created_by="system",
created_at=datetime.now(timezone.utc),
source_snapshot_ref=candidate.source_snapshot_ref,
content_json={"items": [], "summary": {}}
)
repository.save_manifest(manifest)
return ManifestDTO(
id=manifest.id,
candidate_id=manifest.candidate_id,
manifest_version=manifest.manifest_version,
manifest_digest=manifest.manifest_digest,
artifacts_digest=manifest.artifacts_digest,
created_at=manifest.created_at,
created_by=manifest.created_by,
source_snapshot_ref=manifest.source_snapshot_ref,
content_json=manifest.content_json
)
@router.post("/candidates/{candidate_id}/approve")
async def approve_candidate_endpoint(
candidate_id: str,
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
try:
decision = approve_candidate(
repository=repository,
candidate_id=candidate_id,
report_id=str(payload["report_id"]),
decided_by=str(payload["decided_by"]),
comment=payload.get("comment"),
)
except Exception as exc: # noqa: BLE001
raise HTTPException(status_code=409, detail={"message": str(exc), "code": "APPROVAL_GATE_ERROR"})
return {"status": "ok", "decision": decision.decision, "decision_id": decision.id}
@router.post("/candidates/{candidate_id}/reject")
async def reject_candidate_endpoint(
candidate_id: str,
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
try:
decision = reject_candidate(
repository=repository,
candidate_id=candidate_id,
report_id=str(payload["report_id"]),
decided_by=str(payload["decided_by"]),
comment=payload.get("comment"),
)
except Exception as exc: # noqa: BLE001
raise HTTPException(status_code=409, detail={"message": str(exc), "code": "APPROVAL_GATE_ERROR"})
return {"status": "ok", "decision": decision.decision, "decision_id": decision.id}
@router.post("/candidates/{candidate_id}/publish")
async def publish_candidate_endpoint(
candidate_id: str,
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
try:
publication = publish_candidate(
repository=repository,
candidate_id=candidate_id,
report_id=str(payload["report_id"]),
published_by=str(payload["published_by"]),
target_channel=str(payload["target_channel"]),
publication_ref=payload.get("publication_ref"),
)
except Exception as exc: # noqa: BLE001
raise HTTPException(status_code=409, detail={"message": str(exc), "code": "PUBLICATION_GATE_ERROR"})
return {
"status": "ok",
"publication": {
"id": publication.id,
"candidate_id": publication.candidate_id,
"report_id": publication.report_id,
"published_by": publication.published_by,
"published_at": publication.published_at.isoformat() if publication.published_at else None,
"target_channel": publication.target_channel,
"publication_ref": publication.publication_ref,
"status": publication.status,
},
}
@router.post("/publications/{publication_id}/revoke")
async def revoke_publication_endpoint(
publication_id: str,
payload: Dict[str, Any],
repository: CleanReleaseRepository = Depends(get_clean_release_repository),
):
try:
publication = revoke_publication(
repository=repository,
publication_id=publication_id,
revoked_by=str(payload["revoked_by"]),
comment=payload.get("comment"),
)
except Exception as exc: # noqa: BLE001
raise HTTPException(status_code=409, detail={"message": str(exc), "code": "PUBLICATION_GATE_ERROR"})
return {
"status": "ok",
"publication": {
"id": publication.id,
"candidate_id": publication.candidate_id,
"report_id": publication.report_id,
"published_by": publication.published_by,
"published_at": publication.published_at.isoformat() if publication.published_at else None,
"target_channel": publication.target_channel,
"publication_ref": publication.publication_ref,
"status": publication.status,
},
}
# [/DEF:backend.src.api.routes.clean_release_v2:Module]

View File

@@ -1,6 +1,6 @@
# [DEF:backend.src.api.routes.dashboards:Module]
#
# @TIER: CRITICAL
# @TIER: STANDARD
# @SEMANTICS: api, dashboards, resources, hub
# @PURPOSE: API endpoints for the Dashboard Hub - listing dashboards with Git and task status
# @LAYER: API
@@ -9,53 +9,18 @@
# @RELATION: DEPENDS_ON -> backend.src.core.superset_client
#
# @INVARIANT: All dashboard responses include git_status and last_task metadata
#
# @TEST_CONTRACT: DashboardsAPI -> {
# required_fields: {env_id: string, page: integer, page_size: integer},
# optional_fields: {search: string},
# invariants: ["Pagination must be valid", "Environment must exist"]
# }
#
# @TEST_FIXTURE: dashboard_list_happy -> {
# "env_id": "prod",
# "expected_count": 1,
# "dashboards": [{"id": 1, "title": "Main Revenue"}]
# }
#
# @TEST_EDGE: pagination_zero_page -> {"env_id": "prod", "page": 0, "status": 400}
# @TEST_EDGE: pagination_oversize -> {"env_id": "prod", "page_size": 101, "status": 400}
# @TEST_EDGE: missing_env -> {"env_id": "ghost", "status": 404}
# @TEST_EDGE: empty_dashboards -> {"env_id": "empty_env", "expected_total": 0}
# @TEST_EDGE: external_superset_failure -> {"env_id": "bad_conn", "status": 503}
#
# @TEST_INVARIANT: metadata_consistency -> verifies: [dashboard_list_happy, empty_dashboards]
#
# [SECTION: IMPORTS]
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from fastapi.responses import JSONResponse
from typing import List, Optional, Dict, Any, Literal
from typing import List, Optional, Dict, Any
import re
from urllib.parse import urlparse
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from ...dependencies import (
get_config_manager,
get_task_manager,
get_resource_service,
get_mapping_service,
get_current_user,
has_permission,
)
from ...core.database import get_db
from ...core.async_superset_client import AsyncSupersetClient
from ...dependencies import get_config_manager, get_task_manager, get_resource_service, get_mapping_service, has_permission
from ...core.logger import logger, belief_scope
from ...core.superset_client import SupersetClient
from ...core.superset_profile_lookup import SupersetAccountLookupAdapter
from ...core.utils.network import DashboardNotFoundError
from ...models.auth import User
from ...services.profile_service import ProfileService
from ...services.resource_service import ResourceService
# [/SECTION]
router = APIRouter(prefix="/api/dashboards", tags=["Dashboards"])
@@ -71,11 +36,7 @@ class GitStatus(BaseModel):
# [DEF:LastTask:DataClass]
class LastTask(BaseModel):
task_id: Optional[str] = None
status: Optional[str] = Field(
None,
pattern="^PENDING|RUNNING|SUCCESS|FAILED|ERROR|AWAITING_INPUT|WAITING_INPUT|AWAITING_MAPPING$",
)
validation_status: Optional[str] = Field(None, pattern="^PASS|FAIL|WARN|UNKNOWN$")
status: Optional[str] = Field(None, pattern="^RUNNING|SUCCESS|ERROR|WAITING_INPUT$")
# [/DEF:LastTask:DataClass]
# [DEF:DashboardItem:DataClass]
@@ -85,24 +46,10 @@ class DashboardItem(BaseModel):
slug: Optional[str] = None
url: Optional[str] = None
last_modified: Optional[str] = None
created_by: Optional[str] = None
modified_by: Optional[str] = None
owners: Optional[List[str]] = None
git_status: Optional[GitStatus] = None
last_task: Optional[LastTask] = None
# [/DEF:DashboardItem:DataClass]
# [DEF:EffectiveProfileFilter:DataClass]
class EffectiveProfileFilter(BaseModel):
applied: bool
source_page: Literal["dashboards_main", "other"] = "dashboards_main"
override_show_all: bool = False
username: Optional[str] = None
match_logic: Optional[
Literal["owners_or_modified_by", "slug_only", "owners_or_modified_by+slug_only"]
] = None
# [/DEF:EffectiveProfileFilter:DataClass]
# [DEF:DashboardsResponse:DataClass]
class DashboardsResponse(BaseModel):
dashboards: List[DashboardItem]
@@ -110,7 +57,6 @@ class DashboardsResponse(BaseModel):
page: int
page_size: int
total_pages: int
effective_profile_filter: Optional[EffectiveProfileFilter] = None
# [/DEF:DashboardsResponse:DataClass]
# [DEF:DashboardChartItem:DataClass]
@@ -180,304 +126,6 @@ class DatabaseMappingsResponse(BaseModel):
mappings: List[DatabaseMapping]
# [/DEF:DatabaseMappingsResponse:DataClass]
# [DEF:_find_dashboard_id_by_slug:Function]
# @PURPOSE: Resolve dashboard numeric ID by slug using Superset list endpoint.
# @PRE: `dashboard_slug` is non-empty.
# @POST: Returns dashboard ID when found, otherwise None.
def _find_dashboard_id_by_slug(
client: SupersetClient,
dashboard_slug: str,
) -> Optional[int]:
query_variants = [
{"filters": [{"col": "slug", "opr": "eq", "value": dashboard_slug}], "page": 0, "page_size": 1},
{"filters": [{"col": "slug", "op": "eq", "value": dashboard_slug}], "page": 0, "page_size": 1},
]
for query in query_variants:
try:
_count, dashboards = client.get_dashboards_page(query=query)
if dashboards:
resolved_id = dashboards[0].get("id")
if resolved_id is not None:
return int(resolved_id)
except Exception:
continue
return None
# [/DEF:_find_dashboard_id_by_slug:Function]
# [DEF:_resolve_dashboard_id_from_ref:Function]
# @PURPOSE: Resolve dashboard ID from slug-first reference with numeric fallback.
# @PRE: `dashboard_ref` is provided in route path.
# @POST: Returns a valid dashboard ID or raises HTTPException(404).
def _resolve_dashboard_id_from_ref(
dashboard_ref: str,
client: SupersetClient,
) -> int:
normalized_ref = str(dashboard_ref or "").strip()
if not normalized_ref:
raise HTTPException(status_code=404, detail="Dashboard not found")
# Slug-first: even if ref looks numeric, try slug first.
slug_match_id = _find_dashboard_id_by_slug(client, normalized_ref)
if slug_match_id is not None:
return slug_match_id
if normalized_ref.isdigit():
return int(normalized_ref)
raise HTTPException(status_code=404, detail="Dashboard not found")
# [/DEF:_resolve_dashboard_id_from_ref:Function]
# [DEF:_find_dashboard_id_by_slug_async:Function]
# @PURPOSE: Resolve dashboard numeric ID by slug using async Superset list endpoint.
# @PRE: dashboard_slug is non-empty.
# @POST: Returns dashboard ID when found, otherwise None.
async def _find_dashboard_id_by_slug_async(
client: AsyncSupersetClient,
dashboard_slug: str,
) -> Optional[int]:
query_variants = [
{"filters": [{"col": "slug", "opr": "eq", "value": dashboard_slug}], "page": 0, "page_size": 1},
{"filters": [{"col": "slug", "op": "eq", "value": dashboard_slug}], "page": 0, "page_size": 1},
]
for query in query_variants:
try:
_count, dashboards = await client.get_dashboards_page_async(query=query)
if dashboards:
resolved_id = dashboards[0].get("id")
if resolved_id is not None:
return int(resolved_id)
except Exception:
continue
return None
# [/DEF:_find_dashboard_id_by_slug_async:Function]
# [DEF:_resolve_dashboard_id_from_ref_async:Function]
# @PURPOSE: Resolve dashboard ID from slug-first reference using async Superset client.
# @PRE: dashboard_ref is provided in route path.
# @POST: Returns valid dashboard ID or raises HTTPException(404).
async def _resolve_dashboard_id_from_ref_async(
dashboard_ref: str,
client: AsyncSupersetClient,
) -> int:
normalized_ref = str(dashboard_ref or "").strip()
if not normalized_ref:
raise HTTPException(status_code=404, detail="Dashboard not found")
slug_match_id = await _find_dashboard_id_by_slug_async(client, normalized_ref)
if slug_match_id is not None:
return slug_match_id
if normalized_ref.isdigit():
return int(normalized_ref)
raise HTTPException(status_code=404, detail="Dashboard not found")
# [/DEF:_resolve_dashboard_id_from_ref_async:Function]
# [DEF:_normalize_filter_values:Function]
# @PURPOSE: Normalize query filter values to lower-cased non-empty tokens.
# @PRE: values may be None or list of strings.
# @POST: Returns trimmed normalized list preserving input order.
def _normalize_filter_values(values: Optional[List[str]]) -> List[str]:
if not values:
return []
normalized: List[str] = []
for value in values:
token = str(value or "").strip().lower()
if token:
normalized.append(token)
return normalized
# [/DEF:_normalize_filter_values:Function]
# [DEF:_dashboard_git_filter_value:Function]
# @PURPOSE: Build comparable git status token for dashboards filtering.
# @PRE: dashboard payload may contain git_status or None.
# @POST: Returns one of ok|diff|no_repo|error|pending.
def _dashboard_git_filter_value(dashboard: Dict[str, Any]) -> str:
git_status = dashboard.get("git_status") or {}
sync_status = str(git_status.get("sync_status") or "").strip().upper()
has_repo = git_status.get("has_repo")
if has_repo is False or sync_status == "NO_REPO":
return "no_repo"
if sync_status == "DIFF":
return "diff"
if sync_status == "OK":
return "ok"
if sync_status == "ERROR":
return "error"
return "pending"
# [/DEF:_dashboard_git_filter_value:Function]
# [DEF:_normalize_actor_alias_token:Function]
# @PURPOSE: Normalize actor alias token to comparable trim+lower text.
# @PRE: value can be scalar/None.
# @POST: Returns normalized token or None.
def _normalize_actor_alias_token(value: Any) -> Optional[str]:
token = str(value or "").strip().lower()
return token or None
# [/DEF:_normalize_actor_alias_token:Function]
# [DEF:_normalize_owner_display_token:Function]
# @PURPOSE: Project owner payload value into stable display string for API response contracts.
# @PRE: owner can be scalar, dict or None.
# @POST: Returns trimmed non-empty owner display token or None.
def _normalize_owner_display_token(owner: Any) -> Optional[str]:
if owner is None:
return None
if isinstance(owner, dict):
username = str(owner.get("username") or owner.get("user_name") or owner.get("name") or "").strip()
full_name = str(owner.get("full_name") or "").strip()
first_name = str(owner.get("first_name") or "").strip()
last_name = str(owner.get("last_name") or "").strip()
combined = " ".join(part for part in [first_name, last_name] if part).strip()
email = str(owner.get("email") or "").strip()
for candidate in [username, full_name, combined, email]:
if candidate:
return candidate
return None
normalized = str(owner).strip()
return normalized or None
# [/DEF:_normalize_owner_display_token:Function]
# [DEF:_normalize_dashboard_owner_values:Function]
# @PURPOSE: Normalize dashboard owners payload to optional list of display strings.
# @PRE: owners payload can be None, scalar, or list with mixed values.
# @POST: Returns deduplicated owner labels preserving order, or None when absent.
def _normalize_dashboard_owner_values(owners: Any) -> Optional[List[str]]:
if owners is None:
return None
raw_items: List[Any]
if isinstance(owners, list):
raw_items = owners
else:
raw_items = [owners]
normalized: List[str] = []
for owner in raw_items:
token = _normalize_owner_display_token(owner)
if token and token not in normalized:
normalized.append(token)
return normalized
# [/DEF:_normalize_dashboard_owner_values:Function]
# [DEF:_project_dashboard_response_items:Function]
# @PURPOSE: Project dashboard payloads to response-contract-safe shape.
# @PRE: dashboards is a list of dict-like dashboard payloads.
# @POST: Returned items satisfy DashboardItem owners=list[str]|None contract.
def _project_dashboard_response_items(dashboards: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
projected: List[Dict[str, Any]] = []
for dashboard in dashboards:
projected_dashboard = dict(dashboard)
projected_dashboard["owners"] = _normalize_dashboard_owner_values(
projected_dashboard.get("owners")
)
projected.append(projected_dashboard)
return projected
# [/DEF:_project_dashboard_response_items:Function]
# [DEF:_resolve_profile_actor_aliases:Function]
# @PURPOSE: Resolve stable actor aliases for profile filtering without per-dashboard detail fan-out.
# @PRE: bound username is available and env is valid.
# @POST: Returns at least normalized username; may include Superset display-name alias.
# @SIDE_EFFECT: Performs at most one Superset users-lookup request.
def _resolve_profile_actor_aliases(env: Any, bound_username: str) -> List[str]:
normalized_bound = _normalize_actor_alias_token(bound_username)
if not normalized_bound:
return []
aliases: List[str] = [normalized_bound]
try:
client = SupersetClient(env)
adapter = SupersetAccountLookupAdapter(
network_client=client.network,
environment_id=str(getattr(env, "id", "")),
)
lookup_payload = adapter.get_users_page(
search=normalized_bound,
page_index=0,
page_size=20,
sort_column="username",
sort_order="asc",
)
lookup_items = (
lookup_payload.get("items", [])
if isinstance(lookup_payload, dict)
else []
)
matched_item: Optional[Dict[str, Any]] = None
for item in lookup_items:
if not isinstance(item, dict):
continue
if _normalize_actor_alias_token(item.get("username")) == normalized_bound:
matched_item = item
break
if matched_item is None:
for item in lookup_items:
if isinstance(item, dict):
matched_item = item
break
display_alias = _normalize_actor_alias_token(
(matched_item or {}).get("display_name")
)
if display_alias and display_alias not in aliases:
aliases.append(display_alias)
logger.reflect(
"[REFLECT] Resolved profile actor aliases "
f"(env={getattr(env, 'id', None)}, bound_username={normalized_bound!r}, "
f"lookup_items={len(lookup_items)}, aliases={aliases!r})"
)
except Exception as alias_error:
logger.explore(
"[EXPLORE] Failed to resolve profile actor aliases via Superset users lookup "
f"(env={getattr(env, 'id', None)}, bound_username={normalized_bound!r}): {alias_error}"
)
return aliases
# [/DEF:_resolve_profile_actor_aliases:Function]
# [DEF:_matches_dashboard_actor_aliases:Function]
# @PURPOSE: Apply profile actor matching against multiple aliases (username + optional display name).
# @PRE: actor_aliases contains normalized non-empty tokens.
# @POST: Returns True when any alias matches owners OR modified_by.
def _matches_dashboard_actor_aliases(
profile_service: ProfileService,
actor_aliases: List[str],
owners: Optional[Any],
modified_by: Optional[str],
) -> bool:
for actor_alias in actor_aliases:
if profile_service.matches_dashboard_actor(
bound_username=actor_alias,
owners=owners,
modified_by=modified_by,
):
return True
return False
# [/DEF:_matches_dashboard_actor_aliases:Function]
# [DEF:get_dashboards:Function]
# @PURPOSE: Fetch list of dashboards from a specific environment with Git status and last task status
# @PRE: env_id must be a valid environment ID
@@ -485,7 +133,6 @@ def _matches_dashboard_actor_aliases(
# @PRE: page_size must be between 1 and 100 if provided
# @POST: Returns a list of dashboards with enhanced metadata and pagination info
# @POST: Response includes pagination metadata (page, page_size, total, total_pages)
# @POST: Response includes effective profile filter metadata for main dashboards page context
# @PARAM: env_id (str) - The environment ID to fetch dashboards from
# @PARAM: search (Optional[str]) - Filter by title/slug
# @PARAM: page (Optional[int]) - Page number (default: 1)
@@ -498,283 +145,62 @@ async def get_dashboards(
search: Optional[str] = None,
page: int = 1,
page_size: int = 10,
page_context: Literal["dashboards_main", "other"] = Query(default="dashboards_main"),
apply_profile_default: bool = Query(default=True),
override_show_all: bool = Query(default=False),
filter_title: Optional[List[str]] = Query(default=None),
filter_git_status: Optional[List[str]] = Query(default=None),
filter_llm_status: Optional[List[str]] = Query(default=None),
filter_changed_on: Optional[List[str]] = Query(default=None),
filter_actor: Optional[List[str]] = Query(default=None),
config_manager=Depends(get_config_manager),
task_manager=Depends(get_task_manager),
resource_service=Depends(get_resource_service),
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
_ = Depends(has_permission("plugin:migration", "READ"))
):
with belief_scope(
"get_dashboards",
(
f"env_id={env_id}, search={search}, page={page}, page_size={page_size}, "
f"page_context={page_context}, apply_profile_default={apply_profile_default}, "
f"override_show_all={override_show_all}"
),
):
with belief_scope("get_dashboards", f"env_id={env_id}, search={search}, page={page}, page_size={page_size}"):
# Validate pagination parameters
if page < 1:
logger.error(f"[get_dashboards][Coherence:Failed] Invalid page: {page}")
raise HTTPException(status_code=400, detail="Page must be >= 1")
if page_size < 1 or page_size > 100:
logger.error(f"[get_dashboards][Coherence:Failed] Invalid page_size: {page_size}")
raise HTTPException(status_code=400, detail="Page size must be between 1 and 100")
# Validate environment exists
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
logger.error(f"[get_dashboards][Coherence:Failed] Environment not found: {env_id}")
raise HTTPException(status_code=404, detail="Environment not found")
profile_service = ProfileService(db=db, config_manager=config_manager)
bound_username: Optional[str] = None
can_apply_profile_filter = False
can_apply_slug_filter = False
effective_profile_filter = EffectiveProfileFilter(
applied=False,
source_page=page_context,
override_show_all=bool(override_show_all),
username=None,
match_logic=None,
)
try:
profile_preference = profile_service.get_my_preference(current_user).preference
normalized_username = str(
getattr(profile_preference, "superset_username_normalized", None) or ""
).strip().lower()
raw_username = str(
getattr(profile_preference, "superset_username", None) or ""
).strip().lower()
bound_username = normalized_username or raw_username or None
can_apply_profile_filter = (
page_context == "dashboards_main"
and bool(apply_profile_default)
and not bool(override_show_all)
and bool(getattr(profile_preference, "show_only_my_dashboards", False))
and bool(bound_username)
)
can_apply_slug_filter = (
page_context == "dashboards_main"
and bool(apply_profile_default)
and not bool(override_show_all)
and bool(getattr(profile_preference, "show_only_slug_dashboards", True))
)
profile_match_logic = None
if can_apply_profile_filter and can_apply_slug_filter:
profile_match_logic = "owners_or_modified_by+slug_only"
elif can_apply_profile_filter:
profile_match_logic = "owners_or_modified_by"
elif can_apply_slug_filter:
profile_match_logic = "slug_only"
effective_profile_filter = EffectiveProfileFilter(
applied=bool(can_apply_profile_filter or can_apply_slug_filter),
source_page=page_context,
override_show_all=bool(override_show_all),
username=bound_username if can_apply_profile_filter else None,
match_logic=profile_match_logic,
)
except Exception as profile_error:
logger.explore(
f"[EXPLORE] Profile preference unavailable; continuing without profile-default filter: {profile_error}"
)
try:
# Get all tasks for status lookup
all_tasks = task_manager.get_all_tasks()
title_filters = _normalize_filter_values(filter_title)
git_filters = _normalize_filter_values(filter_git_status)
llm_filters = _normalize_filter_values(filter_llm_status)
changed_on_filters = _normalize_filter_values(filter_changed_on)
actor_filters = _normalize_filter_values(filter_actor)
has_column_filters = any(
(
title_filters,
git_filters,
llm_filters,
changed_on_filters,
actor_filters,
)
)
needs_full_scan = has_column_filters or bool(can_apply_profile_filter) or bool(can_apply_slug_filter)
if isinstance(resource_service, ResourceService) and not needs_full_scan:
try:
page_payload = await resource_service.get_dashboards_page_with_status(
env,
all_tasks,
page=page,
page_size=page_size,
search=search,
include_git_status=False,
require_slug=bool(can_apply_slug_filter),
)
paginated_dashboards = page_payload["dashboards"]
total = page_payload["total"]
total_pages = page_payload["total_pages"]
except Exception as page_error:
logger.warning(
"[get_dashboards][Action] Page-based fetch failed; using compatibility fallback: %s",
page_error,
)
dashboards = await resource_service.get_dashboards_with_status(
env,
all_tasks,
include_git_status=False,
require_slug=bool(can_apply_slug_filter),
)
if search:
search_lower = search.lower()
dashboards = [
d for d in dashboards
if search_lower in d.get("title", "").lower()
or search_lower in d.get("slug", "").lower()
]
total = len(dashboards)
total_pages = (total + page_size - 1) // page_size if total > 0 else 1
start_idx = (page - 1) * page_size
end_idx = start_idx + page_size
paginated_dashboards = dashboards[start_idx:end_idx]
else:
dashboards = await resource_service.get_dashboards_with_status(
env,
all_tasks,
include_git_status=bool(git_filters),
require_slug=bool(can_apply_slug_filter),
)
if can_apply_profile_filter and bound_username:
actor_aliases = _resolve_profile_actor_aliases(env, bound_username)
if not actor_aliases:
actor_aliases = [bound_username]
logger.reason(
"[REASON] Applying profile actor filter "
f"(env={env_id}, bound_username={bound_username}, actor_aliases={actor_aliases!r}, "
f"dashboards_before={len(dashboards)})"
)
filtered_dashboards: List[Dict[str, Any]] = []
max_actor_samples = 15
for index, dashboard in enumerate(dashboards):
owners_value = dashboard.get("owners")
created_by_value = dashboard.get("created_by")
modified_by_value = dashboard.get("modified_by")
matches_actor = _matches_dashboard_actor_aliases(
profile_service=profile_service,
actor_aliases=actor_aliases,
owners=owners_value,
modified_by=modified_by_value,
)
if index < max_actor_samples:
logger.reflect(
"[REFLECT] Profile actor filter sample "
f"(env={env_id}, dashboard_id={dashboard.get('id')}, "
f"bound_username={bound_username!r}, actor_aliases={actor_aliases!r}, "
f"owners={owners_value!r}, created_by={created_by_value!r}, "
f"modified_by={modified_by_value!r}, matches={matches_actor})"
)
if matches_actor:
filtered_dashboards.append(dashboard)
logger.reflect(
"[REFLECT] Profile actor filter summary "
f"(env={env_id}, bound_username={bound_username!r}, "
f"dashboards_before={len(dashboards)}, dashboards_after={len(filtered_dashboards)})"
)
dashboards = filtered_dashboards
if can_apply_slug_filter:
dashboards = [
dashboard
for dashboard in dashboards
if str(dashboard.get("slug") or "").strip()
]
if search:
search_lower = search.lower()
dashboards = [
d for d in dashboards
if search_lower in d.get("title", "").lower()
or search_lower in d.get("slug", "").lower()
]
def _matches_dashboard_filters(dashboard: Dict[str, Any]) -> bool:
title_value = str(dashboard.get("title") or "").strip().lower()
if title_filters and title_value not in title_filters:
return False
if git_filters:
git_value = _dashboard_git_filter_value(dashboard)
if git_value not in git_filters:
return False
llm_value = str(
((dashboard.get("last_task") or {}).get("validation_status"))
or "UNKNOWN"
).strip().lower()
if llm_filters and llm_value not in llm_filters:
return False
changed_on_raw = str(dashboard.get("last_modified") or "").strip().lower()
changed_on_prefix = (
changed_on_raw[:10] if len(changed_on_raw) >= 10 else changed_on_raw
)
if (
changed_on_filters
and changed_on_raw not in changed_on_filters
and changed_on_prefix not in changed_on_filters
):
return False
owners = dashboard.get("owners") or []
if isinstance(owners, list):
actor_value = ", ".join(
str(item).strip() for item in owners if str(item).strip()
).lower()
else:
actor_value = str(owners).strip().lower()
if not actor_value:
actor_value = "-"
if actor_filters and actor_value not in actor_filters:
return False
return True
if has_column_filters:
dashboards = [d for d in dashboards if _matches_dashboard_filters(d)]
total = len(dashboards)
total_pages = (total + page_size - 1) // page_size if total > 0 else 1
start_idx = (page - 1) * page_size
end_idx = start_idx + page_size
paginated_dashboards = dashboards[start_idx:end_idx]
logger.info(
f"[get_dashboards][Coherence:OK] Returning {len(paginated_dashboards)} dashboards "
f"(page {page}/{total_pages}, total: {total}, profile_filter_applied={effective_profile_filter.applied})"
)
response_dashboards = _project_dashboard_response_items(paginated_dashboards)
# Fetch dashboards with status using ResourceService
dashboards = await resource_service.get_dashboards_with_status(env, all_tasks)
# Apply search filter if provided
if search:
search_lower = search.lower()
dashboards = [
d for d in dashboards
if search_lower in d.get('title', '').lower()
or search_lower in d.get('slug', '').lower()
]
# Calculate pagination
total = len(dashboards)
total_pages = (total + page_size - 1) // page_size if total > 0 else 1
start_idx = (page - 1) * page_size
end_idx = start_idx + page_size
# Slice dashboards for current page
paginated_dashboards = dashboards[start_idx:end_idx]
logger.info(f"[get_dashboards][Coherence:OK] Returning {len(paginated_dashboards)} dashboards (page {page}/{total_pages}, total: {total})")
return DashboardsResponse(
dashboards=response_dashboards,
dashboards=paginated_dashboards,
total=total,
page=page,
page_size=page_size,
total_pages=total_pages,
effective_profile_filter=effective_profile_filter,
total_pages=total_pages
)
except Exception as e:
logger.error(f"[get_dashboards][Coherence:Failed] Failed to fetch dashboards: {e}")
raise HTTPException(status_code=503, detail=f"Failed to fetch dashboards: {str(e)}")
@@ -793,23 +219,10 @@ async def get_dashboards(
async def get_database_mappings(
source_env_id: str,
target_env_id: str,
config_manager=Depends(get_config_manager),
mapping_service=Depends(get_mapping_service),
_ = Depends(has_permission("plugin:migration", "READ"))
):
with belief_scope("get_database_mappings", f"source={source_env_id}, target={target_env_id}"):
# Validate environments exist
environments = config_manager.get_environments()
source_env = next((e for e in environments if e.id == source_env_id), None)
target_env = next((e for e in environments if e.id == target_env_id), None)
if not source_env:
logger.error(f"[get_database_mappings][Coherence:Failed] Source environment not found: {source_env_id}")
raise HTTPException(status_code=404, detail="Source environment not found")
if not target_env:
logger.error(f"[get_database_mappings][Coherence:Failed] Target environment not found: {target_env_id}")
raise HTTPException(status_code=404, detail="Target environment not found")
try:
# Get mapping suggestions using MappingService
suggestions = await mapping_service.get_suggestions(source_env_id, target_env_id)
@@ -837,29 +250,28 @@ async def get_database_mappings(
# [DEF:get_dashboard_detail:Function]
# @PURPOSE: Fetch detailed dashboard info with related charts and datasets
# @PRE: env_id must be valid and dashboard ref (slug or id) must exist
# @PRE: env_id must be valid and dashboard_id must exist
# @POST: Returns dashboard detail payload for overview page
# @RELATION: CALLS -> SupersetClient.get_dashboard_detail
@router.get("/{dashboard_ref}", response_model=DashboardDetailResponse)
@router.get("/{dashboard_id:int}", response_model=DashboardDetailResponse)
async def get_dashboard_detail(
dashboard_ref: str,
dashboard_id: int,
env_id: str,
config_manager=Depends(get_config_manager),
_ = Depends(has_permission("plugin:migration", "READ"))
):
with belief_scope("get_dashboard_detail", f"dashboard_ref={dashboard_ref}, env_id={env_id}"):
with belief_scope("get_dashboard_detail", f"dashboard_id={dashboard_id}, env_id={env_id}"):
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
logger.error(f"[get_dashboard_detail][Coherence:Failed] Environment not found: {env_id}")
raise HTTPException(status_code=404, detail="Environment not found")
client = AsyncSupersetClient(env)
try:
dashboard_id = await _resolve_dashboard_id_from_ref_async(dashboard_ref, client)
detail = await client.get_dashboard_detail_async(dashboard_id)
client = SupersetClient(env)
detail = client.get_dashboard_detail(dashboard_id)
logger.info(
f"[get_dashboard_detail][Coherence:OK] Dashboard ref={dashboard_ref} resolved_id={dashboard_id}: {detail.get('chart_count', 0)} charts, {detail.get('dataset_count', 0)} datasets"
f"[get_dashboard_detail][Coherence:OK] Dashboard {dashboard_id}: {detail.get('chart_count', 0)} charts, {detail.get('dataset_count', 0)} datasets"
)
return DashboardDetailResponse(**detail)
except HTTPException:
@@ -867,8 +279,6 @@ async def get_dashboard_detail(
except Exception as e:
logger.error(f"[get_dashboard_detail][Coherence:Failed] Failed to fetch dashboard detail: {e}")
raise HTTPException(status_code=503, detail=f"Failed to fetch dashboard detail: {str(e)}")
finally:
await client.aclose()
# [/DEF:get_dashboard_detail:Function]
@@ -907,87 +317,61 @@ def _task_matches_dashboard(task: Any, dashboard_id: int, env_id: Optional[str])
# [DEF:get_dashboard_tasks_history:Function]
# @PURPOSE: Returns history of backup and LLM validation tasks for a dashboard.
# @PRE: dashboard ref (slug or id) is valid.
# @PRE: dashboard_id is valid integer.
# @POST: Response contains sorted task history (newest first).
@router.get("/{dashboard_ref}/tasks", response_model=DashboardTaskHistoryResponse)
@router.get("/{dashboard_id:int}/tasks", response_model=DashboardTaskHistoryResponse)
async def get_dashboard_tasks_history(
dashboard_ref: str,
dashboard_id: int,
env_id: Optional[str] = None,
limit: int = Query(20, ge=1, le=100),
config_manager=Depends(get_config_manager),
task_manager=Depends(get_task_manager),
_ = Depends(has_permission("tasks", "READ"))
):
with belief_scope("get_dashboard_tasks_history", f"dashboard_ref={dashboard_ref}, env_id={env_id}, limit={limit}"):
dashboard_id: Optional[int] = None
client: Optional[AsyncSupersetClient] = None
try:
if dashboard_ref.isdigit():
dashboard_id = int(dashboard_ref)
elif env_id:
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
logger.error(f"[get_dashboard_tasks_history][Coherence:Failed] Environment not found: {env_id}")
raise HTTPException(status_code=404, detail="Environment not found")
client = AsyncSupersetClient(env)
dashboard_id = await _resolve_dashboard_id_from_ref_async(dashboard_ref, client)
else:
logger.error(
"[get_dashboard_tasks_history][Coherence:Failed] Non-numeric dashboard ref requires env_id"
with belief_scope("get_dashboard_tasks_history", f"dashboard_id={dashboard_id}, env_id={env_id}, limit={limit}"):
matching_tasks = []
for task in task_manager.get_all_tasks():
if _task_matches_dashboard(task, dashboard_id, env_id):
matching_tasks.append(task)
def _sort_key(task_obj: Any) -> str:
return (
str(getattr(task_obj, "started_at", "") or "")
or str(getattr(task_obj, "finished_at", "") or "")
)
matching_tasks.sort(key=_sort_key, reverse=True)
selected = matching_tasks[:limit]
items = []
for task in selected:
result = getattr(task, "result", None)
summary = None
validation_status = None
if isinstance(result, dict):
raw_validation_status = result.get("status")
if raw_validation_status is not None:
validation_status = str(raw_validation_status)
summary = (
result.get("summary")
or result.get("status")
or result.get("message")
)
raise HTTPException(
status_code=400,
detail="env_id is required when dashboard reference is a slug",
params = getattr(task, "params", {}) or {}
items.append(
DashboardTaskHistoryItem(
id=str(getattr(task, "id", "")),
plugin_id=str(getattr(task, "plugin_id", "")),
status=str(getattr(task, "status", "")),
validation_status=validation_status,
started_at=getattr(task, "started_at", None).isoformat() if getattr(task, "started_at", None) else None,
finished_at=getattr(task, "finished_at", None).isoformat() if getattr(task, "finished_at", None) else None,
env_id=str(params.get("environment_id") or params.get("env")) if (params.get("environment_id") or params.get("env")) else None,
summary=summary,
)
)
matching_tasks = []
for task in task_manager.get_all_tasks():
if _task_matches_dashboard(task, dashboard_id, env_id):
matching_tasks.append(task)
def _sort_key(task_obj: Any) -> str:
return (
str(getattr(task_obj, "started_at", "") or "")
or str(getattr(task_obj, "finished_at", "") or "")
)
matching_tasks.sort(key=_sort_key, reverse=True)
selected = matching_tasks[:limit]
items = []
for task in selected:
result = getattr(task, "result", None)
summary = None
validation_status = None
if isinstance(result, dict):
raw_validation_status = result.get("status")
if raw_validation_status is not None:
validation_status = str(raw_validation_status)
summary = (
result.get("summary")
or result.get("status")
or result.get("message")
)
params = getattr(task, "params", {}) or {}
items.append(
DashboardTaskHistoryItem(
id=str(getattr(task, "id", "")),
plugin_id=str(getattr(task, "plugin_id", "")),
status=str(getattr(task, "status", "")),
validation_status=validation_status,
started_at=getattr(task, "started_at", None).isoformat() if getattr(task, "started_at", None) else None,
finished_at=getattr(task, "finished_at", None).isoformat() if getattr(task, "finished_at", None) else None,
env_id=str(params.get("environment_id") or params.get("env")) if (params.get("environment_id") or params.get("env")) else None,
summary=summary,
)
)
logger.info(f"[get_dashboard_tasks_history][Coherence:OK] Found {len(items)} tasks for dashboard_ref={dashboard_ref}, dashboard_id={dashboard_id}")
return DashboardTaskHistoryResponse(dashboard_id=dashboard_id, items=items)
finally:
if client is not None:
await client.aclose()
logger.info(f"[get_dashboard_tasks_history][Coherence:OK] Found {len(items)} tasks for dashboard {dashboard_id}")
return DashboardTaskHistoryResponse(dashboard_id=dashboard_id, items=items)
# [/DEF:get_dashboard_tasks_history:Function]
@@ -995,30 +379,29 @@ async def get_dashboard_tasks_history(
# @PURPOSE: Proxies Superset dashboard thumbnail with cache support.
# @PRE: env_id must exist.
# @POST: Returns image bytes or 202 when thumbnail is being prepared by Superset.
@router.get("/{dashboard_ref}/thumbnail")
@router.get("/{dashboard_id:int}/thumbnail")
async def get_dashboard_thumbnail(
dashboard_ref: str,
dashboard_id: int,
env_id: str,
force: bool = Query(False),
config_manager=Depends(get_config_manager),
_ = Depends(has_permission("plugin:migration", "READ"))
):
with belief_scope("get_dashboard_thumbnail", f"dashboard_ref={dashboard_ref}, env_id={env_id}, force={force}"):
with belief_scope("get_dashboard_thumbnail", f"dashboard_id={dashboard_id}, env_id={env_id}, force={force}"):
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
logger.error(f"[get_dashboard_thumbnail][Coherence:Failed] Environment not found: {env_id}")
raise HTTPException(status_code=404, detail="Environment not found")
client = AsyncSupersetClient(env)
try:
dashboard_id = await _resolve_dashboard_id_from_ref_async(dashboard_ref, client)
client = SupersetClient(env)
digest = None
thumb_endpoint = None
# Preferred flow (newer Superset): ask server to cache screenshot and return digest/image_url.
try:
screenshot_payload = await client.network.request(
screenshot_payload = client.network.request(
method="POST",
endpoint=f"/dashboard/{dashboard_id}/cache_dashboard_screenshot/",
json={"force": force},
@@ -1036,7 +419,7 @@ async def get_dashboard_thumbnail(
# Fallback flow (older Superset): read thumbnail_url from dashboard payload.
if not digest:
dashboard_payload = await client.network.request(
dashboard_payload = client.network.request(
method="GET",
endpoint=f"/dashboard/{dashboard_id}",
)
@@ -1055,7 +438,7 @@ async def get_dashboard_thumbnail(
if not thumb_endpoint:
thumb_endpoint = f"/dashboard/{dashboard_id}/thumbnail/{digest or 'latest'}/"
thumb_response = await client.network.request(
thumb_response = client.network.request(
method="GET",
endpoint=thumb_endpoint,
raw_response=True,
@@ -1080,8 +463,6 @@ async def get_dashboard_thumbnail(
except Exception as e:
logger.error(f"[get_dashboard_thumbnail][Coherence:Failed] Failed to fetch dashboard thumbnail: {e}")
raise HTTPException(status_code=503, detail=f"Failed to fetch dashboard thumbnail: {str(e)}")
finally:
await client.aclose()
# [/DEF:get_dashboard_thumbnail:Function]
# [DEF:MigrateRequest:DataClass]

View File

@@ -20,18 +20,6 @@ from ...core.logger import belief_scope
router = APIRouter(prefix="/api/environments", tags=["Environments"])
# [DEF:_normalize_superset_env_url:Function]
# @PURPOSE: Canonicalize Superset environment URL to base host/path without trailing /api/v1.
# @PRE: raw_url can be empty.
# @POST: Returns normalized base URL.
def _normalize_superset_env_url(raw_url: str) -> str:
normalized = str(raw_url or "").strip().rstrip("/")
if normalized.lower().endswith("/api/v1"):
normalized = normalized[:-len("/api/v1")]
return normalized.rstrip("/")
# [/DEF:_normalize_superset_env_url:Function]
# [DEF:ScheduleSchema:DataClass]
class ScheduleSchema(BaseModel):
enabled: bool = False
@@ -43,7 +31,6 @@ class EnvironmentResponse(BaseModel):
id: str
name: str
url: str
stage: str = "DEV"
is_production: bool = False
backup_schedule: Optional[ScheduleSchema] = None
# [/DEF:EnvironmentResponse:DataClass]
@@ -72,26 +59,18 @@ async def get_environments(
# Ensure envs is a list
if not isinstance(envs, list):
envs = []
response_items = []
for e in envs:
resolved_stage = str(
getattr(e, "stage", "")
or ("PROD" if bool(getattr(e, "is_production", False)) else "DEV")
).upper()
response_items.append(
EnvironmentResponse(
id=e.id,
name=e.name,
url=_normalize_superset_env_url(e.url),
stage=resolved_stage,
is_production=(resolved_stage == "PROD"),
backup_schedule=ScheduleSchema(
enabled=e.backup_schedule.enabled,
cron_expression=e.backup_schedule.cron_expression
) if getattr(e, 'backup_schedule', None) else None
)
)
return response_items
return [
EnvironmentResponse(
id=e.id,
name=e.name,
url=e.url,
is_production=getattr(e, "is_production", False),
backup_schedule=ScheduleSchema(
enabled=e.backup_schedule.enabled,
cron_expression=e.backup_schedule.cron_expression
) if getattr(e, 'backup_schedule', None) else None
) for e in envs
]
# [/DEF:get_environments:Function]
# [DEF:update_environment_schedule:Function]

File diff suppressed because it is too large Load Diff

View File

@@ -9,7 +9,7 @@
# @INVARIANT: All schemas must be compatible with the FastAPI router.
from pydantic import BaseModel, Field
from typing import Any, Dict, List, Optional
from typing import List, Optional
from datetime import datetime
from src.models.git import GitProvider, GitStatus, SyncStatus
@@ -21,27 +21,14 @@ class GitServerConfigBase(BaseModel):
provider: GitProvider = Field(..., description="Git provider (GITHUB, GITLAB, GITEA)")
url: str = Field(..., description="Server base URL")
pat: str = Field(..., description="Personal Access Token")
pat: str = Field(..., description="Personal Access Token")
default_repository: Optional[str] = Field(None, description="Default repository path (org/repo)")
default_branch: Optional[str] = Field("main", description="Default branch logic/name")
# [/DEF:GitServerConfigBase:Class]
# [DEF:GitServerConfigUpdate:Class]
# @PURPOSE: Schema for updating an existing Git server configuration.
class GitServerConfigUpdate(BaseModel):
name: Optional[str] = Field(None, description="Display name for the Git server")
provider: Optional[GitProvider] = Field(None, description="Git provider (GITHUB, GITLAB, GITEA)")
url: Optional[str] = Field(None, description="Server base URL")
pat: Optional[str] = Field(None, description="Personal Access Token")
default_repository: Optional[str] = Field(None, description="Default repository path (org/repo)")
default_branch: Optional[str] = Field(None, description="Default branch logic/name")
# [/DEF:GitServerConfigUpdate:Class]
# [DEF:GitServerConfigCreate:Class]
# @PURPOSE: Schema for creating a new Git server configuration.
class GitServerConfigCreate(GitServerConfigBase):
"""Schema for creating a new Git server configuration."""
config_id: Optional[str] = Field(None, description="Optional config ID, useful for testing an existing config without sending its full PAT")
pass
# [/DEF:GitServerConfigCreate:Class]
# [DEF:GitServerConfigSchema:Class]
@@ -126,42 +113,6 @@ class ConflictResolution(BaseModel):
content: Optional[str] = None
# [/DEF:ConflictResolution:Class]
# [DEF:MergeStatusSchema:Class]
# @PURPOSE: Schema representing unfinished merge status for repository.
class MergeStatusSchema(BaseModel):
has_unfinished_merge: bool
repository_path: str
git_dir: str
current_branch: str
merge_head: Optional[str] = None
merge_message_preview: Optional[str] = None
conflicts_count: int = 0
# [/DEF:MergeStatusSchema:Class]
# [DEF:MergeConflictFileSchema:Class]
# @PURPOSE: Schema describing one conflicted file with optional side snapshots.
class MergeConflictFileSchema(BaseModel):
file_path: str
mine: Optional[str] = None
theirs: Optional[str] = None
# [/DEF:MergeConflictFileSchema:Class]
# [DEF:MergeResolveRequest:Class]
# @PURPOSE: Request schema for resolving one or multiple merge conflicts.
class MergeResolveRequest(BaseModel):
resolutions: List[ConflictResolution] = Field(default_factory=list)
# [/DEF:MergeResolveRequest:Class]
# [DEF:MergeContinueRequest:Class]
# @PURPOSE: Request schema for finishing merge with optional explicit commit message.
class MergeContinueRequest(BaseModel):
message: Optional[str] = None
# [/DEF:MergeContinueRequest:Class]
# [DEF:DeploymentEnvironmentSchema:Class]
# @PURPOSE: Schema for representing a target deployment environment.
class DeploymentEnvironmentSchema(BaseModel):
@@ -190,104 +141,4 @@ class RepoInitRequest(BaseModel):
remote_url: str
# [/DEF:RepoInitRequest:Class]
# [DEF:RepositoryBindingSchema:Class]
# @PURPOSE: Schema describing repository-to-config binding and provider metadata.
class RepositoryBindingSchema(BaseModel):
dashboard_id: int
config_id: str
provider: GitProvider
remote_url: str
local_path: str
# [/DEF:RepositoryBindingSchema:Class]
# [DEF:RepoStatusBatchRequest:Class]
# @PURPOSE: Schema for requesting repository statuses for multiple dashboards in a single call.
class RepoStatusBatchRequest(BaseModel):
dashboard_ids: List[int] = Field(default_factory=list, description="Dashboard IDs to resolve repository statuses for")
# [/DEF:RepoStatusBatchRequest:Class]
# [DEF:RepoStatusBatchResponse:Class]
# @PURPOSE: Schema for returning repository statuses keyed by dashboard ID.
class RepoStatusBatchResponse(BaseModel):
statuses: Dict[str, Dict[str, Any]]
# [/DEF:RepoStatusBatchResponse:Class]
# [DEF:GiteaRepoSchema:Class]
# @PURPOSE: Schema describing a Gitea repository.
class GiteaRepoSchema(BaseModel):
name: str
full_name: str
private: bool = False
clone_url: Optional[str] = None
html_url: Optional[str] = None
ssh_url: Optional[str] = None
default_branch: Optional[str] = None
# [/DEF:GiteaRepoSchema:Class]
# [DEF:GiteaRepoCreateRequest:Class]
# @PURPOSE: Request schema for creating a Gitea repository.
class GiteaRepoCreateRequest(BaseModel):
name: str = Field(..., min_length=1, max_length=255)
private: bool = True
description: Optional[str] = None
auto_init: bool = True
default_branch: Optional[str] = "main"
# [/DEF:GiteaRepoCreateRequest:Class]
# [DEF:RemoteRepoSchema:Class]
# @PURPOSE: Provider-agnostic remote repository payload.
class RemoteRepoSchema(BaseModel):
provider: GitProvider
name: str
full_name: str
private: bool = False
clone_url: Optional[str] = None
html_url: Optional[str] = None
ssh_url: Optional[str] = None
default_branch: Optional[str] = None
# [/DEF:RemoteRepoSchema:Class]
# [DEF:RemoteRepoCreateRequest:Class]
# @PURPOSE: Provider-agnostic repository creation request.
class RemoteRepoCreateRequest(BaseModel):
name: str = Field(..., min_length=1, max_length=255)
private: bool = True
description: Optional[str] = None
auto_init: bool = True
default_branch: Optional[str] = "main"
# [/DEF:RemoteRepoCreateRequest:Class]
# [DEF:PromoteRequest:Class]
# @PURPOSE: Request schema for branch promotion workflow.
class PromoteRequest(BaseModel):
from_branch: str = Field(..., min_length=1, max_length=255)
to_branch: str = Field(..., min_length=1, max_length=255)
mode: str = Field(default="mr", pattern="^(mr|direct)$")
title: Optional[str] = None
description: Optional[str] = None
reason: Optional[str] = None
draft: bool = False
remove_source_branch: bool = False
# [/DEF:PromoteRequest:Class]
# [DEF:PromoteResponse:Class]
# @PURPOSE: Response schema for promotion operation result.
class PromoteResponse(BaseModel):
mode: str
from_branch: str
to_branch: str
status: str
url: Optional[str] = None
reference_id: Optional[str] = None
policy_violation: bool = False
# [/DEF:PromoteResponse:Class]
# [/DEF:backend.src.api.routes.git_schemas:Module]
# [/DEF:backend.src.api.routes.git_schemas:Module]

View File

@@ -1,31 +0,0 @@
# [DEF:health_router:Module]
# @TIER: STANDARD
# @SEMANTICS: health, monitoring, dashboards
# @PURPOSE: API endpoints for dashboard health monitoring and status aggregation.
# @LAYER: UI/API
# @RELATION: DEPENDS_ON -> health_service
from fastapi import APIRouter, Depends, Query
from typing import List, Optional
from sqlalchemy.orm import Session
from ...core.database import get_db
from ...services.health_service import HealthService
from ...schemas.health import HealthSummaryResponse
from ...dependencies import has_permission
router = APIRouter(prefix="/api/health", tags=["Health"])
@router.get("/summary", response_model=HealthSummaryResponse)
async def get_health_summary(
environment_id: Optional[str] = Query(None),
db: Session = Depends(get_db),
_ = Depends(has_permission("plugin:migration", "READ"))
):
"""
@PURPOSE: Get aggregated health status for all dashboards.
@POST: Returns HealthSummaryResponse
"""
service = HealthService(db)
return await service.get_health_summary(environment_id=environment_id)
# [/DEF:health_router:Module]

View File

@@ -1,23 +1,10 @@
# [DEF:backend.src.api.routes.migration:Module]
# @TIER: CRITICAL
# @SEMANTICS: api, migration, dashboards, sync, dry-run
# @PURPOSE: HTTP contract layer for migration orchestration, settings, dry-run, and mapping sync endpoints.
# @LAYER: Infra
# @RELATION: [DEPENDS_ON] ->[backend.src.dependencies]
# @RELATION: [DEPENDS_ON] ->[backend.src.core.database]
# @RELATION: [DEPENDS_ON] ->[backend.src.core.superset_client]
# @RELATION: [DEPENDS_ON] ->[backend.src.core.migration.dry_run_orchestrator]
# @RELATION: [DEPENDS_ON] ->[backend.src.core.mapping_service]
# @RELATION: [DEPENDS_ON] ->[backend.src.models.dashboard]
# @RELATION: [DEPENDS_ON] ->[backend.src.models.mapping]
# @INVARIANT: Migration endpoints never execute with invalid environment references and always return explicit HTTP errors on guard failures.
# @TEST_CONTRACT: [DashboardSelection + configured envs] -> [task_id | dry-run result | sync summary]
# @TEST_SCENARIO: [invalid_environment] -> [HTTP_400_or_404]
# @TEST_SCENARIO: [valid_execution] -> [success_payload_with_required_fields]
# @TEST_EDGE: [missing_field] ->[HTTP_400]
# @TEST_EDGE: [invalid_type] ->[validation_error]
# @TEST_EDGE: [external_fail] ->[HTTP_500]
# @TEST_INVARIANT: [EnvironmentValidationBeforeAction] -> VERIFIED_BY: [invalid_environment, valid_execution]
# @TIER: STANDARD
# @SEMANTICS: api, migration, dashboards
# @PURPOSE: API endpoints for migration operations.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.dependencies
# @RELATION: DEPENDS_ON -> backend.src.models.dashboard
from fastapi import APIRouter, Depends, HTTPException, Query
from typing import List, Dict, Any, Optional
@@ -26,19 +13,18 @@ from ...dependencies import get_config_manager, get_task_manager, has_permission
from ...core.database import get_db
from ...models.dashboard import DashboardMetadata, DashboardSelection
from ...core.superset_client import SupersetClient
from ...core.logger import logger, belief_scope
from ...core.migration.dry_run_orchestrator import MigrationDryRunService
from ...core.logger import belief_scope
from ...core.mapping_service import IdMappingService
from ...models.mapping import ResourceMapping
router = APIRouter(prefix="/api", tags=["migration"])
# [DEF:get_dashboards:Function]
# @PURPOSE: Fetch dashboard metadata from a requested environment for migration selection UI.
# @PRE: env_id is provided and exists in configured environments.
# @POST: Returns List[DashboardMetadata] for the resolved environment; emits HTTP_404 when environment is absent.
# @SIDE_EFFECT: Reads environment configuration and performs remote Superset metadata retrieval over network.
# @DATA_CONTRACT: Input[str env_id] -> Output[List[DashboardMetadata]]
# @PURPOSE: Fetch all dashboards from the specified environment for the grid.
# @PRE: Environment ID must be valid.
# @POST: Returns a list of dashboard metadata.
# @PARAM: env_id (str) - The ID of the environment to fetch from.
# @RETURN: List[DashboardMetadata]
@router.get("/environments/{env_id}/dashboards", response_model=List[DashboardMetadata])
async def get_dashboards(
env_id: str,
@@ -46,26 +32,22 @@ async def get_dashboards(
_ = Depends(has_permission("plugin:migration", "EXECUTE"))
):
with belief_scope("get_dashboards", f"env_id={env_id}"):
logger.reason(f"Fetching dashboards for environment: {env_id}")
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
logger.explore(f"Environment {env_id} not found in configuration")
raise HTTPException(status_code=404, detail="Environment not found")
env = next((e for e in environments if e.id == env_id), None)
if not env:
raise HTTPException(status_code=404, detail="Environment not found")
client = SupersetClient(env)
dashboards = client.get_dashboards_summary()
logger.reflect(f"Retrieved {len(dashboards)} dashboards from {env_id}")
return dashboards
client = SupersetClient(env)
dashboards = client.get_dashboards_summary()
return dashboards
# [/DEF:get_dashboards:Function]
# [DEF:execute_migration:Function]
# @PURPOSE: Validate migration selection and enqueue asynchronous migration task execution.
# @PRE: DashboardSelection payload is valid and both source/target environments exist.
# @POST: Returns {"task_id": str, "message": str} when task creation succeeds; emits HTTP_400/HTTP_500 on failure.
# @SIDE_EFFECT: Reads configuration, writes task record through task manager, and writes operational logs.
# @DATA_CONTRACT: Input[DashboardSelection] -> Output[Dict[str, str]]
# @PURPOSE: Execute the migration of selected dashboards.
# @PRE: Selection must be valid and environments must exist.
# @POST: Starts the migration task and returns the task ID.
# @PARAM: selection (DashboardSelection) - The dashboards to migrate.
# @RETURN: Dict - {"task_id": str, "message": str}
@router.post("/migration/execute")
async def execute_migration(
selection: DashboardSelection,
@@ -74,90 +56,35 @@ async def execute_migration(
_ = Depends(has_permission("plugin:migration", "EXECUTE"))
):
with belief_scope("execute_migration"):
logger.reason(f"Initiating migration from {selection.source_env_id} to {selection.target_env_id}")
# Validate environments exist
environments = config_manager.get_environments()
env_ids = {e.id for e in environments}
if selection.source_env_id not in env_ids or selection.target_env_id not in env_ids:
logger.explore("Invalid environment selection", extra={"source": selection.source_env_id, "target": selection.target_env_id})
raise HTTPException(status_code=400, detail="Invalid source or target environment")
env_ids = {e.id for e in environments}
if selection.source_env_id not in env_ids or selection.target_env_id not in env_ids:
raise HTTPException(status_code=400, detail="Invalid source or target environment")
# Include replace_db_config and fix_cross_filters in the task parameters
task_params = selection.dict()
task_params['replace_db_config'] = selection.replace_db_config
task_params['fix_cross_filters'] = selection.fix_cross_filters
logger.reason(f"Creating migration task with {len(selection.selected_ids)} dashboards")
try:
task = await task_manager.create_task("superset-migration", task_params)
logger.reflect(f"Migration task created: {task.id}")
return {"task_id": task.id, "message": "Migration initiated"}
except Exception as e:
logger.explore(f"Task creation failed: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create migration task: {str(e)}")
# Create migration task with debug logging
from ...core.logger import logger
# Include replace_db_config and fix_cross_filters in the task parameters
task_params = selection.dict()
task_params['replace_db_config'] = selection.replace_db_config
task_params['fix_cross_filters'] = selection.fix_cross_filters
logger.info(f"Creating migration task with params: {task_params}")
logger.info(f"Available environments: {env_ids}")
logger.info(f"Source env: {selection.source_env_id}, Target env: {selection.target_env_id}")
try:
task = await task_manager.create_task("superset-migration", task_params)
logger.info(f"Task created successfully: {task.id}")
return {"task_id": task.id, "message": "Migration initiated"}
except Exception as e:
logger.error(f"Task creation failed: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create migration task: {str(e)}")
# [/DEF:execute_migration:Function]
# [DEF:dry_run_migration:Function]
# @PURPOSE: Build pre-flight migration diff and risk summary without mutating target systems.
# @PRE: DashboardSelection is valid, source and target environments exist, differ, and selected_ids is non-empty.
# @POST: Returns deterministic dry-run payload; emits HTTP_400 for guard violations and HTTP_500 for orchestrator value errors.
# @SIDE_EFFECT: Reads local mappings from DB and fetches source/target metadata via Superset API.
# @DATA_CONTRACT: Input[DashboardSelection] -> Output[Dict[str, Any]]
@router.post("/migration/dry-run", response_model=Dict[str, Any])
async def dry_run_migration(
selection: DashboardSelection,
config_manager=Depends(get_config_manager),
db: Session = Depends(get_db),
_ = Depends(has_permission("plugin:migration", "EXECUTE"))
):
with belief_scope("dry_run_migration"):
logger.reason(f"Starting dry run: {selection.source_env_id} -> {selection.target_env_id}")
environments = config_manager.get_environments()
env_map = {env.id: env for env in environments}
source_env = env_map.get(selection.source_env_id)
target_env = env_map.get(selection.target_env_id)
if not source_env or not target_env:
logger.explore("Invalid environment selection for dry run")
raise HTTPException(status_code=400, detail="Invalid source or target environment")
if selection.source_env_id == selection.target_env_id:
logger.explore("Source and target environments are identical")
raise HTTPException(status_code=400, detail="Source and target environments must be different")
if not selection.selected_ids:
logger.explore("No dashboards selected for dry run")
raise HTTPException(status_code=400, detail="No dashboards selected for dry run")
service = MigrationDryRunService()
source_client = SupersetClient(source_env)
target_client = SupersetClient(target_env)
try:
result = service.run(
selection=selection,
source_client=source_client,
target_client=target_client,
db=db,
)
logger.reflect("Dry run analysis complete")
return result
except ValueError as exc:
logger.explore(f"Dry run orchestrator failed: {exc}")
raise HTTPException(status_code=500, detail=str(exc)) from exc
# [/DEF:dry_run_migration:Function]
# [DEF:get_migration_settings:Function]
# @PURPOSE: Read and return configured migration synchronization cron expression.
# @PRE: Configuration store is available and requester has READ permission.
# @POST: Returns {"cron": str} reflecting current persisted settings value.
# @SIDE_EFFECT: Reads configuration from config manager.
# @DATA_CONTRACT: Input[None] -> Output[Dict[str, str]]
# @PURPOSE: Get current migration Cron string explicitly.
@router.get("/migration/settings", response_model=Dict[str, str])
async def get_migration_settings(
config_manager=Depends(get_config_manager),
@@ -170,11 +97,7 @@ async def get_migration_settings(
# [/DEF:get_migration_settings:Function]
# [DEF:update_migration_settings:Function]
# @PURPOSE: Validate and persist migration synchronization cron expression update.
# @PRE: Payload includes "cron" key and requester has WRITE permission.
# @POST: Returns {"cron": str, "status": "updated"} and persists updated cron value.
# @SIDE_EFFECT: Mutates configuration and writes persisted config through config manager.
# @DATA_CONTRACT: Input[Dict[str, str]] -> Output[Dict[str, str]]
# @PURPOSE: Update migration Cron string.
@router.put("/migration/settings", response_model=Dict[str, str])
async def update_migration_settings(
payload: Dict[str, str],
@@ -195,11 +118,7 @@ async def update_migration_settings(
# [/DEF:update_migration_settings:Function]
# [DEF:get_resource_mappings:Function]
# @PURPOSE: Fetch synchronized resource mappings with optional filters and pagination for migration mappings view.
# @PRE: skip>=0, 1<=limit<=500, DB session is active, requester has READ permission.
# @POST: Returns {"items": [...], "total": int} where items reflect applied filters and pagination.
# @SIDE_EFFECT: Executes database read queries against ResourceMapping table.
# @DATA_CONTRACT: Input[QueryParams] -> Output[Dict[str, Any]]
# @PURPOSE: Fetch synchronized object mappings with search, filtering, and pagination.
@router.get("/migration/mappings-data", response_model=Dict[str, Any])
async def get_resource_mappings(
skip: int = Query(0, ge=0),
@@ -245,11 +164,9 @@ async def get_resource_mappings(
# [/DEF:get_resource_mappings:Function]
# [DEF:trigger_sync_now:Function]
# @PURPOSE: Trigger immediate ID synchronization for every configured environment.
# @PRE: At least one environment is configured and requester has EXECUTE permission.
# @POST: Returns sync summary with synced/failed counts after attempting all environments.
# @SIDE_EFFECT: Upserts Environment rows, commits DB transaction, performs network sync calls, and writes logs.
# @DATA_CONTRACT: Input[None] -> Output[Dict[str, Any]]
# @PURPOSE: Triggers an immediate ID synchronization for all environments.
# @PRE: At least one environment must be configured.
# @POST: Environment rows are ensured in DB; sync_environment is called for each.
@router.post("/migration/sync-now", response_model=Dict[str, Any])
async def trigger_sync_now(
config_manager=Depends(get_config_manager),
@@ -304,4 +221,4 @@ async def trigger_sync_now(
}
# [/DEF:trigger_sync_now:Function]
# [/DEF:backend.src.api.routes.migration:Module]
# [/DEF:backend.src.api.routes.migration:Module]

View File

@@ -1,147 +0,0 @@
# [DEF:backend.src.api.routes.profile:Module]
#
# @TIER: CRITICAL
# @SEMANTICS: api, profile, preferences, self-service, account-lookup
# @PURPOSE: Exposes self-scoped profile preference endpoints and environment-based Superset account lookup.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.services.profile_service
# @RELATION: DEPENDS_ON -> backend.src.dependencies.get_current_user
# @RELATION: DEPENDS_ON -> backend.src.core.database.get_db
#
# @INVARIANT: Endpoints are self-scoped and never mutate another user preference.
# @UX_STATE: ProfileLoad -> Returns stable ProfilePreferenceResponse for authenticated user.
# @UX_STATE: Saving -> Validation errors map to actionable 422 details.
# @UX_STATE: LookupLoading -> Returns success/degraded Superset lookup payload.
# @UX_FEEDBACK: Stable status/message/warning payloads support profile page feedback.
# @UX_RECOVERY: Lookup degradation keeps manual username save path available.
# [SECTION: IMPORTS]
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.orm import Session
from ...core.database import get_db
from ...core.logger import logger, belief_scope
from ...dependencies import (
get_config_manager,
get_current_user,
get_plugin_loader,
)
from ...models.auth import User
from ...schemas.profile import (
ProfilePreferenceResponse,
ProfilePreferenceUpdateRequest,
SupersetAccountLookupRequest,
SupersetAccountLookupResponse,
)
from ...services.profile_service import (
EnvironmentNotFoundError,
ProfileAuthorizationError,
ProfileService,
ProfileValidationError,
)
# [/SECTION]
router = APIRouter(prefix="/api/profile", tags=["profile"])
# [DEF:_get_profile_service:Function]
# @PURPOSE: Build profile service for current request scope.
# @PRE: db session and config manager are available.
# @POST: Returns a ready ProfileService instance.
def _get_profile_service(db: Session, config_manager, plugin_loader=None) -> ProfileService:
return ProfileService(
db=db,
config_manager=config_manager,
plugin_loader=plugin_loader,
)
# [/DEF:_get_profile_service:Function]
# [DEF:get_preferences:Function]
# @PURPOSE: Get authenticated user's dashboard filter preference.
# @PRE: Valid JWT and authenticated user context.
# @POST: Returns preference payload for current user only.
@router.get("/preferences", response_model=ProfilePreferenceResponse)
async def get_preferences(
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
config_manager=Depends(get_config_manager),
plugin_loader=Depends(get_plugin_loader),
):
with belief_scope("profile.get_preferences", f"user_id={current_user.id}"):
logger.reason("[REASON] Resolving current user preference")
service = _get_profile_service(db, config_manager, plugin_loader)
return service.get_my_preference(current_user)
# [/DEF:get_preferences:Function]
# [DEF:update_preferences:Function]
# @PURPOSE: Update authenticated user's dashboard filter preference.
# @PRE: Valid JWT and valid request payload.
# @POST: Persists normalized preference for current user or raises validation/authorization errors.
@router.patch("/preferences", response_model=ProfilePreferenceResponse)
async def update_preferences(
payload: ProfilePreferenceUpdateRequest,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
config_manager=Depends(get_config_manager),
plugin_loader=Depends(get_plugin_loader),
):
with belief_scope("profile.update_preferences", f"user_id={current_user.id}"):
service = _get_profile_service(db, config_manager, plugin_loader)
try:
logger.reason("[REASON] Attempting preference save")
return service.update_my_preference(current_user=current_user, payload=payload)
except ProfileValidationError as exc:
logger.reflect("[REFLECT] Preference validation failed")
raise HTTPException(status_code=422, detail=exc.errors) from exc
except ProfileAuthorizationError as exc:
logger.explore("[EXPLORE] Cross-user mutation guard blocked request")
raise HTTPException(status_code=403, detail=str(exc)) from exc
# [/DEF:update_preferences:Function]
# [DEF:lookup_superset_accounts:Function]
# @PURPOSE: Lookup Superset account candidates in selected environment.
# @PRE: Valid JWT, authenticated context, and environment_id query parameter.
# @POST: Returns success or degraded lookup payload with stable shape.
@router.get("/superset-accounts", response_model=SupersetAccountLookupResponse)
async def lookup_superset_accounts(
environment_id: str = Query(...),
search: Optional[str] = Query(default=None),
page_index: int = Query(default=0, ge=0),
page_size: int = Query(default=20, ge=1, le=100),
sort_column: str = Query(default="username"),
sort_order: str = Query(default="desc"),
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
config_manager=Depends(get_config_manager),
plugin_loader=Depends(get_plugin_loader),
):
with belief_scope(
"profile.lookup_superset_accounts",
f"user_id={current_user.id}, environment_id={environment_id}",
):
service = _get_profile_service(db, config_manager, plugin_loader)
lookup_request = SupersetAccountLookupRequest(
environment_id=environment_id,
search=search,
page_index=page_index,
page_size=page_size,
sort_column=sort_column,
sort_order=sort_order,
)
try:
logger.reason("[REASON] Executing Superset account lookup")
return service.lookup_superset_accounts(
current_user=current_user,
request=lookup_request,
)
except EnvironmentNotFoundError as exc:
logger.explore("[EXPLORE] Lookup request references unknown environment")
raise HTTPException(status_code=404, detail=str(exc)) from exc
# [/DEF:lookup_superset_accounts:Function]
# [/DEF:backend.src.api.routes.profile:Module]

View File

@@ -13,11 +13,10 @@ from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, status
from ...dependencies import get_task_manager, has_permission, get_clean_release_repository
from ...dependencies import get_task_manager, has_permission
from ...core.task_manager import TaskManager
from ...core.logger import belief_scope
from ...models.report import ReportCollection, ReportDetailView, ReportQuery, ReportStatus, TaskType
from ...services.clean_release.repository import CleanReleaseRepository
from ...services.reports.report_service import ReportsService
# [/SECTION]
@@ -63,20 +62,6 @@ def _parse_csv_enum_list(raw: Optional[str], enum_cls, field_name: str) -> List:
# @PRE: authenticated/authorized request and validated query params.
# @POST: returns {items,total,page,page_size,has_next,applied_filters}.
# @POST: deterministic error payload for invalid filters.
#
# @TEST_CONTRACT: ListReportsApi ->
# {
# required_fields: {page: int, page_size: int, sort_by: str, sort_order: str},
# optional_fields: {task_types: str, statuses: str, search: str},
# invariants: [
# "Returns ReportCollection on success",
# "Raises HTTPException 400 for invalid query parameters"
# ]
# }
# @TEST_FIXTURE: valid_list_request -> {"page": 1, "page_size": 20}
# @TEST_EDGE: invalid_task_type_filter -> raises HTTPException(400)
# @TEST_EDGE: malformed_query -> raises HTTPException(400)
# @TEST_INVARIANT: consistent_list_payload -> verifies: [valid_list_request]
@router.get("", response_model=ReportCollection)
async def list_reports(
page: int = Query(1, ge=1),
@@ -89,7 +74,6 @@ async def list_reports(
sort_by: str = Query("updated_at"),
sort_order: str = Query("desc"),
task_manager: TaskManager = Depends(get_task_manager),
clean_release_repository: CleanReleaseRepository = Depends(get_clean_release_repository),
_=Depends(has_permission("tasks", "READ")),
):
with belief_scope("list_reports"):
@@ -119,7 +103,7 @@ async def list_reports(
},
)
service = ReportsService(task_manager, clean_release_repository=clean_release_repository)
service = ReportsService(task_manager)
return service.list_reports(query)
# [/DEF:list_reports:Function]
@@ -132,11 +116,10 @@ async def list_reports(
async def get_report_detail(
report_id: str,
task_manager: TaskManager = Depends(get_task_manager),
clean_release_repository: CleanReleaseRepository = Depends(get_clean_release_repository),
_=Depends(has_permission("tasks", "READ")),
):
with belief_scope("get_report_detail", f"report_id={report_id}"):
service = ReportsService(task_manager, clean_release_repository=clean_release_repository)
service = ReportsService(task_manager)
detail = service.get_report_detail(report_id)
if not detail:
raise HTTPException(

View File

@@ -16,15 +16,10 @@ from pydantic import BaseModel
from ...core.config_models import AppConfig, Environment, GlobalSettings, LoggingConfig
from ...models.storage import StorageConfig
from ...dependencies import get_config_manager, has_permission
from ...core.config_manager import ConfigManager
from ...core.logger import logger, belief_scope
from ...core.superset_client import SupersetClient
from ...services.llm_prompt_templates import normalize_llm_settings
from ...models.llm import ValidationPolicy
from ...models.config import AppConfigRecord
from ...schemas.settings import ValidationPolicyCreate, ValidationPolicyUpdate, ValidationPolicyResponse
from ...core.database import get_db
from sqlalchemy.orm import Session
from ...core.config_manager import ConfigManager
from ...core.logger import logger, belief_scope
from ...core.superset_client import SupersetClient
from ...services.llm_prompt_templates import normalize_llm_settings
# [/SECTION]
# [DEF:LoggingConfigResponse:Class]
@@ -38,51 +33,20 @@ class LoggingConfigResponse(BaseModel):
router = APIRouter()
# [DEF:_normalize_superset_env_url:Function]
# @PURPOSE: Canonicalize Superset environment URL to base host/path without trailing /api/v1.
# @PRE: raw_url can be empty.
# @POST: Returns normalized base URL.
def _normalize_superset_env_url(raw_url: str) -> str:
normalized = str(raw_url or "").strip().rstrip("/")
if normalized.lower().endswith("/api/v1"):
normalized = normalized[:-len("/api/v1")]
return normalized.rstrip("/")
# [/DEF:_normalize_superset_env_url:Function]
# [DEF:_validate_superset_connection_fast:Function]
# @PURPOSE: Run lightweight Superset connectivity validation without full pagination scan.
# @PRE: env contains valid URL and credentials.
# @POST: Raises on auth/API failures; returns None on success.
def _validate_superset_connection_fast(env: Environment) -> None:
client = SupersetClient(env)
# 1) Explicit auth check
client.authenticate()
# 2) Single lightweight API call to ensure read access
client.get_dashboards_page(
query={
"page": 0,
"page_size": 1,
"columns": ["id"],
}
)
# [/DEF:_validate_superset_connection_fast:Function]
# [DEF:get_settings:Function]
# @PURPOSE: Retrieves all application settings.
# @PRE: Config manager is available.
# @POST: Returns masked AppConfig.
# @RETURN: AppConfig - The current configuration.
@router.get("", response_model=AppConfig)
async def get_settings(
async def get_settings(
config_manager: ConfigManager = Depends(get_config_manager),
_ = Depends(has_permission("admin:settings", "READ"))
):
with belief_scope("get_settings"):
logger.info("[get_settings][Entry] Fetching all settings")
config = config_manager.get_config().copy(deep=True)
config.settings.llm = normalize_llm_settings(config.settings.llm)
config = config_manager.get_config().copy(deep=True)
config.settings.llm = normalize_llm_settings(config.settings.llm)
# Mask passwords
for env in config.environments:
if env.password:
@@ -155,11 +119,7 @@ async def get_environments(
):
with belief_scope("get_environments"):
logger.info("[get_environments][Entry] Fetching environments")
environments = config_manager.get_environments()
return [
env.copy(update={"url": _normalize_superset_env_url(env.url)})
for env in environments
]
return config_manager.get_environments()
# [/DEF:get_environments:Function]
# [DEF:add_environment:Function]
@@ -176,11 +136,11 @@ async def add_environment(
):
with belief_scope("add_environment"):
logger.info(f"[add_environment][Entry] Adding environment {env.id}")
env = env.copy(update={"url": _normalize_superset_env_url(env.url)})
# Validate connection before adding (fast path)
# Validate connection before adding
try:
_validate_superset_connection_fast(env)
client = SupersetClient(env)
client.get_dashboards(query={"page_size": 1})
except Exception as e:
logger.error(f"[add_environment][Coherence:Failed] Connection validation failed: {e}")
raise HTTPException(status_code=400, detail=f"Connection validation failed: {e}")
@@ -205,8 +165,6 @@ async def update_environment(
with belief_scope("update_environment"):
logger.info(f"[update_environment][Entry] Updating environment {id}")
env = env.copy(update={"url": _normalize_superset_env_url(env.url)})
# If password is masked, we need the real one for validation
env_to_validate = env.copy(deep=True)
if env_to_validate.password == "********":
@@ -214,9 +172,10 @@ async def update_environment(
if old_env:
env_to_validate.password = old_env.password
# Validate connection before updating (fast path)
# Validate connection before updating
try:
_validate_superset_connection_fast(env_to_validate)
client = SupersetClient(env_to_validate)
client.get_dashboards(query={"page_size": 1})
except Exception as e:
logger.error(f"[update_environment][Coherence:Failed] Connection validation failed: {e}")
raise HTTPException(status_code=400, detail=f"Connection validation failed: {e}")
@@ -262,7 +221,11 @@ async def test_environment_connection(
raise HTTPException(status_code=404, detail=f"Environment {id} not found")
try:
_validate_superset_connection_fast(env)
# Initialize client (this will trigger authentication)
client = SupersetClient(env)
# Try a simple request to verify
client.get_dashboards(query={"page_size": 1})
logger.info(f"[test_environment_connection][Coherence:OK] Connection successful for {id}")
return {"status": "success", "message": "Connection successful"}
@@ -318,14 +281,13 @@ async def update_logging_config(
# [/DEF:update_logging_config:Function]
# [DEF:ConsolidatedSettingsResponse:Class]
class ConsolidatedSettingsResponse(BaseModel):
class ConsolidatedSettingsResponse(BaseModel):
environments: List[dict]
connections: List[dict]
llm: dict
llm_providers: List[dict]
logging: dict
storage: dict
notifications: dict = {}
# [/DEF:ConsolidatedSettingsResponse:Class]
# [DEF:get_consolidated_settings:Function]
@@ -334,7 +296,7 @@ class ConsolidatedSettingsResponse(BaseModel):
# @POST: Returns all consolidated settings.
# @RETURN: ConsolidatedSettingsResponse - All settings categories.
@router.get("/consolidated", response_model=ConsolidatedSettingsResponse)
async def get_consolidated_settings(
async def get_consolidated_settings(
config_manager: ConfigManager = Depends(get_config_manager),
_ = Depends(has_permission("admin:settings", "READ"))
):
@@ -346,7 +308,6 @@ async def get_consolidated_settings(
from ...services.llm_provider import LLMProviderService
from ...core.database import SessionLocal
db = SessionLocal()
notifications_payload = {}
try:
llm_service = LLMProviderService(db)
providers = llm_service.get_all_providers()
@@ -361,24 +322,19 @@ async def get_consolidated_settings(
"is_active": p.is_active
} for p in providers
]
config_record = db.query(AppConfigRecord).filter(AppConfigRecord.id == "global").first()
if config_record and isinstance(config_record.payload, dict):
notifications_payload = config_record.payload.get("notifications", {}) or {}
finally:
db.close()
normalized_llm = normalize_llm_settings(config.settings.llm)
return ConsolidatedSettingsResponse(
environments=[env.dict() for env in config.environments],
connections=config.settings.connections,
llm=normalized_llm,
llm_providers=llm_providers_list,
logging=config.settings.logging.dict(),
storage=config.settings.storage.dict(),
notifications=notifications_payload
)
normalized_llm = normalize_llm_settings(config.settings.llm)
return ConsolidatedSettingsResponse(
environments=[env.dict() for env in config.environments],
connections=config.settings.connections,
llm=normalized_llm,
llm_providers=llm_providers_list,
logging=config.settings.logging.dict(),
storage=config.settings.storage.dict()
)
# [/DEF:get_consolidated_settings:Function]
# [DEF:update_consolidated_settings:Function]
@@ -401,9 +357,9 @@ async def update_consolidated_settings(
if "connections" in settings_patch:
current_settings.connections = settings_patch["connections"]
# Update LLM if provided
if "llm" in settings_patch:
current_settings.llm = normalize_llm_settings(settings_patch["llm"])
# Update LLM if provided
if "llm" in settings_patch:
current_settings.llm = normalize_llm_settings(settings_patch["llm"])
# Update Logging if provided
if "logging" in settings_patch:
@@ -417,88 +373,8 @@ async def update_consolidated_settings(
raise HTTPException(status_code=400, detail=message)
current_settings.storage = new_storage
if "notifications" in settings_patch:
payload = config_manager.get_payload()
payload["notifications"] = settings_patch["notifications"]
config_manager.save_config(payload)
config_manager.update_global_settings(current_settings)
return {"status": "success", "message": "Settings updated"}
# [/DEF:update_consolidated_settings:Function]
# [DEF:get_validation_policies:Function]
# @PURPOSE: Lists all validation policies.
# @RETURN: List[ValidationPolicyResponse] - List of policies.
@router.get("/automation/policies", response_model=List[ValidationPolicyResponse])
async def get_validation_policies(
db: Session = Depends(get_db),
_ = Depends(has_permission("admin:settings", "READ"))
):
with belief_scope("get_validation_policies"):
return db.query(ValidationPolicy).all()
# [/DEF:get_validation_policies:Function]
# [DEF:create_validation_policy:Function]
# @PURPOSE: Creates a new validation policy.
# @PARAM: policy (ValidationPolicyCreate) - The policy data.
# @RETURN: ValidationPolicyResponse - The created policy.
@router.post("/automation/policies", response_model=ValidationPolicyResponse)
async def create_validation_policy(
policy: ValidationPolicyCreate,
db: Session = Depends(get_db),
_ = Depends(has_permission("admin:settings", "WRITE"))
):
with belief_scope("create_validation_policy"):
db_policy = ValidationPolicy(**policy.dict())
db.add(db_policy)
db.commit()
db.refresh(db_policy)
return db_policy
# [/DEF:create_validation_policy:Function]
# [DEF:update_validation_policy:Function]
# @PURPOSE: Updates an existing validation policy.
# @PARAM: id (str) - The ID of the policy to update.
# @PARAM: policy (ValidationPolicyUpdate) - The updated policy data.
# @RETURN: ValidationPolicyResponse - The updated policy.
@router.patch("/automation/policies/{id}", response_model=ValidationPolicyResponse)
async def update_validation_policy(
id: str,
policy: ValidationPolicyUpdate,
db: Session = Depends(get_db),
_ = Depends(has_permission("admin:settings", "WRITE"))
):
with belief_scope("update_validation_policy"):
db_policy = db.query(ValidationPolicy).filter(ValidationPolicy.id == id).first()
if not db_policy:
raise HTTPException(status_code=404, detail="Policy not found")
update_data = policy.dict(exclude_unset=True)
for key, value in update_data.items():
setattr(db_policy, key, value)
db.commit()
db.refresh(db_policy)
return db_policy
# [/DEF:update_validation_policy:Function]
# [DEF:delete_validation_policy:Function]
# @PURPOSE: Deletes a validation policy.
# @PARAM: id (str) - The ID of the policy to delete.
@router.delete("/automation/policies/{id}")
async def delete_validation_policy(
id: str,
db: Session = Depends(get_db),
_ = Depends(has_permission("admin:settings", "WRITE"))
):
with belief_scope("delete_validation_policy"):
db_policy = db.query(ValidationPolicy).filter(ValidationPolicy.id == id).first()
if not db_policy:
raise HTTPException(status_code=404, detail="Policy not found")
db.delete(db_policy)
db.commit()
return {"message": "Policy deleted"}
# [/DEF:delete_validation_policy:Function]
# [/DEF:SettingsRouter:Module]

View File

@@ -4,30 +4,30 @@
# @PURPOSE: Defines the FastAPI router for task-related endpoints, allowing clients to create, list, and get the status of tasks.
# @LAYER: UI (API)
# @RELATION: Depends on the TaskManager. It is included by the main app.
from typing import List, Dict, Any, Optional
from typing import List, Dict, Any, Optional
from fastapi import APIRouter, Depends, HTTPException, status, Query
from pydantic import BaseModel
from ...core.logger import belief_scope
from ...core.task_manager import TaskManager, Task, TaskStatus, LogEntry
from ...core.task_manager.models import LogFilter, LogStats
from ...dependencies import get_task_manager, has_permission, get_current_user, get_config_manager
from ...core.config_manager import ConfigManager
from ...services.llm_prompt_templates import (
is_multimodal_model,
normalize_llm_settings,
resolve_bound_provider_id,
)
from ...core.task_manager import TaskManager, Task, TaskStatus, LogEntry
from ...core.task_manager.models import LogFilter, LogStats
from ...dependencies import get_task_manager, has_permission, get_current_user, get_config_manager
from ...core.config_manager import ConfigManager
from ...services.llm_prompt_templates import (
is_multimodal_model,
normalize_llm_settings,
resolve_bound_provider_id,
)
router = APIRouter()
TASK_TYPE_PLUGIN_MAP = {
"llm_validation": ["llm_dashboard_validation"],
"backup": ["superset-backup"],
"migration": ["superset-migration"],
}
class CreateTaskRequest(BaseModel):
router = APIRouter()
TASK_TYPE_PLUGIN_MAP = {
"llm_validation": ["llm_dashboard_validation"],
"backup": ["superset-backup"],
"migration": ["superset-migration"],
}
class CreateTaskRequest(BaseModel):
plugin_id: str
params: Dict[str, Any]
@@ -45,54 +45,54 @@ class ResumeTaskRequest(BaseModel):
# @PRE: plugin_id must exist and params must be valid for that plugin.
# @POST: A new task is created and started.
# @RETURN: Task - The created task instance.
async def create_task(
request: CreateTaskRequest,
task_manager: TaskManager = Depends(get_task_manager),
current_user = Depends(get_current_user),
config_manager: ConfigManager = Depends(get_config_manager),
):
async def create_task(
request: CreateTaskRequest,
task_manager: TaskManager = Depends(get_task_manager),
current_user = Depends(get_current_user),
config_manager: ConfigManager = Depends(get_config_manager),
):
# Dynamic permission check based on plugin_id
has_permission(f"plugin:{request.plugin_id}", "EXECUTE")(current_user)
"""
Create and start a new task for a given plugin.
"""
with belief_scope("create_task"):
try:
# Special handling for LLM tasks to resolve provider config by task binding.
if request.plugin_id in {"llm_dashboard_validation", "llm_documentation"}:
from ...core.database import SessionLocal
from ...services.llm_provider import LLMProviderService
db = SessionLocal()
try:
llm_service = LLMProviderService(db)
provider_id = request.params.get("provider_id")
if not provider_id:
llm_settings = normalize_llm_settings(config_manager.get_config().settings.llm)
binding_key = "dashboard_validation" if request.plugin_id == "llm_dashboard_validation" else "documentation"
provider_id = resolve_bound_provider_id(llm_settings, binding_key)
if provider_id:
request.params["provider_id"] = provider_id
if not provider_id:
providers = llm_service.get_all_providers()
active_provider = next((p for p in providers if p.is_active), None)
if active_provider:
provider_id = active_provider.id
request.params["provider_id"] = provider_id
if provider_id:
db_provider = llm_service.get_provider(provider_id)
if not db_provider:
raise ValueError(f"LLM Provider {provider_id} not found")
if request.plugin_id == "llm_dashboard_validation" and not is_multimodal_model(
db_provider.default_model,
db_provider.provider_type,
):
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail="Selected provider model is not multimodal for dashboard validation",
)
finally:
db.close()
try:
# Special handling for LLM tasks to resolve provider config by task binding.
if request.plugin_id in {"llm_dashboard_validation", "llm_documentation"}:
from ...core.database import SessionLocal
from ...services.llm_provider import LLMProviderService
db = SessionLocal()
try:
llm_service = LLMProviderService(db)
provider_id = request.params.get("provider_id")
if not provider_id:
llm_settings = normalize_llm_settings(config_manager.get_config().settings.llm)
binding_key = "dashboard_validation" if request.plugin_id == "llm_dashboard_validation" else "documentation"
provider_id = resolve_bound_provider_id(llm_settings, binding_key)
if provider_id:
request.params["provider_id"] = provider_id
if not provider_id:
providers = llm_service.get_all_providers()
active_provider = next((p for p in providers if p.is_active), None)
if active_provider:
provider_id = active_provider.id
request.params["provider_id"] = provider_id
if provider_id:
db_provider = llm_service.get_provider(provider_id)
if not db_provider:
raise ValueError(f"LLM Provider {provider_id} not found")
if request.plugin_id == "llm_dashboard_validation" and not is_multimodal_model(
db_provider.default_model,
db_provider.provider_type,
):
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail="Selected provider model is not multimodal for dashboard validation",
)
finally:
db.close()
task = await task_manager.create_task(
plugin_id=request.plugin_id,
@@ -113,36 +113,36 @@ async def create_task(
# @PRE: task_manager must be available.
# @POST: Returns a list of tasks.
# @RETURN: List[Task] - List of tasks.
async def list_tasks(
limit: int = 10,
offset: int = 0,
status_filter: Optional[TaskStatus] = Query(None, alias="status"),
task_type: Optional[str] = Query(None, description="Task category: llm_validation, backup, migration"),
plugin_id: Optional[List[str]] = Query(None, description="Filter by plugin_id (repeatable query param)"),
completed_only: bool = Query(False, description="Return only completed tasks (SUCCESS/FAILED)"),
task_manager: TaskManager = Depends(get_task_manager),
_ = Depends(has_permission("tasks", "READ"))
):
"""
Retrieve a list of tasks with pagination and optional status filter.
"""
with belief_scope("list_tasks"):
plugin_filters = list(plugin_id) if plugin_id else []
if task_type:
if task_type not in TASK_TYPE_PLUGIN_MAP:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Unsupported task_type '{task_type}'. Allowed: {', '.join(TASK_TYPE_PLUGIN_MAP.keys())}"
)
plugin_filters.extend(TASK_TYPE_PLUGIN_MAP[task_type])
return task_manager.get_tasks(
limit=limit,
offset=offset,
status=status_filter,
plugin_ids=plugin_filters or None,
completed_only=completed_only
)
async def list_tasks(
limit: int = 10,
offset: int = 0,
status_filter: Optional[TaskStatus] = Query(None, alias="status"),
task_type: Optional[str] = Query(None, description="Task category: llm_validation, backup, migration"),
plugin_id: Optional[List[str]] = Query(None, description="Filter by plugin_id (repeatable query param)"),
completed_only: bool = Query(False, description="Return only completed tasks (SUCCESS/FAILED)"),
task_manager: TaskManager = Depends(get_task_manager),
_ = Depends(has_permission("tasks", "READ"))
):
"""
Retrieve a list of tasks with pagination and optional status filter.
"""
with belief_scope("list_tasks"):
plugin_filters = list(plugin_id) if plugin_id else []
if task_type:
if task_type not in TASK_TYPE_PLUGIN_MAP:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Unsupported task_type '{task_type}'. Allowed: {', '.join(TASK_TYPE_PLUGIN_MAP.keys())}"
)
plugin_filters.extend(TASK_TYPE_PLUGIN_MAP[task_type])
return task_manager.get_tasks(
limit=limit,
offset=offset,
status=status_filter,
plugin_ids=plugin_filters or None,
completed_only=completed_only
)
# [/DEF:list_tasks:Function]
@router.get("/{task_id}", response_model=Task)
@@ -182,23 +182,6 @@ async def get_task(
# @POST: Returns a list of log entries or raises 404.
# @RETURN: List[LogEntry] - List of log entries.
# @TIER: CRITICAL
# @TEST_CONTRACT get_task_logs_api ->
# {
# required_params: {task_id: str},
# optional_params: {level: str, source: str, search: str},
# invariants: ["returns 404 for non-existent task", "applies filters correctly"]
# }
# @TEST_FIXTURE valid_task_logs_request -> {"task_id": "test_1", "level": "INFO"}
# @TEST_EDGE task_not_found -> raises 404
# @TEST_EDGE invalid_limit -> Query(limit=0) returns 422
# @TEST_INVARIANT response_purity -> verifies: [valid_task_logs_request]
# @TEST_CONTRACT: TaskLogQueryInput -> List[LogEntry]
# @TEST_SCENARIO: existing_task_logs_filtered -> Returns filtered logs by level/source/search with pagination.
# @TEST_FIXTURE: valid_task_with_mixed_logs -> backend/tests/fixtures/task_logs/valid_task_with_mixed_logs.json
# @TEST_EDGE: missing_task -> Unknown task_id returns 404 Task not found.
# @TEST_EDGE: invalid_level_type -> Non-string/invalid level query rejected by validation or yields empty result.
# @TEST_EDGE: pagination_bounds -> offset=0 and limit=1000 remain within API bounds and do not overflow.
# @TEST_INVARIANT: logs_only_for_existing_task -> VERIFIED_BY: [existing_task_logs_filtered, missing_task]
async def get_task_logs(
task_id: str,
level: Optional[str] = Query(None, description="Filter by log level (DEBUG, INFO, WARNING, ERROR)"),
@@ -345,4 +328,4 @@ async def clear_tasks(
task_manager.clear_tasks(status)
return
# [/DEF:clear_tasks:Function]
# [/DEF:TasksRouter:Module]
# [/DEF:TasksRouter:Module]

View File

@@ -21,7 +21,7 @@ import asyncio
from .dependencies import get_task_manager, get_scheduler_service
from .core.utils.network import NetworkError
from .core.logger import logger, belief_scope
from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git, storage, admin, llm, dashboards, datasets, reports, assistant, clean_release, clean_release_v2, profile, health
from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git, storage, admin, llm, dashboards, datasets, reports, assistant
from .api import auth
# [DEF:App:Global]
@@ -133,10 +133,6 @@ app.include_router(dashboards.router)
app.include_router(datasets.router)
app.include_router(reports.router)
app.include_router(assistant.router, prefix="/api/assistant", tags=["Assistant"])
app.include_router(clean_release.router)
app.include_router(clean_release_v2.router)
app.include_router(profile.router)
app.include_router(health.router)
# [DEF:api.include_routers:Action]
@@ -151,21 +147,6 @@ app.include_router(health.router)
# @POST: WebSocket connection is managed and logs are streamed until disconnect.
# @TIER: CRITICAL
# @UX_STATE: Connecting -> Streaming -> (Disconnected)
#
# @TEST_CONTRACT: WebSocketLogStreamApi ->
# {
# required_fields: {websocket: WebSocket, task_id: str},
# optional_fields: {source: str, level: str},
# invariants: [
# "Accepts the WebSocket connection",
# "Applies source and level filters correctly to streamed logs",
# "Cleans up subscriptions on disconnect"
# ]
# }
# @TEST_FIXTURE: valid_ws_connection -> {"task_id": "test_1", "source": "plugin"}
# @TEST_EDGE: task_not_found_ws -> closes connection or sends error
# @TEST_EDGE: empty_task_logs -> waits for new logs
# @TEST_INVARIANT: consistent_streaming -> verifies: [valid_ws_connection]
@app.websocket("/ws/logs/{task_id}")
async def websocket_endpoint(
websocket: WebSocket,

View File

@@ -1,3 +0,0 @@
# [DEF:src.core:Package]
# @PURPOSE: Backend core services and infrastructure package root.
# [/DEF:src.core:Package]

View File

@@ -1,128 +0,0 @@
# [DEF:backend.src.core.__tests__.test_superset_profile_lookup:Module]
# @TIER: STANDARD
# @SEMANTICS: tests, superset, profile, lookup, fallback, sorting
# @PURPOSE: Verifies Superset profile lookup adapter payload normalization and fallback error precedence.
# @LAYER: Domain
# @RELATION: TESTS -> backend.src.core.superset_profile_lookup
# [SECTION: IMPORTS]
import json
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional
import pytest
backend_dir = str(Path(__file__).parent.parent.parent.parent.resolve())
if backend_dir not in sys.path:
sys.path.insert(0, backend_dir)
from src.core.superset_profile_lookup import SupersetAccountLookupAdapter
from src.core.utils.network import AuthenticationError, SupersetAPIError
# [/SECTION]
# [DEF:_RecordingNetworkClient:Class]
# @PURPOSE: Records request payloads and returns scripted responses for deterministic adapter tests.
class _RecordingNetworkClient:
# [DEF:__init__:Function]
# @PURPOSE: Initializes scripted network responses.
# @PRE: scripted_responses is ordered per expected request sequence.
# @POST: Instance stores response script and captures subsequent request calls.
def __init__(self, scripted_responses: List[Any]):
self._scripted_responses = scripted_responses
self.calls: List[Dict[str, Any]] = []
# [/DEF:__init__:Function]
# [DEF:request:Function]
# @PURPOSE: Mimics APIClient.request while capturing call arguments.
# @PRE: method and endpoint are provided.
# @POST: Returns scripted response or raises scripted exception.
def request(
self,
method: str,
endpoint: str,
params: Optional[Dict[str, Any]] = None,
**kwargs,
) -> Dict[str, Any]:
self.calls.append(
{
"method": method,
"endpoint": endpoint,
"params": params or {},
}
)
index = len(self.calls) - 1
response = self._scripted_responses[index]
if isinstance(response, Exception):
raise response
return response
# [/DEF:request:Function]
# [/DEF:_RecordingNetworkClient:Class]
# [DEF:test_get_users_page_sends_lowercase_order_direction:Function]
# @PURPOSE: Ensures adapter sends lowercase order_direction compatible with Superset rison schema.
# @PRE: Adapter is initialized with recording network client.
# @POST: First request query payload contains order_direction='asc' for asc sort.
def test_get_users_page_sends_lowercase_order_direction():
client = _RecordingNetworkClient(
scripted_responses=[{"result": [{"username": "admin"}], "count": 1}]
)
adapter = SupersetAccountLookupAdapter(network_client=client, environment_id="ss-dev")
adapter.get_users_page(
search="admin",
page_index=0,
page_size=20,
sort_column="username",
sort_order="asc",
)
sent_query = json.loads(client.calls[0]["params"]["q"])
assert sent_query["order_direction"] == "asc"
# [/DEF:test_get_users_page_sends_lowercase_order_direction:Function]
# [DEF:test_get_users_page_preserves_primary_schema_error_over_fallback_auth_error:Function]
# @PURPOSE: Ensures fallback auth error does not mask primary schema/query failure.
# @PRE: Primary endpoint fails with SupersetAPIError and fallback fails with AuthenticationError.
# @POST: Raised exception remains primary SupersetAPIError (non-auth) to preserve root cause.
def test_get_users_page_preserves_primary_schema_error_over_fallback_auth_error():
client = _RecordingNetworkClient(
scripted_responses=[
SupersetAPIError("API Error 400: bad rison schema"),
AuthenticationError(),
]
)
adapter = SupersetAccountLookupAdapter(network_client=client, environment_id="ss-dev")
with pytest.raises(SupersetAPIError) as exc_info:
adapter.get_users_page(sort_order="asc")
assert "API Error 400" in str(exc_info.value)
assert not isinstance(exc_info.value, AuthenticationError)
# [/DEF:test_get_users_page_preserves_primary_schema_error_over_fallback_auth_error:Function]
# [DEF:test_get_users_page_uses_fallback_endpoint_when_primary_fails:Function]
# @PURPOSE: Verifies adapter retries second users endpoint and succeeds when fallback is healthy.
# @PRE: Primary endpoint fails; fallback returns valid users payload.
# @POST: Result status is success and both endpoints were attempted in order.
def test_get_users_page_uses_fallback_endpoint_when_primary_fails():
client = _RecordingNetworkClient(
scripted_responses=[
SupersetAPIError("Primary endpoint failed"),
{"result": [{"username": "admin"}], "count": 1},
]
)
adapter = SupersetAccountLookupAdapter(network_client=client, environment_id="ss-dev")
result = adapter.get_users_page()
assert result["status"] == "success"
assert [call["endpoint"] for call in client.calls] == ["/security/users/", "/security/users"]
# [/DEF:test_get_users_page_uses_fallback_endpoint_when_primary_fails:Function]
# [/DEF:backend.src.core.__tests__.test_superset_profile_lookup:Module]

View File

@@ -1,99 +0,0 @@
import pytest
from datetime import time, date, datetime, timedelta
from src.core.scheduler import ThrottledSchedulerConfigurator
# [DEF:test_throttled_scheduler:Module]
# @TIER: STANDARD
# @PURPOSE: Unit tests for ThrottledSchedulerConfigurator distribution logic.
def test_calculate_schedule_even_distribution():
"""
@TEST_SCENARIO: 3 tasks in a 2-hour window should be spaced 1 hour apart.
"""
start = time(1, 0)
end = time(3, 0)
dashboards = ["d1", "d2", "d3"]
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert len(schedule) == 3
assert schedule[0] == datetime(2024, 1, 1, 1, 0)
assert schedule[1] == datetime(2024, 1, 1, 2, 0)
assert schedule[2] == datetime(2024, 1, 1, 3, 0)
def test_calculate_schedule_midnight_crossing():
"""
@TEST_SCENARIO: Window from 23:00 to 01:00 (next day).
"""
start = time(23, 0)
end = time(1, 0)
dashboards = ["d1", "d2", "d3"]
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert len(schedule) == 3
assert schedule[0] == datetime(2024, 1, 1, 23, 0)
assert schedule[1] == datetime(2024, 1, 2, 0, 0)
assert schedule[2] == datetime(2024, 1, 2, 1, 0)
def test_calculate_schedule_single_task():
"""
@TEST_SCENARIO: Single task should be scheduled at start time.
"""
start = time(1, 0)
end = time(2, 0)
dashboards = ["d1"]
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert len(schedule) == 1
assert schedule[0] == datetime(2024, 1, 1, 1, 0)
def test_calculate_schedule_empty_list():
"""
@TEST_SCENARIO: Empty dashboard list returns empty schedule.
"""
start = time(1, 0)
end = time(2, 0)
dashboards = []
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert schedule == []
def test_calculate_schedule_zero_window():
"""
@TEST_SCENARIO: Window start == end. All tasks at start time.
"""
start = time(1, 0)
end = time(1, 0)
dashboards = ["d1", "d2"]
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert len(schedule) == 2
assert schedule[0] == datetime(2024, 1, 1, 1, 0)
assert schedule[1] == datetime(2024, 1, 1, 1, 0)
def test_calculate_schedule_very_small_window():
"""
@TEST_SCENARIO: Window smaller than number of tasks (in seconds).
"""
start = time(1, 0, 0)
end = time(1, 0, 1) # 1 second window
dashboards = ["d1", "d2", "d3"]
today = date(2024, 1, 1)
schedule = ThrottledSchedulerConfigurator.calculate_schedule(start, end, dashboards, today)
assert len(schedule) == 3
assert schedule[0] == datetime(2024, 1, 1, 1, 0, 0)
assert schedule[1] == datetime(2024, 1, 1, 1, 0, 0, 500000) # 0.5s
assert schedule[2] == datetime(2024, 1, 1, 1, 0, 1)
# [/DEF:test_throttled_scheduler:Module]

View File

@@ -1,298 +0,0 @@
# [DEF:backend.src.core.async_superset_client:Module]
#
# @TIER: CRITICAL
# @SEMANTICS: superset, async, client, httpx, dashboards, datasets
# @PURPOSE: Async Superset client for dashboard hot-path requests without blocking FastAPI event loop.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> backend.src.core.superset_client
# @RELATION: DEPENDS_ON -> backend.src.core.utils.async_network.AsyncAPIClient
# @INVARIANT: Async dashboard operations reuse shared auth cache and avoid sync requests in async routes.
# [SECTION: IMPORTS]
import asyncio
import json
import re
from typing import Any, Dict, List, Optional, Tuple, cast
from .config_models import Environment
from .logger import logger as app_logger, belief_scope
from .superset_client import SupersetClient
from .utils.async_network import AsyncAPIClient
# [/SECTION]
# [DEF:AsyncSupersetClient:Class]
# @PURPOSE: Async sibling of SupersetClient for dashboard read paths.
class AsyncSupersetClient(SupersetClient):
# [DEF:__init__:Function]
# @PURPOSE: Initialize async Superset client with AsyncAPIClient transport.
# @PRE: env is valid.
# @POST: Client uses async network transport and inherited projection helpers.
def __init__(self, env: Environment):
self.env = env
auth_payload = {
"username": env.username,
"password": env.password,
"provider": "db",
"refresh": "true",
}
self.network = AsyncAPIClient(
config={"base_url": env.url, "auth": auth_payload},
verify_ssl=env.verify_ssl,
timeout=env.timeout,
)
self.delete_before_reimport = False
# [/DEF:__init__:Function]
# [DEF:aclose:Function]
# @PURPOSE: Close async transport resources.
# @POST: Underlying AsyncAPIClient is closed.
async def aclose(self) -> None:
await self.network.aclose()
# [/DEF:aclose:Function]
# [DEF:get_dashboards_page_async:Function]
# @PURPOSE: Fetch one dashboards page asynchronously.
# @POST: Returns total count and page result list.
async def get_dashboards_page_async(self, query: Optional[Dict] = None) -> Tuple[int, List[Dict]]:
with belief_scope("AsyncSupersetClient.get_dashboards_page_async"):
validated_query = self._validate_query_params(query or {})
if "columns" not in validated_query:
validated_query["columns"] = [
"slug",
"id",
"url",
"changed_on_utc",
"dashboard_title",
"published",
"created_by",
"changed_by",
"changed_by_name",
"owners",
]
response_json = cast(
Dict[str, Any],
await self.network.request(
method="GET",
endpoint="/dashboard/",
params={"q": json.dumps(validated_query)},
),
)
result = response_json.get("result", [])
total_count = response_json.get("count", len(result))
return total_count, result
# [/DEF:get_dashboards_page_async:Function]
# [DEF:get_dashboard_async:Function]
# @PURPOSE: Fetch one dashboard payload asynchronously.
# @POST: Returns raw dashboard payload from Superset API.
async def get_dashboard_async(self, dashboard_id: int) -> Dict:
with belief_scope("AsyncSupersetClient.get_dashboard_async", f"id={dashboard_id}"):
response = await self.network.request(method="GET", endpoint=f"/dashboard/{dashboard_id}")
return cast(Dict, response)
# [/DEF:get_dashboard_async:Function]
# [DEF:get_chart_async:Function]
# @PURPOSE: Fetch one chart payload asynchronously.
# @POST: Returns raw chart payload from Superset API.
async def get_chart_async(self, chart_id: int) -> Dict:
with belief_scope("AsyncSupersetClient.get_chart_async", f"id={chart_id}"):
response = await self.network.request(method="GET", endpoint=f"/chart/{chart_id}")
return cast(Dict, response)
# [/DEF:get_chart_async:Function]
# [DEF:get_dashboard_detail_async:Function]
# @PURPOSE: Fetch dashboard detail asynchronously with concurrent charts/datasets requests.
# @POST: Returns dashboard detail payload for overview page.
async def get_dashboard_detail_async(self, dashboard_id: int) -> Dict:
with belief_scope("AsyncSupersetClient.get_dashboard_detail_async", f"id={dashboard_id}"):
dashboard_response = await self.get_dashboard_async(dashboard_id)
dashboard_data = dashboard_response.get("result", dashboard_response)
charts: List[Dict] = []
datasets: List[Dict] = []
def extract_dataset_id_from_form_data(form_data: Optional[Dict]) -> Optional[int]:
if not isinstance(form_data, dict):
return None
datasource = form_data.get("datasource")
if isinstance(datasource, str):
matched = re.match(r"^(\d+)__", datasource)
if matched:
try:
return int(matched.group(1))
except ValueError:
return None
if isinstance(datasource, dict):
ds_id = datasource.get("id")
try:
return int(ds_id) if ds_id is not None else None
except (TypeError, ValueError):
return None
ds_id = form_data.get("datasource_id")
try:
return int(ds_id) if ds_id is not None else None
except (TypeError, ValueError):
return None
chart_task = self.network.request(
method="GET",
endpoint=f"/dashboard/{dashboard_id}/charts",
)
dataset_task = self.network.request(
method="GET",
endpoint=f"/dashboard/{dashboard_id}/datasets",
)
charts_response, datasets_response = await asyncio.gather(
chart_task,
dataset_task,
return_exceptions=True,
)
if not isinstance(charts_response, Exception):
charts_payload = charts_response.get("result", []) if isinstance(charts_response, dict) else []
for chart_obj in charts_payload:
if not isinstance(chart_obj, dict):
continue
chart_id = chart_obj.get("id")
if chart_id is None:
continue
form_data = chart_obj.get("form_data")
if isinstance(form_data, str):
try:
form_data = json.loads(form_data)
except Exception:
form_data = {}
dataset_id = extract_dataset_id_from_form_data(form_data) or chart_obj.get("datasource_id")
charts.append({
"id": int(chart_id),
"title": chart_obj.get("slice_name") or chart_obj.get("name") or f"Chart {chart_id}",
"viz_type": (form_data.get("viz_type") if isinstance(form_data, dict) else None),
"dataset_id": int(dataset_id) if dataset_id is not None else None,
"last_modified": chart_obj.get("changed_on"),
"overview": chart_obj.get("description") or (form_data.get("viz_type") if isinstance(form_data, dict) else None) or "Chart",
})
else:
app_logger.warning("[get_dashboard_detail_async][Warning] Failed to fetch dashboard charts: %s", charts_response)
if not isinstance(datasets_response, Exception):
datasets_payload = datasets_response.get("result", []) if isinstance(datasets_response, dict) else []
for dataset_obj in datasets_payload:
if not isinstance(dataset_obj, dict):
continue
dataset_id = dataset_obj.get("id")
if dataset_id is None:
continue
db_payload = dataset_obj.get("database")
db_name = db_payload.get("database_name") if isinstance(db_payload, dict) else None
table_name = dataset_obj.get("table_name") or dataset_obj.get("datasource_name") or dataset_obj.get("name") or f"Dataset {dataset_id}"
schema = dataset_obj.get("schema")
fq_name = f"{schema}.{table_name}" if schema else table_name
datasets.append({
"id": int(dataset_id),
"table_name": table_name,
"schema": schema,
"database": db_name or dataset_obj.get("database_name") or "Unknown",
"last_modified": dataset_obj.get("changed_on"),
"overview": fq_name,
})
else:
app_logger.warning("[get_dashboard_detail_async][Warning] Failed to fetch dashboard datasets: %s", datasets_response)
if not charts:
raw_position_json = dashboard_data.get("position_json")
chart_ids_from_position = set()
if isinstance(raw_position_json, str) and raw_position_json:
try:
parsed_position = json.loads(raw_position_json)
chart_ids_from_position.update(self._extract_chart_ids_from_layout(parsed_position))
except Exception:
pass
elif isinstance(raw_position_json, dict):
chart_ids_from_position.update(self._extract_chart_ids_from_layout(raw_position_json))
raw_json_metadata = dashboard_data.get("json_metadata")
if isinstance(raw_json_metadata, str) and raw_json_metadata:
try:
parsed_metadata = json.loads(raw_json_metadata)
chart_ids_from_position.update(self._extract_chart_ids_from_layout(parsed_metadata))
except Exception:
pass
elif isinstance(raw_json_metadata, dict):
chart_ids_from_position.update(self._extract_chart_ids_from_layout(raw_json_metadata))
fallback_chart_tasks = [
self.get_chart_async(int(chart_id))
for chart_id in sorted(chart_ids_from_position)
]
fallback_chart_responses = await asyncio.gather(
*fallback_chart_tasks,
return_exceptions=True,
)
for chart_id, chart_response in zip(sorted(chart_ids_from_position), fallback_chart_responses):
if isinstance(chart_response, Exception):
app_logger.warning("[get_dashboard_detail_async][Warning] Failed to resolve fallback chart %s: %s", chart_id, chart_response)
continue
chart_data = chart_response.get("result", chart_response)
charts.append({
"id": int(chart_id),
"title": chart_data.get("slice_name") or chart_data.get("name") or f"Chart {chart_id}",
"viz_type": chart_data.get("viz_type"),
"dataset_id": chart_data.get("datasource_id"),
"last_modified": chart_data.get("changed_on"),
"overview": chart_data.get("description") or chart_data.get("viz_type") or "Chart",
})
dataset_ids_from_charts = {
c.get("dataset_id")
for c in charts
if c.get("dataset_id") is not None
}
known_dataset_ids = {d.get("id") for d in datasets if d.get("id") is not None}
missing_dataset_ids = sorted(int(item) for item in dataset_ids_from_charts if item not in known_dataset_ids)
if missing_dataset_ids:
dataset_fetch_tasks = [
self.network.request(method="GET", endpoint=f"/dataset/{dataset_id}")
for dataset_id in missing_dataset_ids
]
dataset_fetch_responses = await asyncio.gather(
*dataset_fetch_tasks,
return_exceptions=True,
)
for dataset_id, dataset_response in zip(missing_dataset_ids, dataset_fetch_responses):
if isinstance(dataset_response, Exception):
app_logger.warning("[get_dashboard_detail_async][Warning] Failed to backfill dataset %s: %s", dataset_id, dataset_response)
continue
dataset_data = dataset_response.get("result", dataset_response) if isinstance(dataset_response, dict) else {}
db_payload = dataset_data.get("database")
db_name = db_payload.get("database_name") if isinstance(db_payload, dict) else None
table_name = dataset_data.get("table_name") or dataset_data.get("datasource_name") or dataset_data.get("name") or f"Dataset {dataset_id}"
schema = dataset_data.get("schema")
fq_name = f"{schema}.{table_name}" if schema else table_name
datasets.append({
"id": int(dataset_id),
"table_name": table_name,
"schema": schema,
"database": db_name or dataset_data.get("database_name") or "Unknown",
"last_modified": dataset_data.get("changed_on"),
"overview": fq_name,
})
return {
"id": int(dashboard_data.get("id") or dashboard_id),
"title": dashboard_data.get("dashboard_title") or dashboard_data.get("title") or f"Dashboard {dashboard_id}",
"slug": dashboard_data.get("slug"),
"url": dashboard_data.get("url"),
"description": dashboard_data.get("description"),
"last_modified": dashboard_data.get("changed_on_utc") or dashboard_data.get("changed_on"),
"published": dashboard_data.get("published"),
"charts": charts,
"datasets": datasets,
"chart_count": len(charts),
"dataset_count": len(datasets),
}
# [/DEF:get_dashboard_detail_async:Function]
# [/DEF:AsyncSupersetClient:Class]
# [/DEF:backend.src.core.async_superset_client:Module]

View File

@@ -1,3 +0,0 @@
# [DEF:src.core.auth:Package]
# @PURPOSE: Authentication and authorization package root.
# [/DEF:src.core.auth:Package]

View File

@@ -14,8 +14,6 @@ import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from src.core.database import Base
# Import all models to ensure they are registered with Base before create_all - must import both auth and mapping to ensure Base knows about all tables
from src.models import mapping, auth, task, report
from src.models.auth import User, Role, Permission, ADGroupMapping
from src.services.auth_service import AuthService
from src.core.auth.repository import AuthRepository
@@ -178,94 +176,4 @@ def test_ad_group_mapping(auth_repo):
assert retrieved_mapping.role_id == role.id
def test_authenticate_user_updates_last_login(auth_service, auth_repo):
"""@SIDE_EFFECT: authenticate_user updates last_login timestamp on success."""
user = User(
username="loginuser",
email="login@example.com",
password_hash=get_password_hash("mypassword"),
auth_source="LOCAL"
)
auth_repo.db.add(user)
auth_repo.db.commit()
assert user.last_login is None
authenticated = auth_service.authenticate_user("loginuser", "mypassword")
assert authenticated is not None
assert authenticated.last_login is not None
def test_authenticate_inactive_user(auth_service, auth_repo):
"""@PRE: User with is_active=False should not authenticate."""
user = User(
username="inactive_user",
email="inactive@example.com",
password_hash=get_password_hash("testpass"),
auth_source="LOCAL",
is_active=False
)
auth_repo.db.add(user)
auth_repo.db.commit()
result = auth_service.authenticate_user("inactive_user", "testpass")
assert result is None
def test_verify_password_empty_hash():
"""@PRE: verify_password with empty/None hash returns False."""
assert verify_password("anypassword", "") is False
assert verify_password("anypassword", None) is False
def test_provision_adfs_user_new(auth_service, auth_repo):
"""@POST: provision_adfs_user creates a new ADFS user with correct roles."""
# Set up a role and AD group mapping
role = Role(name="ADFS_Viewer", description="ADFS viewer role")
auth_repo.db.add(role)
auth_repo.db.commit()
mapping = ADGroupMapping(ad_group="DOMAIN\\Viewers", role_id=role.id)
auth_repo.db.add(mapping)
auth_repo.db.commit()
user_info = {
"upn": "newadfsuser@domain.com",
"email": "newadfsuser@domain.com",
"groups": ["DOMAIN\\Viewers"]
}
user = auth_service.provision_adfs_user(user_info)
assert user is not None
assert user.username == "newadfsuser@domain.com"
assert user.auth_source == "ADFS"
assert user.is_active is True
assert len(user.roles) == 1
assert user.roles[0].name == "ADFS_Viewer"
def test_provision_adfs_user_existing(auth_service, auth_repo):
"""@POST: provision_adfs_user updates roles for existing user."""
# Create existing user
existing = User(
username="existingadfs@domain.com",
email="existingadfs@domain.com",
auth_source="ADFS",
is_active=True
)
auth_repo.db.add(existing)
auth_repo.db.commit()
user_info = {
"upn": "existingadfs@domain.com",
"email": "existingadfs@domain.com",
"groups": []
}
user = auth_service.provision_adfs_user(user_info)
assert user is not None
assert user.username == "existingadfs@domain.com"
assert len(user.roles) == 0 # No matching group mappings
# [/DEF:test_auth:Module]

View File

@@ -1,146 +1,106 @@
# [DEF:backend.src.core.auth.repository:Module]
#
# @TIER: CRITICAL
# @SEMANTICS: auth, repository, database, user, role, permission
# @PURPOSE: Data access layer for authentication and user preference entities.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[sqlalchemy.orm.Session]
# @RELATION: [DEPENDS_ON] ->[backend.src.models.auth]
# @RELATION: [DEPENDS_ON] ->[backend.src.models.profile]
# @RELATION: [DEPENDS_ON] ->[backend.src.core.logger.belief_scope]
# @INVARIANT: All database read/write operations must execute via the injected SQLAlchemy session boundary.
# @SEMANTICS: auth, repository, database, user, role
# @PURPOSE: Data access layer for authentication-related entities.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> sqlalchemy
# @RELATION: USES -> backend.src.models.auth
#
# @INVARIANT: All database operations must be performed within a session.
# [SECTION: IMPORTS]
from typing import List, Optional
from typing import Optional, List
from sqlalchemy.orm import Session
from ...models.auth import Permission, Role, User
from ...models.profile import UserDashboardPreference
from ..logger import belief_scope, logger
from ...models.auth import User, Role, Permission
from ..logger import belief_scope
# [/SECTION]
# [DEF:AuthRepository:Class]
# @PURPOSE: Encapsulates database operations for authentication-related entities.
# @RELATION: [DEPENDS_ON] ->[sqlalchemy.orm.Session]
# @PURPOSE: Encapsulates database operations for authentication.
class AuthRepository:
# [DEF:__init__:Function]
# @PURPOSE: Bind repository instance to an existing SQLAlchemy session.
# @PRE: db is an initialized sqlalchemy.orm.Session instance.
# @POST: self.db points to the provided session and is used by all repository methods.
# @SIDE_EFFECT: Stores session reference on repository instance state.
# @DATA_CONTRACT: Input[Session] -> Output[None]
# @PURPOSE: Initializes the repository with a database session.
# @PARAM: db (Session) - SQLAlchemy session.
def __init__(self, db: Session):
with belief_scope("AuthRepository.__init__"):
if not isinstance(db, Session):
logger.explore("Invalid session provided to AuthRepository", extra={"type": type(db)})
raise TypeError("db must be an instance of sqlalchemy.orm.Session")
logger.reason("Binding AuthRepository to database session")
self.db = db
logger.reflect("AuthRepository initialized")
self.db = db
# [/DEF:__init__:Function]
# [DEF:get_user_by_username:Function]
# @PURPOSE: Retrieve a user entity by unique username.
# @PRE: username is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching User entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[User]]
# @PURPOSE: Retrieves a user by their username.
# @PRE: username is a string.
# @POST: Returns User object if found, else None.
# @PARAM: username (str) - The username to search for.
# @RETURN: Optional[User] - The found user or None.
def get_user_by_username(self, username: str) -> Optional[User]:
with belief_scope("AuthRepository.get_user_by_username"):
if not username or not isinstance(username, str):
raise ValueError("username must be a non-empty string")
logger.reason(f"Querying user by username: {username}")
user = self.db.query(User).filter(User.username == username).first()
if user:
logger.reflect(f"User found: {username}")
else:
logger.explore(f"User not found: {username}")
return user
return self.db.query(User).filter(User.username == username).first()
# [/DEF:get_user_by_username:Function]
# [DEF:get_user_by_id:Function]
# @PURPOSE: Retrieve a user entity by identifier.
# @PRE: user_id is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching User entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[User]]
# @PURPOSE: Retrieves a user by their unique ID.
# @PRE: user_id is a valid UUID string.
# @POST: Returns User object if found, else None.
# @PARAM: user_id (str) - The user's unique identifier.
# @RETURN: Optional[User] - The found user or None.
def get_user_by_id(self, user_id: str) -> Optional[User]:
with belief_scope("AuthRepository.get_user_by_id"):
if not user_id or not isinstance(user_id, str):
raise ValueError("user_id must be a non-empty string")
logger.reason(f"Querying user by ID: {user_id}")
user = self.db.query(User).filter(User.id == user_id).first()
if user:
logger.reflect(f"User found by ID: {user_id}")
else:
logger.explore(f"User not found by ID: {user_id}")
return user
return self.db.query(User).filter(User.id == user_id).first()
# [/DEF:get_user_by_id:Function]
# [DEF:get_role_by_name:Function]
# @PURPOSE: Retrieve a role entity by role name.
# @PRE: name is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching Role entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[Role]]
# @PURPOSE: Retrieves a role by its name.
# @PRE: name is a string.
# @POST: Returns Role object if found, else None.
# @PARAM: name (str) - The role name to search for.
# @RETURN: Optional[Role] - The found role or None.
def get_role_by_name(self, name: str) -> Optional[Role]:
with belief_scope("AuthRepository.get_role_by_name"):
return self.db.query(Role).filter(Role.name == name).first()
# [/DEF:get_role_by_name:Function]
# [DEF:update_last_login:Function]
# @PURPOSE: Update last_login timestamp for the provided user entity.
# @PRE: user is a managed User instance and self.db is a valid open Session.
# @POST: user.last_login is set to current UTC timestamp and transaction is committed.
# @SIDE_EFFECT: Mutates user entity state and commits database transaction.
# @DATA_CONTRACT: Input[User] -> Output[None]
# @PURPOSE: Updates the last_login timestamp for a user.
# @PRE: user object is a valid User instance.
# @POST: User's last_login is updated in the database.
# @SIDE_EFFECT: Commits the transaction.
# @PARAM: user (User) - The user to update.
def update_last_login(self, user: User):
with belief_scope("AuthRepository.update_last_login"):
if not isinstance(user, User):
raise TypeError("user must be an instance of User")
from datetime import datetime
logger.reason(f"Updating last login for user: {user.username}")
user.last_login = datetime.utcnow()
self.db.add(user)
self.db.commit()
logger.reflect(f"Last login updated and committed for user: {user.username}")
# [/DEF:update_last_login:Function]
# [DEF:get_role_by_id:Function]
# @PURPOSE: Retrieve a role entity by identifier.
# @PRE: role_id is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching Role entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[Role]]
# @PURPOSE: Retrieves a role by its unique ID.
# @PRE: role_id is a string.
# @POST: Returns Role object if found, else None.
# @PARAM: role_id (str) - The role's unique identifier.
# @RETURN: Optional[Role] - The found role or None.
def get_role_by_id(self, role_id: str) -> Optional[Role]:
with belief_scope("AuthRepository.get_role_by_id"):
return self.db.query(Role).filter(Role.id == role_id).first()
# [/DEF:get_role_by_id:Function]
# [DEF:get_permission_by_id:Function]
# @PURPOSE: Retrieve a permission entity by identifier.
# @PRE: perm_id is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching Permission entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[Permission]]
# @PURPOSE: Retrieves a permission by its unique ID.
# @PRE: perm_id is a string.
# @POST: Returns Permission object if found, else None.
# @PARAM: perm_id (str) - The permission's unique identifier.
# @RETURN: Optional[Permission] - The found permission or None.
def get_permission_by_id(self, perm_id: str) -> Optional[Permission]:
with belief_scope("AuthRepository.get_permission_by_id"):
return self.db.query(Permission).filter(Permission.id == perm_id).first()
# [/DEF:get_permission_by_id:Function]
# [DEF:get_permission_by_resource_action:Function]
# @PURPOSE: Retrieve a permission entity by resource and action pair.
# @PRE: resource and action are non-empty str values; self.db is a valid open Session.
# @POST: Returns matching Permission entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str, str] -> Output[Optional[Permission]]
# @PURPOSE: Retrieves a permission by resource and action.
# @PRE: resource and action are strings.
# @POST: Returns Permission object if found, else None.
# @PARAM: resource (str) - The resource name.
# @PARAM: action (str) - The action name.
# @RETURN: Optional[Permission] - The found permission or None.
def get_permission_by_resource_action(self, resource: str, action: str) -> Optional[Permission]:
with belief_scope("AuthRepository.get_permission_by_resource_action"):
return self.db.query(Permission).filter(
@@ -149,54 +109,15 @@ class AuthRepository:
).first()
# [/DEF:get_permission_by_resource_action:Function]
# [DEF:get_user_dashboard_preference:Function]
# @PURPOSE: Retrieve dashboard preference entity owned by specified user.
# @PRE: user_id is a non-empty str and self.db is a valid open Session.
# @POST: Returns matching UserDashboardPreference entity when present, otherwise None.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[str] -> Output[Optional[UserDashboardPreference]]
def get_user_dashboard_preference(self, user_id: str) -> Optional[UserDashboardPreference]:
with belief_scope("AuthRepository.get_user_dashboard_preference"):
return (
self.db.query(UserDashboardPreference)
.filter(UserDashboardPreference.user_id == user_id)
.first()
)
# [/DEF:get_user_dashboard_preference:Function]
# [DEF:save_user_dashboard_preference:Function]
# @PURPOSE: Persist dashboard preference entity and return refreshed persistent row.
# @PRE: preference is a valid UserDashboardPreference entity and self.db is a valid open Session.
# @POST: preference is committed to DB, refreshed from DB state, and returned.
# @SIDE_EFFECT: Performs INSERT/UPDATE commit and refresh via active DB session.
# @DATA_CONTRACT: Input[UserDashboardPreference] -> Output[UserDashboardPreference]
def save_user_dashboard_preference(
self,
preference: UserDashboardPreference,
) -> UserDashboardPreference:
with belief_scope("AuthRepository.save_user_dashboard_preference"):
if not isinstance(preference, UserDashboardPreference):
raise TypeError("preference must be an instance of UserDashboardPreference")
logger.reason(f"Saving dashboard preference for user: {preference.user_id}")
self.db.add(preference)
self.db.commit()
self.db.refresh(preference)
logger.reflect(f"Dashboard preference saved and refreshed for user: {preference.user_id}")
return preference
# [/DEF:save_user_dashboard_preference:Function]
# [DEF:list_permissions:Function]
# @PURPOSE: List all permission entities available in storage.
# @PRE: self.db is a valid open Session.
# @POST: Returns list containing all Permission entities visible to the session.
# @SIDE_EFFECT: Executes read-only SELECT query through active DB session.
# @DATA_CONTRACT: Input[None] -> Output[List[Permission]]
# @PURPOSE: Lists all available permissions.
# @POST: Returns a list of all Permission objects.
# @RETURN: List[Permission] - List of permissions.
def list_permissions(self) -> List[Permission]:
with belief_scope("AuthRepository.list_permissions"):
return self.db.query(Permission).all()
# [/DEF:list_permissions:Function]
# [/DEF:AuthRepository:Class]
# [/DEF:backend.src.core.auth.repository:Module]

View File

@@ -1,17 +1,17 @@
# [DEF:ConfigManagerModule:Module]
#
# @TIER: CRITICAL
# @SEMANTICS: config, manager, persistence, migration, postgresql
# @PURPOSE: Manages application configuration persistence in DB with one-time migration from legacy JSON.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[ConfigModels]
# @RELATION: [DEPENDS_ON] ->[SessionLocal]
# @RELATION: [DEPENDS_ON] ->[AppConfigRecord]
# @RELATION: [CALLS] ->[logger]
# @RELATION: [CALLS] ->[configure_logger]
# @RELATION: [BINDS_TO] ->[ConfigManager]
# @INVARIANT: Configuration must always be representable by AppConfig and persisted under global record id.
# @TIER: STANDARD
# @SEMANTICS: config, manager, persistence, postgresql
# @PURPOSE: Manages application configuration persisted in database with one-time migration from JSON.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> ConfigModels
# @RELATION: DEPENDS_ON -> AppConfigRecord
# @RELATION: CALLS -> logger
#
# @INVARIANT: Configuration must always be valid according to AppConfig model.
# @PUBLIC_API: ConfigManager
# [SECTION: IMPORTS]
import json
import os
from pathlib import Path
@@ -23,58 +23,47 @@ from .config_models import AppConfig, Environment, GlobalSettings, StorageConfig
from .database import SessionLocal
from ..models.config import AppConfigRecord
from .logger import logger, configure_logger, belief_scope
# [/SECTION]
# [DEF:ConfigManager:Class]
# @TIER: CRITICAL
# @PURPOSE: Handles application configuration load, validation, mutation, and persistence lifecycle.
# @TIER: STANDARD
# @PURPOSE: A class to handle application configuration persistence and management.
class ConfigManager:
# [DEF:__init__:Function]
# @PURPOSE: Initialize manager state from persisted or migrated configuration.
# @PRE: config_path is a non-empty string path.
# @POST: self.config is initialized as AppConfig and logger is configured.
# @SIDE_EFFECT: Reads config sources and updates logging configuration.
# @DATA_CONTRACT: Input(str config_path) -> Output(None; self.config: AppConfig)
# @TIER: STANDARD
# @PURPOSE: Initializes the ConfigManager.
# @PRE: isinstance(config_path, str) and len(config_path) > 0
# @POST: self.config is an instance of AppConfig
# @PARAM: config_path (str) - Path to legacy JSON config (used only for initial migration fallback).
def __init__(self, config_path: str = "config.json"):
with belief_scope("ConfigManager.__init__"):
if not isinstance(config_path, str) or not config_path:
logger.explore("Invalid config_path provided", extra={"path": config_path})
raise ValueError("config_path must be a non-empty string")
with belief_scope("__init__"):
assert isinstance(config_path, str) and config_path, "config_path must be a non-empty string"
logger.reason(f"Initializing ConfigManager with legacy path: {config_path}")
logger.info(f"[ConfigManager][Entry] Initializing with legacy path {config_path}")
self.config_path = Path(config_path)
self.config: AppConfig = self._load_config()
configure_logger(self.config.settings.logging)
if not isinstance(self.config, AppConfig):
logger.explore("Config loading resulted in invalid type", extra={"type": type(self.config)})
raise TypeError("self.config must be an instance of AppConfig")
assert isinstance(self.config, AppConfig), "self.config must be an instance of AppConfig"
logger.reflect("ConfigManager initialization complete")
logger.info("[ConfigManager][Exit] Initialized")
# [/DEF:__init__:Function]
# [DEF:_default_config:Function]
# @PURPOSE: Build default application configuration fallback.
# @PRE: None.
# @POST: Returns valid AppConfig with empty environments and default storage settings.
# @SIDE_EFFECT: None.
# @DATA_CONTRACT: Input(None) -> Output(AppConfig)
# @PURPOSE: Returns default application configuration.
# @RETURN: AppConfig - Default configuration.
def _default_config(self) -> AppConfig:
with belief_scope("_default_config"):
return AppConfig(
environments=[],
settings=GlobalSettings(storage=StorageConfig()),
)
return AppConfig(
environments=[],
settings=GlobalSettings(storage=StorageConfig()),
)
# [/DEF:_default_config:Function]
# [DEF:_load_from_legacy_file:Function]
# @PURPOSE: Load legacy JSON configuration for migration fallback path.
# @PRE: self.config_path is initialized.
# @POST: Returns AppConfig from file payload or safe default.
# @SIDE_EFFECT: Filesystem read and error logging.
# @DATA_CONTRACT: Input(Path self.config_path) -> Output(AppConfig)
# @PURPOSE: Loads legacy configuration from config.json for migration fallback.
# @RETURN: AppConfig - Loaded or default configuration.
def _load_from_legacy_file(self) -> AppConfig:
with belief_scope("_load_from_legacy_file"):
if not self.config_path.exists():
@@ -92,55 +81,47 @@ class ConfigManager:
# [/DEF:_load_from_legacy_file:Function]
# [DEF:_get_record:Function]
# @PURPOSE: Resolve global configuration record from DB.
# @PRE: session is an active SQLAlchemy Session.
# @POST: Returns record when present, otherwise None.
# @SIDE_EFFECT: Database read query.
# @DATA_CONTRACT: Input(Session) -> Output(Optional[AppConfigRecord])
# @PURPOSE: Loads config record from DB.
# @PARAM: session (Session) - DB session.
# @RETURN: Optional[AppConfigRecord] - Existing record or None.
def _get_record(self, session: Session) -> Optional[AppConfigRecord]:
with belief_scope("_get_record"):
return session.query(AppConfigRecord).filter(AppConfigRecord.id == "global").first()
return session.query(AppConfigRecord).filter(AppConfigRecord.id == "global").first()
# [/DEF:_get_record:Function]
# [DEF:_load_config:Function]
# @PURPOSE: Load configuration from DB or perform one-time migration from legacy JSON.
# @PRE: SessionLocal factory is available and AppConfigRecord schema is accessible.
# @POST: Returns valid AppConfig and closes opened DB session.
# @SIDE_EFFECT: Database read/write, possible migration write, logging.
# @DATA_CONTRACT: Input(None) -> Output(AppConfig)
# @PURPOSE: Loads the configuration from DB or performs one-time migration from JSON file.
# @PRE: DB session factory is available.
# @POST: isinstance(return, AppConfig)
# @RETURN: AppConfig - Loaded configuration.
def _load_config(self) -> AppConfig:
with belief_scope("ConfigManager._load_config"):
with belief_scope("_load_config"):
session: Session = SessionLocal()
try:
record = self._get_record(session)
if record and record.payload:
logger.reason("Configuration found in database")
config = AppConfig(**record.payload)
logger.reflect("Database configuration validated")
return config
logger.info("[_load_config][Coherence:OK] Configuration loaded from database")
return AppConfig(**record.payload)
logger.reason("No database config found, initiating legacy migration")
logger.info("[_load_config][Action] No database config found, migrating legacy config")
config = self._load_from_legacy_file()
self._save_config_to_db(config, session=session)
logger.reflect("Legacy configuration migrated to database")
return config
except Exception as e:
logger.explore(f"Error loading config from DB: {e}")
logger.error(f"[_load_config][Coherence:Failed] Error loading config from DB: {e}")
return self._default_config()
finally:
session.close()
# [/DEF:_load_config:Function]
# [DEF:_save_config_to_db:Function]
# @PURPOSE: Persist provided AppConfig into the global DB configuration record.
# @PRE: config is AppConfig; session is either None or an active Session.
# @POST: Global DB record payload equals config.model_dump() when commit succeeds.
# @SIDE_EFFECT: Database insert/update, commit/rollback, logging.
# @DATA_CONTRACT: Input(AppConfig, Optional[Session]) -> Output(None)
# @PURPOSE: Saves the provided configuration object to DB.
# @PRE: isinstance(config, AppConfig)
# @POST: Configuration saved to database.
# @PARAM: config (AppConfig) - The configuration to save.
# @PARAM: session (Optional[Session]) - Existing DB session for transactional reuse.
def _save_config_to_db(self, config: AppConfig, session: Optional[Session] = None):
with belief_scope("ConfigManager._save_config_to_db"):
if not isinstance(config, AppConfig):
raise TypeError("config must be an instance of AppConfig")
with belief_scope("_save_config_to_db"):
assert isinstance(config, AppConfig), "config must be an instance of AppConfig"
owns_session = session is None
db = session or SessionLocal()
@@ -148,17 +129,15 @@ class ConfigManager:
record = self._get_record(db)
payload = config.model_dump()
if record is None:
logger.reason("Creating new global configuration record")
record = AppConfigRecord(id="global", payload=payload)
db.add(record)
else:
logger.reason("Updating existing global configuration record")
record.payload = payload
db.commit()
logger.reflect("Configuration successfully committed to database")
logger.info("[_save_config_to_db][Action] Configuration saved to database")
except Exception as e:
db.rollback()
logger.explore(f"Failed to save configuration: {e}")
logger.error(f"[_save_config_to_db][Coherence:Failed] Failed to save: {e}")
raise
finally:
if owns_session:
@@ -166,51 +145,42 @@ class ConfigManager:
# [/DEF:_save_config_to_db:Function]
# [DEF:save:Function]
# @PURPOSE: Persist current in-memory configuration state.
# @PRE: self.config is initialized.
# @POST: Current self.config is written to DB global record.
# @SIDE_EFFECT: Database write and logging via delegated persistence call.
# @DATA_CONTRACT: Input(None; self.config: AppConfig) -> Output(None)
# @PURPOSE: Saves the current configuration state to DB.
# @PRE: self.config is set.
# @POST: self._save_config_to_db called.
def save(self):
with belief_scope("save"):
self._save_config_to_db(self.config)
# [/DEF:save:Function]
# [DEF:get_config:Function]
# @PURPOSE: Return current in-memory configuration snapshot.
# @PRE: self.config is initialized.
# @POST: Returns AppConfig reference stored in manager.
# @SIDE_EFFECT: None.
# @DATA_CONTRACT: Input(None) -> Output(AppConfig)
# @PURPOSE: Returns the current configuration.
# @RETURN: AppConfig - The current configuration.
def get_config(self) -> AppConfig:
with belief_scope("get_config"):
return self.config
# [/DEF:get_config:Function]
# [DEF:update_global_settings:Function]
# @PURPOSE: Replace global settings and persist the resulting configuration.
# @PRE: settings is GlobalSettings.
# @POST: self.config.settings equals provided settings and DB state is updated.
# @SIDE_EFFECT: Mutates self.config, DB write, logger reconfiguration, logging.
# @DATA_CONTRACT: Input(GlobalSettings) -> Output(None)
# @PURPOSE: Updates the global settings and persists the change.
# @PRE: isinstance(settings, GlobalSettings)
# @POST: self.config.settings updated and saved.
# @PARAM: settings (GlobalSettings) - The new global settings.
def update_global_settings(self, settings: GlobalSettings):
with belief_scope("ConfigManager.update_global_settings"):
if not isinstance(settings, GlobalSettings):
raise TypeError("settings must be an instance of GlobalSettings")
logger.reason("Updating global settings and persisting")
with belief_scope("update_global_settings"):
logger.info("[update_global_settings][Entry] Updating settings")
assert isinstance(settings, GlobalSettings), "settings must be an instance of GlobalSettings"
self.config.settings = settings
self.save()
configure_logger(settings.logging)
logger.reflect("Global settings updated and logger reconfigured")
logger.info("[update_global_settings][Exit] Settings updated")
# [/DEF:update_global_settings:Function]
# [DEF:validate_path:Function]
# @PURPOSE: Validate that path exists and is writable, creating it when absent.
# @PRE: path is a string path candidate.
# @POST: Returns (True, msg) for writable path, else (False, reason).
# @SIDE_EFFECT: Filesystem directory creation attempt and OS permission checks.
# @DATA_CONTRACT: Input(str path) -> Output(tuple[bool, str])
# @PURPOSE: Validates if a path exists and is writable.
# @PARAM: path (str) - The path to validate.
# @RETURN: tuple (bool, str) - (is_valid, message)
def validate_path(self, path: str) -> tuple[bool, str]:
with belief_scope("validate_path"):
p = os.path.abspath(path)
@@ -227,33 +197,25 @@ class ConfigManager:
# [/DEF:validate_path:Function]
# [DEF:get_environments:Function]
# @PURPOSE: Return all configured environments.
# @PRE: self.config is initialized.
# @POST: Returns list of Environment models from current configuration.
# @SIDE_EFFECT: None.
# @DATA_CONTRACT: Input(None) -> Output(List[Environment])
# @PURPOSE: Returns the list of configured environments.
# @RETURN: List[Environment] - List of environments.
def get_environments(self) -> List[Environment]:
with belief_scope("get_environments"):
return self.config.environments
# [/DEF:get_environments:Function]
# [DEF:has_environments:Function]
# @PURPOSE: Check whether at least one environment exists in configuration.
# @PRE: self.config is initialized.
# @POST: Returns True iff environment list length is greater than zero.
# @SIDE_EFFECT: None.
# @DATA_CONTRACT: Input(None) -> Output(bool)
# @PURPOSE: Checks if at least one environment is configured.
# @RETURN: bool - True if at least one environment exists.
def has_environments(self) -> bool:
with belief_scope("has_environments"):
return len(self.config.environments) > 0
# [/DEF:has_environments:Function]
# [DEF:get_environment:Function]
# @PURPOSE: Resolve a configured environment by identifier.
# @PRE: env_id is string identifier.
# @POST: Returns matching Environment when found; otherwise None.
# @SIDE_EFFECT: None.
# @DATA_CONTRACT: Input(str env_id) -> Output(Optional[Environment])
# @PURPOSE: Returns a single environment by ID.
# @PARAM: env_id (str) - The ID of the environment to retrieve.
# @RETURN: Optional[Environment] - The environment with the given ID, or None.
def get_environment(self, env_id: str) -> Optional[Environment]:
with belief_scope("get_environment"):
for env in self.config.environments:
@@ -263,72 +225,60 @@ class ConfigManager:
# [/DEF:get_environment:Function]
# [DEF:add_environment:Function]
# @PURPOSE: Upsert environment by id into configuration and persist.
# @PRE: env is Environment.
# @POST: Configuration contains provided env id with new payload persisted.
# @SIDE_EFFECT: Mutates environment list, DB write, logging.
# @DATA_CONTRACT: Input(Environment) -> Output(None)
# @PURPOSE: Adds a new environment to the configuration.
# @PARAM: env (Environment) - The environment to add.
def add_environment(self, env: Environment):
with belief_scope("ConfigManager.add_environment"):
if not isinstance(env, Environment):
raise TypeError("env must be an instance of Environment")
with belief_scope("add_environment"):
logger.info(f"[add_environment][Entry] Adding environment {env.id}")
assert isinstance(env, Environment), "env must be an instance of Environment"
logger.reason(f"Adding/Updating environment: {env.id}")
self.config.environments = [e for e in self.config.environments if e.id != env.id]
self.config.environments.append(env)
self.save()
logger.reflect(f"Environment {env.id} persisted")
logger.info("[add_environment][Exit] Environment added")
# [/DEF:add_environment:Function]
# [DEF:update_environment:Function]
# @PURPOSE: Update existing environment by id and preserve masked password placeholder behavior.
# @PRE: env_id is non-empty string and updated_env is Environment.
# @POST: Returns True and persists update when target exists; else returns False.
# @SIDE_EFFECT: May mutate environment list, DB write, logging.
# @DATA_CONTRACT: Input(str env_id, Environment updated_env) -> Output(bool)
# @PURPOSE: Updates an existing environment.
# @PARAM: env_id (str) - The ID of the environment to update.
# @PARAM: updated_env (Environment) - The updated environment data.
# @RETURN: bool - True if updated, False otherwise.
def update_environment(self, env_id: str, updated_env: Environment) -> bool:
with belief_scope("ConfigManager.update_environment"):
if not env_id or not isinstance(env_id, str):
raise ValueError("env_id must be a non-empty string")
if not isinstance(updated_env, Environment):
raise TypeError("updated_env must be an instance of Environment")
with belief_scope("update_environment"):
logger.info(f"[update_environment][Entry] Updating {env_id}")
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
assert isinstance(updated_env, Environment), "updated_env must be an instance of Environment"
logger.reason(f"Attempting to update environment: {env_id}")
for i, env in enumerate(self.config.environments):
if env.id == env_id:
if updated_env.password == "********":
logger.reason("Preserving existing password for masked update")
updated_env.password = env.password
self.config.environments[i] = updated_env
self.save()
logger.reflect(f"Environment {env_id} updated and saved")
logger.info(f"[update_environment][Coherence:OK] Updated {env_id}")
return True
logger.explore(f"Environment {env_id} not found for update")
logger.warning(f"[update_environment][Coherence:Failed] Environment {env_id} not found")
return False
# [/DEF:update_environment:Function]
# [DEF:delete_environment:Function]
# @PURPOSE: Delete environment by id and persist when deletion occurs.
# @PRE: env_id is non-empty string.
# @POST: Environment is removed when present; otherwise configuration is unchanged.
# @SIDE_EFFECT: May mutate environment list, conditional DB write, logging.
# @DATA_CONTRACT: Input(str env_id) -> Output(None)
# @PURPOSE: Deletes an environment by ID.
# @PARAM: env_id (str) - The ID of the environment to delete.
def delete_environment(self, env_id: str):
with belief_scope("ConfigManager.delete_environment"):
if not env_id or not isinstance(env_id, str):
raise ValueError("env_id must be a non-empty string")
with belief_scope("delete_environment"):
logger.info(f"[delete_environment][Entry] Deleting {env_id}")
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
logger.reason(f"Attempting to delete environment: {env_id}")
original_count = len(self.config.environments)
self.config.environments = [e for e in self.config.environments if e.id != env_id]
if len(self.config.environments) < original_count:
self.save()
logger.reflect(f"Environment {env_id} deleted and configuration saved")
logger.info(f"[delete_environment][Action] Deleted {env_id}")
else:
logger.explore(f"Environment {env_id} not found for deletion")
logger.warning(f"[delete_environment][Coherence:Failed] Environment {env_id} not found")
# [/DEF:delete_environment:Function]

View File

@@ -24,19 +24,18 @@ class Schedule(BaseModel):
# [DEF:Environment:DataClass]
# @PURPOSE: Represents a Superset environment configuration.
class Environment(BaseModel):
id: str
name: str
url: str
username: str
password: str # Will be masked in UI
stage: str = Field(default="DEV", pattern="^(DEV|PREPROD|PROD)$")
verify_ssl: bool = True
timeout: int = 30
is_default: bool = False
is_production: bool = False
backup_schedule: Schedule = Field(default_factory=Schedule)
# [/DEF:Environment:DataClass]
class Environment(BaseModel):
id: str
name: str
url: str
username: str
password: str # Will be masked in UI
verify_ssl: bool = True
timeout: int = 30
is_default: bool = False
is_production: bool = False
backup_schedule: Schedule = Field(default_factory=Schedule)
# [/DEF:Environment:DataClass]
# [DEF:LoggingConfig:DataClass]
# @PURPOSE: Defines the configuration for the application's logging system.
@@ -49,18 +48,10 @@ class LoggingConfig(BaseModel):
enable_belief_state: bool = True
# [/DEF:LoggingConfig:DataClass]
# [DEF:CleanReleaseConfig:DataClass]
# @PURPOSE: Configuration for clean release compliance subsystem.
class CleanReleaseConfig(BaseModel):
active_policy_id: Optional[str] = None
active_registry_id: Optional[str] = None
# [/DEF:CleanReleaseConfig:DataClass]
# [DEF:GlobalSettings:DataClass]
# @PURPOSE: Represents global application settings.
class GlobalSettings(BaseModel):
storage: StorageConfig = Field(default_factory=StorageConfig)
clean_release: CleanReleaseConfig = Field(default_factory=CleanReleaseConfig)
default_environment_id: Optional[str] = None
logging: LoggingConfig = Field(default_factory=LoggingConfig)
connections: List[dict] = []

View File

@@ -11,7 +11,7 @@
# @INVARIANT: A single engine instance is used for the entire application.
# [SECTION: IMPORTS]
from sqlalchemy import create_engine, inspect, text
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from ..models.mapping import Base
# Import models to ensure they're registered with Base
@@ -20,9 +20,7 @@ from ..models import auth as _auth_models # noqa: F401
from ..models import config as _config_models # noqa: F401
from ..models import llm as _llm_models # noqa: F401
from ..models import assistant as _assistant_models # noqa: F401
from ..models import profile as _profile_models # noqa: F401
from ..models import clean_release as _clean_release_models # noqa: F401
from .logger import belief_scope, logger
from .logger import belief_scope
from .auth.config import auth_config
import os
from pathlib import Path
@@ -96,191 +94,6 @@ TasksSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=tasks_e
AuthSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=auth_engine)
# [/DEF:AuthSessionLocal:Class]
# [DEF:_ensure_user_dashboard_preferences_columns:Function]
# @PURPOSE: Applies additive schema upgrades for user_dashboard_preferences table.
# @PRE: bind_engine points to application database where profile table is stored.
# @POST: Missing columns are added without data loss.
def _ensure_user_dashboard_preferences_columns(bind_engine):
with belief_scope("_ensure_user_dashboard_preferences_columns"):
table_name = "user_dashboard_preferences"
inspector = inspect(bind_engine)
if table_name not in inspector.get_table_names():
return
existing_columns = {
str(column.get("name") or "").strip()
for column in inspector.get_columns(table_name)
}
alter_statements = []
if "git_username" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences ADD COLUMN git_username VARCHAR"
)
if "git_email" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences ADD COLUMN git_email VARCHAR"
)
if "git_personal_access_token_encrypted" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences "
"ADD COLUMN git_personal_access_token_encrypted VARCHAR"
)
if "start_page" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences "
"ADD COLUMN start_page VARCHAR NOT NULL DEFAULT 'dashboards'"
)
if "auto_open_task_drawer" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences "
"ADD COLUMN auto_open_task_drawer BOOLEAN NOT NULL DEFAULT TRUE"
)
if "dashboards_table_density" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences "
"ADD COLUMN dashboards_table_density VARCHAR NOT NULL DEFAULT 'comfortable'"
)
if "show_only_slug_dashboards" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences "
"ADD COLUMN show_only_slug_dashboards BOOLEAN NOT NULL DEFAULT TRUE"
)
if not alter_statements:
return
try:
with bind_engine.begin() as connection:
for statement in alter_statements:
connection.execute(text(statement))
except Exception as migration_error:
logger.warning(
"[database][EXPLORE] Profile preference additive migration failed: %s",
migration_error,
)
# [/DEF:_ensure_user_dashboard_preferences_columns:Function]
# [DEF:_ensure_user_dashboard_preferences_health_columns:Function]
# @PURPOSE: Applies additive schema upgrades for user_dashboard_preferences table (health fields).
def _ensure_user_dashboard_preferences_health_columns(bind_engine):
with belief_scope("_ensure_user_dashboard_preferences_health_columns"):
table_name = "user_dashboard_preferences"
inspector = inspect(bind_engine)
if table_name not in inspector.get_table_names():
return
existing_columns = {
str(column.get("name") or "").strip()
for column in inspector.get_columns(table_name)
}
alter_statements = []
if "telegram_id" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences ADD COLUMN telegram_id VARCHAR"
)
if "email_address" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences ADD COLUMN email_address VARCHAR"
)
if "notify_on_fail" not in existing_columns:
alter_statements.append(
"ALTER TABLE user_dashboard_preferences ADD COLUMN notify_on_fail BOOLEAN NOT NULL DEFAULT TRUE"
)
if not alter_statements:
return
try:
with bind_engine.begin() as connection:
for statement in alter_statements:
connection.execute(text(statement))
except Exception as migration_error:
logger.warning(
"[database][EXPLORE] Profile health preference additive migration failed: %s",
migration_error,
)
# [/DEF:_ensure_user_dashboard_preferences_health_columns:Function]
# [DEF:_ensure_llm_validation_results_columns:Function]
# @PURPOSE: Applies additive schema upgrades for llm_validation_results table.
def _ensure_llm_validation_results_columns(bind_engine):
with belief_scope("_ensure_llm_validation_results_columns"):
table_name = "llm_validation_results"
inspector = inspect(bind_engine)
if table_name not in inspector.get_table_names():
return
existing_columns = {
str(column.get("name") or "").strip()
for column in inspector.get_columns(table_name)
}
alter_statements = []
if "task_id" not in existing_columns:
alter_statements.append(
"ALTER TABLE llm_validation_results ADD COLUMN task_id VARCHAR"
)
if "environment_id" not in existing_columns:
alter_statements.append(
"ALTER TABLE llm_validation_results ADD COLUMN environment_id VARCHAR"
)
if not alter_statements:
return
try:
with bind_engine.begin() as connection:
for statement in alter_statements:
connection.execute(text(statement))
except Exception as migration_error:
logger.warning(
"[database][EXPLORE] ValidationRecord additive migration failed: %s",
migration_error,
)
# [/DEF:_ensure_llm_validation_results_columns:Function]
# [DEF:_ensure_git_server_configs_columns:Function]
# @PURPOSE: Applies additive schema upgrades for git_server_configs table.
# @PRE: bind_engine points to application database.
# @POST: Missing columns are added without data loss.
def _ensure_git_server_configs_columns(bind_engine):
with belief_scope("_ensure_git_server_configs_columns"):
table_name = "git_server_configs"
inspector = inspect(bind_engine)
if table_name not in inspector.get_table_names():
return
existing_columns = {
str(column.get("name") or "").strip()
for column in inspector.get_columns(table_name)
}
alter_statements = []
if "default_branch" not in existing_columns:
alter_statements.append(
"ALTER TABLE git_server_configs ADD COLUMN default_branch VARCHAR NOT NULL DEFAULT 'main'"
)
if not alter_statements:
return
try:
with bind_engine.begin() as connection:
for statement in alter_statements:
connection.execute(text(statement))
except Exception as migration_error:
logger.warning(
"[database][EXPLORE] GitServerConfig preference additive migration failed: %s",
migration_error,
)
# [/DEF:_ensure_git_server_configs_columns:Function]
# [DEF:init_db:Function]
# @PURPOSE: Initializes the database by creating all tables.
# @PRE: engine, tasks_engine and auth_engine are initialized.
@@ -291,10 +104,6 @@ def init_db():
Base.metadata.create_all(bind=engine)
Base.metadata.create_all(bind=tasks_engine)
Base.metadata.create_all(bind=auth_engine)
_ensure_user_dashboard_preferences_columns(engine)
_ensure_llm_validation_results_columns(engine)
_ensure_user_dashboard_preferences_health_columns(engine)
_ensure_git_server_configs_columns(engine)
# [/DEF:init_db:Function]
# [DEF:get_db:Function]

View File

@@ -225,7 +225,7 @@ def test_enable_belief_state_flag(caplog):
assert not any("[DisabledFunction][Exit]" in msg for msg in log_messages), "Exit should not be logged when disabled"
# Coherence:OK should still be logged (internal tracking)
assert any("[DisabledFunction][COHERENCE:OK]" in msg for msg in log_messages), "Coherence should still be logged"
# [/DEF:test_enable_belief_state_flag:Function]
# [DEF:test_belief_scope_missing_anchor:Function]

View File

@@ -23,21 +23,6 @@ from src.core.logger import logger, belief_scope
# [DEF:IdMappingService:Class]
# @TIER: CRITICAL
# @PURPOSE: Service handling the cataloging and retrieval of remote Superset Integer IDs.
#
# @TEST_CONTRACT: IdMappingServiceModel ->
# {
# required_fields: {db_session: Session},
# invariants: [
# "sync_environment correctly creates or updates ResourceMapping records",
# "get_remote_id returns an integer or None",
# "get_remote_ids_batch returns a dictionary of valid UUIDs to integers"
# ]
# }
# @TEST_FIXTURE: valid_mapping_service -> {"db_session": "MockSession()"}
# @TEST_EDGE: sync_api_failure -> handles exception gracefully
# @TEST_EDGE: get_remote_id_not_found -> returns None
# @TEST_EDGE: get_batch_empty_list -> returns empty dict
# @TEST_INVARIANT: resilient_fetching -> verifies: [sync_api_failure]
class IdMappingService:
# [DEF:__init__:Function]

View File

@@ -1,12 +0,0 @@
# [DEF:backend.src.core.migration.__init__:Module]
# @TIER: TRIVIAL
# @SEMANTICS: migration, package, exports
# @PURPOSE: Namespace package for migration pre-flight orchestration components.
# @LAYER: Core
from .dry_run_orchestrator import MigrationDryRunService
from .archive_parser import MigrationArchiveParser
__all__ = ["MigrationDryRunService", "MigrationArchiveParser"]
# [/DEF:backend.src.core.migration.__init__:Module]

View File

@@ -1,139 +0,0 @@
# [DEF:backend.src.core.migration.archive_parser:Module]
# @TIER: STANDARD
# @SEMANTICS: migration, zip, parser, yaml, metadata
# @PURPOSE: Parse Superset export ZIP archives into normalized object catalogs for diffing.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> backend.src.core.logger
# @INVARIANT: Parsing is read-only and never mutates archive files.
import json
import tempfile
import zipfile
from pathlib import Path
from typing import Any, Dict, List, Optional
import yaml
from ..logger import logger, belief_scope
# [DEF:MigrationArchiveParser:Class]
# @PURPOSE: Extract normalized dashboards/charts/datasets metadata from ZIP archives.
class MigrationArchiveParser:
# [DEF:extract_objects_from_zip:Function]
# @PURPOSE: Extract object catalogs from Superset archive.
# @PRE: zip_path points to a valid readable ZIP.
# @POST: Returns object lists grouped by resource type.
# @RETURN: Dict[str, List[Dict[str, Any]]]
def extract_objects_from_zip(self, zip_path: str) -> Dict[str, List[Dict[str, Any]]]:
with belief_scope("MigrationArchiveParser.extract_objects_from_zip"):
result: Dict[str, List[Dict[str, Any]]] = {
"dashboards": [],
"charts": [],
"datasets": [],
}
with tempfile.TemporaryDirectory() as temp_dir_str:
temp_dir = Path(temp_dir_str)
with zipfile.ZipFile(zip_path, "r") as zip_file:
zip_file.extractall(temp_dir)
result["dashboards"] = self._collect_yaml_objects(temp_dir, "dashboards")
result["charts"] = self._collect_yaml_objects(temp_dir, "charts")
result["datasets"] = self._collect_yaml_objects(temp_dir, "datasets")
return result
# [/DEF:extract_objects_from_zip:Function]
# [DEF:_collect_yaml_objects:Function]
# @PURPOSE: Read and normalize YAML manifests for one object type.
# @PRE: object_type is one of dashboards/charts/datasets.
# @POST: Returns only valid normalized objects.
def _collect_yaml_objects(self, root_dir: Path, object_type: str) -> List[Dict[str, Any]]:
with belief_scope("MigrationArchiveParser._collect_yaml_objects"):
files = list(root_dir.glob(f"**/{object_type}/**/*.yaml")) + list(root_dir.glob(f"**/{object_type}/*.yaml"))
objects: List[Dict[str, Any]] = []
for file_path in set(files):
try:
with open(file_path, "r") as file_obj:
payload = yaml.safe_load(file_obj) or {}
normalized = self._normalize_object_payload(payload, object_type)
if normalized:
objects.append(normalized)
except Exception as exc:
logger.reflect(
"[MigrationArchiveParser._collect_yaml_objects][REFLECT] skip_invalid_yaml path=%s error=%s",
file_path,
exc,
)
return objects
# [/DEF:_collect_yaml_objects:Function]
# [DEF:_normalize_object_payload:Function]
# @PURPOSE: Convert raw YAML payload to stable diff signature shape.
# @PRE: payload is parsed YAML mapping.
# @POST: Returns normalized descriptor with `uuid`, `title`, and `signature`.
def _normalize_object_payload(self, payload: Dict[str, Any], object_type: str) -> Optional[Dict[str, Any]]:
with belief_scope("MigrationArchiveParser._normalize_object_payload"):
if not isinstance(payload, dict):
return None
uuid = payload.get("uuid")
if not uuid:
return None
if object_type == "dashboards":
title = payload.get("dashboard_title") or payload.get("title")
signature = {
"title": title,
"slug": payload.get("slug"),
"position_json": payload.get("position_json"),
"json_metadata": payload.get("json_metadata"),
"description": payload.get("description"),
"owners": payload.get("owners"),
}
return {
"uuid": str(uuid),
"title": title or f"Dashboard {uuid}",
"signature": json.dumps(signature, sort_keys=True, default=str),
"owners": payload.get("owners") or [],
}
if object_type == "charts":
title = payload.get("slice_name") or payload.get("name")
signature = {
"title": title,
"viz_type": payload.get("viz_type"),
"params": payload.get("params"),
"query_context": payload.get("query_context"),
"datasource_uuid": payload.get("datasource_uuid"),
"dataset_uuid": payload.get("dataset_uuid"),
}
return {
"uuid": str(uuid),
"title": title or f"Chart {uuid}",
"signature": json.dumps(signature, sort_keys=True, default=str),
"dataset_uuid": payload.get("datasource_uuid") or payload.get("dataset_uuid"),
}
if object_type == "datasets":
title = payload.get("table_name") or payload.get("name")
signature = {
"title": title,
"schema": payload.get("schema"),
"database_uuid": payload.get("database_uuid"),
"sql": payload.get("sql"),
"columns": payload.get("columns"),
"metrics": payload.get("metrics"),
}
return {
"uuid": str(uuid),
"title": title or f"Dataset {uuid}",
"signature": json.dumps(signature, sort_keys=True, default=str),
"database_uuid": payload.get("database_uuid"),
}
return None
# [/DEF:_normalize_object_payload:Function]
# [/DEF:MigrationArchiveParser:Class]
# [/DEF:backend.src.core.migration.archive_parser:Module]

View File

@@ -1,235 +0,0 @@
# [DEF:backend.src.core.migration.dry_run_orchestrator:Module]
# @TIER: STANDARD
# @SEMANTICS: migration, dry_run, diff, risk, superset
# @PURPOSE: Compute pre-flight migration diff and risk scoring without apply.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> backend.src.core.superset_client
# @RELATION: DEPENDS_ON -> backend.src.core.migration_engine
# @RELATION: DEPENDS_ON -> backend.src.core.migration.archive_parser
# @RELATION: DEPENDS_ON -> backend.src.core.migration.risk_assessor
# @INVARIANT: Dry run is informative only and must not mutate target environment.
from datetime import datetime, timezone
import json
from typing import Any, Dict, List
from sqlalchemy.orm import Session
from ...models.dashboard import DashboardSelection
from ...models.mapping import DatabaseMapping
from ..logger import logger, belief_scope
from .archive_parser import MigrationArchiveParser
from .risk_assessor import build_risks, score_risks
from ..migration_engine import MigrationEngine
from ..superset_client import SupersetClient
from ..utils.fileio import create_temp_file
# [DEF:MigrationDryRunService:Class]
# @PURPOSE: Build deterministic diff/risk payload for migration pre-flight.
class MigrationDryRunService:
# [DEF:__init__:Function]
# @PURPOSE: Wire parser dependency for archive object extraction.
# @PRE: parser can be omitted to use default implementation.
# @POST: Service is ready to calculate dry-run payload.
def __init__(self, parser: MigrationArchiveParser | None = None):
self.parser = parser or MigrationArchiveParser()
# [/DEF:__init__:Function]
# [DEF:run:Function]
# @PURPOSE: Execute full dry-run computation for selected dashboards.
# @PRE: source/target clients are authenticated and selection validated by caller.
# @POST: Returns JSON-serializable pre-flight payload with summary, diff and risk.
# @SIDE_EFFECT: Reads source export archives and target metadata via network.
def run(
self,
selection: DashboardSelection,
source_client: SupersetClient,
target_client: SupersetClient,
db: Session,
) -> Dict[str, Any]:
with belief_scope("MigrationDryRunService.run"):
logger.explore("[MigrationDryRunService.run][EXPLORE] starting dry-run pipeline")
engine = MigrationEngine()
db_mapping = self._load_db_mapping(db, selection) if selection.replace_db_config else {}
transformed = {"dashboards": {}, "charts": {}, "datasets": {}}
dashboards_preview = source_client.get_dashboards_summary()
selected_preview = {
item["id"]: item
for item in dashboards_preview
if item.get("id") in selection.selected_ids
}
for dashboard_id in selection.selected_ids:
exported_content, _ = source_client.export_dashboard(int(dashboard_id))
with create_temp_file(content=exported_content, suffix=".zip") as source_zip:
with create_temp_file(suffix=".zip") as transformed_zip:
success = engine.transform_zip(
str(source_zip),
str(transformed_zip),
db_mapping,
strip_databases=False,
target_env_id=selection.target_env_id,
fix_cross_filters=selection.fix_cross_filters,
)
if not success:
raise ValueError(f"Failed to transform export archive for dashboard {dashboard_id}")
extracted = self.parser.extract_objects_from_zip(str(transformed_zip))
self._accumulate_objects(transformed, extracted)
source_objects = {key: list(value.values()) for key, value in transformed.items()}
target_objects = self._build_target_signatures(target_client)
diff = {
"dashboards": self._build_object_diff(source_objects["dashboards"], target_objects["dashboards"]),
"charts": self._build_object_diff(source_objects["charts"], target_objects["charts"]),
"datasets": self._build_object_diff(source_objects["datasets"], target_objects["datasets"]),
}
risk = self._build_risks(source_objects, target_objects, diff, target_client)
summary = {
"dashboards": {action: len(diff["dashboards"][action]) for action in ("create", "update", "delete")},
"charts": {action: len(diff["charts"][action]) for action in ("create", "update", "delete")},
"datasets": {action: len(diff["datasets"][action]) for action in ("create", "update", "delete")},
"selected_dashboards": len(selection.selected_ids),
}
selected_titles = [
selected_preview[dash_id]["title"]
for dash_id in selection.selected_ids
if dash_id in selected_preview
]
logger.reason("[MigrationDryRunService.run][REASON] dry-run payload assembled")
return {
"generated_at": datetime.now(timezone.utc).isoformat(),
"selection": selection.model_dump(),
"selected_dashboard_titles": selected_titles,
"diff": diff,
"summary": summary,
"risk": score_risks(risk),
}
# [/DEF:run:Function]
# [DEF:_load_db_mapping:Function]
# @PURPOSE: Resolve UUID mapping for optional DB config replacement.
def _load_db_mapping(self, db: Session, selection: DashboardSelection) -> Dict[str, str]:
rows = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == selection.source_env_id,
DatabaseMapping.target_env_id == selection.target_env_id,
).all()
return {row.source_db_uuid: row.target_db_uuid for row in rows}
# [/DEF:_load_db_mapping:Function]
# [DEF:_accumulate_objects:Function]
# @PURPOSE: Merge extracted resources by UUID to avoid duplicates.
def _accumulate_objects(self, target: Dict[str, Dict[str, Dict[str, Any]]], source: Dict[str, List[Dict[str, Any]]]) -> None:
for object_type in ("dashboards", "charts", "datasets"):
for item in source.get(object_type, []):
uuid = item.get("uuid")
if uuid:
target[object_type][str(uuid)] = item
# [/DEF:_accumulate_objects:Function]
# [DEF:_index_by_uuid:Function]
# @PURPOSE: Build UUID-index map for normalized resources.
def _index_by_uuid(self, objects: List[Dict[str, Any]]) -> Dict[str, Dict[str, Any]]:
indexed: Dict[str, Dict[str, Any]] = {}
for obj in objects:
uuid = obj.get("uuid")
if uuid:
indexed[str(uuid)] = obj
return indexed
# [/DEF:_index_by_uuid:Function]
# [DEF:_build_object_diff:Function]
# @PURPOSE: Compute create/update/delete buckets by UUID+signature.
def _build_object_diff(self, source_objects: List[Dict[str, Any]], target_objects: List[Dict[str, Any]]) -> Dict[str, List[Dict[str, Any]]]:
target_index = self._index_by_uuid(target_objects)
created: List[Dict[str, Any]] = []
updated: List[Dict[str, Any]] = []
deleted: List[Dict[str, Any]] = []
for source_obj in source_objects:
source_uuid = str(source_obj.get("uuid"))
target_obj = target_index.get(source_uuid)
if not target_obj:
created.append({"uuid": source_uuid, "title": source_obj.get("title")})
continue
if source_obj.get("signature") != target_obj.get("signature"):
updated.append({
"uuid": source_uuid,
"title": source_obj.get("title"),
"target_title": target_obj.get("title"),
})
return {"create": created, "update": updated, "delete": deleted}
# [/DEF:_build_object_diff:Function]
# [DEF:_build_target_signatures:Function]
# @PURPOSE: Pull target metadata and normalize it into comparable signatures.
def _build_target_signatures(self, client: SupersetClient) -> Dict[str, List[Dict[str, Any]]]:
_, dashboards = client.get_dashboards(query={
"columns": ["uuid", "dashboard_title", "slug", "position_json", "json_metadata", "description", "owners"],
})
_, datasets = client.get_datasets(query={
"columns": ["uuid", "table_name", "schema", "database_uuid", "sql", "columns", "metrics"],
})
_, charts = client.get_charts(query={
"columns": ["uuid", "slice_name", "viz_type", "params", "query_context", "datasource_uuid", "dataset_uuid"],
})
return {
"dashboards": [{
"uuid": str(item.get("uuid")),
"title": item.get("dashboard_title"),
"owners": item.get("owners") or [],
"signature": json.dumps({
"title": item.get("dashboard_title"),
"slug": item.get("slug"),
"position_json": item.get("position_json"),
"json_metadata": item.get("json_metadata"),
"description": item.get("description"),
"owners": item.get("owners"),
}, sort_keys=True, default=str),
} for item in dashboards if item.get("uuid")],
"datasets": [{
"uuid": str(item.get("uuid")),
"title": item.get("table_name"),
"database_uuid": item.get("database_uuid"),
"signature": json.dumps({
"title": item.get("table_name"),
"schema": item.get("schema"),
"database_uuid": item.get("database_uuid"),
"sql": item.get("sql"),
"columns": item.get("columns"),
"metrics": item.get("metrics"),
}, sort_keys=True, default=str),
} for item in datasets if item.get("uuid")],
"charts": [{
"uuid": str(item.get("uuid")),
"title": item.get("slice_name") or item.get("name"),
"dataset_uuid": item.get("datasource_uuid") or item.get("dataset_uuid"),
"signature": json.dumps({
"title": item.get("slice_name") or item.get("name"),
"viz_type": item.get("viz_type"),
"params": item.get("params"),
"query_context": item.get("query_context"),
"datasource_uuid": item.get("datasource_uuid"),
"dataset_uuid": item.get("dataset_uuid"),
}, sort_keys=True, default=str),
} for item in charts if item.get("uuid")],
}
# [/DEF:_build_target_signatures:Function]
# [DEF:_build_risks:Function]
# @PURPOSE: Build risk items for missing datasource, broken refs, overwrite, owner mismatch.
def _build_risks(
self,
source_objects: Dict[str, List[Dict[str, Any]]],
target_objects: Dict[str, List[Dict[str, Any]]],
diff: Dict[str, Dict[str, List[Dict[str, Any]]]],
target_client: SupersetClient,
) -> List[Dict[str, Any]]:
return build_risks(source_objects, target_objects, diff, target_client)
# [/DEF:_build_risks:Function]
# [/DEF:MigrationDryRunService:Class]
# [/DEF:backend.src.core.migration.dry_run_orchestrator:Module]

View File

@@ -1,171 +0,0 @@
# [DEF:backend.src.core.migration.risk_assessor:Module]
# @TIER: CRITICAL
# @SEMANTICS: migration, dry_run, risk, scoring, preflight
# @PURPOSE: Compute deterministic migration risk items and aggregate score for dry-run reporting.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[backend.src.core.superset_client.SupersetClient]
# @RELATION: [DISPATCHES] ->[backend.src.core.migration.dry_run_orchestrator.MigrationDryRunService.run]
# @INVARIANT: Risk scoring must remain bounded to [0,100] and preserve severity-to-weight mapping.
# @TEST_CONTRACT: [source_objects,target_objects,diff,target_client] -> [List[RiskItem]]
# @TEST_SCENARIO: [overwrite_update_objects] -> [medium overwrite_existing risk is emitted for each update diff item]
# @TEST_SCENARIO: [missing_datasource_dataset] -> [high missing_datasource risk is emitted]
# @TEST_SCENARIO: [owner_mismatch_dashboard] -> [low owner_mismatch risk is emitted]
# @TEST_EDGE: [missing_field] -> [object without uuid is ignored by indexer]
# @TEST_EDGE: [invalid_type] -> [non-list owners input normalizes to empty identifiers]
# @TEST_EDGE: [external_fail] -> [target_client get_databases exception propagates to caller]
# @TEST_INVARIANT: [score_upper_bound_100] -> VERIFIED_BY: [severity_weight_aggregation]
# @UX_STATE: [Idle] -> [N/A backend domain module]
# @UX_FEEDBACK: [N/A] -> [No direct UI side effects in this module]
# @UX_RECOVERY: [N/A] -> [Caller-level retry/recovery]
# @UX_REACTIVITY: [N/A] -> [Backend synchronous function contracts]
from typing import Any, Dict, List
from ..logger import logger, belief_scope
from ..superset_client import SupersetClient
# [DEF:index_by_uuid:Function]
# @PURPOSE: Build UUID-index from normalized objects.
# @PRE: Input list items are dict-like payloads potentially containing "uuid".
# @POST: Returns mapping keyed by string uuid; only truthy uuid values are included.
# @SIDE_EFFECT: Emits reasoning/reflective logs only.
# @DATA_CONTRACT: List[Dict[str, Any]] -> Dict[str, Dict[str, Any]]
def index_by_uuid(objects: List[Dict[str, Any]]) -> Dict[str, Dict[str, Any]]:
with belief_scope("risk_assessor.index_by_uuid"):
logger.reason("Building UUID index", extra={"objects_count": len(objects)})
indexed: Dict[str, Dict[str, Any]] = {}
for obj in objects:
uuid = obj.get("uuid")
if uuid:
indexed[str(uuid)] = obj
logger.reflect("UUID index built", extra={"indexed_count": len(indexed)})
return indexed
# [/DEF:index_by_uuid:Function]
# [DEF:extract_owner_identifiers:Function]
# @PURPOSE: Normalize owner payloads for stable comparison.
# @PRE: Owners may be list payload, scalar values, or None.
# @POST: Returns sorted unique owner identifiers as strings.
# @SIDE_EFFECT: Emits reasoning/reflective logs only.
# @DATA_CONTRACT: Any -> List[str]
def extract_owner_identifiers(owners: Any) -> List[str]:
with belief_scope("risk_assessor.extract_owner_identifiers"):
logger.reason("Normalizing owner identifiers")
if not isinstance(owners, list):
logger.reflect("Owners payload is not list; returning empty identifiers")
return []
ids: List[str] = []
for owner in owners:
if isinstance(owner, dict):
if owner.get("username"):
ids.append(str(owner["username"]))
elif owner.get("id") is not None:
ids.append(str(owner["id"]))
elif owner is not None:
ids.append(str(owner))
normalized_ids = sorted(set(ids))
logger.reflect("Owner identifiers normalized", extra={"owner_count": len(normalized_ids)})
return normalized_ids
# [/DEF:extract_owner_identifiers:Function]
# [DEF:build_risks:Function]
# @PURPOSE: Build risk list from computed diffs and target catalog state.
# @PRE: source_objects/target_objects/diff contain dashboards/charts/datasets keys with expected list structures.
# @PRE: target_client is authenticated/usable for database list retrieval.
# @POST: Returns list of deterministic risk items derived from overwrite, missing datasource, reference, and owner mismatch checks.
# @SIDE_EFFECT: Calls target Superset API for databases metadata and emits logs.
# @DATA_CONTRACT: (
# @DATA_CONTRACT: Dict[str, List[Dict[str, Any]]],
# @DATA_CONTRACT: Dict[str, List[Dict[str, Any]]],
# @DATA_CONTRACT: Dict[str, Dict[str, List[Dict[str, Any]]]],
# @DATA_CONTRACT: SupersetClient
# @DATA_CONTRACT: ) -> List[Dict[str, Any]]
def build_risks(
source_objects: Dict[str, List[Dict[str, Any]]],
target_objects: Dict[str, List[Dict[str, Any]]],
diff: Dict[str, Dict[str, List[Dict[str, Any]]]],
target_client: SupersetClient,
) -> List[Dict[str, Any]]:
with belief_scope("risk_assessor.build_risks"):
logger.reason("Building migration risks from diff payload")
risks: List[Dict[str, Any]] = []
for object_type in ("dashboards", "charts", "datasets"):
for item in diff[object_type]["update"]:
risks.append({
"code": "overwrite_existing",
"severity": "medium",
"object_type": object_type[:-1],
"object_uuid": item["uuid"],
"message": f"Object will be updated in target: {item.get('title') or item['uuid']}",
})
target_dataset_uuids = set(index_by_uuid(target_objects["datasets"]).keys())
_, target_databases = target_client.get_databases(query={"columns": ["uuid"]})
target_database_uuids = {str(item.get("uuid")) for item in target_databases if item.get("uuid")}
for dataset in source_objects["datasets"]:
db_uuid = dataset.get("database_uuid")
if db_uuid and str(db_uuid) not in target_database_uuids:
risks.append({
"code": "missing_datasource",
"severity": "high",
"object_type": "dataset",
"object_uuid": dataset.get("uuid"),
"message": f"Target datasource is missing for dataset {dataset.get('title') or dataset.get('uuid')}",
})
for chart in source_objects["charts"]:
ds_uuid = chart.get("dataset_uuid")
if ds_uuid and str(ds_uuid) not in target_dataset_uuids:
risks.append({
"code": "breaking_reference",
"severity": "high",
"object_type": "chart",
"object_uuid": chart.get("uuid"),
"message": f"Chart references dataset not found on target: {ds_uuid}",
})
source_dash = index_by_uuid(source_objects["dashboards"])
target_dash = index_by_uuid(target_objects["dashboards"])
for item in diff["dashboards"]["update"]:
source_obj = source_dash.get(item["uuid"])
target_obj = target_dash.get(item["uuid"])
if not source_obj or not target_obj:
continue
source_owners = extract_owner_identifiers(source_obj.get("owners"))
target_owners = extract_owner_identifiers(target_obj.get("owners"))
if source_owners and target_owners and source_owners != target_owners:
risks.append({
"code": "owner_mismatch",
"severity": "low",
"object_type": "dashboard",
"object_uuid": item["uuid"],
"message": f"Owner mismatch for dashboard {item.get('title') or item['uuid']}",
})
logger.reflect("Risk list assembled", extra={"risk_count": len(risks)})
return risks
# [/DEF:build_risks:Function]
# [DEF:score_risks:Function]
# @PURPOSE: Aggregate risk list into score and level.
# @PRE: risk_items contains optional severity fields expected in {high,medium,low} or defaults to low weight.
# @POST: Returns dict with score in [0,100], derived level, and original items.
# @SIDE_EFFECT: Emits reasoning/reflective logs only.
# @DATA_CONTRACT: List[Dict[str, Any]] -> Dict[str, Any]
def score_risks(risk_items: List[Dict[str, Any]]) -> Dict[str, Any]:
with belief_scope("risk_assessor.score_risks"):
logger.reason("Scoring risk items", extra={"risk_items_count": len(risk_items)})
weights = {"high": 25, "medium": 10, "low": 5}
score = min(100, sum(weights.get(item.get("severity", "low"), 5) for item in risk_items))
level = "low" if score < 25 else "medium" if score < 60 else "high"
result = {"score": score, "level": level, "items": risk_items}
logger.reflect("Risk score computed", extra={"score": score, "level": level})
return result
# [/DEF:score_risks:Function]
# [/DEF:backend.src.core.migration.risk_assessor:Module]

View File

@@ -1,15 +1,11 @@
# [DEF:backend.src.core.migration_engine:Module]
#
# @TIER: CRITICAL
# @SEMANTICS: migration, engine, zip, yaml, transformation, cross-filter, id-mapping
# @PURPOSE: Transforms Superset export ZIP archives while preserving archive integrity and patching mapped identifiers.
# @LAYER: Domain
# @RELATION: [DEPENDS_ON] ->[src.core.logger]
# @RELATION: [DEPENDS_ON] ->[src.core.mapping_service.IdMappingService]
# @RELATION: [DEPENDS_ON] ->[src.models.mapping.ResourceType]
# @RELATION: [DEPENDS_ON] ->[yaml]
# @SEMANTICS: migration, engine, zip, yaml, transformation
# @PURPOSE: Handles the interception and transformation of Superset asset ZIP archives.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> PyYAML
#
# @INVARIANT: ZIP structure and non-targeted metadata must remain valid after transformation.
# @INVARIANT: ZIP structure must be preserved after transformation.
# [SECTION: IMPORTS]
import zipfile
@@ -30,17 +26,10 @@ from src.models.mapping import ResourceType
class MigrationEngine:
# [DEF:__init__:Function]
# @PURPOSE: Initializes migration orchestration dependencies for ZIP/YAML metadata transformations.
# @PRE: mapping_service is None or implements batch remote ID lookup for ResourceType.CHART.
# @POST: self.mapping_service is assigned and available for optional cross-filter patching flows.
# @SIDE_EFFECT: Mutates in-memory engine state by storing dependency reference.
# @DATA_CONTRACT: Input[Optional[IdMappingService]] -> Output[MigrationEngine]
# @PURPOSE: Initializes the migration engine with optional ID mapping service.
# @PARAM: mapping_service (Optional[IdMappingService]) - Used for resolving target environment integer IDs.
def __init__(self, mapping_service: Optional[IdMappingService] = None):
with belief_scope("MigrationEngine.__init__"):
logger.reason("Initializing MigrationEngine")
self.mapping_service = mapping_service
logger.reflect("MigrationEngine initialized")
self.mapping_service = mapping_service
# [/DEF:__init__:Function]
# [DEF:transform_zip:Function]
@@ -51,24 +40,20 @@ class MigrationEngine:
# @PARAM: strip_databases (bool) - Whether to remove the databases directory from the archive.
# @PARAM: target_env_id (Optional[str]) - Used if fix_cross_filters is True to know which environment map to use.
# @PARAM: fix_cross_filters (bool) - Whether to patch dashboard json_metadata.
# @PRE: zip_path points to a readable ZIP; output_path parent is writable; db_mapping keys/values are UUID strings.
# @POST: Returns True only when extraction, transformation, and packaging complete without exception.
# @SIDE_EFFECT: Reads/writes filesystem archives, creates temporary directory, emits structured logs.
# @DATA_CONTRACT: Input[(str zip_path, str output_path, Dict[str,str] db_mapping, bool strip_databases, Optional[str] target_env_id, bool fix_cross_filters)] -> Output[bool]
# @RETURN: bool - True if successful.
# @PRE: zip_path must point to a valid Superset export archive.
# @POST: Transformed archive is saved to output_path.
# @RETURN: bool - True if successful.
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str], strip_databases: bool = True, target_env_id: Optional[str] = None, fix_cross_filters: bool = False) -> bool:
"""
Transform a Superset export ZIP by replacing database UUIDs and optionally fixing cross-filters.
"""
with belief_scope("MigrationEngine.transform_zip"):
logger.reason(f"Starting ZIP transformation: {zip_path} -> {output_path}")
with tempfile.TemporaryDirectory() as temp_dir_str:
temp_dir = Path(temp_dir_str)
try:
# 1. Extract
logger.reason(f"Extracting source archive to {temp_dir}")
logger.info(f"[MigrationEngine.transform_zip][Action] Extracting ZIP: {zip_path}")
with zipfile.ZipFile(zip_path, 'r') as zf:
zf.extractall(temp_dir)
@@ -76,33 +61,33 @@ class MigrationEngine:
dataset_files = list(temp_dir.glob("**/datasets/**/*.yaml")) + list(temp_dir.glob("**/datasets/*.yaml"))
dataset_files = list(set(dataset_files))
logger.reason(f"Transforming {len(dataset_files)} dataset YAML files")
logger.info(f"[MigrationEngine.transform_zip][State] Found {len(dataset_files)} dataset files.")
for ds_file in dataset_files:
logger.info(f"[MigrationEngine.transform_zip][Action] Transforming dataset: {ds_file}")
self._transform_yaml(ds_file, db_mapping)
# 2.5 Patch Cross-Filters (Dashboards)
if fix_cross_filters:
if self.mapping_service and target_env_id:
dash_files = list(temp_dir.glob("**/dashboards/**/*.yaml")) + list(temp_dir.glob("**/dashboards/*.yaml"))
dash_files = list(set(dash_files))
logger.reason(f"Patching cross-filters for {len(dash_files)} dashboards")
# Gather all source UUID-to-ID mappings from the archive first
source_id_to_uuid_map = self._extract_chart_uuids_from_archive(temp_dir)
for dash_file in dash_files:
self._patch_dashboard_metadata(dash_file, target_env_id, source_id_to_uuid_map)
else:
logger.explore("Cross-filter patching requested but mapping service or target_env_id is missing")
if fix_cross_filters and self.mapping_service and target_env_id:
dash_files = list(temp_dir.glob("**/dashboards/**/*.yaml")) + list(temp_dir.glob("**/dashboards/*.yaml"))
dash_files = list(set(dash_files))
logger.info(f"[MigrationEngine.transform_zip][State] Found {len(dash_files)} dashboard files for patching.")
# Gather all source UUID-to-ID mappings from the archive first
source_id_to_uuid_map = self._extract_chart_uuids_from_archive(temp_dir)
for dash_file in dash_files:
logger.info(f"[MigrationEngine.transform_zip][Action] Patching dashboard: {dash_file}")
self._patch_dashboard_metadata(dash_file, target_env_id, source_id_to_uuid_map)
# 3. Re-package
logger.reason(f"Re-packaging transformed archive (strip_databases={strip_databases})")
logger.info(f"[MigrationEngine.transform_zip][Action] Re-packaging ZIP to: {output_path} (strip_databases={strip_databases})")
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
for root, dirs, files in os.walk(temp_dir):
rel_root = Path(root).relative_to(temp_dir)
if strip_databases and "databases" in rel_root.parts:
logger.info(f"[MigrationEngine.transform_zip][Action] Skipping file in databases directory: {rel_root}")
continue
for file in files:
@@ -110,10 +95,9 @@ class MigrationEngine:
arcname = file_path.relative_to(temp_dir)
zf.write(file_path, arcname)
logger.reflect("ZIP transformation completed successfully")
return True
except Exception as e:
logger.explore(f"Error transforming ZIP: {e}")
logger.error(f"[MigrationEngine.transform_zip][Coherence:Failed] Error transforming ZIP: {e}")
return False
# [/DEF:transform_zip:Function]
@@ -121,73 +105,54 @@ class MigrationEngine:
# @PURPOSE: Replaces database_uuid in a single YAML file.
# @PARAM: file_path (Path) - Path to the YAML file.
# @PARAM: db_mapping (Dict[str, str]) - UUID mapping dictionary.
# @PRE: file_path exists, is readable YAML, and db_mapping contains source->target UUID pairs.
# @POST: database_uuid is replaced in-place only when source UUID is present in db_mapping.
# @SIDE_EFFECT: Reads and conditionally rewrites YAML file on disk.
# @DATA_CONTRACT: Input[(Path file_path, Dict[str,str] db_mapping)] -> Output[None]
# @PRE: file_path must exist and be readable.
# @POST: File is modified in-place if source UUID matches mapping.
def _transform_yaml(self, file_path: Path, db_mapping: Dict[str, str]):
with belief_scope("MigrationEngine._transform_yaml"):
if not file_path.exists():
logger.explore(f"YAML file not found: {file_path}")
return
with open(file_path, 'r') as f:
data = yaml.safe_load(f)
with open(file_path, 'r') as f:
data = yaml.safe_load(f)
if not data:
return
if not data:
return
source_uuid = data.get('database_uuid')
if source_uuid in db_mapping:
logger.reason(f"Replacing database UUID in {file_path.name}")
data['database_uuid'] = db_mapping[source_uuid]
with open(file_path, 'w') as f:
yaml.dump(data, f)
logger.reflect(f"Database UUID patched in {file_path.name}")
# Superset dataset YAML structure:
# database_uuid: ...
source_uuid = data.get('database_uuid')
if source_uuid in db_mapping:
data['database_uuid'] = db_mapping[source_uuid]
with open(file_path, 'w') as f:
yaml.dump(data, f)
# [/DEF:_transform_yaml:Function]
# [DEF:_extract_chart_uuids_from_archive:Function]
# @PURPOSE: Scans extracted chart YAML files and builds a source chart ID to UUID lookup map.
# @PRE: temp_dir exists and points to extracted archive root with optional chart YAML resources.
# @POST: Returns a best-effort Dict[int, str] containing only parseable chart id/uuid pairs.
# @SIDE_EFFECT: Reads chart YAML files from filesystem; suppresses per-file parsing failures.
# @DATA_CONTRACT: Input[Path] -> Output[Dict[int,str]]
# @PARAM: temp_dir (Path) - Root dir of unpacked archive.
# @PURPOSE: Scans the unpacked ZIP to map local exported integer IDs back to their UUIDs.
# @PARAM: temp_dir (Path) - Root dir of unpacked archive
# @RETURN: Dict[int, str] - Mapping of source Integer ID to UUID.
def _extract_chart_uuids_from_archive(self, temp_dir: Path) -> Dict[int, str]:
with belief_scope("MigrationEngine._extract_chart_uuids_from_archive"):
# Implementation Note: This is a placeholder for the logic that extracts
# actual Source IDs. In a real scenario, this involves parsing chart YAMLs
# or manifesting the export metadata structure where source IDs are stored.
# For simplicity in US1 MVP, we assume it's read from chart files if present.
mapping = {}
chart_files = list(temp_dir.glob("**/charts/**/*.yaml")) + list(temp_dir.glob("**/charts/*.yaml"))
for cf in set(chart_files):
try:
with open(cf, 'r') as f:
cdata = yaml.safe_load(f)
if cdata and 'id' in cdata and 'uuid' in cdata:
mapping[cdata['id']] = cdata['uuid']
except Exception:
pass
return mapping
# Implementation Note: This is a placeholder for the logic that extracts
# actual Source IDs. In a real scenario, this involves parsing chart YAMLs
# or manifesting the export metadata structure where source IDs are stored.
# For simplicity in US1 MVP, we assume it's read from chart files if present.
mapping = {}
chart_files = list(temp_dir.glob("**/charts/**/*.yaml")) + list(temp_dir.glob("**/charts/*.yaml"))
for cf in set(chart_files):
try:
with open(cf, 'r') as f:
cdata = yaml.safe_load(f)
if cdata and 'id' in cdata and 'uuid' in cdata:
mapping[cdata['id']] = cdata['uuid']
except Exception:
pass
return mapping
# [/DEF:_extract_chart_uuids_from_archive:Function]
# [DEF:_patch_dashboard_metadata:Function]
# @PURPOSE: Rewrites dashboard json_metadata chart/dataset integer identifiers using target environment mappings.
# @PRE: file_path points to dashboard YAML with json_metadata; target_env_id is non-empty; source_map contains source id->uuid.
# @POST: json_metadata is re-serialized with mapped integer IDs when remote mappings are available; otherwise file remains unchanged.
# @SIDE_EFFECT: Reads/writes YAML file, performs mapping lookup via mapping_service, emits logs for recoverable/terminal failures.
# @DATA_CONTRACT: Input[(Path file_path, str target_env_id, Dict[int,str] source_map)] -> Output[None]
# @PURPOSE: Replaces integer IDs in json_metadata.
# @PARAM: file_path (Path)
# @PARAM: target_env_id (str)
# @PARAM: source_map (Dict[int, str])
def _patch_dashboard_metadata(self, file_path: Path, target_env_id: str, source_map: Dict[int, str]):
with belief_scope("MigrationEngine._patch_dashboard_metadata"):
try:
if not file_path.exists():
return
with open(file_path, 'r') as f:
data = yaml.safe_load(f)
@@ -198,13 +163,18 @@ class MigrationEngine:
if not metadata_str:
return
metadata = json.loads(metadata_str)
modified = False
# We need to deeply traverse and replace. For MVP, string replacement over the raw JSON is an option,
# but careful dict traversal is safer.
# Fetch target UUIDs for everything we know:
uuids_needed = list(source_map.values())
logger.reason(f"Resolving {len(uuids_needed)} remote IDs for dashboard metadata patching")
target_ids = self.mapping_service.get_remote_ids_batch(target_env_id, ResourceType.CHART, uuids_needed)
if not target_ids:
logger.reflect("No remote target IDs found in mapping database for this dashboard.")
logger.info("[MigrationEngine._patch_dashboard_metadata][Reflect] No remote target IDs found in mapping database.")
return
# Map Source Int -> Target Int
@@ -217,16 +187,21 @@ class MigrationEngine:
missing_targets.append(s_id)
if missing_targets:
logger.explore(f"Missing target IDs for source IDs: {missing_targets}. Cross-filters might break.")
logger.warning(f"[MigrationEngine._patch_dashboard_metadata][Coherence:Recoverable] Missing target IDs for source IDs: {missing_targets}. Cross-filters for these IDs might break.")
if not source_to_target:
logger.reflect("No source IDs matched remotely. Skipping patch.")
logger.info("[MigrationEngine._patch_dashboard_metadata][Reflect] No source IDs matched remotely. Skipping patch.")
return
logger.reason(f"Patching {len(source_to_target)} ID references in json_metadata")
# Complex metadata traversal would go here (e.g. for native_filter_configuration)
# We use regex replacement over the string for safety over unknown nested dicts.
new_metadata_str = metadata_str
# Replace chartId and datasetId assignments explicitly.
# Pattern: "datasetId": 42 or "chartId": 42
for s_id, t_id in source_to_target.items():
# Replace in native_filter_configuration targets
new_metadata_str = re.sub(r'("datasetId"\s*:\s*)' + str(s_id) + r'(\b)', r'\g<1>' + str(t_id) + r'\g<2>', new_metadata_str)
new_metadata_str = re.sub(r'("chartId"\s*:\s*)' + str(s_id) + r'(\b)', r'\g<1>' + str(t_id) + r'\g<2>', new_metadata_str)
@@ -235,10 +210,10 @@ class MigrationEngine:
with open(file_path, 'w') as f:
yaml.dump(data, f)
logger.reflect(f"Dashboard metadata patched and saved: {file_path.name}")
logger.info(f"[MigrationEngine._patch_dashboard_metadata][Reason] Re-serialized modified JSON metadata for dashboard.")
except Exception as e:
logger.explore(f"Metadata patch failed for {file_path.name}: {e}")
logger.error(f"[MigrationEngine._patch_dashboard_metadata][Coherence:Failed] Metadata patch failed: {e}")
# [/DEF:_patch_dashboard_metadata:Function]

View File

@@ -76,8 +76,17 @@ class PluginLoader:
"""
Loads a single Python module and extracts PluginBase subclasses.
"""
# All runtime code is imported through the canonical `src` package root.
package_name = f"src.plugins.{module_name}"
# Try to determine the correct package prefix based on how the app is running
# For standalone execution, we need to handle the import differently
if __name__ == "__main__" or "test" in __name__:
# When running as standalone or in tests, use relative import
package_name = f"plugins.{module_name}"
elif "backend.src" in __name__:
package_prefix = "backend.src.plugins"
package_name = f"{package_prefix}.{module_name}"
else:
package_prefix = "src.plugins"
package_name = f"{package_prefix}.{module_name}"
# print(f"DEBUG: Loading plugin {module_name} as {package_name}")
spec = importlib.util.spec_from_file_location(package_name, file_path)
@@ -189,4 +198,4 @@ class PluginLoader:
return plugin_id in self._plugins
# [/DEF:has_plugin:Function]
# [/DEF:PluginLoader:Class]
# [/DEF:PluginLoader:Class]

View File

@@ -8,13 +8,9 @@
# [SECTION: IMPORTS]
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from apscheduler.triggers.date import DateTrigger
from .logger import logger, belief_scope
from .config_manager import ConfigManager
from .database import SessionLocal
from ..models.llm import ValidationPolicy
import asyncio
from datetime import datetime, time, timedelta, date
# [/SECTION]
# [DEF:SchedulerService:Class]
@@ -121,63 +117,4 @@ class SchedulerService:
# [/DEF:_trigger_backup:Function]
# [/DEF:SchedulerService:Class]
# [DEF:ThrottledSchedulerConfigurator:Class]
# @TIER: CRITICAL
# @SEMANTICS: scheduler, throttling, distribution
# @PURPOSE: Distributes validation tasks evenly within an execution window.
class ThrottledSchedulerConfigurator:
# [DEF:calculate_schedule:Function]
# @PURPOSE: Calculates execution times for N tasks within a window.
# @PRE: window_start, window_end (time), dashboard_ids (List), current_date (date).
# @POST: Returns List[datetime] of scheduled times.
# @INVARIANT: Tasks are distributed with near-even spacing.
@staticmethod
def calculate_schedule(
window_start: time,
window_end: time,
dashboard_ids: list,
current_date: date
) -> list:
with belief_scope("ThrottledSchedulerConfigurator.calculate_schedule"):
n = len(dashboard_ids)
if n == 0:
return []
start_dt = datetime.combine(current_date, window_start)
end_dt = datetime.combine(current_date, window_end)
# Handle window crossing midnight
if end_dt < start_dt:
end_dt += timedelta(days=1)
total_seconds = (end_dt - start_dt).total_seconds()
# Minimum interval of 1 second to avoid division by zero or negative
if total_seconds <= 0:
logger.warning(f"[calculate_schedule] Window size is zero or negative. Falling back to start time for all {n} tasks.")
return [start_dt] * n
# If window is too small for even distribution (e.g. 10 tasks in 5 seconds),
# we still distribute them but they might be very close.
# The requirement says "near-even spacing".
if n == 1:
return [start_dt]
interval = total_seconds / (n - 1) if n > 1 else 0
# If interval is too small (e.g. < 1s), we might want a fallback,
# but the spec says "handle too-small windows with explicit fallback/warning".
if interval < 1:
logger.warning(f"[calculate_schedule] Window too small for {n} tasks (interval {interval:.2f}s). Tasks will be highly concentrated.")
scheduled_times = []
for i in range(n):
scheduled_times.append(start_dt + timedelta(seconds=i * interval))
return scheduled_times
# [/DEF:calculate_schedule:Function]
# [/DEF:ThrottledSchedulerConfigurator:Class]
# [/DEF:SchedulerModule:Module]

View File

@@ -14,7 +14,7 @@ import json
import re
import zipfile
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union, cast
from typing import Dict, List, Optional, Tuple, Union, cast
from requests import Response
from datetime import datetime
from .logger import logger as app_logger, belief_scope
@@ -87,18 +87,7 @@ class SupersetClient:
app_logger.info("[get_dashboards][Enter] Fetching dashboards.")
validated_query = self._validate_query_params(query or {})
if 'columns' not in validated_query:
validated_query['columns'] = [
"slug",
"id",
"url",
"changed_on_utc",
"dashboard_title",
"published",
"created_by",
"changed_by",
"changed_by_name",
"owners",
]
validated_query['columns'] = ["slug", "id", "changed_on_utc", "dashboard_title", "published"]
paginated_data = self._fetch_all_pages(
endpoint="/dashboard/",
@@ -109,267 +98,30 @@ class SupersetClient:
return total_count, paginated_data
# [/DEF:get_dashboards:Function]
# [DEF:get_dashboards_page:Function]
# @PURPOSE: Fetches a single dashboards page from Superset without iterating all pages.
# @PARAM: query (Optional[Dict]) - Query with page/page_size and optional columns.
# @PRE: Client is authenticated.
# @POST: Returns total count and one page of dashboards.
# @RETURN: Tuple[int, List[Dict]]
def get_dashboards_page(self, query: Optional[Dict] = None) -> Tuple[int, List[Dict]]:
with belief_scope("get_dashboards_page"):
validated_query = self._validate_query_params(query or {})
if "columns" not in validated_query:
validated_query["columns"] = [
"slug",
"id",
"url",
"changed_on_utc",
"dashboard_title",
"published",
"created_by",
"changed_by",
"changed_by_name",
"owners",
]
response_json = cast(
Dict[str, Any],
self.network.request(
method="GET",
endpoint="/dashboard/",
params={"q": json.dumps(validated_query)},
),
)
result = response_json.get("result", [])
total_count = response_json.get("count", len(result))
return total_count, result
# [/DEF:get_dashboards_page:Function]
# [DEF:get_dashboards_summary:Function]
# @PURPOSE: Fetches dashboard metadata optimized for the grid.
# @PRE: Client is authenticated.
# @POST: Returns a list of dashboard metadata summaries.
# @RETURN: List[Dict]
def get_dashboards_summary(self, require_slug: bool = False) -> List[Dict]:
def get_dashboards_summary(self) -> List[Dict]:
with belief_scope("SupersetClient.get_dashboards_summary"):
# Rely on list endpoint default projection to stay compatible
# across Superset versions and preserve owners in one request.
query: Dict[str, Any] = {}
if require_slug:
query["filters"] = [
{
"col": "slug",
"opr": "neq",
"value": "",
}
]
query = {
"columns": ["id", "dashboard_title", "changed_on_utc", "published"]
}
_, dashboards = self.get_dashboards(query=query)
# Map fields to DashboardMetadata schema
result = []
max_debug_samples = 12
for index, dash in enumerate(dashboards):
raw_owners = dash.get("owners")
raw_created_by = dash.get("created_by")
raw_changed_by = dash.get("changed_by")
raw_changed_by_name = dash.get("changed_by_name")
owners = self._extract_owner_labels(raw_owners)
# No per-dashboard detail requests here: keep list endpoint O(1).
if not owners:
owners = self._extract_owner_labels(
[raw_created_by, raw_changed_by],
)
projected_created_by = self._extract_user_display(
None,
raw_created_by,
)
projected_modified_by = self._extract_user_display(
raw_changed_by_name,
raw_changed_by,
)
raw_owner_usernames: List[str] = []
if isinstance(raw_owners, list):
for owner_payload in raw_owners:
if isinstance(owner_payload, dict):
owner_username = self._sanitize_user_text(owner_payload.get("username"))
if owner_username:
raw_owner_usernames.append(owner_username)
for dash in dashboards:
result.append({
"id": dash.get("id"),
"slug": dash.get("slug"),
"title": dash.get("dashboard_title"),
"url": dash.get("url"),
"last_modified": dash.get("changed_on_utc"),
"status": "published" if dash.get("published") else "draft",
"created_by": projected_created_by,
"modified_by": projected_modified_by,
"owners": owners,
"status": "published" if dash.get("published") else "draft"
})
if index < max_debug_samples:
app_logger.reflect(
"[REFLECT] Dashboard actor projection sample "
f"(env={getattr(self.env, 'id', None)}, dashboard_id={dash.get('id')}, "
f"raw_owners={raw_owners!r}, raw_owner_usernames={raw_owner_usernames!r}, "
f"raw_created_by={raw_created_by!r}, raw_changed_by={raw_changed_by!r}, "
f"raw_changed_by_name={raw_changed_by_name!r}, projected_owners={owners!r}, "
f"projected_created_by={projected_created_by!r}, projected_modified_by={projected_modified_by!r})"
)
app_logger.reflect(
"[REFLECT] Dashboard actor projection summary "
f"(env={getattr(self.env, 'id', None)}, dashboards={len(result)}, "
f"sampled={min(len(result), max_debug_samples)})"
)
return result
# [/DEF:get_dashboards_summary:Function]
# [DEF:get_dashboards_summary_page:Function]
# @PURPOSE: Fetches one page of dashboard metadata optimized for the grid.
# @PARAM: page (int) - 1-based page number from API route contract.
# @PARAM: page_size (int) - Number of items per page.
# @PRE: page >= 1 and page_size > 0.
# @POST: Returns mapped summaries and total dashboard count.
# @RETURN: Tuple[int, List[Dict]]
def get_dashboards_summary_page(
self,
page: int,
page_size: int,
search: Optional[str] = None,
require_slug: bool = False,
) -> Tuple[int, List[Dict]]:
with belief_scope("SupersetClient.get_dashboards_summary_page"):
query: Dict[str, Any] = {
"page": max(page - 1, 0),
"page_size": page_size,
}
filters: List[Dict[str, Any]] = []
if require_slug:
filters.append(
{
"col": "slug",
"opr": "neq",
"value": "",
}
)
normalized_search = (search or "").strip()
if normalized_search:
# Superset list API supports filter objects with `opr` operator.
# `ct` -> contains (ILIKE on most Superset backends).
filters.append(
{
"col": "dashboard_title",
"opr": "ct",
"value": normalized_search,
}
)
if filters:
query["filters"] = filters
total_count, dashboards = self.get_dashboards_page(query=query)
result = []
for dash in dashboards:
owners = self._extract_owner_labels(dash.get("owners"))
if not owners:
owners = self._extract_owner_labels(
[dash.get("created_by"), dash.get("changed_by")],
)
result.append({
"id": dash.get("id"),
"slug": dash.get("slug"),
"title": dash.get("dashboard_title"),
"url": dash.get("url"),
"last_modified": dash.get("changed_on_utc"),
"status": "published" if dash.get("published") else "draft",
"created_by": self._extract_user_display(
None,
dash.get("created_by"),
),
"modified_by": self._extract_user_display(
dash.get("changed_by_name"),
dash.get("changed_by"),
),
"owners": owners,
})
return total_count, result
# [/DEF:get_dashboards_summary_page:Function]
# [DEF:_extract_owner_labels:Function]
# @PURPOSE: Normalize dashboard owners payload to stable display labels.
# @PRE: owners payload can be scalar, object or list.
# @POST: Returns deduplicated non-empty owner labels preserving order.
# @RETURN: List[str]
def _extract_owner_labels(self, owners_payload: Any) -> List[str]:
if owners_payload is None:
return []
owners_list: List[Any]
if isinstance(owners_payload, list):
owners_list = owners_payload
else:
owners_list = [owners_payload]
normalized: List[str] = []
for owner in owners_list:
label: Optional[str] = None
if isinstance(owner, dict):
label = self._extract_user_display(None, owner)
else:
label = self._sanitize_user_text(owner)
if label and label not in normalized:
normalized.append(label)
return normalized
# [/DEF:_extract_owner_labels:Function]
# [DEF:_extract_user_display:Function]
# @PURPOSE: Normalize user payload to a stable display name.
# @PRE: user payload can be string, dict or None.
# @POST: Returns compact non-empty display value or None.
# @RETURN: Optional[str]
def _extract_user_display(self, preferred_value: Optional[str], user_payload: Optional[Dict]) -> Optional[str]:
preferred = self._sanitize_user_text(preferred_value)
if preferred:
return preferred
if isinstance(user_payload, dict):
full_name = self._sanitize_user_text(user_payload.get("full_name"))
if full_name:
return full_name
first_name = self._sanitize_user_text(user_payload.get("first_name")) or ""
last_name = self._sanitize_user_text(user_payload.get("last_name")) or ""
combined = " ".join(part for part in [first_name, last_name] if part).strip()
if combined:
return combined
username = self._sanitize_user_text(user_payload.get("username"))
if username:
return username
email = self._sanitize_user_text(user_payload.get("email"))
if email:
return email
return None
# [/DEF:_extract_user_display:Function]
# [DEF:_sanitize_user_text:Function]
# @PURPOSE: Convert scalar value to non-empty user-facing text.
# @PRE: value can be any scalar type.
# @POST: Returns trimmed string or None.
# @RETURN: Optional[str]
def _sanitize_user_text(self, value: Optional[Union[str, int]]) -> Optional[str]:
if value is None:
return None
normalized = str(value).strip()
if not normalized:
return None
return normalized
# [/DEF:_sanitize_user_text:Function]
# [DEF:get_dashboard:Function]
# @PURPOSE: Fetches a single dashboard by ID.
# @PRE: Client is authenticated and dashboard_id exists.
@@ -584,25 +336,6 @@ class SupersetClient:
}
# [/DEF:get_dashboard_detail:Function]
# [DEF:get_charts:Function]
# @PURPOSE: Fetches all charts with pagination support.
# @PARAM: query (Optional[Dict]) - Optional query params/columns/filters.
# @PRE: Client is authenticated.
# @POST: Returns total count and charts list.
# @RETURN: Tuple[int, List[Dict]]
def get_charts(self, query: Optional[Dict] = None) -> Tuple[int, List[Dict]]:
with belief_scope("get_charts"):
validated_query = self._validate_query_params(query or {})
if "columns" not in validated_query:
validated_query["columns"] = ["id", "uuid", "slice_name", "viz_type"]
paginated_data = self._fetch_all_pages(
endpoint="/chart/",
pagination_options={"base_query": validated_query, "results_field": "result"},
)
return len(paginated_data), paginated_data
# [/DEF:get_charts:Function]
# [DEF:_extract_chart_ids_from_layout:Function]
# @PURPOSE: Traverses dashboard layout metadata and extracts chart IDs from common keys.
# @PRE: payload can be dict/list/scalar.

View File

@@ -1,238 +0,0 @@
# [DEF:backend.src.core.superset_profile_lookup:Module]
#
# @TIER: STANDARD
# @SEMANTICS: superset, users, lookup, profile, pagination, normalization
# @PURPOSE: Provides environment-scoped Superset account lookup adapter with stable normalized output.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> backend.src.core.utils.network.APIClient
# @RELATION: DEPENDS_ON -> backend.src.core.logger
#
# @INVARIANT: Adapter never leaks raw upstream payload shape to API consumers.
# [SECTION: IMPORTS]
import json
from typing import Any, Dict, List, Optional
from .logger import logger, belief_scope
from .utils.network import APIClient, AuthenticationError, SupersetAPIError
# [/SECTION]
# [DEF:SupersetAccountLookupAdapter:Class]
# @TIER: STANDARD
# @PURPOSE: Lookup Superset users and normalize candidates for profile binding.
class SupersetAccountLookupAdapter:
# [DEF:__init__:Function]
# @PURPOSE: Initializes lookup adapter with authenticated API client and environment context.
# @PRE: network_client supports request(method, endpoint, params=...).
# @POST: Adapter is ready to perform users lookup requests.
def __init__(self, network_client: APIClient, environment_id: str):
self.network_client = network_client
self.environment_id = str(environment_id or "")
# [/DEF:__init__:Function]
# [DEF:get_users_page:Function]
# @PURPOSE: Fetch one users page from Superset with passthrough search/sort parameters.
# @PRE: page_index >= 0 and page_size >= 1.
# @POST: Returns deterministic payload with normalized items and total count.
# @RETURN: Dict[str, Any]
def get_users_page(
self,
search: Optional[str] = None,
page_index: int = 0,
page_size: int = 20,
sort_column: str = "username",
sort_order: str = "desc",
) -> Dict[str, Any]:
with belief_scope("SupersetAccountLookupAdapter.get_users_page"):
normalized_page_index = max(int(page_index), 0)
normalized_page_size = max(int(page_size), 1)
normalized_sort_column = str(sort_column or "username").strip().lower() or "username"
normalized_sort_order = str(sort_order or "desc").strip().lower()
if normalized_sort_order not in {"asc", "desc"}:
normalized_sort_order = "desc"
query: Dict[str, Any] = {
"page": normalized_page_index,
"page_size": normalized_page_size,
"order_column": normalized_sort_column,
"order_direction": normalized_sort_order,
}
normalized_search = str(search or "").strip()
if normalized_search:
query["filters"] = [{"col": "username", "opr": "ct", "value": normalized_search}]
logger.reason(
"[REASON] Lookup Superset users "
f"(env={self.environment_id}, page={normalized_page_index}, page_size={normalized_page_size})"
)
logger.reflect(
"[REFLECT] Prepared Superset users lookup query "
f"(env={self.environment_id}, order_column={normalized_sort_column}, "
f"normalized_sort_order={normalized_sort_order}, "
f"payload_order_direction={query.get('order_direction')})"
)
primary_error: Optional[Exception] = None
last_error: Optional[Exception] = None
for attempt_index, endpoint in enumerate(("/security/users/", "/security/users"), start=1):
try:
logger.reason(
"[REASON] Users lookup request attempt "
f"(env={self.environment_id}, attempt={attempt_index}, endpoint={endpoint})"
)
response = self.network_client.request(
method="GET",
endpoint=endpoint,
params={"q": json.dumps(query)},
)
logger.reflect(
"[REFLECT] Users lookup endpoint succeeded "
f"(env={self.environment_id}, attempt={attempt_index}, endpoint={endpoint})"
)
return self._normalize_lookup_payload(
response=response,
page_index=normalized_page_index,
page_size=normalized_page_size,
)
except Exception as exc:
if primary_error is None:
primary_error = exc
last_error = exc
cause = getattr(exc, "__cause__", None)
cause_response = getattr(cause, "response", None)
status_code = getattr(cause_response, "status_code", None)
logger.explore(
"[EXPLORE] Users lookup endpoint failed "
f"(env={self.environment_id}, attempt={attempt_index}, endpoint={endpoint}, "
f"error_type={type(exc).__name__}, status_code={status_code}, "
f"payload_order_direction={query.get('order_direction')}): {exc}"
)
if last_error is not None:
selected_error: Exception = last_error
if (
primary_error is not None
and primary_error is not last_error
and isinstance(last_error, AuthenticationError)
and not isinstance(primary_error, AuthenticationError)
):
selected_error = primary_error
logger.reflect(
"[REFLECT] Preserving primary lookup failure over fallback auth error "
f"(env={self.environment_id}, primary_error_type={type(primary_error).__name__}, "
f"fallback_error_type={type(last_error).__name__})"
)
logger.explore(
"[EXPLORE] All Superset users lookup endpoints failed "
f"(env={self.environment_id}, payload_order_direction={query.get('order_direction')}, "
f"selected_error_type={type(selected_error).__name__})"
)
raise selected_error
raise SupersetAPIError("Superset users lookup failed without explicit error")
# [/DEF:get_users_page:Function]
# [DEF:_normalize_lookup_payload:Function]
# @PURPOSE: Convert Superset users response variants into stable candidates payload.
# @PRE: response can be dict/list in any supported upstream shape.
# @POST: Output contains canonical keys: status, environment_id, page_index, page_size, total, items.
# @RETURN: Dict[str, Any]
def _normalize_lookup_payload(
self,
response: Any,
page_index: int,
page_size: int,
) -> Dict[str, Any]:
with belief_scope("SupersetAccountLookupAdapter._normalize_lookup_payload"):
payload = response
if isinstance(payload, dict) and isinstance(payload.get("result"), dict):
payload = payload.get("result")
raw_items: List[Any] = []
total = 0
if isinstance(payload, dict):
if isinstance(payload.get("result"), list):
raw_items = payload.get("result") or []
total = int(payload.get("count", len(raw_items)) or 0)
elif isinstance(payload.get("users"), list):
raw_items = payload.get("users") or []
total = int(payload.get("total", len(raw_items)) or 0)
elif isinstance(payload.get("items"), list):
raw_items = payload.get("items") or []
total = int(payload.get("total", len(raw_items)) or 0)
elif isinstance(payload, list):
raw_items = payload
total = len(raw_items)
normalized_items: List[Dict[str, Any]] = []
seen_usernames = set()
for raw_user in raw_items:
candidate = self.normalize_user_payload(raw_user)
username_key = str(candidate.get("username") or "").strip().lower()
if not username_key:
continue
if username_key in seen_usernames:
continue
seen_usernames.add(username_key)
normalized_items.append(candidate)
logger.reflect(
"[REFLECT] Normalized lookup payload "
f"(env={self.environment_id}, items={len(normalized_items)}, total={max(total, len(normalized_items))})"
)
return {
"status": "success",
"environment_id": self.environment_id,
"page_index": max(int(page_index), 0),
"page_size": max(int(page_size), 1),
"total": max(int(total), len(normalized_items)),
"items": normalized_items,
}
# [/DEF:_normalize_lookup_payload:Function]
# [DEF:normalize_user_payload:Function]
# @PURPOSE: Project raw Superset user object to canonical candidate shape.
# @PRE: raw_user may have heterogenous key names between Superset versions.
# @POST: Returns normalized candidate keys (environment_id, username, display_name, email, is_active).
# @RETURN: Dict[str, Any]
def normalize_user_payload(self, raw_user: Any) -> Dict[str, Any]:
if not isinstance(raw_user, dict):
raw_user = {}
username = str(
raw_user.get("username")
or raw_user.get("userName")
or raw_user.get("name")
or ""
).strip()
full_name = str(raw_user.get("full_name") or "").strip()
first_name = str(raw_user.get("first_name") or "").strip()
last_name = str(raw_user.get("last_name") or "").strip()
display_name = full_name or " ".join(
part for part in [first_name, last_name] if part
).strip()
if not display_name:
display_name = username or None
email = str(raw_user.get("email") or "").strip() or None
is_active_raw = raw_user.get("is_active")
is_active = bool(is_active_raw) if is_active_raw is not None else None
return {
"environment_id": self.environment_id,
"username": username,
"display_name": display_name,
"email": email,
"is_active": is_active,
}
# [/DEF:normalize_user_payload:Function]
# [/DEF:SupersetAccountLookupAdapter:Class]
# [/DEF:backend.src.core.superset_profile_lookup:Module]

View File

@@ -1,102 +0,0 @@
# [DEF:__tests__/test_task_logger:Module]
# @RELATION: VERIFIES -> ../task_logger.py
# @PURPOSE: Contract testing for TaskLogger
# [/DEF:__tests__/test_task_logger:Module]
import pytest
from unittest.mock import MagicMock
from src.core.task_manager.task_logger import TaskLogger
# @TEST_FIXTURE: valid_task_logger -> {"task_id": "test_123", "add_log_fn": lambda *args: None, "source": "test_plugin"}
@pytest.fixture
def mock_add_log():
return MagicMock()
@pytest.fixture
def task_logger(mock_add_log):
return TaskLogger(task_id="test_123", add_log_fn=mock_add_log, source="test_plugin")
# @TEST_CONTRACT: TaskLoggerModel -> Invariants
def test_task_logger_initialization(task_logger):
"""Verify TaskLogger is bound to specific task_id and source."""
assert task_logger._task_id == "test_123"
assert task_logger._default_source == "test_plugin"
# @TEST_CONTRACT: invariants -> "All specific log methods (info, error) delegate to _log"
def test_log_methods_delegation(task_logger, mock_add_log):
"""Verify info, error, warning, debug delegate to internal _log."""
task_logger.info("info message", metadata={"k": "v"})
mock_add_log.assert_called_with(
task_id="test_123",
level="INFO",
message="info message",
source="test_plugin",
metadata={"k": "v"}
)
task_logger.error("error message", source="override")
mock_add_log.assert_called_with(
task_id="test_123",
level="ERROR",
message="error message",
source="override",
metadata=None
)
task_logger.warning("warning message")
mock_add_log.assert_called_with(
task_id="test_123",
level="WARNING",
message="warning message",
source="test_plugin",
metadata=None
)
task_logger.debug("debug message")
mock_add_log.assert_called_with(
task_id="test_123",
level="DEBUG",
message="debug message",
source="test_plugin",
metadata=None
)
# @TEST_CONTRACT: invariants -> "with_source creates a new logger with the same task_id"
def test_with_source(task_logger):
"""Verify with_source returns a new instance with updated default source."""
new_logger = task_logger.with_source("new_source")
assert isinstance(new_logger, TaskLogger)
assert new_logger._task_id == "test_123"
assert new_logger._default_source == "new_source"
assert new_logger is not task_logger
# @TEST_EDGE: missing_task_id -> raises TypeError
def test_missing_task_id():
with pytest.raises(TypeError):
TaskLogger(add_log_fn=lambda x: x)
# @TEST_EDGE: invalid_add_log_fn -> raises TypeError
# (Python doesn't strictly enforce this at init, but let's verify it fails on call if not callable)
def test_invalid_add_log_fn():
logger = TaskLogger(task_id="msg", add_log_fn=None)
with pytest.raises(TypeError):
logger.info("test")
# @TEST_INVARIANT: consistent_delegation
def test_progress_log(task_logger, mock_add_log):
"""Verify progress method correctly formats metadata."""
task_logger.progress("Step 1", 45.5)
mock_add_log.assert_called_with(
task_id="test_123",
level="INFO",
message="Step 1",
source="test_plugin",
metadata={"progress": 45.5}
)
# Boundary checks
task_logger.progress("Step high", 150)
assert mock_add_log.call_args[1]["metadata"]["progress"] == 100
task_logger.progress("Step low", -10)
assert mock_add_log.call_args[1]["metadata"]["progress"] == 0

View File

@@ -19,20 +19,6 @@ from ..logger import belief_scope
# @TIER: CRITICAL
# @INVARIANT: logger is always a valid TaskLogger instance.
# @UX_STATE: Idle -> Active -> Complete
#
# @TEST_CONTRACT: TaskContextInit ->
# {
# required_fields: {task_id: str, add_log_fn: Callable, params: dict},
# optional_fields: {default_source: str},
# invariants: [
# "task_id matches initialized logger's task_id",
# "logger is a valid TaskLogger instance"
# ]
# }
# @TEST_FIXTURE: valid_context -> {"task_id": "123", "add_log_fn": lambda *args: None, "params": {"k": "v"}, "default_source": "plugin"}
# @TEST_EDGE: missing_task_id -> raises TypeError
# @TEST_EDGE: missing_add_log_fn -> raises TypeError
# @TEST_INVARIANT: logger_initialized -> verifies: [valid_context]
class TaskContext:
"""
Execution context provided to plugins during task execution.

View File

@@ -39,21 +39,17 @@ from ..logger import logger, belief_scope, should_log_task_level
# @INVARIANT: Task IDs are unique within the registry.
# @INVARIANT: Each task has exactly one status at any time.
# @INVARIANT: Log entries are never deleted after being added to a task.
#
# @TEST_CONTRACT: TaskManagerModel ->
# {
# required_fields: {plugin_loader: PluginLoader},
# invariants: [
# "Tasks are persisted immediately upon creation",
# "Running tasks use a thread pool or asyncio event loop based on executor type",
# "Log flushing runs on a background thread"
# ]
# @TEST_CONTRACT: TaskManager -> {
# required_fields: {plugin_id: str, params: dict},
# optional_fields: {user_id: str},
# invariants: ["Task IDs are unique within the registry", "Each task has exactly one status at any time"]
# }
# @TEST_FIXTURE: valid_manager -> {"plugin_loader": "MockPluginLoader()"}
# @TEST_EDGE: create_task_invalid_plugin -> raises ValueError
# @TEST_EDGE: create_task_invalid_params -> raises ValueError
# @TEST_INVARIANT: lifecycle_management -> verifies: [valid_manager]
class TaskManager:
# @TEST_FIXTURE: create_valid_task -> {"plugin_id": "migration_plugin", "params": {"source": "A", "target": "B"}}
# @TEST_EDGE: missing_required_field -> {"plugin_id": null}
# @TEST_EDGE: empty_response -> {"params": {}}
# @TEST_EDGE: invalid_type -> {"params": "string_instead_of_dict"}
# @TEST_EDGE: external_failure -> {"plugin_not_found": true}
# @TEST_INVARIANT: single_status -> verifies: [create_valid_task, external_failure]
"""
Manages the lifecycle of tasks, including their creation, execution, and state tracking.
"""

Some files were not shown because too many files have changed in this diff Show More