Compare commits

...

5 Commits

Author SHA1 Message Date
077daa0245 mappings+migrate 2025-12-27 10:16:41 +03:00
d38cda09dd tech_lead / coder 2roles 2025-12-27 08:02:59 +03:00
1a893c0bc0 semantic add 2025-12-27 07:14:08 +03:00
40ed375aa4 new loggers logic in constitution 2025-12-27 06:51:28 +03:00
5fdc92fcdf tasks ready 2025-12-27 06:37:03 +03:00
35 changed files with 1505 additions and 402 deletions

View File

@@ -9,6 +9,10 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
- Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic (005-fix-ui-ws-validation)
- N/A (Configuration based) (005-fix-ui-ws-validation)
- Filesystem (plugins, logs, backups), SQLite (optional, for job history if needed) (005-fix-ui-ws-validation)
- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS (007-migration-dashboard-grid)
- N/A (Superset API integration) (007-migration-dashboard-grid)
- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API (007-migration-dashboard-grid)
- N/A (Superset API integration - read-only for metadata) (007-migration-dashboard-grid)
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
@@ -29,10 +33,9 @@ cd src; pytest; ruff check .
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
## Recent Changes
- 006-configurable-belief-logs: Added Python 3.9+ + FastAPI (Backend), Pydantic (Config), Svelte (Frontend)
- 005-fix-ui-ws-validation: Added Python 3.9+ (Backend), Node.js 18+ (Frontend Build)
- 005-fix-ui-ws-validation: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic
- 005-fix-ui-ws-validation: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic
- 007-migration-dashboard-grid: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API
- 007-migration-dashboard-grid: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS
- 007-migration-dashboard-grid: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
<!-- MANUAL ADDITIONS START -->

27
.kilocodemodes Normal file
View File

@@ -0,0 +1,27 @@
customModes:
- slug: tech-lead
name: Tech Lead
description: Architect for contracts and scaffolding
roleDefinition: >-
You are Kilo Code, acting as a Technical Lead and System Architect.
Your primary responsibility is to define the "Structure" and "Contracts" of the system before implementation, following the Semantic Code Generation Protocol.
You operate primarily on 'tasks-arch.md' task lists.
YOUR DUTIES:
1. Create new files and directory structures.
2. Define Modules, Classes, and Functions using `[DEF]` anchors.
3. Write clear Headers with `@PURPOSE`, `@LAYER`, `@RELATION`.
4. Define strict Contracts using `@PRE`, `@POST`, `@PARAM`, `@RETURN`.
5. Leave the implementation body empty or with a placeholder (e.g., `pass`, `return ...`).
YOU DO NOT WRITE BUSINESS LOGIC. Your output is the "Skeleton" and "Rules" that the Developer Agent will fill in.
whenToUse: >-
Use this mode during the "Architecture Phase" of a feature. Select this mode when you need to create new files, define API surfaces, or set up the project structure before coding begins.
groups:
- read
- edit
- command
- list_files
- search_files

View File

@@ -1,14 +1,14 @@
<!--
SYNC IMPACT REPORT
Version: 1.1.0 (Svelte Support)
Version: 1.5.0 (Fractal Complexity Limit)
Changes:
- Added Svelte Component semantic markup standards.
- Updated File Structure Standards to include `.svelte` files.
- Refined File Structure Standards to distinguish between Python Modules and Svelte Components.
- Added Section VI (Fractal Complexity Limit) to enforce strict module (~300 lines) and function (~30-50 lines) size limits.
- Aims to maintain semantic coherence and avoid "Attention Sink".
Templates Status:
- .specify/templates/plan-template.md: ⚠ Pending (Needs update to include Component headers in checks).
- .specify/templates/plan-template.md: ✅ Aligned.
- .specify/templates/spec-template.md: ✅ Aligned.
- .specify/templates/tasks-template.md: ⚠ Pending (Needs update to include Component definition tasks).
- .specify/templates/tasks-arch-template.md: ✅ Aligned (New role-based split).
- .specify/templates/tasks-dev-template.md: ✅ Aligned (New role-based split).
-->
# Semantic Code Generation Constitution
@@ -21,13 +21,31 @@ Semantic definitions (Contracts) must ALWAYS precede implementation code. Logic
Once defined, architectural decisions in the Module Header (`@LAYER`, `@INVARIANT`, `@CONSTRAINT`) are treated as immutable constraints for that module. Changes to these require an explicit refactoring step, not ad-hoc modification during implementation.
### III. Semantic Format Compliance
All output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax with specific Metadata Tags (`@KEY`) and Graph Relations (`@RELATION`). This structure is non-negotiable as it ensures the codebase remains machine-readable, fractal-structured, and optimized for Sparse Attention navigation by AI agents.
All output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax with specific Metadata Tags (`@KEY`) and Graph Relations (`@RELATION`). **Crucially, the closing anchor must strictly match the full content of the opening anchor (e.g., `[DEF:module_name:Module]` must close with `[/DEF:module_name:Module]`).**
**Standardized Graph Relations**
To ensure the integrity of the Semantic Graph, `@RELATION` must use a strict taxonomy:
- `DEPENDS_ON` (Structural dependency)
- `CALLS` (Flow control)
- `CREATES` (Instantiation)
- `INHERITS_FROM` / `IMPLEMENTS` (OOP hierarchy)
- `READS_STATE` / `WRITES_STATE` (Data flow)
- `DISPATCHES` / `HANDLES` (Event flow)
Ad-hoc relationships are forbidden. This structure is non-negotiable as it ensures the codebase remains machine-readable, fractal-structured, and optimized for Sparse Attention navigation by AI agents.
### IV. Design by Contract (DbC)
Contracts are the Source of Truth. Functions and Classes must define their purpose, specifications, and constraints (`@PRE`, `@POST`, `@THROW`) in the metadata block before implementation. Implementation must strictly satisfy these contracts.
### V. Belief State Logging
Logs must define the agent's internal state for debugging and coherence checks. We use a strict format: `logger.level(f"[{ANCHOR_ID}][{STATE}] {MESSAGE} context={...}")` to track transitions between `Entry`, `Validation`, `Action`, and `Coherence` states.
Logs must define the agent's internal state for debugging and coherence checks. We use a strict format: `[{ANCHOR_ID}][{STATE}] {MESSAGE}`. For Python, a **Context Manager** pattern MUST be used to automatically handle `Entry`, `Exit`, and `Coherence` states, ensuring structural integrity and error capturing.
### VI. Fractal Complexity Limit
To maintain semantic coherence and avoid "Attention Sink" issues:
- **Module Size**: If a Module body exceeds ~300 lines (or logical complexity), it MUST be refactored into sub-modules or a package structure.
- **Function Size**: Functions should fit within a standard attention "chunk" (approx. 30-50 lines). If larger, logic MUST be decomposed into helper functions with their own contracts.
This ensures every vector embedding remains sharp and focused.
## File Structure Standards
@@ -51,11 +69,24 @@ Every `.svelte` file must start with a Component definition header (`[DEF:Compon
- `@INVARIANT`: Immutable UI/State rules.
## Generation Workflow
The development process follows a strict sequence:
1. **Analyze Request**: Identify target module and graph position.
2. **Define Structure**: Generate `[DEF]` anchors and Contracts FIRST.
3. **Implement Logic**: Write code satisfying Contracts.
4. **Validate**: If logic conflicts with Contract -> Stop -> Report Error.
The development process follows a strict sequence enforced by Agent Roles:
### 1. Architecture Phase (Mode: `tech-lead`)
**Input**: `tasks-arch.md`
**Responsibility**:
- Analyze request and graph position.
- Generate `[DEF]` anchors, Headers, and Contracts (`@PRE`, `@POST`).
- **Output**: Scaffolding files with no implementation logic.
### 2. Implementation Phase (Mode: `code`)
**Input**: `tasks-dev.md` + Scaffolding files
**Responsibility**:
- Read contracts defined by Architect.
- Write implementation code that strictly satisfies contracts.
- **Output**: Working code with passing tests.
### 3. Validation
If logic conflicts with Contract -> Stop -> Report Error.
## Governance
This Constitution establishes the "Semantic Code Generation Protocol" as the supreme law of this repository.
@@ -63,6 +94,6 @@ This Constitution establishes the "Semantic Code Generation Protocol" as the sup
- **Automated Enforcement**: All code generation tools and agents must parse and validate adherence to the `[DEF]` syntax and Contract requirements.
- **Amendments**: Changes to the syntax or core principles require a formal amendment to this Constitution and a corresponding update to the constitution
- **Review**: Code reviews must verify that implementation matches the preceding contracts and that no "naked code" exists outside of semantic anchors.
- **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure constitutes a build failure.
- **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure (including matching closing tags) constitutes a build failure.
**Version**: 1.1.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-19
**Version**: 1.5.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-27

View File

@@ -9,8 +9,8 @@
#
# OPTIONS:
# --json Output in JSON format
# --require-tasks Require tasks.md to exist (for implementation phase)
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
# --require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
# --include-tasks Include task files in AVAILABLE_DOCS list
# --paths-only Only output path variables (no validation)
# --help, -h Show help message
#
@@ -49,8 +49,8 @@ Consolidated prerequisite checking for Spec-Driven Development workflow.
OPTIONS:
--json Output in JSON format
--require-tasks Require tasks.md to exist (for implementation phase)
--include-tasks Include tasks.md in AVAILABLE_DOCS list
--require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
--include-tasks Include task files in AVAILABLE_DOCS list
--paths-only Only output path variables (no prerequisite validation)
--help, -h Show this help message
@@ -58,7 +58,7 @@ EXAMPLES:
# Check task prerequisites (plan.md required)
./check-prerequisites.sh --json
# Check implementation prerequisites (plan.md + tasks.md required)
# Check implementation prerequisites (plan.md + task files required)
./check-prerequisites.sh --json --require-tasks --include-tasks
# Get feature paths only (no validation)
@@ -86,15 +86,16 @@ check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
if $PATHS_ONLY; then
if $JSON_MODE; then
# Minimal JSON paths payload (no validation performed)
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS_ARCH":"%s","TASKS_DEV":"%s"}\n' \
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS_ARCH" "$TASKS_DEV"
else
echo "REPO_ROOT: $REPO_ROOT"
echo "BRANCH: $CURRENT_BRANCH"
echo "FEATURE_DIR: $FEATURE_DIR"
echo "FEATURE_SPEC: $FEATURE_SPEC"
echo "IMPL_PLAN: $IMPL_PLAN"
echo "TASKS: $TASKS"
echo "TASKS_ARCH: $TASKS_ARCH"
echo "TASKS_DEV: $TASKS_DEV"
fi
exit 0
fi
@@ -112,11 +113,18 @@ if [[ ! -f "$IMPL_PLAN" ]]; then
exit 1
fi
# Check for tasks.md if required
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
echo "Run /speckit.tasks first to create the task list." >&2
exit 1
# Check for task files if required
if $REQUIRE_TASKS; then
if [[ ! -f "$TASKS_ARCH" ]]; then
echo "ERROR: tasks-arch.md not found in $FEATURE_DIR" >&2
echo "Run /speckit.tasks first to create the task lists." >&2
exit 1
fi
if [[ ! -f "$TASKS_DEV" ]]; then
echo "ERROR: tasks-dev.md not found in $FEATURE_DIR" >&2
echo "Run /speckit.tasks first to create the task lists." >&2
exit 1
fi
fi
# Build list of available documents
@@ -133,9 +141,10 @@ fi
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
# Include tasks.md if requested and it exists
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
docs+=("tasks.md")
# Include task files if requested and they exist
if $INCLUDE_TASKS; then
[[ -f "$TASKS_ARCH" ]] && docs+=("tasks-arch.md")
[[ -f "$TASKS_DEV" ]] && docs+=("tasks-dev.md")
fi
# Output results
@@ -161,6 +170,7 @@ else
check_file "$QUICKSTART" "quickstart.md"
if $INCLUDE_TASKS; then
check_file "$TASKS" "tasks.md"
check_file "$TASKS_ARCH" "tasks-arch.md"
check_file "$TASKS_DEV" "tasks-dev.md"
fi
fi

View File

@@ -143,7 +143,9 @@ HAS_GIT='$has_git_repo'
FEATURE_DIR='$feature_dir'
FEATURE_SPEC='$feature_dir/spec.md'
IMPL_PLAN='$feature_dir/plan.md'
TASKS='$feature_dir/tasks.md'
TASKS_ARCH='$feature_dir/tasks-arch.md'
TASKS_DEV='$feature_dir/tasks-dev.md'
TASKS='$feature_dir/tasks.md' # Deprecated
RESEARCH='$feature_dir/research.md'
DATA_MODEL='$feature_dir/data-model.md'
QUICKSTART='$feature_dir/quickstart.md'

View File

@@ -0,0 +1,35 @@
---
description: "Architecture task list template (Contracts & Scaffolding)"
---
# Architecture Tasks: [FEATURE NAME]
**Role**: Architect Agent
**Goal**: Define the "What" and "Why" (Contracts, Scaffolding, Models) before implementation.
**Input**: Design documents from `/specs/[###-feature-name]/`
**Output**: Files with `[DEF]` anchors, `@PRE`/`@POST` contracts, and `@RELATION` mappings. No business logic.
## Phase 1: Setup & Models
- [ ] A001 Create/Update data models in [path] with `[DEF]` and contracts
- [ ] A002 Define API route structure/contracts in [path]
- [ ] A003 Define shared utilities/interfaces
## Phase 2: User Story 1 - [Title]
- [ ] A004 [US1] Define contracts for [Component/Service] in [path]
- [ ] A005 [US1] Define contracts for [Endpoint] in [path]
- [ ] A006 [US1] Define contracts for [Frontend Component] in [path]
## Phase 3: User Story 2 - [Title]
- [ ] A007 [US2] Define contracts for [Component/Service] in [path]
- [ ] A008 [US2] Define contracts for [Endpoint] in [path]
## Handover Checklist
- [ ] All new files created with `[DEF]` anchors
- [ ] All functions/classes have `@PURPOSE`, `@PRE`, `@POST` tags
- [ ] No "naked code" (logic outside of anchors)
- [ ] `tasks-dev.md` is ready for the Developer Agent

View File

@@ -0,0 +1,35 @@
---
description: "Developer task list template (Implementation Logic)"
---
# Developer Tasks: [FEATURE NAME]
**Role**: Developer Agent
**Goal**: Implement the "How" (Logic, State, Error Handling) inside the defined contracts.
**Input**: `tasks-arch.md` (completed), Scaffolding files with `[DEF]` anchors.
**Output**: Working code that satisfies `@PRE`/`@POST` conditions.
## Phase 1: Setup & Models
- [ ] D001 Implement logic for [Model] in [path]
- [ ] D002 Implement logic for [API Route] in [path]
- [ ] D003 Implement shared utilities
## Phase 2: User Story 1 - [Title]
- [ ] D004 [US1] Implement logic for [Component/Service] in [path]
- [ ] D005 [US1] Implement logic for [Endpoint] in [path]
- [ ] D006 [US1] Implement logic for [Frontend Component] in [path]
- [ ] D007 [US1] Verify semantic compliance and belief state logging
## Phase 3: User Story 2 - [Title]
- [ ] D008 [US2] Implement logic for [Component/Service] in [path]
- [ ] D009 [US2] Implement logic for [Endpoint] in [path]
## Polish & Quality Assurance
- [ ] DXXX Verify all tests pass
- [ ] DXXX Check error handling and edge cases
- [ ] DXXX Ensure code style compliance

View File

@@ -1,251 +0,0 @@
---
description: "Task list template for feature implementation"
---
# Tasks: [FEATURE NAME]
**Input**: Design documents from `/specs/[###-feature-name]/`
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/
**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification.
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
- Include exact file paths in descriptions
## Path Conventions
- **Single project**: `src/`, `tests/` at repository root
- **Web app**: `backend/src/`, `frontend/src/`
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
- Paths shown below assume single project - adjust based on plan.md structure
<!--
============================================================================
IMPORTANT: The tasks below are SAMPLE TASKS for illustration purposes only.
The /speckit.tasks command MUST replace these with actual tasks based on:
- User stories from spec.md (with their priorities P1, P2, P3...)
- Feature requirements from plan.md
- Entities from data-model.md
- Endpoints from contracts/
Tasks MUST be organized by user story so each story can be:
- Implemented independently
- Tested independently
- Delivered as an MVP increment
DO NOT keep these sample tasks in the generated tasks.md file.
============================================================================
-->
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Project initialization and basic structure
- [ ] T001 Create project structure per implementation plan
- [ ] T002 Initialize [language] project with [framework] dependencies
- [ ] T003 [P] Configure linting and formatting tools
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
Examples of foundational tasks (adjust based on your project):
- [ ] T004 Setup database schema and migrations framework
- [ ] T005 [P] Implement authentication/authorization framework
- [ ] T006 [P] Setup API routing and middleware structure
- [ ] T007 Create base models/entities that all stories depend on
- [ ] T008 Configure error handling and logging infrastructure
- [ ] T009 Setup environment configuration management
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
---
## Phase 3: User Story 1 - [Title] (Priority: P1) 🎯 MVP
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️
> **NOTE: Write these tests FIRST, ensure they FAIL before implementation**
- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 1
- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py
- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py
- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013)
- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py
- [ ] T016 [US1] Add validation and error handling
- [ ] T017 [US1] Add logging for user story 1 operations
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently
---
## Phase 4: User Story 2 - [Title] (Priority: P2)
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 2 (OPTIONAL - only if tests requested) ⚠️
- [ ] T018 [P] [US2] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T019 [P] [US2] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 2
- [ ] T020 [P] [US2] Create [Entity] model in src/models/[entity].py
- [ ] T021 [US2] Implement [Service] in src/services/[service].py
- [ ] T022 [US2] Implement [endpoint/feature] in src/[location]/[file].py
- [ ] T023 [US2] Integrate with User Story 1 components (if needed)
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently
---
## Phase 5: User Story 3 - [Title] (Priority: P3)
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 3 (OPTIONAL - only if tests requested) ⚠️
- [ ] T024 [P] [US3] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T025 [P] [US3] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 3
- [ ] T026 [P] [US3] Create [Entity] model in src/models/[entity].py
- [ ] T027 [US3] Implement [Service] in src/services/[service].py
- [ ] T028 [US3] Implement [endpoint/feature] in src/[location]/[file].py
**Checkpoint**: All user stories should now be independently functional
---
[Add more user story phases as needed, following the same pattern]
---
## Phase N: Polish & Cross-Cutting Concerns
**Purpose**: Improvements that affect multiple user stories
- [ ] TXXX [P] Documentation updates in docs/
- [ ] TXXX Code cleanup and refactoring
- [ ] TXXX Performance optimization across all stories
- [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/
- [ ] TXXX Security hardening
- [ ] TXXX Run quickstart.md validation
---
## Dependencies & Execution Order
### Phase Dependencies
- **Setup (Phase 1)**: No dependencies - can start immediately
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
- User stories can then proceed in parallel (if staffed)
- Or sequentially in priority order (P1 → P2 → P3)
- **Polish (Final Phase)**: Depends on all desired user stories being complete
### User Story Dependencies
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
- **User Story 2 (P2)**: Can start after Foundational (Phase 2) - May integrate with US1 but should be independently testable
- **User Story 3 (P3)**: Can start after Foundational (Phase 2) - May integrate with US1/US2 but should be independently testable
### Within Each User Story
- Tests (if included) MUST be written and FAIL before implementation
- Models before services
- Services before endpoints
- Core implementation before integration
- Story complete before moving to next priority
### Parallel Opportunities
- All Setup tasks marked [P] can run in parallel
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
- Once Foundational phase completes, all user stories can start in parallel (if team capacity allows)
- All tests for a user story marked [P] can run in parallel
- Models within a story marked [P] can run in parallel
- Different user stories can be worked on in parallel by different team members
---
## Parallel Example: User Story 1
```bash
# Launch all tests for User Story 1 together (if tests requested):
Task: "Contract test for [endpoint] in tests/contract/test_[name].py"
Task: "Integration test for [user journey] in tests/integration/test_[name].py"
# Launch all models for User Story 1 together:
Task: "Create [Entity1] model in src/models/[entity1].py"
Task: "Create [Entity2] model in src/models/[entity2].py"
```
---
## Implementation Strategy
### MVP First (User Story 1 Only)
1. Complete Phase 1: Setup
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
3. Complete Phase 3: User Story 1
4. **STOP and VALIDATE**: Test User Story 1 independently
5. Deploy/demo if ready
### Incremental Delivery
1. Complete Setup + Foundational → Foundation ready
2. Add User Story 1 → Test independently → Deploy/Demo (MVP!)
3. Add User Story 2 → Test independently → Deploy/Demo
4. Add User Story 3 → Test independently → Deploy/Demo
5. Each story adds value without breaking previous stories
### Parallel Team Strategy
With multiple developers:
1. Team completes Setup + Foundational together
2. Once Foundational is done:
- Developer A: User Story 1
- Developer B: User Story 2
- Developer C: User Story 3
3. Stories complete and integrate independently
---
## Notes
- [P] tasks = different files, no dependencies
- [Story] label maps task to specific user story for traceability
- Each user story should be independently completable and testable
- Verify tests fail before implementing
- Commit after each task or logical group
- Stop at any checkpoint to validate story independently
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence

View File

@@ -0,0 +1,71 @@
# [DEF:backend.src.api.routes.migration:Module]
# @SEMANTICS: api, migration, dashboards
# @PURPOSE: API endpoints for migration operations.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.dependencies
# @RELATION: DEPENDS_ON -> backend.src.models.dashboard
from fastapi import APIRouter, Depends, HTTPException
from typing import List, Dict
from backend.src.dependencies import get_config_manager, get_task_manager
from backend.src.models.dashboard import DashboardMetadata, DashboardSelection
from backend.src.core.superset_client import SupersetClient
from superset_tool.models import SupersetConfig
router = APIRouter(prefix="/api", tags=["migration"])
# [DEF:get_dashboards:Function]
# @PURPOSE: Fetch all dashboards from the specified environment for the grid.
# @PRE: Environment ID must be valid.
# @POST: Returns a list of dashboard metadata.
# @PARAM: env_id (str) - The ID of the environment to fetch from.
# @RETURN: List[DashboardMetadata]
@router.get("/environments/{env_id}/dashboards", response_model=List[DashboardMetadata])
async def get_dashboards(env_id: str, config_manager=Depends(get_config_manager)):
environments = config_manager.get_environments()
env = next((e for e in environments if e.id == env_id), None)
if not env:
raise HTTPException(status_code=404, detail="Environment not found")
config = SupersetConfig(
env=env.name,
base_url=env.url,
auth={'provider': 'db', 'username': env.username, 'password': env.password, 'refresh': False},
verify_ssl=True,
timeout=30
)
client = SupersetClient(config)
dashboards = client.get_dashboards_summary()
return dashboards
# [/DEF:get_dashboards]
# [DEF:execute_migration:Function]
# @PURPOSE: Execute the migration of selected dashboards.
# @PRE: Selection must be valid and environments must exist.
# @POST: Starts the migration task and returns the task ID.
# @PARAM: selection (DashboardSelection) - The dashboards to migrate.
# @RETURN: Dict - {"task_id": str, "message": str}
@router.post("/migration/execute")
async def execute_migration(selection: DashboardSelection, config_manager=Depends(get_config_manager), task_manager=Depends(get_task_manager)):
# Validate environments exist
environments = config_manager.get_environments()
env_ids = {e.id for e in environments}
if selection.source_env_id not in env_ids or selection.target_env_id not in env_ids:
raise HTTPException(status_code=400, detail="Invalid source or target environment")
# Create migration task with debug logging
from ...core.logger import logger
logger.info(f"Creating migration task with selection: {selection.dict()}")
logger.info(f"Available environments: {env_ids}")
logger.info(f"Source env: {selection.source_env_id}, Target env: {selection.target_env_id}")
try:
task = await task_manager.create_task("superset-migration", selection.dict())
logger.info(f"Task created successfully: {task.id}")
return {"task_id": task.id, "message": "Migration initiated"}
except Exception as e:
logger.error(f"Task creation failed: {e}")
raise HTTPException(status_code=500, detail=f"Failed to create migration task: {str(e)}")
# [/DEF:execute_migration]
# [/DEF:backend.src.api.routes.migration]

View File

@@ -20,7 +20,7 @@ import os
from .dependencies import get_task_manager
from .core.logger import logger
from .api.routes import plugins, tasks, settings, environments, mappings
from .api.routes import plugins, tasks, settings, environments, mappings, migration
from .core.database import init_db
# Initialize database
@@ -51,6 +51,7 @@ app.include_router(tasks.router, prefix="/api/tasks", tags=["Tasks"])
app.include_router(settings.router, prefix="/api/settings", tags=["Settings"])
app.include_router(environments.router)
app.include_router(mappings.router)
app.include_router(migration.router)
# [DEF:WebSocketEndpoint:Endpoint]
# @SEMANTICS: websocket, logs, streaming, real-time

View File

@@ -15,6 +15,8 @@ import shutil
import tempfile
from pathlib import Path
from typing import Dict
from .logger import logger, belief_scope
import yaml
# [/SECTION]
# [DEF:MigrationEngine:Class]
@@ -26,37 +28,51 @@ class MigrationEngine:
# @PARAM: zip_path (str) - Path to the source ZIP file.
# @PARAM: output_path (str) - Path where the transformed ZIP will be saved.
# @PARAM: db_mapping (Dict[str, str]) - Mapping of source UUID to target UUID.
# @PARAM: strip_databases (bool) - Whether to remove the databases directory from the archive.
# @RETURN: bool - True if successful.
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str]) -> bool:
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str], strip_databases: bool = True) -> bool:
"""
Transform a Superset export ZIP by replacing database UUIDs.
"""
with tempfile.TemporaryDirectory() as temp_dir_str:
temp_dir = Path(temp_dir_str)
with belief_scope("MigrationEngine.transform_zip"):
with tempfile.TemporaryDirectory() as temp_dir_str:
temp_dir = Path(temp_dir_str)
try:
# 1. Extract
with zipfile.ZipFile(zip_path, 'r') as zf:
zf.extractall(temp_dir)
try:
# 1. Extract
logger.info(f"[MigrationEngine.transform_zip][Action] Extracting ZIP: {zip_path}")
with zipfile.ZipFile(zip_path, 'r') as zf:
zf.extractall(temp_dir)
# 2. Transform YAMLs
# Datasets are usually in datasets/*.yaml
dataset_files = list(temp_dir.glob("**/datasets/*.yaml"))
for ds_file in dataset_files:
self._transform_yaml(ds_file, db_mapping)
# 2. Transform YAMLs
# Datasets are usually in datasets/*.yaml
dataset_files = list(temp_dir.glob("**/datasets/**/*.yaml")) + list(temp_dir.glob("**/datasets/*.yaml"))
dataset_files = list(set(dataset_files))
logger.info(f"[MigrationEngine.transform_zip][State] Found {len(dataset_files)} dataset files.")
for ds_file in dataset_files:
logger.info(f"[MigrationEngine.transform_zip][Action] Transforming dataset: {ds_file}")
self._transform_yaml(ds_file, db_mapping)
# 3. Re-package
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
for root, dirs, files in os.walk(temp_dir):
for file in files:
file_path = Path(root) / file
arcname = file_path.relative_to(temp_dir)
zf.write(file_path, arcname)
return True
except Exception as e:
print(f"Error transforming ZIP: {e}")
return False
# 3. Re-package
logger.info(f"[MigrationEngine.transform_zip][Action] Re-packaging ZIP to: {output_path} (strip_databases={strip_databases})")
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
for root, dirs, files in os.walk(temp_dir):
rel_root = Path(root).relative_to(temp_dir)
if strip_databases and "databases" in rel_root.parts:
logger.info(f"[MigrationEngine.transform_zip][Action] Skipping file in databases directory: {rel_root}")
continue
for file in files:
file_path = Path(root) / file
arcname = file_path.relative_to(temp_dir)
zf.write(file_path, arcname)
return True
except Exception as e:
logger.error(f"[MigrationEngine.transform_zip][Coherence:Failed] Error transforming ZIP: {e}")
return False
# [DEF:MigrationEngine._transform_yaml:Function]
# @PURPOSE: Replaces database_uuid in a single YAML file.

View File

@@ -47,12 +47,17 @@ class PluginLoader:
Loads a single Python module and extracts PluginBase subclasses.
"""
# Try to determine the correct package prefix based on how the app is running
if "backend.src" in __name__:
# For standalone execution, we need to handle the import differently
if __name__ == "__main__" or "test" in __name__:
# When running as standalone or in tests, use relative import
package_name = f"plugins.{module_name}"
elif "backend.src" in __name__:
package_prefix = "backend.src.plugins"
package_name = f"{package_prefix}.{module_name}"
else:
package_prefix = "src.plugins"
package_name = f"{package_prefix}.{module_name}"
package_name = f"{package_prefix}.{module_name}"
# print(f"DEBUG: Loading plugin {module_name} as {package_name}")
spec = importlib.util.spec_from_file_location(package_name, file_path)
if spec is None or spec.loader is None:
@@ -106,9 +111,11 @@ class PluginLoader:
# validate(instance={}, schema=schema)
self._plugins[plugin_id] = plugin_instance
self._plugin_configs[plugin_id] = plugin_config
print(f"Plugin '{plugin_instance.name}' (ID: {plugin_id}) loaded successfully.") # Replace with proper logging
from ..core.logger import logger
logger.info(f"Plugin '{plugin_instance.name}' (ID: {plugin_id}) loaded successfully.")
except Exception as e:
print(f"Error validating plugin '{plugin_instance.name}' (ID: {plugin_id}): {e}") # Replace with proper logging
from ..core.logger import logger
logger.error(f"Error validating plugin '{plugin_instance.name}' (ID: {plugin_id}): {e}")
def get_plugin(self, plugin_id: str) -> Optional[PluginBase]:

View File

@@ -52,6 +52,32 @@ class SupersetClient(BaseSupersetClient):
return databases[0] if databases else None
# [/DEF:SupersetClient.get_database_by_uuid]
# [DEF:SupersetClient.get_dashboards_summary:Function]
# @PURPOSE: Fetches dashboard metadata optimized for the grid.
# @POST: Returns a list of dashboard dictionaries.
# @RETURN: List[Dict]
def get_dashboards_summary(self) -> List[Dict]:
"""
Fetches dashboard metadata optimized for the grid.
Returns a list of dictionaries mapped to DashboardMetadata fields.
"""
query = {
"columns": ["id", "dashboard_title", "changed_on_utc", "published"]
}
_, dashboards = self.get_dashboards(query=query)
# Map fields to DashboardMetadata schema
result = []
for dash in dashboards:
result.append({
"id": dash.get("id"),
"title": dash.get("dashboard_title"),
"last_modified": dash.get("changed_on_utc"),
"status": "published" if dash.get("published") else "draft"
})
return result
# [/DEF:SupersetClient.get_dashboards_summary]
# [/DEF:SupersetClient]
# [/DEF:backend.src.core.superset_client]

View File

@@ -64,7 +64,10 @@ class TaskManager:
self.tasks: Dict[str, Task] = {}
self.subscribers: Dict[str, List[asyncio.Queue]] = {}
self.executor = ThreadPoolExecutor(max_workers=5) # For CPU-bound plugin execution
self.loop = asyncio.get_event_loop()
try:
self.loop = asyncio.get_running_loop()
except RuntimeError:
self.loop = asyncio.get_event_loop()
self.task_futures: Dict[str, asyncio.Future] = {}
# [/DEF]
@@ -72,17 +75,25 @@ class TaskManager:
"""
Creates and queues a new task for execution.
"""
from ..core.logger import logger
logger.info(f"TaskManager: Creating task for plugin '{plugin_id}' with params: {params}")
if not self.plugin_loader.has_plugin(plugin_id):
logger.error(f"TaskManager: Plugin with ID '{plugin_id}' not found.")
raise ValueError(f"Plugin with ID '{plugin_id}' not found.")
plugin = self.plugin_loader.get_plugin(plugin_id)
logger.info(f"TaskManager: Found plugin '{plugin.name}' for task creation")
# Validate params against plugin schema (this will be done at a higher level, e.g., API route)
# For now, a basic check
if not isinstance(params, dict):
logger.error("TaskManager: Task parameters must be a dictionary.")
raise ValueError("Task parameters must be a dictionary.")
task = Task(plugin_id=plugin_id, params=params, user_id=user_id)
self.tasks[task.id] = task
logger.info(f"TaskManager: Task {task.id} created and scheduled for execution")
self.loop.create_task(self._run_task(task.id)) # Schedule task for execution
return task
@@ -90,9 +101,11 @@ class TaskManager:
"""
Internal method to execute a task.
"""
from ..core.logger import logger
task = self.tasks[task_id]
plugin = self.plugin_loader.get_plugin(task.plugin_id)
logger.info(f"TaskManager: Starting execution of task {task_id} for plugin '{plugin.name}'")
task.status = TaskStatus.RUNNING
task.started_at = datetime.utcnow()
self._add_log(task_id, "INFO", f"Task started for plugin '{plugin.name}'")
@@ -103,17 +116,27 @@ class TaskManager:
# If the plugin's execute method is already async, this can be simplified.
# Pass task_id to plugin so it can signal pause
params = {**task.params, "_task_id": task_id}
await self.loop.run_in_executor(
self.executor,
lambda: asyncio.run(plugin.execute(params)) if asyncio.iscoroutinefunction(plugin.execute) else plugin.execute(params)
)
logger.info(f"TaskManager: Executing plugin '{plugin.name}' with params: {params}")
if asyncio.iscoroutinefunction(plugin.execute):
logger.info(f"TaskManager: Executing async plugin '{plugin.name}'")
await plugin.execute(params)
else:
logger.info(f"TaskManager: Executing sync plugin '{plugin.name}' in executor")
await self.loop.run_in_executor(
self.executor,
plugin.execute,
params
)
logger.info(f"TaskManager: Task {task_id} completed successfully for plugin '{plugin.name}'")
task.status = TaskStatus.SUCCESS
self._add_log(task_id, "INFO", f"Task completed successfully for plugin '{plugin.name}'")
except Exception as e:
logger.error(f"TaskManager: Task {task_id} failed for plugin '{plugin.name}': {e}")
task.status = TaskStatus.FAILED
self._add_log(task_id, "ERROR", f"Task failed: {e}", {"error_type": type(e).__name__})
finally:
task.finished_at = datetime.utcnow()
logger.info(f"TaskManager: Task {task_id} execution finished with status: {task.status}")
# In a real system, you might notify clients via WebSocket here
async def resolve_task(self, task_id: str, resolution_params: Dict[str, Any]):

View File

@@ -21,7 +21,12 @@ def get_config_manager() -> ConfigManager:
plugin_dir = Path(__file__).parent / "plugins"
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
from .core.logger import logger
logger.info(f"PluginLoader initialized with directory: {plugin_dir}")
logger.info(f"Available plugins: {[config.name for config in plugin_loader.get_all_plugin_configs()]}")
task_manager = TaskManager(plugin_loader)
logger.info("TaskManager initialized")
def get_plugin_loader() -> PluginLoader:
"""Dependency injector for the PluginLoader."""

View File

@@ -0,0 +1,27 @@
# [DEF:backend.src.models.dashboard:Module]
# @SEMANTICS: dashboard, model, metadata, migration
# @PURPOSE: Defines data models for dashboard metadata and selection.
# @LAYER: Model
# @RELATION: USED_BY -> backend.src.api.routes.migration
from pydantic import BaseModel
from typing import List
# [DEF:DashboardMetadata:Class]
# @PURPOSE: Represents a dashboard available for migration.
class DashboardMetadata(BaseModel):
id: int
title: str
last_modified: str
status: str
# [/DEF:DashboardMetadata]
# [DEF:DashboardSelection:Class]
# @PURPOSE: Represents the user's selection of dashboards to migrate.
class DashboardSelection(BaseModel):
selected_ids: List[int]
source_env_id: str
target_env_id: str
# [/DEF:DashboardSelection]
# [/DEF:backend.src.models.dashboard]

View File

@@ -87,34 +87,72 @@ class MigrationPlugin(PluginBase):
}
async def execute(self, params: Dict[str, Any]):
from_env = params["from_env"]
to_env = params["to_env"]
dashboard_regex = params["dashboard_regex"]
source_env_id = params.get("source_env_id")
target_env_id = params.get("target_env_id")
selected_ids = params.get("selected_ids")
# Legacy support or alternative params
from_env_name = params.get("from_env")
to_env_name = params.get("to_env")
dashboard_regex = params.get("dashboard_regex")
replace_db_config = params.get("replace_db_config", False)
from_db_id = params.get("from_db_id")
to_db_id = params.get("to_db_id")
logger = SupersetLogger(log_dir=Path.cwd() / "logs", console=True)
logger.info(f"[MigrationPlugin][Entry] Starting migration from {from_env} to {to_env}.")
logger.info(f"[MigrationPlugin][Entry] Starting migration task.")
logger.info(f"[MigrationPlugin][Action] Params: {params}")
try:
config_manager = get_config_manager()
all_clients = setup_clients(logger, custom_envs=config_manager.get_environments())
from_c = all_clients.get(from_env)
to_c = all_clients.get(to_env)
environments = config_manager.get_environments()
# Resolve environments
src_env = None
tgt_env = None
if source_env_id:
src_env = next((e for e in environments if e.id == source_env_id), None)
elif from_env_name:
src_env = next((e for e in environments if e.name == from_env_name), None)
if target_env_id:
tgt_env = next((e for e in environments if e.id == target_env_id), None)
elif to_env_name:
tgt_env = next((e for e in environments if e.name == to_env_name), None)
if not src_env or not tgt_env:
raise ValueError(f"Could not resolve source or target environment. Source: {source_env_id or from_env_name}, Target: {target_env_id or to_env_name}")
from_env_name = src_env.name
to_env_name = tgt_env.name
logger.info(f"[MigrationPlugin][State] Resolved environments: {from_env_name} -> {to_env_name}")
all_clients = setup_clients(logger, custom_envs=environments)
from_c = all_clients.get(from_env_name)
to_c = all_clients.get(to_env_name)
if not from_c or not to_c:
raise ValueError(f"One or both environments ('{from_env}', '{to_env}') not found in configuration.")
raise ValueError(f"Clients not initialized for environments: {from_env_name}, {to_env_name}")
_, all_dashboards = from_c.get_dashboards()
regex_str = str(dashboard_regex)
dashboards_to_migrate = [
d for d in all_dashboards if re.search(regex_str, d["dashboard_title"], re.IGNORECASE)
]
dashboards_to_migrate = []
if selected_ids:
dashboards_to_migrate = [d for d in all_dashboards if d["id"] in selected_ids]
elif dashboard_regex:
regex_str = str(dashboard_regex)
dashboards_to_migrate = [
d for d in all_dashboards if re.search(regex_str, d["dashboard_title"], re.IGNORECASE)
]
else:
logger.warning("[MigrationPlugin][State] No selection criteria provided (selected_ids or dashboard_regex).")
return
if not dashboards_to_migrate:
logger.warning("[MigrationPlugin][State] No dashboards found matching the regex.")
logger.warning("[MigrationPlugin][State] No dashboards found matching criteria.")
return
# Fetch mappings from database
@@ -123,8 +161,8 @@ class MigrationPlugin(PluginBase):
db = SessionLocal()
try:
# Find environment IDs by name
src_env = db.query(Environment).filter(Environment.name == from_env).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
src_env = db.query(Environment).filter(Environment.name == from_env_name).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env_name).first()
if src_env and tgt_env:
mappings = db.query(DatabaseMapping).filter(
@@ -144,46 +182,39 @@ class MigrationPlugin(PluginBase):
try:
exported_content, _ = from_c.export_dashboard(dash_id)
with create_temp_file(content=exported_content, dry_run=True, suffix=".zip", logger=logger) as tmp_zip_path:
if not replace_db_config:
to_c.import_dashboard(file_name=tmp_zip_path, dash_id=dash_id, dash_slug=dash_slug)
else:
# Check for missing mappings before transformation
# This is a simplified check, in reality we'd check all YAMLs
# For US3, we'll just use the engine and handle missing ones there
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
# If we have missing mappings, we might need to pause
# For now, let's assume the engine can tell us what's missing
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
if not success:
# Signal missing mapping and wait
task_id = params.get("_task_id")
if task_id:
from ..dependencies import get_task_manager
tm = get_task_manager()
logger.info(f"[MigrationPlugin][Action] Pausing for missing mapping in task {task_id}")
# In a real scenario, we'd pass the missing DB info to the frontend
# For this task, we'll just simulate the wait
await tm.wait_for_resolution(task_id)
# After resolution, retry transformation with updated mappings
# (Mappings would be updated in task.params by resolve_task)
db = SessionLocal()
try:
src_env = db.query(Environment).filter(Environment.name == from_env).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
mappings = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == src_env.id,
DatabaseMapping.target_env_id == tgt_env.id
).all()
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
finally:
db.close()
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
# Always transform to strip databases to avoid password errors
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping, strip_databases=False)
if not success and replace_db_config:
# Signal missing mapping and wait (only if we care about mappings)
task_id = params.get("_task_id")
if task_id:
from ..dependencies import get_task_manager
tm = get_task_manager()
logger.info(f"[MigrationPlugin][Action] Pausing for missing mapping in task {task_id}")
# In a real scenario, we'd pass the missing DB info to the frontend
# For this task, we'll just simulate the wait
await tm.wait_for_resolution(task_id)
# After resolution, retry transformation with updated mappings
# (Mappings would be updated in task.params by resolve_task)
db = SessionLocal()
try:
src_env = db.query(Environment).filter(Environment.name == from_env_name).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env_name).first()
mappings = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == src_env.id,
DatabaseMapping.target_env_id == tgt_env.id
).all()
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
finally:
db.close()
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping, strip_databases=False)
if success:
to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
else:
logger.error(f"[MigrationPlugin][Failure] Failed to transform ZIP for dashboard {title}")
if success:
to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
else:
logger.error(f"[MigrationPlugin][Failure] Failed to transform ZIP for dashboard {title}")
logger.info(f"[MigrationPlugin][Success] Dashboard {title} imported.")
except Exception as exc:

View File

@@ -24,7 +24,7 @@ export const options = {
app: ({ head, body, assets, nonce, env }) => "<!DOCTYPE html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<link rel=\"icon\" href=\"" + assets + "/favicon.png\" />\n\t\t<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n\t\t" + head + "\n\t</head>\n\t<body data-sveltekit-preload-data=\"hover\">\n\t\t<div style=\"display: contents\">" + body + "</div>\n\t</body>\n</html>\n",
error: ({ status, message }) => "<!doctype html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<title>" + message + "</title>\n\n\t\t<style>\n\t\t\tbody {\n\t\t\t\t--bg: white;\n\t\t\t\t--fg: #222;\n\t\t\t\t--divider: #ccc;\n\t\t\t\tbackground: var(--bg);\n\t\t\t\tcolor: var(--fg);\n\t\t\t\tfont-family:\n\t\t\t\t\tsystem-ui,\n\t\t\t\t\t-apple-system,\n\t\t\t\t\tBlinkMacSystemFont,\n\t\t\t\t\t'Segoe UI',\n\t\t\t\t\tRoboto,\n\t\t\t\t\tOxygen,\n\t\t\t\t\tUbuntu,\n\t\t\t\t\tCantarell,\n\t\t\t\t\t'Open Sans',\n\t\t\t\t\t'Helvetica Neue',\n\t\t\t\t\tsans-serif;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tjustify-content: center;\n\t\t\t\theight: 100vh;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t.error {\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tmax-width: 32rem;\n\t\t\t\tmargin: 0 1rem;\n\t\t\t}\n\n\t\t\t.status {\n\t\t\t\tfont-weight: 200;\n\t\t\t\tfont-size: 3rem;\n\t\t\t\tline-height: 1;\n\t\t\t\tposition: relative;\n\t\t\t\ttop: -0.05rem;\n\t\t\t}\n\n\t\t\t.message {\n\t\t\t\tborder-left: 1px solid var(--divider);\n\t\t\t\tpadding: 0 0 0 1rem;\n\t\t\t\tmargin: 0 0 0 1rem;\n\t\t\t\tmin-height: 2.5rem;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t}\n\n\t\t\t.message h1 {\n\t\t\t\tfont-weight: 400;\n\t\t\t\tfont-size: 1em;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t@media (prefers-color-scheme: dark) {\n\t\t\t\tbody {\n\t\t\t\t\t--bg: #222;\n\t\t\t\t\t--fg: #ddd;\n\t\t\t\t\t--divider: #666;\n\t\t\t\t}\n\t\t\t}\n\t\t</style>\n\t</head>\n\t<body>\n\t\t<div class=\"error\">\n\t\t\t<span class=\"status\">" + status + "</span>\n\t\t\t<div class=\"message\">\n\t\t\t\t<h1>" + message + "</h1>\n\t\t\t</div>\n\t\t</div>\n\t</body>\n</html>\n"
},
version_hash: "n7gbte"
version_hash: "oj9twc"
};
export async function get_hooks() {

View File

@@ -0,0 +1,205 @@
<!-- [DEF:DashboardGrid:Component] -->
<!--
@SEMANTICS: dashboard, grid, selection, pagination
@PURPOSE: Displays a grid of dashboards with selection and pagination.
@LAYER: Component
@RELATION: USED_BY -> frontend/src/routes/migration/+page.svelte
@INVARIANT: Selected IDs must be a subset of available dashboards.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { createEventDispatcher } from 'svelte';
import type { DashboardMetadata } from '../types/dashboard';
// [/SECTION]
// [SECTION: PROPS]
export let dashboards: DashboardMetadata[] = [];
export let selectedIds: number[] = [];
// [/SECTION]
// [SECTION: STATE]
let filterText = "";
let currentPage = 0;
let pageSize = 20;
let sortColumn: keyof DashboardMetadata = "title";
let sortDirection: "asc" | "desc" = "asc";
// [/SECTION]
// [SECTION: DERIVED]
$: filteredDashboards = dashboards.filter(d =>
d.title.toLowerCase().includes(filterText.toLowerCase())
);
$: sortedDashboards = [...filteredDashboards].sort((a, b) => {
let aVal = a[sortColumn];
let bVal = b[sortColumn];
if (sortColumn === "id") {
aVal = Number(aVal);
bVal = Number(bVal);
}
if (aVal < bVal) return sortDirection === "asc" ? -1 : 1;
if (aVal > bVal) return sortDirection === "asc" ? 1 : -1;
return 0;
});
$: paginatedDashboards = sortedDashboards.slice(
currentPage * pageSize,
(currentPage + 1) * pageSize
);
$: totalPages = Math.ceil(sortedDashboards.length / pageSize);
$: allSelected = paginatedDashboards.length > 0 && paginatedDashboards.every(d => selectedIds.includes(d.id));
$: someSelected = paginatedDashboards.some(d => selectedIds.includes(d.id));
// [/SECTION]
// [SECTION: EVENTS]
const dispatch = createEventDispatcher<{ selectionChanged: number[] }>();
// [/SECTION]
// [DEF:handleSort:Function]
// @PURPOSE: Toggles sort direction or changes sort column.
function handleSort(column: keyof DashboardMetadata) {
if (sortColumn === column) {
sortDirection = sortDirection === "asc" ? "desc" : "asc";
} else {
sortColumn = column;
sortDirection = "asc";
}
}
// [/DEF:handleSort]
// [DEF:handleSelectionChange:Function]
// @PURPOSE: Handles individual checkbox changes.
function handleSelectionChange(id: number, checked: boolean) {
let newSelected = [...selectedIds];
if (checked) {
if (!newSelected.includes(id)) newSelected.push(id);
} else {
newSelected = newSelected.filter(sid => sid !== id);
}
selectedIds = newSelected;
dispatch('selectionChanged', newSelected);
}
// [/DEF:handleSelectionChange]
// [DEF:handleSelectAll:Function]
// @PURPOSE: Handles select all checkbox.
function handleSelectAll(checked: boolean) {
let newSelected = [...selectedIds];
if (checked) {
paginatedDashboards.forEach(d => {
if (!newSelected.includes(d.id)) newSelected.push(d.id);
});
} else {
paginatedDashboards.forEach(d => {
newSelected = newSelected.filter(sid => sid !== d.id);
});
}
selectedIds = newSelected;
dispatch('selectionChanged', newSelected);
}
// [/DEF:handleSelectAll]
// [DEF:goToPage:Function]
// @PURPOSE: Changes current page.
function goToPage(page: number) {
if (page >= 0 && page < totalPages) {
currentPage = page;
}
}
// [/DEF:goToPage]
</script>
<!-- [SECTION: TEMPLATE] -->
<div class="dashboard-grid">
<!-- Filter Input -->
<div class="mb-4">
<input
type="text"
bind:value={filterText}
placeholder="Search dashboards..."
class="w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500"
/>
</div>
<!-- Grid/Table -->
<div class="overflow-x-auto">
<table class="min-w-full bg-white border border-gray-300">
<thead class="bg-gray-50">
<tr>
<th class="px-4 py-2 border-b">
<input
type="checkbox"
checked={allSelected}
indeterminate={someSelected && !allSelected}
on:change={(e) => handleSelectAll((e.target as HTMLInputElement).checked)}
/>
</th>
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('title')}>
Title {sortColumn === 'title' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
</th>
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('last_modified')}>
Last Modified {sortColumn === 'last_modified' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
</th>
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('status')}>
Status {sortColumn === 'status' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
</th>
</tr>
</thead>
<tbody>
{#each paginatedDashboards as dashboard (dashboard.id)}
<tr class="hover:bg-gray-50">
<td class="px-4 py-2 border-b">
<input
type="checkbox"
checked={selectedIds.includes(dashboard.id)}
on:change={(e) => handleSelectionChange(dashboard.id, (e.target as HTMLInputElement).checked)}
/>
</td>
<td class="px-4 py-2 border-b">{dashboard.title}</td>
<td class="px-4 py-2 border-b">{new Date(dashboard.last_modified).toLocaleDateString()}</td>
<td class="px-4 py-2 border-b">
<span class="px-2 py-1 text-xs font-medium rounded-full {dashboard.status === 'published' ? 'bg-green-100 text-green-800' : 'bg-gray-100 text-gray-800'}">
{dashboard.status}
</span>
</td>
</tr>
{/each}
</tbody>
</table>
</div>
<!-- Pagination Controls -->
<div class="flex items-center justify-between mt-4">
<div class="text-sm text-gray-700">
Showing {currentPage * pageSize + 1} to {Math.min((currentPage + 1) * pageSize, sortedDashboards.length)} of {sortedDashboards.length} dashboards
</div>
<div class="flex space-x-2">
<button
class="px-3 py-1 text-sm border border-gray-300 rounded-md hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
disabled={currentPage === 0}
on:click={() => goToPage(currentPage - 1)}
>
Previous
</button>
<button
class="px-3 py-1 text-sm border border-gray-300 rounded-md hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
disabled={currentPage >= totalPages - 1}
on:click={() => goToPage(currentPage + 1)}
>
Next
</button>
</div>
</div>
</div>
<!-- [/SECTION] -->
<style>
/* Component styles */
</style>
<!-- [/DEF:DashboardGrid] -->

View File

@@ -36,8 +36,9 @@
<!-- [SECTION: TEMPLATE] -->
<div class="flex flex-col space-y-1">
<label class="text-sm font-medium text-gray-700">{label}</label>
<label for="env-select" class="text-sm font-medium text-gray-700">{label}</label>
<select
id="env-select"
class="block w-full pl-3 pr-10 py-2 text-base border-gray-300 focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm rounded-md"
value={selectedId}
on:change={handleSelect}

View File

@@ -12,16 +12,26 @@
// [SECTION: IMPORTS]
import { onMount } from 'svelte';
import EnvSelector from '../../components/EnvSelector.svelte';
import DashboardGrid from '../../components/DashboardGrid.svelte';
import MappingTable from '../../components/MappingTable.svelte';
import MissingMappingModal from '../../components/MissingMappingModal.svelte';
import type { DashboardMetadata, DashboardSelection } from '../../types/dashboard';
// [/SECTION]
// [SECTION: STATE]
let environments = [];
let environments: any[] = [];
let sourceEnvId = "";
let targetEnvId = "";
let dashboardRegex = ".*";
let replaceDb = false;
let loading = true;
let error = "";
let dashboards: DashboardMetadata[] = [];
let selectedDashboardIds: number[] = [];
let sourceDatabases: any[] = [];
let targetDatabases: any[] = [];
let mappings: any[] = [];
let suggestions: any[] = [];
let fetchingDbs = false;
// [/SECTION]
// [DEF:fetchEnvironments:Function]
@@ -42,8 +52,100 @@
}
// [/DEF:fetchEnvironments]
// [DEF:fetchDashboards:Function]
/**
* @purpose Fetches dashboards for the selected source environment.
* @param envId The environment ID.
* @post dashboards state is updated.
*/
async function fetchDashboards(envId: string) {
try {
const response = await fetch(`/api/environments/${envId}/dashboards`);
if (!response.ok) throw new Error('Failed to fetch dashboards');
dashboards = await response.json();
selectedDashboardIds = []; // Reset selection when env changes
} catch (e) {
error = e.message;
dashboards = [];
}
}
// [/DEF:fetchDashboards]
onMount(fetchEnvironments);
// Reactive: fetch dashboards when source env changes
$: if (sourceEnvId) fetchDashboards(sourceEnvId);
// [DEF:fetchDatabases:Function]
/**
* @purpose Fetches databases from both environments and gets suggestions.
*/
async function fetchDatabases() {
if (!sourceEnvId || !targetEnvId) return;
fetchingDbs = true;
error = "";
try {
const [srcRes, tgtRes, mapRes, sugRes] = await Promise.all([
fetch(`/api/environments/${sourceEnvId}/databases`),
fetch(`/api/environments/${targetEnvId}/databases`),
fetch(`/api/mappings?source_env_id=${sourceEnvId}&target_env_id=${targetEnvId}`),
fetch(`/api/mappings/suggest`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ source_env_id: sourceEnvId, target_env_id: targetEnvId })
})
]);
if (!srcRes.ok || !tgtRes.ok) throw new Error('Failed to fetch databases from environments');
sourceDatabases = await srcRes.json();
targetDatabases = await tgtRes.json();
mappings = await mapRes.json();
suggestions = await sugRes.json();
} catch (e) {
error = e.message;
} finally {
fetchingDbs = false;
}
}
// [/DEF:fetchDatabases]
// [DEF:handleMappingUpdate:Function]
/**
* @purpose Saves a mapping to the backend.
*/
async function handleMappingUpdate(event: CustomEvent) {
const { sourceUuid, targetUuid } = event.detail;
const sDb = sourceDatabases.find(d => d.uuid === sourceUuid);
const tDb = targetDatabases.find(d => d.uuid === targetUuid);
if (!sDb || !tDb) return;
try {
const response = await fetch('/api/mappings', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
source_env_id: sourceEnvId,
target_env_id: targetEnvId,
source_db_uuid: sourceUuid,
target_db_uuid: targetUuid,
source_db_name: sDb.database_name,
target_db_name: tDb.database_name
})
});
if (!response.ok) throw new Error('Failed to save mapping');
const savedMapping = await response.json();
mappings = [...mappings.filter(m => m.source_db_uuid !== sourceUuid), savedMapping];
} catch (e) {
error = e.message;
}
}
// [/DEF:handleMappingUpdate]
// [DEF:startMigration:Function]
/**
* @purpose Starts the migration process.
@@ -58,10 +160,33 @@
error = "Source and target environments must be different.";
return;
}
if (selectedDashboardIds.length === 0) {
error = "Please select at least one dashboard to migrate.";
return;
}
error = "";
console.log(`[MigrationDashboard][Action] Starting migration from ${sourceEnvId} to ${targetEnvId} (Replace DB: ${replaceDb})`);
// TODO: Implement actual migration trigger in US3
try {
const selection: DashboardSelection = {
selected_ids: selectedDashboardIds,
source_env_id: sourceEnvId,
target_env_id: targetEnvId
};
console.log(`[MigrationDashboard][Action] Starting migration with selection:`, selection);
const response = await fetch('/api/migration/execute', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(selection)
});
console.log(`[MigrationDashboard][Action] API response status: ${response.status}`);
if (!response.ok) throw new Error(`Failed to start migration: ${response.status} ${response.statusText}`);
const result = await response.json();
console.log(`[MigrationDashboard][Action] Migration started: ${result.task_id} - ${result.message}`);
// TODO: Show success message or redirect to task status
} catch (e) {
console.error(`[MigrationDashboard][Failure] Migration failed:`, e);
error = e.message;
}
}
// [/DEF:startMigration]
</script>
@@ -91,23 +216,28 @@
/>
</div>
<!-- [DEF:DashboardSelectionSection] -->
<div class="mb-8">
<label for="dashboard-regex" class="block text-sm font-medium text-gray-700 mb-1">Dashboard Regex</label>
<input
id="dashboard-regex"
type="text"
bind:value={dashboardRegex}
placeholder="e.g. ^Finance Dashboard$"
class="shadow-sm focus:ring-indigo-500 focus:border-indigo-500 block w-full sm:text-sm border-gray-300 rounded-md"
/>
<p class="mt-1 text-sm text-gray-500">Regular expression to filter dashboards to migrate.</p>
<h2 class="text-lg font-medium mb-4">Select Dashboards</h2>
{#if sourceEnvId}
<DashboardGrid
{dashboards}
bind:selectedIds={selectedDashboardIds}
/>
{:else}
<p class="text-gray-500 italic">Select a source environment to view dashboards.</p>
{/if}
</div>
<!-- [/DEF:DashboardSelectionSection] -->
<div class="flex items-center mb-8">
<div class="flex items-center mb-4">
<input
id="replace-db"
type="checkbox"
bind:checked={replaceDb}
on:change={() => { if (replaceDb && sourceDatabases.length === 0) fetchDatabases(); }}
class="h-4 w-4 text-indigo-600 focus:ring-indigo-500 border-gray-300 rounded"
/>
<label for="replace-db" class="ml-2 block text-sm text-gray-900">
@@ -115,9 +245,33 @@
</label>
</div>
{#if replaceDb}
<div class="mb-8 p-4 border rounded-md bg-gray-50">
<h3 class="text-md font-medium mb-4">Database Mappings</h3>
{#if fetchingDbs}
<p>Loading databases and suggestions...</p>
{:else if sourceDatabases.length > 0}
<MappingTable
{sourceDatabases}
{targetDatabases}
{mappings}
{suggestions}
on:update={handleMappingUpdate}
/>
{:else if sourceEnvId && targetEnvId}
<button
on:click={fetchDatabases}
class="text-indigo-600 hover:text-indigo-500 text-sm font-medium"
>
Refresh Databases & Suggestions
</button>
{/if}
</div>
{/if}
<button
on:click={startMigration}
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId}
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId || selectedDashboardIds.length === 0}
class="inline-flex items-center px-4 py-2 border border-transparent text-sm font-medium rounded-md shadow-sm text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 disabled:bg-gray-400"
>
Start Migration

View File

@@ -0,0 +1,12 @@
export interface DashboardMetadata {
id: number;
title: string;
last_modified: string;
status: string;
}
export interface DashboardSelection {
selected_ids: number[];
source_env_id: string;
target_env_id: string;
}

BIN
mappings.db Normal file

Binary file not shown.

View File

@@ -13,6 +13,7 @@ This protocol standardizes the "Semantic Bridge" between the two languages using
2. **Immutability:** Architectural decisions defined in the Module/Component Header are treated as immutable constraints.
3. **Format Compliance:** Output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax for structure.
4. **Logic over Assertion:** Contracts define the *logic flow*. Do not generate explicit `assert` statements unless requested. The code logic itself must inherently satisfy the Pre/Post conditions (e.g., via control flow, guards, or types).
5. **Fractal Complexity:** Modules and functions must adhere to strict size limits (~300 lines/module, ~30-50 lines/function) to maintain semantic focus.
---
@@ -154,15 +155,16 @@ async function updateUserProfile(profileData) {
Logs delineate the agent's internal state.
* **Python:** `logger.info(f"[{ANCHOR_ID}][{STATE}] Msg")`
* **Python:** MUST use a Context Manager (e.g., `with belief_scope("ANCHOR_ID"):`) to ensure state consistency and automatic handling of Entry/Exit/Error states.
* Manual logging (inside scope): `logger.info(f"[{ANCHOR_ID}][{STATE}] Msg")`
* **Svelte/JS:** `console.log(\`[${ANCHOR_ID}][${STATE}] Msg\`)`
**Required States:**
1. `Entry` (Start of block)
2. `Action` (Key business logic)
3. `Coherence:OK` (Logic successfully completed)
4. `Coherence:Failed` (Error handling)
5. `Exit` (End of block)
1. `Entry` (Start of block - Auto-logged by Context Manager)
2. `Action` (Key business logic - Manual log)
3. `Coherence:OK` (Logic successfully completed - Auto-logged by Context Manager)
4. `Coherence:Failed` (Exception/Error - Auto-logged by Context Manager)
5. `Exit` (End of block - Auto-logged by Context Manager)
---

View File

@@ -0,0 +1,36 @@
# Specification Quality Checklist: Migration Plugin Dashboard Grid
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-27
**Feature**: [specs/007-migration-dashboard-grid/spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- The specification clearly defines the UI requirements for the dashboard selection grid.
- "Superset API" is mentioned as the source of truth, which is acceptable as it defines the data boundary.
- Success criteria include specific performance metrics (<200ms filtering).

View File

@@ -0,0 +1,58 @@
# API Contracts: Migration Dashboard Grid
## Endpoints
### 1. List Dashboards
**Method**: `GET`
**Path**: `/api/environments/{env_id}/dashboards`
**Purpose**: Fetch all dashboards from the specified environment for the grid.
**Request Parameters**:
- `env_id` (path): The ID of the environment to fetch from.
**Response**:
- **200 OK**:
```json
[
{
"id": 123,
"title": "Sales Dashboard",
"last_modified": "2023-10-27T10:00:00Z",
"status": "published"
},
{
"id": 124,
"title": "Draft Metrics",
"last_modified": "2023-10-26T15:30:00Z",
"status": "draft"
}
]
```
- **404 Not Found**: Environment not found.
- **500 Internal Server Error**: Superset API error.
## Components (Frontend)
### DashboardGrid
**Props**:
- `dashboards`: `DashboardMetadata[]` - List of dashboards to display.
- `selectedIds`: `number[]` - IDs of currently selected dashboards.
**Events**:
- `selectionChanged`: Emitted when selection changes. Payload: `number[]` (new list of selected IDs).
**State**:
- `filterText`: string - Current filter text.
- `currentPage`: number - Current page index (0-based).
- `pageSize`: number - Items per page (default 20).
- `sortColumn`: string - 'title' | 'last_modified' | 'status'.
- `sortDirection`: 'asc' | 'desc'.
## Superset Client Extension
### `get_dashboards_summary`
**Signature**: `def get_dashboards_summary(self) -> List[Dict]`
**Purpose**: Fetches dashboard metadata optimized for the grid.
**Implementation Detail**:
- Calls `GET /api/v1/dashboard/` with query params `q=(columns:!(id,dashboard_title,changed_on_utc,published))`.
- Maps response fields to `DashboardMetadata` schema.

View File

@@ -0,0 +1,25 @@
# Data Model: Migration Dashboard Grid
## Entities
### DashboardMetadata
**Source**: Superset API (`/api/v1/dashboard/`)
**Purpose**: Represents a dashboard available for migration.
| Field | Type | Description | Source Mapping |
|-------|------|-------------|----------------|
| `id` | Integer | Unique identifier | `id` |
| `title` | String | Display name of the dashboard | `dashboard_title` |
| `last_modified` | String (ISO 8601) | Timestamp of last modification | `changed_on_utc` |
| `status` | Enum ('published', 'draft') | Publication status | `published` (boolean) -> 'published'/'draft' |
## Value Objects
### DashboardSelection
**Purpose**: Represents the user's selection of dashboards to migrate.
| Field | Type | Description |
|-------|------|-------------|
| `selected_ids` | List[Integer] | List of dashboard IDs selected for migration |
| `source_env_id` | String | ID of the source environment |
| `target_env_id` | String | ID of the target environment |

View File

@@ -0,0 +1,85 @@
# Implementation Plan: [FEATURE]
**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link]
**Input**: Feature specification from `/specs/[###-feature-name]/spec.md`
**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
## Summary
[Extract from feature spec: primary requirement + technical approach from research]
## Technical Context
**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API
**Storage**: N/A (Superset API integration - read-only for metadata)
**Testing**: pytest (Backend), vitest (Frontend - inferred)
**Target Platform**: Linux server / Containerized
**Project Type**: web application (Backend + Frontend)
**Performance Goals**: Client-side filtering < 200ms for 100+ items
**Constraints**: Must handle large lists via pagination (Client-side). Spec says "Client-side (Fetch all, filter locally)" and "Pagination (e.g., 20 per page)". *RESOLVED: Fetch all, paginate locally.*
**Scale/Scope**: ~100s of dashboards per environment.
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
- [x] **Causal Validity**: Contracts (API/Data Model) defined before implementation.
- [x] **Immutability**: Module headers (`[DEF]`) preserved/added.
- [x] **Semantic Format**: All new code uses `[DEF]` anchors and metadata.
- [x] **Fractal Complexity**: New components (Grid) kept modular; `SupersetClient` extensions are small methods.
**Status**: PASSED
## Project Structure
### Documentation (this feature)
```text
specs/[###-feature]/
├── plan.md # This file (/speckit.plan command output)
├── research.md # Phase 0 output (/speckit.plan command)
├── data-model.md # Phase 1 output (/speckit.plan command)
├── quickstart.md # Phase 1 output (/speckit.plan command)
├── contracts/ # Phase 1 output (/speckit.plan command)
└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
```
### Source Code (repository root)
```text
backend/
├── src/
│ ├── api/
│ │ └── routes/
│ │ └── environments.py # Update to support dashboard fetching
│ ├── core/
│ │ └── superset_client.py # Update to fetch extended dashboard metadata
│ └── models/
│ └── dashboard.py # New model for Dashboard metadata
└── tests/
└── test_superset_client.py
frontend/
├── src/
│ ├── components/
│ │ ├── DashboardGrid.svelte # New component
│ │ └── Pagination.svelte # New component (if not exists)
│ ├── routes/
│ │ └── migration/
│ │ └── +page.svelte # Update to use DashboardGrid
│ └── types/
│ └── dashboard.ts # New type definitions
```
**Structure Decision**: Standard Web Application structure. Backend updates to `SupersetClient` and API routes to serve dashboard metadata. Frontend updates to include a new `DashboardGrid` component and integrate it into the migration flow.
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |

View File

@@ -0,0 +1,31 @@
# Quickstart: Migration Dashboard Grid
## Prerequisites
- Backend running (`uvicorn backend.src.app:app --reload`)
- Frontend running (`npm run dev`)
- Superset instance accessible and configured in `config.yaml`
## Steps to Verify
1. **Navigate to Migration Page**:
- Open browser to `http://localhost:5173/migration`
- Select a Source Environment from the dropdown.
2. **Verify Dashboard Grid**:
- The grid should appear below the environment selectors.
- It should list dashboards with columns: Title, Last Modified, Status.
- Status pills should be green (Published) or gray (Draft).
3. **Test Filtering**:
- Type in the "Search dashboards..." input.
- The list should filter instantly (client-side).
4. **Test Pagination**:
- If >20 dashboards, check pagination controls at the bottom.
- Navigate to next page.
5. **Test Selection**:
- Select a few dashboards.
- Change filter (hide selected).
- Clear filter -> Selection should persist.
- Click "Select All" -> Should select all matching current filter.

View File

@@ -0,0 +1,48 @@
# Research: Migration Dashboard Grid
## Unknowns & Clarifications
### 1. Pagination vs Client-side Filtering
**Context**: The spec mentions "Client-side (Fetch all, filter locally)" (FR-004) but also "Pagination (e.g., 20 per page)" (FR-008).
**Resolution**:
- We will fetch ALL dashboard metadata from the Superset API in one go. The metadata (ID, Title, Status, Date) is lightweight. Even for 1000 dashboards, the payload is small (~100KB).
- **Client-side Pagination**: We will implement pagination purely on the frontend. This satisfies "Pagination" for UI performance/usability while keeping the "Fetch all" requirement for fast filtering.
- **Decision**: Fetch all, paginate locally.
### 2. Superset API for Dashboard Metadata
**Context**: Need to fetch `title`, `changed_on`, `published`.
**Research**:
- Superset API endpoint: `/api/v1/dashboard/`
- Standard response includes `result` array with `dashboard_title`, `changed_on_utc`, `published`.
- **Decision**: Use `GET /api/v1/dashboard/` with `q` parameter to select specific columns to minimize payload.
- Columns: `id`, `dashboard_title`, `changed_on_utc`, `published`.
### 3. Grid Component
**Context**: Need a grid with sorting, filtering, and selection.
**Options**:
- **Custom Svelte Table**: Lightweight, full control.
- **3rd Party Lib (e.g. svelte-headless-table)**: Powerful but maybe overkill.
- **Decision**: **Custom Svelte Component** (`DashboardGrid.svelte`).
- Why: Requirements are specific (Select All across pages, custom status pill, specific columns). A custom component using standard HTML table + Tailwind is simple and maintainable for this scope.
## Design Decisions
### Data Model
- **Dashboard**:
- `id`: string (or int, depends on Superset version, usually int for dashboards but we treat as ID)
- `title`: string
- `last_modified`: string (ISO date)
- `status`: 'published' | 'draft'
### Architecture
- **Backend**:
- `SupersetClient.get_dashboards()`: Fetches list from Superset.
- `GET /api/environments/{id}/dashboards`: Proxy endpoint.
- **Frontend**:
- `DashboardGrid.svelte`: Handles display, sorting, pagination, and selection logic.
- `migration/+page.svelte`: Orchestrates fetching and passes data to Grid.
### UX/UI
- **Status Column**: Badge (Green for Published, Gray for Draft).
- **Selection**: Checkbox in first column.
- **Pagination**: Simple "Prev 1 of 5 Next" controls at bottom.

View File

@@ -0,0 +1,81 @@
# Feature Specification: Migration Plugin Dashboard Grid
**Feature Branch**: `007-migration-dashboard-grid`
**Created**: 2025-12-27
**Status**: Draft
**Input**: User description: "Я хочу доработать плагин миграции. Выбор дашбордов должен осуществляться из списка-грида, с возможностью его фильтровать по наименованию. В гриде должны быть поля наименования дашборда, дата последнего изменения дашборда, плюс статус - опубликован или черновик"
## Clarifications
### Session 2025-12-27
- Q: How should the grid handle data loading and filtering to ensure performance and usability? → A: **Client-side** (Fetch all, filter locally).
- Q: Should the grid include a "Select All" checkbox in the header for bulk operations? → A: **Yes, include "Select All"**.
- Q: How should the grid handle large lists of dashboards (e.g., >50)? → A: **Pagination** (e.g., 20 per page).
- Q: Does the "Select All" checkbox select only the currently visible page of dashboards, or all dashboards that match the current filter? → A: **All matching filter** (Selects all filtered results, not just the visible page).
- Q: What should happen if the user changes the filter while some items are already selected? → A: **Preserve selection** (Selected items remain selected even if hidden by new filter).
- Q: What should be the default sort order when the dashboard grid first loads? → A: **Last Modified Date (Newest first)**.
- Q: Should the grid include an "Owners" column to help distinguish dashboards with the same name? → A: **Yes, include Owners**.
- Q: How should the "Owners" column display multiple owners? → A: **Show first owner + count (e.g., "admin + 2") with tooltip**.
- Q: How should the "Status" (Draft/Published) be visually represented in the grid? → A: **Colored Badges/Chips**.
- Q: Should the grid include a "Preview" action (e.g., link to open the dashboard in Superset)? → A: **Yes, open in new tab**.
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Advanced Dashboard Selection (Priority: P1)
As a migration engineer, I want to select dashboards from a detailed grid view that includes status and modification dates, so that I can easily distinguish between draft/published versions and identify the most recent changes before migrating.
**Why this priority**: Current selection mechanisms (likely simple dropdowns or lists) lack critical context (status, freshness), making it error-prone to select the right assets for migration.
**Independent Test**: Can be tested by connecting to a Superset instance with known dashboards (some drafts, some published) and verifying the grid correctly displays their metadata and allows filtering/selection.
**Acceptance Scenarios**:
1. **Given** I have selected a source environment in the migration plugin, **When** the dashboard list loads, **Then** I see a grid view displaying "Dashboard Name", "Last Modified", and "Status" columns.
2. **Given** the dashboard grid is displayed, **When** I type "Sales" into the filter input, **Then** the grid updates to show only dashboards containing "Sales" in their name.
3. **Given** a dashboard is in "Draft" state in Superset, **When** it appears in the grid, **Then** the Status column clearly indicates "Draft" (vs "Published").
4. **Given** I want to migrate multiple dashboards, **When** I check the boxes next to several rows, **Then** they are added to the selection for the migration job.
5. **Given** the grid is populated, **When** I click the "Select All" checkbox in the header, **Then** all visible dashboards are selected.
---
### Edge Cases
- **Empty Environment**: What happens if the source environment has no dashboards? System should display a "No dashboards found" message in the grid area.
- **Missing Metadata**: What if the Superset API returns null for `changed_on` or `published`? System should display "N/A" or a default value (e.g., "Unknown") rather than crashing.
- **Large Dataset**: How does the grid handle 1000+ dashboards? The grid MUST use pagination (default 20 items per page) to manage display density.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST fetch extended metadata for dashboards from the Superset API, specifically: Title, Last Modified Date (`changed_on`), and Published Status (`published`).
- **FR-002**: The Migration Plugin UI MUST display a data grid component to list these dashboards.
- **FR-003**: The grid MUST include sortable columns for:
- Name (Dashboard Title)
- Last Modified (Date/Time)
- Status (Published/Draft)
- Owners (List of owner names)
- **FR-004**: The UI MUST provide a text filter input that filters the grid rows by Dashboard Name in real-time using client-side logic (fetching all dashboards once).
- **FR-005**: The grid MUST support multi-row selection to allow migrating batches of dashboards.
- **FR-006**: The selection state MUST be passed to the migration execution logic when the user initiates the migration.
- **FR-007**: The grid header MUST include a "Select All" checkbox. When checked, it MUST select ALL dashboards matching the current filter criteria (spanning across all pages), not just the currently visible page.
- **FR-008**: The grid MUST support pagination, displaying 20 rows per page by default, with navigation controls (Next/Prev/Page numbers).
- **FR-009**: The selection state MUST be preserved across filter changes. Items selected before a filter change MUST remain selected even if they are hidden by the new filter.
### Key Entities
- **Dashboard Metadata**:
- `id`: Unique identifier from Superset.
- `title`: Display name.
- `changed_on`: Timestamp of last edit.
- `is_published`: Boolean status.
- `owners`: List of owner objects/names.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Users can identify the status (Draft/Published) of any dashboard in the list with 100% accuracy.
- **SC-002**: Filtering a list of 100 dashboards takes less than 200ms to update the view.
- **SC-003**: Users can successfully select and initiate migration for a mix of Draft and Published dashboards in a single operation.

View File

@@ -0,0 +1,29 @@
---
description: "Architecture tasks for Migration Plugin Dashboard Grid"
---
# Architecture Tasks: Migration Plugin Dashboard Grid
**Role**: Architect Agent
**Goal**: Define the "What" and "Why" (Contracts, Scaffolding, Models) before implementation.
## Phase 1: Setup & Models
- [x] A001 Define contracts/scaffolding for migration route in backend/src/api/routes/migration.py
- [x] A002 Define contracts/scaffolding for Dashboard model in backend/src/models/dashboard.py
## Phase 2: User Story 1 - Advanced Dashboard Selection
- [x] A003 [US1] Define contracts/scaffolding for SupersetClient extensions in backend/src/core/superset_client.py
- [x] A004 [US1] Define contracts/scaffolding for GET /api/migration/dashboards endpoint in backend/src/api/routes/migration.py
- [x] A005 [US1] Define contracts/scaffolding for DashboardGrid component in frontend/src/components/DashboardGrid.svelte
- [x] A006 [US1] Define contracts/scaffolding for migration page integration in frontend/src/routes/migration/+page.svelte
- [x] A007 [US1] Define contracts/scaffolding for POST /api/migration/execute endpoint in backend/src/api/routes/migration.py
## Handover Checklist
- [x] All new files created with `[DEF]` anchors
- [x] All functions/classes have `@PURPOSE`, `@PRE`, `@POST` tags
- [x] No "naked code" (logic outside of anchors)
- [x] `tasks-dev.md` is ready for the Developer Agent

View File

@@ -0,0 +1,49 @@
---
description: "Development tasks for Migration Plugin Dashboard Grid"
---
# Development Tasks: Migration Plugin Dashboard Grid
**Role**: Developer Agent
**Goal**: Implement the logic defined in the architecture contracts.
## Phase 1: Backend Implementation
- [x] D001 [US1] Implement `SupersetClient.get_dashboards_summary` in `backend/src/core/superset_client.py`
- **Context**: Fetch dashboards from Superset API with specific columns (`id`, `dashboard_title`, `changed_on_utc`, `published`).
- **Input**: None (uses instance config).
- **Output**: List of dictionaries mapped to `DashboardMetadata` fields.
- [x] D002 [US1] Implement `get_dashboards` endpoint in `backend/src/api/routes/migration.py`
- **Context**: Initialize `SupersetClient` with environment config and call `get_dashboards_summary`.
- **Input**: `env_id` (path param).
- **Output**: JSON list of `DashboardMetadata`.
- [x] D003 [US1] Implement `execute_migration` endpoint in `backend/src/api/routes/migration.py`
- **Context**: Validate selection and initiate migration task (placeholder or TaskManager integration).
- **Input**: `DashboardSelection` body.
- **Output**: Task ID and status message.
## Phase 2: Frontend Implementation
- [x] D004 [US1] Implement `DashboardGrid.svelte` logic
- **Context**: `frontend/src/components/DashboardGrid.svelte`
- **Requirements**:
- Client-side pagination (default 20 items).
- Sorting by Title, Last Modified, Status.
- Text filtering (search by title).
- Multi-selection with "Select All" capability.
- Emit selection events.
- [x] D005 [US1] Integrate `DashboardGrid` into Migration Page
- **Context**: `frontend/src/routes/migration/+page.svelte`
- **Requirements**:
- Fetch dashboards when `sourceEnvId` changes.
- Bind `dashboards` data to `DashboardGrid`.
- Bind `selectedDashboardIds`.
- Update `startMigration` to send `selectedDashboardIds` to backend.
## Phase 3: Verification
- [ ] D006 Verify Dashboard Grid functionality (Sort, Filter, Page).
- [ ] D007 Verify API integration (Fetch dashboards, Start migration).

View File

@@ -20,7 +20,7 @@ from .utils.logger import SupersetLogger
class SupersetConfig(BaseModel):
env: str = Field(..., description="Название окружения (например, dev, prod).")
base_url: str = Field(..., description="Базовый URL Superset API, включая /api/v1.")
auth: Dict[str, str] = Field(..., description="Словарь с данными для аутентификации (provider, username, password, refresh).")
auth: Dict[str, Any] = Field(..., description="Словарь с данными для аутентификации (provider, username, password, refresh).")
verify_ssl: bool = Field(True, description="Флаг для проверки SSL-сертификатов.")
timeout: int = Field(30, description="Таймаут в секундах для HTTP-запросов.")
logger: Optional[SupersetLogger] = Field(None, description="Экземпляр логгера для логирования.")
@@ -32,7 +32,7 @@ class SupersetConfig(BaseModel):
# @THROW: ValueError - Если отсутствуют обязательные поля.
# @PARAM: v (Dict[str, str]) - Значение поля auth.
@validator('auth')
def validate_auth(cls, v: Dict[str, str]) -> Dict[str, str]:
def validate_auth(cls, v: Dict[str, Any]) -> Dict[str, Any]:
required = {'provider', 'username', 'password', 'refresh'}
if not required.issubset(v.keys()):
raise ValueError(f"Словарь 'auth' должен содержать поля: {required}. Отсутствующие: {required - v.keys()}")

188
test_migration_debug.py Normal file
View File

@@ -0,0 +1,188 @@
#!/usr/bin/env python3
"""
Debug script to test the migration flow and identify where the issue occurs.
"""
import sys
import os
from pathlib import Path
# Add project root to sys.path
project_root = Path(__file__).parent
sys.path.insert(0, str(project_root))
def test_plugin_loader():
"""Test if the plugin loader can find the migration plugin."""
print("=== Testing Plugin Loader ===")
try:
from backend.src.core.plugin_loader import PluginLoader
from pathlib import Path
plugin_dir = Path(__file__).parent / "backend" / "src" / "plugins"
print(f"Plugin directory: {plugin_dir}")
print(f"Directory exists: {plugin_dir.exists()}")
if plugin_dir.exists():
print(f"Files in plugin directory: {list(plugin_dir.iterdir())}")
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
configs = plugin_loader.get_all_plugin_configs()
print(f"Found {len(configs)} plugins:")
for config in configs:
print(f" - {config.name} (ID: {config.id})")
migration_plugin = plugin_loader.get_plugin("superset-migration")
if migration_plugin:
print("✓ Migration plugin found")
print(f" Name: {migration_plugin.name}")
print(f" Description: {migration_plugin.description}")
print(f" Schema: {migration_plugin.get_schema()}")
else:
print("✗ Migration plugin NOT found")
return migration_plugin is not None
except Exception as e:
print(f"✗ Plugin loader test failed: {e}")
import traceback
traceback.print_exc()
return False
def test_task_manager():
"""Test if the task manager can create tasks."""
print("\n=== Testing Task Manager ===")
try:
from backend.src.core.plugin_loader import PluginLoader
from backend.src.core.task_manager import TaskManager
from pathlib import Path
plugin_dir = Path(__file__).parent / "backend" / "src" / "plugins"
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
task_manager = TaskManager(plugin_loader)
# Test task creation
test_params = {
"from_env": "dev",
"to_env": "prod",
"dashboard_regex": ".*",
"replace_db_config": False
}
print(f"Creating test task with params: {test_params}")
import asyncio
task = asyncio.run(task_manager.create_task("superset-migration", test_params))
print(f"✓ Task created successfully: {task.id}")
print(f" Status: {task.status}")
print(f" Plugin ID: {task.plugin_id}")
return True
except Exception as e:
print(f"✗ Task manager test failed: {e}")
import traceback
traceback.print_exc()
return False
def test_migration_endpoint():
"""Test the migration endpoint directly."""
print("\n=== Testing Migration Endpoint ===")
try:
from backend.src.api.routes.migration import execute_migration
from backend.src.models.dashboard import DashboardSelection
from backend.src.dependencies import get_config_manager, get_task_manager
# Create a test selection
selection = DashboardSelection(
selected_ids=[1, 2, 3],
source_env_id="ss2",
target_env_id="ss1"
)
print(f"Testing migration with selection: {selection.dict()}")
# This would normally be called by FastAPI with dependencies
# For testing, we'll call it directly
config_manager = get_config_manager()
task_manager = get_task_manager()
import asyncio
# We need to ensure TaskManager uses the SAME loop as run_and_wait
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# Re-initialize TaskManager with the new loop
task_manager.loop = loop
async def run_and_wait():
result = await execute_migration(selection, config_manager, task_manager)
print(f"✓ Migration endpoint test successful: {result}")
task_id = result["task_id"]
print(f"Waiting for task {task_id} to complete...")
for i in range(20):
await asyncio.sleep(1)
task = task_manager.get_task(task_id)
print(f" [{i}] Task status: {task.status}")
if task.status in ["SUCCESS", "FAILED"]:
print(f"Task finished with status: {task.status}")
for log in task.logs:
print(f" [{log.level}] {log.message}")
return task.status == "SUCCESS"
print("Task timed out. Current status: " + task_manager.get_task(task_id).status)
return False
try:
return loop.run_until_complete(run_and_wait())
finally:
loop.close()
except Exception as e:
print(f"✗ Migration endpoint test failed: {e}")
import traceback
traceback.print_exc()
return False
def main():
"""Run all tests."""
print("Migration Debug Test Script")
print("=" * 50)
results = []
# Test 1: Plugin Loader
results.append(("Plugin Loader", test_plugin_loader()))
# Test 2: Task Manager
results.append(("Task Manager", test_task_manager()))
# Test 3: Migration Endpoint
results.append(("Migration Endpoint", test_migration_endpoint()))
# Summary
print("\n" + "=" * 50)
print("TEST RESULTS SUMMARY:")
print("=" * 50)
passed = 0
for test_name, result in results:
status = "PASS" if result else "FAIL"
print(f"{test_name}: {status}")
if result:
passed += 1
print(f"\nTotal: {passed}/{len(results)} tests passed")
if passed == len(results):
print("✓ All tests passed! The migration system appears to be working correctly.")
else:
print("✗ Some tests failed. Check the logs above for details.")
return passed == len(results)
if __name__ == "__main__":
success = main()
sys.exit(0 if success else 1)