Compare commits
9 Commits
006-config
...
007-migrat
| Author | SHA1 | Date | |
|---|---|---|---|
| 077daa0245 | |||
| d38cda09dd | |||
| 1a893c0bc0 | |||
| 40ed375aa4 | |||
| 5fdc92fcdf | |||
| e83328b4ff | |||
| 8da906738b | |||
| d5a1c0e091 | |||
| ef7a0fcf92 |
@@ -9,6 +9,10 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
|
||||
- Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic (005-fix-ui-ws-validation)
|
||||
- N/A (Configuration based) (005-fix-ui-ws-validation)
|
||||
- Filesystem (plugins, logs, backups), SQLite (optional, for job history if needed) (005-fix-ui-ws-validation)
|
||||
- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS (007-migration-dashboard-grid)
|
||||
- N/A (Superset API integration) (007-migration-dashboard-grid)
|
||||
- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API (007-migration-dashboard-grid)
|
||||
- N/A (Superset API integration - read-only for metadata) (007-migration-dashboard-grid)
|
||||
|
||||
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
|
||||
|
||||
@@ -29,9 +33,9 @@ cd src; pytest; ruff check .
|
||||
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
|
||||
|
||||
## Recent Changes
|
||||
- 006-configurable-belief-logs: Added Python 3.9+ + FastAPI (Backend), Pydantic (Config), Svelte (Frontend)
|
||||
- 005-fix-ui-ws-validation: Added Python 3.9+ (Backend), Node.js 18+ (Frontend Build)
|
||||
- 005-fix-ui-ws-validation: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic
|
||||
- 007-migration-dashboard-grid: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API
|
||||
- 007-migration-dashboard-grid: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS
|
||||
- 007-migration-dashboard-grid: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
|
||||
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
|
||||
27
.kilocodemodes
Normal file
27
.kilocodemodes
Normal file
@@ -0,0 +1,27 @@
|
||||
customModes:
|
||||
- slug: tech-lead
|
||||
name: Tech Lead
|
||||
description: Architect for contracts and scaffolding
|
||||
roleDefinition: >-
|
||||
You are Kilo Code, acting as a Technical Lead and System Architect.
|
||||
|
||||
Your primary responsibility is to define the "Structure" and "Contracts" of the system before implementation, following the Semantic Code Generation Protocol.
|
||||
|
||||
You operate primarily on 'tasks-arch.md' task lists.
|
||||
|
||||
YOUR DUTIES:
|
||||
1. Create new files and directory structures.
|
||||
2. Define Modules, Classes, and Functions using `[DEF]` anchors.
|
||||
3. Write clear Headers with `@PURPOSE`, `@LAYER`, `@RELATION`.
|
||||
4. Define strict Contracts using `@PRE`, `@POST`, `@PARAM`, `@RETURN`.
|
||||
5. Leave the implementation body empty or with a placeholder (e.g., `pass`, `return ...`).
|
||||
|
||||
YOU DO NOT WRITE BUSINESS LOGIC. Your output is the "Skeleton" and "Rules" that the Developer Agent will fill in.
|
||||
whenToUse: >-
|
||||
Use this mode during the "Architecture Phase" of a feature. Select this mode when you need to create new files, define API surfaces, or set up the project structure before coding begins.
|
||||
groups:
|
||||
- read
|
||||
- edit
|
||||
- command
|
||||
- list_files
|
||||
- search_files
|
||||
@@ -1,14 +1,14 @@
|
||||
<!--
|
||||
SYNC IMPACT REPORT
|
||||
Version: 1.1.0 (Svelte Support)
|
||||
Version: 1.5.0 (Fractal Complexity Limit)
|
||||
Changes:
|
||||
- Added Svelte Component semantic markup standards.
|
||||
- Updated File Structure Standards to include `.svelte` files.
|
||||
- Refined File Structure Standards to distinguish between Python Modules and Svelte Components.
|
||||
- Added Section VI (Fractal Complexity Limit) to enforce strict module (~300 lines) and function (~30-50 lines) size limits.
|
||||
- Aims to maintain semantic coherence and avoid "Attention Sink".
|
||||
Templates Status:
|
||||
- .specify/templates/plan-template.md: ⚠ Pending (Needs update to include Component headers in checks).
|
||||
- .specify/templates/plan-template.md: ✅ Aligned.
|
||||
- .specify/templates/spec-template.md: ✅ Aligned.
|
||||
- .specify/templates/tasks-template.md: ⚠ Pending (Needs update to include Component definition tasks).
|
||||
- .specify/templates/tasks-arch-template.md: ✅ Aligned (New role-based split).
|
||||
- .specify/templates/tasks-dev-template.md: ✅ Aligned (New role-based split).
|
||||
-->
|
||||
# Semantic Code Generation Constitution
|
||||
|
||||
@@ -21,13 +21,31 @@ Semantic definitions (Contracts) must ALWAYS precede implementation code. Logic
|
||||
Once defined, architectural decisions in the Module Header (`@LAYER`, `@INVARIANT`, `@CONSTRAINT`) are treated as immutable constraints for that module. Changes to these require an explicit refactoring step, not ad-hoc modification during implementation.
|
||||
|
||||
### III. Semantic Format Compliance
|
||||
All output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax with specific Metadata Tags (`@KEY`) and Graph Relations (`@RELATION`). This structure is non-negotiable as it ensures the codebase remains machine-readable, fractal-structured, and optimized for Sparse Attention navigation by AI agents.
|
||||
All output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax with specific Metadata Tags (`@KEY`) and Graph Relations (`@RELATION`). **Crucially, the closing anchor must strictly match the full content of the opening anchor (e.g., `[DEF:module_name:Module]` must close with `[/DEF:module_name:Module]`).**
|
||||
|
||||
**Standardized Graph Relations**
|
||||
To ensure the integrity of the Semantic Graph, `@RELATION` must use a strict taxonomy:
|
||||
- `DEPENDS_ON` (Structural dependency)
|
||||
- `CALLS` (Flow control)
|
||||
- `CREATES` (Instantiation)
|
||||
- `INHERITS_FROM` / `IMPLEMENTS` (OOP hierarchy)
|
||||
- `READS_STATE` / `WRITES_STATE` (Data flow)
|
||||
- `DISPATCHES` / `HANDLES` (Event flow)
|
||||
|
||||
Ad-hoc relationships are forbidden. This structure is non-negotiable as it ensures the codebase remains machine-readable, fractal-structured, and optimized for Sparse Attention navigation by AI agents.
|
||||
|
||||
### IV. Design by Contract (DbC)
|
||||
Contracts are the Source of Truth. Functions and Classes must define their purpose, specifications, and constraints (`@PRE`, `@POST`, `@THROW`) in the metadata block before implementation. Implementation must strictly satisfy these contracts.
|
||||
|
||||
### V. Belief State Logging
|
||||
Logs must define the agent's internal state for debugging and coherence checks. We use a strict format: `logger.level(f"[{ANCHOR_ID}][{STATE}] {MESSAGE} context={...}")` to track transitions between `Entry`, `Validation`, `Action`, and `Coherence` states.
|
||||
Logs must define the agent's internal state for debugging and coherence checks. We use a strict format: `[{ANCHOR_ID}][{STATE}] {MESSAGE}`. For Python, a **Context Manager** pattern MUST be used to automatically handle `Entry`, `Exit`, and `Coherence` states, ensuring structural integrity and error capturing.
|
||||
|
||||
### VI. Fractal Complexity Limit
|
||||
To maintain semantic coherence and avoid "Attention Sink" issues:
|
||||
- **Module Size**: If a Module body exceeds ~300 lines (or logical complexity), it MUST be refactored into sub-modules or a package structure.
|
||||
- **Function Size**: Functions should fit within a standard attention "chunk" (approx. 30-50 lines). If larger, logic MUST be decomposed into helper functions with their own contracts.
|
||||
|
||||
This ensures every vector embedding remains sharp and focused.
|
||||
|
||||
## File Structure Standards
|
||||
|
||||
@@ -51,11 +69,24 @@ Every `.svelte` file must start with a Component definition header (`[DEF:Compon
|
||||
- `@INVARIANT`: Immutable UI/State rules.
|
||||
|
||||
## Generation Workflow
|
||||
The development process follows a strict sequence:
|
||||
1. **Analyze Request**: Identify target module and graph position.
|
||||
2. **Define Structure**: Generate `[DEF]` anchors and Contracts FIRST.
|
||||
3. **Implement Logic**: Write code satisfying Contracts.
|
||||
4. **Validate**: If logic conflicts with Contract -> Stop -> Report Error.
|
||||
The development process follows a strict sequence enforced by Agent Roles:
|
||||
|
||||
### 1. Architecture Phase (Mode: `tech-lead`)
|
||||
**Input**: `tasks-arch.md`
|
||||
**Responsibility**:
|
||||
- Analyze request and graph position.
|
||||
- Generate `[DEF]` anchors, Headers, and Contracts (`@PRE`, `@POST`).
|
||||
- **Output**: Scaffolding files with no implementation logic.
|
||||
|
||||
### 2. Implementation Phase (Mode: `code`)
|
||||
**Input**: `tasks-dev.md` + Scaffolding files
|
||||
**Responsibility**:
|
||||
- Read contracts defined by Architect.
|
||||
- Write implementation code that strictly satisfies contracts.
|
||||
- **Output**: Working code with passing tests.
|
||||
|
||||
### 3. Validation
|
||||
If logic conflicts with Contract -> Stop -> Report Error.
|
||||
|
||||
## Governance
|
||||
This Constitution establishes the "Semantic Code Generation Protocol" as the supreme law of this repository.
|
||||
@@ -63,6 +94,6 @@ This Constitution establishes the "Semantic Code Generation Protocol" as the sup
|
||||
- **Automated Enforcement**: All code generation tools and agents must parse and validate adherence to the `[DEF]` syntax and Contract requirements.
|
||||
- **Amendments**: Changes to the syntax or core principles require a formal amendment to this Constitution and a corresponding update to the constitution
|
||||
- **Review**: Code reviews must verify that implementation matches the preceding contracts and that no "naked code" exists outside of semantic anchors.
|
||||
- **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure constitutes a build failure.
|
||||
- **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure (including matching closing tags) constitutes a build failure.
|
||||
|
||||
**Version**: 1.1.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-19
|
||||
**Version**: 1.5.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-27
|
||||
|
||||
@@ -9,8 +9,8 @@
|
||||
#
|
||||
# OPTIONS:
|
||||
# --json Output in JSON format
|
||||
# --require-tasks Require tasks.md to exist (for implementation phase)
|
||||
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
# --require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
|
||||
# --include-tasks Include task files in AVAILABLE_DOCS list
|
||||
# --paths-only Only output path variables (no validation)
|
||||
# --help, -h Show help message
|
||||
#
|
||||
@@ -49,8 +49,8 @@ Consolidated prerequisite checking for Spec-Driven Development workflow.
|
||||
|
||||
OPTIONS:
|
||||
--json Output in JSON format
|
||||
--require-tasks Require tasks.md to exist (for implementation phase)
|
||||
--include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
--require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
|
||||
--include-tasks Include task files in AVAILABLE_DOCS list
|
||||
--paths-only Only output path variables (no prerequisite validation)
|
||||
--help, -h Show this help message
|
||||
|
||||
@@ -58,7 +58,7 @@ EXAMPLES:
|
||||
# Check task prerequisites (plan.md required)
|
||||
./check-prerequisites.sh --json
|
||||
|
||||
# Check implementation prerequisites (plan.md + tasks.md required)
|
||||
# Check implementation prerequisites (plan.md + task files required)
|
||||
./check-prerequisites.sh --json --require-tasks --include-tasks
|
||||
|
||||
# Get feature paths only (no validation)
|
||||
@@ -86,15 +86,16 @@ check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
if $PATHS_ONLY; then
|
||||
if $JSON_MODE; then
|
||||
# Minimal JSON paths payload (no validation performed)
|
||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
|
||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
|
||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS_ARCH":"%s","TASKS_DEV":"%s"}\n' \
|
||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS_ARCH" "$TASKS_DEV"
|
||||
else
|
||||
echo "REPO_ROOT: $REPO_ROOT"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "FEATURE_DIR: $FEATURE_DIR"
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "TASKS: $TASKS"
|
||||
echo "TASKS_ARCH: $TASKS_ARCH"
|
||||
echo "TASKS_DEV: $TASKS_DEV"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
@@ -112,11 +113,18 @@ if [[ ! -f "$IMPL_PLAN" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for tasks.md if required
|
||||
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
|
||||
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.tasks first to create the task list." >&2
|
||||
exit 1
|
||||
# Check for task files if required
|
||||
if $REQUIRE_TASKS; then
|
||||
if [[ ! -f "$TASKS_ARCH" ]]; then
|
||||
echo "ERROR: tasks-arch.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.tasks first to create the task lists." >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -f "$TASKS_DEV" ]]; then
|
||||
echo "ERROR: tasks-dev.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.tasks first to create the task lists." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Build list of available documents
|
||||
@@ -133,9 +141,10 @@ fi
|
||||
|
||||
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
|
||||
|
||||
# Include tasks.md if requested and it exists
|
||||
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
|
||||
docs+=("tasks.md")
|
||||
# Include task files if requested and they exist
|
||||
if $INCLUDE_TASKS; then
|
||||
[[ -f "$TASKS_ARCH" ]] && docs+=("tasks-arch.md")
|
||||
[[ -f "$TASKS_DEV" ]] && docs+=("tasks-dev.md")
|
||||
fi
|
||||
|
||||
# Output results
|
||||
@@ -161,6 +170,7 @@ else
|
||||
check_file "$QUICKSTART" "quickstart.md"
|
||||
|
||||
if $INCLUDE_TASKS; then
|
||||
check_file "$TASKS" "tasks.md"
|
||||
check_file "$TASKS_ARCH" "tasks-arch.md"
|
||||
check_file "$TASKS_DEV" "tasks-dev.md"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -143,7 +143,9 @@ HAS_GIT='$has_git_repo'
|
||||
FEATURE_DIR='$feature_dir'
|
||||
FEATURE_SPEC='$feature_dir/spec.md'
|
||||
IMPL_PLAN='$feature_dir/plan.md'
|
||||
TASKS='$feature_dir/tasks.md'
|
||||
TASKS_ARCH='$feature_dir/tasks-arch.md'
|
||||
TASKS_DEV='$feature_dir/tasks-dev.md'
|
||||
TASKS='$feature_dir/tasks.md' # Deprecated
|
||||
RESEARCH='$feature_dir/research.md'
|
||||
DATA_MODEL='$feature_dir/data-model.md'
|
||||
QUICKSTART='$feature_dir/quickstart.md'
|
||||
|
||||
35
.specify/templates/tasks-arch-template.md
Normal file
35
.specify/templates/tasks-arch-template.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
|
||||
description: "Architecture task list template (Contracts & Scaffolding)"
|
||||
---
|
||||
|
||||
# Architecture Tasks: [FEATURE NAME]
|
||||
|
||||
**Role**: Architect Agent
|
||||
**Goal**: Define the "What" and "Why" (Contracts, Scaffolding, Models) before implementation.
|
||||
**Input**: Design documents from `/specs/[###-feature-name]/`
|
||||
**Output**: Files with `[DEF]` anchors, `@PRE`/`@POST` contracts, and `@RELATION` mappings. No business logic.
|
||||
|
||||
## Phase 1: Setup & Models
|
||||
|
||||
- [ ] A001 Create/Update data models in [path] with `[DEF]` and contracts
|
||||
- [ ] A002 Define API route structure/contracts in [path]
|
||||
- [ ] A003 Define shared utilities/interfaces
|
||||
|
||||
## Phase 2: User Story 1 - [Title]
|
||||
|
||||
- [ ] A004 [US1] Define contracts for [Component/Service] in [path]
|
||||
- [ ] A005 [US1] Define contracts for [Endpoint] in [path]
|
||||
- [ ] A006 [US1] Define contracts for [Frontend Component] in [path]
|
||||
|
||||
## Phase 3: User Story 2 - [Title]
|
||||
|
||||
- [ ] A007 [US2] Define contracts for [Component/Service] in [path]
|
||||
- [ ] A008 [US2] Define contracts for [Endpoint] in [path]
|
||||
|
||||
## Handover Checklist
|
||||
|
||||
- [ ] All new files created with `[DEF]` anchors
|
||||
- [ ] All functions/classes have `@PURPOSE`, `@PRE`, `@POST` tags
|
||||
- [ ] No "naked code" (logic outside of anchors)
|
||||
- [ ] `tasks-dev.md` is ready for the Developer Agent
|
||||
35
.specify/templates/tasks-dev-template.md
Normal file
35
.specify/templates/tasks-dev-template.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
|
||||
description: "Developer task list template (Implementation Logic)"
|
||||
---
|
||||
|
||||
# Developer Tasks: [FEATURE NAME]
|
||||
|
||||
**Role**: Developer Agent
|
||||
**Goal**: Implement the "How" (Logic, State, Error Handling) inside the defined contracts.
|
||||
**Input**: `tasks-arch.md` (completed), Scaffolding files with `[DEF]` anchors.
|
||||
**Output**: Working code that satisfies `@PRE`/`@POST` conditions.
|
||||
|
||||
## Phase 1: Setup & Models
|
||||
|
||||
- [ ] D001 Implement logic for [Model] in [path]
|
||||
- [ ] D002 Implement logic for [API Route] in [path]
|
||||
- [ ] D003 Implement shared utilities
|
||||
|
||||
## Phase 2: User Story 1 - [Title]
|
||||
|
||||
- [ ] D004 [US1] Implement logic for [Component/Service] in [path]
|
||||
- [ ] D005 [US1] Implement logic for [Endpoint] in [path]
|
||||
- [ ] D006 [US1] Implement logic for [Frontend Component] in [path]
|
||||
- [ ] D007 [US1] Verify semantic compliance and belief state logging
|
||||
|
||||
## Phase 3: User Story 2 - [Title]
|
||||
|
||||
- [ ] D008 [US2] Implement logic for [Component/Service] in [path]
|
||||
- [ ] D009 [US2] Implement logic for [Endpoint] in [path]
|
||||
|
||||
## Polish & Quality Assurance
|
||||
|
||||
- [ ] DXXX Verify all tests pass
|
||||
- [ ] DXXX Check error handling and edge cases
|
||||
- [ ] DXXX Ensure code style compliance
|
||||
@@ -1,251 +0,0 @@
|
||||
---
|
||||
|
||||
description: "Task list template for feature implementation"
|
||||
---
|
||||
|
||||
# Tasks: [FEATURE NAME]
|
||||
|
||||
**Input**: Design documents from `/specs/[###-feature-name]/`
|
||||
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/
|
||||
|
||||
**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification.
|
||||
|
||||
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
|
||||
|
||||
## Format: `[ID] [P?] [Story] Description`
|
||||
|
||||
- **[P]**: Can run in parallel (different files, no dependencies)
|
||||
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
|
||||
- Include exact file paths in descriptions
|
||||
|
||||
## Path Conventions
|
||||
|
||||
- **Single project**: `src/`, `tests/` at repository root
|
||||
- **Web app**: `backend/src/`, `frontend/src/`
|
||||
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
|
||||
- Paths shown below assume single project - adjust based on plan.md structure
|
||||
|
||||
<!--
|
||||
============================================================================
|
||||
IMPORTANT: The tasks below are SAMPLE TASKS for illustration purposes only.
|
||||
|
||||
The /speckit.tasks command MUST replace these with actual tasks based on:
|
||||
- User stories from spec.md (with their priorities P1, P2, P3...)
|
||||
- Feature requirements from plan.md
|
||||
- Entities from data-model.md
|
||||
- Endpoints from contracts/
|
||||
|
||||
Tasks MUST be organized by user story so each story can be:
|
||||
- Implemented independently
|
||||
- Tested independently
|
||||
- Delivered as an MVP increment
|
||||
|
||||
DO NOT keep these sample tasks in the generated tasks.md file.
|
||||
============================================================================
|
||||
-->
|
||||
|
||||
## Phase 1: Setup (Shared Infrastructure)
|
||||
|
||||
**Purpose**: Project initialization and basic structure
|
||||
|
||||
- [ ] T001 Create project structure per implementation plan
|
||||
- [ ] T002 Initialize [language] project with [framework] dependencies
|
||||
- [ ] T003 [P] Configure linting and formatting tools
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Foundational (Blocking Prerequisites)
|
||||
|
||||
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
|
||||
|
||||
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
|
||||
|
||||
Examples of foundational tasks (adjust based on your project):
|
||||
|
||||
- [ ] T004 Setup database schema and migrations framework
|
||||
- [ ] T005 [P] Implement authentication/authorization framework
|
||||
- [ ] T006 [P] Setup API routing and middleware structure
|
||||
- [ ] T007 Create base models/entities that all stories depend on
|
||||
- [ ] T008 Configure error handling and logging infrastructure
|
||||
- [ ] T009 Setup environment configuration management
|
||||
|
||||
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: User Story 1 - [Title] (Priority: P1) 🎯 MVP
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
> **NOTE: Write these tests FIRST, ensure they FAIL before implementation**
|
||||
|
||||
- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 1
|
||||
|
||||
- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py
|
||||
- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py
|
||||
- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013)
|
||||
- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
- [ ] T016 [US1] Add validation and error handling
|
||||
- [ ] T017 [US1] Add logging for user story 1 operations
|
||||
|
||||
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: User Story 2 - [Title] (Priority: P2)
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 2 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
- [ ] T018 [P] [US2] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T019 [P] [US2] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 2
|
||||
|
||||
- [ ] T020 [P] [US2] Create [Entity] model in src/models/[entity].py
|
||||
- [ ] T021 [US2] Implement [Service] in src/services/[service].py
|
||||
- [ ] T022 [US2] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
- [ ] T023 [US2] Integrate with User Story 1 components (if needed)
|
||||
|
||||
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: User Story 3 - [Title] (Priority: P3)
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 3 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
- [ ] T024 [P] [US3] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T025 [P] [US3] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T026 [P] [US3] Create [Entity] model in src/models/[entity].py
|
||||
- [ ] T027 [US3] Implement [Service] in src/services/[service].py
|
||||
- [ ] T028 [US3] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
|
||||
**Checkpoint**: All user stories should now be independently functional
|
||||
|
||||
---
|
||||
|
||||
[Add more user story phases as needed, following the same pattern]
|
||||
|
||||
---
|
||||
|
||||
## Phase N: Polish & Cross-Cutting Concerns
|
||||
|
||||
**Purpose**: Improvements that affect multiple user stories
|
||||
|
||||
- [ ] TXXX [P] Documentation updates in docs/
|
||||
- [ ] TXXX Code cleanup and refactoring
|
||||
- [ ] TXXX Performance optimization across all stories
|
||||
- [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/
|
||||
- [ ] TXXX Security hardening
|
||||
- [ ] TXXX Run quickstart.md validation
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Execution Order
|
||||
|
||||
### Phase Dependencies
|
||||
|
||||
- **Setup (Phase 1)**: No dependencies - can start immediately
|
||||
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
|
||||
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
|
||||
- User stories can then proceed in parallel (if staffed)
|
||||
- Or sequentially in priority order (P1 → P2 → P3)
|
||||
- **Polish (Final Phase)**: Depends on all desired user stories being complete
|
||||
|
||||
### User Story Dependencies
|
||||
|
||||
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
|
||||
- **User Story 2 (P2)**: Can start after Foundational (Phase 2) - May integrate with US1 but should be independently testable
|
||||
- **User Story 3 (P3)**: Can start after Foundational (Phase 2) - May integrate with US1/US2 but should be independently testable
|
||||
|
||||
### Within Each User Story
|
||||
|
||||
- Tests (if included) MUST be written and FAIL before implementation
|
||||
- Models before services
|
||||
- Services before endpoints
|
||||
- Core implementation before integration
|
||||
- Story complete before moving to next priority
|
||||
|
||||
### Parallel Opportunities
|
||||
|
||||
- All Setup tasks marked [P] can run in parallel
|
||||
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
|
||||
- Once Foundational phase completes, all user stories can start in parallel (if team capacity allows)
|
||||
- All tests for a user story marked [P] can run in parallel
|
||||
- Models within a story marked [P] can run in parallel
|
||||
- Different user stories can be worked on in parallel by different team members
|
||||
|
||||
---
|
||||
|
||||
## Parallel Example: User Story 1
|
||||
|
||||
```bash
|
||||
# Launch all tests for User Story 1 together (if tests requested):
|
||||
Task: "Contract test for [endpoint] in tests/contract/test_[name].py"
|
||||
Task: "Integration test for [user journey] in tests/integration/test_[name].py"
|
||||
|
||||
# Launch all models for User Story 1 together:
|
||||
Task: "Create [Entity1] model in src/models/[entity1].py"
|
||||
Task: "Create [Entity2] model in src/models/[entity2].py"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### MVP First (User Story 1 Only)
|
||||
|
||||
1. Complete Phase 1: Setup
|
||||
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
|
||||
3. Complete Phase 3: User Story 1
|
||||
4. **STOP and VALIDATE**: Test User Story 1 independently
|
||||
5. Deploy/demo if ready
|
||||
|
||||
### Incremental Delivery
|
||||
|
||||
1. Complete Setup + Foundational → Foundation ready
|
||||
2. Add User Story 1 → Test independently → Deploy/Demo (MVP!)
|
||||
3. Add User Story 2 → Test independently → Deploy/Demo
|
||||
4. Add User Story 3 → Test independently → Deploy/Demo
|
||||
5. Each story adds value without breaking previous stories
|
||||
|
||||
### Parallel Team Strategy
|
||||
|
||||
With multiple developers:
|
||||
|
||||
1. Team completes Setup + Foundational together
|
||||
2. Once Foundational is done:
|
||||
- Developer A: User Story 1
|
||||
- Developer B: User Story 2
|
||||
- Developer C: User Story 3
|
||||
3. Stories complete and integrate independently
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- [P] tasks = different files, no dependencies
|
||||
- [Story] label maps task to specific user story for traceability
|
||||
- Each user story should be independently completable and testable
|
||||
- Verify tests fail before implementing
|
||||
- Commit after each task or logical group
|
||||
- Stop at any checkpoint to validate story independently
|
||||
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence
|
||||
@@ -15,7 +15,6 @@ from backend.src.dependencies import get_config_manager
|
||||
from backend.src.core.superset_client import SupersetClient
|
||||
from superset_tool.models import SupersetConfig
|
||||
from pydantic import BaseModel
|
||||
from backend.src.core.logger import logger
|
||||
# [/SECTION]
|
||||
|
||||
router = APIRouter(prefix="/api/environments", tags=["environments"])
|
||||
@@ -39,9 +38,7 @@ class DatabaseResponse(BaseModel):
|
||||
# @RETURN: List[EnvironmentResponse]
|
||||
@router.get("", response_model=List[EnvironmentResponse])
|
||||
async def get_environments(config_manager=Depends(get_config_manager)):
|
||||
logger.info(f"[get_environments][Debug] Config path: {config_manager.config_path}")
|
||||
envs = config_manager.get_environments()
|
||||
logger.info(f"[get_environments][Debug] Found {len(envs)} environments")
|
||||
return [EnvironmentResponse(id=e.id, name=e.name, url=e.url) for e in envs]
|
||||
# [/DEF:get_environments]
|
||||
|
||||
|
||||
71
backend/src/api/routes/migration.py
Normal file
71
backend/src/api/routes/migration.py
Normal file
@@ -0,0 +1,71 @@
|
||||
# [DEF:backend.src.api.routes.migration:Module]
|
||||
# @SEMANTICS: api, migration, dashboards
|
||||
# @PURPOSE: API endpoints for migration operations.
|
||||
# @LAYER: API
|
||||
# @RELATION: DEPENDS_ON -> backend.src.dependencies
|
||||
# @RELATION: DEPENDS_ON -> backend.src.models.dashboard
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from typing import List, Dict
|
||||
from backend.src.dependencies import get_config_manager, get_task_manager
|
||||
from backend.src.models.dashboard import DashboardMetadata, DashboardSelection
|
||||
from backend.src.core.superset_client import SupersetClient
|
||||
from superset_tool.models import SupersetConfig
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["migration"])
|
||||
|
||||
# [DEF:get_dashboards:Function]
|
||||
# @PURPOSE: Fetch all dashboards from the specified environment for the grid.
|
||||
# @PRE: Environment ID must be valid.
|
||||
# @POST: Returns a list of dashboard metadata.
|
||||
# @PARAM: env_id (str) - The ID of the environment to fetch from.
|
||||
# @RETURN: List[DashboardMetadata]
|
||||
@router.get("/environments/{env_id}/dashboards", response_model=List[DashboardMetadata])
|
||||
async def get_dashboards(env_id: str, config_manager=Depends(get_config_manager)):
|
||||
environments = config_manager.get_environments()
|
||||
env = next((e for e in environments if e.id == env_id), None)
|
||||
if not env:
|
||||
raise HTTPException(status_code=404, detail="Environment not found")
|
||||
|
||||
config = SupersetConfig(
|
||||
env=env.name,
|
||||
base_url=env.url,
|
||||
auth={'provider': 'db', 'username': env.username, 'password': env.password, 'refresh': False},
|
||||
verify_ssl=True,
|
||||
timeout=30
|
||||
)
|
||||
client = SupersetClient(config)
|
||||
dashboards = client.get_dashboards_summary()
|
||||
return dashboards
|
||||
# [/DEF:get_dashboards]
|
||||
|
||||
# [DEF:execute_migration:Function]
|
||||
# @PURPOSE: Execute the migration of selected dashboards.
|
||||
# @PRE: Selection must be valid and environments must exist.
|
||||
# @POST: Starts the migration task and returns the task ID.
|
||||
# @PARAM: selection (DashboardSelection) - The dashboards to migrate.
|
||||
# @RETURN: Dict - {"task_id": str, "message": str}
|
||||
@router.post("/migration/execute")
|
||||
async def execute_migration(selection: DashboardSelection, config_manager=Depends(get_config_manager), task_manager=Depends(get_task_manager)):
|
||||
# Validate environments exist
|
||||
environments = config_manager.get_environments()
|
||||
env_ids = {e.id for e in environments}
|
||||
if selection.source_env_id not in env_ids or selection.target_env_id not in env_ids:
|
||||
raise HTTPException(status_code=400, detail="Invalid source or target environment")
|
||||
|
||||
# Create migration task with debug logging
|
||||
from ...core.logger import logger
|
||||
logger.info(f"Creating migration task with selection: {selection.dict()}")
|
||||
logger.info(f"Available environments: {env_ids}")
|
||||
logger.info(f"Source env: {selection.source_env_id}, Target env: {selection.target_env_id}")
|
||||
|
||||
try:
|
||||
task = await task_manager.create_task("superset-migration", selection.dict())
|
||||
logger.info(f"Task created successfully: {task.id}")
|
||||
return {"task_id": task.id, "message": "Migration initiated"}
|
||||
except Exception as e:
|
||||
logger.error(f"Task creation failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create migration task: {str(e)}")
|
||||
# [/DEF:execute_migration]
|
||||
|
||||
# [/DEF:backend.src.api.routes.migration]
|
||||
@@ -20,7 +20,7 @@ import os
|
||||
|
||||
from .dependencies import get_task_manager
|
||||
from .core.logger import logger
|
||||
from .api.routes import plugins, tasks, settings, environments, mappings
|
||||
from .api.routes import plugins, tasks, settings, environments, mappings, migration
|
||||
from .core.database import init_db
|
||||
|
||||
# Initialize database
|
||||
@@ -51,6 +51,7 @@ app.include_router(tasks.router, prefix="/api/tasks", tags=["Tasks"])
|
||||
app.include_router(settings.router, prefix="/api/settings", tags=["Settings"])
|
||||
app.include_router(environments.router)
|
||||
app.include_router(mappings.router)
|
||||
app.include_router(migration.router)
|
||||
|
||||
# [DEF:WebSocketEndpoint:Endpoint]
|
||||
# @SEMANTICS: websocket, logs, streaming, real-time
|
||||
|
||||
@@ -15,6 +15,8 @@ import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict
|
||||
from .logger import logger, belief_scope
|
||||
import yaml
|
||||
# [/SECTION]
|
||||
|
||||
# [DEF:MigrationEngine:Class]
|
||||
@@ -26,37 +28,51 @@ class MigrationEngine:
|
||||
# @PARAM: zip_path (str) - Path to the source ZIP file.
|
||||
# @PARAM: output_path (str) - Path where the transformed ZIP will be saved.
|
||||
# @PARAM: db_mapping (Dict[str, str]) - Mapping of source UUID to target UUID.
|
||||
# @PARAM: strip_databases (bool) - Whether to remove the databases directory from the archive.
|
||||
# @RETURN: bool - True if successful.
|
||||
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str]) -> bool:
|
||||
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str], strip_databases: bool = True) -> bool:
|
||||
"""
|
||||
Transform a Superset export ZIP by replacing database UUIDs.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir_str:
|
||||
temp_dir = Path(temp_dir_str)
|
||||
with belief_scope("MigrationEngine.transform_zip"):
|
||||
with tempfile.TemporaryDirectory() as temp_dir_str:
|
||||
temp_dir = Path(temp_dir_str)
|
||||
|
||||
try:
|
||||
# 1. Extract
|
||||
with zipfile.ZipFile(zip_path, 'r') as zf:
|
||||
zf.extractall(temp_dir)
|
||||
try:
|
||||
# 1. Extract
|
||||
logger.info(f"[MigrationEngine.transform_zip][Action] Extracting ZIP: {zip_path}")
|
||||
with zipfile.ZipFile(zip_path, 'r') as zf:
|
||||
zf.extractall(temp_dir)
|
||||
|
||||
# 2. Transform YAMLs
|
||||
# Datasets are usually in datasets/*.yaml
|
||||
dataset_files = list(temp_dir.glob("**/datasets/*.yaml"))
|
||||
for ds_file in dataset_files:
|
||||
self._transform_yaml(ds_file, db_mapping)
|
||||
# 2. Transform YAMLs
|
||||
# Datasets are usually in datasets/*.yaml
|
||||
dataset_files = list(temp_dir.glob("**/datasets/**/*.yaml")) + list(temp_dir.glob("**/datasets/*.yaml"))
|
||||
dataset_files = list(set(dataset_files))
|
||||
|
||||
logger.info(f"[MigrationEngine.transform_zip][State] Found {len(dataset_files)} dataset files.")
|
||||
for ds_file in dataset_files:
|
||||
logger.info(f"[MigrationEngine.transform_zip][Action] Transforming dataset: {ds_file}")
|
||||
self._transform_yaml(ds_file, db_mapping)
|
||||
|
||||
# 3. Re-package
|
||||
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
|
||||
for root, dirs, files in os.walk(temp_dir):
|
||||
for file in files:
|
||||
file_path = Path(root) / file
|
||||
arcname = file_path.relative_to(temp_dir)
|
||||
zf.write(file_path, arcname)
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error transforming ZIP: {e}")
|
||||
return False
|
||||
# 3. Re-package
|
||||
logger.info(f"[MigrationEngine.transform_zip][Action] Re-packaging ZIP to: {output_path} (strip_databases={strip_databases})")
|
||||
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
|
||||
for root, dirs, files in os.walk(temp_dir):
|
||||
rel_root = Path(root).relative_to(temp_dir)
|
||||
|
||||
if strip_databases and "databases" in rel_root.parts:
|
||||
logger.info(f"[MigrationEngine.transform_zip][Action] Skipping file in databases directory: {rel_root}")
|
||||
continue
|
||||
|
||||
for file in files:
|
||||
file_path = Path(root) / file
|
||||
arcname = file_path.relative_to(temp_dir)
|
||||
zf.write(file_path, arcname)
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"[MigrationEngine.transform_zip][Coherence:Failed] Error transforming ZIP: {e}")
|
||||
return False
|
||||
|
||||
# [DEF:MigrationEngine._transform_yaml:Function]
|
||||
# @PURPOSE: Replaces database_uuid in a single YAML file.
|
||||
|
||||
@@ -47,12 +47,17 @@ class PluginLoader:
|
||||
Loads a single Python module and extracts PluginBase subclasses.
|
||||
"""
|
||||
# Try to determine the correct package prefix based on how the app is running
|
||||
if "backend.src" in __name__:
|
||||
# For standalone execution, we need to handle the import differently
|
||||
if __name__ == "__main__" or "test" in __name__:
|
||||
# When running as standalone or in tests, use relative import
|
||||
package_name = f"plugins.{module_name}"
|
||||
elif "backend.src" in __name__:
|
||||
package_prefix = "backend.src.plugins"
|
||||
package_name = f"{package_prefix}.{module_name}"
|
||||
else:
|
||||
package_prefix = "src.plugins"
|
||||
package_name = f"{package_prefix}.{module_name}"
|
||||
|
||||
package_name = f"{package_prefix}.{module_name}"
|
||||
# print(f"DEBUG: Loading plugin {module_name} as {package_name}")
|
||||
spec = importlib.util.spec_from_file_location(package_name, file_path)
|
||||
if spec is None or spec.loader is None:
|
||||
@@ -106,9 +111,11 @@ class PluginLoader:
|
||||
# validate(instance={}, schema=schema)
|
||||
self._plugins[plugin_id] = plugin_instance
|
||||
self._plugin_configs[plugin_id] = plugin_config
|
||||
print(f"Plugin '{plugin_instance.name}' (ID: {plugin_id}) loaded successfully.") # Replace with proper logging
|
||||
from ..core.logger import logger
|
||||
logger.info(f"Plugin '{plugin_instance.name}' (ID: {plugin_id}) loaded successfully.")
|
||||
except Exception as e:
|
||||
print(f"Error validating plugin '{plugin_instance.name}' (ID: {plugin_id}): {e}") # Replace with proper logging
|
||||
from ..core.logger import logger
|
||||
logger.error(f"Error validating plugin '{plugin_instance.name}' (ID: {plugin_id}): {e}")
|
||||
|
||||
|
||||
def get_plugin(self, plugin_id: str) -> Optional[PluginBase]:
|
||||
|
||||
@@ -11,19 +11,12 @@
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
from superset_tool.client import SupersetClient as BaseSupersetClient
|
||||
from superset_tool.models import SupersetConfig
|
||||
from backend.src.core.logger import logger
|
||||
from superset_tool.utils.logger import SupersetLogger
|
||||
# [/SECTION]
|
||||
|
||||
# [DEF:SupersetClient:Class]
|
||||
# @PURPOSE: Extended SupersetClient for migration-specific operations.
|
||||
class SupersetClient(BaseSupersetClient):
|
||||
def __init__(self, config: SupersetConfig):
|
||||
# Initialize with the application's logger wrapped in SupersetLogger
|
||||
# to ensure BeliefFormatter is used.
|
||||
sl_logger = SupersetLogger(logger=logger)
|
||||
super().__init__(config=config, logger=sl_logger)
|
||||
|
||||
|
||||
# [DEF:SupersetClient.get_databases_summary:Function]
|
||||
# @PURPOSE: Fetch a summary of databases including uuid, name, and engine.
|
||||
# @POST: Returns a list of database dictionaries with 'engine' field.
|
||||
@@ -59,6 +52,32 @@ class SupersetClient(BaseSupersetClient):
|
||||
return databases[0] if databases else None
|
||||
# [/DEF:SupersetClient.get_database_by_uuid]
|
||||
|
||||
# [DEF:SupersetClient.get_dashboards_summary:Function]
|
||||
# @PURPOSE: Fetches dashboard metadata optimized for the grid.
|
||||
# @POST: Returns a list of dashboard dictionaries.
|
||||
# @RETURN: List[Dict]
|
||||
def get_dashboards_summary(self) -> List[Dict]:
|
||||
"""
|
||||
Fetches dashboard metadata optimized for the grid.
|
||||
Returns a list of dictionaries mapped to DashboardMetadata fields.
|
||||
"""
|
||||
query = {
|
||||
"columns": ["id", "dashboard_title", "changed_on_utc", "published"]
|
||||
}
|
||||
_, dashboards = self.get_dashboards(query=query)
|
||||
|
||||
# Map fields to DashboardMetadata schema
|
||||
result = []
|
||||
for dash in dashboards:
|
||||
result.append({
|
||||
"id": dash.get("id"),
|
||||
"title": dash.get("dashboard_title"),
|
||||
"last_modified": dash.get("changed_on_utc"),
|
||||
"status": "published" if dash.get("published") else "draft"
|
||||
})
|
||||
return result
|
||||
# [/DEF:SupersetClient.get_dashboards_summary]
|
||||
|
||||
# [/DEF:SupersetClient]
|
||||
|
||||
# [/DEF:backend.src.core.superset_client]
|
||||
|
||||
@@ -64,7 +64,10 @@ class TaskManager:
|
||||
self.tasks: Dict[str, Task] = {}
|
||||
self.subscribers: Dict[str, List[asyncio.Queue]] = {}
|
||||
self.executor = ThreadPoolExecutor(max_workers=5) # For CPU-bound plugin execution
|
||||
self.loop = asyncio.get_event_loop()
|
||||
try:
|
||||
self.loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
self.loop = asyncio.get_event_loop()
|
||||
self.task_futures: Dict[str, asyncio.Future] = {}
|
||||
# [/DEF]
|
||||
|
||||
@@ -72,17 +75,25 @@ class TaskManager:
|
||||
"""
|
||||
Creates and queues a new task for execution.
|
||||
"""
|
||||
from ..core.logger import logger
|
||||
logger.info(f"TaskManager: Creating task for plugin '{plugin_id}' with params: {params}")
|
||||
|
||||
if not self.plugin_loader.has_plugin(plugin_id):
|
||||
logger.error(f"TaskManager: Plugin with ID '{plugin_id}' not found.")
|
||||
raise ValueError(f"Plugin with ID '{plugin_id}' not found.")
|
||||
|
||||
plugin = self.plugin_loader.get_plugin(plugin_id)
|
||||
logger.info(f"TaskManager: Found plugin '{plugin.name}' for task creation")
|
||||
|
||||
# Validate params against plugin schema (this will be done at a higher level, e.g., API route)
|
||||
# For now, a basic check
|
||||
if not isinstance(params, dict):
|
||||
logger.error("TaskManager: Task parameters must be a dictionary.")
|
||||
raise ValueError("Task parameters must be a dictionary.")
|
||||
|
||||
task = Task(plugin_id=plugin_id, params=params, user_id=user_id)
|
||||
self.tasks[task.id] = task
|
||||
logger.info(f"TaskManager: Task {task.id} created and scheduled for execution")
|
||||
self.loop.create_task(self._run_task(task.id)) # Schedule task for execution
|
||||
return task
|
||||
|
||||
@@ -90,9 +101,11 @@ class TaskManager:
|
||||
"""
|
||||
Internal method to execute a task.
|
||||
"""
|
||||
from ..core.logger import logger
|
||||
task = self.tasks[task_id]
|
||||
plugin = self.plugin_loader.get_plugin(task.plugin_id)
|
||||
|
||||
logger.info(f"TaskManager: Starting execution of task {task_id} for plugin '{plugin.name}'")
|
||||
task.status = TaskStatus.RUNNING
|
||||
task.started_at = datetime.utcnow()
|
||||
self._add_log(task_id, "INFO", f"Task started for plugin '{plugin.name}'")
|
||||
@@ -103,17 +116,27 @@ class TaskManager:
|
||||
# If the plugin's execute method is already async, this can be simplified.
|
||||
# Pass task_id to plugin so it can signal pause
|
||||
params = {**task.params, "_task_id": task_id}
|
||||
await self.loop.run_in_executor(
|
||||
self.executor,
|
||||
lambda: asyncio.run(plugin.execute(params)) if asyncio.iscoroutinefunction(plugin.execute) else plugin.execute(params)
|
||||
)
|
||||
logger.info(f"TaskManager: Executing plugin '{plugin.name}' with params: {params}")
|
||||
if asyncio.iscoroutinefunction(plugin.execute):
|
||||
logger.info(f"TaskManager: Executing async plugin '{plugin.name}'")
|
||||
await plugin.execute(params)
|
||||
else:
|
||||
logger.info(f"TaskManager: Executing sync plugin '{plugin.name}' in executor")
|
||||
await self.loop.run_in_executor(
|
||||
self.executor,
|
||||
plugin.execute,
|
||||
params
|
||||
)
|
||||
logger.info(f"TaskManager: Task {task_id} completed successfully for plugin '{plugin.name}'")
|
||||
task.status = TaskStatus.SUCCESS
|
||||
self._add_log(task_id, "INFO", f"Task completed successfully for plugin '{plugin.name}'")
|
||||
except Exception as e:
|
||||
logger.error(f"TaskManager: Task {task_id} failed for plugin '{plugin.name}': {e}")
|
||||
task.status = TaskStatus.FAILED
|
||||
self._add_log(task_id, "ERROR", f"Task failed: {e}", {"error_type": type(e).__name__})
|
||||
finally:
|
||||
task.finished_at = datetime.utcnow()
|
||||
logger.info(f"TaskManager: Task {task_id} execution finished with status: {task.status}")
|
||||
# In a real system, you might notify clients via WebSocket here
|
||||
|
||||
async def resolve_task(self, task_id: str, resolution_params: Dict[str, Any]):
|
||||
|
||||
@@ -21,7 +21,12 @@ def get_config_manager() -> ConfigManager:
|
||||
|
||||
plugin_dir = Path(__file__).parent / "plugins"
|
||||
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
|
||||
from .core.logger import logger
|
||||
logger.info(f"PluginLoader initialized with directory: {plugin_dir}")
|
||||
logger.info(f"Available plugins: {[config.name for config in plugin_loader.get_all_plugin_configs()]}")
|
||||
|
||||
task_manager = TaskManager(plugin_loader)
|
||||
logger.info("TaskManager initialized")
|
||||
|
||||
def get_plugin_loader() -> PluginLoader:
|
||||
"""Dependency injector for the PluginLoader."""
|
||||
|
||||
27
backend/src/models/dashboard.py
Normal file
27
backend/src/models/dashboard.py
Normal file
@@ -0,0 +1,27 @@
|
||||
# [DEF:backend.src.models.dashboard:Module]
|
||||
# @SEMANTICS: dashboard, model, metadata, migration
|
||||
# @PURPOSE: Defines data models for dashboard metadata and selection.
|
||||
# @LAYER: Model
|
||||
# @RELATION: USED_BY -> backend.src.api.routes.migration
|
||||
|
||||
from pydantic import BaseModel
|
||||
from typing import List
|
||||
|
||||
# [DEF:DashboardMetadata:Class]
|
||||
# @PURPOSE: Represents a dashboard available for migration.
|
||||
class DashboardMetadata(BaseModel):
|
||||
id: int
|
||||
title: str
|
||||
last_modified: str
|
||||
status: str
|
||||
# [/DEF:DashboardMetadata]
|
||||
|
||||
# [DEF:DashboardSelection:Class]
|
||||
# @PURPOSE: Represents the user's selection of dashboards to migrate.
|
||||
class DashboardSelection(BaseModel):
|
||||
selected_ids: List[int]
|
||||
source_env_id: str
|
||||
target_env_id: str
|
||||
# [/DEF:DashboardSelection]
|
||||
|
||||
# [/DEF:backend.src.models.dashboard]
|
||||
@@ -87,34 +87,72 @@ class MigrationPlugin(PluginBase):
|
||||
}
|
||||
|
||||
async def execute(self, params: Dict[str, Any]):
|
||||
from_env = params["from_env"]
|
||||
to_env = params["to_env"]
|
||||
dashboard_regex = params["dashboard_regex"]
|
||||
source_env_id = params.get("source_env_id")
|
||||
target_env_id = params.get("target_env_id")
|
||||
selected_ids = params.get("selected_ids")
|
||||
|
||||
# Legacy support or alternative params
|
||||
from_env_name = params.get("from_env")
|
||||
to_env_name = params.get("to_env")
|
||||
dashboard_regex = params.get("dashboard_regex")
|
||||
|
||||
replace_db_config = params.get("replace_db_config", False)
|
||||
from_db_id = params.get("from_db_id")
|
||||
to_db_id = params.get("to_db_id")
|
||||
|
||||
logger = SupersetLogger(log_dir=Path.cwd() / "logs", console=True)
|
||||
logger.info(f"[MigrationPlugin][Entry] Starting migration from {from_env} to {to_env}.")
|
||||
logger.info(f"[MigrationPlugin][Entry] Starting migration task.")
|
||||
logger.info(f"[MigrationPlugin][Action] Params: {params}")
|
||||
|
||||
try:
|
||||
config_manager = get_config_manager()
|
||||
all_clients = setup_clients(logger, custom_envs=config_manager.get_environments())
|
||||
from_c = all_clients.get(from_env)
|
||||
to_c = all_clients.get(to_env)
|
||||
environments = config_manager.get_environments()
|
||||
|
||||
# Resolve environments
|
||||
src_env = None
|
||||
tgt_env = None
|
||||
|
||||
if source_env_id:
|
||||
src_env = next((e for e in environments if e.id == source_env_id), None)
|
||||
elif from_env_name:
|
||||
src_env = next((e for e in environments if e.name == from_env_name), None)
|
||||
|
||||
if target_env_id:
|
||||
tgt_env = next((e for e in environments if e.id == target_env_id), None)
|
||||
elif to_env_name:
|
||||
tgt_env = next((e for e in environments if e.name == to_env_name), None)
|
||||
|
||||
if not src_env or not tgt_env:
|
||||
raise ValueError(f"Could not resolve source or target environment. Source: {source_env_id or from_env_name}, Target: {target_env_id or to_env_name}")
|
||||
|
||||
from_env_name = src_env.name
|
||||
to_env_name = tgt_env.name
|
||||
|
||||
logger.info(f"[MigrationPlugin][State] Resolved environments: {from_env_name} -> {to_env_name}")
|
||||
|
||||
all_clients = setup_clients(logger, custom_envs=environments)
|
||||
from_c = all_clients.get(from_env_name)
|
||||
to_c = all_clients.get(to_env_name)
|
||||
|
||||
if not from_c or not to_c:
|
||||
raise ValueError(f"One or both environments ('{from_env}', '{to_env}') not found in configuration.")
|
||||
raise ValueError(f"Clients not initialized for environments: {from_env_name}, {to_env_name}")
|
||||
|
||||
_, all_dashboards = from_c.get_dashboards()
|
||||
|
||||
regex_str = str(dashboard_regex)
|
||||
dashboards_to_migrate = [
|
||||
d for d in all_dashboards if re.search(regex_str, d["dashboard_title"], re.IGNORECASE)
|
||||
]
|
||||
dashboards_to_migrate = []
|
||||
if selected_ids:
|
||||
dashboards_to_migrate = [d for d in all_dashboards if d["id"] in selected_ids]
|
||||
elif dashboard_regex:
|
||||
regex_str = str(dashboard_regex)
|
||||
dashboards_to_migrate = [
|
||||
d for d in all_dashboards if re.search(regex_str, d["dashboard_title"], re.IGNORECASE)
|
||||
]
|
||||
else:
|
||||
logger.warning("[MigrationPlugin][State] No selection criteria provided (selected_ids or dashboard_regex).")
|
||||
return
|
||||
|
||||
if not dashboards_to_migrate:
|
||||
logger.warning("[MigrationPlugin][State] No dashboards found matching the regex.")
|
||||
logger.warning("[MigrationPlugin][State] No dashboards found matching criteria.")
|
||||
return
|
||||
|
||||
# Fetch mappings from database
|
||||
@@ -123,8 +161,8 @@ class MigrationPlugin(PluginBase):
|
||||
db = SessionLocal()
|
||||
try:
|
||||
# Find environment IDs by name
|
||||
src_env = db.query(Environment).filter(Environment.name == from_env).first()
|
||||
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
|
||||
src_env = db.query(Environment).filter(Environment.name == from_env_name).first()
|
||||
tgt_env = db.query(Environment).filter(Environment.name == to_env_name).first()
|
||||
|
||||
if src_env and tgt_env:
|
||||
mappings = db.query(DatabaseMapping).filter(
|
||||
@@ -144,46 +182,39 @@ class MigrationPlugin(PluginBase):
|
||||
try:
|
||||
exported_content, _ = from_c.export_dashboard(dash_id)
|
||||
with create_temp_file(content=exported_content, dry_run=True, suffix=".zip", logger=logger) as tmp_zip_path:
|
||||
if not replace_db_config:
|
||||
to_c.import_dashboard(file_name=tmp_zip_path, dash_id=dash_id, dash_slug=dash_slug)
|
||||
else:
|
||||
# Check for missing mappings before transformation
|
||||
# This is a simplified check, in reality we'd check all YAMLs
|
||||
# For US3, we'll just use the engine and handle missing ones there
|
||||
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
|
||||
# If we have missing mappings, we might need to pause
|
||||
# For now, let's assume the engine can tell us what's missing
|
||||
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
|
||||
|
||||
if not success:
|
||||
# Signal missing mapping and wait
|
||||
task_id = params.get("_task_id")
|
||||
if task_id:
|
||||
from ..dependencies import get_task_manager
|
||||
tm = get_task_manager()
|
||||
logger.info(f"[MigrationPlugin][Action] Pausing for missing mapping in task {task_id}")
|
||||
# In a real scenario, we'd pass the missing DB info to the frontend
|
||||
# For this task, we'll just simulate the wait
|
||||
await tm.wait_for_resolution(task_id)
|
||||
# After resolution, retry transformation with updated mappings
|
||||
# (Mappings would be updated in task.params by resolve_task)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
src_env = db.query(Environment).filter(Environment.name == from_env).first()
|
||||
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
|
||||
mappings = db.query(DatabaseMapping).filter(
|
||||
DatabaseMapping.source_env_id == src_env.id,
|
||||
DatabaseMapping.target_env_id == tgt_env.id
|
||||
).all()
|
||||
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
|
||||
finally:
|
||||
db.close()
|
||||
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
|
||||
# Always transform to strip databases to avoid password errors
|
||||
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
|
||||
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping, strip_databases=False)
|
||||
|
||||
if not success and replace_db_config:
|
||||
# Signal missing mapping and wait (only if we care about mappings)
|
||||
task_id = params.get("_task_id")
|
||||
if task_id:
|
||||
from ..dependencies import get_task_manager
|
||||
tm = get_task_manager()
|
||||
logger.info(f"[MigrationPlugin][Action] Pausing for missing mapping in task {task_id}")
|
||||
# In a real scenario, we'd pass the missing DB info to the frontend
|
||||
# For this task, we'll just simulate the wait
|
||||
await tm.wait_for_resolution(task_id)
|
||||
# After resolution, retry transformation with updated mappings
|
||||
# (Mappings would be updated in task.params by resolve_task)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
src_env = db.query(Environment).filter(Environment.name == from_env_name).first()
|
||||
tgt_env = db.query(Environment).filter(Environment.name == to_env_name).first()
|
||||
mappings = db.query(DatabaseMapping).filter(
|
||||
DatabaseMapping.source_env_id == src_env.id,
|
||||
DatabaseMapping.target_env_id == tgt_env.id
|
||||
).all()
|
||||
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
|
||||
finally:
|
||||
db.close()
|
||||
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping, strip_databases=False)
|
||||
|
||||
if success:
|
||||
to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
|
||||
else:
|
||||
logger.error(f"[MigrationPlugin][Failure] Failed to transform ZIP for dashboard {title}")
|
||||
if success:
|
||||
to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
|
||||
else:
|
||||
logger.error(f"[MigrationPlugin][Failure] Failed to transform ZIP for dashboard {title}")
|
||||
|
||||
logger.info(f"[MigrationPlugin][Success] Dashboard {title} imported.")
|
||||
except Exception as exc:
|
||||
|
||||
@@ -24,7 +24,7 @@ export const options = {
|
||||
app: ({ head, body, assets, nonce, env }) => "<!DOCTYPE html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<link rel=\"icon\" href=\"" + assets + "/favicon.png\" />\n\t\t<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n\t\t" + head + "\n\t</head>\n\t<body data-sveltekit-preload-data=\"hover\">\n\t\t<div style=\"display: contents\">" + body + "</div>\n\t</body>\n</html>\n",
|
||||
error: ({ status, message }) => "<!doctype html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<title>" + message + "</title>\n\n\t\t<style>\n\t\t\tbody {\n\t\t\t\t--bg: white;\n\t\t\t\t--fg: #222;\n\t\t\t\t--divider: #ccc;\n\t\t\t\tbackground: var(--bg);\n\t\t\t\tcolor: var(--fg);\n\t\t\t\tfont-family:\n\t\t\t\t\tsystem-ui,\n\t\t\t\t\t-apple-system,\n\t\t\t\t\tBlinkMacSystemFont,\n\t\t\t\t\t'Segoe UI',\n\t\t\t\t\tRoboto,\n\t\t\t\t\tOxygen,\n\t\t\t\t\tUbuntu,\n\t\t\t\t\tCantarell,\n\t\t\t\t\t'Open Sans',\n\t\t\t\t\t'Helvetica Neue',\n\t\t\t\t\tsans-serif;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tjustify-content: center;\n\t\t\t\theight: 100vh;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t.error {\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tmax-width: 32rem;\n\t\t\t\tmargin: 0 1rem;\n\t\t\t}\n\n\t\t\t.status {\n\t\t\t\tfont-weight: 200;\n\t\t\t\tfont-size: 3rem;\n\t\t\t\tline-height: 1;\n\t\t\t\tposition: relative;\n\t\t\t\ttop: -0.05rem;\n\t\t\t}\n\n\t\t\t.message {\n\t\t\t\tborder-left: 1px solid var(--divider);\n\t\t\t\tpadding: 0 0 0 1rem;\n\t\t\t\tmargin: 0 0 0 1rem;\n\t\t\t\tmin-height: 2.5rem;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t}\n\n\t\t\t.message h1 {\n\t\t\t\tfont-weight: 400;\n\t\t\t\tfont-size: 1em;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t@media (prefers-color-scheme: dark) {\n\t\t\t\tbody {\n\t\t\t\t\t--bg: #222;\n\t\t\t\t\t--fg: #ddd;\n\t\t\t\t\t--divider: #666;\n\t\t\t\t}\n\t\t\t}\n\t\t</style>\n\t</head>\n\t<body>\n\t\t<div class=\"error\">\n\t\t\t<span class=\"status\">" + status + "</span>\n\t\t\t<div class=\"message\">\n\t\t\t\t<h1>" + message + "</h1>\n\t\t\t</div>\n\t\t</div>\n\t</body>\n</html>\n"
|
||||
},
|
||||
version_hash: "n7gbte"
|
||||
version_hash: "oj9twc"
|
||||
};
|
||||
|
||||
export async function get_hooks() {
|
||||
|
||||
205
frontend/src/components/DashboardGrid.svelte
Normal file
205
frontend/src/components/DashboardGrid.svelte
Normal file
@@ -0,0 +1,205 @@
|
||||
<!-- [DEF:DashboardGrid:Component] -->
|
||||
<!--
|
||||
@SEMANTICS: dashboard, grid, selection, pagination
|
||||
@PURPOSE: Displays a grid of dashboards with selection and pagination.
|
||||
@LAYER: Component
|
||||
@RELATION: USED_BY -> frontend/src/routes/migration/+page.svelte
|
||||
|
||||
@INVARIANT: Selected IDs must be a subset of available dashboards.
|
||||
-->
|
||||
|
||||
<script lang="ts">
|
||||
// [SECTION: IMPORTS]
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
import type { DashboardMetadata } from '../types/dashboard';
|
||||
// [/SECTION]
|
||||
|
||||
// [SECTION: PROPS]
|
||||
export let dashboards: DashboardMetadata[] = [];
|
||||
export let selectedIds: number[] = [];
|
||||
// [/SECTION]
|
||||
|
||||
// [SECTION: STATE]
|
||||
let filterText = "";
|
||||
let currentPage = 0;
|
||||
let pageSize = 20;
|
||||
let sortColumn: keyof DashboardMetadata = "title";
|
||||
let sortDirection: "asc" | "desc" = "asc";
|
||||
// [/SECTION]
|
||||
|
||||
// [SECTION: DERIVED]
|
||||
$: filteredDashboards = dashboards.filter(d =>
|
||||
d.title.toLowerCase().includes(filterText.toLowerCase())
|
||||
);
|
||||
|
||||
$: sortedDashboards = [...filteredDashboards].sort((a, b) => {
|
||||
let aVal = a[sortColumn];
|
||||
let bVal = b[sortColumn];
|
||||
if (sortColumn === "id") {
|
||||
aVal = Number(aVal);
|
||||
bVal = Number(bVal);
|
||||
}
|
||||
if (aVal < bVal) return sortDirection === "asc" ? -1 : 1;
|
||||
if (aVal > bVal) return sortDirection === "asc" ? 1 : -1;
|
||||
return 0;
|
||||
});
|
||||
|
||||
$: paginatedDashboards = sortedDashboards.slice(
|
||||
currentPage * pageSize,
|
||||
(currentPage + 1) * pageSize
|
||||
);
|
||||
|
||||
$: totalPages = Math.ceil(sortedDashboards.length / pageSize);
|
||||
|
||||
$: allSelected = paginatedDashboards.length > 0 && paginatedDashboards.every(d => selectedIds.includes(d.id));
|
||||
$: someSelected = paginatedDashboards.some(d => selectedIds.includes(d.id));
|
||||
// [/SECTION]
|
||||
|
||||
// [SECTION: EVENTS]
|
||||
const dispatch = createEventDispatcher<{ selectionChanged: number[] }>();
|
||||
// [/SECTION]
|
||||
|
||||
// [DEF:handleSort:Function]
|
||||
// @PURPOSE: Toggles sort direction or changes sort column.
|
||||
function handleSort(column: keyof DashboardMetadata) {
|
||||
if (sortColumn === column) {
|
||||
sortDirection = sortDirection === "asc" ? "desc" : "asc";
|
||||
} else {
|
||||
sortColumn = column;
|
||||
sortDirection = "asc";
|
||||
}
|
||||
}
|
||||
// [/DEF:handleSort]
|
||||
|
||||
// [DEF:handleSelectionChange:Function]
|
||||
// @PURPOSE: Handles individual checkbox changes.
|
||||
function handleSelectionChange(id: number, checked: boolean) {
|
||||
let newSelected = [...selectedIds];
|
||||
if (checked) {
|
||||
if (!newSelected.includes(id)) newSelected.push(id);
|
||||
} else {
|
||||
newSelected = newSelected.filter(sid => sid !== id);
|
||||
}
|
||||
selectedIds = newSelected;
|
||||
dispatch('selectionChanged', newSelected);
|
||||
}
|
||||
// [/DEF:handleSelectionChange]
|
||||
|
||||
// [DEF:handleSelectAll:Function]
|
||||
// @PURPOSE: Handles select all checkbox.
|
||||
function handleSelectAll(checked: boolean) {
|
||||
let newSelected = [...selectedIds];
|
||||
if (checked) {
|
||||
paginatedDashboards.forEach(d => {
|
||||
if (!newSelected.includes(d.id)) newSelected.push(d.id);
|
||||
});
|
||||
} else {
|
||||
paginatedDashboards.forEach(d => {
|
||||
newSelected = newSelected.filter(sid => sid !== d.id);
|
||||
});
|
||||
}
|
||||
selectedIds = newSelected;
|
||||
dispatch('selectionChanged', newSelected);
|
||||
}
|
||||
// [/DEF:handleSelectAll]
|
||||
|
||||
// [DEF:goToPage:Function]
|
||||
// @PURPOSE: Changes current page.
|
||||
function goToPage(page: number) {
|
||||
if (page >= 0 && page < totalPages) {
|
||||
currentPage = page;
|
||||
}
|
||||
}
|
||||
// [/DEF:goToPage]
|
||||
|
||||
</script>
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="dashboard-grid">
|
||||
<!-- Filter Input -->
|
||||
<div class="mb-4">
|
||||
<input
|
||||
type="text"
|
||||
bind:value={filterText}
|
||||
placeholder="Search dashboards..."
|
||||
class="w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<!-- Grid/Table -->
|
||||
<div class="overflow-x-auto">
|
||||
<table class="min-w-full bg-white border border-gray-300">
|
||||
<thead class="bg-gray-50">
|
||||
<tr>
|
||||
<th class="px-4 py-2 border-b">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={allSelected}
|
||||
indeterminate={someSelected && !allSelected}
|
||||
on:change={(e) => handleSelectAll((e.target as HTMLInputElement).checked)}
|
||||
/>
|
||||
</th>
|
||||
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('title')}>
|
||||
Title {sortColumn === 'title' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
|
||||
</th>
|
||||
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('last_modified')}>
|
||||
Last Modified {sortColumn === 'last_modified' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
|
||||
</th>
|
||||
<th class="px-4 py-2 border-b cursor-pointer" on:click={() => handleSort('status')}>
|
||||
Status {sortColumn === 'status' ? (sortDirection === 'asc' ? '↑' : '↓') : ''}
|
||||
</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{#each paginatedDashboards as dashboard (dashboard.id)}
|
||||
<tr class="hover:bg-gray-50">
|
||||
<td class="px-4 py-2 border-b">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={selectedIds.includes(dashboard.id)}
|
||||
on:change={(e) => handleSelectionChange(dashboard.id, (e.target as HTMLInputElement).checked)}
|
||||
/>
|
||||
</td>
|
||||
<td class="px-4 py-2 border-b">{dashboard.title}</td>
|
||||
<td class="px-4 py-2 border-b">{new Date(dashboard.last_modified).toLocaleDateString()}</td>
|
||||
<td class="px-4 py-2 border-b">
|
||||
<span class="px-2 py-1 text-xs font-medium rounded-full {dashboard.status === 'published' ? 'bg-green-100 text-green-800' : 'bg-gray-100 text-gray-800'}">
|
||||
{dashboard.status}
|
||||
</span>
|
||||
</td>
|
||||
</tr>
|
||||
{/each}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Pagination Controls -->
|
||||
<div class="flex items-center justify-between mt-4">
|
||||
<div class="text-sm text-gray-700">
|
||||
Showing {currentPage * pageSize + 1} to {Math.min((currentPage + 1) * pageSize, sortedDashboards.length)} of {sortedDashboards.length} dashboards
|
||||
</div>
|
||||
<div class="flex space-x-2">
|
||||
<button
|
||||
class="px-3 py-1 text-sm border border-gray-300 rounded-md hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
disabled={currentPage === 0}
|
||||
on:click={() => goToPage(currentPage - 1)}
|
||||
>
|
||||
Previous
|
||||
</button>
|
||||
<button
|
||||
class="px-3 py-1 text-sm border border-gray-300 rounded-md hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
disabled={currentPage >= totalPages - 1}
|
||||
on:click={() => goToPage(currentPage + 1)}
|
||||
>
|
||||
Next
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- [/SECTION] -->
|
||||
|
||||
<style>
|
||||
/* Component styles */
|
||||
</style>
|
||||
|
||||
<!-- [/DEF:DashboardGrid] -->
|
||||
@@ -36,8 +36,9 @@
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="flex flex-col space-y-1">
|
||||
<label class="text-sm font-medium text-gray-700">{label}</label>
|
||||
<label for="env-select" class="text-sm font-medium text-gray-700">{label}</label>
|
||||
<select
|
||||
id="env-select"
|
||||
class="block w-full pl-3 pr-10 py-2 text-base border-gray-300 focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm rounded-md"
|
||||
value={selectedId}
|
||||
on:change={handleSelect}
|
||||
|
||||
@@ -12,16 +12,26 @@
|
||||
// [SECTION: IMPORTS]
|
||||
import { onMount } from 'svelte';
|
||||
import EnvSelector from '../../components/EnvSelector.svelte';
|
||||
import DashboardGrid from '../../components/DashboardGrid.svelte';
|
||||
import MappingTable from '../../components/MappingTable.svelte';
|
||||
import MissingMappingModal from '../../components/MissingMappingModal.svelte';
|
||||
import type { DashboardMetadata, DashboardSelection } from '../../types/dashboard';
|
||||
// [/SECTION]
|
||||
|
||||
// [SECTION: STATE]
|
||||
let environments = [];
|
||||
let environments: any[] = [];
|
||||
let sourceEnvId = "";
|
||||
let targetEnvId = "";
|
||||
let dashboardRegex = ".*";
|
||||
let replaceDb = false;
|
||||
let loading = true;
|
||||
let error = "";
|
||||
let dashboards: DashboardMetadata[] = [];
|
||||
let selectedDashboardIds: number[] = [];
|
||||
let sourceDatabases: any[] = [];
|
||||
let targetDatabases: any[] = [];
|
||||
let mappings: any[] = [];
|
||||
let suggestions: any[] = [];
|
||||
let fetchingDbs = false;
|
||||
// [/SECTION]
|
||||
|
||||
// [DEF:fetchEnvironments:Function]
|
||||
@@ -42,8 +52,100 @@
|
||||
}
|
||||
// [/DEF:fetchEnvironments]
|
||||
|
||||
// [DEF:fetchDashboards:Function]
|
||||
/**
|
||||
* @purpose Fetches dashboards for the selected source environment.
|
||||
* @param envId The environment ID.
|
||||
* @post dashboards state is updated.
|
||||
*/
|
||||
async function fetchDashboards(envId: string) {
|
||||
try {
|
||||
const response = await fetch(`/api/environments/${envId}/dashboards`);
|
||||
if (!response.ok) throw new Error('Failed to fetch dashboards');
|
||||
dashboards = await response.json();
|
||||
selectedDashboardIds = []; // Reset selection when env changes
|
||||
} catch (e) {
|
||||
error = e.message;
|
||||
dashboards = [];
|
||||
}
|
||||
}
|
||||
// [/DEF:fetchDashboards]
|
||||
|
||||
onMount(fetchEnvironments);
|
||||
|
||||
// Reactive: fetch dashboards when source env changes
|
||||
$: if (sourceEnvId) fetchDashboards(sourceEnvId);
|
||||
|
||||
// [DEF:fetchDatabases:Function]
|
||||
/**
|
||||
* @purpose Fetches databases from both environments and gets suggestions.
|
||||
*/
|
||||
async function fetchDatabases() {
|
||||
if (!sourceEnvId || !targetEnvId) return;
|
||||
fetchingDbs = true;
|
||||
error = "";
|
||||
|
||||
try {
|
||||
const [srcRes, tgtRes, mapRes, sugRes] = await Promise.all([
|
||||
fetch(`/api/environments/${sourceEnvId}/databases`),
|
||||
fetch(`/api/environments/${targetEnvId}/databases`),
|
||||
fetch(`/api/mappings?source_env_id=${sourceEnvId}&target_env_id=${targetEnvId}`),
|
||||
fetch(`/api/mappings/suggest`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ source_env_id: sourceEnvId, target_env_id: targetEnvId })
|
||||
})
|
||||
]);
|
||||
|
||||
if (!srcRes.ok || !tgtRes.ok) throw new Error('Failed to fetch databases from environments');
|
||||
|
||||
sourceDatabases = await srcRes.json();
|
||||
targetDatabases = await tgtRes.json();
|
||||
mappings = await mapRes.json();
|
||||
suggestions = await sugRes.json();
|
||||
} catch (e) {
|
||||
error = e.message;
|
||||
} finally {
|
||||
fetchingDbs = false;
|
||||
}
|
||||
}
|
||||
// [/DEF:fetchDatabases]
|
||||
|
||||
// [DEF:handleMappingUpdate:Function]
|
||||
/**
|
||||
* @purpose Saves a mapping to the backend.
|
||||
*/
|
||||
async function handleMappingUpdate(event: CustomEvent) {
|
||||
const { sourceUuid, targetUuid } = event.detail;
|
||||
const sDb = sourceDatabases.find(d => d.uuid === sourceUuid);
|
||||
const tDb = targetDatabases.find(d => d.uuid === targetUuid);
|
||||
|
||||
if (!sDb || !tDb) return;
|
||||
|
||||
try {
|
||||
const response = await fetch('/api/mappings', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
source_env_id: sourceEnvId,
|
||||
target_env_id: targetEnvId,
|
||||
source_db_uuid: sourceUuid,
|
||||
target_db_uuid: targetUuid,
|
||||
source_db_name: sDb.database_name,
|
||||
target_db_name: tDb.database_name
|
||||
})
|
||||
});
|
||||
|
||||
if (!response.ok) throw new Error('Failed to save mapping');
|
||||
|
||||
const savedMapping = await response.json();
|
||||
mappings = [...mappings.filter(m => m.source_db_uuid !== sourceUuid), savedMapping];
|
||||
} catch (e) {
|
||||
error = e.message;
|
||||
}
|
||||
}
|
||||
// [/DEF:handleMappingUpdate]
|
||||
|
||||
// [DEF:startMigration:Function]
|
||||
/**
|
||||
* @purpose Starts the migration process.
|
||||
@@ -58,10 +160,33 @@
|
||||
error = "Source and target environments must be different.";
|
||||
return;
|
||||
}
|
||||
|
||||
if (selectedDashboardIds.length === 0) {
|
||||
error = "Please select at least one dashboard to migrate.";
|
||||
return;
|
||||
}
|
||||
|
||||
error = "";
|
||||
console.log(`[MigrationDashboard][Action] Starting migration from ${sourceEnvId} to ${targetEnvId} (Replace DB: ${replaceDb})`);
|
||||
// TODO: Implement actual migration trigger in US3
|
||||
try {
|
||||
const selection: DashboardSelection = {
|
||||
selected_ids: selectedDashboardIds,
|
||||
source_env_id: sourceEnvId,
|
||||
target_env_id: targetEnvId
|
||||
};
|
||||
console.log(`[MigrationDashboard][Action] Starting migration with selection:`, selection);
|
||||
const response = await fetch('/api/migration/execute', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(selection)
|
||||
});
|
||||
console.log(`[MigrationDashboard][Action] API response status: ${response.status}`);
|
||||
if (!response.ok) throw new Error(`Failed to start migration: ${response.status} ${response.statusText}`);
|
||||
const result = await response.json();
|
||||
console.log(`[MigrationDashboard][Action] Migration started: ${result.task_id} - ${result.message}`);
|
||||
// TODO: Show success message or redirect to task status
|
||||
} catch (e) {
|
||||
console.error(`[MigrationDashboard][Failure] Migration failed:`, e);
|
||||
error = e.message;
|
||||
}
|
||||
}
|
||||
// [/DEF:startMigration]
|
||||
</script>
|
||||
@@ -91,42 +216,62 @@
|
||||
/>
|
||||
</div>
|
||||
|
||||
<!-- [DEF:DashboardSelectionSection] -->
|
||||
<div class="mb-8">
|
||||
<label for="dashboard-regex" class="block text-sm font-medium text-gray-700 mb-1">Dashboard Regex</label>
|
||||
<h2 class="text-lg font-medium mb-4">Select Dashboards</h2>
|
||||
|
||||
{#if sourceEnvId}
|
||||
<DashboardGrid
|
||||
{dashboards}
|
||||
bind:selectedIds={selectedDashboardIds}
|
||||
/>
|
||||
{:else}
|
||||
<p class="text-gray-500 italic">Select a source environment to view dashboards.</p>
|
||||
{/if}
|
||||
</div>
|
||||
<!-- [/DEF:DashboardSelectionSection] -->
|
||||
|
||||
|
||||
<div class="flex items-center mb-4">
|
||||
<input
|
||||
id="dashboard-regex"
|
||||
type="text"
|
||||
bind:value={dashboardRegex}
|
||||
placeholder="e.g. ^Finance Dashboard$"
|
||||
class="shadow-sm focus:ring-indigo-500 focus:border-indigo-500 block w-full sm:text-sm border-gray-300 rounded-md"
|
||||
id="replace-db"
|
||||
type="checkbox"
|
||||
bind:checked={replaceDb}
|
||||
on:change={() => { if (replaceDb && sourceDatabases.length === 0) fetchDatabases(); }}
|
||||
class="h-4 w-4 text-indigo-600 focus:ring-indigo-500 border-gray-300 rounded"
|
||||
/>
|
||||
<p class="mt-1 text-sm text-gray-500">Regular expression to filter dashboards to migrate.</p>
|
||||
<label for="replace-db" class="ml-2 block text-sm text-gray-900">
|
||||
Replace Database (Apply Mappings)
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center justify-between mb-8">
|
||||
<div class="flex items-center">
|
||||
<input
|
||||
id="replace-db"
|
||||
type="checkbox"
|
||||
bind:checked={replaceDb}
|
||||
class="h-4 w-4 text-indigo-600 focus:ring-indigo-500 border-gray-300 rounded"
|
||||
/>
|
||||
<label for="replace-db" class="ml-2 block text-sm text-gray-900">
|
||||
Replace Database (Apply Mappings)
|
||||
</label>
|
||||
{#if replaceDb}
|
||||
<div class="mb-8 p-4 border rounded-md bg-gray-50">
|
||||
<h3 class="text-md font-medium mb-4">Database Mappings</h3>
|
||||
{#if fetchingDbs}
|
||||
<p>Loading databases and suggestions...</p>
|
||||
{:else if sourceDatabases.length > 0}
|
||||
<MappingTable
|
||||
{sourceDatabases}
|
||||
{targetDatabases}
|
||||
{mappings}
|
||||
{suggestions}
|
||||
on:update={handleMappingUpdate}
|
||||
/>
|
||||
{:else if sourceEnvId && targetEnvId}
|
||||
<button
|
||||
on:click={fetchDatabases}
|
||||
class="text-indigo-600 hover:text-indigo-500 text-sm font-medium"
|
||||
>
|
||||
Refresh Databases & Suggestions
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<a
|
||||
href="/migration/mappings"
|
||||
class="text-sm font-medium text-indigo-600 hover:text-indigo-500"
|
||||
>
|
||||
Manage Mappings →
|
||||
</a>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
<button
|
||||
on:click={startMigration}
|
||||
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId}
|
||||
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId || selectedDashboardIds.length === 0}
|
||||
class="inline-flex items-center px-4 py-2 border border-transparent text-sm font-medium rounded-md shadow-sm text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 disabled:bg-gray-400"
|
||||
>
|
||||
Start Migration
|
||||
|
||||
12
frontend/src/types/dashboard.ts
Normal file
12
frontend/src/types/dashboard.ts
Normal file
@@ -0,0 +1,12 @@
|
||||
export interface DashboardMetadata {
|
||||
id: number;
|
||||
title: string;
|
||||
last_modified: string;
|
||||
status: string;
|
||||
}
|
||||
|
||||
export interface DashboardSelection {
|
||||
selected_ids: number[];
|
||||
source_env_id: string;
|
||||
target_env_id: string;
|
||||
}
|
||||
BIN
mappings.db
Normal file
BIN
mappings.db
Normal file
Binary file not shown.
@@ -13,6 +13,7 @@ This protocol standardizes the "Semantic Bridge" between the two languages using
|
||||
2. **Immutability:** Architectural decisions defined in the Module/Component Header are treated as immutable constraints.
|
||||
3. **Format Compliance:** Output must strictly follow the `[DEF]` / `[/DEF]` anchor syntax for structure.
|
||||
4. **Logic over Assertion:** Contracts define the *logic flow*. Do not generate explicit `assert` statements unless requested. The code logic itself must inherently satisfy the Pre/Post conditions (e.g., via control flow, guards, or types).
|
||||
5. **Fractal Complexity:** Modules and functions must adhere to strict size limits (~300 lines/module, ~30-50 lines/function) to maintain semantic focus.
|
||||
|
||||
---
|
||||
|
||||
@@ -154,15 +155,16 @@ async function updateUserProfile(profileData) {
|
||||
|
||||
Logs delineate the agent's internal state.
|
||||
|
||||
* **Python:** `logger.info(f"[{ANCHOR_ID}][{STATE}] Msg")`
|
||||
* **Python:** MUST use a Context Manager (e.g., `with belief_scope("ANCHOR_ID"):`) to ensure state consistency and automatic handling of Entry/Exit/Error states.
|
||||
* Manual logging (inside scope): `logger.info(f"[{ANCHOR_ID}][{STATE}] Msg")`
|
||||
* **Svelte/JS:** `console.log(\`[${ANCHOR_ID}][${STATE}] Msg\`)`
|
||||
|
||||
**Required States:**
|
||||
1. `Entry` (Start of block)
|
||||
2. `Action` (Key business logic)
|
||||
3. `Coherence:OK` (Logic successfully completed)
|
||||
4. `Coherence:Failed` (Error handling)
|
||||
5. `Exit` (End of block)
|
||||
1. `Entry` (Start of block - Auto-logged by Context Manager)
|
||||
2. `Action` (Key business logic - Manual log)
|
||||
3. `Coherence:OK` (Logic successfully completed - Auto-logged by Context Manager)
|
||||
4. `Coherence:Failed` (Exception/Error - Auto-logged by Context Manager)
|
||||
5. `Exit` (End of block - Auto-logged by Context Manager)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Specification Quality Checklist: Migration Plugin Dashboard Grid
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: 2025-12-27
|
||||
**Feature**: [specs/007-migration-dashboard-grid/spec.md](../spec.md)
|
||||
|
||||
## Content Quality
|
||||
|
||||
- [x] No implementation details (languages, frameworks, APIs)
|
||||
- [x] Focused on user value and business needs
|
||||
- [x] Written for non-technical stakeholders
|
||||
- [x] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
|
||||
- [x] No [NEEDS CLARIFICATION] markers remain
|
||||
- [x] Requirements are testable and unambiguous
|
||||
- [x] Success criteria are measurable
|
||||
- [x] Success criteria are technology-agnostic (no implementation details)
|
||||
- [x] All acceptance scenarios are defined
|
||||
- [x] Edge cases are identified
|
||||
- [x] Scope is clearly bounded
|
||||
- [x] Dependencies and assumptions identified
|
||||
|
||||
## Feature Readiness
|
||||
|
||||
- [x] All functional requirements have clear acceptance criteria
|
||||
- [x] User scenarios cover primary flows
|
||||
- [x] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [x] No implementation details leak into specification
|
||||
|
||||
## Notes
|
||||
|
||||
- The specification clearly defines the UI requirements for the dashboard selection grid.
|
||||
- "Superset API" is mentioned as the source of truth, which is acceptable as it defines the data boundary.
|
||||
- Success criteria include specific performance metrics (<200ms filtering).
|
||||
58
specs/007-migration-dashboard-grid/contracts/api.md
Normal file
58
specs/007-migration-dashboard-grid/contracts/api.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# API Contracts: Migration Dashboard Grid
|
||||
|
||||
## Endpoints
|
||||
|
||||
### 1. List Dashboards
|
||||
**Method**: `GET`
|
||||
**Path**: `/api/environments/{env_id}/dashboards`
|
||||
**Purpose**: Fetch all dashboards from the specified environment for the grid.
|
||||
|
||||
**Request Parameters**:
|
||||
- `env_id` (path): The ID of the environment to fetch from.
|
||||
|
||||
**Response**:
|
||||
- **200 OK**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 123,
|
||||
"title": "Sales Dashboard",
|
||||
"last_modified": "2023-10-27T10:00:00Z",
|
||||
"status": "published"
|
||||
},
|
||||
{
|
||||
"id": 124,
|
||||
"title": "Draft Metrics",
|
||||
"last_modified": "2023-10-26T15:30:00Z",
|
||||
"status": "draft"
|
||||
}
|
||||
]
|
||||
```
|
||||
- **404 Not Found**: Environment not found.
|
||||
- **500 Internal Server Error**: Superset API error.
|
||||
|
||||
## Components (Frontend)
|
||||
|
||||
### DashboardGrid
|
||||
**Props**:
|
||||
- `dashboards`: `DashboardMetadata[]` - List of dashboards to display.
|
||||
- `selectedIds`: `number[]` - IDs of currently selected dashboards.
|
||||
|
||||
**Events**:
|
||||
- `selectionChanged`: Emitted when selection changes. Payload: `number[]` (new list of selected IDs).
|
||||
|
||||
**State**:
|
||||
- `filterText`: string - Current filter text.
|
||||
- `currentPage`: number - Current page index (0-based).
|
||||
- `pageSize`: number - Items per page (default 20).
|
||||
- `sortColumn`: string - 'title' | 'last_modified' | 'status'.
|
||||
- `sortDirection`: 'asc' | 'desc'.
|
||||
|
||||
## Superset Client Extension
|
||||
|
||||
### `get_dashboards_summary`
|
||||
**Signature**: `def get_dashboards_summary(self) -> List[Dict]`
|
||||
**Purpose**: Fetches dashboard metadata optimized for the grid.
|
||||
**Implementation Detail**:
|
||||
- Calls `GET /api/v1/dashboard/` with query params `q=(columns:!(id,dashboard_title,changed_on_utc,published))`.
|
||||
- Maps response fields to `DashboardMetadata` schema.
|
||||
25
specs/007-migration-dashboard-grid/data-model.md
Normal file
25
specs/007-migration-dashboard-grid/data-model.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# Data Model: Migration Dashboard Grid
|
||||
|
||||
## Entities
|
||||
|
||||
### DashboardMetadata
|
||||
**Source**: Superset API (`/api/v1/dashboard/`)
|
||||
**Purpose**: Represents a dashboard available for migration.
|
||||
|
||||
| Field | Type | Description | Source Mapping |
|
||||
|-------|------|-------------|----------------|
|
||||
| `id` | Integer | Unique identifier | `id` |
|
||||
| `title` | String | Display name of the dashboard | `dashboard_title` |
|
||||
| `last_modified` | String (ISO 8601) | Timestamp of last modification | `changed_on_utc` |
|
||||
| `status` | Enum ('published', 'draft') | Publication status | `published` (boolean) -> 'published'/'draft' |
|
||||
|
||||
## Value Objects
|
||||
|
||||
### DashboardSelection
|
||||
**Purpose**: Represents the user's selection of dashboards to migrate.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `selected_ids` | List[Integer] | List of dashboard IDs selected for migration |
|
||||
| `source_env_id` | String | ID of the source environment |
|
||||
| `target_env_id` | String | ID of the target environment |
|
||||
85
specs/007-migration-dashboard-grid/plan.md
Normal file
85
specs/007-migration-dashboard-grid/plan.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# Implementation Plan: [FEATURE]
|
||||
|
||||
**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link]
|
||||
**Input**: Feature specification from `/specs/[###-feature-name]/spec.md`
|
||||
|
||||
**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
|
||||
|
||||
## Summary
|
||||
|
||||
[Extract from feature spec: primary requirement + technical approach from research]
|
||||
|
||||
## Technical Context
|
||||
|
||||
**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
|
||||
**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API
|
||||
**Storage**: N/A (Superset API integration - read-only for metadata)
|
||||
**Testing**: pytest (Backend), vitest (Frontend - inferred)
|
||||
**Target Platform**: Linux server / Containerized
|
||||
**Project Type**: web application (Backend + Frontend)
|
||||
**Performance Goals**: Client-side filtering < 200ms for 100+ items
|
||||
**Constraints**: Must handle large lists via pagination (Client-side). Spec says "Client-side (Fetch all, filter locally)" and "Pagination (e.g., 20 per page)". *RESOLVED: Fetch all, paginate locally.*
|
||||
**Scale/Scope**: ~100s of dashboards per environment.
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
- [x] **Causal Validity**: Contracts (API/Data Model) defined before implementation.
|
||||
- [x] **Immutability**: Module headers (`[DEF]`) preserved/added.
|
||||
- [x] **Semantic Format**: All new code uses `[DEF]` anchors and metadata.
|
||||
- [x] **Fractal Complexity**: New components (Grid) kept modular; `SupersetClient` extensions are small methods.
|
||||
|
||||
**Status**: PASSED
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
|
||||
```text
|
||||
specs/[###-feature]/
|
||||
├── plan.md # This file (/speckit.plan command output)
|
||||
├── research.md # Phase 0 output (/speckit.plan command)
|
||||
├── data-model.md # Phase 1 output (/speckit.plan command)
|
||||
├── quickstart.md # Phase 1 output (/speckit.plan command)
|
||||
├── contracts/ # Phase 1 output (/speckit.plan command)
|
||||
└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
|
||||
```text
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── api/
|
||||
│ │ └── routes/
|
||||
│ │ └── environments.py # Update to support dashboard fetching
|
||||
│ ├── core/
|
||||
│ │ └── superset_client.py # Update to fetch extended dashboard metadata
|
||||
│ └── models/
|
||||
│ └── dashboard.py # New model for Dashboard metadata
|
||||
└── tests/
|
||||
└── test_superset_client.py
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ │ ├── DashboardGrid.svelte # New component
|
||||
│ │ └── Pagination.svelte # New component (if not exists)
|
||||
│ ├── routes/
|
||||
│ │ └── migration/
|
||||
│ │ └── +page.svelte # Update to use DashboardGrid
|
||||
│ └── types/
|
||||
│ └── dashboard.ts # New type definitions
|
||||
```
|
||||
|
||||
**Structure Decision**: Standard Web Application structure. Backend updates to `SupersetClient` and API routes to serve dashboard metadata. Frontend updates to include a new `DashboardGrid` component and integrate it into the migration flow.
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
> **Fill ONLY if Constitution Check has violations that must be justified**
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
|
||||
| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |
|
||||
31
specs/007-migration-dashboard-grid/quickstart.md
Normal file
31
specs/007-migration-dashboard-grid/quickstart.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Quickstart: Migration Dashboard Grid
|
||||
|
||||
## Prerequisites
|
||||
- Backend running (`uvicorn backend.src.app:app --reload`)
|
||||
- Frontend running (`npm run dev`)
|
||||
- Superset instance accessible and configured in `config.yaml`
|
||||
|
||||
## Steps to Verify
|
||||
|
||||
1. **Navigate to Migration Page**:
|
||||
- Open browser to `http://localhost:5173/migration`
|
||||
- Select a Source Environment from the dropdown.
|
||||
|
||||
2. **Verify Dashboard Grid**:
|
||||
- The grid should appear below the environment selectors.
|
||||
- It should list dashboards with columns: Title, Last Modified, Status.
|
||||
- Status pills should be green (Published) or gray (Draft).
|
||||
|
||||
3. **Test Filtering**:
|
||||
- Type in the "Search dashboards..." input.
|
||||
- The list should filter instantly (client-side).
|
||||
|
||||
4. **Test Pagination**:
|
||||
- If >20 dashboards, check pagination controls at the bottom.
|
||||
- Navigate to next page.
|
||||
|
||||
5. **Test Selection**:
|
||||
- Select a few dashboards.
|
||||
- Change filter (hide selected).
|
||||
- Clear filter -> Selection should persist.
|
||||
- Click "Select All" -> Should select all matching current filter.
|
||||
48
specs/007-migration-dashboard-grid/research.md
Normal file
48
specs/007-migration-dashboard-grid/research.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Research: Migration Dashboard Grid
|
||||
|
||||
## Unknowns & Clarifications
|
||||
|
||||
### 1. Pagination vs Client-side Filtering
|
||||
**Context**: The spec mentions "Client-side (Fetch all, filter locally)" (FR-004) but also "Pagination (e.g., 20 per page)" (FR-008).
|
||||
**Resolution**:
|
||||
- We will fetch ALL dashboard metadata from the Superset API in one go. The metadata (ID, Title, Status, Date) is lightweight. Even for 1000 dashboards, the payload is small (~100KB).
|
||||
- **Client-side Pagination**: We will implement pagination purely on the frontend. This satisfies "Pagination" for UI performance/usability while keeping the "Fetch all" requirement for fast filtering.
|
||||
- **Decision**: Fetch all, paginate locally.
|
||||
|
||||
### 2. Superset API for Dashboard Metadata
|
||||
**Context**: Need to fetch `title`, `changed_on`, `published`.
|
||||
**Research**:
|
||||
- Superset API endpoint: `/api/v1/dashboard/`
|
||||
- Standard response includes `result` array with `dashboard_title`, `changed_on_utc`, `published`.
|
||||
- **Decision**: Use `GET /api/v1/dashboard/` with `q` parameter to select specific columns to minimize payload.
|
||||
- Columns: `id`, `dashboard_title`, `changed_on_utc`, `published`.
|
||||
|
||||
### 3. Grid Component
|
||||
**Context**: Need a grid with sorting, filtering, and selection.
|
||||
**Options**:
|
||||
- **Custom Svelte Table**: Lightweight, full control.
|
||||
- **3rd Party Lib (e.g. svelte-headless-table)**: Powerful but maybe overkill.
|
||||
- **Decision**: **Custom Svelte Component** (`DashboardGrid.svelte`).
|
||||
- Why: Requirements are specific (Select All across pages, custom status pill, specific columns). A custom component using standard HTML table + Tailwind is simple and maintainable for this scope.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Data Model
|
||||
- **Dashboard**:
|
||||
- `id`: string (or int, depends on Superset version, usually int for dashboards but we treat as ID)
|
||||
- `title`: string
|
||||
- `last_modified`: string (ISO date)
|
||||
- `status`: 'published' | 'draft'
|
||||
|
||||
### Architecture
|
||||
- **Backend**:
|
||||
- `SupersetClient.get_dashboards()`: Fetches list from Superset.
|
||||
- `GET /api/environments/{id}/dashboards`: Proxy endpoint.
|
||||
- **Frontend**:
|
||||
- `DashboardGrid.svelte`: Handles display, sorting, pagination, and selection logic.
|
||||
- `migration/+page.svelte`: Orchestrates fetching and passes data to Grid.
|
||||
|
||||
### UX/UI
|
||||
- **Status Column**: Badge (Green for Published, Gray for Draft).
|
||||
- **Selection**: Checkbox in first column.
|
||||
- **Pagination**: Simple "Prev 1 of 5 Next" controls at bottom.
|
||||
81
specs/007-migration-dashboard-grid/spec.md
Normal file
81
specs/007-migration-dashboard-grid/spec.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Feature Specification: Migration Plugin Dashboard Grid
|
||||
|
||||
**Feature Branch**: `007-migration-dashboard-grid`
|
||||
**Created**: 2025-12-27
|
||||
**Status**: Draft
|
||||
**Input**: User description: "Я хочу доработать плагин миграции. Выбор дашбордов должен осуществляться из списка-грида, с возможностью его фильтровать по наименованию. В гриде должны быть поля наименования дашборда, дата последнего изменения дашборда, плюс статус - опубликован или черновик"
|
||||
|
||||
## Clarifications
|
||||
|
||||
### Session 2025-12-27
|
||||
- Q: How should the grid handle data loading and filtering to ensure performance and usability? → A: **Client-side** (Fetch all, filter locally).
|
||||
- Q: Should the grid include a "Select All" checkbox in the header for bulk operations? → A: **Yes, include "Select All"**.
|
||||
- Q: How should the grid handle large lists of dashboards (e.g., >50)? → A: **Pagination** (e.g., 20 per page).
|
||||
- Q: Does the "Select All" checkbox select only the currently visible page of dashboards, or all dashboards that match the current filter? → A: **All matching filter** (Selects all filtered results, not just the visible page).
|
||||
- Q: What should happen if the user changes the filter while some items are already selected? → A: **Preserve selection** (Selected items remain selected even if hidden by new filter).
|
||||
- Q: What should be the default sort order when the dashboard grid first loads? → A: **Last Modified Date (Newest first)**.
|
||||
- Q: Should the grid include an "Owners" column to help distinguish dashboards with the same name? → A: **Yes, include Owners**.
|
||||
- Q: How should the "Owners" column display multiple owners? → A: **Show first owner + count (e.g., "admin + 2") with tooltip**.
|
||||
- Q: How should the "Status" (Draft/Published) be visually represented in the grid? → A: **Colored Badges/Chips**.
|
||||
- Q: Should the grid include a "Preview" action (e.g., link to open the dashboard in Superset)? → A: **Yes, open in new tab**.
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - Advanced Dashboard Selection (Priority: P1)
|
||||
|
||||
As a migration engineer, I want to select dashboards from a detailed grid view that includes status and modification dates, so that I can easily distinguish between draft/published versions and identify the most recent changes before migrating.
|
||||
|
||||
**Why this priority**: Current selection mechanisms (likely simple dropdowns or lists) lack critical context (status, freshness), making it error-prone to select the right assets for migration.
|
||||
|
||||
**Independent Test**: Can be tested by connecting to a Superset instance with known dashboards (some drafts, some published) and verifying the grid correctly displays their metadata and allows filtering/selection.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** I have selected a source environment in the migration plugin, **When** the dashboard list loads, **Then** I see a grid view displaying "Dashboard Name", "Last Modified", and "Status" columns.
|
||||
2. **Given** the dashboard grid is displayed, **When** I type "Sales" into the filter input, **Then** the grid updates to show only dashboards containing "Sales" in their name.
|
||||
3. **Given** a dashboard is in "Draft" state in Superset, **When** it appears in the grid, **Then** the Status column clearly indicates "Draft" (vs "Published").
|
||||
4. **Given** I want to migrate multiple dashboards, **When** I check the boxes next to several rows, **Then** they are added to the selection for the migration job.
|
||||
5. **Given** the grid is populated, **When** I click the "Select All" checkbox in the header, **Then** all visible dashboards are selected.
|
||||
|
||||
---
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- **Empty Environment**: What happens if the source environment has no dashboards? System should display a "No dashboards found" message in the grid area.
|
||||
- **Missing Metadata**: What if the Superset API returns null for `changed_on` or `published`? System should display "N/A" or a default value (e.g., "Unknown") rather than crashing.
|
||||
- **Large Dataset**: How does the grid handle 1000+ dashboards? The grid MUST use pagination (default 20 items per page) to manage display density.
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: The system MUST fetch extended metadata for dashboards from the Superset API, specifically: Title, Last Modified Date (`changed_on`), and Published Status (`published`).
|
||||
- **FR-002**: The Migration Plugin UI MUST display a data grid component to list these dashboards.
|
||||
- **FR-003**: The grid MUST include sortable columns for:
|
||||
- Name (Dashboard Title)
|
||||
- Last Modified (Date/Time)
|
||||
- Status (Published/Draft)
|
||||
- Owners (List of owner names)
|
||||
- **FR-004**: The UI MUST provide a text filter input that filters the grid rows by Dashboard Name in real-time using client-side logic (fetching all dashboards once).
|
||||
- **FR-005**: The grid MUST support multi-row selection to allow migrating batches of dashboards.
|
||||
- **FR-006**: The selection state MUST be passed to the migration execution logic when the user initiates the migration.
|
||||
- **FR-007**: The grid header MUST include a "Select All" checkbox. When checked, it MUST select ALL dashboards matching the current filter criteria (spanning across all pages), not just the currently visible page.
|
||||
- **FR-008**: The grid MUST support pagination, displaying 20 rows per page by default, with navigation controls (Next/Prev/Page numbers).
|
||||
- **FR-009**: The selection state MUST be preserved across filter changes. Items selected before a filter change MUST remain selected even if they are hidden by the new filter.
|
||||
|
||||
### Key Entities
|
||||
|
||||
- **Dashboard Metadata**:
|
||||
- `id`: Unique identifier from Superset.
|
||||
- `title`: Display name.
|
||||
- `changed_on`: Timestamp of last edit.
|
||||
- `is_published`: Boolean status.
|
||||
- `owners`: List of owner objects/names.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: Users can identify the status (Draft/Published) of any dashboard in the list with 100% accuracy.
|
||||
- **SC-002**: Filtering a list of 100 dashboards takes less than 200ms to update the view.
|
||||
- **SC-003**: Users can successfully select and initiate migration for a mix of Draft and Published dashboards in a single operation.
|
||||
29
specs/007-migration-dashboard-grid/tasks-arch.md
Normal file
29
specs/007-migration-dashboard-grid/tasks-arch.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
|
||||
description: "Architecture tasks for Migration Plugin Dashboard Grid"
|
||||
---
|
||||
|
||||
# Architecture Tasks: Migration Plugin Dashboard Grid
|
||||
|
||||
**Role**: Architect Agent
|
||||
**Goal**: Define the "What" and "Why" (Contracts, Scaffolding, Models) before implementation.
|
||||
|
||||
## Phase 1: Setup & Models
|
||||
|
||||
- [x] A001 Define contracts/scaffolding for migration route in backend/src/api/routes/migration.py
|
||||
- [x] A002 Define contracts/scaffolding for Dashboard model in backend/src/models/dashboard.py
|
||||
|
||||
## Phase 2: User Story 1 - Advanced Dashboard Selection
|
||||
|
||||
- [x] A003 [US1] Define contracts/scaffolding for SupersetClient extensions in backend/src/core/superset_client.py
|
||||
- [x] A004 [US1] Define contracts/scaffolding for GET /api/migration/dashboards endpoint in backend/src/api/routes/migration.py
|
||||
- [x] A005 [US1] Define contracts/scaffolding for DashboardGrid component in frontend/src/components/DashboardGrid.svelte
|
||||
- [x] A006 [US1] Define contracts/scaffolding for migration page integration in frontend/src/routes/migration/+page.svelte
|
||||
- [x] A007 [US1] Define contracts/scaffolding for POST /api/migration/execute endpoint in backend/src/api/routes/migration.py
|
||||
|
||||
## Handover Checklist
|
||||
|
||||
- [x] All new files created with `[DEF]` anchors
|
||||
- [x] All functions/classes have `@PURPOSE`, `@PRE`, `@POST` tags
|
||||
- [x] No "naked code" (logic outside of anchors)
|
||||
- [x] `tasks-dev.md` is ready for the Developer Agent
|
||||
49
specs/007-migration-dashboard-grid/tasks-dev.md
Normal file
49
specs/007-migration-dashboard-grid/tasks-dev.md
Normal file
@@ -0,0 +1,49 @@
|
||||
---
|
||||
description: "Development tasks for Migration Plugin Dashboard Grid"
|
||||
---
|
||||
|
||||
# Development Tasks: Migration Plugin Dashboard Grid
|
||||
|
||||
**Role**: Developer Agent
|
||||
**Goal**: Implement the logic defined in the architecture contracts.
|
||||
|
||||
## Phase 1: Backend Implementation
|
||||
|
||||
- [x] D001 [US1] Implement `SupersetClient.get_dashboards_summary` in `backend/src/core/superset_client.py`
|
||||
- **Context**: Fetch dashboards from Superset API with specific columns (`id`, `dashboard_title`, `changed_on_utc`, `published`).
|
||||
- **Input**: None (uses instance config).
|
||||
- **Output**: List of dictionaries mapped to `DashboardMetadata` fields.
|
||||
|
||||
- [x] D002 [US1] Implement `get_dashboards` endpoint in `backend/src/api/routes/migration.py`
|
||||
- **Context**: Initialize `SupersetClient` with environment config and call `get_dashboards_summary`.
|
||||
- **Input**: `env_id` (path param).
|
||||
- **Output**: JSON list of `DashboardMetadata`.
|
||||
|
||||
- [x] D003 [US1] Implement `execute_migration` endpoint in `backend/src/api/routes/migration.py`
|
||||
- **Context**: Validate selection and initiate migration task (placeholder or TaskManager integration).
|
||||
- **Input**: `DashboardSelection` body.
|
||||
- **Output**: Task ID and status message.
|
||||
|
||||
## Phase 2: Frontend Implementation
|
||||
|
||||
- [x] D004 [US1] Implement `DashboardGrid.svelte` logic
|
||||
- **Context**: `frontend/src/components/DashboardGrid.svelte`
|
||||
- **Requirements**:
|
||||
- Client-side pagination (default 20 items).
|
||||
- Sorting by Title, Last Modified, Status.
|
||||
- Text filtering (search by title).
|
||||
- Multi-selection with "Select All" capability.
|
||||
- Emit selection events.
|
||||
|
||||
- [x] D005 [US1] Integrate `DashboardGrid` into Migration Page
|
||||
- **Context**: `frontend/src/routes/migration/+page.svelte`
|
||||
- **Requirements**:
|
||||
- Fetch dashboards when `sourceEnvId` changes.
|
||||
- Bind `dashboards` data to `DashboardGrid`.
|
||||
- Bind `selectedDashboardIds`.
|
||||
- Update `startMigration` to send `selectedDashboardIds` to backend.
|
||||
|
||||
## Phase 3: Verification
|
||||
|
||||
- [ ] D006 Verify Dashboard Grid functionality (Sort, Filter, Page).
|
||||
- [ ] D007 Verify API integration (Fetch dashboards, Start migration).
|
||||
@@ -20,7 +20,7 @@ from .utils.logger import SupersetLogger
|
||||
class SupersetConfig(BaseModel):
|
||||
env: str = Field(..., description="Название окружения (например, dev, prod).")
|
||||
base_url: str = Field(..., description="Базовый URL Superset API, включая /api/v1.")
|
||||
auth: Dict[str, str] = Field(..., description="Словарь с данными для аутентификации (provider, username, password, refresh).")
|
||||
auth: Dict[str, Any] = Field(..., description="Словарь с данными для аутентификации (provider, username, password, refresh).")
|
||||
verify_ssl: bool = Field(True, description="Флаг для проверки SSL-сертификатов.")
|
||||
timeout: int = Field(30, description="Таймаут в секундах для HTTP-запросов.")
|
||||
logger: Optional[SupersetLogger] = Field(None, description="Экземпляр логгера для логирования.")
|
||||
@@ -32,7 +32,7 @@ class SupersetConfig(BaseModel):
|
||||
# @THROW: ValueError - Если отсутствуют обязательные поля.
|
||||
# @PARAM: v (Dict[str, str]) - Значение поля auth.
|
||||
@validator('auth')
|
||||
def validate_auth(cls, v: Dict[str, str]) -> Dict[str, str]:
|
||||
def validate_auth(cls, v: Dict[str, Any]) -> Dict[str, Any]:
|
||||
required = {'provider', 'username', 'password', 'refresh'}
|
||||
if not required.issubset(v.keys()):
|
||||
raise ValueError(f"Словарь 'auth' должен содержать поля: {required}. Отсутствующие: {required - v.keys()}")
|
||||
|
||||
188
test_migration_debug.py
Normal file
188
test_migration_debug.py
Normal file
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Debug script to test the migration flow and identify where the issue occurs.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Add project root to sys.path
|
||||
project_root = Path(__file__).parent
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
def test_plugin_loader():
|
||||
"""Test if the plugin loader can find the migration plugin."""
|
||||
print("=== Testing Plugin Loader ===")
|
||||
try:
|
||||
from backend.src.core.plugin_loader import PluginLoader
|
||||
from pathlib import Path
|
||||
|
||||
plugin_dir = Path(__file__).parent / "backend" / "src" / "plugins"
|
||||
print(f"Plugin directory: {plugin_dir}")
|
||||
print(f"Directory exists: {plugin_dir.exists()}")
|
||||
|
||||
if plugin_dir.exists():
|
||||
print(f"Files in plugin directory: {list(plugin_dir.iterdir())}")
|
||||
|
||||
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
|
||||
configs = plugin_loader.get_all_plugin_configs()
|
||||
|
||||
print(f"Found {len(configs)} plugins:")
|
||||
for config in configs:
|
||||
print(f" - {config.name} (ID: {config.id})")
|
||||
|
||||
migration_plugin = plugin_loader.get_plugin("superset-migration")
|
||||
if migration_plugin:
|
||||
print("✓ Migration plugin found")
|
||||
print(f" Name: {migration_plugin.name}")
|
||||
print(f" Description: {migration_plugin.description}")
|
||||
print(f" Schema: {migration_plugin.get_schema()}")
|
||||
else:
|
||||
print("✗ Migration plugin NOT found")
|
||||
|
||||
return migration_plugin is not None
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Plugin loader test failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
def test_task_manager():
|
||||
"""Test if the task manager can create tasks."""
|
||||
print("\n=== Testing Task Manager ===")
|
||||
try:
|
||||
from backend.src.core.plugin_loader import PluginLoader
|
||||
from backend.src.core.task_manager import TaskManager
|
||||
from pathlib import Path
|
||||
|
||||
plugin_dir = Path(__file__).parent / "backend" / "src" / "plugins"
|
||||
plugin_loader = PluginLoader(plugin_dir=str(plugin_dir))
|
||||
task_manager = TaskManager(plugin_loader)
|
||||
|
||||
# Test task creation
|
||||
test_params = {
|
||||
"from_env": "dev",
|
||||
"to_env": "prod",
|
||||
"dashboard_regex": ".*",
|
||||
"replace_db_config": False
|
||||
}
|
||||
|
||||
print(f"Creating test task with params: {test_params}")
|
||||
import asyncio
|
||||
task = asyncio.run(task_manager.create_task("superset-migration", test_params))
|
||||
print(f"✓ Task created successfully: {task.id}")
|
||||
print(f" Status: {task.status}")
|
||||
print(f" Plugin ID: {task.plugin_id}")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Task manager test failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
def test_migration_endpoint():
|
||||
"""Test the migration endpoint directly."""
|
||||
print("\n=== Testing Migration Endpoint ===")
|
||||
try:
|
||||
from backend.src.api.routes.migration import execute_migration
|
||||
from backend.src.models.dashboard import DashboardSelection
|
||||
from backend.src.dependencies import get_config_manager, get_task_manager
|
||||
|
||||
# Create a test selection
|
||||
selection = DashboardSelection(
|
||||
selected_ids=[1, 2, 3],
|
||||
source_env_id="ss2",
|
||||
target_env_id="ss1"
|
||||
)
|
||||
|
||||
print(f"Testing migration with selection: {selection.dict()}")
|
||||
|
||||
# This would normally be called by FastAPI with dependencies
|
||||
# For testing, we'll call it directly
|
||||
config_manager = get_config_manager()
|
||||
task_manager = get_task_manager()
|
||||
|
||||
import asyncio
|
||||
|
||||
# We need to ensure TaskManager uses the SAME loop as run_and_wait
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
|
||||
# Re-initialize TaskManager with the new loop
|
||||
task_manager.loop = loop
|
||||
|
||||
async def run_and_wait():
|
||||
result = await execute_migration(selection, config_manager, task_manager)
|
||||
print(f"✓ Migration endpoint test successful: {result}")
|
||||
|
||||
task_id = result["task_id"]
|
||||
print(f"Waiting for task {task_id} to complete...")
|
||||
|
||||
for i in range(20):
|
||||
await asyncio.sleep(1)
|
||||
task = task_manager.get_task(task_id)
|
||||
print(f" [{i}] Task status: {task.status}")
|
||||
if task.status in ["SUCCESS", "FAILED"]:
|
||||
print(f"Task finished with status: {task.status}")
|
||||
for log in task.logs:
|
||||
print(f" [{log.level}] {log.message}")
|
||||
return task.status == "SUCCESS"
|
||||
|
||||
print("Task timed out. Current status: " + task_manager.get_task(task_id).status)
|
||||
return False
|
||||
|
||||
try:
|
||||
return loop.run_until_complete(run_and_wait())
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Migration endpoint test failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
def main():
|
||||
"""Run all tests."""
|
||||
print("Migration Debug Test Script")
|
||||
print("=" * 50)
|
||||
|
||||
results = []
|
||||
|
||||
# Test 1: Plugin Loader
|
||||
results.append(("Plugin Loader", test_plugin_loader()))
|
||||
|
||||
# Test 2: Task Manager
|
||||
results.append(("Task Manager", test_task_manager()))
|
||||
|
||||
# Test 3: Migration Endpoint
|
||||
results.append(("Migration Endpoint", test_migration_endpoint()))
|
||||
|
||||
# Summary
|
||||
print("\n" + "=" * 50)
|
||||
print("TEST RESULTS SUMMARY:")
|
||||
print("=" * 50)
|
||||
|
||||
passed = 0
|
||||
for test_name, result in results:
|
||||
status = "PASS" if result else "FAIL"
|
||||
print(f"{test_name}: {status}")
|
||||
if result:
|
||||
passed += 1
|
||||
|
||||
print(f"\nTotal: {passed}/{len(results)} tests passed")
|
||||
|
||||
if passed == len(results):
|
||||
print("✓ All tests passed! The migration system appears to be working correctly.")
|
||||
else:
|
||||
print("✗ Some tests failed. Check the logs above for details.")
|
||||
|
||||
return passed == len(results)
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
Reference in New Issue
Block a user