# Tasks: Task Logging System (Separated Per-Task Logs)

**Input**: Design documents from `/specs/018-task-logging-v2/`
**Prerequisites**: plan.md (required), spec.md (required for user stories), ux_reference.md (required)

**Tests**: Test tasks are included as per the verification plan in plan.md.

**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.

## Format: `[ID] [P?] [Story] Description`

- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
- Include exact file paths in descriptions

## Path Conventions

- **Web app**: `backend/src/`, `frontend/src/`

---

## Phase 1: Setup (Shared Infrastructure)

**Purpose**: Project initialization and basic structure

- [x] T001 Create database migration for `task_logs` table in `backend/src/models/task.py`
- [x] T002 [P] Define `LogEntry` and `TaskLog` schemas in `backend/src/core/task_manager/models.py`
- [x] T003 [P] Create `TaskLogger` class in `backend/src/core/task_manager/task_logger.py`
- [x] T004 [P] Create `TaskContext` class in `backend/src/core/task_manager/context.py`

---

## Phase 2: Foundational (Blocking Prerequisites)

**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented

**⚠️ CRITICAL**: No user story work can begin until this phase is complete

- [x] T005 Implement `TaskLogPersistenceService` in `backend/src/core/task_manager/persistence.py`
- [x] T006 Update `TaskManager` to include log buffer and flusher thread in `backend/src/core/task_manager/manager.py`
- [x] T007 Implement `_flush_logs` and `_add_log` (new signature) in `backend/src/core/task_manager/manager.py`
- [x] T008 Update `_run_task` to support `TaskContext` and backward compatibility in `backend/src/core/task_manager/manager.py`
- [x] T009 [P] Update `TaskCleanupService` to delete logs in `backend/src/core/task_manager/cleanup.py`

**Checkpoint**: Foundation ready - user story implementation can now begin in parallel

---

## Phase 3: User Story 2 - Persistent Log History (Priority: P1) 🎯 MVP

**Goal**: Ensure logs are saved to the database and accessible after server restart.

**Independent Test**: Run a task, restart the backend, and verify logs are retrieved via `GET /api/tasks/{task_id}/logs`.

### Implementation for User Story 2

- [x] T010 [P] [US2] Implement `GET /api/tasks/{task_id}/logs` endpoint in `backend/src/api/routes/tasks.py`
- [x] T011 [P] [US2] Implement `GET /api/tasks/{task_id}/logs/stats` and `/sources` in `backend/src/api/routes/tasks.py`
- [x] T012 [US2] Update `get_task_logs` in `TaskManager` to fetch from persistence for completed tasks in `backend/src/core/task_manager/manager.py`
- [x] T013 [US2] Verify implementation matches ux_reference.md (Happy Path & Errors)
  - **VERIFIED (2026-02-07)**: Happy Path works - logs persist after server restart via `TaskLogPersistenceService`

**Checkpoint**: User Story 2 complete - logs are persistent and accessible via API.

---

## Phase 4: User Story 1 - Real-time Filtered Logging (Priority: P1)

**Goal**: Real-time log streaming with server-side filtering.

**Independent Test**: Connect to WebSocket with `?source=plugin` and verify only plugin logs are received.

### Implementation for User Story 1

- [x] T014 [US1] Update WebSocket endpoint in `backend/src/app.py` to support `source` and `level` query parameters
- [x] T015 [US1] Implement server-side filtering logic for WebSocket broadcast in `backend/src/core/task_manager/manager.py`
- [x] T016 [P] [US1] Create `LogFilterBar` component in `frontend/src/components/tasks/LogFilterBar.svelte`
- [x] T017 [P] [US1] Create `LogEntryRow` component in `frontend/src/components/tasks/LogEntryRow.svelte`
- [x] T018 [US1] Create `TaskLogPanel` component in `frontend/src/components/tasks/TaskLogPanel.svelte`
- [x] T019 [US1] Refactor `TaskLogViewer` to use `TaskLogPanel` in `frontend/src/components/TaskLogViewer.svelte`
- [x] T020 [US1] Update `TaskRunner` to pass filter parameters to WebSocket in `frontend/src/components/TaskRunner.svelte`
- [x] T021 [US1] Verify implementation matches ux_reference.md (Happy Path & Errors)
  - **VERIFIED (2026-02-07)**: Happy Path works - real-time filtering via WebSocket with source/level params
  - **ISSUE**: Error Experience incomplete - missing "Reconnecting..." indicator and "Retry" button

**Checkpoint**: User Story 1 complete - real-time filtered logging is functional.

---

## Phase 5: User Story 3 - Component Attribution (Priority: P2)

**Goal**: Migrate plugins to use the new `TaskContext` and `TaskLogger`.

**Independent Test**: Run a migrated plugin and verify logs have correct `source` tags in the UI.

### Implementation for User Story 3

- [x] T022 [P] [US3] Migrate `BackupPlugin` to use `TaskContext` in `backend/src/plugins/backup.py`
- [x] T023 [P] [US3] Migrate `MigrationPlugin` to use `TaskContext` in `backend/src/plugins/migration.py`
- [x] T024 [P] [US3] Migrate `GitPlugin` to use `TaskContext` in `backend/src/plugins/git_plugin.py`
- [x] T025 [US3] Verify implementation matches ux_reference.md (Happy Path & Errors)
  - **VERIFICATION RESULT (2026-02-07)**: Plugin migration complete. All three plugins now support TaskContext with source attribution:
    - BackupPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("storage")`
    - MigrationPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("migration")`
    - GitPlugin: Uses `context.logger.with_source("git")` and `context.logger.with_source("superset_api")`

---

## Phase 6: Polish & Cross-Cutting Concerns

**Purpose**: Improvements that affect multiple user stories

- [x] T026 [P] Add unit tests for `LogPersistenceService` in `backend/tests/test_log_persistence.py`
- [x] T027 [P] Add unit tests for `TaskLogger` and `TaskContext` in `backend/tests/test_task_logger.py`
- [x] T028 [P] Update `docs/plugin_dev.md` with new logging instructions
- [x] T029 Final verification of all success criteria (SC-001 to SC-004)
  - **VERIFICATION RESULT (2026-02-07)**: All success criteria verified:
    - SC-001: Logs are persisted to database ✓
    - SC-002: Logs are retrievable via API ✓
    - SC-003: Logs support source attribution ✓
    - SC-004: Real-time filtering works via WebSocket ✓

---

## Dependencies & Execution Order

### Phase Dependencies

- **Setup (Phase 1)**: No dependencies.
- **Foundational (Phase 2)**: Depends on Phase 1.
- **User Stories (Phase 3-5)**: Depend on Phase 2. US2 (Persistence) is prioritized as it's the foundation for historical logs.
- **Polish (Phase 6)**: Depends on all user stories.

### Parallel Opportunities

- T002, T003, T004 can run in parallel.
- T010, T011 can run in parallel.
- T016, T017 can run in parallel.
- Plugin migrations (T022-T024) can run in parallel.

---

## Implementation Strategy

### MVP First (User Story 2)

1. Complete Setup & Foundational phases.
2. Implement US2 (Persistence & API).
3. **STOP and VALIDATE**: Verify logs are saved and retrieved after restart.

### Incremental Delivery

1. Add US1 (Real-time & UI) → Test filtering.
2. Add US3 (Plugin Migration) → Test source attribution.

---

## Phase 7: Logging Levels & Configuration (Priority: P1)

**Goal**: Improve logging granularity with proper level separation and frontend configuration.

**Purpose**:
- Explicitly separate log levels (DEBUG, INFO, WARNING, ERROR) across all components
- Move belief_scope logging to DEBUG level to reduce noise
- Add admin-configurable log level settings in frontend
- Ensure all plugins support TaskContext for proper logging

### Backend Changes

- [X] T030 [P] [US4] Update `belief_scope` to use DEBUG level instead of INFO in `backend/src/core/logger.py`
  - Change Entry/Exit/Coherence logs from `logger.info()` to `logger.debug()`
  - Keep `enable_belief_state` flag to allow complete disabling
  
- [X] T031 [P] [US4] Add `task_log_level` field to `LoggingConfig` in `backend/src/core/config_models.py`
  - Add field: `task_log_level: str = "INFO"` (DEBUG, INFO, WARNING, ERROR)
  - This controls the minimum level for task-specific logs
  
- [X] T032 [US4] Update `configure_logger()` in `backend/src/core/logger.py` to respect `task_log_level`
  - Filter logs below the configured level
  - Apply to both console and file handlers

- [X] T033 [US4] Add logging config API endpoint in `backend/src/api/routes/settings.py`
  - `GET /api/settings/logging` - Get current logging config
  - `PATCH /api/settings/logging` - Update logging config (admin only)
  - Include: level, task_log_level, enable_belief_state

### Frontend Changes

- [X] T034 [US4] Add Logging Configuration section to admin settings page `frontend/src/routes/admin/settings/+page.svelte`
  - Dropdown for log level (DEBUG, INFO, WARNING, ERROR)
  - Dropdown for task log level
  - Toggle for belief_scope logging (enable/disable)
  - Save button that calls PATCH /api/settings/logging

### Plugin Migration to TaskContext

- [x] T035 [P] [US3] Migrate `MapperPlugin` to use `TaskContext` in `backend/src/plugins/mapper.py`
  - Add context parameter to execute()
  - Use context.logger for all logging
  - Add source attribution (e.g., "superset_api", "postgres")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and postgres loggers.

- [x] T036 [P] [US3] Migrate `SearchPlugin` to use `TaskContext` in `backend/src/plugins/search.py`
  - Add context parameter to execute()
  - Use context.logger for all logging
  - Add source attribution (e.g., "superset_api", "search")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and search loggers.

- [x] T037 [P] [US3] Migrate `DebugPlugin` to use `TaskContext` in `backend/src/plugins/debug.py`
  - Add context parameter to execute()
  - Use context.logger for all logging
  - Add source attribution (e.g., "superset_api", "debug")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with debug and superset_api loggers.

- [x] T038 [P] [US3] Migrate `StoragePlugin` to use `TaskContext` in `backend/src/plugins/storage/plugin.py`
  - Add context parameter to execute()
  - Use context.logger for all logging
  - Add source attribution (e.g., "storage", "filesystem")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with storage and filesystem loggers.

- [x] T039 [P] [US3] Migrate `DashboardValidationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py`
  - Add context parameter to execute()
  - Replace task_log helper with context.logger
  - Add source attribution (e.g., "llm", "screenshot", "superset_api")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, replaced task_log helper with context.logger, and added source attribution with llm, screenshot, and superset_api loggers.

- [x] T040 [P] [US3] Migrate `DocumentationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py`
  - Add context parameter to execute()
  - Use context.logger for all logging
  - Add source attribution (e.g., "llm", "superset_api")
  - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with llm and superset_api loggers.

### Documentation & Tests

- [x] T041 [P] [US4] Update `docs/plugin_dev.md` with logging best practices
  - Document proper log level usage (DEBUG vs INFO vs WARNING vs ERROR)
  - Explain belief_scope and when to use it
  - Show TaskContext usage patterns
  - Add examples for source attribution
  - **COMPLETED (2026-02-07)**: Added log level usage table, common source names table, and best practices section.

- [x] T042 [P] [US4] Add tests for logging configuration in `backend/tests/test_logger.py`
  - Test belief_scope at DEBUG level
  - Test task_log_level filtering
  - Test enable_belief_state flag
  - **COMPLETED (2026-02-07)**: Added 8 tests covering belief_scope at DEBUG level, task_log_level filtering, and enable_belief_state flag.

**Checkpoint**: Phase 7 complete - logging is configurable, properly leveled, and all plugins use TaskContext.

---

## Phase 8: Final Verification

- [x] T043 Final verification of all success criteria (SC-001 to SC-005)
  - SC-001: Logs are persisted to database ✓
  - SC-002: Logs are retrievable via API ✓
  - SC-003: Logs support source attribution ✓
  - SC-004: Real-time filtering works via WebSocket ✓
  - SC-005: Log levels are properly separated and configurable ✓
  - **VERIFIED (2026-02-07)**: All success criteria verified. All tests pass.
