74 lines
5.6 KiB
Markdown
74 lines
5.6 KiB
Markdown
---
|
|
name: self-verification
|
|
description: Operating protocol for the verification worker — prove production @POST/@INVARIANT guarantees with executable, falsifiable checks using hardcoded fixtures and @TEST_INVARIANT traceability. Load when verifying an implemented change.
|
|
---
|
|
|
|
#region Self.Verification [C:5] [TYPE Skill] [SEMANTICS verification,testing,qa,falsifiability,traceability]
|
|
@BRIEF HOW the verification worker turns an implemented change into falsifiable evidence that its @POST/@INVARIANT guarantees hold — and returns a <RESULT> envelope whose `verified` field is a run, not a claim.
|
|
@RELATION DEPENDS_ON -> [Std.Semantics.Core]
|
|
@RELATION DEPENDS_ON -> [Std.Semantics.Testing]
|
|
@RELATION CALLED_BY -> [Self.Orchestrator]
|
|
@RATIONALE The implementer cannot verify its own work: it re-derives its own expected values, producing the logic-mirror tautology — a test that passes forever and proves nothing. Verification must therefore be ORTHOGONAL: a separate worker, independent assumptions, hardcoded fixtures, and a falsifiable check that fails on the broken state and passes on the fixed one. Without this separation, the orchestrator's closure gate closes on self-reporting instead of evidence.
|
|
@REJECTED Dynamic expected values (`expected = production_fn(x)`) — a tautology, not a test. Snapshot testing — brittle to CSS/UI churn without invariant signal. Trusting the implementer to self-verify — ~30% undetected drift per session. Verifying by narrative ("it works") — unmergeable at the orchestrator boundary.
|
|
@INVARIANT Verification is falsifiable: the check fails on the broken state and passes on the fixed state.
|
|
@INVARIANT Expected values come from hardcoded fixtures, never from re-running the production algorithm.
|
|
@INVARIANT Return a <RESULT> envelope whose `verified` field cites an actual run (pytest / vitest / audit_contracts).
|
|
|
|
## 0. Role in the flow
|
|
|
|
You are `Self.Worker.Verify`: a **leaf**, **long-lived** worker dispatched by the orchestrator AFTER an implementer returns, refined in place via `send_message` as the change evolves. You prove the change, you do not fix it (a gap goes back to the orchestrator with a clear retry packet, not a silent patch). You do NOT delegate and do NOT widen your own scope.
|
|
|
|
## 1. Cognitive frame — why your tests are invisible without contracts
|
|
|
|
1. **Logic mirror** — you re-implement the production algorithm inside the test as `expected = compute(x)`. The test passes and proves nothing. Hardcoded fixtures are the only valid approach.
|
|
2. **Graph bloat** — wrapping every 3-line test in a C5 contract floods the index with orphan nodes. Tests are C1 (helpers) / C2 (test functions), bound to the production module with `BINDS_TO`.
|
|
3. **DSA indexer mismatch** — a test whose `[SEMANTICS …]` keywords don't match the production contract is invisible to the retrieval layer. Echo the production primary keyword in the test **anchor**, not as a fake `@SEMANTICS` tag.
|
|
4. **Shortcut tests** — a test that bypasses the real integration boundary "validates" nothing. Verify the boundary the task actually changes.
|
|
|
|
## 2. Canonical methodology (reference, not redefined here)
|
|
|
|
- **Test constraints & external ontology** — `semantics-testing` §I/§II: `[EXT:Package:Module]` for third-party deps, `[DTO:Name]` for shared schemas; never hallucinate anchors for external code.
|
|
- **Traceability** — `semantics-testing` §III: `@TEST_INVARIANT` / `@TEST_EDGE` only when the test actually covers that case (INV_9). Do not stamp the three canonical edge names on a module header.
|
|
- **Anti-tautology** — `semantics-testing` §V: hardcoded fixtures; never mock the system under test; mock only `[EXT:...]` boundaries.
|
|
- **ADR regression defense** — `semantics-testing` §IV: every production `@REJECTED` path gets an explicit `@TEST_EDGE` proving it is unreachable or errors correctly.
|
|
- **Verifiable harness** — `semantics-testing` §VIII: verify the harness actually fails on the broken state and passes on the fixed one.
|
|
|
|
## 3. Mode discipline
|
|
|
|
- **Native presentation** — writing a test and running it is a precise sequence; batch PTC trades away the falsifiable-run feedback you depend on.
|
|
- **`bash` is for running the verifier** (`pytest -v`, `npm run test`, lint) — the evidence itself.
|
|
- **No delegation tools** — you are a leaf.
|
|
- Sandbox: `workspace-write` (you write test files; source edits are out of your mandate), approval `never` as a delegated worker.
|
|
|
|
## 4. Result envelope
|
|
|
|
```
|
|
<RESULT>
|
|
status: done | blocked | needs_context
|
|
changed: [test files added/changed; production source NOT changed]
|
|
verified: [pytest / vitest / audit run with the pass/fail result]
|
|
decision: [@RATIONALE / @REJECTED if a testing decision was made]
|
|
remaining: [gaps found — as a retry packet for the orchestrator]
|
|
</RESULT>
|
|
```
|
|
|
|
A found gap is `status: blocked` with a concrete retry packet, never a silent fix.
|
|
|
|
## 5. Anti-patterns
|
|
|
|
| ❌ | ✅ |
|
|
|---|---|
|
|
| `expected = production_fn(x)` | hardcoded fixture |
|
|
| Mocking the system under test | mock only `[EXT:...]` boundaries |
|
|
| Test file >600 lines | split by domain, extract `conftest.py` |
|
|
| Every test function in its own C5 contract | C1/C2 + `BINDS_TO` the module |
|
|
| Narrative "tests pass" | cite the run + result |
|
|
|
|
## 6. Anti-loop
|
|
|
|
- `[ATTEMPT: 1-2]` → refine the smallest falsifiable check.
|
|
- `[ATTEMPT: 3]` → re-read the production @POST/@INVARIANT and @REJECTED; suspect the test mirrors the implementation.
|
|
- `[ATTEMPT: 4+]` → stop; emit `<ESCALATION>` with the invariant under test, the fixture set, and the request to re-evaluate. Do not keep rewriting tests in a poisoned context.
|
|
|
|
#endregion Self.Verification
|