docs: add stats dashboard design docs, plans, and knowledge base

- Stats dashboard redesign design and implementation plans
- Episode detail and Anki card link design
- Internal knowledge base restructure
- Backlog tasks for testing, verification, and occurrence tracking
This commit is contained in:
2026-03-14 22:13:24 -07:00
parent 9eed37420e
commit ee95e86ad5
35 changed files with 5139 additions and 0 deletions

View File

@@ -0,0 +1,64 @@
---
id: TASK-165
title: Rewrite SubMiner agentic testing automation plan
status: Done
assignee: []
created_date: '2026-03-13 04:45'
updated_date: '2026-03-13 04:47'
labels:
- planning
- testing
- agents
dependencies: []
references:
- /home/sudacode/projects/japanese/SubMiner/testing-plan.md
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-change-verification/SKILL.md
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-scrum-master/SKILL.md
documentation:
- /home/sudacode/projects/japanese/SubMiner/docs-site/development.md
- /home/sudacode/projects/japanese/SubMiner/docs-site/architecture.md
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Replace the current generic Electron/mpv testing plan with a SubMiner-specific plan that uses the existing skills as the source of truth, treats real launcher/plugin/mpv runtime verification as primary, and defines a non-interference contract for parallel agent work.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 `testing-plan.md` is rewritten for SubMiner rather than a generic Electron+mpv app
- [x] #2 The plan keeps `subminer-scrum-master` and `subminer-change-verification` as the primary orchestration and verification entrypoints
- [x] #3 The plan defines real launcher/plugin/mpv runtime verification as the authoritative lane for runtime bug claims
- [x] #4 The plan defines explicit session isolation and non-interference rules for parallel agent work
- [x] #5 The plan defines artifact/reporting expectations and phased rollout, with synthetic/headless verification clearly secondary to real-runtime verification
<!-- AC:END -->
## Implementation Plan
<!-- SECTION:PLAN:BEGIN -->
1. Review the existing testing plan and compare it against current SubMiner architecture, verification lanes, and skills.
2. Replace the generic Electron/mpv harness framing with a SubMiner-specific control plane centered on existing skills.
3. Define the authoritative real-runtime lane, session isolation rules, concurrency classes, and reporting contract.
4. Sanity-check the rewritten document against current repo docs and skill contracts before handoff.
<!-- SECTION:PLAN:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
Rewrote `testing-plan.md` around existing `subminer-scrum-master` and `subminer-change-verification` responsibilities instead of proposing a competing new top-level testing skill.
Set real launcher/plugin/mpv/runtime verification as the authoritative lane for runtime bug claims and made synthetic/headless verification explicitly secondary.
Defined session-scoped paths, unique mutable resources, concurrency classes, and an exclusive lease for conflicting real-runtime verification to prevent parallel interference.
Sanity-checked the final document by inspecting the rewritten file content and diff.
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Rewrote `testing-plan.md` into a SubMiner-specific agentic verification plan. The new document keeps `subminer-scrum-master` and `subminer-change-verification` as the primary orchestration and verification entrypoints, treats the real launcher/plugin/mpv/runtime path as authoritative for runtime bug claims, and defines a hard non-interference contract for parallel work through session isolation and an exclusive real-runtime lease. The plan now also includes an explicit reporting schema, capture policy, phased rollout, and a clear statement that true parallel full-app instances are not a phase-1 requirement. Verification for this task was a document sanity pass against the current repo docs, skills, and the resulting file diff.
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,64 @@
---
id: TASK-166
title: Harden SubMiner change verification for authoritative agentic runtime checks
status: Done
assignee: []
created_date: '2026-03-13 05:19'
updated_date: '2026-03-13 05:36'
labels:
- testing
- agents
- verification
dependencies: []
references:
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-change-verification/scripts/verify_subminer_change.sh
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-change-verification/scripts/classify_subminer_diff.sh
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-change-verification/SKILL.md
documentation:
- /home/sudacode/projects/japanese/SubMiner/testing-plan.md
- /home/sudacode/projects/japanese/SubMiner/docs-site/development.md
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Tighten the SubMiner change-verification classifier and verifier so the implementation matches the approved agentic verification plan: authoritative runtime verification must fail closed when unavailable, lane naming should use real-runtime semantics, session and artifact identities must be unique, and the verifier must be safer for parallel agent use.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 The verifier uses `real-runtime` terminology instead of `real-gui` for authoritative runtime verification
- [x] #2 Requested authoritative runtime verification fails closed with a non-green outcome when it cannot run, and unknown lanes do not pass open
- [x] #3 The verifier allocates a unique session identifier and artifact root that does not rely on second-level timestamp uniqueness alone
- [x] #4 The verifier summary/report output includes explicit top-level status and session metadata needed for agent aggregation
- [x] #5 The classifier and verifier better reflect runtime-escalation cases for launcher/plugin/socket/runtime-sensitive changes
- [x] #6 Regression tests cover the new verifier/classifier behavior
<!-- AC:END -->
## Implementation Plan
<!-- SECTION:PLAN:BEGIN -->
1. Add regression tests for classifier/verifier behavior before changing the scripts.
2. Harden `verify_subminer_change.sh` to use `real-runtime` terminology, fail closed for blocked/unknown authoritative verification, and emit unique session metadata in summaries.
3. Update `classify_subminer_diff.sh` and the skill doc to use `real-runtime` escalation language and better flag launcher/plugin/runtime-sensitive paths.
4. Run targeted regression tests plus a focused dry-run verifier check, then record outcomes and blockers in the task.
<!-- SECTION:PLAN:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
Added `scripts/subminer-change-verification.test.ts` to regression-test classifier/verifier behavior around `real-runtime` naming, fail-closed authoritative verification, unknown lanes, and unique session metadata.
Reworked `verify_subminer_change.sh` to normalize `real-gui` to `real-runtime`, emit unique session IDs and richer summary metadata, block authoritative runtime verification when unavailable, and fail closed for unknown lanes.
Updated `classify_subminer_diff.sh` to emit `real-runtime-candidate` for launcher/plugin/runtime-sensitive paths, and updated the active skill doc wording to match the new lane terminology.
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Hardened the SubMiner change-verification tooling to match the approved agentic verification plan. The verifier now uses `real-runtime` terminology for authoritative runtime verification, preserves compatibility with the deprecated `real-gui` alias, fails closed for unknown lanes, and returns a blocked non-green outcome when requested authoritative runtime verification cannot run. It now allocates a unique session ID and artifact root by default, writes richer session metadata and top-level status into `summary.json`/`summary.txt` plus `reports/summary.*`, and records path selection mode, blockers, and session-local env roots for agent aggregation. The classifier now emits `real-runtime-candidate` for launcher/plugin/runtime-sensitive paths, and the active skill doc uses the same terminology. Verification ran via `bun test scripts/subminer-change-verification.test.ts`, direct dry-run smoke checks for blocked `real-runtime` and failed unknown-lane execution, and a targeted classifier invocation for launcher/plugin paths.
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,51 @@
---
id: TASK-167
title: Track shared SubMiner agent skills in git and clean up ignore rules
status: Done
assignee: []
created_date: '2026-03-13 05:46'
updated_date: '2026-03-13 05:47'
labels:
- git
- agents
- repo-hygiene
dependencies: []
references:
- /home/sudacode/projects/japanese/SubMiner/.gitignore
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-change-verification/SKILL.md
- >-
/home/sudacode/projects/japanese/SubMiner/.agents/skills/subminer-scrum-master/SKILL.md
documentation:
- /home/sudacode/projects/japanese/SubMiner/testing-plan.md
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Adjust the repository ignore rules so the shared SubMiner agent skill files can be committed while keeping unrelated local agent state ignored. Also ensure generated local verification artifacts like `.tmp/` do not pollute git status.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 Root ignore rules allow the shared SubMiner skill files under `.agents/skills/` to be tracked without broadly unignoring local agent state
- [x] #2 The changed shared skill files appear in git status as trackable files after the ignore update
- [x] #3 Local generated verification artifact directories remain ignored so git status stays clean
- [x] #4 The updated ignore rules are minimal and scoped to the repo-shared skill files
<!-- AC:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
Updated `.gitignore` to keep `.agents` ignored by default while narrowly unignoring the repo-shared SubMiner skill files and verifier scripts.
Added `.tmp/` to the root ignore rules so local verification artifacts stop polluting `git status`.
Verified the result with `git status --untracked-files=all` and `git check-ignore -v`, confirming the shared skill files are now trackable and `.tmp/` remains ignored.
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Adjusted the root `.gitignore` so the shared SubMiner agent skill files can be committed cleanly without broadly unignoring local agent state. The repo now tracks the shared `subminer-change-verification` skill files and the `subminer-scrum-master` skill doc, while `.tmp/` is ignored so generated verification artifacts do not pollute git status. Verified with `git status --untracked-files=all` and `git check-ignore -v` that the intended skill files are commit-ready and `.tmp/` remains ignored.
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,45 @@
---
id: TASK-170
title: 'Fix imm_words POS filtering and add stats cleanup maintenance command'
status: Done
assignee: []
created_date: '2026-03-13 00:00'
updated_date: '2026-03-14 18:31'
labels: []
dependencies: []
priority: high
ordinal: 9010
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
`imm_words` is currently populated from raw subtitle text instead of tokenized subtitle metadata, so ignored functional/noise tokens leak into stats and no POS metadata is stored. Fix live persistence to follow the existing token annotation exclusion rules and add an on-demand stats cleanup command to remove stale bad vocabulary rows from the stats DB.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 New `imm_words` inserts use tokenized subtitle data, persist POS metadata, and skip tokens excluded by existing POS-based vocabulary ignore rules.
- [x] #2 `subminer stats cleanup` supports `-v` / `--vocab`, defaults to vocab cleanup, and removes stale bad `imm_words` rows on demand.
- [x] #3 Regression coverage exists for persistence filtering, cleanup behavior, and stats cleanup CLI wiring.
<!-- AC:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Fixed `imm_words` persistence so the tracker now consumes tokenized subtitle data, stores POS metadata (`part_of_speech`, `pos1`, `pos2`, `pos3`), preserves distinct surface/lemma fields (`word` vs `headword`) when tokenization provides them, and skips vocabulary rows excluded by the existing POS/noise rules instead of mining raw subtitle fragments. Added `subminer stats cleanup` with default vocab cleanup plus `-v/--vocab`; the cleanup pass now repairs stale `headword`, `reading`, and `part_of_speech` values, attempts best-effort MeCab backfill for legacy rows, and removes rows that still have no usable POS metadata or fail the vocab filters.
Verification:
- `bun run typecheck`
- `bun test src/core/services/immersion-tracker-service.test.ts src/core/services/immersion-tracker/__tests__/query.test.ts src/core/services/immersion-tracker/storage-session.test.ts launcher/parse-args.test.ts launcher/commands/command-modules.test.ts src/main/runtime/stats-cli-command.test.ts src/main/runtime/mpv-main-event-main-deps.test.ts src/core/services/cli-command.test.ts`
- `bun run docs:test`
- `bun run docs:build`
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,77 @@
---
id: TASK-171
title: Add normalized immersion word and kanji occurrence tracking
status: Done
assignee:
- codex
created_date: '2026-03-14 11:30'
updated_date: '2026-03-14 11:48'
labels:
- immersion
- stats
- database
dependencies: []
references:
- /home/sudacode/projects/japanese/SubMiner/docs/plans/2026-03-14-immersion-occurrence-tracking-design.md
- /home/sudacode/projects/japanese/SubMiner/docs/plans/2026-03-14-immersion-occurrence-tracking.md
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Add normalized occurrence tables for immersion-tracked words and kanji so stats can map vocabulary back to the exact anime, episode, timestamp, and subtitle line where each item appeared. Preserve repeated tokens within the same line via counted occurrences instead of deduping, while avoiding duplicated token text storage.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 The immersion schema adds normalized subtitle-line and counted occurrence tables for words and kanji, with additive migration support for existing databases.
- [x] #2 Subtitle-line tracking writes one subtitle-line row per seen line plus counted word/kanji occurrences linked back to the line, session, video, and anime context.
- [x] #3 Query surfaces can map a word or kanji back to anime/episode/line/timestamp rows without breaking current top-level vocabulary and kanji stats.
- [x] #4 Focused regression coverage exists for schema, counted occurrence persistence, and reverse-mapping queries.
- [x] #5 Verification covers the SQLite immersion lane and any broader lanes required by touched service/API files.
<!-- AC:END -->
## Implementation Plan
<!-- SECTION:PLAN:BEGIN -->
1. Add red tests for new line/occurrence schema and migration shape in the SQLite immersion lane.
2. Add red tests for service-level subtitle persistence that writes one line row plus counted word/kanji occurrences.
3. Implement additive schema, write-path plumbing, and counted occurrence upserts with minimal disruption to existing aggregate tables.
4. Add reverse-mapping query surfaces for word and kanji occurrences, plus focused API/service exposure only where needed.
5. Run focused SQLite verification first, then broader verification only if touched runtime/API files require it.
<!-- SECTION:PLAN:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
2026-03-14: Design approved in-thread. Chosen shape: `imm_subtitle_lines` plus counted bridge tables `imm_word_line_occurrences` and `imm_kanji_line_occurrences`, retaining repeated tokens within a line via `occurrence_count`.
2026-03-14: Implemented additive schema version bump to 7. `recordSubtitleLine(...)` now queues one normalized subtitle-line write that owns aggregate word/kanji upserts plus counted bridge-row inserts.
2026-03-14: Added reverse-mapping query surfaces for exact word triples and single kanji lookups. No stats API/UI consumer was widened in this change.
2026-03-14: Verification commands run:
- `bun test src/core/services/immersion-tracker-service.test.ts`
- `bun test src/core/services/immersion-tracker/storage-session.test.ts`
- `bun test src/core/services/immersion-tracker/__tests__/query.test.ts`
- `bun run typecheck`
- `bash .agents/skills/subminer-change-verification/scripts/classify_subminer_diff.sh src/core/services/immersion-tracker/types.ts src/core/services/immersion-tracker/storage.ts src/core/services/immersion-tracker/query.ts src/core/services/immersion-tracker-service.ts src/core/services/immersion-tracker/storage-session.test.ts src/core/services/immersion-tracker-service.test.ts src/core/services/immersion-tracker/__tests__/query.test.ts`
- `bash .agents/skills/subminer-change-verification/scripts/verify_subminer_change.sh --lane core src/core/services/immersion-tracker/types.ts src/core/services/immersion-tracker/storage.ts src/core/services/immersion-tracker/query.ts src/core/services/immersion-tracker-service.ts src/core/services/immersion-tracker/storage-session.test.ts src/core/services/immersion-tracker-service.test.ts src/core/services/immersion-tracker/__tests__/query.test.ts`
- `bun run test:immersion:sqlite:src`
2026-03-14: Verification results:
- targeted tracker/query tests: passed
- verifier lane selection: `core`
- verifier result: passed (`typecheck`, `test:fast`)
- verifier artifacts: `.tmp/skill-verification/subminer-verify-20260314-114630-abO7mb/`
- maintained immersion SQLite lane: passed
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Added normalized subtitle-line occurrence tracking to immersion stats with three additive tables: `imm_subtitle_lines`, `imm_word_line_occurrences`, and `imm_kanji_line_occurrences`.
`recordSubtitleLine(...)` now preserves repeated allowed tokens and repeated kanji within the same subtitle line via `occurrence_count`, while still updating canonical `imm_words` and `imm_kanji` aggregates.
Added reverse-mapping queries for exact word triples and kanji so callers can fetch anime/video/session/line/timestamp context for each occurrence without duplicating token text storage.
Verified with targeted tracker/query tests, `bun run typecheck`, verifier-selected `core` coverage, and the maintained `bun run test:immersion:sqlite:src` lane.
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,65 @@
---
id: TASK-172
title: Wire immersion occurrence drilldown into stats API and vocabulary drawer
status: Done
assignee:
- codex
created_date: '2026-03-14 12:05'
updated_date: '2026-03-14 12:11'
labels:
- immersion
- stats
- ui
dependencies:
- TASK-171
references: []
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Expose the new immersion word/kanji occurrence queries through the stats server and add a right-side Vocabulary drawer that shows recent occurrence rows with paging when a word or kanji is clicked.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 Stats server exposes word and kanji occurrence endpoints with bounded recent-first paging.
- [x] #2 Stats client/types support loading occurrence pages for a selected word or kanji.
- [x] #3 Vocabulary tab opens a right drawer for the selected word/kanji, shows recent occurrences, and supports loading more.
- [x] #4 Focused regression coverage exists for the server endpoint contract, and the stats UI still typechecks/builds.
- [x] #5 Verification covers the cheapest sufficient backend and stats-UI lanes.
<!-- AC:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
2026-03-14: Design approved in-thread. Chosen UX: click a word chip or kanji glyph to open a right-side drawer with recent-first occurrences, initial cap 50, plus “Load more”.
2026-03-14: Implemented `/api/stats/vocabulary/occurrences` and `/api/stats/kanji/occurrences` with `limit` + `offset` paging. The drawer uses direct stats HTTP client calls and keeps existing aggregate vocabulary data flow intact.
2026-03-14: Verification commands run:
- `bun test src/core/services/__tests__/stats-server.test.ts`
- `bun run typecheck`
- `cd stats && bun run build`
- `bun run docs:test`
- `bun run docs:build`
- `cd stats && bunx tsc --noEmit`
- `bash .agents/skills/subminer-change-verification/scripts/verify_subminer_change.sh --lane core src/core/services/stats-server.ts src/core/services/__tests__/stats-server.test.ts`
2026-03-14: Verification results:
- stats server endpoint tests: passed
- root typecheck: passed
- stats UI production build: passed
- docs-site test/build: passed
- `cd stats && bunx tsc --noEmit`: passed after removing stale `hasCoverArt` prop usage in the library stats UI
- verifier result: passed (`typecheck`, `test:fast`)
- verifier artifacts: `.tmp/skill-verification/subminer-verify-20260314-120900-J0VvB0/`
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Wired occurrence drilldown into the stats server and Vocabulary tab. Words and kanji now open a right-side drawer that loads recent occurrence rows 50 at a time and supports “Load more”.
Added bounded recent-first occurrence endpoints to the stats HTTP API, extended the stats client/type surface, and made word chips plus kanji glyphs selectable with active-state styling.
Updated the immersion-tracking docs to mention vocabulary occurrence drilldown. Verified with focused stats-server tests, root typecheck, stats UI production build, docs-site test/build, the repo verifier core lane, and a direct `stats` package typecheck after removing the stale `MediaHeader` prop mismatch.
<!-- SECTION:FINAL_SUMMARY:END -->