mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-03-20 03:16:46 -07:00
Harden stats APIs and fix Electron Yomitan debug runtime
- Validate stats session IDs/limits and add AnkiConnect request timeouts - Stabilize stats window/runtime lifecycle and tighten window security defaults - Fix Electron CLI debug startup by unsetting `ELECTRON_RUN_AS_NODE` and wiring Yomitan session state - Expand regression coverage for tracker queries/events ordering and session aggregates - Update docs for stats dashboard usage and Yomitan lookup troubleshooting
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
---
|
||||
id: TASK-174
|
||||
title: Fix missing frequency highlights for merged tokenizer tokens
|
||||
status: In Progress
|
||||
assignee:
|
||||
- codex
|
||||
created_date: '2026-03-15 10:18'
|
||||
updated_date: '2026-03-15 10:40'
|
||||
labels:
|
||||
- bug
|
||||
- tokenizer
|
||||
- frequency-highlighting
|
||||
dependencies: []
|
||||
references:
|
||||
- /Users/sudacode/projects/japanese/SubMiner/src/core/services/tokenizer.ts
|
||||
- >-
|
||||
/Users/sudacode/projects/japanese/SubMiner/src/core/services/tokenizer/parser-selection-stage.ts
|
||||
- >-
|
||||
/Users/sudacode/projects/japanese/SubMiner/src/core/services/tokenizer/yomitan-parser-runtime.ts
|
||||
- /Users/sudacode/projects/japanese/SubMiner/scripts/get_frequency.ts
|
||||
- /Users/sudacode/projects/japanese/SubMiner/scripts/test-yomitan-parser.ts
|
||||
priority: high
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
Frequency highlighting can miss words that should color within the configured top-X limit when tokenizer candidate selection keeps merged Yomitan units that combine a content word with trailing function text. The annotation stage then conservatively clears frequency for the whole merged token, so visible high-frequency words lose highlighting. The standalone debug CLIs are also failing to initialize the shared Yomitan runtime, which blocks reliable repro for this class of bug.
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
<!-- AC:BEGIN -->
|
||||
- [ ] #1 Tokenizer no longer drops frequency highlighting for content words in merged-token cases where a better scanning parse candidate would preserve highlightable tokens.
|
||||
- [ ] #2 A regression test covers the reported sentence shape and fails before the fix.
|
||||
- [ ] #3 The standalone frequency/parser debug path can initialize the shared Yomitan runtime well enough to reproduce tokenizer output instead of immediately reporting runtime/session wiring errors.
|
||||
<!-- AC:END -->
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
<!-- SECTION:PLAN:BEGIN -->
|
||||
1. Add a regression test for the reported merged-token frequency miss, centered on Yomitan scanning candidate selection and downstream frequency annotation.
|
||||
2. Update tokenizer candidate selection so merged content+function tokens do not win over candidates that preserve highlightable content tokens.
|
||||
3. Repair the standalone frequency/parser debug scripts so their Electron/Yomitan runtime wiring matches current shared runtime expectations.
|
||||
4. Verify with targeted tokenizer/parser tests and the standalone debug repro command.
|
||||
<!-- SECTION:PLAN:END -->
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
<!-- SECTION:NOTES:BEGIN -->
|
||||
Initial triage: shared frequency class logic looks correct; likely failure is upstream tokenizer candidate selection producing merged content+function tokens that annotation later excludes from frequency. Standalone debug scripts also fail to initialize a usable Electron/Yomitan runtime, blocking reliable repro from the current CLI path.
|
||||
|
||||
Repro after fixing the standalone Electron wrapper does not support the original highlight claim for `誰でもいいから かかってこいよ`: the tokenizer reports `かかってこい` with `frequencyRank` 63098, so it correctly stays uncolored at `--color-top-x 10000` and becomes colorable once the threshold is raised above that rank. The concrete bug fixed in this pass is the standalone Electron debug path: package scripts now unset `ELECTRON_RUN_AS_NODE`, and the scripts normalize Electron imports/guards so `get-frequency:electron` can reach real Electron/Yomitan runtime state instead of immediately falling back to Node-mode diagnostics. `test-yomitan-parser:electron` still shows extension/service-worker issues against the existing profile and was not stabilized in this pass.
|
||||
<!-- SECTION:NOTES:END -->
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: TASK-175
|
||||
title: Address latest PR 19 review comments
|
||||
status: In Progress
|
||||
assignee: []
|
||||
created_date: '2026-03-15 10:25'
|
||||
labels:
|
||||
- pr-review
|
||||
- stats-dashboard
|
||||
dependencies: []
|
||||
references:
|
||||
- src/core/services/ipc.ts
|
||||
- src/core/services/stats-server.ts
|
||||
- src/core/services/immersion-tracker/__tests__/query.test.ts
|
||||
- src/core/services/stats-window-runtime.ts
|
||||
- src/core/services/stats-window.test.ts
|
||||
- src/shared/ipc/contracts.ts
|
||||
- src/main.ts
|
||||
priority: medium
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
<!-- SECTION:DESCRIPTION:BEGIN -->
|
||||
Validate the latest automated review comments on PR #19 against the current branch, implement the technically valid fixes, and document any items intentionally left unchanged.
|
||||
<!-- SECTION:DESCRIPTION:END -->
|
||||
|
||||
## Acceptance Criteria
|
||||
<!-- AC:BEGIN -->
|
||||
- [ ] #1 Validated the latest PR #19 review comments against current branch behavior and existing architecture
|
||||
- [ ] #2 Implemented the accepted fixes with regression coverage where it fits
|
||||
- [ ] #3 Documented which latest review items were intentionally not changed because they were already addressed or not technically warranted
|
||||
<!-- AC:END -->
|
||||
Reference in New Issue
Block a user