Files
SubMiner/backlog/tasks/task-58 - Add-standalone-script-to-exercise-SubMiner-Yomitan-parser-and-candidate-selection.md
kyasuda 457e6f0f10 feat(tokenizer): refine Yomitan grouping and parser tooling
- map segmented Yomitan lines into single logical tokens and improve candidate selection heuristics

- limit frequency lookup to selected token text with POS-based exclusions and add debug logging hook

- add standalone Yomitan parser test script, deterministic utility-script shutdown, and docs/backlog updates
2026-02-16 17:41:24 -08:00

1.4 KiB

id, title, status, assignee, created_date, updated_date, labels, dependencies
id title status assignee created_date updated_date labels dependencies
TASK-58 Add standalone script to exercise SubMiner Yomitan parser and candidate selection Done
2026-02-16 22:04 2026-02-16 22:06

Description

Create scripts/test-yomitan-parser.ts as a standalone CLI tool that reuses SubMiner's Yomitan parser logic to inspect parse output and candidate selection behavior for test inputs.

Acceptance Criteria

  • #1 A new script exists at scripts/test-yomitan-parser.ts.
  • #2 The script can be run standalone and accepts input text for parsing.
  • #3 The script uses existing SubMiner parser logic rather than duplicating parser behavior.
  • #4 The script prints parsed results and candidate selection details (including which candidate is chosen).

Final Summary

Added a standalone parser test script at scripts/test-yomitan-parser.ts that reuses tokenizeSubtitleService from SubMiner, initializes optional Yomitan+Electron runtime, fetches raw parse candidates via Yomitan parseText, and reports which candidate(s) match the final selected token output. Added package scripts test-yomitan-parser and test-yomitan-parser:electron for direct and Electron-backed runs.