refactor(tokenizer): address review feedback on Yomitan scan runtime

- extract the injected scan runtime (helpers, install script, call-script
  builder) into tokenizer/yomitan-scan-runtime-script.ts; the host module drops
  from ~2700 to ~1900 lines
- append the kana run to the reading as well as the surface when an unparsed
  run extends the previous token, so the reading keeps covering the surface and
  the known-word reading fallback stays enabled (bumps scan runtime version)
- stop annotateMs before character-image resolution so the stage timing
  measures the annotation stage only
This commit is contained in:
2026-08-03 23:02:52 -07:00
parent e7cef039f3
commit 030c94934e
5 changed files with 853 additions and 786 deletions
@@ -8,3 +8,4 @@ area: subtitles
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused across a provisional raw-subtitle emit and resumes only after the tokenized payload lands, so it never competes with the on-screen line for the parser window.
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.