Commit Graph

2 Commits

Author SHA1 Message Date
sudacode 030c94934e refactor(tokenizer): address review feedback on Yomitan scan runtime
- extract the injected scan runtime (helpers, install script, call-script
  builder) into tokenizer/yomitan-scan-runtime-script.ts; the host module drops
  from ~2700 to ~1900 lines
- append the kana run to the reading as well as the surface when an unparsed
  run extends the previous token, so the reading keeps covering the surface and
  the known-word reading fallback stays enabled (bumps scan runtime version)
- stop annotateMs before character-image resolution so the stage timing
  measures the annotation stage only
2026-08-04 01:56:44 -07:00
sudacode e7cef039f3 perf(tokenizer): single-pass Yomitan scan with install-once runtime and cross-line cache
- drop the duplicate parseText full parse per line; the termsFind scanner walk
  is now authoritative and emits its own unparsed filler runs (parseText kept
  only as error fallback)
- install scan helpers once per parser window (__subminerYomitanScan) instead
  of re-shipping ~500 lines of script per subtitle line
- persist termsFind results across lines in a window-scoped LRU keyed by
  substring, invalidated via a cache epoch on dictionary/settings changes
- skip lookups at punctuation/whitespace positions and cap the shrinking-window
  retry ladder at 4 lookups per position
- build tokenizer runtime deps once (JLPT lookup cache never hit before; mecab
  availability check ran per line)
- stop restarting the prefetch run on every subtitle change; resume prefetch
  only after the tokenized payload lands, not on provisional raw emits
- add per-stage debug timings (scanMs/mecabMs/frequencyMs/annotateMs)
2026-08-04 01:56:44 -07:00