mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-04 19:21:33 -07:00
refactor(tokenizer): address review feedback on Yomitan scan runtime
- extract the injected scan runtime (helpers, install script, call-script builder) into tokenizer/yomitan-scan-runtime-script.ts; the host module drops from ~2700 to ~1900 lines - append the kana run to the reading as well as the surface when an unparsed run extends the previous token, so the reading keeps covering the surface and the known-word reading fallback stays enabled (bumps scan runtime version) - stop annotateMs before character-image resolution so the stage timing measures the annotation stage only
This commit is contained in:
@@ -8,3 +8,4 @@ area: subtitles
|
||||
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
|
||||
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused across a provisional raw-subtitle emit and resumes only after the tokenized payload lands, so it never competes with the on-screen line for the parser window.
|
||||
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
||||
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
||||
|
||||
Reference in New Issue
Block a user