SubMiner/changes/subtitle-tokenization-performance.md

type: changed area: subtitles

• Subtitle tokenization no longer runs a duplicate full parseText pass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line.
• The Yomitan scanning helpers are now installed once per parser window (__subminerYomitanScan) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call.
• termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
• The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ). The shrinking-window retry ladder keeps following the consumed lengths the backend reports, and only blind guesses (windows the backend consumed whole, which tell it nothing) are capped at four per position. A line that hits that cap escalates to a single parseText for the whole line, so a hard line still resolves to dictionary tokens instead of an unparsed run, without letting the ladder run to one lookup per window length.
• Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a which mecab availability check that re-ran synchronously on every line when MeCab is absent.
• Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused for the whole time the subtitle processing controller is working on the line, including the provisional raw emit that precedes tokenization, so it never competes with the on-screen line for the parser window. The pause is released when the controller reports it has settled, which also covers the lines that finish without an emit (a suppressed duplicate or a failed tokenization) and used to leave prefetching paused indefinitely.
• Added per-stage debug timings (scanMs, mecabMs, frequencyMs, annotateMs) to the subtitle tokenization pipeline log.
• Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
• Subtitle prefetching no longer stays paused for the rest of a cue when the same subtitle text is reported twice and there is nothing to tokenize. This covers the startup and overlay priming paths as well as ordinary subtitle changes.
• Character name and image lookups are now refreshed centrally whenever a character dictionary sync changes its content, so a newly added name can no longer be skipped by a stale candidate list.
• A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
• Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
• The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
• The unnamed-mob disambiguator filter (Girl A / Girl B) now targets the letters those labels are split into, instead of every one-character term: a character whose name really is one character (𠮷, or a single kana) keeps it. The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
• A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it.