fix(tokenizer): compose halfwidth kana readings, memoize entry classific

- Halfwidth katakana now folds to hiragana/katakana like fullwidth kana, so a name written that way carries a reading (voiced pairs like ガ compose correctly instead of leaving a stray mark).
- Mob-label filter (Girl A / Girl B) now only drops a single letter/digit split off a name, keeping genuine one-character names in any script (별, ア・ベ, 山田 空).
- Dictionary-entry classification (source names, media ids) is memoized per entry object instead of recomputed on every lookup and retry window.
- Autoplay priming now tells the processing controller it already painted the plain line, so the controller skips its own duplicate plain emit.
This commit is contained in:
2026-08-05 19:13:03 -07:00
parent afa66ee508
commit f5689e60a3
13 changed files with 261 additions and 42 deletions
+4 -2
View File
@@ -14,5 +14,7 @@ area: subtitles
- A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
- The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
- The unnamed-mob disambiguator filter (Girl A / Girl B) now targets the letters those labels are split into, instead of every one-character term: a character whose name really is one character (𠮷, or a single kana) keeps it. The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it.
- The unnamed-mob disambiguator filter (Girl A / Girl B) now only drops a single letter or digit split off a name, instead of every one-character term: a name that is genuinely one character keeps its terms whatever the script (𠮷, あ, 별 김, ア・ベ). The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it, and it now carries a reading (it used to come out blank, which disables known-word matching and frequency lookups for the token). Voiced halfwidth kana compose properly, so ガク reads ガク rather than ガク, and kana normalization folds halfwidth throughout so those tokens compare equal to the same word written fullwidth.
- Dictionary-entry classification (source dictionaries, character-dictionary media ids) is memoized per entry object for as long as the entry is cached, instead of being recomputed for every headword comparison and every retry window.
- Autoplay priming no longer broadcasts the plain subtitle twice: it tells the processing controller the line has already been painted, so the controller goes straight to the annotated payload.