mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-07 07:21:35 -07:00
fix(tokenizer): admit halfwidth katakana names to prefilter, split scan
- Fold halfwidth kana throughout so the character-name prefilter treats halfwidth spellings like any other; only unfoldable voiced marks in a candidate region bypass it, and kanji-then-halfwidth names (山ガク) now match too - Split yomitan-scanning-helpers-script.ts into focused injected-script modules: kana, furigana, frequency, dictionary-classification, match-selection - Extract shared vm-context test harness (yomitan-scan-test-harness.ts) out of yomitan-parser-runtime.test.ts and add yomitan-scan-runtime.test.ts
This commit is contained in:
@@ -15,6 +15,6 @@ area: subtitles
|
||||
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
||||
- The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
|
||||
- The unnamed-mob disambiguator filter (Girl A / Girl B) now only drops a single letter or digit split off a name, instead of every one-character term: a name that is genuinely one character keeps its terms whatever the script (𠮷, あ, 별 김, ア・ベ). The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
|
||||
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it, and it now carries a reading (it used to come out blank, which disables known-word matching and frequency lookups for the token). Voiced halfwidth kana compose properly, so ガク reads ガク rather than ガク, and kana normalization folds halfwidth throughout so those tokens compare equal to the same word written fullwidth.
|
||||
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it, and it now carries a reading (it used to come out blank, which disables known-word matching and frequency lookups for the token). Voiced halfwidth kana compose properly, so ガク reads ガク rather than ガク, and kana normalization folds halfwidth throughout so those tokens compare equal to the same word written fullwidth. Because the fold makes halfwidth text indexable, the character-name prefilter now judges halfwidth spellings like any other, and only a position whose candidate-sized region holds an unfoldable voiced mark bypasses it. That also covers a name that starts on a kanji and turns halfwidth later (山ガク), which the earlier first-character rule dropped.
|
||||
- Dictionary-entry classification (source dictionaries, character-dictionary media ids) is memoized per entry object for as long as the entry is cached, instead of being recomputed for every headword comparison and every retry window.
|
||||
- Autoplay priming no longer broadcasts the plain subtitle twice: it tells the processing controller the line has already been painted, so the controller goes straight to the annotated payload.
|
||||
|
||||
Reference in New Issue
Block a user