mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-04 19:21:33 -07:00
perf(tokenizer): skip character-name lookups where no name can start
The greedy name pre-pass asked the Yomitan backend at every Japanese position, because a character name can begin mid-token. With the character dictionary enabled that roughly doubled the round trips per line (measured 10 -> 21 on a 23-char line). SubMiner generates the character dictionary, so the cached snapshots already list every form a character entry can be matched by (term and reading). Those forms are installed into the scan runtime once per media and the pre-pass now probes only positions where one of them starts, compared after kana normalization so a katakana name still matches a hiragana reading form. The overhead drops to zero (21 -> 10, the same as with the dictionary disabled). Fail-safe: with no candidate list (no media id, no cached snapshot, failed install) the pre-pass keeps its exhaustive behavior, so stale character data costs speed rather than a missing name. Halfwidth katakana positions bypass the filter since kana normalization does not fold them. The candidate lookup is consulted per subtitle line, so it caches its snapshot directory signature for 5s; dictionary writes still call invalidate().
This commit is contained in:
@@ -9,3 +9,4 @@ area: subtitles
|
||||
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused across a provisional raw-subtitle emit and resumes only after the tokenized payload lands, so it never competes with the on-screen line for the parser window.
|
||||
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
||||
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
||||
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
||||
|
||||
Reference in New Issue
Block a user