fix(tokenizer): bound retry ladder and cache, fix name-match edge cases

- Cap only blind shrinking-window retries (not backend-guided shrinks); a line that exhausts the cap escalates to one parseText fallback instead of dropping to raw text
- Bound the cross-line termsFind cache by retained dictionary-entry weight, not just key count, and re-check it when a lookup resolves
- Keep halfwidth katakana character names in the greedy pre-pass, and let a generic word beat a name it fully contains
- Share one Han code-point table between the character dictionary and the scanner's name pre-pass; narrow the mob-disambiguator filter to the split letters, not every one-character term
- Release the subtitle prefetch pause on a new onProcessingSettled signal instead of the tokenized emit, so duplicate/suppressed/failed lines no longer pause prefetch indefinitely
- Split the Yomitan scan runtime's injected helper script into its own file
This commit is contained in:
2026-08-05 01:22:10 -07:00
parent 3c24597724
commit afa66ee508
19 changed files with 1352 additions and 587 deletions
@@ -56,10 +56,12 @@ background prefetch work. Prefetch is not re-centered here: restarting the run p
(`onSeek`) discarded the in-flight tokenization every time the subtitle changed, so only real
seeks restart it (see `onTimePosUpdate` in `src/main.ts`).
The pause is released by the emit that carries the tokenized payload. Both controller methods
return whether an emit is expected, and the caller resumes immediately when it is not — otherwise
a repeated subtitle (which schedules no work) would leave prefetching idle for the rest of the
cue.
The pause is released by the controller's `onProcessingSettled` callback, which fires once it has
no work left. Emits do not release it: the first emit for an uncached line is the plain payload
that precedes tokenization, and a run can finish without emitting at all (a suppressed duplicate,
a failed tokenization). Both controller methods return whether processing is now pending, and the
caller resumes immediately when it is not — a repeated subtitle schedules no work, so no settle is
coming and prefetching would otherwise idle for the rest of the cue.
## Live Cue Delivery