mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-12 01:55:55 -07:00
fix(tokenizer): bound retry ladder and cache, fix name-match edge cases
- Cap only blind shrinking-window retries (not backend-guided shrinks); a line that exhausts the cap escalates to one parseText fallback instead of dropping to raw text - Bound the cross-line termsFind cache by retained dictionary-entry weight, not just key count, and re-check it when a lookup resolves - Keep halfwidth katakana character names in the greedy pre-pass, and let a generic word beat a name it fully contains - Share one Han code-point table between the character dictionary and the scanner's name pre-pass; narrow the mob-disambiguator filter to the split letters, not every one-character term - Release the subtitle prefetch pause on a new onProcessingSettled signal instead of the tokenized emit, so duplicate/suppressed/failed lines no longer pause prefetch indefinitely - Split the Yomitan scan runtime's injected helper script into its own file
This commit is contained in:
@@ -1042,6 +1042,19 @@ export async function requestYomitanScanTokens(
|
||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||
rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||
}
|
||||
// The scanner reports a line where a position ran out of shrinking-window
|
||||
// retries: it stopped short of windows an uncapped ladder would have tried,
|
||||
// so a real term may be sitting in an unparsed run. One parseText for the
|
||||
// line is the bounded way to get the exhaustive answer back (this is the
|
||||
// parse the scanner replaced, and it only runs for these rare lines).
|
||||
if (isObject(rawResult) && rawResult.retryBudgetExhausted === true) {
|
||||
logger.info?.('Yomitan scanner exhausted its retry budget; parsing the line as a fallback.');
|
||||
const fallbackTokens = await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
if (fallbackTokens) {
|
||||
return fallbackTokens;
|
||||
}
|
||||
rawResult = rawResult.tokens;
|
||||
}
|
||||
if (isScanTokenArray(rawResult)) {
|
||||
// Filler-only results carry no dictionary match; keep the historical
|
||||
// contract of returning null so callers fall back to raw text.
|
||||
|
||||
Reference in New Issue
Block a user