mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-06 07:21:33 -07:00
fix(tokenizer): bound retry ladder and cache, fix name-match edge cases
- Cap only blind shrinking-window retries (not backend-guided shrinks); a line that exhausts the cap escalates to one parseText fallback instead of dropping to raw text - Bound the cross-line termsFind cache by retained dictionary-entry weight, not just key count, and re-check it when a lookup resolves - Keep halfwidth katakana character names in the greedy pre-pass, and let a generic word beat a name it fully contains - Share one Han code-point table between the character dictionary and the scanner's name pre-pass; narrow the mob-disambiguator filter to the split letters, not every one-character term - Release the subtitle prefetch pause on a new onProcessingSettled signal instead of the tokenized emit, so duplicate/suppressed/failed lines no longer pause prefetch indefinitely - Split the Yomitan scan runtime's injected helper script into its own file
This commit is contained in:
@@ -4,12 +4,15 @@ area: subtitles
|
|||||||
- Subtitle tokenization no longer runs a duplicate full `parseText` pass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line.
|
- Subtitle tokenization no longer runs a duplicate full `parseText` pass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line.
|
||||||
- The Yomitan scanning helpers are now installed once per parser window (`__subminerYomitanScan`) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call.
|
- The Yomitan scanning helpers are now installed once per parser window (`__subminerYomitanScan`) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call.
|
||||||
- termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
|
- termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
|
||||||
- The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ) and caps the shrinking-window retry ladder at four extra lookups per position.
|
- The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ). The shrinking-window retry ladder keeps following the consumed lengths the backend reports, and only blind guesses (windows the backend consumed whole, which tell it nothing) are capped at four per position. A line that hits that cap escalates to a single `parseText` for the whole line, so a hard line still resolves to dictionary tokens instead of an unparsed run, without letting the ladder run to one lookup per window length.
|
||||||
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
|
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
|
||||||
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused across a provisional raw-subtitle emit and resumes only after the tokenized payload lands, so it never competes with the on-screen line for the parser window.
|
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused for the whole time the subtitle processing controller is working on the line, including the provisional raw emit that precedes tokenization, so it never competes with the on-screen line for the parser window. The pause is released when the controller reports it has settled, which also covers the lines that finish without an emit (a suppressed duplicate or a failed tokenization) and used to leave prefetching paused indefinitely.
|
||||||
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
||||||
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
||||||
- Subtitle prefetching no longer stays paused for the rest of a cue when the same subtitle text is reported twice and there is nothing to tokenize. This covers the startup and overlay priming paths as well as ordinary subtitle changes.
|
- Subtitle prefetching no longer stays paused for the rest of a cue when the same subtitle text is reported twice and there is nothing to tokenize. This covers the startup and overlay priming paths as well as ordinary subtitle changes.
|
||||||
- Character name and image lookups are now refreshed centrally whenever a character dictionary sync changes its content, so a newly added name can no longer be skipped by a stale candidate list.
|
- Character name and image lookups are now refreshed centrally whenever a character dictionary sync changes its content, so a newly added name can no longer be skipped by a stale candidate list.
|
||||||
- A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
|
- A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
|
||||||
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
||||||
|
- The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
|
||||||
|
- The unnamed-mob disambiguator filter (Girl A / Girl B) now targets the letters those labels are split into, instead of every one-character term: a character whose name really is one character (𠮷, or a single kana) keeps it. The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
|
||||||
|
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it.
|
||||||
|
|||||||
@@ -56,10 +56,12 @@ background prefetch work. Prefetch is not re-centered here: restarting the run p
|
|||||||
(`onSeek`) discarded the in-flight tokenization every time the subtitle changed, so only real
|
(`onSeek`) discarded the in-flight tokenization every time the subtitle changed, so only real
|
||||||
seeks restart it (see `onTimePosUpdate` in `src/main.ts`).
|
seeks restart it (see `onTimePosUpdate` in `src/main.ts`).
|
||||||
|
|
||||||
The pause is released by the emit that carries the tokenized payload. Both controller methods
|
The pause is released by the controller's `onProcessingSettled` callback, which fires once it has
|
||||||
return whether an emit is expected, and the caller resumes immediately when it is not — otherwise
|
no work left. Emits do not release it: the first emit for an uncached line is the plain payload
|
||||||
a repeated subtitle (which schedules no work) would leave prefetching idle for the rest of the
|
that precedes tokenization, and a run can finish without emitting at all (a suppressed duplicate,
|
||||||
cue.
|
a failed tokenization). Both controller methods return whether processing is now pending, and the
|
||||||
|
caller resumes immediately when it is not — a repeated subtitle schedules no work, so no settle is
|
||||||
|
coming and prefetching would otherwise idle for the rest of the cue.
|
||||||
|
|
||||||
## Live Cue Delivery
|
## Live Cue Delivery
|
||||||
|
|
||||||
|
|||||||
@@ -593,6 +593,48 @@ test('refreshCurrentSubtitle reports the empty-text emit that an in-flight run w
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('onProcessingSettled fires once after the queue drains, including runs that emit nothing', async () => {
|
||||||
|
const events: string[] = [];
|
||||||
|
let resolveFirst: ((value: SubtitleData | null) => void) | undefined;
|
||||||
|
let tokenizationFails = false;
|
||||||
|
const controller = createSubtitleProcessingController({
|
||||||
|
tokenizeSubtitle: async (text) => {
|
||||||
|
if (tokenizationFails) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (text === '一行目') {
|
||||||
|
return await new Promise<SubtitleData | null>((resolve) => {
|
||||||
|
resolveFirst = resolve;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return { text, tokens: [] };
|
||||||
|
},
|
||||||
|
emitSubtitle: (payload) => events.push(`emit:${payload.text}`),
|
||||||
|
onProcessingSettled: () => events.push('settled'),
|
||||||
|
});
|
||||||
|
|
||||||
|
controller.onSubtitleChange('一行目');
|
||||||
|
await flushMicrotasks();
|
||||||
|
// A second line arrives before the first finishes: the controller still has
|
||||||
|
// work, so it must not report itself settled between the two.
|
||||||
|
controller.onSubtitleChange('二行目');
|
||||||
|
resolveFirst?.({ text: '一行目', tokens: [] });
|
||||||
|
await flushMicrotasks();
|
||||||
|
await flushMicrotasks();
|
||||||
|
|
||||||
|
assert.deepEqual(events, ['emit:一行目', 'emit:二行目', 'emit:二行目', 'settled']);
|
||||||
|
|
||||||
|
// Tokenization failure on a line already shown plain: nothing is emitted, and
|
||||||
|
// the settle signal is the only way a caller learns the work is over.
|
||||||
|
events.length = 0;
|
||||||
|
tokenizationFails = true;
|
||||||
|
controller.invalidateTokenizationCache();
|
||||||
|
assert.equal(controller.refreshCurrentSubtitle('二行目'), true);
|
||||||
|
await flushMicrotasks();
|
||||||
|
await flushMicrotasks();
|
||||||
|
assert.deepEqual(events, ['settled']);
|
||||||
|
});
|
||||||
|
|
||||||
test('refreshCurrentSubtitle reports no emit for empty text when nothing is running', async () => {
|
test('refreshCurrentSubtitle reports no emit for empty text when nothing is running', async () => {
|
||||||
const emitted: SubtitleData[] = [];
|
const emitted: SubtitleData[] = [];
|
||||||
const controller = createSubtitleProcessingController({
|
const controller = createSubtitleProcessingController({
|
||||||
|
|||||||
@@ -3,6 +3,14 @@ import type { SubtitleData } from '../../types';
|
|||||||
export interface SubtitleProcessingControllerDeps {
|
export interface SubtitleProcessingControllerDeps {
|
||||||
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
||||||
emitSubtitle: (payload: SubtitleData) => void;
|
emitSubtitle: (payload: SubtitleData) => void;
|
||||||
|
/**
|
||||||
|
* Fires when the controller runs out of work: every scheduled line has been
|
||||||
|
* processed, whether it ended in an emit, a suppressed duplicate, or a
|
||||||
|
* tokenizer failure. Callers that hold a resource for the duration of
|
||||||
|
* processing (prefetch pausing) release it here rather than on an emit,
|
||||||
|
* which is not guaranteed to happen.
|
||||||
|
*/
|
||||||
|
onProcessingSettled?: () => void;
|
||||||
logDebug?: (message: string) => void;
|
logDebug?: (message: string) => void;
|
||||||
now?: () => number;
|
now?: () => number;
|
||||||
cacheLimit?: number;
|
cacheLimit?: number;
|
||||||
@@ -18,12 +26,13 @@ export const DEFAULT_SUBTITLE_TOKENIZATION_CACHE_LIMIT = 2500;
|
|||||||
|
|
||||||
export interface SubtitleProcessingController {
|
export interface SubtitleProcessingController {
|
||||||
/**
|
/**
|
||||||
* Returns whether the text was new and processing was scheduled. A false
|
* Returns whether processing is now scheduled or already in flight for this
|
||||||
* return means nothing will be emitted for this event, which callers that
|
* event. A false return means the controller is idle and will do nothing, so
|
||||||
* gate work on the emit (such as pausing subtitle prefetching) need to know.
|
* onProcessingSettled will not fire; callers that pause work for the duration
|
||||||
|
* of processing (such as subtitle prefetching) must release it themselves.
|
||||||
*/
|
*/
|
||||||
onSubtitleChange: (text: string) => boolean;
|
onSubtitleChange: (text: string) => boolean;
|
||||||
/** Same contract as onSubtitleChange: whether an emit is expected. */
|
/** Same contract as onSubtitleChange: whether processing is pending. */
|
||||||
refreshCurrentSubtitle: (textOverride?: string) => boolean;
|
refreshCurrentSubtitle: (textOverride?: string) => boolean;
|
||||||
invalidateTokenizationCache: () => void;
|
invalidateTokenizationCache: () => void;
|
||||||
preCacheTokenization: (text: string, data: SubtitleData) => void;
|
preCacheTokenization: (text: string, data: SubtitleData) => void;
|
||||||
@@ -170,7 +179,12 @@ export function createSubtitleProcessingController(
|
|||||||
(latestText.trim() && cacheGeneration !== lastEmittedGeneration)
|
(latestText.trim() && cacheGeneration !== lastEmittedGeneration)
|
||||||
) {
|
) {
|
||||||
processLatest();
|
processLatest();
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
|
// Nothing left to do: signal completion even when this run emitted
|
||||||
|
// nothing (suppressed duplicate, tokenizer failure), or callers waiting
|
||||||
|
// on the controller would wait forever.
|
||||||
|
deps.onProcessingSettled?.();
|
||||||
});
|
});
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -121,7 +121,10 @@ const NAME_SCAN_WORDS: Array<[string, string, string, boolean]> = [
|
|||||||
['いない', 'いる', 'いる', false],
|
['いない', 'いる', 'いる', false],
|
||||||
];
|
];
|
||||||
|
|
||||||
function createNameScanDeps(lookups: string[]) {
|
function createNameScanDeps(
|
||||||
|
lookups: string[],
|
||||||
|
words: Array<[string, string, string, boolean]> = NAME_SCAN_WORDS,
|
||||||
|
) {
|
||||||
return createScanDeps((action, params) => {
|
return createScanDeps((action, params) => {
|
||||||
if (action === 'optionsGetFull') {
|
if (action === 'optionsGetFull') {
|
||||||
return {
|
return {
|
||||||
@@ -151,7 +154,7 @@ function createNameScanDeps(lookups: string[]) {
|
|||||||
}
|
}
|
||||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
lookups.push(text);
|
lookups.push(text);
|
||||||
for (const [surface, term, reading, isName] of NAME_SCAN_WORDS) {
|
for (const [surface, term, reading, isName] of words) {
|
||||||
if (text.startsWith(surface)) {
|
if (text.startsWith(surface)) {
|
||||||
return {
|
return {
|
||||||
originalTextLength: surface.length,
|
originalTextLength: surface.length,
|
||||||
@@ -234,6 +237,39 @@ test('requestYomitanScanTokens matches a katakana name from its kana-normalized
|
|||||||
assert.equal(result?.[0]?.isNameMatch, true);
|
assert.equal(result?.[0]?.isNameMatch, true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Kana normalization does not fold halfwidth katakana, so a name written that
|
||||||
|
// way can never prefix-match a candidate form; the pre-pass has a bypass for
|
||||||
|
// those positions, which only runs if they count as Japanese in the first place.
|
||||||
|
// The generic word here reaches into the name, so only a pre-pass reservation
|
||||||
|
// can keep the name whole.
|
||||||
|
const HALFWIDTH_NAME_SCAN_WORDS: Array<[string, string, string, boolean]> = [
|
||||||
|
['まだﾐ', 'まだミ', 'まだみ', false],
|
||||||
|
['まだ', 'まだ', 'まだ', false],
|
||||||
|
['ﾐﾅﾄ', 'ミナト', 'みなと', true],
|
||||||
|
];
|
||||||
|
|
||||||
|
test('requestYomitanScanTokens probes halfwidth katakana positions during the name pre-pass', async () => {
|
||||||
|
const lookups: string[] = [];
|
||||||
|
const result = await requestYomitanScanTokens(
|
||||||
|
'まだﾐﾅﾄ',
|
||||||
|
createNameScanDeps(lookups, HALFWIDTH_NAME_SCAN_WORDS),
|
||||||
|
{ error: () => undefined },
|
||||||
|
{
|
||||||
|
includeNameMatchMetadata: true,
|
||||||
|
currentCharacterDictionaryMediaId: 1,
|
||||||
|
// Fullwidth forms only, as the generated dictionary stores them.
|
||||||
|
nameCandidates: { key: 'media-1', forms: ['ミナト', 'みなと'] },
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(countTermsFindLookups(lookups, 'ﾐﾅﾄ'), 1);
|
||||||
|
assert.deepEqual(
|
||||||
|
result?.map((token) => token.surface),
|
||||||
|
['まだ', 'ﾐﾅﾄ'],
|
||||||
|
);
|
||||||
|
assert.equal(result?.[1]?.isNameMatch, true);
|
||||||
|
});
|
||||||
|
|
||||||
test('requestYomitanScanTokens falls back to the exhaustive name scan without candidates', async () => {
|
test('requestYomitanScanTokens falls back to the exhaustive name scan without candidates', async () => {
|
||||||
const withoutLookups: string[] = [];
|
const withoutLookups: string[] = [];
|
||||||
const withoutCandidates = await requestYomitanScanTokens(
|
const withoutCandidates = await requestYomitanScanTokens(
|
||||||
@@ -2061,6 +2097,102 @@ test('requestYomitanScanTokens lets a longer generic word beat a shorter name at
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('requestYomitanScanTokens lets a generic word beat a name it fully contains', async () => {
|
||||||
|
const nameEntry = (term: string, reading: string) => ({
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term,
|
||||||
|
reading,
|
||||||
|
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
definitions: [
|
||||||
|
{
|
||||||
|
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
|
||||||
|
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
|
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term,
|
||||||
|
reading,
|
||||||
|
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
const deps = createScanDeps((action, params) => {
|
||||||
|
if (action === 'optionsGetFull') {
|
||||||
|
return {
|
||||||
|
profileCurrent: 0,
|
||||||
|
profiles: [
|
||||||
|
{
|
||||||
|
options: {
|
||||||
|
scanning: { length: 40 },
|
||||||
|
dictionaries: [
|
||||||
|
{ name: 'JMdict', enabled: true },
|
||||||
|
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (action === 'getDictionaryInfo') {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
if (action !== 'termsFind') {
|
||||||
|
throw new Error(`unexpected action: ${action}`);
|
||||||
|
}
|
||||||
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
|
if (text.startsWith('写真')) {
|
||||||
|
return {
|
||||||
|
originalTextLength: 2,
|
||||||
|
dictionaryEntries: [jmdictEntry('写真', 'しゃしん', '写真')],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (text.startsWith('写')) {
|
||||||
|
return { originalTextLength: 1, dictionaryEntries: [jmdictEntry('写', 'しゃ', '写')] };
|
||||||
|
}
|
||||||
|
if (text.startsWith('真')) {
|
||||||
|
// The given name of 安田真 also matches the second half of 写真.
|
||||||
|
return {
|
||||||
|
originalTextLength: 1,
|
||||||
|
dictionaryEntries: [nameEntry('真', 'しん'), jmdictEntry('真', 'しん', '真')],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (text.startsWith('は')) {
|
||||||
|
return { originalTextLength: 1, dictionaryEntries: [jmdictEntry('は', 'は', 'は')] };
|
||||||
|
}
|
||||||
|
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await requestYomitanScanTokens(
|
||||||
|
'写真は',
|
||||||
|
deps,
|
||||||
|
{ error: () => undefined },
|
||||||
|
{ includeNameMatchMetadata: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(Array.isArray(result), true);
|
||||||
|
assert.deepEqual(
|
||||||
|
result?.map(({ surface, headword, startPos, endPos, isNameMatch }) => ({
|
||||||
|
surface,
|
||||||
|
headword,
|
||||||
|
startPos,
|
||||||
|
endPos,
|
||||||
|
isNameMatch,
|
||||||
|
})),
|
||||||
|
[
|
||||||
|
{ surface: '写真', headword: '写真', startPos: 0, endPos: 2, isNameMatch: false },
|
||||||
|
{ surface: 'は', headword: 'は', startPos: 2, endPos: 3, isNameMatch: false },
|
||||||
|
],
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
test('requestYomitanScanTokens skips greedy name scan without an enabled character dictionary', async () => {
|
test('requestYomitanScanTokens skips greedy name scan without an enabled character dictionary', async () => {
|
||||||
let scanCallScript = '';
|
let scanCallScript = '';
|
||||||
const deps = createScanDeps(
|
const deps = createScanDeps(
|
||||||
@@ -2388,6 +2520,103 @@ test('clearYomitanParserCachesForWindow invalidates the cross-line termsFind cac
|
|||||||
assert.equal(countTermsFindLookups(lookups, '猫'), 2);
|
assert.equal(countTermsFindLookups(lookups, '猫'), 2);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('an oversized termsFind result is dropped from the cache instead of being reused', async () => {
|
||||||
|
const lookups: string[] = [];
|
||||||
|
// One entry over the runtime's 20,000 retained-entry budget: the weight is
|
||||||
|
// only known once the lookup resolves, so the cache has to re-check then.
|
||||||
|
const oversizedEntries = Array.from({ length: 20_001 }, () => ({
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term: '猫',
|
||||||
|
reading: 'ねこ',
|
||||||
|
sources: [{ originalText: '猫', isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}));
|
||||||
|
const deps = createScanDeps((action, params) => {
|
||||||
|
if (action === 'optionsGetFull') {
|
||||||
|
return {
|
||||||
|
profileCurrent: 0,
|
||||||
|
profiles: [{ options: { scanning: { length: 40 } } }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (action === 'getDictionaryInfo') {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
|
lookups.push(text);
|
||||||
|
if (text.startsWith('猫')) {
|
||||||
|
return { originalTextLength: 1, dictionaryEntries: oversizedEntries };
|
||||||
|
}
|
||||||
|
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||||
|
});
|
||||||
|
|
||||||
|
await requestYomitanScanTokens('猫', deps, { error: () => undefined });
|
||||||
|
await requestYomitanScanTokens('猫', deps, { error: () => undefined });
|
||||||
|
|
||||||
|
assert.equal(countTermsFindLookups(lookups, '猫'), 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scanner tokens survive a retry-budget escalation whose parseText finds nothing', async () => {
|
||||||
|
const parsedTexts: string[] = [];
|
||||||
|
const deps = createScanDeps((action, params) => {
|
||||||
|
if (action === 'optionsGetFull') {
|
||||||
|
return {
|
||||||
|
profileCurrent: 0,
|
||||||
|
profiles: [{ options: { scanning: { length: 40 } } }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (action === 'getDictionaryInfo') {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
|
if (action === 'parseText') {
|
||||||
|
parsedTexts.push(text);
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
if (text.startsWith('猫')) {
|
||||||
|
return {
|
||||||
|
originalTextLength: 1,
|
||||||
|
dictionaryEntries: [
|
||||||
|
{
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term: '猫',
|
||||||
|
reading: 'ねこ',
|
||||||
|
sources: [{ originalText: '猫', isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
// The rest of the line burns the blind-retry budget at every position.
|
||||||
|
return {
|
||||||
|
originalTextLength: text.length,
|
||||||
|
dictionaryEntries: [
|
||||||
|
{
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term: 'ミスマッチ',
|
||||||
|
reading: 'みすまっち',
|
||||||
|
sources: [{ originalText: 'ZZZ', isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await requestYomitanScanTokens('猫あいうえおかきくけこ', deps, {
|
||||||
|
error: () => undefined,
|
||||||
|
});
|
||||||
|
|
||||||
|
// The escalation ran exactly once and found nothing, so the tokens the
|
||||||
|
// scanner did resolve are kept instead of dropping the line to raw text.
|
||||||
|
assert.deepEqual(parsedTexts, ['猫あいうえおかきくけこ']);
|
||||||
|
assert.equal(result?.[0]?.surface, '猫');
|
||||||
|
});
|
||||||
|
|
||||||
test('requestYomitanScanTokens skips termsFind lookups at punctuation and whitespace positions', async () => {
|
test('requestYomitanScanTokens skips termsFind lookups at punctuation and whitespace positions', async () => {
|
||||||
const lookups: string[] = [];
|
const lookups: string[] = [];
|
||||||
const deps = createScanDeps(createSingleTermScanHandler(lookups));
|
const deps = createScanDeps(createSingleTermScanHandler(lookups));
|
||||||
@@ -2402,8 +2631,9 @@ test('requestYomitanScanTokens skips termsFind lookups at punctuation and whites
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
test('requestYomitanScanTokens caps the shrinking-window retry ladder per position', async () => {
|
test('requestYomitanScanTokens caps blind retries and escalates the line to parseText', async () => {
|
||||||
const lookups: string[] = [];
|
const lookups: string[] = [];
|
||||||
|
const parsedTexts: string[] = [];
|
||||||
const deps = createScanDeps((action, params) => {
|
const deps = createScanDeps((action, params) => {
|
||||||
if (action === 'optionsGetFull') {
|
if (action === 'optionsGetFull') {
|
||||||
return {
|
return {
|
||||||
@@ -2415,9 +2645,22 @@ test('requestYomitanScanTokens caps the shrinking-window retry ladder per positi
|
|||||||
return [];
|
return [];
|
||||||
}
|
}
|
||||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
|
if (action === 'parseText') {
|
||||||
|
parsedTexts.push(text);
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
source: 'scanning-parser',
|
||||||
|
index: 0,
|
||||||
|
content: [
|
||||||
|
[{ text: 'あいうえお', reading: 'あいうえお', headwords: [[{ term: 'あい' }]] }],
|
||||||
|
],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}
|
||||||
lookups.push(text);
|
lookups.push(text);
|
||||||
// Every window "matches" its whole length but never yields an
|
// Every window "matches" its whole length but never yields an
|
||||||
// exact-source headword, the worst case for the retry ladder.
|
// exact-source headword, the worst case for the retry ladder: each step
|
||||||
|
// down is a blind guess with nothing shorter reported to aim at.
|
||||||
return {
|
return {
|
||||||
originalTextLength: text.length,
|
originalTextLength: text.length,
|
||||||
dictionaryEntries: [
|
dictionaryEntries: [
|
||||||
@@ -2438,9 +2681,73 @@ test('requestYomitanScanTokens caps the shrinking-window retry ladder per positi
|
|||||||
error: () => undefined,
|
error: () => undefined,
|
||||||
});
|
});
|
||||||
|
|
||||||
assert.equal(result, null);
|
// Position 0: one initial window lookup plus at most four blind retries, so
|
||||||
// Position 0: one initial window lookup plus at most four shrinking retries.
|
// the ladder cannot degrade into a lookup per window length.
|
||||||
assert.equal(countTermsFindLookups(lookups, 'あいうえお'), 5);
|
assert.equal(countTermsFindLookups(lookups, 'あいうえお'), 5);
|
||||||
|
// Giving up there would leave the line unparsed, so it escalates to the one
|
||||||
|
// full parse the scanner normally replaces.
|
||||||
|
assert.deepEqual(parsedTexts, ['あいうえおかきくけこ']);
|
||||||
|
assert.equal(result?.[0]?.headword, 'あい');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('requestYomitanScanTokens keeps shrinking while the backend guides the retry ladder', async () => {
|
||||||
|
const lookups: string[] = [];
|
||||||
|
const deps = createScanDeps((action, params) => {
|
||||||
|
if (action === 'optionsGetFull') {
|
||||||
|
return {
|
||||||
|
profileCurrent: 0,
|
||||||
|
profiles: [{ options: { scanning: { length: 40 } } }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (action === 'getDictionaryInfo') {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||||
|
lookups.push(text);
|
||||||
|
// Normalization keeps eating one character past the term, so every window
|
||||||
|
// reports a shorter consumed length: informative steps that must not be
|
||||||
|
// spent from the blind-retry budget. The term only surfaces at length 2,
|
||||||
|
// six lookups down the ladder.
|
||||||
|
if (text.length === 2) {
|
||||||
|
return {
|
||||||
|
originalTextLength: 2,
|
||||||
|
dictionaryEntries: [
|
||||||
|
{
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term: 'あい',
|
||||||
|
reading: 'あい',
|
||||||
|
sources: [{ originalText: 'あい', isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
originalTextLength: Math.max(text.length - 1, 0),
|
||||||
|
dictionaryEntries: [
|
||||||
|
{
|
||||||
|
headwords: [
|
||||||
|
{
|
||||||
|
term: 'ミスマッチ',
|
||||||
|
reading: 'みすまっち',
|
||||||
|
sources: [{ originalText: 'ZZZ', isPrimary: true, matchType: 'exact' }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await requestYomitanScanTokens('あいうえおかきくけこさしすせ', deps, {
|
||||||
|
error: () => undefined,
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(result?.[0]?.surface, 'あい');
|
||||||
|
// Windows of 14, 12, 10, 8, 6, 4 characters, then the match at 2: a ladder
|
||||||
|
// capped at four lookups would stop at 6 and leave the line unparsed.
|
||||||
|
assert.equal(countTermsFindLookups(lookups, 'あい'), 7);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('requestYomitanScanTokens falls back to parseText when the scanner eval fails', async () => {
|
test('requestYomitanScanTokens falls back to parseText when the scanner eval fails', async () => {
|
||||||
|
|||||||
@@ -1042,6 +1042,19 @@ export async function requestYomitanScanTokens(
|
|||||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||||
rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||||
}
|
}
|
||||||
|
// The scanner reports a line where a position ran out of shrinking-window
|
||||||
|
// retries: it stopped short of windows an uncapped ladder would have tried,
|
||||||
|
// so a real term may be sitting in an unparsed run. One parseText for the
|
||||||
|
// line is the bounded way to get the exhaustive answer back (this is the
|
||||||
|
// parse the scanner replaced, and it only runs for these rare lines).
|
||||||
|
if (isObject(rawResult) && rawResult.retryBudgetExhausted === true) {
|
||||||
|
logger.info?.('Yomitan scanner exhausted its retry budget; parsing the line as a fallback.');
|
||||||
|
const fallbackTokens = await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||||
|
if (fallbackTokens) {
|
||||||
|
return fallbackTokens;
|
||||||
|
}
|
||||||
|
rawResult = rawResult.tokens;
|
||||||
|
}
|
||||||
if (isScanTokenArray(rawResult)) {
|
if (isScanTokenArray(rawResult)) {
|
||||||
// Filler-only results carry no dictionary match; keep the historical
|
// Filler-only results carry no dictionary match; keep the historical
|
||||||
// contract of returning null so callers fall back to raw text.
|
// contract of returning null so callers fall back to raw text.
|
||||||
|
|||||||
@@ -1,501 +1,17 @@
|
|||||||
// In-page Yomitan scan runtime: the helper bundle and scan walk that get
|
// In-page Yomitan scan runtime: the scan walk that gets installed once per
|
||||||
// installed once per parser window as globalThis.__subminerYomitanScan, plus
|
// parser window as globalThis.__subminerYomitanScan, plus the tiny per-line
|
||||||
// the tiny per-line call script. Kept separate from the host runtime module so
|
// call script. Kept separate from the host runtime module so the injected
|
||||||
// the injected-script text (which is data, not executed here) does not
|
// script text (which is data, not executed here) does not dominate that file;
|
||||||
// dominate that file.
|
// the helper bundle it embeds lives in yomitan-scanning-helpers-script.ts.
|
||||||
|
import { YOMITAN_SCANNING_HELPERS } from './yomitan-scanning-helpers-script';
|
||||||
|
|
||||||
|
export { CHARACTER_DICTIONARY_TITLE_PREFIX } from './yomitan-scanning-helpers-script';
|
||||||
|
|
||||||
export type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
export type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
||||||
|
|
||||||
export const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
|
||||||
|
|
||||||
const YOMITAN_SCANNING_HELPERS = String.raw`
|
|
||||||
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
|
||||||
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
|
||||||
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
|
||||||
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
|
||||||
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
|
||||||
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
|
||||||
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0x3400, 0x9fff]];
|
|
||||||
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
|
||||||
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
|
||||||
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
|
||||||
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
|
||||||
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
|
||||||
function getSegmentReadingContribution(segment) {
|
|
||||||
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
|
||||||
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
|
||||||
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
|
||||||
return isKanaOnly ? segmentText : "";
|
|
||||||
}
|
|
||||||
function getProlongedHiragana(previousCharacter) {
|
|
||||||
switch (previousCharacter) {
|
|
||||||
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
|
||||||
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
|
||||||
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
|
||||||
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
|
||||||
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
|
||||||
default: return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function getFuriganaKanaSegments(text, reading) {
|
|
||||||
const newSegments = [];
|
|
||||||
let start = 0;
|
|
||||||
let state = (reading[0] === text[0]);
|
|
||||||
for (let i = 1; i < text.length; ++i) {
|
|
||||||
const newState = (reading[i] === text[i]);
|
|
||||||
if (state === newState) { continue; }
|
|
||||||
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
|
||||||
state = newState;
|
|
||||||
start = i;
|
|
||||||
}
|
|
||||||
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
|
||||||
return newSegments;
|
|
||||||
}
|
|
||||||
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
|
||||||
let result = '';
|
|
||||||
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
|
||||||
for (let char of text) {
|
|
||||||
const codePoint = char.codePointAt(0);
|
|
||||||
switch (codePoint) {
|
|
||||||
case KATAKANA_SMALL_KA_CODE_POINT:
|
|
||||||
case KATAKANA_SMALL_KE_CODE_POINT:
|
|
||||||
break;
|
|
||||||
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
|
||||||
if (!keepProlongedSoundMarks && result.length > 0) {
|
|
||||||
const char2 = getProlongedHiragana(result[result.length - 1]);
|
|
||||||
if (char2 !== null) { char = char2; }
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
|
||||||
char = String.fromCodePoint(codePoint + offset);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
result += char;
|
|
||||||
}
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
|
||||||
const groupCount = groups.length - groupsStart;
|
|
||||||
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
|
||||||
const group = groups[groupsStart];
|
|
||||||
const {isKana, text} = group;
|
|
||||||
if (isKana) {
|
|
||||||
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
|
||||||
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
|
||||||
if (segments !== null) {
|
|
||||||
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
|
||||||
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
|
||||||
return segments;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
let result = null;
|
|
||||||
for (let i = reading.length; i >= text.length; --i) {
|
|
||||||
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
|
||||||
if (segments !== null) {
|
|
||||||
if (result !== null) { return null; }
|
|
||||||
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
|
||||||
result = segments;
|
|
||||||
}
|
|
||||||
if (groupCount === 1) { break; }
|
|
||||||
}
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
function distributeFurigana(term, reading) {
|
|
||||||
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
|
||||||
const groups = [];
|
|
||||||
let groupPre = null;
|
|
||||||
let isKanaPre = null;
|
|
||||||
for (const c of term) {
|
|
||||||
const isKana = isCodePointKana(c.codePointAt(0));
|
|
||||||
if (isKana === isKanaPre) { groupPre.text += c; }
|
|
||||||
else {
|
|
||||||
groupPre = {isKana, text: c, textNormalized: null};
|
|
||||||
groups.push(groupPre);
|
|
||||||
isKanaPre = isKana;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (const group of groups) {
|
|
||||||
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
|
||||||
}
|
|
||||||
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
|
||||||
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
|
||||||
}
|
|
||||||
function getStemLength(text1, text2) {
|
|
||||||
const minLength = Math.min(text1.length, text2.length);
|
|
||||||
if (minLength === 0) { return 0; }
|
|
||||||
let i = 0;
|
|
||||||
while (true) {
|
|
||||||
const char1 = text1.codePointAt(i);
|
|
||||||
const char2 = text2.codePointAt(i);
|
|
||||||
if (char1 !== char2) { break; }
|
|
||||||
const charLength = String.fromCodePoint(char1).length;
|
|
||||||
i += charLength;
|
|
||||||
if (i >= minLength) {
|
|
||||||
if (i > minLength) { i -= charLength; }
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return i;
|
|
||||||
}
|
|
||||||
function distributeFuriganaInflected(term, reading, source) {
|
|
||||||
const termNormalized = convertKatakanaToHiragana(term);
|
|
||||||
const readingNormalized = convertKatakanaToHiragana(reading);
|
|
||||||
const sourceNormalized = convertKatakanaToHiragana(source);
|
|
||||||
let mainText = term;
|
|
||||||
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
|
||||||
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
|
||||||
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
|
||||||
mainText = reading;
|
|
||||||
stemLength = readingStemLength;
|
|
||||||
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
|
||||||
}
|
|
||||||
const segments = [];
|
|
||||||
if (stemLength > 0) {
|
|
||||||
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
|
||||||
const segments2 = distributeFurigana(mainText, reading);
|
|
||||||
let consumed = 0;
|
|
||||||
for (const segment of segments2) {
|
|
||||||
const start = consumed;
|
|
||||||
consumed += segment.text.length;
|
|
||||||
if (consumed < stemLength) { segments.push(segment); }
|
|
||||||
else if (consumed === stemLength) { segments.push(segment); break; }
|
|
||||||
else {
|
|
||||||
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (stemLength < source.length) {
|
|
||||||
const remainder = source.substring(stemLength);
|
|
||||||
const last = segments[segments.length - 1];
|
|
||||||
if (last && last.reading.length === 0) { last.text += remainder; }
|
|
||||||
else { segments.push(createFuriganaSegment(remainder, '')); }
|
|
||||||
}
|
|
||||||
return segments;
|
|
||||||
}
|
|
||||||
function parsePositiveFrequencyNumber(value) {
|
|
||||||
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
|
||||||
return Math.max(1, Math.floor(value));
|
|
||||||
}
|
|
||||||
if (typeof value === 'string') {
|
|
||||||
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
|
||||||
if (!numericMatch) { return null; }
|
|
||||||
const parsed = Number.parseFloat(numericMatch);
|
|
||||||
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
|
||||||
return Math.max(1, Math.floor(parsed));
|
|
||||||
}
|
|
||||||
if (Array.isArray(value)) {
|
|
||||||
for (const item of value) {
|
|
||||||
const parsed = parsePositiveFrequencyNumber(item);
|
|
||||||
if (parsed !== null) { return parsed; }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
function parseDisplayFrequencyNumber(value) {
|
|
||||||
if (typeof value === 'string') {
|
|
||||||
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
|
||||||
if (!leadingDigits) { return null; }
|
|
||||||
const parsed = Number.parseInt(leadingDigits, 10);
|
|
||||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
|
||||||
}
|
|
||||||
return parsePositiveFrequencyNumber(value);
|
|
||||||
}
|
|
||||||
function getFrequencyDictionaryName(frequency) {
|
|
||||||
const candidates = [
|
|
||||||
frequency?.dictionary,
|
|
||||||
frequency?.dictionaryName,
|
|
||||||
frequency?.name,
|
|
||||||
frequency?.title,
|
|
||||||
frequency?.dictionaryTitle,
|
|
||||||
frequency?.dictionaryAlias
|
|
||||||
];
|
|
||||||
for (const candidate of candidates) {
|
|
||||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
|
||||||
return candidate.trim();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
|
||||||
let best = null;
|
|
||||||
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
|
||||||
for (const frequency of dictionaryEntry?.frequencies || []) {
|
|
||||||
if (!frequency || typeof frequency !== 'object') { continue; }
|
|
||||||
const frequencyHeadwordIndex = frequency.headwordIndex;
|
|
||||||
if (typeof frequencyHeadwordIndex === 'number') {
|
|
||||||
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
|
||||||
} else if (headwordCount > 1) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
const dictionary = getFrequencyDictionaryName(frequency);
|
|
||||||
if (!dictionary) { continue; }
|
|
||||||
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
|
||||||
const rank =
|
|
||||||
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
|
||||||
parsePositiveFrequencyNumber(frequency.frequency);
|
|
||||||
if (rank === null) { continue; }
|
|
||||||
const priorityRaw = dictionaryPriorityByName[dictionary];
|
|
||||||
const fallbackPriority =
|
|
||||||
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
|
||||||
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
|
||||||
: Number.MAX_SAFE_INTEGER;
|
|
||||||
const priority =
|
|
||||||
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
|
||||||
? Math.max(0, Math.floor(priorityRaw))
|
|
||||||
: fallbackPriority;
|
|
||||||
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
|
||||||
best = { priority, rank };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return best?.rank ?? null;
|
|
||||||
}
|
|
||||||
function hasExactSource(headword, token, requirePrimary) {
|
|
||||||
for (const src of headword.sources || []) {
|
|
||||||
if (src.originalText !== token) { continue; }
|
|
||||||
if (requirePrimary && !src.isPrimary) { continue; }
|
|
||||||
if (src.matchType !== 'exact') { continue; }
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
|
||||||
const matches = [];
|
|
||||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
|
||||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
|
||||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
|
||||||
const headword = headwords[headwordIndex];
|
|
||||||
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
|
||||||
matches.push({ dictionaryEntry, headword, headwordIndex });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return matches;
|
|
||||||
}
|
|
||||||
function sameHeadword(match, preferredMatch) {
|
|
||||||
if (!match || !preferredMatch) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if (match.headword?.term !== preferredMatch.headword?.term) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
|
||||||
const preferredReading =
|
|
||||||
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
|
||||||
if (!matchReading || !preferredReading) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
return matchReading === preferredReading;
|
|
||||||
}
|
|
||||||
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
|
||||||
let best = null;
|
|
||||||
for (const match of matches) {
|
|
||||||
const rank = getBestFrequencyRank(
|
|
||||||
match.dictionaryEntry,
|
|
||||||
match.headwordIndex,
|
|
||||||
dictionaryPriorityByName,
|
|
||||||
dictionaryFrequencyModeByName
|
|
||||||
);
|
|
||||||
if (rank === null) { continue; }
|
|
||||||
if (best === null || rank < best) {
|
|
||||||
best = rank;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return best;
|
|
||||||
}
|
|
||||||
function normalizeWordClasses(headword) {
|
|
||||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
|
||||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
|
||||||
return classes.length > 0 ? classes : undefined;
|
|
||||||
}
|
|
||||||
function appendDictionaryNames(target, value) {
|
|
||||||
if (!value || typeof value !== 'object') {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const candidates = [
|
|
||||||
value.dictionary,
|
|
||||||
value.dictionaryName,
|
|
||||||
value.name,
|
|
||||||
value.title,
|
|
||||||
value.dictionaryTitle,
|
|
||||||
value.dictionaryAlias
|
|
||||||
];
|
|
||||||
for (const candidate of candidates) {
|
|
||||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
|
||||||
target.push(candidate.trim());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function getDictionaryEntryNames(entry) {
|
|
||||||
const names = [];
|
|
||||||
appendDictionaryNames(names, entry);
|
|
||||||
for (const definition of entry?.definitions || []) {
|
|
||||||
appendDictionaryNames(names, definition);
|
|
||||||
}
|
|
||||||
for (const frequency of entry?.frequencies || []) {
|
|
||||||
appendDictionaryNames(names, frequency);
|
|
||||||
}
|
|
||||||
for (const pronunciation of entry?.pronunciations || []) {
|
|
||||||
appendDictionaryNames(names, pronunciation);
|
|
||||||
}
|
|
||||||
return names;
|
|
||||||
}
|
|
||||||
function isNameDictionaryEntry(entry) {
|
|
||||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
|
||||||
}
|
|
||||||
function parseSubMinerMediaIdFromString(value) {
|
|
||||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
|
||||||
if (imageMatch) {
|
|
||||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
|
||||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
|
||||||
}
|
|
||||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
|
||||||
if (titleMatch) {
|
|
||||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
|
||||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
function parseSubMinerMediaIdCandidate(value) {
|
|
||||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
|
||||||
const parsed = Number.parseInt(value.trim(), 10);
|
|
||||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
function collectSubMinerMediaIds(value, target) {
|
|
||||||
if (typeof value === 'string') {
|
|
||||||
const parsed = parseSubMinerMediaIdFromString(value);
|
|
||||||
if (parsed !== null) { target.add(parsed); }
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (!value || typeof value !== 'object') {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (Array.isArray(value)) {
|
|
||||||
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const mediaIdCandidates = [
|
|
||||||
value.subminerMediaId,
|
|
||||||
value.subMinerMediaId,
|
|
||||||
value.characterDictionaryMediaId,
|
|
||||||
value.data?.subminerMediaId,
|
|
||||||
value.data?.subMinerMediaId,
|
|
||||||
value.data?.characterDictionaryMediaId
|
|
||||||
];
|
|
||||||
for (const candidate of mediaIdCandidates) {
|
|
||||||
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
|
||||||
if (parsed !== null) { target.add(parsed); }
|
|
||||||
}
|
|
||||||
for (const child of Object.values(value)) {
|
|
||||||
collectSubMinerMediaIds(child, target);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function getSubMinerMediaIds(entry) {
|
|
||||||
const mediaIds = new Set();
|
|
||||||
collectSubMinerMediaIds(entry, mediaIds);
|
|
||||||
return mediaIds;
|
|
||||||
}
|
|
||||||
function isCurrentMediaNameDictionaryEntry(entry) {
|
|
||||||
if (!isNameDictionaryEntry(entry)) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if (currentCharacterDictionaryMediaId === null) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
const mediaIds = getSubMinerMediaIds(entry);
|
|
||||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
|
||||||
}
|
|
||||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
|
||||||
let best = null;
|
|
||||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
|
||||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
|
||||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
|
||||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
|
||||||
const headword = headwords[headwordIndex];
|
|
||||||
for (const src of headword?.sources || []) {
|
|
||||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
|
||||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
|
||||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
|
||||||
if (best === null || originalText.length > best.sourceLength) {
|
|
||||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return best;
|
|
||||||
}
|
|
||||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
|
||||||
let best = 0;
|
|
||||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
|
||||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
|
||||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
|
||||||
for (const headword of headwords) {
|
|
||||||
for (const src of headword?.sources || []) {
|
|
||||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
|
||||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
|
||||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
|
||||||
if (originalText.length > best) { best = originalText.length; }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return best;
|
|
||||||
}
|
|
||||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
|
||||||
const currentMediaDictionaryEntries =
|
|
||||||
currentCharacterDictionaryMediaId === null
|
|
||||||
? (dictionaryEntries || [])
|
|
||||||
: (dictionaryEntries || []).filter((entry) => {
|
|
||||||
if (!isNameDictionaryEntry(entry)) { return true; }
|
|
||||||
return isCurrentMediaNameDictionaryEntry(entry);
|
|
||||||
});
|
|
||||||
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
|
||||||
let matchedNameDictionary = false;
|
|
||||||
if (includeNameMatchMetadata) {
|
|
||||||
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
|
||||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
|
||||||
for (const match of exactPrimaryMatches) {
|
|
||||||
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
|
||||||
matchedNameDictionary = true;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
if (matchedNameDictionary) { break; }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const preferredMatch = exactPrimaryMatches[0];
|
|
||||||
if (preferredMatch) {
|
|
||||||
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
|
||||||
.filter((match) => sameHeadword(match, preferredMatch));
|
|
||||||
return {
|
|
||||||
term: preferredMatch.headword.term,
|
|
||||||
reading: preferredMatch.headword.reading,
|
|
||||||
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
|
||||||
isNameMatch:
|
|
||||||
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
|
||||||
frequencyRank: getBestFrequencyRankForMatches(
|
|
||||||
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
|
||||||
dictionaryPriorityByName,
|
|
||||||
dictionaryFrequencyModeByName
|
|
||||||
)
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
|
|
||||||
// Bump whenever the install script below changes so already-loaded parser
|
// Bump whenever the install script below changes so already-loaded parser
|
||||||
// windows re-install the new scan runtime instead of running the stale one.
|
// windows re-install the new scan runtime instead of running the stale one.
|
||||||
export const YOMITAN_SCAN_RUNTIME_VERSION = 4;
|
export const YOMITAN_SCAN_RUNTIME_VERSION = 6;
|
||||||
export const YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL = '__subminer-yomitan-scan-runtime-missing__';
|
export const YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL = '__subminer-yomitan-scan-runtime-missing__';
|
||||||
|
|
||||||
export interface YomitanScanRequestParams {
|
export interface YomitanScanRequestParams {
|
||||||
@@ -546,9 +62,38 @@ export const YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT = String.raw`
|
|||||||
// repeat particles and inflections constantly, so most lookups hit here.
|
// repeat particles and inflections constantly, so most lookups hit here.
|
||||||
// Entries hold in-flight promises so concurrent identical lookups dedupe.
|
// Entries hold in-flight promises so concurrent identical lookups dedupe.
|
||||||
const termsFindCache = new Map();
|
const termsFindCache = new Map();
|
||||||
|
// Two bounds. The key count keeps the map itself small; the accumulated
|
||||||
|
// dictionary-entry count stands in for retained bytes, because a single
|
||||||
|
// lookup over a common prefix can hold hundreds of entries with their full
|
||||||
|
// glossaries and a key-count cap alone would not bound that.
|
||||||
const TERMS_FIND_CACHE_LIMIT = 2000;
|
const TERMS_FIND_CACHE_LIMIT = 2000;
|
||||||
|
const TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT = 20000;
|
||||||
|
let termsFindCacheDictionaryEntries = 0;
|
||||||
let termsFindCacheEpoch = -1;
|
let termsFindCacheEpoch = -1;
|
||||||
const MAX_SHRINKING_WINDOW_RETRY_LOOKUPS = 4;
|
function dropCachedTermsFind(cacheKey, entry) {
|
||||||
|
if (termsFindCache.get(cacheKey) !== entry) { return; }
|
||||||
|
termsFindCache.delete(cacheKey);
|
||||||
|
termsFindCacheDictionaryEntries -= entry.dictionaryEntryCount;
|
||||||
|
}
|
||||||
|
// Runs on insert and again once a lookup resolves: an entry is only worth
|
||||||
|
// its estimated weight of 1 until then, so a single oversized response
|
||||||
|
// would otherwise sit in the cache forever, over the limit and reused.
|
||||||
|
function evictOverflowingTermsFindEntries() {
|
||||||
|
while (
|
||||||
|
termsFindCache.size > TERMS_FIND_CACHE_LIMIT ||
|
||||||
|
termsFindCacheDictionaryEntries > TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT
|
||||||
|
) {
|
||||||
|
const oldest = termsFindCache.entries().next().value;
|
||||||
|
if (oldest === undefined) { break; }
|
||||||
|
dropCachedTermsFind(oldest[0], oldest[1]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Only blind ladder steps are capped (see the retry loop): those are the
|
||||||
|
// ones that would otherwise degrade into O(scanLength) lookups at a single
|
||||||
|
// position. Steps the backend guides by reporting a shorter consumed length
|
||||||
|
// stay uncapped, so a valid prefix term is still found on lines where
|
||||||
|
// normalization eats a long tail.
|
||||||
|
const MAX_BLIND_SHRINKING_WINDOW_RETRIES = 4;
|
||||||
// Character-name candidate forms for the current media, installed
|
// Character-name candidate forms for the current media, installed
|
||||||
// separately from the per-line scan call so the per-line script stays tiny.
|
// separately from the per-line scan call so the per-line script stays tiny.
|
||||||
// Stored raw here; the normalized lookup index is built inside the scan,
|
// Stored raw here; the normalized lookup index is built inside the scan,
|
||||||
@@ -581,6 +126,7 @@ export const YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT = String.raw`
|
|||||||
} = scanParams;
|
} = scanParams;
|
||||||
if (cacheEpoch !== termsFindCacheEpoch) {
|
if (cacheEpoch !== termsFindCacheEpoch) {
|
||||||
termsFindCache.clear();
|
termsFindCache.clear();
|
||||||
|
termsFindCacheDictionaryEntries = 0;
|
||||||
termsFindCacheEpoch = cacheEpoch;
|
termsFindCacheEpoch = cacheEpoch;
|
||||||
}
|
}
|
||||||
${YOMITAN_SCANNING_HELPERS}
|
${YOMITAN_SCANNING_HELPERS}
|
||||||
@@ -602,24 +148,37 @@ ${YOMITAN_SCANNING_HELPERS}
|
|||||||
const tokens = [];
|
const tokens = [];
|
||||||
async function termsFindAt(position, windowLength) {
|
async function termsFindAt(position, windowLength) {
|
||||||
const substring = text.substring(position, position + windowLength);
|
const substring = text.substring(position, position + windowLength);
|
||||||
const cacheKey = profileIndex + " " + substring;
|
const cacheKey = profileIndex + "\u0000" + substring;
|
||||||
const cached = termsFindCache.get(cacheKey);
|
const cached = termsFindCache.get(cacheKey);
|
||||||
if (cached !== undefined) {
|
if (cached !== undefined) {
|
||||||
termsFindCache.delete(cacheKey);
|
termsFindCache.delete(cacheKey);
|
||||||
termsFindCache.set(cacheKey, cached);
|
termsFindCache.set(cacheKey, cached);
|
||||||
return await cached;
|
return await cached.promise;
|
||||||
}
|
|
||||||
const pending = invoke("termsFind", { text: substring, details, optionsContext: { index: profileIndex } });
|
|
||||||
termsFindCache.set(cacheKey, pending);
|
|
||||||
while (termsFindCache.size > TERMS_FIND_CACHE_LIMIT) {
|
|
||||||
const oldestKey = termsFindCache.keys().next().value;
|
|
||||||
if (oldestKey === undefined) { break; }
|
|
||||||
termsFindCache.delete(oldestKey);
|
|
||||||
}
|
}
|
||||||
|
// An in-flight lookup counts as one entry until it resolves; the real
|
||||||
|
// weight replaces that estimate once the result is known.
|
||||||
|
const entry = { promise: null, dictionaryEntryCount: 1 };
|
||||||
|
entry.promise = invoke("termsFind", { text: substring, details, optionsContext: { index: profileIndex } })
|
||||||
|
.then((result) => {
|
||||||
|
const resolvedCount =
|
||||||
|
1 + (Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries.length : 0);
|
||||||
|
const isCached = termsFindCache.get(cacheKey) === entry;
|
||||||
|
if (isCached) {
|
||||||
|
termsFindCacheDictionaryEntries += resolvedCount - entry.dictionaryEntryCount;
|
||||||
|
}
|
||||||
|
entry.dictionaryEntryCount = resolvedCount;
|
||||||
|
// The real weight can push the cache over its budget, and a single
|
||||||
|
// response can exceed it on its own, so re-check here.
|
||||||
|
if (isCached) { evictOverflowingTermsFindEntries(); }
|
||||||
|
return result;
|
||||||
|
});
|
||||||
|
termsFindCache.set(cacheKey, entry);
|
||||||
|
termsFindCacheDictionaryEntries += entry.dictionaryEntryCount;
|
||||||
|
evictOverflowingTermsFindEntries();
|
||||||
try {
|
try {
|
||||||
return await pending;
|
return await entry.promise;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
termsFindCache.delete(cacheKey);
|
dropCachedTermsFind(cacheKey, entry);
|
||||||
throw error;
|
throw error;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -682,6 +241,43 @@ ${YOMITAN_SCANNING_HELPERS}
|
|||||||
}
|
}
|
||||||
return tokenPayload;
|
return tokenPayload;
|
||||||
}
|
}
|
||||||
|
// findTokenAt plus the shrinking-window ladder below it: Yomitan text
|
||||||
|
// normalization can consume characters (whitespace, punctuation) beyond
|
||||||
|
// the matched term, leaving no headword whose source equals the consumed
|
||||||
|
// text. Retry with shorter windows so a valid prefix term (e.g. a
|
||||||
|
// character name before a paren) still tokenizes instead of the position
|
||||||
|
// being skipped.
|
||||||
|
// Every window at or above the consumed length repeats the same result,
|
||||||
|
// so the next informative window sits just below it. A lookup that
|
||||||
|
// consumed its whole window reports nothing to aim at, and the step down
|
||||||
|
// from it is a blind guess: only those are budgeted.
|
||||||
|
// The window can run past the end of the line, so blindness is judged
|
||||||
|
// against the text the lookup actually saw.
|
||||||
|
// Set when a position stopped short of windows an uncapped ladder would
|
||||||
|
// still have tried; the line then escalates to parseText at the end.
|
||||||
|
let blindRetryBudgetExhausted = false;
|
||||||
|
async function resolveTokenAt(position, windowLength) {
|
||||||
|
let attempt = await findTokenAt(position, windowLength);
|
||||||
|
const scannedLength = Math.min(windowLength, text.length - position);
|
||||||
|
let retryLength = Math.min(attempt.matchedLength, scannedLength) - 1;
|
||||||
|
let stepIsBlind = attempt.matchedLength >= scannedLength;
|
||||||
|
let blindRetriesRemaining = MAX_BLIND_SHRINKING_WINDOW_RETRIES;
|
||||||
|
while (!attempt.token && retryLength >= 1) {
|
||||||
|
if (stepIsBlind) {
|
||||||
|
if (blindRetriesRemaining <= 0) {
|
||||||
|
blindRetryBudgetExhausted = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
blindRetriesRemaining -= 1;
|
||||||
|
}
|
||||||
|
const retry = await findTokenAt(position, retryLength);
|
||||||
|
if (retry.token) { return retry; }
|
||||||
|
const guidedLength = retry.matchedLength - 1;
|
||||||
|
stepIsBlind = guidedLength >= retryLength - 1;
|
||||||
|
retryLength = Math.min(retryLength - 1, guidedLength);
|
||||||
|
}
|
||||||
|
return attempt;
|
||||||
|
}
|
||||||
async function findTokenAt(position, windowLength) {
|
async function findTokenAt(position, windowLength) {
|
||||||
const codePoint = text.codePointAt(position);
|
const codePoint = text.codePointAt(position);
|
||||||
const character = String.fromCodePoint(codePoint);
|
const character = String.fromCodePoint(codePoint);
|
||||||
@@ -804,6 +400,17 @@ ${YOMITAN_SCANNING_HELPERS}
|
|||||||
namePos += nameMatch.sourceLength;
|
namePos += nameMatch.sourceLength;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// First reserved name span that a match ending at endPos would leave
|
||||||
|
// half-consumed. Spans the match covers entirely are not returned: those
|
||||||
|
// lose to the longer word instead of splitting it.
|
||||||
|
function findSplitNameToken(startIndex, endPos) {
|
||||||
|
for (let index = startIndex; index < nameTokens.length; index += 1) {
|
||||||
|
const nameToken = nameTokens[index];
|
||||||
|
if (nameToken.startPos >= endPos) { return null; }
|
||||||
|
if (nameToken.endPos > endPos) { return nameToken; }
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
let i = 0;
|
let i = 0;
|
||||||
let nameIndex = 0;
|
let nameIndex = 0;
|
||||||
let unparsedRunStart = null;
|
let unparsedRunStart = null;
|
||||||
@@ -827,27 +434,18 @@ ${YOMITAN_SCANNING_HELPERS}
|
|||||||
i += String.fromCodePoint(codePoint).length;
|
i += String.fromCodePoint(codePoint).length;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// Cap the window at the next reserved name span so a generic match
|
// A reservation only outranks generic matches that would cut into it.
|
||||||
// cannot consume into it.
|
// Look the position up unrestricted first: a generic word that starts
|
||||||
const windowLength = nextNameToken ? Math.min(scanLength, nextNameToken.startPos - i) : scanLength;
|
// earlier and covers the whole name span (写真 over a character named
|
||||||
let attempt = await findTokenAt(i, windowLength);
|
// 真) is the better reading, so the reservation yields rather than
|
||||||
// Yomitan text normalization can consume characters (whitespace,
|
// splitting the word. Only a match that ends inside a name span gets
|
||||||
// punctuation) beyond the matched term, leaving no headword whose
|
// re-run against a window capped at that span.
|
||||||
// source equals the consumed text. Retry with shorter windows so a
|
let attempt = await resolveTokenAt(i, scanLength);
|
||||||
// valid prefix term (e.g. a character name before a paren) still
|
if (attempt.token) {
|
||||||
// tokenizes instead of the position being skipped. The ladder is
|
const splitNameToken = findSplitNameToken(nameIndex, attempt.token.endPos);
|
||||||
// capped: without a cap it degrades to O(scanLength) lookups at a
|
if (splitNameToken) {
|
||||||
// single position.
|
attempt = await resolveTokenAt(i, splitNameToken.startPos - i);
|
||||||
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
|
|
||||||
let retryLookupsRemaining = MAX_SHRINKING_WINDOW_RETRY_LOOKUPS;
|
|
||||||
while (!attempt.token && retryLength >= 1 && retryLookupsRemaining > 0) {
|
|
||||||
retryLookupsRemaining -= 1;
|
|
||||||
const retry = await findTokenAt(i, retryLength);
|
|
||||||
if (retry.token) {
|
|
||||||
attempt = retry;
|
|
||||||
break;
|
|
||||||
}
|
}
|
||||||
retryLength = Math.min(retryLength - 1, retry.matchedLength - 1);
|
|
||||||
}
|
}
|
||||||
if (attempt.token) {
|
if (attempt.token) {
|
||||||
flushUnparsedRun(unparsedRunStart, i);
|
flushUnparsedRun(unparsedRunStart, i);
|
||||||
@@ -860,6 +458,13 @@ ${YOMITAN_SCANNING_HELPERS}
|
|||||||
i += String.fromCodePoint(text.codePointAt(i)).length;
|
i += String.fromCodePoint(text.codePointAt(i)).length;
|
||||||
}
|
}
|
||||||
flushUnparsedRun(unparsedRunStart, text.length);
|
flushUnparsedRun(unparsedRunStart, text.length);
|
||||||
|
if (blindRetryBudgetExhausted) {
|
||||||
|
// A position gave up with shorter windows still worth trying. The walk
|
||||||
|
// is the only tokenizer now, so stopping there would leave a real term
|
||||||
|
// as an unparsed run; report it so the host can spend one parseText on
|
||||||
|
// the line instead of letting the ladder run to O(scanLength) lookups.
|
||||||
|
return { tokens, retryBudgetExhausted: true };
|
||||||
|
}
|
||||||
return tokens;
|
return tokens;
|
||||||
};
|
};
|
||||||
return true;
|
return true;
|
||||||
|
|||||||
@@ -0,0 +1,496 @@
|
|||||||
|
// Helper bundle for the in-page Yomitan scan runtime: kana/furigana handling,
|
||||||
|
// headword preference, and frequency-rank resolution. Injected as text into the
|
||||||
|
// parser window by yomitan-scan-runtime-script.ts, so it is data here, not code
|
||||||
|
// this process runs.
|
||||||
|
import { HAN_CODE_POINT_RANGES } from '../../text/han-code-points';
|
||||||
|
|
||||||
|
export const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||||
|
|
||||||
|
export const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||||
|
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
||||||
|
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
||||||
|
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
||||||
|
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
||||||
|
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
||||||
|
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
||||||
|
// Han ranges come from the shared table so the scan walk and the character
|
||||||
|
// dictionary agree on what a kanji is (supplementary planes included).
|
||||||
|
// Halfwidth katakana counts as Japanese text: a name written that way has
|
||||||
|
// to reach the greedy pre-pass, which has its own handling for it.
|
||||||
|
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0xff66, 0xff9f], ...${JSON.stringify(HAN_CODE_POINT_RANGES)}];
|
||||||
|
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
||||||
|
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
||||||
|
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
||||||
|
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
||||||
|
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
||||||
|
function getSegmentReadingContribution(segment) {
|
||||||
|
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
||||||
|
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
||||||
|
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
||||||
|
return isKanaOnly ? segmentText : "";
|
||||||
|
}
|
||||||
|
function getProlongedHiragana(previousCharacter) {
|
||||||
|
switch (previousCharacter) {
|
||||||
|
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
||||||
|
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
||||||
|
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
||||||
|
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
||||||
|
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
||||||
|
default: return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function getFuriganaKanaSegments(text, reading) {
|
||||||
|
const newSegments = [];
|
||||||
|
let start = 0;
|
||||||
|
let state = (reading[0] === text[0]);
|
||||||
|
for (let i = 1; i < text.length; ++i) {
|
||||||
|
const newState = (reading[i] === text[i]);
|
||||||
|
if (state === newState) { continue; }
|
||||||
|
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
||||||
|
state = newState;
|
||||||
|
start = i;
|
||||||
|
}
|
||||||
|
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
||||||
|
return newSegments;
|
||||||
|
}
|
||||||
|
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
||||||
|
let result = '';
|
||||||
|
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
||||||
|
for (let char of text) {
|
||||||
|
const codePoint = char.codePointAt(0);
|
||||||
|
switch (codePoint) {
|
||||||
|
case KATAKANA_SMALL_KA_CODE_POINT:
|
||||||
|
case KATAKANA_SMALL_KE_CODE_POINT:
|
||||||
|
break;
|
||||||
|
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
||||||
|
if (!keepProlongedSoundMarks && result.length > 0) {
|
||||||
|
const char2 = getProlongedHiragana(result[result.length - 1]);
|
||||||
|
if (char2 !== null) { char = char2; }
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
||||||
|
char = String.fromCodePoint(codePoint + offset);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
result += char;
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
||||||
|
const groupCount = groups.length - groupsStart;
|
||||||
|
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
||||||
|
const group = groups[groupsStart];
|
||||||
|
const {isKana, text} = group;
|
||||||
|
if (isKana) {
|
||||||
|
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
||||||
|
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
||||||
|
if (segments !== null) {
|
||||||
|
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
||||||
|
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
||||||
|
return segments;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
let result = null;
|
||||||
|
for (let i = reading.length; i >= text.length; --i) {
|
||||||
|
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
||||||
|
if (segments !== null) {
|
||||||
|
if (result !== null) { return null; }
|
||||||
|
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
||||||
|
result = segments;
|
||||||
|
}
|
||||||
|
if (groupCount === 1) { break; }
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
function distributeFurigana(term, reading) {
|
||||||
|
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
||||||
|
const groups = [];
|
||||||
|
let groupPre = null;
|
||||||
|
let isKanaPre = null;
|
||||||
|
for (const c of term) {
|
||||||
|
const isKana = isCodePointKana(c.codePointAt(0));
|
||||||
|
if (isKana === isKanaPre) { groupPre.text += c; }
|
||||||
|
else {
|
||||||
|
groupPre = {isKana, text: c, textNormalized: null};
|
||||||
|
groups.push(groupPre);
|
||||||
|
isKanaPre = isKana;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const group of groups) {
|
||||||
|
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
||||||
|
}
|
||||||
|
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
||||||
|
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
||||||
|
}
|
||||||
|
function getStemLength(text1, text2) {
|
||||||
|
const minLength = Math.min(text1.length, text2.length);
|
||||||
|
if (minLength === 0) { return 0; }
|
||||||
|
let i = 0;
|
||||||
|
while (true) {
|
||||||
|
const char1 = text1.codePointAt(i);
|
||||||
|
const char2 = text2.codePointAt(i);
|
||||||
|
if (char1 !== char2) { break; }
|
||||||
|
const charLength = String.fromCodePoint(char1).length;
|
||||||
|
i += charLength;
|
||||||
|
if (i >= minLength) {
|
||||||
|
if (i > minLength) { i -= charLength; }
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return i;
|
||||||
|
}
|
||||||
|
function distributeFuriganaInflected(term, reading, source) {
|
||||||
|
const termNormalized = convertKatakanaToHiragana(term);
|
||||||
|
const readingNormalized = convertKatakanaToHiragana(reading);
|
||||||
|
const sourceNormalized = convertKatakanaToHiragana(source);
|
||||||
|
let mainText = term;
|
||||||
|
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
||||||
|
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
||||||
|
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
||||||
|
mainText = reading;
|
||||||
|
stemLength = readingStemLength;
|
||||||
|
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
||||||
|
}
|
||||||
|
const segments = [];
|
||||||
|
if (stemLength > 0) {
|
||||||
|
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
||||||
|
const segments2 = distributeFurigana(mainText, reading);
|
||||||
|
let consumed = 0;
|
||||||
|
for (const segment of segments2) {
|
||||||
|
const start = consumed;
|
||||||
|
consumed += segment.text.length;
|
||||||
|
if (consumed < stemLength) { segments.push(segment); }
|
||||||
|
else if (consumed === stemLength) { segments.push(segment); break; }
|
||||||
|
else {
|
||||||
|
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (stemLength < source.length) {
|
||||||
|
const remainder = source.substring(stemLength);
|
||||||
|
const last = segments[segments.length - 1];
|
||||||
|
if (last && last.reading.length === 0) { last.text += remainder; }
|
||||||
|
else { segments.push(createFuriganaSegment(remainder, '')); }
|
||||||
|
}
|
||||||
|
return segments;
|
||||||
|
}
|
||||||
|
function parsePositiveFrequencyNumber(value) {
|
||||||
|
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||||
|
return Math.max(1, Math.floor(value));
|
||||||
|
}
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
||||||
|
if (!numericMatch) { return null; }
|
||||||
|
const parsed = Number.parseFloat(numericMatch);
|
||||||
|
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
||||||
|
return Math.max(1, Math.floor(parsed));
|
||||||
|
}
|
||||||
|
if (Array.isArray(value)) {
|
||||||
|
for (const item of value) {
|
||||||
|
const parsed = parsePositiveFrequencyNumber(item);
|
||||||
|
if (parsed !== null) { return parsed; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
function parseDisplayFrequencyNumber(value) {
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
||||||
|
if (!leadingDigits) { return null; }
|
||||||
|
const parsed = Number.parseInt(leadingDigits, 10);
|
||||||
|
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
||||||
|
}
|
||||||
|
return parsePositiveFrequencyNumber(value);
|
||||||
|
}
|
||||||
|
function getFrequencyDictionaryName(frequency) {
|
||||||
|
const candidates = [
|
||||||
|
frequency?.dictionary,
|
||||||
|
frequency?.dictionaryName,
|
||||||
|
frequency?.name,
|
||||||
|
frequency?.title,
|
||||||
|
frequency?.dictionaryTitle,
|
||||||
|
frequency?.dictionaryAlias
|
||||||
|
];
|
||||||
|
for (const candidate of candidates) {
|
||||||
|
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||||
|
return candidate.trim();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||||
|
let best = null;
|
||||||
|
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
||||||
|
for (const frequency of dictionaryEntry?.frequencies || []) {
|
||||||
|
if (!frequency || typeof frequency !== 'object') { continue; }
|
||||||
|
const frequencyHeadwordIndex = frequency.headwordIndex;
|
||||||
|
if (typeof frequencyHeadwordIndex === 'number') {
|
||||||
|
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
||||||
|
} else if (headwordCount > 1) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const dictionary = getFrequencyDictionaryName(frequency);
|
||||||
|
if (!dictionary) { continue; }
|
||||||
|
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
||||||
|
const rank =
|
||||||
|
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
||||||
|
parsePositiveFrequencyNumber(frequency.frequency);
|
||||||
|
if (rank === null) { continue; }
|
||||||
|
const priorityRaw = dictionaryPriorityByName[dictionary];
|
||||||
|
const fallbackPriority =
|
||||||
|
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
||||||
|
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
||||||
|
: Number.MAX_SAFE_INTEGER;
|
||||||
|
const priority =
|
||||||
|
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
||||||
|
? Math.max(0, Math.floor(priorityRaw))
|
||||||
|
: fallbackPriority;
|
||||||
|
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
||||||
|
best = { priority, rank };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best?.rank ?? null;
|
||||||
|
}
|
||||||
|
function hasExactSource(headword, token, requirePrimary) {
|
||||||
|
for (const src of headword.sources || []) {
|
||||||
|
if (src.originalText !== token) { continue; }
|
||||||
|
if (requirePrimary && !src.isPrimary) { continue; }
|
||||||
|
if (src.matchType !== 'exact') { continue; }
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
||||||
|
const matches = [];
|
||||||
|
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||||
|
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||||
|
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||||
|
const headword = headwords[headwordIndex];
|
||||||
|
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
||||||
|
matches.push({ dictionaryEntry, headword, headwordIndex });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return matches;
|
||||||
|
}
|
||||||
|
function sameHeadword(match, preferredMatch) {
|
||||||
|
if (!match || !preferredMatch) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (match.headword?.term !== preferredMatch.headword?.term) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
||||||
|
const preferredReading =
|
||||||
|
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
||||||
|
if (!matchReading || !preferredReading) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return matchReading === preferredReading;
|
||||||
|
}
|
||||||
|
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||||
|
let best = null;
|
||||||
|
for (const match of matches) {
|
||||||
|
const rank = getBestFrequencyRank(
|
||||||
|
match.dictionaryEntry,
|
||||||
|
match.headwordIndex,
|
||||||
|
dictionaryPriorityByName,
|
||||||
|
dictionaryFrequencyModeByName
|
||||||
|
);
|
||||||
|
if (rank === null) { continue; }
|
||||||
|
if (best === null || rank < best) {
|
||||||
|
best = rank;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
function normalizeWordClasses(headword) {
|
||||||
|
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||||
|
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||||
|
return classes.length > 0 ? classes : undefined;
|
||||||
|
}
|
||||||
|
function appendDictionaryNames(target, value) {
|
||||||
|
if (!value || typeof value !== 'object') {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const candidates = [
|
||||||
|
value.dictionary,
|
||||||
|
value.dictionaryName,
|
||||||
|
value.name,
|
||||||
|
value.title,
|
||||||
|
value.dictionaryTitle,
|
||||||
|
value.dictionaryAlias
|
||||||
|
];
|
||||||
|
for (const candidate of candidates) {
|
||||||
|
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||||
|
target.push(candidate.trim());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function getDictionaryEntryNames(entry) {
|
||||||
|
const names = [];
|
||||||
|
appendDictionaryNames(names, entry);
|
||||||
|
for (const definition of entry?.definitions || []) {
|
||||||
|
appendDictionaryNames(names, definition);
|
||||||
|
}
|
||||||
|
for (const frequency of entry?.frequencies || []) {
|
||||||
|
appendDictionaryNames(names, frequency);
|
||||||
|
}
|
||||||
|
for (const pronunciation of entry?.pronunciations || []) {
|
||||||
|
appendDictionaryNames(names, pronunciation);
|
||||||
|
}
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
function isNameDictionaryEntry(entry) {
|
||||||
|
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||||
|
}
|
||||||
|
function parseSubMinerMediaIdFromString(value) {
|
||||||
|
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||||
|
if (imageMatch) {
|
||||||
|
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||||
|
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||||
|
}
|
||||||
|
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||||
|
if (titleMatch) {
|
||||||
|
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||||
|
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
function parseSubMinerMediaIdCandidate(value) {
|
||||||
|
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
||||||
|
const parsed = Number.parseInt(value.trim(), 10);
|
||||||
|
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
function collectSubMinerMediaIds(value, target) {
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
const parsed = parseSubMinerMediaIdFromString(value);
|
||||||
|
if (parsed !== null) { target.add(parsed); }
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!value || typeof value !== 'object') {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (Array.isArray(value)) {
|
||||||
|
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const mediaIdCandidates = [
|
||||||
|
value.subminerMediaId,
|
||||||
|
value.subMinerMediaId,
|
||||||
|
value.characterDictionaryMediaId,
|
||||||
|
value.data?.subminerMediaId,
|
||||||
|
value.data?.subMinerMediaId,
|
||||||
|
value.data?.characterDictionaryMediaId
|
||||||
|
];
|
||||||
|
for (const candidate of mediaIdCandidates) {
|
||||||
|
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
||||||
|
if (parsed !== null) { target.add(parsed); }
|
||||||
|
}
|
||||||
|
for (const child of Object.values(value)) {
|
||||||
|
collectSubMinerMediaIds(child, target);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function getSubMinerMediaIds(entry) {
|
||||||
|
const mediaIds = new Set();
|
||||||
|
collectSubMinerMediaIds(entry, mediaIds);
|
||||||
|
return mediaIds;
|
||||||
|
}
|
||||||
|
function isCurrentMediaNameDictionaryEntry(entry) {
|
||||||
|
if (!isNameDictionaryEntry(entry)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (currentCharacterDictionaryMediaId === null) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const mediaIds = getSubMinerMediaIds(entry);
|
||||||
|
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||||
|
}
|
||||||
|
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||||
|
let best = null;
|
||||||
|
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||||
|
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||||
|
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||||
|
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||||
|
const headword = headwords[headwordIndex];
|
||||||
|
for (const src of headword?.sources || []) {
|
||||||
|
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||||
|
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||||
|
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||||
|
if (best === null || originalText.length > best.sourceLength) {
|
||||||
|
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||||
|
let best = 0;
|
||||||
|
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||||
|
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||||
|
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||||
|
for (const headword of headwords) {
|
||||||
|
for (const src of headword?.sources || []) {
|
||||||
|
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||||
|
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||||
|
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||||
|
if (originalText.length > best) { best = originalText.length; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||||
|
const currentMediaDictionaryEntries =
|
||||||
|
currentCharacterDictionaryMediaId === null
|
||||||
|
? (dictionaryEntries || [])
|
||||||
|
: (dictionaryEntries || []).filter((entry) => {
|
||||||
|
if (!isNameDictionaryEntry(entry)) { return true; }
|
||||||
|
return isCurrentMediaNameDictionaryEntry(entry);
|
||||||
|
});
|
||||||
|
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
||||||
|
let matchedNameDictionary = false;
|
||||||
|
if (includeNameMatchMetadata) {
|
||||||
|
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
||||||
|
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||||
|
for (const match of exactPrimaryMatches) {
|
||||||
|
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
||||||
|
matchedNameDictionary = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (matchedNameDictionary) { break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const preferredMatch = exactPrimaryMatches[0];
|
||||||
|
if (preferredMatch) {
|
||||||
|
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
||||||
|
.filter((match) => sameHeadword(match, preferredMatch));
|
||||||
|
return {
|
||||||
|
term: preferredMatch.headword.term,
|
||||||
|
reading: preferredMatch.headword.reading,
|
||||||
|
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
||||||
|
isNameMatch:
|
||||||
|
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
||||||
|
frequencyRank: getBestFrequencyRankForMatches(
|
||||||
|
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
||||||
|
dictionaryPriorityByName,
|
||||||
|
dictionaryFrequencyModeByName
|
||||||
|
)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
`;
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import test from 'node:test';
|
||||||
|
import { HAN_CODE_POINT_RANGES, HAN_REGEXP_CLASS_BODY, isHanCodePoint } from './han-code-points';
|
||||||
|
|
||||||
|
test('every range boundary is inside the table', () => {
|
||||||
|
for (const [start, end] of HAN_CODE_POINT_RANGES) {
|
||||||
|
for (const codePoint of [start, end]) {
|
||||||
|
assert.ok(isHanCodePoint(codePoint), `expected U+${codePoint.toString(16)} to be Han`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extension J (Unicode 17) and the Compatibility blocks are the ones a
|
||||||
|
// BMP-only table used to miss.
|
||||||
|
assert.ok(isHanCodePoint(0x323b0));
|
||||||
|
assert.ok(isHanCodePoint(0x33479));
|
||||||
|
assert.ok(isHanCodePoint(0xf900));
|
||||||
|
assert.ok(isHanCodePoint(0x2f800));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('no unified ideograph the runtime knows about falls outside the table', () => {
|
||||||
|
// One direction only: a runtime with older Unicode data simply checks fewer
|
||||||
|
// code points, where asserting the reverse would fail on Extension J.
|
||||||
|
const unifiedIdeograph = /\p{Unified_Ideograph}/u;
|
||||||
|
|
||||||
|
for (let codePoint = 0x3000; codePoint <= 0x40000; codePoint += 1) {
|
||||||
|
if (unifiedIdeograph.test(String.fromCodePoint(codePoint))) {
|
||||||
|
assert.ok(
|
||||||
|
isHanCodePoint(codePoint),
|
||||||
|
`expected unified ideograph U+${codePoint.toString(16)} to be in the table`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('code points just outside the table are rejected', () => {
|
||||||
|
for (const codePoint of [0x33ff, 0x4dc0, 0xa000, 0x1f000, 0x3347a]) {
|
||||||
|
assert.equal(
|
||||||
|
isHanCodePoint(codePoint),
|
||||||
|
false,
|
||||||
|
`expected U+${codePoint.toString(16)} not to be Han`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the regexp class body matches the same code points as the predicate', () => {
|
||||||
|
const classRegExp = new RegExp(`^[${HAN_REGEXP_CLASS_BODY}]$`, 'u');
|
||||||
|
|
||||||
|
for (const codePoint of [0x3400, 0x4e00, 0x9fff, 0xf900, 0x20000, 0x323b0, 0x33479]) {
|
||||||
|
assert.match(String.fromCodePoint(codePoint), classRegExp);
|
||||||
|
}
|
||||||
|
for (const codePoint of [0x3040, 0x30ff, 0x33fa, 0x3347a]) {
|
||||||
|
assert.doesNotMatch(String.fromCodePoint(codePoint), classRegExp);
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
// Single source of truth for "this code point is a Han character", shared by
|
||||||
|
// the main-process character dictionary and the in-page Yomitan scan runtime.
|
||||||
|
// The two used to carry separate range lists, and they drifted: a name written
|
||||||
|
// with a supplementary-plane kanji could enter the generated dictionary while
|
||||||
|
// the scanner's greedy name pre-pass refused to probe the position.
|
||||||
|
//
|
||||||
|
// Ranges rather than \p{Script=Han}: the scan walk tests one code point per
|
||||||
|
// character of every subtitle line, where an integer compare beats building a
|
||||||
|
// string for a regex, and the script is injected as text into a page where a
|
||||||
|
// shared helper cannot be imported.
|
||||||
|
export const HAN_CODE_POINT_RANGES: ReadonlyArray<readonly [number, number]> = [
|
||||||
|
[0x3400, 0x4dbf], // Extension A
|
||||||
|
[0x4e00, 0x9fff], // CJK Unified Ideographs
|
||||||
|
[0xf900, 0xfaff], // Compatibility Ideographs
|
||||||
|
[0x20000, 0x2a6df], // Extension B
|
||||||
|
[0x2a700, 0x2ebef], // Extensions C-F
|
||||||
|
[0x2ebf0, 0x2ee5f], // Extension I
|
||||||
|
[0x2f800, 0x2fa1f], // Compatibility Ideographs Supplement
|
||||||
|
[0x30000, 0x3134f], // Extension G
|
||||||
|
[0x31350, 0x323af], // Extension H
|
||||||
|
[0x323b0, 0x33479], // Extension J (Unicode 17)
|
||||||
|
];
|
||||||
|
|
||||||
|
export function isHanCodePoint(codePoint: number): boolean {
|
||||||
|
return HAN_CODE_POINT_RANGES.some(([start, end]) => codePoint >= start && codePoint <= end);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The same ranges as a regular expression character class body (needs the `u` flag). */
|
||||||
|
export const HAN_REGEXP_CLASS_BODY = HAN_CODE_POINT_RANGES.map(
|
||||||
|
([start, end]) => `\\u{${start.toString(16)}}-\\u{${end.toString(16)}}`,
|
||||||
|
).join('');
|
||||||
+20
-6
@@ -1834,9 +1834,10 @@ function emitSubtitlePayload(payload: SubtitleData, options?: { resumePrefetch?:
|
|||||||
}
|
}
|
||||||
annotationSubtitleWsService.broadcast(timedPayload, frequencyOptions);
|
annotationSubtitleWsService.broadcast(timedPayload, frequencyOptions);
|
||||||
autoplayReadyGate.maybeSignalPluginAutoplayReady(timedPayload, { forceWhilePaused: true });
|
autoplayReadyGate.maybeSignalPluginAutoplayReady(timedPayload, { forceWhilePaused: true });
|
||||||
// resumePrefetch: false marks a provisional pre-tokenization emit; prefetch
|
// resumePrefetch: false marks an emit that is not the end of the work for
|
||||||
// stays paused until the tokenized payload for the line lands so it does not
|
// this line; prefetch stays paused until the subtitle processing controller
|
||||||
// compete with the on-screen line for the single Yomitan parser window.
|
// settles so it does not compete with the on-screen line for the single
|
||||||
|
// Yomitan parser window.
|
||||||
if (options?.resumePrefetch !== false) {
|
if (options?.resumePrefetch !== false) {
|
||||||
subtitlePrefetchService?.resume();
|
subtitlePrefetchService?.resume();
|
||||||
}
|
}
|
||||||
@@ -1895,7 +1896,17 @@ const buildSubtitleProcessingControllerMainDepsHandler =
|
|||||||
createBuildSubtitleProcessingControllerMainDepsHandler({
|
createBuildSubtitleProcessingControllerMainDepsHandler({
|
||||||
tokenizeSubtitle: async (text: string) =>
|
tokenizeSubtitle: async (text: string) =>
|
||||||
tokenizeSubtitleDeferred ? await tokenizeSubtitleDeferred(text) : { text, tokens: null },
|
tokenizeSubtitleDeferred ? await tokenizeSubtitleDeferred(text) : { text, tokens: null },
|
||||||
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
// Controller emits never release the prefetch pause: the first emit for an
|
||||||
|
// uncached line is the provisional plain payload, sent before tokenization
|
||||||
|
// starts, so resuming on it would put prefetch back in contention with the
|
||||||
|
// on-screen line for the single parser window.
|
||||||
|
emitSubtitle: (payload) => emitSubtitlePayload(payload, { resumePrefetch: false }),
|
||||||
|
// The pause is released once the controller has no work left, which covers
|
||||||
|
// the runs that end without an emit (suppressed duplicate, failed
|
||||||
|
// tokenization) as well as the ones that deliver a payload.
|
||||||
|
onProcessingSettled: () => {
|
||||||
|
subtitlePrefetchService?.resume();
|
||||||
|
},
|
||||||
logDebug: (message) => {
|
logDebug: (message) => {
|
||||||
logger.debug(`[subtitle-processing] ${message}`);
|
logger.debug(`[subtitle-processing] ${message}`);
|
||||||
},
|
},
|
||||||
@@ -3988,7 +3999,10 @@ const refreshCurrentSubtitleAfterKnownWordUpdate = (): void => {
|
|||||||
}
|
}
|
||||||
subtitleProcessingController.invalidateTokenizationCache();
|
subtitleProcessingController.invalidateTokenizationCache();
|
||||||
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
||||||
subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);
|
if (!subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)) {
|
||||||
|
// Idle controller: no settle is coming to release the pause above.
|
||||||
|
subtitlePrefetchService?.resume();
|
||||||
|
}
|
||||||
};
|
};
|
||||||
let hasAttemptedImmersionTrackerStartup = false;
|
let hasAttemptedImmersionTrackerStartup = false;
|
||||||
const ensureImmersionTrackerStarted = (): void => {
|
const ensureImmersionTrackerStarted = (): void => {
|
||||||
@@ -4393,7 +4407,7 @@ const {
|
|||||||
// tokenization work on every line. Real seeks restart via onTimePosUpdate.
|
// tokenization work on every line. Real seeks restart via onTimePosUpdate.
|
||||||
subtitlePrefetchService?.pause();
|
subtitlePrefetchService?.pause();
|
||||||
if (!subtitleProcessingController.onSubtitleChange(text)) {
|
if (!subtitleProcessingController.onSubtitleChange(text)) {
|
||||||
// Repeat of the current text: nothing will be tokenized, so no emit is
|
// Repeat of the current text: the controller is idle, so no settle is
|
||||||
// coming to release the pause. Resume now instead of idling prefetch
|
// coming to release the pause. Resume now instead of idling prefetch
|
||||||
// for the rest of the cue.
|
// for the rest of the cue.
|
||||||
subtitlePrefetchService?.resume();
|
subtitlePrefetchService?.resume();
|
||||||
|
|||||||
@@ -33,6 +33,20 @@ function collectSnapshotNameForms(snapshot: CharacterDictionarySnapshot): string
|
|||||||
return [...forms];
|
return [...forms];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The signature grows with the size of the dictionary library, and it rides
|
||||||
|
// along in every per-line scan call, so it is folded into a fixed-width digest
|
||||||
|
// first. Collisions only matter against the immediately previous signature (the
|
||||||
|
// runtime compares keys for equality), and FNV-1a over the file list is far
|
||||||
|
// beyond what that needs.
|
||||||
|
function digestSnapshotDirectorySignature(signature: string): string {
|
||||||
|
let hash = 0x811c9dc5;
|
||||||
|
for (let index = 0; index < signature.length; index += 1) {
|
||||||
|
hash ^= signature.charCodeAt(index);
|
||||||
|
hash = Math.imul(hash, 0x01000193);
|
||||||
|
}
|
||||||
|
return (hash >>> 0).toString(36);
|
||||||
|
}
|
||||||
|
|
||||||
function getSnapshotDirectorySignature(outputDir: string): string {
|
function getSnapshotDirectorySignature(outputDir: string): string {
|
||||||
let entries: fs.Dirent[] = [];
|
let entries: fs.Dirent[] = [];
|
||||||
try {
|
try {
|
||||||
@@ -132,7 +146,10 @@ export function createCharacterNameCandidateLookup(deps: {
|
|||||||
if (!forms || forms.length === 0) {
|
if (!forms || forms.length === 0) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return { key: `${signature ?? ''}:${normalizedMediaId}`, forms };
|
return {
|
||||||
|
key: `${digestSnapshotDirectorySignature(signature ?? '')}:${normalizedMediaId}`,
|
||||||
|
forms,
|
||||||
|
};
|
||||||
},
|
},
|
||||||
invalidate(): void {
|
invalidate(): void {
|
||||||
signature = null;
|
signature = null;
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { isHanCodePoint } from '../../core/text/han-code-points';
|
||||||
import { HONORIFIC_SUFFIXES } from './constants';
|
import { HONORIFIC_SUFFIXES } from './constants';
|
||||||
import type { JapaneseNameParts, NameReadings, ResolvedNameSplits } from './types';
|
import type { JapaneseNameParts, NameReadings, ResolvedNameSplits } from './types';
|
||||||
|
|
||||||
@@ -26,10 +27,12 @@ export function buildReading(term: string): string {
|
|||||||
return katakanaToHiragana(compact);
|
return katakanaToHiragana(compact);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Code points, not code units: a supplementary-plane kanji (𠮷, U+20BB7) is a
|
||||||
|
// surrogate pair, and reading only the high surrogate would classify a real
|
||||||
|
// single-character name as non-kanji and drop it.
|
||||||
export function containsKanji(value: string): boolean {
|
export function containsKanji(value: string): boolean {
|
||||||
for (const char of value) {
|
for (const char of value) {
|
||||||
const code = char.charCodeAt(0);
|
if (isHanCodePoint(char.codePointAt(0) ?? 0)) {
|
||||||
if ((code >= 0x4e00 && code <= 0x9fff) || (code >= 0x3400 && code <= 0x4dbf)) {
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -55,6 +55,26 @@ test('buildNameTerms drops the disambiguator letter of a mob character name', ()
|
|||||||
assert.ok(terms.includes('ジョシア'));
|
assert.ok(terms.includes('ジョシア'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('buildNameTerms keeps a character whose whole name is one kana', () => {
|
||||||
|
const terms = buildNameTerms(
|
||||||
|
characterRecord({
|
||||||
|
firstNameHint: '',
|
||||||
|
lastNameHint: '',
|
||||||
|
fullName: 'A',
|
||||||
|
nativeName: 'あ',
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
// The mob-disambiguator filter targets letters split off a longer name; an
|
||||||
|
// explicit one-character name is the character's actual name.
|
||||||
|
assert.ok(terms.includes('あ'));
|
||||||
|
assert.ok(terms.includes('あさん'));
|
||||||
|
// The romanized "A" is still a label, so it contributes neither itself nor
|
||||||
|
// its single-kana alias.
|
||||||
|
assert.ok(!terms.includes('A'));
|
||||||
|
assert.ok(!terms.includes('ア'));
|
||||||
|
});
|
||||||
|
|
||||||
test('buildNameTerms keeps a single-kanji name part', () => {
|
test('buildNameTerms keeps a single-kanji name part', () => {
|
||||||
// The name is an alias, not the native name, so the parts come from the
|
// The name is an alias, not the native name, so the parts come from the
|
||||||
// space split rather than from the native-name split.
|
// space split rather than from the native-name split.
|
||||||
@@ -71,3 +91,20 @@ test('buildNameTerms keeps a single-kanji name part', () => {
|
|||||||
assert.ok(terms.includes('山田'));
|
assert.ok(terms.includes('山田'));
|
||||||
assert.ok(terms.includes('空'));
|
assert.ok(terms.includes('空'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('buildNameTerms keeps a single supplementary-plane kanji name part', () => {
|
||||||
|
// 𠮷 (U+20BB7) is a surrogate pair: a code-unit kanji check reads only the
|
||||||
|
// high surrogate and drops the part as if it were a mob disambiguator.
|
||||||
|
const terms = buildNameTerms(
|
||||||
|
characterRecord({
|
||||||
|
firstNameHint: 'Tsukasa',
|
||||||
|
lastNameHint: 'Yoshi',
|
||||||
|
fullName: 'Tsukasa Yoshi',
|
||||||
|
nativeName: '',
|
||||||
|
alternativeNames: ['𠮷 司'],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.ok(terms.includes('𠮷'));
|
||||||
|
assert.ok(terms.includes('司'));
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { HAN_REGEXP_CLASS_BODY } from '../../core/text/han-code-points';
|
||||||
import { HONORIFIC_SUFFIXES } from './constants';
|
import { HONORIFIC_SUFFIXES } from './constants';
|
||||||
import {
|
import {
|
||||||
addRomanizedKanaAliases,
|
addRomanizedKanaAliases,
|
||||||
@@ -42,20 +43,34 @@ export function expandRawNameVariants(rawName: string): string[] {
|
|||||||
return [...variants];
|
return [...variants];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Kana, halfwidth included: one of these can stand alone as a name, where a
|
||||||
|
// latin letter or a digit cannot.
|
||||||
|
const SINGLE_KANA_CHARACTER = /^[\u3040-\u30ff\u31f0-\u31ff\uff66-\uff9f]$/u;
|
||||||
|
|
||||||
// AniList disambiguates unnamed mob characters with a trailing letter (女子A /
|
// AniList disambiguates unnamed mob characters with a trailing letter (女子A /
|
||||||
// "Joshi A"), and a single letter romanizes into a single-kana alias (A → ア)
|
// "Joshi A"), and a lone letter romanizes into a single-kana alias (A → ア)
|
||||||
// that collides with interjections (あ〜 matching ア). A one-character form is
|
// that collides with interjections (あ〜 matching ア). That letter is a label,
|
||||||
// only a real lookup target when it is kanji, so every other single-character
|
// not a name, so it is dropped where a name splits into it and before it can
|
||||||
// form is dropped before it can become a term.
|
// become a kana alias. A name that is genuinely one character, a character
|
||||||
function isUsableNameTerm(name: string): boolean {
|
// actually called あ or a single kanji, is a real lookup target and is kept.
|
||||||
return [...name].length > 1 || containsKanji(name);
|
function isNameDisambiguatorLetter(name: string): boolean {
|
||||||
|
return [...name].length === 1 && !containsKanji(name) && !SINGLE_KANA_CHARACTER.test(name);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function isUsableNameTerm(name: string): boolean {
|
||||||
|
return !isNameDisambiguatorLetter(name);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kana, Han (shared ranges), and the marks that only ever appear inside a
|
||||||
|
// Japanese name: iteration marks and the small ka/ke used in place names.
|
||||||
|
const JAPANESE_NAME_CHARACTERS = new RegExp(
|
||||||
|
`^[\\u3040-\\u30ff${HAN_REGEXP_CLASS_BODY}\u3005\u3006\u30f5\u30f6\u30fc]+$`,
|
||||||
|
'u',
|
||||||
|
);
|
||||||
|
|
||||||
export function isJapaneseNameSplitCandidate(name: string): boolean {
|
export function isJapaneseNameSplitCandidate(name: string): boolean {
|
||||||
const compact = name.replace(/[\s\u3000・･·•]/g, '');
|
const compact = name.replace(/[\s\u3000・･·•]/g, '');
|
||||||
return (
|
return containsKanji(compact) && JAPANESE_NAME_CHARACTERS.test(compact);
|
||||||
containsKanji(compact) && /^[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff々〆ヵヶー]+$/.test(compact)
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function addJapaneseNameParts(
|
function addJapaneseNameParts(
|
||||||
@@ -131,7 +146,10 @@ export function buildNameTerms(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const alias of addRomanizedKanaAliases(romanizedBase)) {
|
// Romanized forms that are a bare letter would become a single-kana alias.
|
||||||
|
for (const alias of addRomanizedKanaAliases(
|
||||||
|
[...romanizedBase].filter((entry) => !isNameDisambiguatorLetter(entry)),
|
||||||
|
)) {
|
||||||
base.add(alias);
|
base.add(alias);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -150,7 +168,9 @@ export function buildNameTerms(
|
|||||||
|
|
||||||
const withHonorifics = new Set<string>();
|
const withHonorifics = new Set<string>();
|
||||||
for (const entry of base) {
|
for (const entry of base) {
|
||||||
if (!isUsableNameTerm(entry)) continue;
|
// Only labels split off a longer name are filtered (see above); an explicit
|
||||||
|
// one-character name reaches this point intact.
|
||||||
|
if (isNameDisambiguatorLetter(entry)) continue;
|
||||||
withHonorifics.add(entry);
|
withHonorifics.add(entry);
|
||||||
for (const suffix of HONORIFIC_SUFFIXES) {
|
for (const suffix of HONORIFIC_SUFFIXES) {
|
||||||
withHonorifics.add(`${entry}${suffix.term}`);
|
withHonorifics.add(`${entry}${suffix.term}`);
|
||||||
|
|||||||
@@ -493,16 +493,35 @@ test('known-word updates invalidate prefetched tokenizations before refreshing c
|
|||||||
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
||||||
assert.match(
|
assert.match(
|
||||||
actionBlock,
|
actionBlock,
|
||||||
/subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\);/,
|
/if \(!subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\)\) \{[\s\S]*?subtitlePrefetchService\?\.resume\(\);/,
|
||||||
);
|
);
|
||||||
assert.ok(
|
assert.ok(
|
||||||
actionBlock.indexOf('subtitleProcessingController.invalidateTokenizationCache();') <
|
actionBlock.indexOf('subtitleProcessingController.invalidateTokenizationCache();') <
|
||||||
actionBlock.indexOf(
|
actionBlock.indexOf(
|
||||||
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);',
|
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)',
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('subtitle processing controller resumes prefetch on settle, not on its emits', () => {
|
||||||
|
const source = readMainSource();
|
||||||
|
const depsBlock = source.match(
|
||||||
|
/createBuildSubtitleProcessingControllerMainDepsHandler\(\{(?<body>[\s\S]*?)\n \}\);/,
|
||||||
|
)?.groups?.body;
|
||||||
|
|
||||||
|
assert.ok(depsBlock);
|
||||||
|
// A controller emit can be the provisional plain payload sent before the
|
||||||
|
// scan runs, so it must not release the prefetch pause.
|
||||||
|
assert.match(
|
||||||
|
depsBlock,
|
||||||
|
/emitSubtitle: \(payload\) => emitSubtitlePayload\(payload, \{ resumePrefetch: false \}\),/,
|
||||||
|
);
|
||||||
|
assert.match(
|
||||||
|
depsBlock,
|
||||||
|
/onProcessingSettled: \(\) => \{\s+subtitlePrefetchService\?\.resume\(\);/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
test('manual visible overlay changes notify mpv plugin visibility state', () => {
|
test('manual visible overlay changes notify mpv plugin visibility state', () => {
|
||||||
const source = readMainSource();
|
const source = readMainSource();
|
||||||
const setBlock = source.match(
|
const setBlock = source.match(
|
||||||
|
|||||||
@@ -215,6 +215,7 @@ function createPrimingRuntimeWithRealController(options: {
|
|||||||
text: string;
|
text: string;
|
||||||
calls: string[];
|
calls: string[];
|
||||||
onTokenize: () => void;
|
onTokenize: () => void;
|
||||||
|
tokenize?: (text: string) => SubtitleData | null | Promise<SubtitleData | null>;
|
||||||
cacheLimit?: number;
|
cacheLimit?: number;
|
||||||
}) {
|
}) {
|
||||||
const { text, calls } = options;
|
const { text, calls } = options;
|
||||||
@@ -222,15 +223,42 @@ function createPrimingRuntimeWithRealController(options: {
|
|||||||
let currentSubtitleData: SubtitleData | null = null;
|
let currentSubtitleData: SubtitleData | null = null;
|
||||||
const mediaPath = '/media/video.mkv';
|
const mediaPath = '/media/video.mkv';
|
||||||
|
|
||||||
|
const prefetchService = {
|
||||||
|
pause: () => calls.push('prefetch:pause'),
|
||||||
|
resume: () => calls.push('prefetch:resume'),
|
||||||
|
};
|
||||||
|
// Mirrors main.ts emitSubtitlePayload: an emit resumes prefetching unless it
|
||||||
|
// is explicitly marked as not the end of the work for the line, and every
|
||||||
|
// controller emit is so marked.
|
||||||
|
const emitSubtitlePayload = (
|
||||||
|
payload: SubtitleData,
|
||||||
|
emitOptions?: { resumePrefetch?: boolean },
|
||||||
|
): void => {
|
||||||
|
currentSubtitleData = payload;
|
||||||
|
calls.push(
|
||||||
|
emitOptions?.resumePrefetch === false
|
||||||
|
? `emit-raw:${payload.text}`
|
||||||
|
: `emit-direct:${payload.text}`,
|
||||||
|
);
|
||||||
|
if (emitOptions?.resumePrefetch !== false) {
|
||||||
|
prefetchService.resume();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const subtitleProcessingController = createSubtitleProcessingController({
|
const subtitleProcessingController = createSubtitleProcessingController({
|
||||||
tokenizeSubtitle: async (subtitleText) => {
|
tokenizeSubtitle: async (subtitleText) => {
|
||||||
options.onTokenize();
|
options.onTokenize();
|
||||||
return { text: subtitleText, tokens: [] };
|
return options.tokenize ? options.tokenize(subtitleText) : { text: subtitleText, tokens: [] };
|
||||||
},
|
},
|
||||||
|
// main.ts routes controller emits through emitSubtitlePayload with
|
||||||
|
// resumePrefetch: false, so they never release the pause on their own.
|
||||||
emitSubtitle: (payload) => {
|
emitSubtitle: (payload) => {
|
||||||
currentSubtitleData = payload;
|
currentSubtitleData = payload;
|
||||||
calls.push(`emit:${payload.text}:tokens=${payload.tokens === null ? 'none' : 'yes'}`);
|
calls.push(`emit:${payload.text}:tokens=${payload.tokens === null ? 'none' : 'yes'}`);
|
||||||
},
|
},
|
||||||
|
onProcessingSettled: () => {
|
||||||
|
prefetchService.resume();
|
||||||
|
},
|
||||||
...(options.cacheLimit === undefined ? {} : { cacheLimit: options.cacheLimit }),
|
...(options.cacheLimit === undefined ? {} : { cacheLimit: options.cacheLimit }),
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -249,17 +277,8 @@ function createPrimingRuntimeWithRealController(options: {
|
|||||||
getActiveParsedSubtitleCues: () => [],
|
getActiveParsedSubtitleCues: () => [],
|
||||||
setActiveParsedSubtitleMediaPath: () => {},
|
setActiveParsedSubtitleMediaPath: () => {},
|
||||||
subtitleProcessingController,
|
subtitleProcessingController,
|
||||||
emitSubtitlePayload: (payload, emitOptions) => {
|
emitSubtitlePayload,
|
||||||
if (emitOptions?.resumePrefetch === false) {
|
getSubtitlePrefetchService: () => prefetchService,
|
||||||
calls.push(`emit-raw:${payload.text}`);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
calls.push(`emit-direct:${payload.text}`);
|
|
||||||
},
|
|
||||||
getSubtitlePrefetchService: () => ({
|
|
||||||
pause: () => calls.push('prefetch:pause'),
|
|
||||||
resume: () => calls.push('prefetch:resume'),
|
|
||||||
}),
|
|
||||||
getLastObservedTimePos: () => 12,
|
getLastObservedTimePos: () => 12,
|
||||||
getVisibleOverlayVisible: () => true,
|
getVisibleOverlayVisible: () => true,
|
||||||
emitSecondarySubtitle: () => {},
|
emitSecondarySubtitle: () => {},
|
||||||
@@ -336,3 +355,68 @@ test('primeCurrentSubtitleForAutoplay releases the prefetch pause when nothing i
|
|||||||
|
|
||||||
assert.deepEqual(calls, ['prefetch:pause', `emit-raw:${text}`, 'prefetch:resume']);
|
assert.deepEqual(calls, ['prefetch:pause', `emit-raw:${text}`, 'prefetch:resume']);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('primeCurrentSubtitleForAutoplay releases the prefetch pause when tokenization emits nothing', async () => {
|
||||||
|
const calls: string[] = [];
|
||||||
|
const text = '起動字幕';
|
||||||
|
const { runtime, subtitleProcessingController, mediaPath } =
|
||||||
|
createPrimingRuntimeWithRealController({
|
||||||
|
text,
|
||||||
|
calls,
|
||||||
|
onTokenize: () => {},
|
||||||
|
// Transient tokenizer failure: the controller falls back to plain text it
|
||||||
|
// has already shown, so it suppresses the emit entirely.
|
||||||
|
tokenize: () => null,
|
||||||
|
});
|
||||||
|
|
||||||
|
subtitleProcessingController.onSubtitleChange(text);
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||||
|
subtitleProcessingController.invalidateTokenizationCache();
|
||||||
|
calls.length = 0;
|
||||||
|
|
||||||
|
await runtime.primeCurrentSubtitleForAutoplay(mediaPath);
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||||
|
|
||||||
|
assert.ok(
|
||||||
|
!calls.some((call) => call.startsWith('emit:')),
|
||||||
|
`expected no controller emit, saw ${JSON.stringify(calls)}`,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
calls.filter((call) => call === 'prefetch:resume').length,
|
||||||
|
1,
|
||||||
|
`expected the prefetch pause to be released, saw ${JSON.stringify(calls)}`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('prefetch stays paused until tokenization of an uncached line completes', async () => {
|
||||||
|
const calls: string[] = [];
|
||||||
|
const text = '起動字幕';
|
||||||
|
let finishTokenization = (): void => {};
|
||||||
|
const tokenizationGate = new Promise<void>((resolve) => {
|
||||||
|
finishTokenization = resolve;
|
||||||
|
});
|
||||||
|
const { subtitleProcessingController } = createPrimingRuntimeWithRealController({
|
||||||
|
text,
|
||||||
|
calls,
|
||||||
|
onTokenize: () => {},
|
||||||
|
tokenize: async (subtitleText) => {
|
||||||
|
await tokenizationGate;
|
||||||
|
return { text: subtitleText, tokens: [] };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
subtitleProcessingController.onSubtitleChange(text);
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||||
|
|
||||||
|
// The provisional plain emit must not release the pause: the expensive scan
|
||||||
|
// is still ahead of it and would compete with prefetching for the parser.
|
||||||
|
assert.deepEqual(calls, [`emit:${text}:tokens=none`]);
|
||||||
|
|
||||||
|
finishTokenization();
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||||
|
assert.deepEqual(calls, [
|
||||||
|
`emit:${text}:tokens=none`,
|
||||||
|
`emit:${text}:tokens=yes`,
|
||||||
|
'prefetch:resume',
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ export interface AutoplaySubtitlePrimingRuntimeDeps {
|
|||||||
setActiveParsedSubtitleMediaPath: (mediaPath: string | null) => void;
|
setActiveParsedSubtitleMediaPath: (mediaPath: string | null) => void;
|
||||||
subtitleProcessingController: {
|
subtitleProcessingController: {
|
||||||
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
||||||
// Both report whether an emit is expected; see pausePrefetchUntilEmit.
|
// Both report whether processing is pending; see pausePrefetchUntilProcessed.
|
||||||
onSubtitleChange: (text: string) => boolean;
|
onSubtitleChange: (text: string) => boolean;
|
||||||
refreshCurrentSubtitle: (text: string) => boolean;
|
refreshCurrentSubtitle: (text: string) => boolean;
|
||||||
};
|
};
|
||||||
@@ -65,11 +65,12 @@ export function setMpvCurrentSecondarySubText(
|
|||||||
export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimingRuntimeDeps) {
|
export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimingRuntimeDeps) {
|
||||||
const { subtitleProcessingController, emitSubtitlePayload } = deps;
|
const { subtitleProcessingController, emitSubtitlePayload } = deps;
|
||||||
|
|
||||||
// Prefetching is paused so the on-screen line gets the parser to itself, and
|
// Prefetching is paused so the on-screen line gets the parser to itself; the
|
||||||
// the resume rides on the tokenized emit. When the controller reports that no
|
// resume rides on the controller settling (see onProcessingSettled), not on
|
||||||
// emit is coming (repeat text with nothing scheduled), release it here or
|
// an emit, which a suppressed duplicate or a failed tokenization never sends.
|
||||||
// prefetching idles until some later line happens to complete.
|
// When the controller reports it has nothing scheduled, no settle is coming
|
||||||
function pausePrefetchUntilEmit(scheduleTokenization: () => boolean): void {
|
// either, so release the pause here or prefetching idles indefinitely.
|
||||||
|
function pausePrefetchUntilProcessed(scheduleTokenization: () => boolean): void {
|
||||||
const prefetch = deps.getSubtitlePrefetchService();
|
const prefetch = deps.getSubtitlePrefetchService();
|
||||||
prefetch?.pause();
|
prefetch?.pause();
|
||||||
if (!scheduleTokenization()) {
|
if (!scheduleTokenization()) {
|
||||||
@@ -122,8 +123,8 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Provisional raw emit: keep prefetch paused until the tokenized payload
|
// Provisional raw emit: keep prefetch paused until the processing
|
||||||
// for this line is delivered by the processing controller.
|
// controller is done with this line.
|
||||||
emitSubtitlePayload({ text, tokens: null }, { resumePrefetch: false });
|
emitSubtitlePayload({ text, tokens: null }, { resumePrefetch: false });
|
||||||
// refreshCurrentSubtitle, not onSubtitleChange: the cache miss above can be
|
// refreshCurrentSubtitle, not onSubtitleChange: the cache miss above can be
|
||||||
// an invalidation (mining a card) on text the controller still holds, and
|
// an invalidation (mining a card) on text the controller still holds, and
|
||||||
@@ -131,7 +132,7 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
|||||||
// leave this line permanently unannotated. refreshCurrentSubtitle also
|
// leave this line permanently unannotated. refreshCurrentSubtitle also
|
||||||
// re-tokenizes for a new cache generation.
|
// re-tokenizes for a new cache generation.
|
||||||
if (!subtitleProcessingController.refreshCurrentSubtitle(text)) {
|
if (!subtitleProcessingController.refreshCurrentSubtitle(text)) {
|
||||||
// Nothing scheduled, so no emit is coming to release the pause.
|
// Nothing scheduled, so no settle is coming to release the pause.
|
||||||
deps.getSubtitlePrefetchService()?.resume();
|
deps.getSubtitlePrefetchService()?.resume();
|
||||||
}
|
}
|
||||||
return true;
|
return true;
|
||||||
@@ -177,10 +178,12 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
|||||||
getCurrentSubtitleData: () => deps.getCurrentSubtitleData(),
|
getCurrentSubtitleData: () => deps.getCurrentSubtitleData(),
|
||||||
consumeCachedSubtitle: (text) => subtitleProcessingController.consumeCachedSubtitle(text),
|
consumeCachedSubtitle: (text) => subtitleProcessingController.consumeCachedSubtitle(text),
|
||||||
onSubtitleChange: (text) => {
|
onSubtitleChange: (text) => {
|
||||||
pausePrefetchUntilEmit(() => subtitleProcessingController.onSubtitleChange(text));
|
pausePrefetchUntilProcessed(() => subtitleProcessingController.onSubtitleChange(text));
|
||||||
},
|
},
|
||||||
refreshCurrentSubtitle: (text) => {
|
refreshCurrentSubtitle: (text) => {
|
||||||
pausePrefetchUntilEmit(() => subtitleProcessingController.refreshCurrentSubtitle(text));
|
pausePrefetchUntilProcessed(() =>
|
||||||
|
subtitleProcessingController.refreshCurrentSubtitle(text),
|
||||||
|
);
|
||||||
},
|
},
|
||||||
deferUncachedRefresh: true,
|
deferUncachedRefresh: true,
|
||||||
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
||||||
@@ -224,7 +227,7 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
|||||||
if (!text.trim()) {
|
if (!text.trim()) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
pausePrefetchUntilEmit(() => subtitleProcessingController.refreshCurrentSubtitle(text));
|
pausePrefetchUntilProcessed(() => subtitleProcessingController.refreshCurrentSubtitle(text));
|
||||||
}, VISIBLE_OVERLAY_SUBTITLE_REFRESH_AFTER_FIRST_PAINT_DELAY_MS);
|
}, VISIBLE_OVERLAY_SUBTITLE_REFRESH_AFTER_FIRST_PAINT_DELAY_MS);
|
||||||
visibleOverlaySubtitleRefreshAfterFirstPaintTimer.unref?.();
|
visibleOverlaySubtitleRefreshAfterFirstPaintTimer.unref?.();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ export function createBuildSubtitleProcessingControllerMainDepsHandler(
|
|||||||
return (): SubtitleProcessingControllerDeps => ({
|
return (): SubtitleProcessingControllerDeps => ({
|
||||||
tokenizeSubtitle: (text: string) => deps.tokenizeSubtitle(text),
|
tokenizeSubtitle: (text: string) => deps.tokenizeSubtitle(text),
|
||||||
emitSubtitle: (payload) => deps.emitSubtitle(payload),
|
emitSubtitle: (payload) => deps.emitSubtitle(payload),
|
||||||
|
onProcessingSettled: () => deps.onProcessingSettled?.(),
|
||||||
logDebug: deps.logDebug,
|
logDebug: deps.logDebug,
|
||||||
now: deps.now,
|
now: deps.now,
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user