mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-05 07:21:34 -07:00
Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
afa66ee508
|
|||
|
3c24597724
|
|||
|
d878d8bf4f
|
|||
|
8683961967
|
|||
|
c9baaeea17
|
|||
|
2003efa235
|
|||
|
f43674cc39
|
|||
|
b0a2ce6e8a
|
|||
|
030c94934e
|
|||
|
e7cef039f3
|
@@ -0,0 +1,18 @@
|
||||
type: changed
|
||||
area: subtitles
|
||||
|
||||
- Subtitle tokenization no longer runs a duplicate full `parseText` pass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line.
|
||||
- The Yomitan scanning helpers are now installed once per parser window (`__subminerYomitanScan`) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call.
|
||||
- termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
|
||||
- The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ). The shrinking-window retry ladder keeps following the consumed lengths the backend reports, and only blind guesses (windows the backend consumed whole, which tell it nothing) are capped at four per position. A line that hits that cap escalates to a single `parseText` for the whole line, so a hard line still resolves to dictionary tokens instead of an unparsed run, without letting the ladder run to one lookup per window length.
|
||||
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
|
||||
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused for the whole time the subtitle processing controller is working on the line, including the provisional raw emit that precedes tokenization, so it never competes with the on-screen line for the parser window. The pause is released when the controller reports it has settled, which also covers the lines that finish without an emit (a suppressed duplicate or a failed tokenization) and used to leave prefetching paused indefinitely.
|
||||
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
||||
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
||||
- Subtitle prefetching no longer stays paused for the rest of a cue when the same subtitle text is reported twice and there is nothing to tokenize. This covers the startup and overlay priming paths as well as ordinary subtitle changes.
|
||||
- Character name and image lookups are now refreshed centrally whenever a character dictionary sync changes its content, so a newly added name can no longer be skipped by a stale candidate list.
|
||||
- A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
|
||||
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
||||
- The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
|
||||
- The unnamed-mob disambiguator filter (Girl A / Girl B) now targets the letters those labels are split into, instead of every one-character term: a character whose name really is one character (𠮷, or a single kana) keeps it. The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
|
||||
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it.
|
||||
@@ -50,10 +50,18 @@ subtitles do not draw.
|
||||
7. Cache miss: call `refreshCurrentSubtitle(text)`. Normal processing emits a plain payload
|
||||
synchronously, then replaces it with the tokenized payload when ready.
|
||||
|
||||
In `src/main.ts`, both `onSubtitleChange` and `refreshCurrentSubtitle` pause
|
||||
`subtitlePrefetchService`, notify it with `onSeek(lastObservedTimePos)`, and then call the matching
|
||||
`subtitleProcessingController` method. This gives the visible overlay priority over background
|
||||
prefetch work and re-centers prefetch around the live playback time.
|
||||
Both `onSubtitleChange` and `refreshCurrentSubtitle` pause `subtitlePrefetchService` and then call
|
||||
the matching `subtitleProcessingController` method, giving the visible overlay priority over
|
||||
background prefetch work. Prefetch is not re-centered here: restarting the run per line
|
||||
(`onSeek`) discarded the in-flight tokenization every time the subtitle changed, so only real
|
||||
seeks restart it (see `onTimePosUpdate` in `src/main.ts`).
|
||||
|
||||
The pause is released by the controller's `onProcessingSettled` callback, which fires once it has
|
||||
no work left. Emits do not release it: the first emit for an uncached line is the plain payload
|
||||
that precedes tokenization, and a run can finish without emitting at all (a suppressed duplicate,
|
||||
a failed tokenization). Both controller methods return whether processing is now pending, and the
|
||||
caller resumes immediately when it is not — a repeated subtitle schedules no work, so no settle is
|
||||
coming and prefetching would otherwise idle for the rest of the cue.
|
||||
|
||||
## Live Cue Delivery
|
||||
|
||||
|
||||
@@ -539,3 +539,110 @@ test('default cache limit covers a full-length title without evicting', () => {
|
||||
assert.equal(controller.hasCachedSubtitle('line-0'), true);
|
||||
assert.equal(controller.hasCachedSubtitle('line-1999'), true);
|
||||
});
|
||||
|
||||
test('onSubtitleChange reports whether processing was scheduled', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
// New text schedules work, so an emit (and anything gated on it) will follow.
|
||||
assert.equal(controller.onSubtitleChange('字幕'), true);
|
||||
await flushMicrotasks();
|
||||
|
||||
// A repeat emits nothing, so callers must not wait on an emit that is never
|
||||
// coming (subtitle prefetching would stay paused for the rest of the cue).
|
||||
const emittedCount = emitted.length;
|
||||
assert.equal(controller.onSubtitleChange('字幕'), false);
|
||||
await flushMicrotasks();
|
||||
assert.equal(emitted.length, emittedCount);
|
||||
});
|
||||
|
||||
test('refreshCurrentSubtitle reports the empty-text emit that an in-flight run will deliver', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveFirst: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
if (text === '字幕') {
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveFirst = resolve;
|
||||
});
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('字幕');
|
||||
await flushMicrotasks();
|
||||
|
||||
// Clearing the subtitle while tokenization is in flight: the running loop
|
||||
// picks the empty text up and emits it, so callers gated on that emit (the
|
||||
// prefetch pause) must be told one is coming.
|
||||
assert.equal(controller.refreshCurrentSubtitle(''), true);
|
||||
|
||||
resolveFirst?.({ text: '字幕', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
// '字幕' is the provisional plain emit the in-flight run already made before
|
||||
// the refresh; '' is the emit the refresh promised.
|
||||
assert.deepEqual(
|
||||
emitted.map((payload) => payload.text),
|
||||
['字幕', ''],
|
||||
);
|
||||
});
|
||||
|
||||
test('onProcessingSettled fires once after the queue drains, including runs that emit nothing', async () => {
|
||||
const events: string[] = [];
|
||||
let resolveFirst: ((value: SubtitleData | null) => void) | undefined;
|
||||
let tokenizationFails = false;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
if (tokenizationFails) {
|
||||
return null;
|
||||
}
|
||||
if (text === '一行目') {
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveFirst = resolve;
|
||||
});
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => events.push(`emit:${payload.text}`),
|
||||
onProcessingSettled: () => events.push('settled'),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('一行目');
|
||||
await flushMicrotasks();
|
||||
// A second line arrives before the first finishes: the controller still has
|
||||
// work, so it must not report itself settled between the two.
|
||||
controller.onSubtitleChange('二行目');
|
||||
resolveFirst?.({ text: '一行目', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(events, ['emit:一行目', 'emit:二行目', 'emit:二行目', 'settled']);
|
||||
|
||||
// Tokenization failure on a line already shown plain: nothing is emitted, and
|
||||
// the settle signal is the only way a caller learns the work is over.
|
||||
events.length = 0;
|
||||
tokenizationFails = true;
|
||||
controller.invalidateTokenizationCache();
|
||||
assert.equal(controller.refreshCurrentSubtitle('二行目'), true);
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(events, ['settled']);
|
||||
});
|
||||
|
||||
test('refreshCurrentSubtitle reports no emit for empty text when nothing is running', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
assert.equal(controller.refreshCurrentSubtitle(''), false);
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, []);
|
||||
});
|
||||
|
||||
@@ -3,6 +3,14 @@ import type { SubtitleData } from '../../types';
|
||||
export interface SubtitleProcessingControllerDeps {
|
||||
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
||||
emitSubtitle: (payload: SubtitleData) => void;
|
||||
/**
|
||||
* Fires when the controller runs out of work: every scheduled line has been
|
||||
* processed, whether it ended in an emit, a suppressed duplicate, or a
|
||||
* tokenizer failure. Callers that hold a resource for the duration of
|
||||
* processing (prefetch pausing) release it here rather than on an emit,
|
||||
* which is not guaranteed to happen.
|
||||
*/
|
||||
onProcessingSettled?: () => void;
|
||||
logDebug?: (message: string) => void;
|
||||
now?: () => number;
|
||||
cacheLimit?: number;
|
||||
@@ -17,8 +25,15 @@ export interface SubtitleProcessingControllerDeps {
|
||||
export const DEFAULT_SUBTITLE_TOKENIZATION_CACHE_LIMIT = 2500;
|
||||
|
||||
export interface SubtitleProcessingController {
|
||||
onSubtitleChange: (text: string) => void;
|
||||
refreshCurrentSubtitle: (textOverride?: string) => void;
|
||||
/**
|
||||
* Returns whether processing is now scheduled or already in flight for this
|
||||
* event. A false return means the controller is idle and will do nothing, so
|
||||
* onProcessingSettled will not fire; callers that pause work for the duration
|
||||
* of processing (such as subtitle prefetching) must release it themselves.
|
||||
*/
|
||||
onSubtitleChange: (text: string) => boolean;
|
||||
/** Same contract as onSubtitleChange: whether processing is pending. */
|
||||
refreshCurrentSubtitle: (textOverride?: string) => boolean;
|
||||
invalidateTokenizationCache: () => void;
|
||||
preCacheTokenization: (text: string, data: SubtitleData) => void;
|
||||
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
||||
@@ -164,14 +179,20 @@ export function createSubtitleProcessingController(
|
||||
(latestText.trim() && cacheGeneration !== lastEmittedGeneration)
|
||||
) {
|
||||
processLatest();
|
||||
return;
|
||||
}
|
||||
// Nothing left to do: signal completion even when this run emitted
|
||||
// nothing (suppressed duplicate, tokenizer failure), or callers waiting
|
||||
// on the controller would wait forever.
|
||||
deps.onProcessingSettled?.();
|
||||
});
|
||||
};
|
||||
|
||||
return {
|
||||
onSubtitleChange: (text: string) => {
|
||||
if (text === latestText) {
|
||||
return;
|
||||
// A run already in flight for this text will still emit for it.
|
||||
return processing;
|
||||
}
|
||||
latestText = text;
|
||||
if (
|
||||
@@ -183,21 +204,25 @@ export function createSubtitleProcessingController(
|
||||
lastPlainEmittedText = text;
|
||||
}
|
||||
processLatest();
|
||||
return true;
|
||||
},
|
||||
refreshCurrentSubtitle: (textOverride?: string) => {
|
||||
if (typeof textOverride === 'string') {
|
||||
latestText = textOverride;
|
||||
}
|
||||
if (!latestText.trim()) {
|
||||
return;
|
||||
// A run in flight will pick this up and emit the empty subtitle, so
|
||||
// the caller is still waiting on an emit.
|
||||
return processing;
|
||||
}
|
||||
if (
|
||||
processing ||
|
||||
(latestText === lastEmittedText && cacheGeneration === lastEmittedGeneration)
|
||||
) {
|
||||
return;
|
||||
if (processing) {
|
||||
return true;
|
||||
}
|
||||
if (latestText === lastEmittedText && cacheGeneration === lastEmittedGeneration) {
|
||||
return false;
|
||||
}
|
||||
processLatest();
|
||||
return true;
|
||||
},
|
||||
invalidateTokenizationCache: () => {
|
||||
tokenizationCache.clear();
|
||||
|
||||
@@ -2934,44 +2934,12 @@ test('tokenizeSubtitle preserves Yomitan compound token when MeCab components ar
|
||||
return [];
|
||||
}
|
||||
|
||||
if (script.includes('parseText')) {
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: '取り組んで',
|
||||
surface: '取り組んで',
|
||||
reading: 'とりくんで',
|
||||
headwords: [[{ term: '取り組む' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'もらいます',
|
||||
reading: 'もらいます',
|
||||
headwords: [[{ term: 'もらう' }]],
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
surface: '取り',
|
||||
reading: 'とり',
|
||||
headword: '取る',
|
||||
headword: '取り組む',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
},
|
||||
{
|
||||
surface: '組んで',
|
||||
reading: 'くんで',
|
||||
headword: '組む',
|
||||
startPos: 2,
|
||||
endPos: 5,
|
||||
},
|
||||
{
|
||||
|
||||
@@ -70,6 +70,7 @@ export interface TokenizerServiceDeps {
|
||||
getNameMatchImagesEnabled?: () => boolean;
|
||||
getCharacterNameImage?: (term: string) => CharacterNameImage | null;
|
||||
getCurrentCharacterDictionaryMediaId?: () => number | null;
|
||||
getCharacterNameCandidates?: () => { key: string; forms: string[] } | null;
|
||||
getFrequencyDictionaryEnabled?: () => boolean;
|
||||
getFrequencyDictionaryMatchMode?: () => FrequencyDictionaryMatchMode;
|
||||
getFrequencyRank?: FrequencyDictionaryLookup;
|
||||
@@ -106,6 +107,7 @@ export interface TokenizerDepsRuntimeOptions {
|
||||
getNameMatchImagesEnabled?: () => boolean;
|
||||
getCharacterNameImage?: (term: string) => CharacterNameImage | null;
|
||||
getCurrentCharacterDictionaryMediaId?: () => number | null;
|
||||
getCharacterNameCandidates?: () => { key: string; forms: string[] } | null;
|
||||
getFrequencyDictionaryEnabled?: () => boolean;
|
||||
getFrequencyDictionaryMatchMode?: () => FrequencyDictionaryMatchMode;
|
||||
getFrequencyRank?: FrequencyDictionaryLookup;
|
||||
@@ -266,6 +268,7 @@ export function createTokenizerDepsRuntime(
|
||||
getNameMatchImagesEnabled: options.getNameMatchImagesEnabled,
|
||||
getCharacterNameImage: options.getCharacterNameImage,
|
||||
getCurrentCharacterDictionaryMediaId: options.getCurrentCharacterDictionaryMediaId,
|
||||
getCharacterNameCandidates: options.getCharacterNameCandidates,
|
||||
getFrequencyDictionaryEnabled: options.getFrequencyDictionaryEnabled,
|
||||
getFrequencyDictionaryMatchMode: options.getFrequencyDictionaryMatchMode ?? (() => 'headword'),
|
||||
getFrequencyRank: options.getFrequencyRank,
|
||||
@@ -716,15 +719,30 @@ function getAnnotationOptions(deps: TokenizerServiceDeps): TokenizerAnnotationOp
|
||||
};
|
||||
}
|
||||
|
||||
// Per-line stage durations for the pipeline debug log; every field is filled in
|
||||
// by the stage that awaits the corresponding work.
|
||||
interface TokenizationStageTimings {
|
||||
scanMs?: number;
|
||||
mecabMs?: number;
|
||||
frequencyMs?: number;
|
||||
annotateMs?: number;
|
||||
}
|
||||
|
||||
async function parseWithYomitanInternalParser(
|
||||
text: string,
|
||||
deps: TokenizerServiceDeps,
|
||||
options: TokenizerAnnotationOptions,
|
||||
stageTimings?: TokenizationStageTimings,
|
||||
): Promise<MergedToken[] | null> {
|
||||
const scanStartedAtMs = Date.now();
|
||||
const selectedTokens = await requestYomitanScanTokens(text, deps, logger, {
|
||||
includeNameMatchMetadata: options.nameMatchEnabled,
|
||||
currentCharacterDictionaryMediaId: deps.getCurrentCharacterDictionaryMediaId?.() ?? null,
|
||||
nameCandidates: deps.getCharacterNameCandidates?.() ?? null,
|
||||
});
|
||||
if (stageTimings) {
|
||||
stageTimings.scanMs = Date.now() - scanStartedAtMs;
|
||||
}
|
||||
if (!selectedTokens || selectedTokens.length === 0) {
|
||||
return null;
|
||||
}
|
||||
@@ -757,6 +775,7 @@ async function parseWithYomitanInternalParser(
|
||||
|
||||
const frequencyRankPromise: Promise<YomitanFrequencyIndex> = options.frequencyEnabled
|
||||
? (async () => {
|
||||
const frequencyStartedAtMs = Date.now();
|
||||
const frequencyMatchMode = options.frequencyMatchMode;
|
||||
const termReadingList = buildYomitanFrequencyTermReadingList(
|
||||
normalizedSelectedTokens,
|
||||
@@ -767,12 +786,17 @@ async function parseWithYomitanInternalParser(
|
||||
deps,
|
||||
logger,
|
||||
);
|
||||
return buildYomitanFrequencyIndex(yomitanFrequencies);
|
||||
const frequencyIndex = buildYomitanFrequencyIndex(yomitanFrequencies);
|
||||
if (stageTimings) {
|
||||
stageTimings.frequencyMs = Date.now() - frequencyStartedAtMs;
|
||||
}
|
||||
return frequencyIndex;
|
||||
})()
|
||||
: Promise.resolve({ byPair: new Map(), byTerm: new Map() });
|
||||
|
||||
const mecabEnrichmentPromise: Promise<MergedToken[]> = needsMecabPosEnrichment(options)
|
||||
? (async () => {
|
||||
const mecabStartedAtMs = Date.now();
|
||||
try {
|
||||
const mecabTokens = await deps.tokenizeWithMecab(text);
|
||||
const enrichTokensWithMecab = deps.enrichTokensWithMecab ?? enrichTokensWithMecabAsync;
|
||||
@@ -786,6 +810,10 @@ async function parseWithYomitanInternalParser(
|
||||
`textLength=${text.length}`,
|
||||
);
|
||||
return normalizedSelectedTokens;
|
||||
} finally {
|
||||
if (stageTimings) {
|
||||
stageTimings.mecabMs = Date.now() - mecabStartedAtMs;
|
||||
}
|
||||
}
|
||||
})()
|
||||
: Promise.resolve(normalizedSelectedTokens);
|
||||
@@ -876,15 +904,35 @@ export async function tokenizeSubtitle(
|
||||
const annotationOptions = getAnnotationOptions(deps);
|
||||
annotationOptions.sourceText = tokenizeText;
|
||||
|
||||
const yomitanTokens = await parseWithYomitanInternalParser(tokenizeText, deps, annotationOptions);
|
||||
const stageTimings: TokenizationStageTimings = {};
|
||||
const startedAtMs = Date.now();
|
||||
const logStageTimings = (tokenCount: number): void => {
|
||||
logger.debug(
|
||||
`Subtitle tokenization stages; textLength=${tokenizeText.length}, tokenCount=${tokenCount}, ` +
|
||||
`scanMs=${stageTimings.scanMs ?? '-'}, mecabMs=${stageTimings.mecabMs ?? '-'}, ` +
|
||||
`frequencyMs=${stageTimings.frequencyMs ?? '-'}, annotateMs=${stageTimings.annotateMs ?? '-'}, ` +
|
||||
`totalMs=${Date.now() - startedAtMs}`,
|
||||
);
|
||||
};
|
||||
|
||||
const yomitanTokens = await parseWithYomitanInternalParser(
|
||||
tokenizeText,
|
||||
deps,
|
||||
annotationOptions,
|
||||
stageTimings,
|
||||
);
|
||||
if (yomitanTokens && yomitanTokens.length > 0) {
|
||||
const annotateStartedAtMs = Date.now();
|
||||
const annotatedTokens = await applyAnnotationStage(yomitanTokens, deps, annotationOptions);
|
||||
stageTimings.annotateMs = Date.now() - annotateStartedAtMs;
|
||||
const renderedTokens = applyCharacterNameImages(annotatedTokens, deps, annotationOptions);
|
||||
logStageTimings(renderedTokens.length);
|
||||
return {
|
||||
text: displayText,
|
||||
tokens: renderedTokens.length > 0 ? renderedTokens : null,
|
||||
};
|
||||
}
|
||||
|
||||
logStageTimings(0);
|
||||
return { text: displayText, tokens: null };
|
||||
}
|
||||
|
||||
@@ -366,8 +366,11 @@ export function createReplayMessageStore(messages: GoldenRecordedMessage[]): Rep
|
||||
};
|
||||
}
|
||||
|
||||
async function runInjectedScriptInVm(script: string, store: ReplayMessageStore): Promise<unknown> {
|
||||
return await vm.runInNewContext(script, {
|
||||
// One persistent context per fixture, matching the real parser window: the
|
||||
// scan runtime installs itself once into globalThis and later per-line call
|
||||
// scripts reuse it.
|
||||
function createInjectedScriptVm(store: ReplayMessageStore): (script: string) => Promise<unknown> {
|
||||
const context = vm.createContext({
|
||||
chrome: {
|
||||
runtime: {
|
||||
lastError: null,
|
||||
@@ -393,6 +396,7 @@ async function runInjectedScriptInVm(script: string, store: ReplayMessageStore):
|
||||
Set,
|
||||
String,
|
||||
});
|
||||
return async (script: string) => await vm.runInContext(script, context);
|
||||
}
|
||||
|
||||
export function createReplayTokenizerDeps(fixture: GoldenFixture): TokenizerServiceDeps {
|
||||
@@ -400,13 +404,14 @@ export function createReplayTokenizerDeps(fixture: GoldenFixture): TokenizerServ
|
||||
const scriptResults = new Map(
|
||||
fixture.recording.scripts.map((entry) => [entry.sha256, entry] as const),
|
||||
);
|
||||
const runInjectedScriptInVm = createInjectedScriptVm(store);
|
||||
|
||||
const parserWindow = {
|
||||
isDestroyed: () => false,
|
||||
webContents: {
|
||||
executeJavaScript: async (script: string) => {
|
||||
try {
|
||||
return await runInjectedScriptInVm(script, store);
|
||||
return await runInjectedScriptInVm(script);
|
||||
} catch (vmError) {
|
||||
const recorded = scriptResults.get(hashInjectedScript(script));
|
||||
if (recorded) {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3,6 +3,14 @@ import * as fs from 'fs';
|
||||
import * as http from 'http';
|
||||
import * as path from 'path';
|
||||
import { selectYomitanParseTokens } from './parser-selection-stage';
|
||||
import {
|
||||
buildYomitanScanCallScript,
|
||||
buildYomitanScanNameCandidatesScript,
|
||||
CHARACTER_DICTIONARY_TITLE_PREFIX,
|
||||
YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT,
|
||||
YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL,
|
||||
type YomitanFrequencyMode,
|
||||
} from './yomitan-scan-runtime-script';
|
||||
|
||||
interface LoggerLike {
|
||||
error: (message: string, ...args: unknown[]) => void;
|
||||
@@ -22,8 +30,6 @@ interface YomitanParserRuntimeDeps {
|
||||
createYomitanExtensionWindow?: (pageName: string) => Promise<BrowserWindow | null>;
|
||||
}
|
||||
|
||||
type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
||||
|
||||
export interface YomitanDictionaryInfo {
|
||||
title: string;
|
||||
revision?: string | number;
|
||||
@@ -74,13 +80,19 @@ export interface YomitanAddNoteResult {
|
||||
}
|
||||
|
||||
const DEFAULT_YOMITAN_SCAN_LENGTH = 40;
|
||||
const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
const yomitanProfileMetadataByWindow = new WeakMap<BrowserWindow, YomitanProfileMetadata>();
|
||||
const yomitanProfileDiagnosticsLoggedByWindow = new WeakSet<BrowserWindow>();
|
||||
const yomitanFrequencyCacheByWindow = new WeakMap<
|
||||
BrowserWindow,
|
||||
Map<string, YomitanTermFrequency[]>
|
||||
>();
|
||||
// Epoch passed with every scan request; the in-window termsFind cache clears
|
||||
// itself when the epoch changes (dictionary imports, settings changes).
|
||||
const yomitanScanCacheEpochByWindow = new WeakMap<BrowserWindow, number>();
|
||||
|
||||
function getYomitanScanCacheEpoch(window: BrowserWindow): number {
|
||||
return yomitanScanCacheEpochByWindow.get(window) ?? 0;
|
||||
}
|
||||
|
||||
function isObject(value: unknown): value is Record<string, unknown> {
|
||||
return Boolean(value && typeof value === 'object');
|
||||
@@ -99,6 +111,7 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
typeof entry.startPos === 'number' &&
|
||||
typeof entry.endPos === 'number' &&
|
||||
(entry.isNameMatch === undefined || typeof entry.isNameMatch === 'boolean') &&
|
||||
(entry.isUnparsedRun === undefined || typeof entry.isUnparsedRun === 'boolean') &&
|
||||
(entry.frequencyRank === undefined || typeof entry.frequencyRank === 'number') &&
|
||||
(entry.wordClasses === undefined ||
|
||||
(Array.isArray(entry.wordClasses) &&
|
||||
@@ -107,13 +120,9 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
);
|
||||
}
|
||||
|
||||
function scanTokenSpanKey(token: YomitanScanToken): string {
|
||||
return `${token.startPos}:${token.endPos}:${token.surface}`;
|
||||
}
|
||||
|
||||
// Maps a parse-selected token to the scanner-token shape carried out of the
|
||||
// parser runtime. Shared by both selectYomitanParseTokens fallback paths so the
|
||||
// projected fields stay in sync as the shape changes.
|
||||
// parser runtime, used by the parseText fallback path when the in-window
|
||||
// scanner is unavailable.
|
||||
function toYomitanScanToken(token: {
|
||||
surface: string;
|
||||
reading: string;
|
||||
@@ -132,66 +141,6 @@ function toYomitanScanToken(token: {
|
||||
};
|
||||
}
|
||||
|
||||
// parseText segmentation is authoritative (it emits filler chunks for text the
|
||||
// termsFind scanner skips), but only the termsFind scanner carries annotation
|
||||
// metadata (isNameMatch, frequencyRank, headwordReading, wordClasses). Graft
|
||||
// scanner tokens onto the parseText segmentation per matching span so one
|
||||
// unmatched chunk degrades only itself instead of dropping the whole line's
|
||||
// metadata.
|
||||
//
|
||||
// Exception: character-name tokens. The greedy name scan can re-segment text
|
||||
// around a name (e.g. とヨータ → と + ヨータ instead of とヨー + タ), so
|
||||
// parseText segmentation cannot be authoritative there. Each name span is
|
||||
// expanded until it aligns with token boundaries in both segmentations, then
|
||||
// the parse tokens inside are replaced with the scanner tokens.
|
||||
function mergeScannerTokensIntoParseTokens(
|
||||
parseScanTokens: YomitanScanToken[],
|
||||
scannerTokens: YomitanScanToken[],
|
||||
): YomitanScanToken[] {
|
||||
const scannerTokensBySpan = new Map<string, YomitanScanToken>();
|
||||
for (const token of scannerTokens) {
|
||||
scannerTokensBySpan.set(scanTokenSpanKey(token), token);
|
||||
}
|
||||
const graftedTokens = parseScanTokens.map(
|
||||
(token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token,
|
||||
);
|
||||
|
||||
const nameTokens = scannerTokens.filter((token) => token.isNameMatch === true);
|
||||
if (nameTokens.length === 0) {
|
||||
return graftedTokens;
|
||||
}
|
||||
|
||||
const regions = nameTokens.map((token) => ({ start: token.startPos, end: token.endPos }));
|
||||
const allTokens = [...parseScanTokens, ...scannerTokens];
|
||||
let expanded = true;
|
||||
while (expanded) {
|
||||
expanded = false;
|
||||
for (const region of regions) {
|
||||
for (const token of allTokens) {
|
||||
const overlaps = token.startPos < region.end && token.endPos > region.start;
|
||||
const extendsBeyond = token.startPos < region.start || token.endPos > region.end;
|
||||
if (overlaps && extendsBeyond) {
|
||||
region.start = Math.min(region.start, token.startPos);
|
||||
region.end = Math.max(region.end, token.endPos);
|
||||
expanded = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isInsideNameRegion = (token: YomitanScanToken): boolean =>
|
||||
regions.some((region) => token.startPos >= region.start && token.endPos <= region.end);
|
||||
|
||||
const merged = graftedTokens.filter((token) => !isInsideNameRegion(token));
|
||||
for (const token of scannerTokens) {
|
||||
if (isInsideNameRegion(token)) {
|
||||
merged.push(token);
|
||||
}
|
||||
}
|
||||
merged.sort((a, b) => a.startPos - b.startPos || a.endPos - b.endPos);
|
||||
return merged;
|
||||
}
|
||||
|
||||
function makeTermReadingCacheKey(term: string, reading: string | null): string {
|
||||
return `${term}\u0000${reading ?? ''}`;
|
||||
}
|
||||
@@ -208,6 +157,7 @@ function getWindowFrequencyCache(window: BrowserWindow): Map<string, YomitanTerm
|
||||
function clearWindowCaches(window: BrowserWindow): void {
|
||||
yomitanProfileMetadataByWindow.delete(window);
|
||||
yomitanFrequencyCacheByWindow.delete(window);
|
||||
yomitanScanCacheEpochByWindow.set(window, getYomitanScanCacheEpoch(window) + 1);
|
||||
}
|
||||
export function clearYomitanParserCachesForWindow(window: BrowserWindow): void {
|
||||
clearWindowCaches(window);
|
||||
@@ -704,6 +654,10 @@ async function ensureYomitanParserWindow(
|
||||
if (readyPromise) {
|
||||
await readyPromise;
|
||||
}
|
||||
// Eagerly install the scan runtime so the first subtitle line does not
|
||||
// pay the install round trip; failures fall back to the per-request
|
||||
// install-and-retry path.
|
||||
await installYomitanScanRuntime(parserWindow).catch(() => {});
|
||||
|
||||
return true;
|
||||
} catch (err) {
|
||||
@@ -877,668 +831,42 @@ async function serveDictionaryZipOnce<T>(
|
||||
}
|
||||
}
|
||||
|
||||
const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
||||
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
||||
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
||||
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
||||
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
||||
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
||||
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0x3400, 0x9fff]];
|
||||
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
||||
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
||||
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
||||
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
||||
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
||||
function getSegmentReadingContribution(segment) {
|
||||
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
||||
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
||||
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
return isKanaOnly ? segmentText : "";
|
||||
}
|
||||
function getProlongedHiragana(previousCharacter) {
|
||||
switch (previousCharacter) {
|
||||
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
||||
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
||||
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
||||
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
||||
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
||||
default: return null;
|
||||
}
|
||||
}
|
||||
function getFuriganaKanaSegments(text, reading) {
|
||||
const newSegments = [];
|
||||
let start = 0;
|
||||
let state = (reading[0] === text[0]);
|
||||
for (let i = 1; i < text.length; ++i) {
|
||||
const newState = (reading[i] === text[i]);
|
||||
if (state === newState) { continue; }
|
||||
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
||||
state = newState;
|
||||
start = i;
|
||||
}
|
||||
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
||||
return newSegments;
|
||||
}
|
||||
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
||||
let result = '';
|
||||
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
||||
for (let char of text) {
|
||||
const codePoint = char.codePointAt(0);
|
||||
switch (codePoint) {
|
||||
case KATAKANA_SMALL_KA_CODE_POINT:
|
||||
case KATAKANA_SMALL_KE_CODE_POINT:
|
||||
break;
|
||||
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
||||
if (!keepProlongedSoundMarks && result.length > 0) {
|
||||
const char2 = getProlongedHiragana(result[result.length - 1]);
|
||||
if (char2 !== null) { char = char2; }
|
||||
}
|
||||
break;
|
||||
default:
|
||||
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
||||
char = String.fromCodePoint(codePoint + offset);
|
||||
}
|
||||
break;
|
||||
}
|
||||
result += char;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
||||
const groupCount = groups.length - groupsStart;
|
||||
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
||||
const group = groups[groupsStart];
|
||||
const {isKana, text} = group;
|
||||
if (isKana) {
|
||||
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
||||
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
||||
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
||||
return segments;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let result = null;
|
||||
for (let i = reading.length; i >= text.length; --i) {
|
||||
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (result !== null) { return null; }
|
||||
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
||||
result = segments;
|
||||
}
|
||||
if (groupCount === 1) { break; }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function distributeFurigana(term, reading) {
|
||||
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
||||
const groups = [];
|
||||
let groupPre = null;
|
||||
let isKanaPre = null;
|
||||
for (const c of term) {
|
||||
const isKana = isCodePointKana(c.codePointAt(0));
|
||||
if (isKana === isKanaPre) { groupPre.text += c; }
|
||||
else {
|
||||
groupPre = {isKana, text: c, textNormalized: null};
|
||||
groups.push(groupPre);
|
||||
isKanaPre = isKana;
|
||||
}
|
||||
}
|
||||
for (const group of groups) {
|
||||
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
||||
}
|
||||
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
||||
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
||||
}
|
||||
function getStemLength(text1, text2) {
|
||||
const minLength = Math.min(text1.length, text2.length);
|
||||
if (minLength === 0) { return 0; }
|
||||
let i = 0;
|
||||
while (true) {
|
||||
const char1 = text1.codePointAt(i);
|
||||
const char2 = text2.codePointAt(i);
|
||||
if (char1 !== char2) { break; }
|
||||
const charLength = String.fromCodePoint(char1).length;
|
||||
i += charLength;
|
||||
if (i >= minLength) {
|
||||
if (i > minLength) { i -= charLength; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
return i;
|
||||
}
|
||||
function distributeFuriganaInflected(term, reading, source) {
|
||||
const termNormalized = convertKatakanaToHiragana(term);
|
||||
const readingNormalized = convertKatakanaToHiragana(reading);
|
||||
const sourceNormalized = convertKatakanaToHiragana(source);
|
||||
let mainText = term;
|
||||
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
||||
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
||||
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
||||
mainText = reading;
|
||||
stemLength = readingStemLength;
|
||||
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
||||
}
|
||||
const segments = [];
|
||||
if (stemLength > 0) {
|
||||
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
||||
const segments2 = distributeFurigana(mainText, reading);
|
||||
let consumed = 0;
|
||||
for (const segment of segments2) {
|
||||
const start = consumed;
|
||||
consumed += segment.text.length;
|
||||
if (consumed < stemLength) { segments.push(segment); }
|
||||
else if (consumed === stemLength) { segments.push(segment); break; }
|
||||
else {
|
||||
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (stemLength < source.length) {
|
||||
const remainder = source.substring(stemLength);
|
||||
const last = segments[segments.length - 1];
|
||||
if (last && last.reading.length === 0) { last.text += remainder; }
|
||||
else { segments.push(createFuriganaSegment(remainder, '')); }
|
||||
}
|
||||
return segments;
|
||||
}
|
||||
function parsePositiveFrequencyNumber(value) {
|
||||
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||
return Math.max(1, Math.floor(value));
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
||||
if (!numericMatch) { return null; }
|
||||
const parsed = Number.parseFloat(numericMatch);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
||||
return Math.max(1, Math.floor(parsed));
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) {
|
||||
const parsed = parsePositiveFrequencyNumber(item);
|
||||
if (parsed !== null) { return parsed; }
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseDisplayFrequencyNumber(value) {
|
||||
if (typeof value === 'string') {
|
||||
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
||||
if (!leadingDigits) { return null; }
|
||||
const parsed = Number.parseInt(leadingDigits, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
||||
}
|
||||
return parsePositiveFrequencyNumber(value);
|
||||
}
|
||||
function getFrequencyDictionaryName(frequency) {
|
||||
const candidates = [
|
||||
frequency?.dictionary,
|
||||
frequency?.dictionaryName,
|
||||
frequency?.name,
|
||||
frequency?.title,
|
||||
frequency?.dictionaryTitle,
|
||||
frequency?.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
return candidate.trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
||||
for (const frequency of dictionaryEntry?.frequencies || []) {
|
||||
if (!frequency || typeof frequency !== 'object') { continue; }
|
||||
const frequencyHeadwordIndex = frequency.headwordIndex;
|
||||
if (typeof frequencyHeadwordIndex === 'number') {
|
||||
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
||||
} else if (headwordCount > 1) {
|
||||
continue;
|
||||
}
|
||||
const dictionary = getFrequencyDictionaryName(frequency);
|
||||
if (!dictionary) { continue; }
|
||||
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
||||
const rank =
|
||||
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
||||
parsePositiveFrequencyNumber(frequency.frequency);
|
||||
if (rank === null) { continue; }
|
||||
const priorityRaw = dictionaryPriorityByName[dictionary];
|
||||
const fallbackPriority =
|
||||
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
||||
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
||||
: Number.MAX_SAFE_INTEGER;
|
||||
const priority =
|
||||
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
||||
? Math.max(0, Math.floor(priorityRaw))
|
||||
: fallbackPriority;
|
||||
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
||||
best = { priority, rank };
|
||||
}
|
||||
}
|
||||
return best?.rank ?? null;
|
||||
}
|
||||
function hasExactSource(headword, token, requirePrimary) {
|
||||
for (const src of headword.sources || []) {
|
||||
if (src.originalText !== token) { continue; }
|
||||
if (requirePrimary && !src.isPrimary) { continue; }
|
||||
if (src.matchType !== 'exact') { continue; }
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
||||
const matches = [];
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
||||
matches.push({ dictionaryEntry, headword, headwordIndex });
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
function sameHeadword(match, preferredMatch) {
|
||||
if (!match || !preferredMatch) {
|
||||
return false;
|
||||
}
|
||||
if (match.headword?.term !== preferredMatch.headword?.term) {
|
||||
return false;
|
||||
}
|
||||
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
||||
const preferredReading =
|
||||
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
||||
if (!matchReading || !preferredReading) {
|
||||
return true;
|
||||
}
|
||||
return matchReading === preferredReading;
|
||||
}
|
||||
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
for (const match of matches) {
|
||||
const rank = getBestFrequencyRank(
|
||||
match.dictionaryEntry,
|
||||
match.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (rank === null) { continue; }
|
||||
if (best === null || rank < best) {
|
||||
best = rank;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
}
|
||||
function appendDictionaryNames(target, value) {
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
const candidates = [
|
||||
value.dictionary,
|
||||
value.dictionaryName,
|
||||
value.name,
|
||||
value.title,
|
||||
value.dictionaryTitle,
|
||||
value.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
target.push(candidate.trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
function getDictionaryEntryNames(entry) {
|
||||
const names = [];
|
||||
appendDictionaryNames(names, entry);
|
||||
for (const definition of entry?.definitions || []) {
|
||||
appendDictionaryNames(names, definition);
|
||||
}
|
||||
for (const frequency of entry?.frequencies || []) {
|
||||
appendDictionaryNames(names, frequency);
|
||||
}
|
||||
for (const pronunciation of entry?.pronunciations || []) {
|
||||
appendDictionaryNames(names, pronunciation);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
function isNameDictionaryEntry(entry) {
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
if (imageMatch) {
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseSubMinerMediaIdCandidate(value) {
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
||||
const parsed = Number.parseInt(value.trim(), 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function collectSubMinerMediaIds(value, target) {
|
||||
if (typeof value === 'string') {
|
||||
const parsed = parseSubMinerMediaIdFromString(value);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
||||
return;
|
||||
}
|
||||
const mediaIdCandidates = [
|
||||
value.subminerMediaId,
|
||||
value.subMinerMediaId,
|
||||
value.characterDictionaryMediaId,
|
||||
value.data?.subminerMediaId,
|
||||
value.data?.subMinerMediaId,
|
||||
value.data?.characterDictionaryMediaId
|
||||
];
|
||||
for (const candidate of mediaIdCandidates) {
|
||||
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
collectSubMinerMediaIds(child, target);
|
||||
}
|
||||
}
|
||||
function getSubMinerMediaIds(entry) {
|
||||
const mediaIds = new Set();
|
||||
collectSubMinerMediaIds(entry, mediaIds);
|
||||
return mediaIds;
|
||||
}
|
||||
function isCurrentMediaNameDictionaryEntry(entry) {
|
||||
if (!isNameDictionaryEntry(entry)) {
|
||||
return false;
|
||||
}
|
||||
if (currentCharacterDictionaryMediaId === null) {
|
||||
return true;
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
: (dictionaryEntries || []).filter((entry) => {
|
||||
if (!isNameDictionaryEntry(entry)) { return true; }
|
||||
return isCurrentMediaNameDictionaryEntry(entry);
|
||||
});
|
||||
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
||||
let matchedNameDictionary = false;
|
||||
if (includeNameMatchMetadata) {
|
||||
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
for (const match of exactPrimaryMatches) {
|
||||
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
||||
matchedNameDictionary = true;
|
||||
break;
|
||||
}
|
||||
if (matchedNameDictionary) { break; }
|
||||
}
|
||||
}
|
||||
const preferredMatch = exactPrimaryMatches[0];
|
||||
if (preferredMatch) {
|
||||
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
||||
.filter((match) => sameHeadword(match, preferredMatch));
|
||||
return {
|
||||
term: preferredMatch.headword.term,
|
||||
reading: preferredMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
||||
isNameMatch:
|
||||
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
||||
frequencyRank: getBestFrequencyRankForMatches(
|
||||
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
`;
|
||||
async function installYomitanScanRuntime(parserWindow: BrowserWindow): Promise<void> {
|
||||
await parserWindow.webContents.executeJavaScript(YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT, true);
|
||||
// A fresh runtime has no candidate list; force the next scan to reinstall it.
|
||||
yomitanScanNameCandidateKeyByWindow.delete(parserWindow);
|
||||
}
|
||||
|
||||
function buildYomitanScanningScript(
|
||||
text: string,
|
||||
profileIndex: number,
|
||||
scanLength: number,
|
||||
includeNameMatchMetadata: boolean,
|
||||
greedyNameScanEnabled: boolean,
|
||||
currentCharacterDictionaryMediaId: number | null,
|
||||
dictionaryPriorityByName: Record<string, number>,
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>,
|
||||
): string {
|
||||
return `
|
||||
(async () => {
|
||||
const invoke = (action, params) =>
|
||||
new Promise((resolve, reject) => {
|
||||
chrome.runtime.sendMessage({ action, params }, (response) => {
|
||||
if (chrome.runtime.lastError) {
|
||||
reject(new Error(chrome.runtime.lastError.message));
|
||||
// Key of the character-name candidate list currently installed in each parser
|
||||
// window, so an unchanged list costs nothing per line.
|
||||
const yomitanScanNameCandidateKeyByWindow = new WeakMap<BrowserWindow, string>();
|
||||
|
||||
async function ensureYomitanScanNameCandidates(
|
||||
parserWindow: BrowserWindow,
|
||||
nameCandidates: { key: string; forms: string[] } | null,
|
||||
logger: LoggerLike,
|
||||
): Promise<void> {
|
||||
const installedKey = yomitanScanNameCandidateKeyByWindow.get(parserWindow);
|
||||
const nextKey = nameCandidates?.key ?? '';
|
||||
if (installedKey === nextKey) {
|
||||
return;
|
||||
}
|
||||
if (!response || typeof response !== "object") {
|
||||
reject(new Error("Invalid response from Yomitan backend"));
|
||||
return;
|
||||
}
|
||||
if (response.error) {
|
||||
reject(new Error(response.error.message || "Yomitan backend error"));
|
||||
return;
|
||||
}
|
||||
resolve(response.result);
|
||||
});
|
||||
});
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const includeNameMatchMetadata = ${includeNameMatchMetadata ? 'true' : 'false'};
|
||||
const greedyNameScanEnabled = ${greedyNameScanEnabled ? 'true' : 'false'};
|
||||
const currentCharacterDictionaryMediaId = ${
|
||||
currentCharacterDictionaryMediaId !== null
|
||||
? String(currentCharacterDictionaryMediaId)
|
||||
: 'null'
|
||||
};
|
||||
const dictionaryPriorityByName = ${JSON.stringify(dictionaryPriorityByName)};
|
||||
const dictionaryFrequencyModeByName = ${JSON.stringify(dictionaryFrequencyModeByName)};
|
||||
const text = ${JSON.stringify(text)};
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
const termsFindCache = new Map();
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const cacheKey = position + ":" + windowLength;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached) { return cached; }
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const result = await invoke("termsFind", { text: substring, details, optionsContext: { index: ${profileIndex} } });
|
||||
termsFindCache.set(cacheKey, result);
|
||||
return result;
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
surface: segments.map((segment) => segment.text).join("") || source,
|
||||
reading: segments.map(getSegmentReadingContribution).join(""),
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
? Math.max(1, Math.floor(preferredHeadword.frequencyRank))
|
||||
: undefined,
|
||||
};
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return tokenPayload;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
|
||||
try {
|
||||
await parserWindow.webContents.executeJavaScript(
|
||||
buildYomitanScanNameCandidatesScript(nameCandidates),
|
||||
true,
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
yomitanScanNameCandidateKeyByWindow.set(parserWindow, nextKey);
|
||||
} catch (err) {
|
||||
// The scan falls back to checking every position when the list is absent,
|
||||
// so a failed install costs speed, never a missed name.
|
||||
logger.warn?.(
|
||||
'Failed to install Yomitan character-name scan candidates:',
|
||||
(err as Error).message,
|
||||
);
|
||||
yomitanScanNameCandidateKeyByWindow.delete(parserWindow);
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, ${scanLength});
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + ${scanLength});
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
while (i < text.length) {
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
// Cap the window at the next reserved name span so a generic match
|
||||
// cannot consume into it.
|
||||
const windowLength = nextNameToken ? Math.min(${scanLength}, nextNameToken.startPos - i) : ${scanLength};
|
||||
let attempt = await findTokenAt(i, windowLength);
|
||||
// Yomitan text normalization can consume characters (whitespace,
|
||||
// punctuation) beyond the matched term, leaving no headword whose
|
||||
// source equals the consumed text. Retry with shorter windows so a
|
||||
// valid prefix term (e.g. a character name before a paren) still
|
||||
// tokenizes instead of the position being skipped.
|
||||
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
const retry = await findTokenAt(i, retryLength);
|
||||
if (retry.token) {
|
||||
attempt = retry;
|
||||
break;
|
||||
}
|
||||
retryLength = Math.min(retryLength - 1, retry.matchedLength - 1);
|
||||
}
|
||||
if (attempt.token) {
|
||||
tokens.push(attempt.token);
|
||||
i += attempt.matchedLength;
|
||||
continue;
|
||||
}
|
||||
i += String.fromCodePoint(text.codePointAt(i)).length;
|
||||
}
|
||||
return tokens;
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
export async function requestYomitanParseResults(
|
||||
@@ -1635,6 +963,20 @@ export async function requestYomitanParseResults(
|
||||
}
|
||||
}
|
||||
|
||||
// parseText fallback for when the in-window scanner cannot run (script eval
|
||||
// failure, unexpected payload). The scanner walk is the primary tokenizer and
|
||||
// emits its own filler runs, so this extra full parse only happens on errors.
|
||||
async function requestYomitanParseFallbackTokens(
|
||||
text: string,
|
||||
deps: YomitanParserRuntimeDeps,
|
||||
logger: LoggerLike,
|
||||
): Promise<YomitanScanToken[] | null> {
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens = selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
return parseScanTokens && parseScanTokens.length > 0 ? parseScanTokens : null;
|
||||
}
|
||||
|
||||
export async function requestYomitanScanTokens(
|
||||
text: string,
|
||||
deps: YomitanParserRuntimeDeps,
|
||||
@@ -1642,6 +984,7 @@ export async function requestYomitanScanTokens(
|
||||
options?: {
|
||||
includeNameMatchMetadata?: boolean;
|
||||
currentCharacterDictionaryMediaId?: number | null;
|
||||
nameCandidates?: { key: string; forms: string[] } | null;
|
||||
},
|
||||
): Promise<YomitanScanToken[] | null> {
|
||||
const yomitanExt = deps.getYomitanExt();
|
||||
@@ -1655,10 +998,6 @@ export async function requestYomitanScanTokens(
|
||||
return null;
|
||||
}
|
||||
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedParseTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens = selectedParseTokens?.map(toYomitanScanToken) ?? null;
|
||||
|
||||
const metadata = await requestYomitanProfileMetadata(parserWindow, logger);
|
||||
const profileIndex = metadata?.profileIndex ?? 0;
|
||||
const scanLength = metadata?.scanLength ?? DEFAULT_YOMITAN_SCAN_LENGTH;
|
||||
@@ -1669,44 +1008,63 @@ export async function requestYomitanScanTokens(
|
||||
name.startsWith(CHARACTER_DICTIONARY_TITLE_PREFIX),
|
||||
);
|
||||
|
||||
try {
|
||||
const rawResult = await parserWindow.webContents.executeJavaScript(
|
||||
buildYomitanScanningScript(
|
||||
// Candidate name forms let the in-page pre-pass skip positions where no
|
||||
// character name can start. Installed only when it changes (per media), so
|
||||
// the per-line call stays a single tiny script.
|
||||
const nameCandidates = greedyNameScanEnabled ? (options?.nameCandidates ?? null) : null;
|
||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||
|
||||
const callScript = buildYomitanScanCallScript({
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
currentCharacterDictionaryMediaId:
|
||||
typeof options?.currentCharacterDictionaryMediaId === 'number' &&
|
||||
Number.isFinite(options.currentCharacterDictionaryMediaId) &&
|
||||
options.currentCharacterDictionaryMediaId > 0
|
||||
? Math.floor(options.currentCharacterDictionaryMediaId)
|
||||
: null,
|
||||
metadata?.dictionaryPriorityByName ?? {},
|
||||
metadata?.dictionaryFrequencyModeByName ?? {},
|
||||
),
|
||||
true,
|
||||
);
|
||||
dictionaryPriorityByName: metadata?.dictionaryPriorityByName ?? {},
|
||||
dictionaryFrequencyModeByName: metadata?.dictionaryFrequencyModeByName ?? {},
|
||||
cacheEpoch: getYomitanScanCacheEpoch(parserWindow),
|
||||
nameCandidateKey: nameCandidates?.key ?? null,
|
||||
});
|
||||
|
||||
try {
|
||||
let rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||
if (rawResult === YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL) {
|
||||
// First request for this window, or the page reloaded and dropped the
|
||||
// installed runtime: install and retry once. The candidate list lives in
|
||||
// the same page state, so it has to be reinstalled alongside it.
|
||||
await installYomitanScanRuntime(parserWindow);
|
||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||
rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||
}
|
||||
// The scanner reports a line where a position ran out of shrinking-window
|
||||
// retries: it stopped short of windows an uncapped ladder would have tried,
|
||||
// so a real term may be sitting in an unparsed run. One parseText for the
|
||||
// line is the bounded way to get the exhaustive answer back (this is the
|
||||
// parse the scanner replaced, and it only runs for these rare lines).
|
||||
if (isObject(rawResult) && rawResult.retryBudgetExhausted === true) {
|
||||
logger.info?.('Yomitan scanner exhausted its retry budget; parsing the line as a fallback.');
|
||||
const fallbackTokens = await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
if (fallbackTokens) {
|
||||
return fallbackTokens;
|
||||
}
|
||||
rawResult = rawResult.tokens;
|
||||
}
|
||||
if (isScanTokenArray(rawResult)) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return mergeScannerTokensIntoParseTokens(parseScanTokens, rawResult);
|
||||
// Filler-only results carry no dictionary match; keep the historical
|
||||
// contract of returning null so callers fall back to raw text.
|
||||
return rawResult.some((token) => token.isUnparsedRun !== true) ? rawResult : null;
|
||||
}
|
||||
return rawResult;
|
||||
}
|
||||
if (Array.isArray(rawResult)) {
|
||||
const selectedTokens = selectYomitanParseTokens(rawResult, () => false, 'headword');
|
||||
return selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
}
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
}
|
||||
return null;
|
||||
logger.error('Yomitan scanner returned an unexpected payload; using parseText fallback.');
|
||||
return await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
} catch (err) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
}
|
||||
logger.error('Yomitan scanner request failed:', (err as Error).message);
|
||||
return null;
|
||||
return await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,513 @@
|
||||
// In-page Yomitan scan runtime: the scan walk that gets installed once per
|
||||
// parser window as globalThis.__subminerYomitanScan, plus the tiny per-line
|
||||
// call script. Kept separate from the host runtime module so the injected
|
||||
// script text (which is data, not executed here) does not dominate that file;
|
||||
// the helper bundle it embeds lives in yomitan-scanning-helpers-script.ts.
|
||||
import { YOMITAN_SCANNING_HELPERS } from './yomitan-scanning-helpers-script';
|
||||
|
||||
export { CHARACTER_DICTIONARY_TITLE_PREFIX } from './yomitan-scanning-helpers-script';
|
||||
|
||||
export type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
||||
|
||||
// Bump whenever the install script below changes so already-loaded parser
|
||||
// windows re-install the new scan runtime instead of running the stale one.
|
||||
export const YOMITAN_SCAN_RUNTIME_VERSION = 6;
|
||||
export const YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL = '__subminer-yomitan-scan-runtime-missing__';
|
||||
|
||||
export interface YomitanScanRequestParams {
|
||||
text: string;
|
||||
profileIndex: number;
|
||||
scanLength: number;
|
||||
includeNameMatchMetadata: boolean;
|
||||
greedyNameScanEnabled: boolean;
|
||||
currentCharacterDictionaryMediaId: number | null;
|
||||
dictionaryPriorityByName: Record<string, number>;
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>;
|
||||
cacheEpoch: number;
|
||||
/**
|
||||
* Key of the character-name candidate list installed for the current media,
|
||||
* or null to scan every Japanese position (see the pre-pass prefilter).
|
||||
*/
|
||||
nameCandidateKey: string | null;
|
||||
}
|
||||
|
||||
// Installed once per parser window (and re-installed after in-page reloads):
|
||||
// keeps V8 from re-parsing the helper bundle on every subtitle line, and hosts
|
||||
// the cross-line termsFind cache. Each subtitle line then only evaluates a tiny
|
||||
// call into globalThis.__subminerYomitanScan.
|
||||
export const YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT = String.raw`
|
||||
(() => {
|
||||
if (globalThis.__subminerYomitanScanVersion === ${YOMITAN_SCAN_RUNTIME_VERSION}) {
|
||||
return true;
|
||||
}
|
||||
const invoke = (action, params) =>
|
||||
new Promise((resolve, reject) => {
|
||||
chrome.runtime.sendMessage({ action, params }, (response) => {
|
||||
if (chrome.runtime.lastError) {
|
||||
reject(new Error(chrome.runtime.lastError.message));
|
||||
return;
|
||||
}
|
||||
if (!response || typeof response !== "object") {
|
||||
reject(new Error("Invalid response from Yomitan backend"));
|
||||
return;
|
||||
}
|
||||
if (response.error) {
|
||||
reject(new Error(response.error.message || "Yomitan backend error"));
|
||||
return;
|
||||
}
|
||||
resolve(response.result);
|
||||
});
|
||||
});
|
||||
// Cross-line termsFind LRU keyed by profile + substring: subtitle lines
|
||||
// repeat particles and inflections constantly, so most lookups hit here.
|
||||
// Entries hold in-flight promises so concurrent identical lookups dedupe.
|
||||
const termsFindCache = new Map();
|
||||
// Two bounds. The key count keeps the map itself small; the accumulated
|
||||
// dictionary-entry count stands in for retained bytes, because a single
|
||||
// lookup over a common prefix can hold hundreds of entries with their full
|
||||
// glossaries and a key-count cap alone would not bound that.
|
||||
const TERMS_FIND_CACHE_LIMIT = 2000;
|
||||
const TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT = 20000;
|
||||
let termsFindCacheDictionaryEntries = 0;
|
||||
let termsFindCacheEpoch = -1;
|
||||
function dropCachedTermsFind(cacheKey, entry) {
|
||||
if (termsFindCache.get(cacheKey) !== entry) { return; }
|
||||
termsFindCache.delete(cacheKey);
|
||||
termsFindCacheDictionaryEntries -= entry.dictionaryEntryCount;
|
||||
}
|
||||
// Runs on insert and again once a lookup resolves: an entry is only worth
|
||||
// its estimated weight of 1 until then, so a single oversized response
|
||||
// would otherwise sit in the cache forever, over the limit and reused.
|
||||
function evictOverflowingTermsFindEntries() {
|
||||
while (
|
||||
termsFindCache.size > TERMS_FIND_CACHE_LIMIT ||
|
||||
termsFindCacheDictionaryEntries > TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT
|
||||
) {
|
||||
const oldest = termsFindCache.entries().next().value;
|
||||
if (oldest === undefined) { break; }
|
||||
dropCachedTermsFind(oldest[0], oldest[1]);
|
||||
}
|
||||
}
|
||||
// Only blind ladder steps are capped (see the retry loop): those are the
|
||||
// ones that would otherwise degrade into O(scanLength) lookups at a single
|
||||
// position. Steps the backend guides by reporting a shorter consumed length
|
||||
// stay uncapped, so a valid prefix term is still found on lines where
|
||||
// normalization eats a long tail.
|
||||
const MAX_BLIND_SHRINKING_WINDOW_RETRIES = 4;
|
||||
// Character-name candidate forms for the current media, installed
|
||||
// separately from the per-line scan call so the per-line script stays tiny.
|
||||
// Stored raw here; the normalized lookup index is built inside the scan,
|
||||
// where the kana-normalization helper is in scope, and reused by key.
|
||||
let rawNameCandidates = null;
|
||||
let nameCandidateIndex = null;
|
||||
globalThis.__subminerYomitanScanSetNameCandidates = (key, forms) => {
|
||||
if (!key || !Array.isArray(forms) || forms.length === 0) {
|
||||
rawNameCandidates = null;
|
||||
nameCandidateIndex = null;
|
||||
return false;
|
||||
}
|
||||
rawNameCandidates = { key, forms };
|
||||
nameCandidateIndex = null;
|
||||
return true;
|
||||
};
|
||||
globalThis.__subminerYomitanScanVersion = ${YOMITAN_SCAN_RUNTIME_VERSION};
|
||||
globalThis.__subminerYomitanScan = async (scanParams) => {
|
||||
const {
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
currentCharacterDictionaryMediaId,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName,
|
||||
cacheEpoch,
|
||||
nameCandidateKey
|
||||
} = scanParams;
|
||||
if (cacheEpoch !== termsFindCacheEpoch) {
|
||||
termsFindCache.clear();
|
||||
termsFindCacheDictionaryEntries = 0;
|
||||
termsFindCacheEpoch = cacheEpoch;
|
||||
}
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const CAPTION_OPENING_BRACKETS = new Set(["(", "(", "[", "[", "{", "{", "「", "『", "【", "〈", "《", "≪", "<", "<"]);
|
||||
function shouldEmitUnparsedRunAsToken(runText) {
|
||||
if (!/[\p{L}\p{N}]/u.test(runText)) { return false; }
|
||||
const firstChar = Array.from(runText.trim())[0];
|
||||
return firstChar !== undefined && !CAPTION_OPENING_BRACKETS.has(firstChar);
|
||||
}
|
||||
function isLookupWorthyCodePoint(codePoint) {
|
||||
if (isCodePointJapanese(codePoint)) { return true; }
|
||||
return /[\p{L}\p{N}]/u.test(String.fromCodePoint(codePoint));
|
||||
}
|
||||
function isKanaOnlyRunText(runText) {
|
||||
const chars = Array.from(runText);
|
||||
return chars.length > 0 && chars.every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
}
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const cacheKey = profileIndex + "\u0000" + substring;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached !== undefined) {
|
||||
termsFindCache.delete(cacheKey);
|
||||
termsFindCache.set(cacheKey, cached);
|
||||
return await cached.promise;
|
||||
}
|
||||
// An in-flight lookup counts as one entry until it resolves; the real
|
||||
// weight replaces that estimate once the result is known.
|
||||
const entry = { promise: null, dictionaryEntryCount: 1 };
|
||||
entry.promise = invoke("termsFind", { text: substring, details, optionsContext: { index: profileIndex } })
|
||||
.then((result) => {
|
||||
const resolvedCount =
|
||||
1 + (Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries.length : 0);
|
||||
const isCached = termsFindCache.get(cacheKey) === entry;
|
||||
if (isCached) {
|
||||
termsFindCacheDictionaryEntries += resolvedCount - entry.dictionaryEntryCount;
|
||||
}
|
||||
entry.dictionaryEntryCount = resolvedCount;
|
||||
// The real weight can push the cache over its budget, and a single
|
||||
// response can exceed it on its own, so re-check here.
|
||||
if (isCached) { evictOverflowingTermsFindEntries(); }
|
||||
return result;
|
||||
});
|
||||
termsFindCache.set(cacheKey, entry);
|
||||
termsFindCacheDictionaryEntries += entry.dictionaryEntryCount;
|
||||
evictOverflowingTermsFindEntries();
|
||||
try {
|
||||
return await entry.promise;
|
||||
} catch (error) {
|
||||
dropCachedTermsFind(cacheKey, entry);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
// Text the walk skips accumulates into unparsed runs, mirroring the
|
||||
// filler chunks the parseText segmentation used to provide: runs stay
|
||||
// hoverable (flagged isUnparsedRun) unless they are punctuation-only or
|
||||
// caption-style asides, and kana continuations of a longer headword
|
||||
// extend the previous token instead.
|
||||
function flushUnparsedRun(runStart, runEnd) {
|
||||
if (runStart === null || runEnd <= runStart) { return; }
|
||||
const runText = text.substring(runStart, runEnd);
|
||||
const previousToken = tokens[tokens.length - 1];
|
||||
if (
|
||||
previousToken &&
|
||||
previousToken.endPos === runStart &&
|
||||
isKanaOnlyRunText(runText) &&
|
||||
typeof previousToken.headword === "string" &&
|
||||
previousToken.headword.length > previousToken.surface.length &&
|
||||
previousToken.headword.startsWith(previousToken.surface + runText)
|
||||
) {
|
||||
previousToken.surface += runText;
|
||||
// The run is kana-only, so its reading is itself: append it or the
|
||||
// reading stops covering the surface, which disables the known-word
|
||||
// reading fallback (isCompleteReadingForSurface) downstream.
|
||||
previousToken.reading += runText;
|
||||
// The run is kana-only, so its reading is itself: append it or the
|
||||
// reading stops covering the surface, which disables the known-word
|
||||
// reading fallback (isCompleteReadingForSurface) downstream.
|
||||
previousToken.endPos = runEnd;
|
||||
return;
|
||||
}
|
||||
if (!shouldEmitUnparsedRunAsToken(runText)) { return; }
|
||||
tokens.push({
|
||||
surface: runText,
|
||||
reading: "",
|
||||
headword: runText,
|
||||
startPos: runStart,
|
||||
endPos: runEnd,
|
||||
isUnparsedRun: true
|
||||
});
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
surface: segments.map((segment) => segment.text).join("") || source,
|
||||
reading: segments.map(getSegmentReadingContribution).join(""),
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
? Math.max(1, Math.floor(preferredHeadword.frequencyRank))
|
||||
: undefined,
|
||||
};
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return tokenPayload;
|
||||
}
|
||||
// findTokenAt plus the shrinking-window ladder below it: Yomitan text
|
||||
// normalization can consume characters (whitespace, punctuation) beyond
|
||||
// the matched term, leaving no headword whose source equals the consumed
|
||||
// text. Retry with shorter windows so a valid prefix term (e.g. a
|
||||
// character name before a paren) still tokenizes instead of the position
|
||||
// being skipped.
|
||||
// Every window at or above the consumed length repeats the same result,
|
||||
// so the next informative window sits just below it. A lookup that
|
||||
// consumed its whole window reports nothing to aim at, and the step down
|
||||
// from it is a blind guess: only those are budgeted.
|
||||
// The window can run past the end of the line, so blindness is judged
|
||||
// against the text the lookup actually saw.
|
||||
// Set when a position stopped short of windows an uncapped ladder would
|
||||
// still have tried; the line then escalates to parseText at the end.
|
||||
let blindRetryBudgetExhausted = false;
|
||||
async function resolveTokenAt(position, windowLength) {
|
||||
let attempt = await findTokenAt(position, windowLength);
|
||||
const scannedLength = Math.min(windowLength, text.length - position);
|
||||
let retryLength = Math.min(attempt.matchedLength, scannedLength) - 1;
|
||||
let stepIsBlind = attempt.matchedLength >= scannedLength;
|
||||
let blindRetriesRemaining = MAX_BLIND_SHRINKING_WINDOW_RETRIES;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
if (stepIsBlind) {
|
||||
if (blindRetriesRemaining <= 0) {
|
||||
blindRetryBudgetExhausted = true;
|
||||
break;
|
||||
}
|
||||
blindRetriesRemaining -= 1;
|
||||
}
|
||||
const retry = await findTokenAt(position, retryLength);
|
||||
if (retry.token) { return retry; }
|
||||
const guidedLength = retry.matchedLength - 1;
|
||||
stepIsBlind = guidedLength >= retryLength - 1;
|
||||
retryLength = Math.min(retryLength - 1, guidedLength);
|
||||
}
|
||||
return attempt;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Halfwidth katakana survives kana normalization unchanged, so a name
|
||||
// written that way would not prefix-match a candidate form. Those
|
||||
// positions bypass the prefilter rather than risk a missed name.
|
||||
function isHalfwidthKatakanaCodePoint(codePoint) {
|
||||
return codePoint >= 0xff66 && codePoint <= 0xff9f;
|
||||
}
|
||||
// Build (once per candidate list) a first-character bucket index of the
|
||||
// normalized name forms, so the pre-pass can reject a position with a
|
||||
// single map hit instead of a backend round trip.
|
||||
if (rawNameCandidates && nameCandidateIndex?.key !== rawNameCandidates.key) {
|
||||
const byFirstChar = new Map();
|
||||
for (const form of rawNameCandidates.forms) {
|
||||
const normalized = typeof form === "string" ? convertKatakanaToHiragana(form.trim()) : "";
|
||||
if (!normalized) { continue; }
|
||||
const bucket = byFirstChar.get(normalized[0]);
|
||||
if (bucket) { bucket.push(normalized); } else { byFirstChar.set(normalized[0], [normalized]); }
|
||||
}
|
||||
nameCandidateIndex = byFirstChar.size > 0 ? { key: rawNameCandidates.key, byFirstChar } : null;
|
||||
} else if (!rawNameCandidates) {
|
||||
nameCandidateIndex = null;
|
||||
}
|
||||
// Only meaningful when the installed list matches the media this scan is
|
||||
// for; otherwise fall back to scanning every position.
|
||||
const activeNameCandidateIndex =
|
||||
nameCandidateKey !== null && nameCandidateIndex?.key === nameCandidateKey
|
||||
? nameCandidateIndex
|
||||
: null;
|
||||
const normalizedText = activeNameCandidateIndex ? convertKatakanaToHiragana(text) : "";
|
||||
// Yomitan collapses emphatic sequences before matching (すっっごーーい →
|
||||
// すごい), so a stretched name still resolves to its entry. Skipping these
|
||||
// characters keeps such spellings candidates; the filter only ever grows
|
||||
// the probe set, so a false positive costs one lookup, never a name.
|
||||
const EMPHATIC_SKIP_CHARS = new Set(["ぁ", "ぃ", "ぅ", "ぇ", "ぉ", "っ", "ゃ", "ゅ", "ょ", "ー"]);
|
||||
function matchesCandidateFormAt(form, position) {
|
||||
let textIndex = position;
|
||||
for (let formIndex = 0; formIndex < form.length; formIndex += 1) {
|
||||
while (
|
||||
textIndex < normalizedText.length &&
|
||||
normalizedText[textIndex] !== form[formIndex] &&
|
||||
EMPHATIC_SKIP_CHARS.has(normalizedText[textIndex])
|
||||
) {
|
||||
textIndex += 1;
|
||||
}
|
||||
if (normalizedText[textIndex] !== form[formIndex]) { return false; }
|
||||
textIndex += 1;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
function couldNameStartAt(position, codePoint) {
|
||||
if (!activeNameCandidateIndex) { return true; }
|
||||
if (isHalfwidthKatakanaCodePoint(codePoint)) { return true; }
|
||||
const bucket = activeNameCandidateIndex.byFirstChar.get(normalizedText[position]);
|
||||
if (!bucket) { return false; }
|
||||
for (const form of bucket) {
|
||||
if (matchesCandidateFormAt(form, position)) { return true; }
|
||||
}
|
||||
return false;
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint) || !couldNameStartAt(namePos, codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, scanLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + scanLength);
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
// First reserved name span that a match ending at endPos would leave
|
||||
// half-consumed. Spans the match covers entirely are not returned: those
|
||||
// lose to the longer word instead of splitting it.
|
||||
function findSplitNameToken(startIndex, endPos) {
|
||||
for (let index = startIndex; index < nameTokens.length; index += 1) {
|
||||
const nameToken = nameTokens[index];
|
||||
if (nameToken.startPos >= endPos) { return null; }
|
||||
if (nameToken.endPos > endPos) { return nameToken; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
let unparsedRunStart = null;
|
||||
while (i < text.length) {
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
flushUnparsedRun(unparsedRunStart, i);
|
||||
unparsedRunStart = null;
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
const codePoint = text.codePointAt(i);
|
||||
// Punctuation and whitespace can never start a token: skip the backend
|
||||
// round trip entirely. Latin letters and digits stay lookup-worthy
|
||||
// (terms like Tシャツ start on an ASCII letter).
|
||||
if (!isLookupWorthyCodePoint(codePoint)) {
|
||||
if (unparsedRunStart === null) { unparsedRunStart = i; }
|
||||
i += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
// A reservation only outranks generic matches that would cut into it.
|
||||
// Look the position up unrestricted first: a generic word that starts
|
||||
// earlier and covers the whole name span (写真 over a character named
|
||||
// 真) is the better reading, so the reservation yields rather than
|
||||
// splitting the word. Only a match that ends inside a name span gets
|
||||
// re-run against a window capped at that span.
|
||||
let attempt = await resolveTokenAt(i, scanLength);
|
||||
if (attempt.token) {
|
||||
const splitNameToken = findSplitNameToken(nameIndex, attempt.token.endPos);
|
||||
if (splitNameToken) {
|
||||
attempt = await resolveTokenAt(i, splitNameToken.startPos - i);
|
||||
}
|
||||
}
|
||||
if (attempt.token) {
|
||||
flushUnparsedRun(unparsedRunStart, i);
|
||||
unparsedRunStart = null;
|
||||
tokens.push(attempt.token);
|
||||
i += attempt.matchedLength;
|
||||
continue;
|
||||
}
|
||||
if (unparsedRunStart === null) { unparsedRunStart = i; }
|
||||
i += String.fromCodePoint(text.codePointAt(i)).length;
|
||||
}
|
||||
flushUnparsedRun(unparsedRunStart, text.length);
|
||||
if (blindRetryBudgetExhausted) {
|
||||
// A position gave up with shorter windows still worth trying. The walk
|
||||
// is the only tokenizer now, so stopping there would leave a real term
|
||||
// as an unparsed run; report it so the host can spend one parseText on
|
||||
// the line instead of letting the ladder run to O(scanLength) lookups.
|
||||
return { tokens, retryBudgetExhausted: true };
|
||||
}
|
||||
return tokens;
|
||||
};
|
||||
return true;
|
||||
})();
|
||||
`;
|
||||
|
||||
// Installs (or clears) the character-name candidate forms for the current
|
||||
// media. Runs only when the list changes, not per line. Passing null restores
|
||||
// the exhaustive every-position pre-pass.
|
||||
export function buildYomitanScanNameCandidatesScript(
|
||||
nameCandidates: { key: string; forms: string[] } | null,
|
||||
): string {
|
||||
if (!nameCandidates) {
|
||||
return `
|
||||
(() => {
|
||||
if (typeof globalThis.__subminerYomitanScanSetNameCandidates !== "function") {
|
||||
return false;
|
||||
}
|
||||
return globalThis.__subminerYomitanScanSetNameCandidates(null, null);
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
return `
|
||||
(() => {
|
||||
if (typeof globalThis.__subminerYomitanScanSetNameCandidates !== "function") {
|
||||
return false;
|
||||
}
|
||||
return globalThis.__subminerYomitanScanSetNameCandidates(
|
||||
${JSON.stringify(nameCandidates.key)},
|
||||
${JSON.stringify(nameCandidates.forms)}
|
||||
);
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
export function buildYomitanScanCallScript(params: YomitanScanRequestParams): string {
|
||||
return `
|
||||
(async () => {
|
||||
if (typeof globalThis.__subminerYomitanScan !== "function") {
|
||||
return ${JSON.stringify(YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL)};
|
||||
}
|
||||
return await globalThis.__subminerYomitanScan(${JSON.stringify(params)});
|
||||
})();
|
||||
`;
|
||||
}
|
||||
@@ -0,0 +1,496 @@
|
||||
// Helper bundle for the in-page Yomitan scan runtime: kana/furigana handling,
|
||||
// headword preference, and frequency-rank resolution. Injected as text into the
|
||||
// parser window by yomitan-scan-runtime-script.ts, so it is data here, not code
|
||||
// this process runs.
|
||||
import { HAN_CODE_POINT_RANGES } from '../../text/han-code-points';
|
||||
|
||||
export const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
|
||||
export const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
||||
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
||||
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
||||
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
||||
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
||||
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
||||
// Han ranges come from the shared table so the scan walk and the character
|
||||
// dictionary agree on what a kanji is (supplementary planes included).
|
||||
// Halfwidth katakana counts as Japanese text: a name written that way has
|
||||
// to reach the greedy pre-pass, which has its own handling for it.
|
||||
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0xff66, 0xff9f], ...${JSON.stringify(HAN_CODE_POINT_RANGES)}];
|
||||
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
||||
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
||||
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
||||
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
||||
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
||||
function getSegmentReadingContribution(segment) {
|
||||
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
||||
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
||||
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
return isKanaOnly ? segmentText : "";
|
||||
}
|
||||
function getProlongedHiragana(previousCharacter) {
|
||||
switch (previousCharacter) {
|
||||
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
||||
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
||||
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
||||
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
||||
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
||||
default: return null;
|
||||
}
|
||||
}
|
||||
function getFuriganaKanaSegments(text, reading) {
|
||||
const newSegments = [];
|
||||
let start = 0;
|
||||
let state = (reading[0] === text[0]);
|
||||
for (let i = 1; i < text.length; ++i) {
|
||||
const newState = (reading[i] === text[i]);
|
||||
if (state === newState) { continue; }
|
||||
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
||||
state = newState;
|
||||
start = i;
|
||||
}
|
||||
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
||||
return newSegments;
|
||||
}
|
||||
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
||||
let result = '';
|
||||
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
||||
for (let char of text) {
|
||||
const codePoint = char.codePointAt(0);
|
||||
switch (codePoint) {
|
||||
case KATAKANA_SMALL_KA_CODE_POINT:
|
||||
case KATAKANA_SMALL_KE_CODE_POINT:
|
||||
break;
|
||||
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
||||
if (!keepProlongedSoundMarks && result.length > 0) {
|
||||
const char2 = getProlongedHiragana(result[result.length - 1]);
|
||||
if (char2 !== null) { char = char2; }
|
||||
}
|
||||
break;
|
||||
default:
|
||||
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
||||
char = String.fromCodePoint(codePoint + offset);
|
||||
}
|
||||
break;
|
||||
}
|
||||
result += char;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
||||
const groupCount = groups.length - groupsStart;
|
||||
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
||||
const group = groups[groupsStart];
|
||||
const {isKana, text} = group;
|
||||
if (isKana) {
|
||||
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
||||
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
||||
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
||||
return segments;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let result = null;
|
||||
for (let i = reading.length; i >= text.length; --i) {
|
||||
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (result !== null) { return null; }
|
||||
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
||||
result = segments;
|
||||
}
|
||||
if (groupCount === 1) { break; }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function distributeFurigana(term, reading) {
|
||||
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
||||
const groups = [];
|
||||
let groupPre = null;
|
||||
let isKanaPre = null;
|
||||
for (const c of term) {
|
||||
const isKana = isCodePointKana(c.codePointAt(0));
|
||||
if (isKana === isKanaPre) { groupPre.text += c; }
|
||||
else {
|
||||
groupPre = {isKana, text: c, textNormalized: null};
|
||||
groups.push(groupPre);
|
||||
isKanaPre = isKana;
|
||||
}
|
||||
}
|
||||
for (const group of groups) {
|
||||
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
||||
}
|
||||
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
||||
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
||||
}
|
||||
function getStemLength(text1, text2) {
|
||||
const minLength = Math.min(text1.length, text2.length);
|
||||
if (minLength === 0) { return 0; }
|
||||
let i = 0;
|
||||
while (true) {
|
||||
const char1 = text1.codePointAt(i);
|
||||
const char2 = text2.codePointAt(i);
|
||||
if (char1 !== char2) { break; }
|
||||
const charLength = String.fromCodePoint(char1).length;
|
||||
i += charLength;
|
||||
if (i >= minLength) {
|
||||
if (i > minLength) { i -= charLength; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
return i;
|
||||
}
|
||||
function distributeFuriganaInflected(term, reading, source) {
|
||||
const termNormalized = convertKatakanaToHiragana(term);
|
||||
const readingNormalized = convertKatakanaToHiragana(reading);
|
||||
const sourceNormalized = convertKatakanaToHiragana(source);
|
||||
let mainText = term;
|
||||
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
||||
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
||||
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
||||
mainText = reading;
|
||||
stemLength = readingStemLength;
|
||||
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
||||
}
|
||||
const segments = [];
|
||||
if (stemLength > 0) {
|
||||
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
||||
const segments2 = distributeFurigana(mainText, reading);
|
||||
let consumed = 0;
|
||||
for (const segment of segments2) {
|
||||
const start = consumed;
|
||||
consumed += segment.text.length;
|
||||
if (consumed < stemLength) { segments.push(segment); }
|
||||
else if (consumed === stemLength) { segments.push(segment); break; }
|
||||
else {
|
||||
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (stemLength < source.length) {
|
||||
const remainder = source.substring(stemLength);
|
||||
const last = segments[segments.length - 1];
|
||||
if (last && last.reading.length === 0) { last.text += remainder; }
|
||||
else { segments.push(createFuriganaSegment(remainder, '')); }
|
||||
}
|
||||
return segments;
|
||||
}
|
||||
function parsePositiveFrequencyNumber(value) {
|
||||
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||
return Math.max(1, Math.floor(value));
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
||||
if (!numericMatch) { return null; }
|
||||
const parsed = Number.parseFloat(numericMatch);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
||||
return Math.max(1, Math.floor(parsed));
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) {
|
||||
const parsed = parsePositiveFrequencyNumber(item);
|
||||
if (parsed !== null) { return parsed; }
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseDisplayFrequencyNumber(value) {
|
||||
if (typeof value === 'string') {
|
||||
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
||||
if (!leadingDigits) { return null; }
|
||||
const parsed = Number.parseInt(leadingDigits, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
||||
}
|
||||
return parsePositiveFrequencyNumber(value);
|
||||
}
|
||||
function getFrequencyDictionaryName(frequency) {
|
||||
const candidates = [
|
||||
frequency?.dictionary,
|
||||
frequency?.dictionaryName,
|
||||
frequency?.name,
|
||||
frequency?.title,
|
||||
frequency?.dictionaryTitle,
|
||||
frequency?.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
return candidate.trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
||||
for (const frequency of dictionaryEntry?.frequencies || []) {
|
||||
if (!frequency || typeof frequency !== 'object') { continue; }
|
||||
const frequencyHeadwordIndex = frequency.headwordIndex;
|
||||
if (typeof frequencyHeadwordIndex === 'number') {
|
||||
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
||||
} else if (headwordCount > 1) {
|
||||
continue;
|
||||
}
|
||||
const dictionary = getFrequencyDictionaryName(frequency);
|
||||
if (!dictionary) { continue; }
|
||||
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
||||
const rank =
|
||||
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
||||
parsePositiveFrequencyNumber(frequency.frequency);
|
||||
if (rank === null) { continue; }
|
||||
const priorityRaw = dictionaryPriorityByName[dictionary];
|
||||
const fallbackPriority =
|
||||
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
||||
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
||||
: Number.MAX_SAFE_INTEGER;
|
||||
const priority =
|
||||
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
||||
? Math.max(0, Math.floor(priorityRaw))
|
||||
: fallbackPriority;
|
||||
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
||||
best = { priority, rank };
|
||||
}
|
||||
}
|
||||
return best?.rank ?? null;
|
||||
}
|
||||
function hasExactSource(headword, token, requirePrimary) {
|
||||
for (const src of headword.sources || []) {
|
||||
if (src.originalText !== token) { continue; }
|
||||
if (requirePrimary && !src.isPrimary) { continue; }
|
||||
if (src.matchType !== 'exact') { continue; }
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
||||
const matches = [];
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
||||
matches.push({ dictionaryEntry, headword, headwordIndex });
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
function sameHeadword(match, preferredMatch) {
|
||||
if (!match || !preferredMatch) {
|
||||
return false;
|
||||
}
|
||||
if (match.headword?.term !== preferredMatch.headword?.term) {
|
||||
return false;
|
||||
}
|
||||
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
||||
const preferredReading =
|
||||
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
||||
if (!matchReading || !preferredReading) {
|
||||
return true;
|
||||
}
|
||||
return matchReading === preferredReading;
|
||||
}
|
||||
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
for (const match of matches) {
|
||||
const rank = getBestFrequencyRank(
|
||||
match.dictionaryEntry,
|
||||
match.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (rank === null) { continue; }
|
||||
if (best === null || rank < best) {
|
||||
best = rank;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
}
|
||||
function appendDictionaryNames(target, value) {
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
const candidates = [
|
||||
value.dictionary,
|
||||
value.dictionaryName,
|
||||
value.name,
|
||||
value.title,
|
||||
value.dictionaryTitle,
|
||||
value.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
target.push(candidate.trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
function getDictionaryEntryNames(entry) {
|
||||
const names = [];
|
||||
appendDictionaryNames(names, entry);
|
||||
for (const definition of entry?.definitions || []) {
|
||||
appendDictionaryNames(names, definition);
|
||||
}
|
||||
for (const frequency of entry?.frequencies || []) {
|
||||
appendDictionaryNames(names, frequency);
|
||||
}
|
||||
for (const pronunciation of entry?.pronunciations || []) {
|
||||
appendDictionaryNames(names, pronunciation);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
function isNameDictionaryEntry(entry) {
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
if (imageMatch) {
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseSubMinerMediaIdCandidate(value) {
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
||||
const parsed = Number.parseInt(value.trim(), 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function collectSubMinerMediaIds(value, target) {
|
||||
if (typeof value === 'string') {
|
||||
const parsed = parseSubMinerMediaIdFromString(value);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
||||
return;
|
||||
}
|
||||
const mediaIdCandidates = [
|
||||
value.subminerMediaId,
|
||||
value.subMinerMediaId,
|
||||
value.characterDictionaryMediaId,
|
||||
value.data?.subminerMediaId,
|
||||
value.data?.subMinerMediaId,
|
||||
value.data?.characterDictionaryMediaId
|
||||
];
|
||||
for (const candidate of mediaIdCandidates) {
|
||||
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
collectSubMinerMediaIds(child, target);
|
||||
}
|
||||
}
|
||||
function getSubMinerMediaIds(entry) {
|
||||
const mediaIds = new Set();
|
||||
collectSubMinerMediaIds(entry, mediaIds);
|
||||
return mediaIds;
|
||||
}
|
||||
function isCurrentMediaNameDictionaryEntry(entry) {
|
||||
if (!isNameDictionaryEntry(entry)) {
|
||||
return false;
|
||||
}
|
||||
if (currentCharacterDictionaryMediaId === null) {
|
||||
return true;
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
: (dictionaryEntries || []).filter((entry) => {
|
||||
if (!isNameDictionaryEntry(entry)) { return true; }
|
||||
return isCurrentMediaNameDictionaryEntry(entry);
|
||||
});
|
||||
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
||||
let matchedNameDictionary = false;
|
||||
if (includeNameMatchMetadata) {
|
||||
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
for (const match of exactPrimaryMatches) {
|
||||
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
||||
matchedNameDictionary = true;
|
||||
break;
|
||||
}
|
||||
if (matchedNameDictionary) { break; }
|
||||
}
|
||||
}
|
||||
const preferredMatch = exactPrimaryMatches[0];
|
||||
if (preferredMatch) {
|
||||
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
||||
.filter((match) => sameHeadword(match, preferredMatch));
|
||||
return {
|
||||
term: preferredMatch.headword.term,
|
||||
reading: preferredMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
||||
isNameMatch:
|
||||
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
||||
frequencyRank: getBestFrequencyRankForMatches(
|
||||
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
`;
|
||||
@@ -0,0 +1,54 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { HAN_CODE_POINT_RANGES, HAN_REGEXP_CLASS_BODY, isHanCodePoint } from './han-code-points';
|
||||
|
||||
test('every range boundary is inside the table', () => {
|
||||
for (const [start, end] of HAN_CODE_POINT_RANGES) {
|
||||
for (const codePoint of [start, end]) {
|
||||
assert.ok(isHanCodePoint(codePoint), `expected U+${codePoint.toString(16)} to be Han`);
|
||||
}
|
||||
}
|
||||
|
||||
// Extension J (Unicode 17) and the Compatibility blocks are the ones a
|
||||
// BMP-only table used to miss.
|
||||
assert.ok(isHanCodePoint(0x323b0));
|
||||
assert.ok(isHanCodePoint(0x33479));
|
||||
assert.ok(isHanCodePoint(0xf900));
|
||||
assert.ok(isHanCodePoint(0x2f800));
|
||||
});
|
||||
|
||||
test('no unified ideograph the runtime knows about falls outside the table', () => {
|
||||
// One direction only: a runtime with older Unicode data simply checks fewer
|
||||
// code points, where asserting the reverse would fail on Extension J.
|
||||
const unifiedIdeograph = /\p{Unified_Ideograph}/u;
|
||||
|
||||
for (let codePoint = 0x3000; codePoint <= 0x40000; codePoint += 1) {
|
||||
if (unifiedIdeograph.test(String.fromCodePoint(codePoint))) {
|
||||
assert.ok(
|
||||
isHanCodePoint(codePoint),
|
||||
`expected unified ideograph U+${codePoint.toString(16)} to be in the table`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('code points just outside the table are rejected', () => {
|
||||
for (const codePoint of [0x33ff, 0x4dc0, 0xa000, 0x1f000, 0x3347a]) {
|
||||
assert.equal(
|
||||
isHanCodePoint(codePoint),
|
||||
false,
|
||||
`expected U+${codePoint.toString(16)} not to be Han`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('the regexp class body matches the same code points as the predicate', () => {
|
||||
const classRegExp = new RegExp(`^[${HAN_REGEXP_CLASS_BODY}]$`, 'u');
|
||||
|
||||
for (const codePoint of [0x3400, 0x4e00, 0x9fff, 0xf900, 0x20000, 0x323b0, 0x33479]) {
|
||||
assert.match(String.fromCodePoint(codePoint), classRegExp);
|
||||
}
|
||||
for (const codePoint of [0x3040, 0x30ff, 0x33fa, 0x3347a]) {
|
||||
assert.doesNotMatch(String.fromCodePoint(codePoint), classRegExp);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,31 @@
|
||||
// Single source of truth for "this code point is a Han character", shared by
|
||||
// the main-process character dictionary and the in-page Yomitan scan runtime.
|
||||
// The two used to carry separate range lists, and they drifted: a name written
|
||||
// with a supplementary-plane kanji could enter the generated dictionary while
|
||||
// the scanner's greedy name pre-pass refused to probe the position.
|
||||
//
|
||||
// Ranges rather than \p{Script=Han}: the scan walk tests one code point per
|
||||
// character of every subtitle line, where an integer compare beats building a
|
||||
// string for a regex, and the script is injected as text into a page where a
|
||||
// shared helper cannot be imported.
|
||||
export const HAN_CODE_POINT_RANGES: ReadonlyArray<readonly [number, number]> = [
|
||||
[0x3400, 0x4dbf], // Extension A
|
||||
[0x4e00, 0x9fff], // CJK Unified Ideographs
|
||||
[0xf900, 0xfaff], // Compatibility Ideographs
|
||||
[0x20000, 0x2a6df], // Extension B
|
||||
[0x2a700, 0x2ebef], // Extensions C-F
|
||||
[0x2ebf0, 0x2ee5f], // Extension I
|
||||
[0x2f800, 0x2fa1f], // Compatibility Ideographs Supplement
|
||||
[0x30000, 0x3134f], // Extension G
|
||||
[0x31350, 0x323af], // Extension H
|
||||
[0x323b0, 0x33479], // Extension J (Unicode 17)
|
||||
];
|
||||
|
||||
export function isHanCodePoint(codePoint: number): boolean {
|
||||
return HAN_CODE_POINT_RANGES.some(([start, end]) => codePoint >= start && codePoint <= end);
|
||||
}
|
||||
|
||||
/** The same ranges as a regular expression character class body (needs the `u` flag). */
|
||||
export const HAN_REGEXP_CLASS_BODY = HAN_CODE_POINT_RANGES.map(
|
||||
([start, end]) => `\\u{${start.toString(16)}}-\\u{${end.toString(16)}}`,
|
||||
).join('');
|
||||
+44
-9
@@ -487,6 +487,7 @@ import { createOverlayVisibilityRuntimeService } from './main/overlay-visibility
|
||||
import { createDiscordPresenceRuntime } from './main/runtime/discord-presence-runtime';
|
||||
import { createCharacterDictionaryRuntimeService } from './main/character-dictionary-runtime';
|
||||
import { createCharacterDictionaryImageLookup } from './main/character-dictionary-runtime/image-lookup';
|
||||
import { createCharacterNameCandidateLookup } from './main/character-dictionary-runtime/name-candidates';
|
||||
import {
|
||||
createCharacterDictionaryAutoSyncRuntimeService,
|
||||
getCharacterDictionaryManagerSnapshot,
|
||||
@@ -1816,7 +1817,7 @@ function withCurrentSubtitleTiming(payload: SubtitleData): SubtitleData {
|
||||
endTime: appState.mpvClient?.currentSubEnd ?? null,
|
||||
};
|
||||
}
|
||||
function emitSubtitlePayload(payload: SubtitleData): void {
|
||||
function emitSubtitlePayload(payload: SubtitleData, options?: { resumePrefetch?: boolean }): void {
|
||||
const timedPayload = withCurrentSubtitleTiming(payload);
|
||||
const currentSubtitleData = appState.currentSubtitleData;
|
||||
const isAnnotationUpgrade = isSubtitleAnnotationUpgrade(currentSubtitleData, timedPayload);
|
||||
@@ -1833,7 +1834,13 @@ function emitSubtitlePayload(payload: SubtitleData): void {
|
||||
}
|
||||
annotationSubtitleWsService.broadcast(timedPayload, frequencyOptions);
|
||||
autoplayReadyGate.maybeSignalPluginAutoplayReady(timedPayload, { forceWhilePaused: true });
|
||||
// resumePrefetch: false marks an emit that is not the end of the work for
|
||||
// this line; prefetch stays paused until the subtitle processing controller
|
||||
// settles so it does not compete with the on-screen line for the single
|
||||
// Yomitan parser window.
|
||||
if (options?.resumePrefetch !== false) {
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
}
|
||||
function getCurrentAutoplaySubtitlePayload(): SubtitleData | null {
|
||||
const payload = appState.currentSubtitleData;
|
||||
@@ -1889,7 +1896,17 @@ const buildSubtitleProcessingControllerMainDepsHandler =
|
||||
createBuildSubtitleProcessingControllerMainDepsHandler({
|
||||
tokenizeSubtitle: async (text: string) =>
|
||||
tokenizeSubtitleDeferred ? await tokenizeSubtitleDeferred(text) : { text, tokens: null },
|
||||
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
||||
// Controller emits never release the prefetch pause: the first emit for an
|
||||
// uncached line is the provisional plain payload, sent before tokenization
|
||||
// starts, so resuming on it would put prefetch back in contention with the
|
||||
// on-screen line for the single parser window.
|
||||
emitSubtitle: (payload) => emitSubtitlePayload(payload, { resumePrefetch: false }),
|
||||
// The pause is released once the controller has no work left, which covers
|
||||
// the runs that end without an emit (suppressed duplicate, failed
|
||||
// tokenization) as well as the ones that deliver a payload.
|
||||
onProcessingSettled: () => {
|
||||
subtitlePrefetchService?.resume();
|
||||
},
|
||||
logDebug: (message) => {
|
||||
logger.debug(`[subtitle-processing] ${message}`);
|
||||
},
|
||||
@@ -1926,7 +1943,7 @@ const autoplaySubtitlePrimingRuntime = createAutoplaySubtitlePrimingRuntime({
|
||||
appState.activeParsedSubtitleMediaPath = mediaPath;
|
||||
},
|
||||
subtitleProcessingController,
|
||||
emitSubtitlePayload: (payload) => emitSubtitlePayload(payload),
|
||||
emitSubtitlePayload: (payload, options) => emitSubtitlePayload(payload, options),
|
||||
getSubtitlePrefetchService: () => subtitlePrefetchService,
|
||||
getLastObservedTimePos: () => lastObservedTimePos,
|
||||
getVisibleOverlayVisible: () => overlayManager.getVisibleOverlayVisible(),
|
||||
@@ -2532,6 +2549,10 @@ const characterDictionaryAutoSyncRuntime = createCharacterDictionaryAutoSyncRunt
|
||||
},
|
||||
{
|
||||
hasParserWindow: () => Boolean(appState.yomitanParserWindow),
|
||||
invalidateCharacterDictionaryLookups: () => {
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
characterNameCandidateLookup.invalidate();
|
||||
},
|
||||
clearParserCaches: () => {
|
||||
if (appState.yomitanParserWindow) {
|
||||
clearYomitanParserCachesForWindow(appState.yomitanParserWindow);
|
||||
@@ -2557,6 +2578,13 @@ const characterDictionaryImageLookup = createCharacterDictionaryImageLookup({
|
||||
getCurrentMediaId: () => characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
});
|
||||
|
||||
// Lets the Yomitan scan runtime skip name lookups at positions where no
|
||||
// character name can start; absent candidates just mean the exhaustive scan.
|
||||
const characterNameCandidateLookup = createCharacterNameCandidateLookup({
|
||||
userDataPath: USER_DATA_PATH,
|
||||
getCurrentMediaId: () => characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
});
|
||||
|
||||
const overlayVisibilityRuntime = createOverlayVisibilityRuntimeService(
|
||||
createBuildOverlayVisibilityRuntimeMainDepsHandler({
|
||||
getMainWindow: () => overlayManager.getMainWindow(),
|
||||
@@ -3971,7 +3999,10 @@ const refreshCurrentSubtitleAfterKnownWordUpdate = (): void => {
|
||||
}
|
||||
subtitleProcessingController.invalidateTokenizationCache();
|
||||
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
||||
subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);
|
||||
if (!subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)) {
|
||||
// Idle controller: no settle is coming to release the pause above.
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
};
|
||||
let hasAttemptedImmersionTrackerStartup = false;
|
||||
const ensureImmersionTrackerStarted = (): void => {
|
||||
@@ -4372,9 +4403,15 @@ const {
|
||||
emitSubtitlePayload(payload);
|
||||
},
|
||||
onSubtitleChange: (text) => {
|
||||
// Pause only; restarting the prefetch run here would discard in-flight
|
||||
// tokenization work on every line. Real seeks restart via onTimePosUpdate.
|
||||
subtitlePrefetchService?.pause();
|
||||
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
if (!subtitleProcessingController.onSubtitleChange(text)) {
|
||||
// Repeat of the current text: the controller is idle, so no settle is
|
||||
// coming to release the pause. Resume now instead of idling prefetch
|
||||
// for the rest of the cue.
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
},
|
||||
refreshDiscordPresence: () => {
|
||||
discordPresenceRuntime.publishDiscordPresence();
|
||||
@@ -4618,6 +4655,7 @@ const {
|
||||
getCharacterNameImage: (term) => characterDictionaryImageLookup.get(term),
|
||||
getCurrentCharacterDictionaryMediaId: () =>
|
||||
characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
getCharacterNameCandidates: () => characterNameCandidateLookup.get(),
|
||||
getFrequencyDictionaryEnabled: () =>
|
||||
getRuntimeBooleanOption(
|
||||
'subtitle.annotation.frequency',
|
||||
@@ -5672,7 +5710,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager override:', error);
|
||||
}
|
||||
@@ -5703,7 +5740,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager removal:', error);
|
||||
}
|
||||
@@ -5720,7 +5756,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager reorder:', error);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export const ANILIST_GRAPHQL_URL = 'https://graphql.anilist.co';
|
||||
export const ANILIST_REQUEST_DELAY_MS = 2000;
|
||||
export const CHARACTER_IMAGE_DOWNLOAD_DELAY_MS = 250;
|
||||
export const CHARACTER_DICTIONARY_FORMAT_VERSION = 19;
|
||||
export const CHARACTER_DICTIONARY_FORMAT_VERSION = 20;
|
||||
export const CHARACTER_DICTIONARY_MERGED_TITLE = 'SubMiner Character Dictionary';
|
||||
|
||||
export const HONORIFIC_SUFFIXES = [
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import test from 'node:test';
|
||||
import { CHARACTER_DICTIONARY_FORMAT_VERSION } from './constants';
|
||||
import { createCharacterNameCandidateLookup } from './name-candidates';
|
||||
|
||||
function writeSnapshot(outputDir: string, mediaId: number, entries: Array<[string, string]>): void {
|
||||
const snapshotsDir = path.join(outputDir, 'snapshots');
|
||||
fs.mkdirSync(snapshotsDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(snapshotsDir, `anilist-${mediaId}.json`),
|
||||
JSON.stringify({
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
mediaId,
|
||||
mediaTitle: `title-${mediaId}`,
|
||||
entryCount: entries.length,
|
||||
updatedAt: 1,
|
||||
termEntries: entries.map(([term, reading]) => [
|
||||
term,
|
||||
reading,
|
||||
'name main',
|
||||
'',
|
||||
100,
|
||||
[],
|
||||
0,
|
||||
'',
|
||||
]),
|
||||
images: [],
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function withTempDir<T>(run: (dir: string) => T): T {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-name-candidates-'));
|
||||
try {
|
||||
return run(dir);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
test('collects terms and readings for the current media', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['湊', 'みなと'],
|
||||
]);
|
||||
writeSnapshot(dir, 2, [['カズマ', 'かずま']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const candidates = lookup.get();
|
||||
|
||||
assert.ok(candidates);
|
||||
assert.deepEqual([...candidates.forms].sort(), ['みなと', 'ミナト', '湊'].sort());
|
||||
// Deduplicated: both entries share the みなと reading.
|
||||
assert.equal(candidates.forms.length, 3);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null without a media scope so the scanner stays exhaustive', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => null,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null for a media with no cached snapshot', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 999,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('key changes when the snapshot content changes', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const first = lookup.get();
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
lookup.invalidate();
|
||||
const second = lookup.get();
|
||||
|
||||
assert.ok(first && second);
|
||||
assert.notEqual(first.key, second.key);
|
||||
assert.equal(second.forms.length, 4);
|
||||
});
|
||||
});
|
||||
|
||||
// The lookup runs once per subtitle line, so it must not stat the snapshot
|
||||
// directory every call. Asserted behaviorally: an unannounced on-disk change is
|
||||
// invisible until the recheck interval elapses, which can only be true if the
|
||||
// filesystem is not consulted per lookup.
|
||||
test('does not re-read the snapshot directory on every lookup', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
|
||||
nowMs += 1000;
|
||||
assert.equal(lookup.get()?.forms.length, 2, 'expected the cached list within the interval');
|
||||
|
||||
nowMs += 10_000;
|
||||
assert.equal(lookup.get()?.forms.length, 4, 'expected a refresh past the interval');
|
||||
});
|
||||
});
|
||||
|
||||
test('invalidate picks up a snapshot change immediately', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
nowMs += 1;
|
||||
lookup.invalidate();
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 4);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,159 @@
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { readCachedSnapshots } from './cache';
|
||||
import type { CharacterDictionarySnapshot } from './types';
|
||||
|
||||
// Candidate name forms for the greedy name pre-pass in the Yomitan scan
|
||||
// runtime. The scanner otherwise has to ask the backend at every Japanese
|
||||
// position, because a character name can start mid-token; knowing which forms
|
||||
// exist lets it look up only where a name can actually begin.
|
||||
//
|
||||
// A form is any string Yomitan could match a character entry by: the term and
|
||||
// its reading. Both come from the dictionary SubMiner generated, so the pair is
|
||||
// the complete matchable set for an entry. Callers treat a missing list as
|
||||
// "scan every position", so a stale or absent snapshot costs speed, never a
|
||||
// missed name.
|
||||
|
||||
function getSnapshotsDir(outputDir: string): string {
|
||||
return path.join(outputDir, 'snapshots');
|
||||
}
|
||||
|
||||
function collectSnapshotNameForms(snapshot: CharacterDictionarySnapshot): string[] {
|
||||
const forms = new Set<string>();
|
||||
for (const entry of snapshot.termEntries) {
|
||||
const term = typeof entry[0] === 'string' ? entry[0].trim() : '';
|
||||
if (term) {
|
||||
forms.add(term);
|
||||
}
|
||||
const reading = typeof entry[1] === 'string' ? entry[1].trim() : '';
|
||||
if (reading) {
|
||||
forms.add(reading);
|
||||
}
|
||||
}
|
||||
return [...forms];
|
||||
}
|
||||
|
||||
// The signature grows with the size of the dictionary library, and it rides
|
||||
// along in every per-line scan call, so it is folded into a fixed-width digest
|
||||
// first. Collisions only matter against the immediately previous signature (the
|
||||
// runtime compares keys for equality), and FNV-1a over the file list is far
|
||||
// beyond what that needs.
|
||||
function digestSnapshotDirectorySignature(signature: string): string {
|
||||
let hash = 0x811c9dc5;
|
||||
for (let index = 0; index < signature.length; index += 1) {
|
||||
hash ^= signature.charCodeAt(index);
|
||||
hash = Math.imul(hash, 0x01000193);
|
||||
}
|
||||
return (hash >>> 0).toString(36);
|
||||
}
|
||||
|
||||
function getSnapshotDirectorySignature(outputDir: string): string {
|
||||
let entries: fs.Dirent[] = [];
|
||||
try {
|
||||
entries = fs.readdirSync(getSnapshotsDir(outputDir), { withFileTypes: true });
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
|
||||
const parts: string[] = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isFile() || !/^anilist-\d+\.json$/.test(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const stat = fs.statSync(path.join(getSnapshotsDir(outputDir), entry.name));
|
||||
parts.push(`${entry.name}:${stat.mtimeMs}:${stat.size}`);
|
||||
} catch {
|
||||
// Ignore files that disappear during a refresh; the next lookup rebuilds.
|
||||
}
|
||||
}
|
||||
return parts.sort().join('|');
|
||||
}
|
||||
|
||||
export interface CharacterNameCandidateSet {
|
||||
/** Identifies this exact form list, so the scan runtime can cache it. */
|
||||
key: string;
|
||||
forms: string[];
|
||||
}
|
||||
|
||||
// This lookup is consulted once per subtitle line, so it must not stat the
|
||||
// snapshot directory every time. Dictionary writes are rare and always call
|
||||
// invalidate(), which forces the next lookup to re-read; the interval only
|
||||
// bounds staleness from changes made behind our back.
|
||||
const SNAPSHOT_SIGNATURE_RECHECK_INTERVAL_MS = 5000;
|
||||
|
||||
export function createCharacterNameCandidateLookup(deps: {
|
||||
userDataPath?: string;
|
||||
outputDir?: string;
|
||||
getCurrentMediaId?: () => number | null | undefined;
|
||||
now?: () => number;
|
||||
}): {
|
||||
get: (mediaId?: number | null) => CharacterNameCandidateSet | null;
|
||||
invalidate: () => void;
|
||||
} {
|
||||
const outputDir =
|
||||
deps.outputDir ??
|
||||
(deps.userDataPath ? path.join(deps.userDataPath, 'character-dictionaries') : '');
|
||||
const now = deps.now ?? (() => Date.now());
|
||||
let signature: string | null = null;
|
||||
let lastSignatureCheckAtMs = 0;
|
||||
let formsByMediaId = new Map<number, string[]>();
|
||||
|
||||
function refreshIfNeeded(): void {
|
||||
if (!outputDir) {
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
signature = '';
|
||||
return;
|
||||
}
|
||||
const nowMs = now();
|
||||
if (
|
||||
signature !== null &&
|
||||
nowMs - lastSignatureCheckAtMs < SNAPSHOT_SIGNATURE_RECHECK_INTERVAL_MS
|
||||
) {
|
||||
return;
|
||||
}
|
||||
lastSignatureCheckAtMs = nowMs;
|
||||
const nextSignature = getSnapshotDirectorySignature(outputDir);
|
||||
if (nextSignature === signature) {
|
||||
return;
|
||||
}
|
||||
signature = nextSignature;
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
for (const snapshot of readCachedSnapshots(outputDir)) {
|
||||
const forms = collectSnapshotNameForms(snapshot);
|
||||
if (forms.length > 0) {
|
||||
formsByMediaId.set(snapshot.mediaId, forms);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
get(mediaId?: number | null): CharacterNameCandidateSet | null {
|
||||
refreshIfNeeded();
|
||||
const rawMediaId = mediaId ?? deps.getCurrentMediaId?.() ?? null;
|
||||
const normalizedMediaId =
|
||||
typeof rawMediaId === 'number' && Number.isFinite(rawMediaId) && rawMediaId > 0
|
||||
? Math.floor(rawMediaId)
|
||||
: null;
|
||||
|
||||
// Without a media scope the pre-pass would need every character of every
|
||||
// cached title, which is both slow to match and pointless: report no
|
||||
// candidates so the scanner keeps its exhaustive behavior.
|
||||
if (normalizedMediaId === null) {
|
||||
return null;
|
||||
}
|
||||
const forms = formsByMediaId.get(normalizedMediaId);
|
||||
if (!forms || forms.length === 0) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
key: `${digestSnapshotDirectorySignature(signature ?? '')}:${normalizedMediaId}`,
|
||||
forms,
|
||||
};
|
||||
},
|
||||
invalidate(): void {
|
||||
signature = null;
|
||||
lastSignatureCheckAtMs = 0;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isHanCodePoint } from '../../core/text/han-code-points';
|
||||
import { HONORIFIC_SUFFIXES } from './constants';
|
||||
import type { JapaneseNameParts, NameReadings, ResolvedNameSplits } from './types';
|
||||
|
||||
@@ -26,10 +27,12 @@ export function buildReading(term: string): string {
|
||||
return katakanaToHiragana(compact);
|
||||
}
|
||||
|
||||
// Code points, not code units: a supplementary-plane kanji (𠮷, U+20BB7) is a
|
||||
// surrogate pair, and reading only the high surrogate would classify a real
|
||||
// single-character name as non-kanji and drop it.
|
||||
export function containsKanji(value: string): boolean {
|
||||
for (const char of value) {
|
||||
const code = char.charCodeAt(0);
|
||||
if ((code >= 0x4e00 && code <= 0x9fff) || (code >= 0x3400 && code <= 0x4dbf)) {
|
||||
if (isHanCodePoint(char.codePointAt(0) ?? 0)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,3 +36,75 @@ test('buildNameTerms adds surname honorifics from Japanese localized aliases', (
|
||||
assert.ok(terms.includes('馬渕さん'));
|
||||
assert.ok(!terms.includes('송치'));
|
||||
});
|
||||
|
||||
test('buildNameTerms drops the disambiguator letter of a mob character name', () => {
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: '',
|
||||
lastNameHint: '',
|
||||
fullName: 'Joshi A',
|
||||
nativeName: '女子A',
|
||||
}),
|
||||
);
|
||||
|
||||
// ア would match every あ〜 in the subtitles; the letter is a disambiguator
|
||||
// (Girl A / Girl B), not a name.
|
||||
assert.ok(!terms.includes('ア'));
|
||||
assert.ok(!terms.includes('アさん'));
|
||||
assert.ok(terms.includes('女子A'));
|
||||
assert.ok(terms.includes('ジョシア'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a character whose whole name is one kana', () => {
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: '',
|
||||
lastNameHint: '',
|
||||
fullName: 'A',
|
||||
nativeName: 'あ',
|
||||
}),
|
||||
);
|
||||
|
||||
// The mob-disambiguator filter targets letters split off a longer name; an
|
||||
// explicit one-character name is the character's actual name.
|
||||
assert.ok(terms.includes('あ'));
|
||||
assert.ok(terms.includes('あさん'));
|
||||
// The romanized "A" is still a label, so it contributes neither itself nor
|
||||
// its single-kana alias.
|
||||
assert.ok(!terms.includes('A'));
|
||||
assert.ok(!terms.includes('ア'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a single-kanji name part', () => {
|
||||
// The name is an alias, not the native name, so the parts come from the
|
||||
// space split rather than from the native-name split.
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: 'Sora',
|
||||
lastNameHint: 'Yamada',
|
||||
fullName: 'Sora Yamada',
|
||||
nativeName: '',
|
||||
alternativeNames: ['山田 空'],
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(terms.includes('山田'));
|
||||
assert.ok(terms.includes('空'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a single supplementary-plane kanji name part', () => {
|
||||
// 𠮷 (U+20BB7) is a surrogate pair: a code-unit kanji check reads only the
|
||||
// high surrogate and drops the part as if it were a mob disambiguator.
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: 'Tsukasa',
|
||||
lastNameHint: 'Yoshi',
|
||||
fullName: 'Tsukasa Yoshi',
|
||||
nativeName: '',
|
||||
alternativeNames: ['𠮷 司'],
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(terms.includes('𠮷'));
|
||||
assert.ok(terms.includes('司'));
|
||||
});
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { HAN_REGEXP_CLASS_BODY } from '../../core/text/han-code-points';
|
||||
import { HONORIFIC_SUFFIXES } from './constants';
|
||||
import {
|
||||
addRomanizedKanaAliases,
|
||||
@@ -42,11 +43,34 @@ export function expandRawNameVariants(rawName: string): string[] {
|
||||
return [...variants];
|
||||
}
|
||||
|
||||
// Kana, halfwidth included: one of these can stand alone as a name, where a
|
||||
// latin letter or a digit cannot.
|
||||
const SINGLE_KANA_CHARACTER = /^[\u3040-\u30ff\u31f0-\u31ff\uff66-\uff9f]$/u;
|
||||
|
||||
// AniList disambiguates unnamed mob characters with a trailing letter (女子A /
|
||||
// "Joshi A"), and a lone letter romanizes into a single-kana alias (A → ア)
|
||||
// that collides with interjections (あ〜 matching ア). That letter is a label,
|
||||
// not a name, so it is dropped where a name splits into it and before it can
|
||||
// become a kana alias. A name that is genuinely one character, a character
|
||||
// actually called あ or a single kanji, is a real lookup target and is kept.
|
||||
function isNameDisambiguatorLetter(name: string): boolean {
|
||||
return [...name].length === 1 && !containsKanji(name) && !SINGLE_KANA_CHARACTER.test(name);
|
||||
}
|
||||
|
||||
function isUsableNameTerm(name: string): boolean {
|
||||
return !isNameDisambiguatorLetter(name);
|
||||
}
|
||||
|
||||
// Kana, Han (shared ranges), and the marks that only ever appear inside a
|
||||
// Japanese name: iteration marks and the small ka/ke used in place names.
|
||||
const JAPANESE_NAME_CHARACTERS = new RegExp(
|
||||
`^[\\u3040-\\u30ff${HAN_REGEXP_CLASS_BODY}\u3005\u3006\u30f5\u30f6\u30fc]+$`,
|
||||
'u',
|
||||
);
|
||||
|
||||
export function isJapaneseNameSplitCandidate(name: string): boolean {
|
||||
const compact = name.replace(/[\s\u3000・・·•]/g, '');
|
||||
return (
|
||||
containsKanji(compact) && /^[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff々〆ヵヶー]+$/.test(compact)
|
||||
);
|
||||
return containsKanji(compact) && JAPANESE_NAME_CHARACTERS.test(compact);
|
||||
}
|
||||
|
||||
function addJapaneseNameParts(
|
||||
@@ -97,8 +121,11 @@ export function buildNameTerms(
|
||||
|
||||
const split = name.split(/[\s\u3000]+/).filter((part) => part.trim().length > 0);
|
||||
if (split.length === 2) {
|
||||
target.add(split[0]!);
|
||||
target.add(split[1]!);
|
||||
for (const part of split) {
|
||||
if (isUsableNameTerm(part)) {
|
||||
target.add(part);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const splitByMiddleDot = name
|
||||
@@ -107,9 +134,11 @@ export function buildNameTerms(
|
||||
.filter((part) => part.length > 0);
|
||||
if (splitByMiddleDot.length >= 2) {
|
||||
for (const part of splitByMiddleDot) {
|
||||
if (isUsableNameTerm(part)) {
|
||||
target.add(part);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (target === base) {
|
||||
addJapaneseNameParts(character, name, base, resolvedSplits);
|
||||
@@ -117,7 +146,10 @@ export function buildNameTerms(
|
||||
}
|
||||
}
|
||||
|
||||
for (const alias of addRomanizedKanaAliases(romanizedBase)) {
|
||||
// Romanized forms that are a bare letter would become a single-kana alias.
|
||||
for (const alias of addRomanizedKanaAliases(
|
||||
[...romanizedBase].filter((entry) => !isNameDisambiguatorLetter(entry)),
|
||||
)) {
|
||||
base.add(alias);
|
||||
}
|
||||
|
||||
@@ -136,6 +168,9 @@ export function buildNameTerms(
|
||||
|
||||
const withHonorifics = new Set<string>();
|
||||
for (const entry of base) {
|
||||
// Only labels split off a longer name are filtered (see above); an explicit
|
||||
// one-character name reaches this point intact.
|
||||
if (isNameDisambiguatorLetter(entry)) continue;
|
||||
withHonorifics.add(entry);
|
||||
for (const suffix of HONORIFIC_SUFFIXES) {
|
||||
withHonorifics.add(`${entry}${suffix.term}`);
|
||||
|
||||
@@ -223,7 +223,7 @@ test('update overlay notification action triggers install flow', () => {
|
||||
assert.match(runtimeSource, /fallbackClient\.openNoteInBrowser\(noteId\)/);
|
||||
});
|
||||
|
||||
test('subtitle change re-prioritizes prefetch around live playback before tokenizing current line', () => {
|
||||
test('subtitle change pauses prefetch without restarting its run before tokenizing current line', () => {
|
||||
const source = readMainSource();
|
||||
const actionBlock = source.match(
|
||||
/onSubtitleChange:\s*\(text\)\s*=>\s*\{(?<body>[\s\S]*?)\n \},\n refreshDiscordPresence:/,
|
||||
@@ -231,15 +231,19 @@ test('subtitle change re-prioritizes prefetch around live playback before tokeni
|
||||
|
||||
assert.ok(actionBlock);
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.pause\(\);/);
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
||||
assert.match(actionBlock, /subtitleProcessingController\.onSubtitleChange\(text\);/);
|
||||
// Restarting the run per line (onSeek) discards in-flight prefetch work;
|
||||
// only real seeks restart via onTimePosUpdate.
|
||||
assert.doesNotMatch(actionBlock, /subtitlePrefetchService\?\.onSeek\(/);
|
||||
assert.match(actionBlock, /subtitleProcessingController\.onSubtitleChange\(text\)/);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitlePrefetchService?.pause();') <
|
||||
actionBlock.indexOf('subtitlePrefetchService?.onSeek(lastObservedTimePos);'),
|
||||
actionBlock.indexOf('subtitleProcessingController.onSubtitleChange(text)'),
|
||||
);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitlePrefetchService?.onSeek(lastObservedTimePos);') <
|
||||
actionBlock.indexOf('subtitleProcessingController.onSubtitleChange(text);'),
|
||||
// A repeated subtitle emits nothing, so the pause has to be released here or
|
||||
// prefetching idles until the next distinct line.
|
||||
assert.match(
|
||||
actionBlock,
|
||||
/if \(!subtitleProcessingController\.onSubtitleChange\(text\)\) \{[\s\S]*?subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -489,16 +493,35 @@ test('known-word updates invalidate prefetched tokenizations before refreshing c
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
||||
assert.match(
|
||||
actionBlock,
|
||||
/subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\);/,
|
||||
/if \(!subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\)\) \{[\s\S]*?subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitleProcessingController.invalidateTokenizationCache();') <
|
||||
actionBlock.indexOf(
|
||||
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);',
|
||||
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)',
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
test('subtitle processing controller resumes prefetch on settle, not on its emits', () => {
|
||||
const source = readMainSource();
|
||||
const depsBlock = source.match(
|
||||
/createBuildSubtitleProcessingControllerMainDepsHandler\(\{(?<body>[\s\S]*?)\n \}\);/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(depsBlock);
|
||||
// A controller emit can be the provisional plain payload sent before the
|
||||
// scan runs, so it must not release the prefetch pause.
|
||||
assert.match(
|
||||
depsBlock,
|
||||
/emitSubtitle: \(payload\) => emitSubtitlePayload\(payload, \{ resumePrefetch: false \}\),/,
|
||||
);
|
||||
assert.match(
|
||||
depsBlock,
|
||||
/onProcessingSettled: \(\) => \{\s+subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
});
|
||||
|
||||
test('manual visible overlay changes notify mpv plugin visibility state', () => {
|
||||
const source = readMainSource();
|
||||
const setBlock = source.match(
|
||||
@@ -593,7 +616,7 @@ test('YouTube media cache lifecycle routes through configured status notificatio
|
||||
test('subtitle broadcasts share one frequency options snapshot per emitted payload', () => {
|
||||
const source = readMainSource();
|
||||
const emitBlock = source.match(
|
||||
/function emitSubtitlePayload\(payload: SubtitleData\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
/function emitSubtitlePayload\([\s\S]*?\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
)?.groups?.body;
|
||||
const frequencyOptionsSnapshot = emitBlock?.match(
|
||||
/const frequencyDictionary = configService\.getConfig\(\)\.subtitleStyle\.frequencyDictionary;(?<body>[\s\S]*?)\n \};/,
|
||||
@@ -616,7 +639,7 @@ test('subtitle broadcasts share one frequency options snapshot per emitted paylo
|
||||
test('annotation upgrades skip the duplicate basic websocket event', () => {
|
||||
const source = readMainSource();
|
||||
const emitBlock = source.match(
|
||||
/function emitSubtitlePayload\(payload: SubtitleData\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
/function emitSubtitlePayload\([\s\S]*?\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(emitBlock);
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { createSubtitleProcessingController } from '../../core/services/subtitle-processing-controller';
|
||||
import type { SubtitleData } from '../../types';
|
||||
import {
|
||||
createAutoplaySubtitlePrimingRuntime,
|
||||
setMpvCurrentSecondarySubText,
|
||||
@@ -42,8 +44,8 @@ test('scheduleSubtitlePrefetchRefresh logs refresh failures from timer callback'
|
||||
setActiveParsedSubtitleMediaPath: () => {},
|
||||
subtitleProcessingController: {
|
||||
consumeCachedSubtitle: () => null,
|
||||
onSubtitleChange: () => {},
|
||||
refreshCurrentSubtitle: () => {},
|
||||
onSubtitleChange: () => true,
|
||||
refreshCurrentSubtitle: () => true,
|
||||
},
|
||||
emitSubtitlePayload: () => {},
|
||||
getSubtitlePrefetchService: () => null,
|
||||
@@ -93,13 +95,24 @@ test('primeCurrentSubtitleForAutoplay refreshes active subtitle cues when mpv su
|
||||
setActiveParsedSubtitleMediaPath: () => {},
|
||||
subtitleProcessingController: {
|
||||
consumeCachedSubtitle: () => null,
|
||||
onSubtitleChange: (text) => calls.push(`change:${text}`),
|
||||
refreshCurrentSubtitle: (text) => calls.push(`refresh:${text ?? ''}`),
|
||||
onSubtitleChange: (text) => {
|
||||
calls.push(`change:${text}`);
|
||||
return true;
|
||||
},
|
||||
emitSubtitlePayload: (payload) => calls.push(`emit:${payload.text}`),
|
||||
refreshCurrentSubtitle: (text) => {
|
||||
calls.push(`refresh:${text ?? ''}`);
|
||||
return true;
|
||||
},
|
||||
},
|
||||
emitSubtitlePayload: (payload, options) =>
|
||||
calls.push(`emit:${payload.text}:resume=${options?.resumePrefetch !== false}`),
|
||||
getSubtitlePrefetchService: () => ({
|
||||
pause: () => calls.push('prefetch:pause'),
|
||||
onSeek: (timePos) => calls.push(`prefetch:seek:${timePos}`),
|
||||
pause: () => {
|
||||
calls.push('prefetch:pause');
|
||||
},
|
||||
resume: () => {
|
||||
calls.push('prefetch:resume');
|
||||
},
|
||||
}),
|
||||
getLastObservedTimePos: () => 12,
|
||||
getVisibleOverlayVisible: () => true,
|
||||
@@ -120,8 +133,10 @@ test('primeCurrentSubtitleForAutoplay refreshes active subtitle cues when mpv su
|
||||
'request:time-pos',
|
||||
'set:起動字幕',
|
||||
'prefetch:pause',
|
||||
'emit:起動字幕',
|
||||
'change:起動字幕',
|
||||
'emit:起動字幕:resume=false',
|
||||
// Uncached priming refreshes rather than announcing a change, so an
|
||||
// invalidated-but-unchanged line is still re-tokenized.
|
||||
'refresh:起動字幕',
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -151,13 +166,24 @@ test('primeCurrentSubtitleForAutoplay emits raw first paint on cache miss before
|
||||
setActiveParsedSubtitleMediaPath: () => {},
|
||||
subtitleProcessingController: {
|
||||
consumeCachedSubtitle: () => null,
|
||||
onSubtitleChange: (text) => calls.push(`change:${text}`),
|
||||
refreshCurrentSubtitle: (text) => calls.push(`refresh:${text ?? ''}`),
|
||||
onSubtitleChange: (text) => {
|
||||
calls.push(`change:${text}`);
|
||||
return true;
|
||||
},
|
||||
emitSubtitlePayload: (payload) => calls.push(`emit:${payload.text}`),
|
||||
refreshCurrentSubtitle: (text) => {
|
||||
calls.push(`refresh:${text ?? ''}`);
|
||||
return true;
|
||||
},
|
||||
},
|
||||
emitSubtitlePayload: (payload, options) =>
|
||||
calls.push(`emit:${payload.text}:resume=${options?.resumePrefetch !== false}`),
|
||||
getSubtitlePrefetchService: () => ({
|
||||
pause: () => calls.push('prefetch:pause'),
|
||||
onSeek: (timePos) => calls.push(`prefetch:seek:${timePos}`),
|
||||
pause: () => {
|
||||
calls.push('prefetch:pause');
|
||||
},
|
||||
resume: () => {
|
||||
calls.push('prefetch:resume');
|
||||
},
|
||||
}),
|
||||
getLastObservedTimePos: () => 12,
|
||||
getVisibleOverlayVisible: () => true,
|
||||
@@ -175,7 +201,222 @@ test('primeCurrentSubtitleForAutoplay emits raw first paint on cache miss before
|
||||
'request:sub-text',
|
||||
'set:起動字幕',
|
||||
'prefetch:pause',
|
||||
'emit:起動字幕',
|
||||
'change:起動字幕',
|
||||
'emit:起動字幕:resume=false',
|
||||
// Uncached priming refreshes rather than announcing a change, so an
|
||||
// invalidated-but-unchanged line is still re-tokenized.
|
||||
'refresh:起動字幕',
|
||||
]);
|
||||
});
|
||||
|
||||
// Driven by the real processing controller rather than a stub: the failure this
|
||||
// covers is a disagreement between the priming path and the controller's own
|
||||
// staleness rules, which a hand-written stub cannot reproduce.
|
||||
function createPrimingRuntimeWithRealController(options: {
|
||||
text: string;
|
||||
calls: string[];
|
||||
onTokenize: () => void;
|
||||
tokenize?: (text: string) => SubtitleData | null | Promise<SubtitleData | null>;
|
||||
cacheLimit?: number;
|
||||
}) {
|
||||
const { text, calls } = options;
|
||||
let currentSubText = '';
|
||||
let currentSubtitleData: SubtitleData | null = null;
|
||||
const mediaPath = '/media/video.mkv';
|
||||
|
||||
const prefetchService = {
|
||||
pause: () => calls.push('prefetch:pause'),
|
||||
resume: () => calls.push('prefetch:resume'),
|
||||
};
|
||||
// Mirrors main.ts emitSubtitlePayload: an emit resumes prefetching unless it
|
||||
// is explicitly marked as not the end of the work for the line, and every
|
||||
// controller emit is so marked.
|
||||
const emitSubtitlePayload = (
|
||||
payload: SubtitleData,
|
||||
emitOptions?: { resumePrefetch?: boolean },
|
||||
): void => {
|
||||
currentSubtitleData = payload;
|
||||
calls.push(
|
||||
emitOptions?.resumePrefetch === false
|
||||
? `emit-raw:${payload.text}`
|
||||
: `emit-direct:${payload.text}`,
|
||||
);
|
||||
if (emitOptions?.resumePrefetch !== false) {
|
||||
prefetchService.resume();
|
||||
}
|
||||
};
|
||||
|
||||
const subtitleProcessingController = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (subtitleText) => {
|
||||
options.onTokenize();
|
||||
return options.tokenize ? options.tokenize(subtitleText) : { text: subtitleText, tokens: [] };
|
||||
},
|
||||
// main.ts routes controller emits through emitSubtitlePayload with
|
||||
// resumePrefetch: false, so they never release the pause on their own.
|
||||
emitSubtitle: (payload) => {
|
||||
currentSubtitleData = payload;
|
||||
calls.push(`emit:${payload.text}:tokens=${payload.tokens === null ? 'none' : 'yes'}`);
|
||||
},
|
||||
onProcessingSettled: () => {
|
||||
prefetchService.resume();
|
||||
},
|
||||
...(options.cacheLimit === undefined ? {} : { cacheLimit: options.cacheLimit }),
|
||||
});
|
||||
|
||||
const runtime = createAutoplaySubtitlePrimingRuntime({
|
||||
getCurrentMediaPath: () => mediaPath,
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentVideoPath: mediaPath,
|
||||
requestProperty: async (name) => (name === 'sub-text' ? text : null),
|
||||
}),
|
||||
setCurrentSubText: (value) => {
|
||||
currentSubText = value;
|
||||
},
|
||||
getCurrentSubText: () => currentSubText,
|
||||
getCurrentSubtitleData: () => currentSubtitleData,
|
||||
getActiveParsedSubtitleCues: () => [],
|
||||
setActiveParsedSubtitleMediaPath: () => {},
|
||||
subtitleProcessingController,
|
||||
emitSubtitlePayload,
|
||||
getSubtitlePrefetchService: () => prefetchService,
|
||||
getLastObservedTimePos: () => 12,
|
||||
getVisibleOverlayVisible: () => true,
|
||||
emitSecondarySubtitle: () => {},
|
||||
initSubtitlePrefetch: async () => {},
|
||||
refreshSubtitlePrefetchFromActiveTrack: async () => {},
|
||||
logDebug: () => {},
|
||||
});
|
||||
|
||||
return { runtime, subtitleProcessingController, mediaPath };
|
||||
}
|
||||
|
||||
test('primeCurrentSubtitleForAutoplay re-tokenizes text whose cached annotation was invalidated', async () => {
|
||||
const calls: string[] = [];
|
||||
let tokenizations = 0;
|
||||
const text = '起動字幕';
|
||||
const { runtime, subtitleProcessingController, mediaPath } =
|
||||
createPrimingRuntimeWithRealController({
|
||||
text,
|
||||
calls,
|
||||
onTokenize: () => {
|
||||
tokenizations += 1;
|
||||
},
|
||||
});
|
||||
|
||||
// The line was already tokenized and cached during normal playback.
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
const tokenizationsBeforeInvalidation = tokenizations;
|
||||
|
||||
// Mining a card drops every cached tokenization.
|
||||
subtitleProcessingController.invalidateTokenizationCache();
|
||||
calls.length = 0;
|
||||
|
||||
await runtime.primeCurrentSubtitleForAutoplay(mediaPath);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
// The cache miss must schedule fresh work, or the line stays unannotated for
|
||||
// as long as it is on screen.
|
||||
assert.equal(
|
||||
tokenizations,
|
||||
tokenizationsBeforeInvalidation + 1,
|
||||
'expected the invalidated subtitle to be tokenized again',
|
||||
);
|
||||
assert.ok(
|
||||
calls.includes(`emit:${text}:tokens=yes`),
|
||||
`expected an annotated emit, saw ${JSON.stringify(calls)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('primeCurrentSubtitleForAutoplay releases the prefetch pause when nothing is scheduled', async () => {
|
||||
const calls: string[] = [];
|
||||
const text = '起動字幕';
|
||||
const { runtime, subtitleProcessingController, mediaPath } =
|
||||
createPrimingRuntimeWithRealController({
|
||||
text,
|
||||
calls,
|
||||
onTokenize: () => {},
|
||||
cacheLimit: 1,
|
||||
});
|
||||
|
||||
// Emitted at the current cache generation, then evicted from the one-entry
|
||||
// cache: priming misses the cache but the controller has nothing to redo, so
|
||||
// no emit is coming and the pause must be released here.
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
subtitleProcessingController.preCacheTokenization('別の字幕', {
|
||||
text: '別の字幕',
|
||||
tokens: [],
|
||||
});
|
||||
calls.length = 0;
|
||||
|
||||
await runtime.primeCurrentSubtitleForAutoplay(mediaPath);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
assert.deepEqual(calls, ['prefetch:pause', `emit-raw:${text}`, 'prefetch:resume']);
|
||||
});
|
||||
|
||||
test('primeCurrentSubtitleForAutoplay releases the prefetch pause when tokenization emits nothing', async () => {
|
||||
const calls: string[] = [];
|
||||
const text = '起動字幕';
|
||||
const { runtime, subtitleProcessingController, mediaPath } =
|
||||
createPrimingRuntimeWithRealController({
|
||||
text,
|
||||
calls,
|
||||
onTokenize: () => {},
|
||||
// Transient tokenizer failure: the controller falls back to plain text it
|
||||
// has already shown, so it suppresses the emit entirely.
|
||||
tokenize: () => null,
|
||||
});
|
||||
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
subtitleProcessingController.invalidateTokenizationCache();
|
||||
calls.length = 0;
|
||||
|
||||
await runtime.primeCurrentSubtitleForAutoplay(mediaPath);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
assert.ok(
|
||||
!calls.some((call) => call.startsWith('emit:')),
|
||||
`expected no controller emit, saw ${JSON.stringify(calls)}`,
|
||||
);
|
||||
assert.equal(
|
||||
calls.filter((call) => call === 'prefetch:resume').length,
|
||||
1,
|
||||
`expected the prefetch pause to be released, saw ${JSON.stringify(calls)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('prefetch stays paused until tokenization of an uncached line completes', async () => {
|
||||
const calls: string[] = [];
|
||||
const text = '起動字幕';
|
||||
let finishTokenization = (): void => {};
|
||||
const tokenizationGate = new Promise<void>((resolve) => {
|
||||
finishTokenization = resolve;
|
||||
});
|
||||
const { subtitleProcessingController } = createPrimingRuntimeWithRealController({
|
||||
text,
|
||||
calls,
|
||||
onTokenize: () => {},
|
||||
tokenize: async (subtitleText) => {
|
||||
await tokenizationGate;
|
||||
return { text: subtitleText, tokens: [] };
|
||||
},
|
||||
});
|
||||
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
// The provisional plain emit must not release the pause: the expensive scan
|
||||
// is still ahead of it and would compete with prefetching for the parser.
|
||||
assert.deepEqual(calls, [`emit:${text}:tokens=none`]);
|
||||
|
||||
finishTokenization();
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
assert.deepEqual(calls, [
|
||||
`emit:${text}:tokens=none`,
|
||||
`emit:${text}:tokens=yes`,
|
||||
'prefetch:resume',
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -17,7 +17,7 @@ type AutoplaySubtitlePrimingMpvClient = {
|
||||
|
||||
type AutoplaySubtitlePrimingPrefetchService = {
|
||||
pause: () => void;
|
||||
onSeek: (timePos: number) => void;
|
||||
resume: () => void;
|
||||
};
|
||||
|
||||
export interface AutoplaySubtitlePrimingRuntimeDeps {
|
||||
@@ -30,10 +30,11 @@ export interface AutoplaySubtitlePrimingRuntimeDeps {
|
||||
setActiveParsedSubtitleMediaPath: (mediaPath: string | null) => void;
|
||||
subtitleProcessingController: {
|
||||
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
||||
onSubtitleChange: (text: string) => void;
|
||||
refreshCurrentSubtitle: (text: string) => void;
|
||||
// Both report whether processing is pending; see pausePrefetchUntilProcessed.
|
||||
onSubtitleChange: (text: string) => boolean;
|
||||
refreshCurrentSubtitle: (text: string) => boolean;
|
||||
};
|
||||
emitSubtitlePayload: (payload: SubtitleData) => void;
|
||||
emitSubtitlePayload: (payload: SubtitleData, options?: { resumePrefetch?: boolean }) => void;
|
||||
getSubtitlePrefetchService: () => AutoplaySubtitlePrimingPrefetchService | null;
|
||||
getLastObservedTimePos: () => number;
|
||||
getVisibleOverlayVisible: () => boolean;
|
||||
@@ -64,6 +65,19 @@ export function setMpvCurrentSecondarySubText(
|
||||
export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimingRuntimeDeps) {
|
||||
const { subtitleProcessingController, emitSubtitlePayload } = deps;
|
||||
|
||||
// Prefetching is paused so the on-screen line gets the parser to itself; the
|
||||
// resume rides on the controller settling (see onProcessingSettled), not on
|
||||
// an emit, which a suppressed duplicate or a failed tokenization never sends.
|
||||
// When the controller reports it has nothing scheduled, no settle is coming
|
||||
// either, so release the pause here or prefetching idles indefinitely.
|
||||
function pausePrefetchUntilProcessed(scheduleTokenization: () => boolean): void {
|
||||
const prefetch = deps.getSubtitlePrefetchService();
|
||||
prefetch?.pause();
|
||||
if (!scheduleTokenization()) {
|
||||
prefetch?.resume();
|
||||
}
|
||||
}
|
||||
|
||||
let subtitlePrefetchRefreshTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let autoplaySubtitlePrimedMediaPath: string | null = null;
|
||||
let visibleOverlaySubtitleRefreshAfterFirstPaintTimer: ReturnType<typeof setTimeout> | null =
|
||||
@@ -104,12 +118,23 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
const cachedPayload = subtitleProcessingController.consumeCachedSubtitle(text);
|
||||
if (cachedPayload) {
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
// This emit resumes prefetching, so no pause is left outstanding.
|
||||
emitSubtitlePayload(cachedPayload);
|
||||
return true;
|
||||
}
|
||||
|
||||
emitSubtitlePayload({ text, tokens: null });
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
// Provisional raw emit: keep prefetch paused until the processing
|
||||
// controller is done with this line.
|
||||
emitSubtitlePayload({ text, tokens: null }, { resumePrefetch: false });
|
||||
// refreshCurrentSubtitle, not onSubtitleChange: the cache miss above can be
|
||||
// an invalidation (mining a card) on text the controller still holds, and
|
||||
// onSubtitleChange treats unchanged text as nothing to do, which would
|
||||
// leave this line permanently unannotated. refreshCurrentSubtitle also
|
||||
// re-tokenizes for a new cache generation.
|
||||
if (!subtitleProcessingController.refreshCurrentSubtitle(text)) {
|
||||
// Nothing scheduled, so no settle is coming to release the pause.
|
||||
deps.getSubtitlePrefetchService()?.resume();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -153,14 +178,12 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
getCurrentSubtitleData: () => deps.getCurrentSubtitleData(),
|
||||
consumeCachedSubtitle: (text) => subtitleProcessingController.consumeCachedSubtitle(text),
|
||||
onSubtitleChange: (text) => {
|
||||
deps.getSubtitlePrefetchService()?.pause();
|
||||
deps.getSubtitlePrefetchService()?.onSeek(deps.getLastObservedTimePos());
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
pausePrefetchUntilProcessed(() => subtitleProcessingController.onSubtitleChange(text));
|
||||
},
|
||||
refreshCurrentSubtitle: (text) => {
|
||||
deps.getSubtitlePrefetchService()?.pause();
|
||||
deps.getSubtitlePrefetchService()?.onSeek(deps.getLastObservedTimePos());
|
||||
subtitleProcessingController.refreshCurrentSubtitle(text);
|
||||
pausePrefetchUntilProcessed(() =>
|
||||
subtitleProcessingController.refreshCurrentSubtitle(text),
|
||||
);
|
||||
},
|
||||
deferUncachedRefresh: true,
|
||||
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
||||
@@ -204,9 +227,7 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
if (!text.trim()) {
|
||||
return;
|
||||
}
|
||||
deps.getSubtitlePrefetchService()?.pause();
|
||||
deps.getSubtitlePrefetchService()?.onSeek(deps.getLastObservedTimePos());
|
||||
subtitleProcessingController.refreshCurrentSubtitle(text);
|
||||
pausePrefetchUntilProcessed(() => subtitleProcessingController.refreshCurrentSubtitle(text));
|
||||
}, VISIBLE_OVERLAY_SUBTITLE_REFRESH_AFTER_FIRST_PAINT_DELAY_MS);
|
||||
visibleOverlaySubtitleRefreshAfterFirstPaintTimer.unref?.();
|
||||
}
|
||||
|
||||
@@ -53,3 +53,52 @@ test('character dictionary sync completion refreshes subtitle state when diction
|
||||
'log:[dictionary:auto-sync] refreshed current subtitle after sync (AniList 1, changed=yes, title=Frieren)',
|
||||
]);
|
||||
});
|
||||
|
||||
test('character dictionary sync completion drops cached dictionary reads before refreshing', () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
handleCharacterDictionaryAutoSyncComplete(
|
||||
{
|
||||
mediaId: 1,
|
||||
mediaTitle: 'Frieren',
|
||||
changed: true,
|
||||
},
|
||||
{
|
||||
hasParserWindow: () => true,
|
||||
invalidateCharacterDictionaryLookups: () => calls.push('invalidate-dictionary-lookups'),
|
||||
clearParserCaches: () => calls.push('clear-parser'),
|
||||
invalidateTokenizationCache: () => calls.push('invalidate'),
|
||||
refreshSubtitlePrefetch: () => calls.push('prefetch'),
|
||||
refreshCurrentSubtitle: () => calls.push('refresh-subtitle'),
|
||||
logInfo: () => {},
|
||||
},
|
||||
);
|
||||
|
||||
// Must run before the refreshes, or they re-tokenize against the character
|
||||
// names and images from the previous dictionary build.
|
||||
assert.equal(calls[0], 'invalidate-dictionary-lookups');
|
||||
assert.ok(calls.indexOf('invalidate-dictionary-lookups') < calls.indexOf('refresh-subtitle'));
|
||||
});
|
||||
|
||||
test('character dictionary sync completion leaves cached dictionary reads alone when unchanged', () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
handleCharacterDictionaryAutoSyncComplete(
|
||||
{
|
||||
mediaId: 1,
|
||||
mediaTitle: 'Frieren',
|
||||
changed: false,
|
||||
},
|
||||
{
|
||||
hasParserWindow: () => true,
|
||||
invalidateCharacterDictionaryLookups: () => calls.push('invalidate-dictionary-lookups'),
|
||||
clearParserCaches: () => calls.push('clear-parser'),
|
||||
invalidateTokenizationCache: () => calls.push('invalidate'),
|
||||
refreshSubtitlePrefetch: () => calls.push('prefetch'),
|
||||
refreshCurrentSubtitle: () => calls.push('refresh-subtitle'),
|
||||
logInfo: () => {},
|
||||
},
|
||||
);
|
||||
|
||||
assert.deepEqual(calls, []);
|
||||
});
|
||||
|
||||
@@ -7,6 +7,12 @@ export function handleCharacterDictionaryAutoSyncComplete(
|
||||
deps: {
|
||||
hasParserWindow: () => boolean;
|
||||
clearParserCaches: () => void;
|
||||
/**
|
||||
* Drops cached reads of the generated dictionary (character images, and the
|
||||
* name candidates the scanner uses to skip lookups). Runs before the
|
||||
* refreshes below so they re-tokenize against the new dictionary content.
|
||||
*/
|
||||
invalidateCharacterDictionaryLookups?: () => void;
|
||||
invalidateTokenizationCache: () => void;
|
||||
refreshSubtitlePrefetch: () => void;
|
||||
refreshCurrentSubtitle: () => void;
|
||||
@@ -14,6 +20,7 @@ export function handleCharacterDictionaryAutoSyncComplete(
|
||||
},
|
||||
): void {
|
||||
if (completion.changed) {
|
||||
deps.invalidateCharacterDictionaryLookups?.();
|
||||
if (deps.hasParserWindow()) {
|
||||
deps.clearParserCaches();
|
||||
}
|
||||
|
||||
@@ -225,24 +225,35 @@ export function composeMpvRuntimeHandlers<
|
||||
}
|
||||
return tokenizationWarmupInFlight;
|
||||
};
|
||||
const tokenizeSubtitle = async (text: string): Promise<TTokenizedSubtitle> => {
|
||||
if (!tokenizationWarmupCompleted) void startTokenizationWarmups();
|
||||
await ensureTokenizationPrerequisites();
|
||||
// Built once and reused for every tokenization: per-call rebuilds create
|
||||
// fresh closures, which defeats identity-keyed caches downstream (the JLPT
|
||||
// lookup cache keys on the getJlptLevel function, and the mecab availability
|
||||
// WeakSet keys on the runtime deps instance).
|
||||
let cachedTokenizerRuntimeDeps: TTokenizerRuntimeDeps | null = null;
|
||||
const getTokenizerRuntimeDeps = (): TTokenizerRuntimeDeps => {
|
||||
if (cachedTokenizerRuntimeDeps) {
|
||||
return cachedTokenizerRuntimeDeps;
|
||||
}
|
||||
const tokenizerMainDeps = buildTokenizerDepsHandler();
|
||||
if (shouldWarmupAnnotationDictionaries()) {
|
||||
const onTokenizationReady = tokenizerMainDeps.onTokenizationReady;
|
||||
const baseOnTokenizationReady = tokenizerMainDeps.onTokenizationReady;
|
||||
tokenizerMainDeps.onTokenizationReady = (tokenizedText: string): void => {
|
||||
if (!shouldWarmupAnnotationDictionaries()) {
|
||||
baseOnTokenizationReady?.(tokenizedText);
|
||||
return;
|
||||
}
|
||||
markTokenizationPlaybackReady();
|
||||
onTokenizationReady?.(tokenizedText);
|
||||
baseOnTokenizationReady?.(tokenizedText);
|
||||
if (!tokenizationWarmupCompleted) {
|
||||
void prewarmSubtitleDictionaries({ showLoadingOsd: true }).catch(() => {});
|
||||
}
|
||||
};
|
||||
}
|
||||
return options.tokenizer.tokenizeSubtitle(
|
||||
text,
|
||||
options.tokenizer.createTokenizerRuntimeDeps(tokenizerMainDeps),
|
||||
);
|
||||
cachedTokenizerRuntimeDeps = options.tokenizer.createTokenizerRuntimeDeps(tokenizerMainDeps);
|
||||
return cachedTokenizerRuntimeDeps;
|
||||
};
|
||||
const tokenizeSubtitle = async (text: string): Promise<TTokenizedSubtitle> => {
|
||||
if (!tokenizationWarmupCompleted) void startTokenizationWarmups();
|
||||
await ensureTokenizationPrerequisites();
|
||||
return options.tokenizer.tokenizeSubtitle(text, getTokenizerRuntimeDeps());
|
||||
};
|
||||
|
||||
const launchBackgroundWarmupTask = createLaunchBackgroundWarmupTaskFromStartup(
|
||||
|
||||
@@ -6,6 +6,7 @@ export function createBuildSubtitleProcessingControllerMainDepsHandler(
|
||||
return (): SubtitleProcessingControllerDeps => ({
|
||||
tokenizeSubtitle: (text: string) => deps.tokenizeSubtitle(text),
|
||||
emitSubtitle: (payload) => deps.emitSubtitle(payload),
|
||||
onProcessingSettled: () => deps.onProcessingSettled?.(),
|
||||
logDebug: deps.logDebug,
|
||||
now: deps.now,
|
||||
});
|
||||
|
||||
@@ -9,6 +9,9 @@ type TokenizerMainDeps = TokenizerDepsRuntimeOptions & {
|
||||
getCurrentCharacterDictionaryMediaId?: NonNullable<
|
||||
TokenizerDepsRuntimeOptions['getCurrentCharacterDictionaryMediaId']
|
||||
>;
|
||||
getCharacterNameCandidates?: NonNullable<
|
||||
TokenizerDepsRuntimeOptions['getCharacterNameCandidates']
|
||||
>;
|
||||
getFrequencyDictionaryEnabled: NonNullable<
|
||||
TokenizerDepsRuntimeOptions['getFrequencyDictionaryEnabled']
|
||||
>;
|
||||
@@ -84,6 +87,11 @@ export function createBuildTokenizerDepsMainHandler(deps: TokenizerMainDeps) {
|
||||
getCurrentCharacterDictionaryMediaId: () => deps.getCurrentCharacterDictionaryMediaId!(),
|
||||
}
|
||||
: {}),
|
||||
...(deps.getCharacterNameCandidates
|
||||
? {
|
||||
getCharacterNameCandidates: () => deps.getCharacterNameCandidates!(),
|
||||
}
|
||||
: {}),
|
||||
getFrequencyDictionaryEnabled: () => deps.getFrequencyDictionaryEnabled(),
|
||||
getFrequencyDictionaryMatchMode: () => deps.getFrequencyDictionaryMatchMode(),
|
||||
getFrequencyRank: (text: string) => deps.getFrequencyRank(text),
|
||||
|
||||
Reference in New Issue
Block a user