Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
84c75f50aa
|
|||
|
8b838f2c7d
|
|||
|
db4139ba0b
|
|||
| 6c251502b3 | |||
| ae40934d3a | |||
|
cdb1475a54
|
|||
|
a2e49b369b
|
|||
| 7f13aed50a | |||
|
8b21a2bca8
|
|
Before Width: | Height: | Size: 436 B After Width: | Height: | Size: 253 B |
|
Before Width: | Height: | Size: 549 B After Width: | Height: | Size: 366 B |
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Removed 確かに and やはり from the annotation stop-word list so they get frequency/JLPT highlighting again and count toward vocabulary stats; the list now only covers interjections, pronouns, and grammar fragments.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed character names being swallowed by longer generic dictionary matches during subtitle tokenization (e.g. ヨータ in 美姫とヨータ was lost because とヨー normalizes to とよう and matched 渡洋), so the name never tokenized and got no highlight, portrait, or hover lookup. Character-dictionary matches now claim their spans in a greedy pre-pass before the left-to-right scan, and re-segmented name spans override the parse tokenization on merge. A name only claims its span when no strictly longer generic word starts at the same position, so a character named 空 no longer splits 空気 (ties still go to the name). The pre-pass runs only when a SubMiner character dictionary is enabled in the active Yomitan profile.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Kanji-bearing nouns that MeCab tags as non-independent (非自立) — e.g. 日 in いい日だったな, 点, 以外 — now keep frequency/JLPT highlighting and count toward vocabulary stats. Yomitan segments them as standalone vocabulary tokens, so the MeCab POS filter only suppresses kana grammar nouns (こと, もの, とき) it was meant for.
|
||||
@@ -1,4 +1,7 @@
|
||||
type: fixed
|
||||
type: changed
|
||||
area: overlay
|
||||
|
||||
- Fixed words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" matching a known 床/ゆか "floor" card). The known-word cache now stores each card's word together with its reading and only matches when the token's reading agrees; cards without a reading field keep matching in any reading as before.
|
||||
- New reading-aware subtitle parsing and known-word matching: the known-word cache now stores each Anki card's word together with its reading (cache format v3), and a token only gets the known-word highlight when its parsed reading agrees with the card. The cache and the stats server upgrade automatically.
|
||||
- Fixes words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" no longer matches a known 床/ゆか "floor" card). Cards without a reading field keep matching in any reading as before.
|
||||
- Single-kana grammar tokens (よ in 全然いいよ, standalone え) no longer borrow the reading of an unrelated card (such as 夜 or 絵) and get painted as known: reading-only matching requires at least two kana, while single-kana cards still match by their word field.
|
||||
- Kanji words no longer count as known just because a different mined kanji word shares their reading (渓谷/けいこく no longer matches a known 警告/けいこく card). Words mined in kana form still highlight their kanji spelling in subtitles, and kana subtitle text still matches cards mined in kanji form.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed character-name annotations dropping for an entire subtitle line when it contained any chunk the dictionary scanner could not match (e.g. an interjection like やほっ before a name): scanner metadata is now merged per token into the parseText segmentation instead of being discarded on any span mismatch.
|
||||
@@ -1,5 +1,4 @@
|
||||
type: fixed
|
||||
type: changed
|
||||
area: overlay
|
||||
|
||||
- Fixed single-kana grammar tokens counting as known words by borrowing the reading of an unrelated Anki note (よ in 全然いいよ matched a card read よ such as 夜, standalone え matched 絵), which painted them with the known-word highlight. Reading-only known-word matching now requires at least two kana; single-kana cards still match by their word field, and tokens genuinely present in the known-words cache (e.g. です) keep their highlight.
|
||||
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Session known-word counts no longer show 0 everywhere. The stats server's known-word cache parser only understood the v1/v2 cache formats, so after the reading-aware v3 cache upgrade it silently treated the cache as missing; it now flattens v3 note entries into the headword set.
|
||||
@@ -1,4 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed subtitle text that Yomitan's parser cannot match (e.g. the truncated volitional in とこ戻ろ…) being rendered as plain, non-interactive text: it was invisible to hover/lookup and excluded from the n+1 word count, which could wrongly mark a sentence as n+1. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation; bracketed SFX/speaker captions and punctuation-only runs are still skipped.
|
||||
- Fixed subtitle text that Yomitan's parser cannot match (truncated inflections like the volitional in とこ戻ろ…, elongation runs like ぅ~/ぉ〜) being rendered as plain, non-interactive text with no hover lookup. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation, while being fully ignored by frequency/JLPT highlighting, the N+1 candidate math, and vocabulary stats. Bracketed SFX/speaker captions and punctuation-only runs are still skipped.
|
||||
|
||||
|
Before Width: | Height: | Size: 9.0 KiB After Width: | Height: | Size: 8.8 KiB |
|
Before Width: | Height: | Size: 862 B After Width: | Height: | Size: 633 B |
|
Before Width: | Height: | Size: 1.6 KiB After Width: | Height: | Size: 1.4 KiB |
@@ -2,7 +2,7 @@
|
||||
"name": "subminer",
|
||||
"productName": "SubMiner",
|
||||
"desktopName": "SubMiner.desktop",
|
||||
"version": "0.18.0-beta.2",
|
||||
"version": "0.18.0-beta.4",
|
||||
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
|
||||
"packageManager": "bun@1.3.5",
|
||||
"main": "dist/main-entry.js",
|
||||
|
||||
@@ -21,17 +21,24 @@
|
||||
### Fixed
|
||||
- **Character Name Highlighting in Subtitles**
|
||||
- Fixed unspaced Japanese names (e.g. 東紫乃, 渡辺真奈美) being split at the wrong point, which left surnames like 東 and 渡辺 without their character portrait or hover lookup.
|
||||
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, or stripped entirely when grammar tagging misclassified the name token.
|
||||
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, misclassified by grammar tagging, or swallowed entirely by a longer generic dictionary match (e.g. ヨータ disappearing inside a false とヨー match).
|
||||
- Fixed a single unrecognized word in a subtitle line (like a stray interjection) causing character-name highlighting to drop for the whole line instead of just that word.
|
||||
- No action needed — existing data upgrades automatically the next time a matching name is seen.
|
||||
- **Known-Word Highlighting**
|
||||
- Words are no longer marked "known" (green) just because they share spelling with a known Anki card that actually teaches a different reading (e.g. 床 read as とこ no longer falsely matches a known 床/ゆか card).
|
||||
- Kanji words are also no longer marked known just because a different mined word happens to share their reading (e.g. 渓谷/けいこく no longer falsely matches a known 警告/けいこく card).
|
||||
- Single-kana grammar tokens (particles like よ, え) no longer borrow an unrelated card's reading and get falsely painted as known.
|
||||
- Stats sessions now correctly reflect known-word counts again after the reading-aware matching upgrade, instead of showing 0 everywhere.
|
||||
- **Annotation Highlighting Refinements**
|
||||
- Restored frequency/JLPT highlighting and vocabulary-stat counting for words like 確かに and やはり, which were wrongly treated as grammar noise.
|
||||
- Kanji nouns that MeCab tags as "non-independent" (e.g. 日, 点, 以外) also keep their highlighting and stats counting again.
|
||||
- Suffix-only tokens (e.g. さん, れる) are now excluded from JLPT/frequency highlighting by default to match how particles and interjections are treated; known-word highlighting for them still works, and this is configurable.
|
||||
- **Unparsed Subtitle Text**
|
||||
- Subtitle text the dictionary can't recognize (like a truncated verb form) is now still hoverable for lookup and correctly counted toward a sentence's difficulty, instead of showing as dead, non-interactive text.
|
||||
- **Kiku Manual Field Grouping**
|
||||
- Fixed the field-grouping dialog getting stuck invisible behind fullscreen video on Hyprland/Wayland, and failing silently on repeated attempts after the first use.
|
||||
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut.
|
||||
- Fixed a timed-out or failed grouping request leaving an invisible, stuck dialog covering the video; it now closes automatically so the overlay recovers.
|
||||
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut, and added a proper error message for the previously-silent case where the original card can no longer be loaded.
|
||||
- **Secondary Subtitles**
|
||||
- Karaoke-style secondary subtitles (common in opening/ending songs) no longer spam dozens of lines down the screen; repeated lines are now collapsed and the subtitle area is capped to a strip at the top.
|
||||
- **YouTube Extraction**
|
||||
@@ -59,7 +66,8 @@
|
||||
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
|
||||
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
|
||||
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
|
||||
- fix(stats): parse v3 reading-aware known-word cache in stats server by @ksyasuda in #149
|
||||
- fix(overlay): keep frequency/JLPT highlight for kanji non-independent nouns by @ksyasuda in #150
|
||||
- fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names by @ksyasuda in #151
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1213,6 +1213,102 @@ test('writeChangelogArtifacts appends contributor attribution and a new-contribu
|
||||
}
|
||||
});
|
||||
|
||||
test('writeChangelogArtifacts skips contributor attribution in GitHub Actions without a token', async () => {
|
||||
const { writeChangelogArtifacts } = await loadModule();
|
||||
const workspace = createWorkspace('release-notes-actions-no-token');
|
||||
const projectRoot = path.join(workspace, 'SubMiner');
|
||||
const originalActions = process.env.GITHUB_ACTIONS;
|
||||
const originalGhToken = process.env.GH_TOKEN;
|
||||
const originalGithubToken = process.env.GITHUB_TOKEN;
|
||||
const originalPath = process.env.PATH;
|
||||
const originalWarn = console.warn;
|
||||
const warnings: string[] = [];
|
||||
|
||||
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
|
||||
fs.writeFileSync(path.join(projectRoot, 'CHANGELOG.md'), '# Changelog\n', 'utf8');
|
||||
fs.writeFileSync(
|
||||
path.join(projectRoot, 'changes', '001.md'),
|
||||
['type: added', 'area: release', '', '- Added a feature.'].join('\n'),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
try {
|
||||
process.env.GITHUB_ACTIONS = 'true';
|
||||
delete process.env.GH_TOKEN;
|
||||
delete process.env.GITHUB_TOKEN;
|
||||
process.env.PATH = workspace;
|
||||
console.warn = (message?: unknown) => {
|
||||
warnings.push(String(message));
|
||||
};
|
||||
|
||||
writeChangelogArtifacts({
|
||||
cwd: projectRoot,
|
||||
version: '0.6.0',
|
||||
date: '2026-05-06',
|
||||
deps: { runClaude: defaultStubClaude().runClaude },
|
||||
});
|
||||
|
||||
assert.deepEqual(warnings, []);
|
||||
const releaseNotes = fs.readFileSync(
|
||||
path.join(projectRoot, 'release', 'release-notes.md'),
|
||||
'utf8',
|
||||
);
|
||||
assert.doesNotMatch(releaseNotes, /## What's Changed/);
|
||||
} finally {
|
||||
console.warn = originalWarn;
|
||||
if (originalActions === undefined) {
|
||||
delete process.env.GITHUB_ACTIONS;
|
||||
} else {
|
||||
process.env.GITHUB_ACTIONS = originalActions;
|
||||
}
|
||||
if (originalGhToken === undefined) {
|
||||
delete process.env.GH_TOKEN;
|
||||
} else {
|
||||
process.env.GH_TOKEN = originalGhToken;
|
||||
}
|
||||
if (originalGithubToken === undefined) {
|
||||
delete process.env.GITHUB_TOKEN;
|
||||
} else {
|
||||
process.env.GITHUB_TOKEN = originalGithubToken;
|
||||
}
|
||||
if (originalPath === undefined) {
|
||||
delete process.env.PATH;
|
||||
} else {
|
||||
process.env.PATH = originalPath;
|
||||
}
|
||||
fs.rmSync(workspace, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('shouldSkipDefaultContributionLookup skips GitHub Actions without a gh token', async () => {
|
||||
const { shouldSkipDefaultContributionLookup } = await loadModule();
|
||||
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: 'true',
|
||||
GH_TOKEN: undefined,
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: 'true',
|
||||
GH_TOKEN: 'ghs_test',
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: undefined,
|
||||
GH_TOKEN: undefined,
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('writeReleaseNotesForVersion preserves committed contributor attribution before installation', async () => {
|
||||
const { writeReleaseNotesForVersion } = await loadModule();
|
||||
const workspace = createWorkspace('release-notes-preserve-attribution');
|
||||
|
||||
@@ -345,6 +345,12 @@ function resolveFragmentRelativePath(fragmentPath: string, cwd: string): string
|
||||
return path.relative(cwd, fragmentPath).split(path.sep).join('/');
|
||||
}
|
||||
|
||||
export function shouldSkipDefaultContributionLookup(
|
||||
env: Partial<Record<'GITHUB_ACTIONS' | 'GH_TOKEN' | 'GITHUB_TOKEN', string>> = process.env,
|
||||
): boolean {
|
||||
return env.GITHUB_ACTIONS === 'true' && !env.GH_TOKEN && !env.GITHUB_TOKEN;
|
||||
}
|
||||
|
||||
// Walks git history + the GitHub API to attribute each released fragment to the
|
||||
// PR (and author) that introduced it. One git call and one gh call per fragment,
|
||||
// plus one gh call per unique author for the first-contribution check. Best
|
||||
@@ -354,6 +360,9 @@ function defaultResolveContributions(fragmentPaths: string[], cwd: string): Cont
|
||||
if (fragmentPaths.length === 0) {
|
||||
return [];
|
||||
}
|
||||
if (shouldSkipDefaultContributionLookup()) {
|
||||
return [];
|
||||
}
|
||||
|
||||
try {
|
||||
const slug = execFileSync(
|
||||
|
||||
@@ -703,8 +703,12 @@ export class AnkiIntegration {
|
||||
});
|
||||
}
|
||||
|
||||
isKnownWord(text: string, reading?: string): boolean {
|
||||
return this.knownWordCache.isKnownWord(text, reading);
|
||||
isKnownWord(
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
): boolean {
|
||||
return this.knownWordCache.isKnownWord(text, reading, options);
|
||||
}
|
||||
|
||||
getKnownWordMatchMode(): NPlusOneMatchMode {
|
||||
|
||||
@@ -734,6 +734,42 @@ test('KnownWordCacheManager disambiguates known words by note reading', async ()
|
||||
}
|
||||
});
|
||||
|
||||
test('KnownWordCacheManager suppresses reading-only matches when disallowed', async () => {
|
||||
const config: AnkiConnectConfig = {
|
||||
fields: {
|
||||
word: 'Word',
|
||||
},
|
||||
knownWords: {
|
||||
highlightEnabled: true,
|
||||
},
|
||||
};
|
||||
const { manager, clientState, cleanup } = createKnownWordCacheHarness(config);
|
||||
|
||||
try {
|
||||
clientState.findNotesResult = [1];
|
||||
clientState.notesInfoResult = [
|
||||
{
|
||||
noteId: 1,
|
||||
fields: {
|
||||
Word: { value: '警告' },
|
||||
'Word Reading': { value: 'けいこく' },
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
await manager.refresh(true);
|
||||
|
||||
// Reading-only match stays available for kana subtitle text…
|
||||
assert.equal(manager.isKnownWord('けいこく'), true);
|
||||
// …but a kanji token's reading (渓谷/けいこく) must not borrow 警告's.
|
||||
assert.equal(manager.isKnownWord('けいこく', undefined, { allowReadingOnlyMatch: false }), false);
|
||||
// Mined word texts still match regardless of the flag.
|
||||
assert.equal(manager.isKnownWord('警告', undefined, { allowReadingOnlyMatch: false }), true);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('KnownWordCacheManager does not match single-kana text by reading alone', async () => {
|
||||
const config: AnkiConnectConfig = {
|
||||
fields: {
|
||||
|
||||
@@ -142,7 +142,11 @@ export class KnownWordCacheManager {
|
||||
);
|
||||
}
|
||||
|
||||
isKnownWord(text: string, reading?: string): boolean {
|
||||
isKnownWord(
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
): boolean {
|
||||
if (!this.isKnownWordCacheEnabled()) {
|
||||
return false;
|
||||
}
|
||||
@@ -163,6 +167,14 @@ export class KnownWordCacheManager {
|
||||
);
|
||||
}
|
||||
|
||||
// Callers that look up a kanji token's reading (not subtitle text) must
|
||||
// opt out of the reading-only fallback: readingCounts holds readings of
|
||||
// every note including kanji words, so 渓谷's けいこく would match a
|
||||
// mined 警告/けいこく.
|
||||
if (options?.allowReadingOnlyMatch === false) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Reading-only fallback, except for single-kana text: particles and
|
||||
// interjections (よ, ね, え…) would otherwise borrow the reading of an
|
||||
// unrelated note (夜「よ」, 絵「え」) and count as known.
|
||||
|
||||
@@ -28,6 +28,7 @@ interface YomitanTokenInput {
|
||||
frequencyRank?: number;
|
||||
isNameMatch?: boolean;
|
||||
wordClasses?: string[];
|
||||
isUnparsedRun?: boolean;
|
||||
}
|
||||
|
||||
function makeDepsFromYomitanTokens(
|
||||
@@ -60,6 +61,7 @@ function makeDepsFromYomitanTokens(
|
||||
isNameMatch: token.isNameMatch ?? false,
|
||||
frequencyRank: token.frequencyRank,
|
||||
wordClasses: token.wordClasses,
|
||||
isUnparsedRun: token.isUnparsedRun,
|
||||
};
|
||||
});
|
||||
},
|
||||
@@ -3486,8 +3488,8 @@ test('tokenizeSubtitle keeps known-word highlight for exact non-independent kanj
|
||||
assert.equal(result.tokens?.[1]?.surface, '点');
|
||||
assert.equal(result.tokens?.[1]?.isKnown, true);
|
||||
assert.equal(result.tokens?.[1]?.isNPlusOneTarget, false);
|
||||
assert.equal(result.tokens?.[1]?.frequencyRank, undefined);
|
||||
assert.equal(result.tokens?.[1]?.jlptLevel, undefined);
|
||||
assert.equal(result.tokens?.[1]?.frequencyRank, 1384);
|
||||
assert.equal(result.tokens?.[1]?.jlptLevel, 'N3');
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle keeps mecab-tagged interjections tokenized while clearing annotation metadata', async () => {
|
||||
@@ -4223,6 +4225,38 @@ test('tokenizeSubtitle clears all annotations for explanatory pondering endings'
|
||||
);
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle ignores unparsed-run tokens for annotations and N+1', async () => {
|
||||
// もう いるぅ~!: the ぅ~ elongation has no Yomitan dictionary entry; it must
|
||||
// not become the sole N+1 candidate or receive frequency/JLPT annotations.
|
||||
const result = await tokenizeSubtitle(
|
||||
'もう いるぅ~!',
|
||||
makeDepsFromYomitanTokens(
|
||||
[
|
||||
{ surface: 'もう', reading: 'もう', headword: 'もう' },
|
||||
{ surface: 'いる', reading: 'いる', headword: 'いる' },
|
||||
{ surface: 'ぅ~', reading: '', headword: 'ぅ~', isUnparsedRun: true, frequencyRank: 999 },
|
||||
],
|
||||
{
|
||||
getFrequencyDictionaryEnabled: () => true,
|
||||
getJlptLevel: (text) => (text === 'ぅ~' ? 'N5' : null),
|
||||
isKnownWord: (text) => text === 'もう' || text === 'いる',
|
||||
getMinSentenceWordsForNPlusOne: () => 2,
|
||||
tokenizeWithMecab: async () => null,
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
const filler = result.tokens?.find((token) => token.surface === 'ぅ~');
|
||||
assert.ok(filler);
|
||||
assert.equal(filler?.isNPlusOneTarget, false);
|
||||
assert.equal(filler?.frequencyRank, undefined);
|
||||
assert.equal(filler?.jlptLevel, undefined);
|
||||
assert.equal(
|
||||
result.tokens?.some((token) => token.isNPlusOneTarget),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle keeps frequency for content-led merged token with trailing colloquial suffixes', async () => {
|
||||
const result = await tokenizeSubtitle(
|
||||
'張り切ってんじゃ',
|
||||
|
||||
@@ -33,6 +33,15 @@ type MecabTokenEnrichmentFn = (
|
||||
mecabTokens: MergedToken[] | null,
|
||||
) => Promise<MergedToken[]>;
|
||||
|
||||
// allowReadingOnlyMatch: false suppresses the cache's reading-only index for
|
||||
// lookups that pass a kanji token's reading as the text (see
|
||||
// computeTokenKnownStatus in annotation-stage).
|
||||
export type KnownWordLookupFn = (
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
) => boolean;
|
||||
|
||||
export interface TokenizerServiceDeps {
|
||||
getYomitanExt: () => Extension | null;
|
||||
getYomitanSession?: () => Session | null;
|
||||
@@ -42,7 +51,7 @@ export interface TokenizerServiceDeps {
|
||||
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
|
||||
getYomitanParserInitPromise: () => Promise<boolean> | null;
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: KnownWordLookupFn;
|
||||
getKnownWordMatchMode: () => NPlusOneMatchMode;
|
||||
getKnownWordsEnabled?: () => boolean;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
@@ -77,7 +86,7 @@ export interface TokenizerDepsRuntimeOptions {
|
||||
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
|
||||
getYomitanParserInitPromise: () => Promise<boolean> | null;
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: KnownWordLookupFn;
|
||||
getKnownWordMatchMode: () => NPlusOneMatchMode;
|
||||
getKnownWordsEnabled?: () => boolean;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
@@ -129,7 +138,7 @@ const INVISIBLE_SEPARATOR_PATTERN = /[\u200b\u2060\ufeff]/g;
|
||||
function getKnownWordLookup(
|
||||
deps: TokenizerServiceDeps,
|
||||
options: TokenizerAnnotationOptions,
|
||||
): (text: string, reading?: string) => boolean {
|
||||
): KnownWordLookupFn {
|
||||
if (!options.knownWordsEnabled && !options.nPlusOneEnabled) {
|
||||
return () => false;
|
||||
}
|
||||
@@ -734,6 +743,7 @@ async function parseWithYomitanInternalParser(
|
||||
isNPlusOneTarget: false,
|
||||
isNameMatch: token.isNameMatch ?? false,
|
||||
frequencyRank: token.frequencyRank,
|
||||
isUnparsedRun: token.isUnparsedRun === true ? true : undefined,
|
||||
};
|
||||
}),
|
||||
);
|
||||
|
||||
@@ -208,6 +208,48 @@ test('annotateTokens hides known-word marks while still using known words for N+
|
||||
assert.equal(result[2]?.isNPlusOneTarget, true);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations excludes unparsed-run tokens', () => {
|
||||
// 戻 from 「…とこ戻ろ…」: Yomitan had no dictionary entry, headword falls back
|
||||
// to the surface. Without the flag the token passes every other filter.
|
||||
const unflagged = makeToken({ surface: '戻', headword: '戻', reading: '' });
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(unflagged), false);
|
||||
|
||||
const flagged = makeToken({ surface: '戻', headword: '戻', reading: '', isUnparsedRun: true });
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(flagged), true);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(flagged), true);
|
||||
});
|
||||
|
||||
test('annotateTokens ignores unparsed-run tokens for annotations and N+1', () => {
|
||||
// もう いるぅ~!-style line: the elongation run is the only unknown token and
|
||||
// used to become the sole N+1 candidate despite having no dictionary entry.
|
||||
const tokens = [
|
||||
makeToken({ surface: 'みんな', headword: '皆', reading: 'みんな', startPos: 0, endPos: 3 }),
|
||||
makeToken({ surface: 'とこ', headword: '所', reading: 'とこ', startPos: 3, endPos: 5 }),
|
||||
makeToken({
|
||||
surface: '戻',
|
||||
headword: '戻',
|
||||
reading: '',
|
||||
startPos: 5,
|
||||
endPos: 6,
|
||||
isUnparsedRun: true,
|
||||
frequencyRank: 12,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
isKnownWord: (text) => text === '皆' || text === '所',
|
||||
getJlptLevel: (text) => (text === '戻' ? 'N5' : null),
|
||||
}),
|
||||
{ minSentenceWordsForNPlusOne: 2 },
|
||||
);
|
||||
|
||||
assert.equal(result[2]?.isNPlusOneTarget, false);
|
||||
assert.equal(result[2]?.jlptLevel, undefined);
|
||||
assert.equal(result[2]?.frequencyRank, undefined);
|
||||
});
|
||||
|
||||
test('annotateTokens falls back to reading for known-word matches when headword lookup misses', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
@@ -231,6 +273,29 @@ test('annotateTokens falls back to reading for known-word matches when headword
|
||||
assert.equal(result[0]?.frequencyRank, 1895);
|
||||
});
|
||||
|
||||
test('annotateTokens reading fallback does not match kanji tokens sharing a mined reading', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '渓谷',
|
||||
headword: '渓谷',
|
||||
reading: 'けいこく',
|
||||
endPos: 2,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
// Mimics the cache with a mined 警告/けいこく: けいこく matches through
|
||||
// the reading-only index unless the lookup opts out of it.
|
||||
isKnownWord: (text, _reading, options) =>
|
||||
text === '警告' || (options?.allowReadingOnlyMatch !== false && text === 'けいこく'),
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, false);
|
||||
});
|
||||
|
||||
test('annotateTokens ignores partial furigana readings for known-word fallback', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
@@ -575,7 +640,9 @@ test('shouldExcludeTokenFromSubtitleAnnotations keeps lexical tokens outside exp
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations still excludes lexical non-independent kanji nouns from non-known annotations', () => {
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations keeps lexical non-independent kanji nouns', () => {
|
||||
// Yomitan segments 以外/日/方 as standalone vocabulary tokens; MeCab's
|
||||
// 非自立 tag must only suppress kana grammar nouns (こと, もの, とき).
|
||||
const token = makeToken({
|
||||
surface: '以外',
|
||||
headword: '以外',
|
||||
@@ -586,6 +653,21 @@ test('shouldExcludeTokenFromSubtitleAnnotations still excludes lexical non-indep
|
||||
pos3: '副詞可能',
|
||||
});
|
||||
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), false);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations still excludes kana non-independent nouns', () => {
|
||||
const token = makeToken({
|
||||
surface: 'こと',
|
||||
headword: 'こと',
|
||||
reading: 'コト',
|
||||
partOfSpeech: PartOfSpeech.noun,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '一般',
|
||||
});
|
||||
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), true);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), true);
|
||||
});
|
||||
@@ -728,15 +810,6 @@ test('shouldExcludeTokenFromVocabularyPersistence excludes common frequency stop
|
||||
pos2: '代名詞|副助詞/並立助詞/終助詞',
|
||||
pos3: '一般|*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: '確かに',
|
||||
headword: '確かに',
|
||||
reading: 'たしかに',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞|助詞',
|
||||
pos2: '形容動詞語幹|副詞化',
|
||||
pos3: '*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: 'あなた',
|
||||
headword: '貴方',
|
||||
@@ -753,6 +826,34 @@ test('shouldExcludeTokenFromVocabularyPersistence excludes common frequency stop
|
||||
}
|
||||
});
|
||||
|
||||
test('content adverbs are not excluded from annotations or vocabulary persistence', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '確かに',
|
||||
headword: '確かに',
|
||||
reading: 'たしかに',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞|助詞',
|
||||
pos2: '形容動詞語幹|副詞化',
|
||||
pos3: '*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: 'やはり',
|
||||
headword: 'やはり',
|
||||
reading: 'ヤハリ',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '副詞',
|
||||
pos2: '一般',
|
||||
pos3: '*',
|
||||
}),
|
||||
];
|
||||
|
||||
for (const token of tokens) {
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false, token.surface);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), false, token.surface);
|
||||
}
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations excludes standalone して grammar helper fragments', () => {
|
||||
const token = makeToken({
|
||||
surface: 'して',
|
||||
@@ -1402,8 +1503,8 @@ test('annotateTokens keeps known-word status for non-independent kanji noun toke
|
||||
|
||||
assert.equal(result[0]?.isKnown, true);
|
||||
assert.equal(result[0]?.isNPlusOneTarget, false);
|
||||
assert.equal(result[0]?.frequencyRank, undefined);
|
||||
assert.equal(result[0]?.jlptLevel, undefined);
|
||||
assert.equal(result[0]?.frequencyRank, 1384);
|
||||
assert.equal(result[0]?.jlptLevel, 'N3');
|
||||
});
|
||||
|
||||
test('annotateTokens keeps known-word status for lexical non-independent kanji nouns', () => {
|
||||
@@ -1431,23 +1532,54 @@ test('annotateTokens keeps known-word status for lexical non-independent kanji n
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, true);
|
||||
assert.equal(result[0]?.frequencyRank, undefined);
|
||||
assert.equal(result[0]?.frequencyRank, 437);
|
||||
assert.equal(result[0]?.isNPlusOneTarget, false);
|
||||
});
|
||||
|
||||
test('annotateTokens clears all annotations for non-independent kanji noun tokens under unified gate', () => {
|
||||
test('annotateTokens keeps frequency for unknown non-independent kanji noun tokens', () => {
|
||||
// 日 in いい日だったな: MeCab tags it 名詞/非自立 but Yomitan segments it as
|
||||
// a standalone vocabulary token, so frequency highlighting must survive.
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '者',
|
||||
reading: 'もの',
|
||||
headword: '者',
|
||||
surface: '日',
|
||||
reading: 'ひ',
|
||||
headword: '日',
|
||||
partOfSpeech: PartOfSpeech.noun,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '副詞可能',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
frequencyRank: 718,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
getJlptLevel: (text) => (text === '日' ? 'N4' : null),
|
||||
}),
|
||||
{ minSentenceWordsForNPlusOne: 1 },
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, false);
|
||||
assert.equal(result[0]?.frequencyRank, 718);
|
||||
assert.equal(result[0]?.jlptLevel, 'N4');
|
||||
});
|
||||
|
||||
test('annotateTokens still clears annotations for kana non-independent noun tokens', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: 'こと',
|
||||
reading: 'こと',
|
||||
headword: 'こと',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '一般',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 475,
|
||||
endPos: 2,
|
||||
frequencyRank: 96,
|
||||
}),
|
||||
];
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
import { JlptLevel, MergedToken, NPlusOneMatchMode, PartOfSpeech } from '../../../types';
|
||||
import { shouldIgnoreJlptByTerm, shouldIgnoreJlptForMecabPos1 } from '../jlpt-token-filter';
|
||||
import {
|
||||
isKanjiNonIndependentNounToken,
|
||||
shouldExcludeTokenFromSubtitleAnnotations as sharedShouldExcludeTokenFromSubtitleAnnotations,
|
||||
stripSubtitleAnnotationMetadata as sharedStripSubtitleAnnotationMetadata,
|
||||
} from './subtitle-annotation-filter';
|
||||
@@ -25,7 +26,11 @@ const jlptLevelLookupCaches = new WeakMap<
|
||||
>();
|
||||
|
||||
export interface AnnotationStageDeps {
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: (
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
) => boolean;
|
||||
knownWordMatchMode: NPlusOneMatchMode;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
}
|
||||
@@ -94,23 +99,6 @@ function normalizePos2Tag(pos2: string | undefined): string {
|
||||
return typeof pos2 === 'string' ? pos2.trim() : '';
|
||||
}
|
||||
|
||||
function hasKanjiChar(text: string): boolean {
|
||||
for (const char of text) {
|
||||
const code = char.codePointAt(0);
|
||||
if (code === undefined) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
(code >= 0x3400 && code <= 0x4dbf) ||
|
||||
(code >= 0x4e00 && code <= 0x9fff) ||
|
||||
(code >= 0xf900 && code <= 0xfaff)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function isExcludedComponent(
|
||||
pos1: string | undefined,
|
||||
pos2: string | undefined,
|
||||
@@ -283,34 +271,6 @@ function isFrequencyExcludedByPos(
|
||||
);
|
||||
}
|
||||
|
||||
function shouldKeepFrequencyForNonIndependentKanjiNoun(
|
||||
token: MergedToken,
|
||||
pos1Exclusions: ReadonlySet<string>,
|
||||
): boolean {
|
||||
if (pos1Exclusions.has('名詞')) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const rank =
|
||||
typeof token.frequencyRank === 'number' && Number.isFinite(token.frequencyRank)
|
||||
? Math.max(1, Math.floor(token.frequencyRank))
|
||||
: null;
|
||||
if (rank === null) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const pos1Parts = splitNormalizedTagParts(normalizePos1Tag(token.pos1));
|
||||
const pos2Parts = splitNormalizedTagParts(normalizePos2Tag(token.pos2));
|
||||
if (pos1Parts.length !== 1 || pos2Parts.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
if (pos1Parts[0] !== '名詞' || pos2Parts[0] !== '非自立') {
|
||||
return false;
|
||||
}
|
||||
|
||||
return hasKanjiChar(token.surface) || hasKanjiChar(token.headword);
|
||||
}
|
||||
|
||||
export function shouldExcludeTokenFromVocabularyPersistence(
|
||||
token: MergedToken,
|
||||
options: Pick<AnnotationStageOptions, 'pos1Exclusions' | 'pos2Exclusions'> = {},
|
||||
@@ -320,7 +280,8 @@ export function shouldExcludeTokenFromVocabularyPersistence(
|
||||
|
||||
return (
|
||||
sharedShouldExcludeTokenFromSubtitleAnnotations(token, { pos1Exclusions, pos2Exclusions }) ||
|
||||
isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions)
|
||||
(isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions) &&
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions))
|
||||
);
|
||||
}
|
||||
|
||||
@@ -697,7 +658,7 @@ function resolveKnownWordReadingForMatch(
|
||||
|
||||
function computeTokenKnownStatus(
|
||||
token: MergedToken,
|
||||
isKnownWord: (text: string, reading?: string) => boolean,
|
||||
isKnownWord: AnnotationStageDeps['isKnownWord'],
|
||||
knownWordMatchMode: NPlusOneMatchMode,
|
||||
): boolean {
|
||||
const matchText = resolveKnownWordText(token.surface, token.headword, knownWordMatchMode);
|
||||
@@ -711,7 +672,14 @@ function computeTokenKnownStatus(
|
||||
return false;
|
||||
}
|
||||
|
||||
return fallbackReading !== matchText.trim() && isKnownWord(fallbackReading);
|
||||
// This fallback covers words mined in kana (token 大体, mined word だいたい),
|
||||
// so the reading must only match mined word texts — the cache's reading-only
|
||||
// index would let any kanji token match an unrelated note that shares its
|
||||
// reading (渓谷/けいこく vs a mined 警告/けいこく).
|
||||
return (
|
||||
fallbackReading !== matchText.trim() &&
|
||||
isKnownWord(fallbackReading, undefined, { allowReadingOnlyMatch: false })
|
||||
);
|
||||
}
|
||||
|
||||
function filterTokenFrequencyRank(
|
||||
@@ -721,7 +689,7 @@ function filterTokenFrequencyRank(
|
||||
): number | undefined {
|
||||
if (
|
||||
isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions) &&
|
||||
!shouldKeepFrequencyForNonIndependentKanjiNoun(token, pos1Exclusions)
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
@@ -146,13 +146,17 @@ test('emits unparsed non-caption text as a token with surface headword', () => {
|
||||
|
||||
const tokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
assert.deepEqual(
|
||||
tokens?.map((token) => ({ surface: token.surface, headword: token.headword })),
|
||||
tokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
headword: token.headword,
|
||||
isUnparsedRun: token.isUnparsedRun ?? false,
|
||||
})),
|
||||
[
|
||||
{ surface: 'みんな', headword: '皆' },
|
||||
{ surface: 'の', headword: 'の' },
|
||||
{ surface: 'とこ', headword: '所' },
|
||||
{ surface: '戻', headword: '戻' },
|
||||
{ surface: 'ろ', headword: '櫓' },
|
||||
{ surface: 'みんな', headword: '皆', isUnparsedRun: false },
|
||||
{ surface: 'の', headword: 'の', isUnparsedRun: false },
|
||||
{ surface: 'とこ', headword: '所', isUnparsedRun: false },
|
||||
{ surface: '戻', headword: '戻', isUnparsedRun: true },
|
||||
{ surface: 'ろ', headword: '櫓', isUnparsedRun: false },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
@@ -239,6 +239,7 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
headword: string,
|
||||
start: number,
|
||||
end: number,
|
||||
isUnparsedRun = false,
|
||||
): void => {
|
||||
tokens.push({
|
||||
surface,
|
||||
@@ -254,6 +255,7 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
const matchText = resolveKnownWordText(surface, headword, knownWordMatchMode);
|
||||
return matchText ? isKnownWord(matchText) : false;
|
||||
})(),
|
||||
...(isUnparsedRun ? { isUnparsedRun: true } : {}),
|
||||
});
|
||||
};
|
||||
|
||||
@@ -275,10 +277,11 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
previousToken.reading += combinedReading;
|
||||
previousToken.endPos = end;
|
||||
} else if (shouldEmitUnparsedRunAsToken(combinedSurface)) {
|
||||
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional). Keep it
|
||||
// as a token with its surface as headword so it stays hoverable and counts in
|
||||
// the n+1 math — matching what the embedded Yomitan actually returns.
|
||||
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end);
|
||||
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional, ぅ~
|
||||
// elongations). Keep it as a token with its surface as headword so it stays
|
||||
// hoverable, but flag it so annotation/N+1/vocab logic ignores it — there is
|
||||
// no dictionary entry behind it.
|
||||
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end, true);
|
||||
}
|
||||
} else {
|
||||
hasDictionaryMatch = true;
|
||||
|
||||
@@ -47,7 +47,6 @@ export const SUBTITLE_ANNOTATION_EXCLUDED_TERMS = new Set([
|
||||
'へえ',
|
||||
'ふう',
|
||||
'ほう',
|
||||
'やはり',
|
||||
'何か',
|
||||
'何だ',
|
||||
'何も',
|
||||
@@ -55,7 +54,6 @@ export const SUBTITLE_ANNOTATION_EXCLUDED_TERMS = new Set([
|
||||
'有る',
|
||||
'在る',
|
||||
'様',
|
||||
'確かに',
|
||||
'誰も',
|
||||
'貴方',
|
||||
'もんか',
|
||||
@@ -139,6 +137,46 @@ function resolvePos2Exclusions(options: SubtitleAnnotationFilterOptions = {}): R
|
||||
return resolveAnnotationPos2ExclusionSet(DEFAULT_ANNOTATION_POS2_EXCLUSION_CONFIG);
|
||||
}
|
||||
|
||||
function hasKanjiChar(text: string): boolean {
|
||||
for (const char of text) {
|
||||
const code = char.codePointAt(0);
|
||||
if (code === undefined) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
(code >= 0x3400 && code <= 0x4dbf) ||
|
||||
(code >= 0x4e00 && code <= 0x9fff) ||
|
||||
(code >= 0xf900 && code <= 0xfaff)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// Kanji-bearing non-independent nouns (日, 方, 上, …) are real vocabulary that
|
||||
// Yomitan segments as standalone tokens; MeCab's 非自立 tag exists to suppress
|
||||
// kana grammar nouns (こと, もの, とき) and must not hide these.
|
||||
export function isKanjiNonIndependentNounToken(
|
||||
token: MergedToken,
|
||||
pos1Exclusions: ReadonlySet<string>,
|
||||
): boolean {
|
||||
if (pos1Exclusions.has('名詞')) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const pos1Parts = splitNormalizedTagParts(normalizePosTag(token.pos1));
|
||||
const pos2Parts = splitNormalizedTagParts(normalizePosTag(token.pos2));
|
||||
if (pos1Parts.length !== 1 || pos2Parts.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
if (pos1Parts[0] !== '名詞' || pos2Parts[0] !== '非自立') {
|
||||
return false;
|
||||
}
|
||||
|
||||
return hasKanjiChar(token.surface) || hasKanjiChar(token.headword);
|
||||
}
|
||||
|
||||
function normalizeKana(text: string): string {
|
||||
const raw = text.trim();
|
||||
if (!raw) {
|
||||
@@ -436,6 +474,13 @@ export function shouldExcludeTokenFromSubtitleAnnotations(
|
||||
token: MergedToken,
|
||||
options: SubtitleAnnotationFilterOptions = {},
|
||||
): boolean {
|
||||
// No Yomitan dictionary entry backs this token (ぅ~ elongations, truncated
|
||||
// inflections) — it exists only to stay hoverable and must never receive
|
||||
// annotations or count in the N+1 math.
|
||||
if (token.isUnparsedRun === true) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const pos1Exclusions = resolvePos1Exclusions(options);
|
||||
const pos2Exclusions = resolvePos2Exclusions(options);
|
||||
const normalizedPos1 = normalizePosTag(token.pos1);
|
||||
@@ -447,7 +492,10 @@ export function shouldExcludeTokenFromSubtitleAnnotations(
|
||||
return true;
|
||||
}
|
||||
|
||||
if (isExcludedByTagSet(normalizedPos2, pos2Exclusions)) {
|
||||
if (
|
||||
isExcludedByTagSet(normalizedPos2, pos2Exclusions) &&
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
@@ -820,6 +820,135 @@ test('requestYomitanScanTokens keeps scanner metadata when parse spans agree', a
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens keeps scanner metadata for matching spans when parse segmentation has filler chunks', async () => {
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
if (script.includes('parseText')) {
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: 'や',
|
||||
reading: '',
|
||||
headwords: [[{ term: 'や' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'ほ',
|
||||
reading: '',
|
||||
headwords: [[{ term: '帆' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'っ ',
|
||||
reading: '',
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'ミナト',
|
||||
reading: '',
|
||||
headwords: [[{ term: 'ミナト' }]],
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
// The termsFind scanner skips the unmatched っ+space chunk, so its spans
|
||||
// do not line up 1:1 with the parseText segmentation above.
|
||||
return [
|
||||
{
|
||||
surface: 'や',
|
||||
reading: 'や',
|
||||
headword: 'や',
|
||||
headwordReading: 'や',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 57,
|
||||
},
|
||||
{
|
||||
surface: 'ほ',
|
||||
reading: 'ほ',
|
||||
headword: '帆',
|
||||
headwordReading: 'ほ',
|
||||
startPos: 1,
|
||||
endPos: 2,
|
||||
frequencyRank: 15414,
|
||||
},
|
||||
{
|
||||
surface: 'ミナト',
|
||||
reading: 'ミナト',
|
||||
headword: 'ミナト',
|
||||
headwordReading: 'みなと',
|
||||
startPos: 4,
|
||||
endPos: 7,
|
||||
isNameMatch: true,
|
||||
frequencyRank: 75133,
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const result = await requestYomitanScanTokens('やほっ ミナト', deps, {
|
||||
error: () => undefined,
|
||||
});
|
||||
|
||||
assert.deepEqual(result, [
|
||||
{
|
||||
surface: 'や',
|
||||
reading: 'や',
|
||||
headword: 'や',
|
||||
headwordReading: 'や',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 57,
|
||||
},
|
||||
{
|
||||
surface: 'ほ',
|
||||
reading: 'ほ',
|
||||
headword: '帆',
|
||||
headwordReading: 'ほ',
|
||||
startPos: 1,
|
||||
endPos: 2,
|
||||
frequencyRank: 15414,
|
||||
},
|
||||
{
|
||||
surface: 'っ ',
|
||||
reading: '',
|
||||
headword: 'っ ',
|
||||
startPos: 2,
|
||||
endPos: 4,
|
||||
isUnparsedRun: true,
|
||||
},
|
||||
{
|
||||
surface: 'ミナト',
|
||||
reading: 'ミナト',
|
||||
headword: 'ミナト',
|
||||
headwordReading: 'みなと',
|
||||
startPos: 4,
|
||||
endPos: 7,
|
||||
isNameMatch: true,
|
||||
frequencyRank: 75133,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens falls back to left-to-right termsFind scanning', async () => {
|
||||
const scripts: string[] = [];
|
||||
const deps = createDeps(async (script) => {
|
||||
@@ -1761,6 +1890,364 @@ test('requestYomitanScanTokens accepts SubMiner character entries with structure
|
||||
assert.equal((result as Array<{ isNameMatch?: boolean }>)[0]?.isNameMatch, true);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens greedily tokenizes character names before longer generic matches', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'美姫とヨータ',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
|
||||
|
||||
const nameEntry = (term: string, reading: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [
|
||||
{
|
||||
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
},
|
||||
],
|
||||
});
|
||||
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
|
||||
});
|
||||
|
||||
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
|
||||
if (action !== 'termsFind') {
|
||||
throw new Error(`unexpected action: ${action}`);
|
||||
}
|
||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||
if (text.startsWith('美姫')) {
|
||||
return { originalTextLength: 2, dictionaryEntries: [nameEntry('美姫', 'みき')] };
|
||||
}
|
||||
if (text.startsWith('とヨータ')) {
|
||||
// Greedy generic match: とヨー normalizes to とよう (渡洋). Without the
|
||||
// name pre-pass this consumes the ヨ of ヨータ.
|
||||
return {
|
||||
originalTextLength: 3,
|
||||
dictionaryEntries: [jmdictEntry('渡洋', 'とよう', 'とヨー'), jmdictEntry('と', 'と', 'と')],
|
||||
};
|
||||
}
|
||||
if (text.startsWith('ヨータ')) {
|
||||
return { originalTextLength: 3, dictionaryEntries: [nameEntry('ヨータ', 'よーた')] };
|
||||
}
|
||||
if (text === 'と') {
|
||||
return { originalTextLength: 1, dictionaryEntries: [jmdictEntry('と', 'と', 'と')] };
|
||||
}
|
||||
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||
});
|
||||
|
||||
assert.equal(Array.isArray(result), true);
|
||||
assert.deepEqual(
|
||||
(result as Array<Record<string, unknown>>).map(
|
||||
({ surface, headword, startPos, endPos, isNameMatch }) => ({
|
||||
surface,
|
||||
headword,
|
||||
startPos,
|
||||
endPos,
|
||||
isNameMatch,
|
||||
}),
|
||||
),
|
||||
[
|
||||
{ surface: '美姫', headword: '美姫', startPos: 0, endPos: 2, isNameMatch: true },
|
||||
{ surface: 'と', headword: 'と', startPos: 2, endPos: 3, isNameMatch: false },
|
||||
{ surface: 'ヨータ', headword: 'ヨータ', startPos: 3, endPos: 6, isNameMatch: true },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens lets a longer generic word beat a shorter name at the same position', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'空気変わって',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
|
||||
|
||||
const nameEntry = (term: string, reading: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [
|
||||
{
|
||||
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
},
|
||||
],
|
||||
});
|
||||
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
|
||||
});
|
||||
|
||||
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
|
||||
if (action !== 'termsFind') {
|
||||
throw new Error(`unexpected action: ${action}`);
|
||||
}
|
||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||
if (text.startsWith('空気')) {
|
||||
// A character named 空 matches here, but the generic 空気 is longer and
|
||||
// must win the position.
|
||||
return {
|
||||
originalTextLength: 2,
|
||||
dictionaryEntries: [nameEntry('空', 'くう'), jmdictEntry('空気', 'くうき', '空気')],
|
||||
};
|
||||
}
|
||||
if (text.startsWith('変わって')) {
|
||||
return {
|
||||
originalTextLength: 4,
|
||||
dictionaryEntries: [jmdictEntry('変わる', 'かわる', '変わって')],
|
||||
};
|
||||
}
|
||||
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||
});
|
||||
|
||||
assert.equal(Array.isArray(result), true);
|
||||
assert.deepEqual(
|
||||
(result as Array<Record<string, unknown>>).map(
|
||||
({ surface, headword, startPos, endPos, isNameMatch }) => ({
|
||||
surface,
|
||||
headword,
|
||||
startPos,
|
||||
endPos,
|
||||
isNameMatch,
|
||||
}),
|
||||
),
|
||||
[
|
||||
{ surface: '空気', headword: '空気', startPos: 0, endPos: 2, isNameMatch: false },
|
||||
{ surface: '変わって', headword: '変わる', startPos: 2, endPos: 6, isNameMatch: false },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens skips greedy name scan without an enabled character dictionary', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [{ name: 'JMdict', enabled: true }],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'アクア',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = false;/);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens replaces parseText segmentation where greedy name tokens re-segment', async () => {
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
if (script.includes('parseText')) {
|
||||
// parseText walks greedily too, so it merges と with ヨー into 渡洋 and
|
||||
// strands the タ.
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: '美姫',
|
||||
reading: 'みき',
|
||||
headwords: [[{ term: '美姫' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'とヨー',
|
||||
reading: 'とよう',
|
||||
headwords: [[{ term: '渡洋' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'タ',
|
||||
reading: '',
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
return [
|
||||
{
|
||||
surface: '美姫',
|
||||
reading: 'みき',
|
||||
headword: '美姫',
|
||||
headwordReading: 'みき',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
isNameMatch: true,
|
||||
},
|
||||
{
|
||||
surface: 'と',
|
||||
reading: 'と',
|
||||
headword: 'と',
|
||||
headwordReading: 'と',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
isNameMatch: false,
|
||||
},
|
||||
{
|
||||
surface: 'ヨータ',
|
||||
reading: 'ヨータ',
|
||||
headword: 'ヨータ',
|
||||
headwordReading: 'よーた',
|
||||
startPos: 3,
|
||||
endPos: 6,
|
||||
isNameMatch: true,
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const result = await requestYomitanScanTokens(
|
||||
'美姫とヨータ',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.deepEqual(result, [
|
||||
{
|
||||
surface: '美姫',
|
||||
reading: 'みき',
|
||||
headword: '美姫',
|
||||
headwordReading: 'みき',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
isNameMatch: true,
|
||||
},
|
||||
{
|
||||
surface: 'と',
|
||||
reading: 'と',
|
||||
headword: 'と',
|
||||
headwordReading: 'と',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
isNameMatch: false,
|
||||
},
|
||||
{
|
||||
surface: 'ヨータ',
|
||||
reading: 'ヨータ',
|
||||
headword: 'ヨータ',
|
||||
headwordReading: 'よーた',
|
||||
startPos: 3,
|
||||
endPos: 6,
|
||||
isNameMatch: true,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens preserves matched headword word classes', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
|
||||
@@ -57,6 +57,7 @@ export interface YomitanScanToken {
|
||||
isNameMatch?: boolean;
|
||||
frequencyRank?: number;
|
||||
wordClasses?: string[];
|
||||
isUnparsedRun?: boolean;
|
||||
}
|
||||
|
||||
interface YomitanProfileMetadata {
|
||||
@@ -73,6 +74,7 @@ export interface YomitanAddNoteResult {
|
||||
}
|
||||
|
||||
const DEFAULT_YOMITAN_SCAN_LENGTH = 40;
|
||||
const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
const yomitanProfileMetadataByWindow = new WeakMap<BrowserWindow, YomitanProfileMetadata>();
|
||||
const yomitanProfileDiagnosticsLoggedByWindow = new WeakSet<BrowserWindow>();
|
||||
const yomitanFrequencyCacheByWindow = new WeakMap<
|
||||
@@ -105,20 +107,89 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
);
|
||||
}
|
||||
|
||||
function hasSameTokenSpans(left: YomitanScanToken[], right: YomitanScanToken[]): boolean {
|
||||
if (left.length !== right.length) {
|
||||
return false;
|
||||
function scanTokenSpanKey(token: YomitanScanToken): string {
|
||||
return `${token.startPos}:${token.endPos}:${token.surface}`;
|
||||
}
|
||||
|
||||
// Maps a parse-selected token to the scanner-token shape carried out of the
|
||||
// parser runtime. Shared by both selectYomitanParseTokens fallback paths so the
|
||||
// projected fields stay in sync as the shape changes.
|
||||
function toYomitanScanToken(token: {
|
||||
surface: string;
|
||||
reading: string;
|
||||
headword: string;
|
||||
startPos: number;
|
||||
endPos: number;
|
||||
isUnparsedRun?: boolean;
|
||||
}): YomitanScanToken {
|
||||
return {
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
...(token.isUnparsedRun === true ? { isUnparsedRun: true } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
// parseText segmentation is authoritative (it emits filler chunks for text the
|
||||
// termsFind scanner skips), but only the termsFind scanner carries annotation
|
||||
// metadata (isNameMatch, frequencyRank, headwordReading, wordClasses). Graft
|
||||
// scanner tokens onto the parseText segmentation per matching span so one
|
||||
// unmatched chunk degrades only itself instead of dropping the whole line's
|
||||
// metadata.
|
||||
//
|
||||
// Exception: character-name tokens. The greedy name scan can re-segment text
|
||||
// around a name (e.g. とヨータ → と + ヨータ instead of とヨー + タ), so
|
||||
// parseText segmentation cannot be authoritative there. Each name span is
|
||||
// expanded until it aligns with token boundaries in both segmentations, then
|
||||
// the parse tokens inside are replaced with the scanner tokens.
|
||||
function mergeScannerTokensIntoParseTokens(
|
||||
parseScanTokens: YomitanScanToken[],
|
||||
scannerTokens: YomitanScanToken[],
|
||||
): YomitanScanToken[] {
|
||||
const scannerTokensBySpan = new Map<string, YomitanScanToken>();
|
||||
for (const token of scannerTokens) {
|
||||
scannerTokensBySpan.set(scanTokenSpanKey(token), token);
|
||||
}
|
||||
const graftedTokens = parseScanTokens.map(
|
||||
(token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token,
|
||||
);
|
||||
|
||||
const nameTokens = scannerTokens.filter((token) => token.isNameMatch === true);
|
||||
if (nameTokens.length === 0) {
|
||||
return graftedTokens;
|
||||
}
|
||||
|
||||
return left.every((token, index) => {
|
||||
const other = right[index];
|
||||
return (
|
||||
other !== undefined &&
|
||||
token.surface === other.surface &&
|
||||
token.startPos === other.startPos &&
|
||||
token.endPos === other.endPos
|
||||
);
|
||||
});
|
||||
const regions = nameTokens.map((token) => ({ start: token.startPos, end: token.endPos }));
|
||||
const allTokens = [...parseScanTokens, ...scannerTokens];
|
||||
let expanded = true;
|
||||
while (expanded) {
|
||||
expanded = false;
|
||||
for (const region of regions) {
|
||||
for (const token of allTokens) {
|
||||
const overlaps = token.startPos < region.end && token.endPos > region.start;
|
||||
const extendsBeyond = token.startPos < region.start || token.endPos > region.end;
|
||||
if (overlaps && extendsBeyond) {
|
||||
region.start = Math.min(region.start, token.startPos);
|
||||
region.end = Math.max(region.end, token.endPos);
|
||||
expanded = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isInsideNameRegion = (token: YomitanScanToken): boolean =>
|
||||
regions.some((region) => token.startPos >= region.start && token.endPos <= region.end);
|
||||
|
||||
const merged = graftedTokens.filter((token) => !isInsideNameRegion(token));
|
||||
for (const token of scannerTokens) {
|
||||
if (isInsideNameRegion(token)) {
|
||||
merged.push(token);
|
||||
}
|
||||
}
|
||||
merged.sort((a, b) => a.startPos - b.startPos || a.endPos - b.endPos);
|
||||
return merged;
|
||||
}
|
||||
|
||||
function makeTermReadingCacheKey(term: string, reading: string | null): string {
|
||||
@@ -1104,8 +1175,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
function normalizeWordClasses(headword) {
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
@@ -1146,7 +1216,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith("SubMiner Character Dictionary"));
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
@@ -1154,7 +1224,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/SubMiner Character Dictionary[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
@@ -1214,7 +1284,43 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
@@ -1261,6 +1367,7 @@ function buildYomitanScanningScript(
|
||||
profileIndex: number,
|
||||
scanLength: number,
|
||||
includeNameMatchMetadata: boolean,
|
||||
greedyNameScanEnabled: boolean,
|
||||
currentCharacterDictionaryMediaId: number | null,
|
||||
dictionaryPriorityByName: Record<string, number>,
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>,
|
||||
@@ -1287,6 +1394,7 @@ function buildYomitanScanningScript(
|
||||
});
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const includeNameMatchMetadata = ${includeNameMatchMetadata ? 'true' : 'false'};
|
||||
const greedyNameScanEnabled = ${greedyNameScanEnabled ? 'true' : 'false'};
|
||||
const currentCharacterDictionaryMediaId = ${
|
||||
currentCharacterDictionaryMediaId !== null
|
||||
? String(currentCharacterDictionaryMediaId)
|
||||
@@ -1297,26 +1405,17 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
const text = ${JSON.stringify(text)};
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const termsFindCache = new Map();
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const cacheKey = position + ":" + windowLength;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached) { return cached; }
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const result = await invoke("termsFind", { text: substring, details, optionsContext: { index: ${profileIndex} } });
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = substring.substring(0, originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
termsFindCache.set(cacheKey, result);
|
||||
return result;
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
@@ -1325,7 +1424,7 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + originalTextLength,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
@@ -1335,17 +1434,93 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return { token: tokenPayload, matchedLength: originalTextLength };
|
||||
return tokenPayload;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, ${scanLength});
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + ${scanLength});
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
while (i < text.length) {
|
||||
let attempt = await findTokenAt(i, ${scanLength});
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
// Cap the window at the next reserved name span so a generic match
|
||||
// cannot consume into it.
|
||||
const windowLength = nextNameToken ? Math.min(${scanLength}, nextNameToken.startPos - i) : ${scanLength};
|
||||
let attempt = await findTokenAt(i, windowLength);
|
||||
// Yomitan text normalization can consume characters (whitespace,
|
||||
// punctuation) beyond the matched term, leaving no headword whose
|
||||
// source equals the consumed text. Retry with shorter windows so a
|
||||
// valid prefix term (e.g. a character name before a paren) still
|
||||
// tokenizes instead of the position being skipped.
|
||||
let retryLength = Math.min(attempt.matchedLength, ${scanLength}) - 1;
|
||||
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
const retry = await findTokenAt(i, retryLength);
|
||||
if (retry.token) {
|
||||
@@ -1482,18 +1657,17 @@ export async function requestYomitanScanTokens(
|
||||
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedParseTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens =
|
||||
selectedParseTokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
})) ?? null;
|
||||
const parseScanTokens = selectedParseTokens?.map(toYomitanScanToken) ?? null;
|
||||
|
||||
const metadata = await requestYomitanProfileMetadata(parserWindow, logger);
|
||||
const profileIndex = metadata?.profileIndex ?? 0;
|
||||
const scanLength = metadata?.scanLength ?? DEFAULT_YOMITAN_SCAN_LENGTH;
|
||||
const includeNameMatchMetadata = options?.includeNameMatchMetadata === true;
|
||||
const greedyNameScanEnabled =
|
||||
includeNameMatchMetadata &&
|
||||
(metadata?.dictionaries ?? []).some((name) =>
|
||||
name.startsWith(CHARACTER_DICTIONARY_TITLE_PREFIX),
|
||||
);
|
||||
|
||||
try {
|
||||
const rawResult = await parserWindow.webContents.executeJavaScript(
|
||||
@@ -1501,7 +1675,8 @@ export async function requestYomitanScanTokens(
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
options?.includeNameMatchMetadata === true,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
typeof options?.currentCharacterDictionaryMediaId === 'number' &&
|
||||
Number.isFinite(options.currentCharacterDictionaryMediaId) &&
|
||||
options.currentCharacterDictionaryMediaId > 0
|
||||
@@ -1514,21 +1689,13 @@ export async function requestYomitanScanTokens(
|
||||
);
|
||||
if (isScanTokenArray(rawResult)) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return hasSameTokenSpans(parseScanTokens, rawResult) ? rawResult : parseScanTokens;
|
||||
return mergeScannerTokensIntoParseTokens(parseScanTokens, rawResult);
|
||||
}
|
||||
return rawResult;
|
||||
}
|
||||
if (Array.isArray(rawResult)) {
|
||||
const selectedTokens = selectYomitanParseTokens(rawResult, () => false, 'headword');
|
||||
return (
|
||||
selectedTokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
})) ?? null
|
||||
);
|
||||
return selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
}
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
|
||||
@@ -4652,7 +4652,8 @@ const {
|
||||
setYomitanParserInitPromise: (promise) => {
|
||||
appState.yomitanParserInitPromise = promise;
|
||||
},
|
||||
isKnownWord: (text, reading) => Boolean(appState.ankiIntegration?.isKnownWord(text, reading)),
|
||||
isKnownWord: (text, reading, options) =>
|
||||
Boolean(appState.ankiIntegration?.isKnownWord(text, reading, options)),
|
||||
recordLookup: (hit) => {
|
||||
ensureImmersionTrackerStarted();
|
||||
appState.immersionTracker?.recordLookup(hit);
|
||||
|
||||
@@ -37,8 +37,8 @@ export function createBuildTokenizerDepsMainHandler(deps: TokenizerMainDeps) {
|
||||
getYomitanParserInitPromise: () => deps.getYomitanParserInitPromise(),
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) =>
|
||||
deps.setYomitanParserInitPromise(promise),
|
||||
isKnownWord: (text: string, reading?: string) => {
|
||||
const hit = deps.isKnownWord(text, reading);
|
||||
isKnownWord: (text, reading, options) => {
|
||||
const hit = deps.isKnownWord(text, reading, options);
|
||||
deps.recordLookup(hit);
|
||||
return hit;
|
||||
},
|
||||
|
||||
@@ -40,6 +40,12 @@ export interface MergedToken {
|
||||
isMerged: boolean;
|
||||
isKnown: boolean;
|
||||
isNPlusOneTarget: boolean;
|
||||
/**
|
||||
* Text Yomitan had no dictionary entry for (e.g. ぅ~ elongation runs,
|
||||
* truncated inflections). Kept as a token so it stays hoverable, but
|
||||
* ignored by annotation, N+1, and vocabulary-stats logic.
|
||||
*/
|
||||
isUnparsedRun?: boolean;
|
||||
isNameMatch?: boolean;
|
||||
characterImage?: CharacterNameImage;
|
||||
jlptLevel?: JlptLevel;
|
||||
|
||||
|
Before Width: | Height: | Size: 6.5 KiB After Width: | Height: | Size: 6.2 KiB |