Compare commits
15 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
84c75f50aa
|
|||
|
8b838f2c7d
|
|||
|
db4139ba0b
|
|||
| 6c251502b3 | |||
| ae40934d3a | |||
|
cdb1475a54
|
|||
|
a2e49b369b
|
|||
| 7f13aed50a | |||
|
8b21a2bca8
|
|||
|
925413adfe
|
|||
| d0644ab2eb | |||
| d253710c2e | |||
|
c3df510e4f
|
|||
| 187f68e5b6 | |||
|
0e254cbbef
|
|
Before Width: | Height: | Size: 436 B After Width: | Height: | Size: 253 B |
|
Before Width: | Height: | Size: 549 B After Width: | Height: | Size: 366 B |
@@ -1,4 +1,4 @@
|
||||
type: fixed
|
||||
type: added
|
||||
area: mining
|
||||
|
||||
- Normalized generated card audio by default during media extraction, with `ankiConnect.media.normalizeAudio` available to keep raw source loudness when needed.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Removed 確かに and やはり from the annotation stop-word list so they get frequency/JLPT highlighting again and count toward vocabulary stats; the list now only covers interjections, pronouns, and grammar fragments.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Cover art is now fetched eagerly when a new series starts playing, instead of waiting for the first visit to its series detail page, so the stats timeline shows the best-guess AniList image right away. The stats covers endpoint also backfills missing series art in the background, so existing series without an image pick one up on the next stats page load.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: changed
|
||||
area: launcher
|
||||
|
||||
- Moved fzf previews below launcher menus so long titles and metadata have more horizontal room.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed character names being swallowed by longer generic dictionary matches during subtitle tokenization (e.g. ヨータ in 美姫とヨータ was lost because とヨー normalizes to とよう and matched 渡洋), so the name never tokenized and got no highlight, portrait, or hover lookup. Character-dictionary matches now claim their spans in a greedy pre-pass before the left-to-right scan, and re-segmented name spans override the parse tokenization on merge. A name only claims its span when no strictly longer generic word starts at the same position, so a character named 空 no longer splits 空気 (ties still go to the name). The pre-pass runs only when a SubMiner character dictionary is enabled in the active Yomitan profile.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Kanji-bearing nouns that MeCab tags as non-independent (非自立) — e.g. 日 in いい日だったな, 点, 以外 — now keep frequency/JLPT highlighting and count toward vocabulary stats. Yomitan segments them as standalone vocabulary tokens, so the MeCab POS filter only suppresses kana grammar nouns (こと, もの, とき) it was meant for.
|
||||
@@ -1,4 +1,7 @@
|
||||
type: fixed
|
||||
type: changed
|
||||
area: overlay
|
||||
|
||||
- Fixed words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" matching a known 床/ゆか "floor" card). The known-word cache now stores each card's word together with its reading and only matches when the token's reading agrees; cards without a reading field keep matching in any reading as before.
|
||||
- New reading-aware subtitle parsing and known-word matching: the known-word cache now stores each Anki card's word together with its reading (cache format v3), and a token only gets the known-word highlight when its parsed reading agrees with the card. The cache and the stats server upgrade automatically.
|
||||
- Fixes words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" no longer matches a known 床/ゆか "floor" card). Cards without a reading field keep matching in any reading as before.
|
||||
- Single-kana grammar tokens (よ in 全然いいよ, standalone え) no longer borrow the reading of an unrelated card (such as 夜 or 絵) and get painted as known: reading-only matching requires at least two kana, while single-kana cards still match by their word field.
|
||||
- Kanji words no longer count as known just because a different mined kanji word shares their reading (渓谷/けいこく no longer matches a known 警告/けいこく card). Words mined in kana form still highlight their kanji spelling in subtitles, and kana subtitle text still matches cards mined in kanji form.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed character-name annotations dropping for an entire subtitle line when it contained any chunk the dictionary scanner could not match (e.g. an interjection like やほっ before a name): scanner metadata is now merged per token into the parseText segmentation instead of being discarded on any span mismatch.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: changed
|
||||
area: overlay
|
||||
|
||||
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.
|
||||
@@ -1,4 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed subtitle text that Yomitan's parser cannot match (e.g. the truncated volitional in とこ戻ろ…) being rendered as plain, non-interactive text: it was invisible to hover/lookup and excluded from the n+1 word count, which could wrongly mark a sentence as n+1. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation; bracketed SFX/speaker captions and punctuation-only runs are still skipped.
|
||||
- Fixed subtitle text that Yomitan's parser cannot match (truncated inflections like the volitional in とこ戻ろ…, elongation runs like ぅ~/ぉ〜) being rendered as plain, non-interactive text with no hover lookup. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation, while being fully ignored by frequency/JLPT highlighting, the N+1 candidate math, and vocabulary stats. Bracketed SFX/speaker captions and punctuation-only runs are still skipped.
|
||||
|
||||
|
Before Width: | Height: | Size: 9.0 KiB After Width: | Height: | Size: 8.8 KiB |
|
Before Width: | Height: | Size: 862 B After Width: | Height: | Size: 633 B |
|
Before Width: | Height: | Size: 1.6 KiB After Width: | Height: | Size: 1.4 KiB |
@@ -56,7 +56,7 @@ export function showFzfFlatMenu(
|
||||
`--prompt=${prompt}`,
|
||||
'--delimiter=\t',
|
||||
'--with-nth=2',
|
||||
'--preview-window=right:50%:wrap',
|
||||
'--preview-window=down:50%:wrap',
|
||||
'--preview',
|
||||
previewCommand,
|
||||
];
|
||||
@@ -468,7 +468,7 @@ thumb=$(get_thumb)
|
||||
'--prompt=Select Video: ',
|
||||
'--delimiter=\t',
|
||||
'--with-nth=1',
|
||||
'--preview-window=right:50%:wrap',
|
||||
'--preview-window=down:50%:wrap',
|
||||
'--preview',
|
||||
previewCmd,
|
||||
],
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"name": "subminer",
|
||||
"productName": "SubMiner",
|
||||
"desktopName": "SubMiner.desktop",
|
||||
"version": "0.18.0-beta.1",
|
||||
"version": "0.18.0-beta.4",
|
||||
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
|
||||
"packageManager": "bun@1.3.5",
|
||||
"main": "dist/main-entry.js",
|
||||
|
||||
@@ -5,37 +5,51 @@
|
||||
## Highlights
|
||||
### Added
|
||||
- **Watch History Browser**
|
||||
- New `subminer -H` / `--history` command to browse your local watch history, replay the last episode, jump to the next one, or pick an episode via fzf/rofi.
|
||||
- New `subminer -H` / `--history` command lets you browse your local watch history, replay the last episode, jump to the next one, or pick an episode via fzf or rofi.
|
||||
- The rofi picker now shows AniList cover art for each show, making it easier to spot the right title at a glance.
|
||||
- **Card Audio Normalization**
|
||||
- Audio extracted for Anki cards is now volume-normalized by default, giving more consistent playback loudness across cards.
|
||||
- Prefer the original source volume? Disable it via the new `ankiConnect.media.normalizeAudio` setting.
|
||||
|
||||
### Changed
|
||||
- **New App Icon**
|
||||
- SubMiner now ships pixel-art submarine artwork, contributed by an anonymous community member.
|
||||
- SubMiner now ships pixel-art submarine artwork contributed by an anonymous community member.
|
||||
- Applied across the app icon, tray icon, notifications, README, docs site, and stats page.
|
||||
- **Launcher Preview Layout**
|
||||
- fzf previews in the launcher now sit below the menu instead of beside it, giving long titles and metadata more horizontal room.
|
||||
|
||||
### Fixed
|
||||
- **Character Name Highlighting in Subtitles**
|
||||
- Fixed unspaced Japanese names (e.g. 東紫乃, 渡辺真奈美) being split at the wrong point, which left surnames like 東 and 渡辺 without their character portrait or hover lookup.
|
||||
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling or by conflicting grammar tagging.
|
||||
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, misclassified by grammar tagging, or swallowed entirely by a longer generic dictionary match (e.g. ヨータ disappearing inside a false とヨー match).
|
||||
- Fixed a single unrecognized word in a subtitle line (like a stray interjection) causing character-name highlighting to drop for the whole line instead of just that word.
|
||||
- No action needed — existing data upgrades automatically the next time a matching name is seen.
|
||||
- **Known-Word Highlighting**
|
||||
- Words are no longer marked "known" (green) just because they share spelling with a known Anki card that actually teaches a different reading (e.g. 床 read as とこ no longer falsely matches a known 床/ゆか card).
|
||||
- Kanji words are also no longer marked known just because a different mined word happens to share their reading (e.g. 渓谷/けいこく no longer falsely matches a known 警告/けいこく card).
|
||||
- Single-kana grammar tokens (particles like よ, え) no longer borrow an unrelated card's reading and get falsely painted as known.
|
||||
- Stats sessions now correctly reflect known-word counts again after the reading-aware matching upgrade, instead of showing 0 everywhere.
|
||||
- **Annotation Highlighting Refinements**
|
||||
- Restored frequency/JLPT highlighting and vocabulary-stat counting for words like 確かに and やはり, which were wrongly treated as grammar noise.
|
||||
- Kanji nouns that MeCab tags as "non-independent" (e.g. 日, 点, 以外) also keep their highlighting and stats counting again.
|
||||
- Suffix-only tokens (e.g. さん, れる) are now excluded from JLPT/frequency highlighting by default to match how particles and interjections are treated; known-word highlighting for them still works, and this is configurable.
|
||||
- **Unparsed Subtitle Text**
|
||||
- Subtitle text the dictionary can't recognize (like a truncated verb form) is now still hoverable for lookup and correctly counted toward a sentence's difficulty, instead of showing as dead, non-interactive text.
|
||||
- **Kiku Manual Field Grouping**
|
||||
- Fixed the field-grouping dialog getting stuck invisible behind fullscreen video on Hyprland/Wayland, and failing silently on repeated attempts after the first use.
|
||||
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut.
|
||||
- Fixed a timed-out or failed grouping request leaving an invisible, stuck dialog covering the video; it now closes automatically so the overlay recovers.
|
||||
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut, and added a proper error message for the previously-silent case where the original card can no longer be loaded.
|
||||
- **Secondary Subtitles**
|
||||
- Karaoke-style secondary subtitles (common in opening/ending songs) no longer spam dozens of lines down the screen; repeated lines are now collapsed and the subtitle area is capped to a strip at the top.
|
||||
- **Card Audio Normalization**
|
||||
- Audio extracted for Anki cards is now volume-normalized by default for more consistent playback loudness.
|
||||
- If you prefer the original source volume, disable it via the new `ankiConnect.media.normalizeAudio` setting.
|
||||
- **YouTube Extraction**
|
||||
- Fixed direct YouTube stream extraction occasionally corrupting the stream URL and causing failed audio/video capture.
|
||||
- **Background Stats Server**
|
||||
- Launching SubMiner in the background now correctly auto-starts the stats server when enabled, and won't start a duplicate if one's already running.
|
||||
- **Stats Trend Charts**
|
||||
- All trend chart titles now show by default, with the ability to hide specific titles (remembered across sessions) and cap how many top titles a chart displays.
|
||||
- **Stats Cover Art**
|
||||
- Cover art now loads as soon as a series starts playing instead of waiting for your first visit to its detail page, so the stats timeline shows artwork right away.
|
||||
- Existing series missing art are backfilled automatically the next time you open the stats page.
|
||||
|
||||
## What's Changed
|
||||
|
||||
@@ -50,6 +64,10 @@
|
||||
- fix(stats): start stats server on background app launch by @ksyasuda in #144
|
||||
- fix(tokenizer): keep unparsed Yomitan tokens hoverable by @ksyasuda in #145
|
||||
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
|
||||
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
|
||||
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
|
||||
- fix(overlay): keep frequency/JLPT highlight for kanji non-independent nouns by @ksyasuda in #150
|
||||
- fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names by @ksyasuda in #151
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1213,6 +1213,102 @@ test('writeChangelogArtifacts appends contributor attribution and a new-contribu
|
||||
}
|
||||
});
|
||||
|
||||
test('writeChangelogArtifacts skips contributor attribution in GitHub Actions without a token', async () => {
|
||||
const { writeChangelogArtifacts } = await loadModule();
|
||||
const workspace = createWorkspace('release-notes-actions-no-token');
|
||||
const projectRoot = path.join(workspace, 'SubMiner');
|
||||
const originalActions = process.env.GITHUB_ACTIONS;
|
||||
const originalGhToken = process.env.GH_TOKEN;
|
||||
const originalGithubToken = process.env.GITHUB_TOKEN;
|
||||
const originalPath = process.env.PATH;
|
||||
const originalWarn = console.warn;
|
||||
const warnings: string[] = [];
|
||||
|
||||
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
|
||||
fs.writeFileSync(path.join(projectRoot, 'CHANGELOG.md'), '# Changelog\n', 'utf8');
|
||||
fs.writeFileSync(
|
||||
path.join(projectRoot, 'changes', '001.md'),
|
||||
['type: added', 'area: release', '', '- Added a feature.'].join('\n'),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
try {
|
||||
process.env.GITHUB_ACTIONS = 'true';
|
||||
delete process.env.GH_TOKEN;
|
||||
delete process.env.GITHUB_TOKEN;
|
||||
process.env.PATH = workspace;
|
||||
console.warn = (message?: unknown) => {
|
||||
warnings.push(String(message));
|
||||
};
|
||||
|
||||
writeChangelogArtifacts({
|
||||
cwd: projectRoot,
|
||||
version: '0.6.0',
|
||||
date: '2026-05-06',
|
||||
deps: { runClaude: defaultStubClaude().runClaude },
|
||||
});
|
||||
|
||||
assert.deepEqual(warnings, []);
|
||||
const releaseNotes = fs.readFileSync(
|
||||
path.join(projectRoot, 'release', 'release-notes.md'),
|
||||
'utf8',
|
||||
);
|
||||
assert.doesNotMatch(releaseNotes, /## What's Changed/);
|
||||
} finally {
|
||||
console.warn = originalWarn;
|
||||
if (originalActions === undefined) {
|
||||
delete process.env.GITHUB_ACTIONS;
|
||||
} else {
|
||||
process.env.GITHUB_ACTIONS = originalActions;
|
||||
}
|
||||
if (originalGhToken === undefined) {
|
||||
delete process.env.GH_TOKEN;
|
||||
} else {
|
||||
process.env.GH_TOKEN = originalGhToken;
|
||||
}
|
||||
if (originalGithubToken === undefined) {
|
||||
delete process.env.GITHUB_TOKEN;
|
||||
} else {
|
||||
process.env.GITHUB_TOKEN = originalGithubToken;
|
||||
}
|
||||
if (originalPath === undefined) {
|
||||
delete process.env.PATH;
|
||||
} else {
|
||||
process.env.PATH = originalPath;
|
||||
}
|
||||
fs.rmSync(workspace, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('shouldSkipDefaultContributionLookup skips GitHub Actions without a gh token', async () => {
|
||||
const { shouldSkipDefaultContributionLookup } = await loadModule();
|
||||
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: 'true',
|
||||
GH_TOKEN: undefined,
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: 'true',
|
||||
GH_TOKEN: 'ghs_test',
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
shouldSkipDefaultContributionLookup({
|
||||
GITHUB_ACTIONS: undefined,
|
||||
GH_TOKEN: undefined,
|
||||
GITHUB_TOKEN: undefined,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('writeReleaseNotesForVersion preserves committed contributor attribution before installation', async () => {
|
||||
const { writeReleaseNotesForVersion } = await loadModule();
|
||||
const workspace = createWorkspace('release-notes-preserve-attribution');
|
||||
|
||||
@@ -345,6 +345,12 @@ function resolveFragmentRelativePath(fragmentPath: string, cwd: string): string
|
||||
return path.relative(cwd, fragmentPath).split(path.sep).join('/');
|
||||
}
|
||||
|
||||
export function shouldSkipDefaultContributionLookup(
|
||||
env: Partial<Record<'GITHUB_ACTIONS' | 'GH_TOKEN' | 'GITHUB_TOKEN', string>> = process.env,
|
||||
): boolean {
|
||||
return env.GITHUB_ACTIONS === 'true' && !env.GH_TOKEN && !env.GITHUB_TOKEN;
|
||||
}
|
||||
|
||||
// Walks git history + the GitHub API to attribute each released fragment to the
|
||||
// PR (and author) that introduced it. One git call and one gh call per fragment,
|
||||
// plus one gh call per unique author for the first-contribution check. Best
|
||||
@@ -354,6 +360,9 @@ function defaultResolveContributions(fragmentPaths: string[], cwd: string): Cont
|
||||
if (fragmentPaths.length === 0) {
|
||||
return [];
|
||||
}
|
||||
if (shouldSkipDefaultContributionLookup()) {
|
||||
return [];
|
||||
}
|
||||
|
||||
try {
|
||||
const slug = execFileSync(
|
||||
|
||||
@@ -703,8 +703,12 @@ export class AnkiIntegration {
|
||||
});
|
||||
}
|
||||
|
||||
isKnownWord(text: string, reading?: string): boolean {
|
||||
return this.knownWordCache.isKnownWord(text, reading);
|
||||
isKnownWord(
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
): boolean {
|
||||
return this.knownWordCache.isKnownWord(text, reading, options);
|
||||
}
|
||||
|
||||
getKnownWordMatchMode(): NPlusOneMatchMode {
|
||||
|
||||
@@ -734,6 +734,85 @@ test('KnownWordCacheManager disambiguates known words by note reading', async ()
|
||||
}
|
||||
});
|
||||
|
||||
test('KnownWordCacheManager suppresses reading-only matches when disallowed', async () => {
|
||||
const config: AnkiConnectConfig = {
|
||||
fields: {
|
||||
word: 'Word',
|
||||
},
|
||||
knownWords: {
|
||||
highlightEnabled: true,
|
||||
},
|
||||
};
|
||||
const { manager, clientState, cleanup } = createKnownWordCacheHarness(config);
|
||||
|
||||
try {
|
||||
clientState.findNotesResult = [1];
|
||||
clientState.notesInfoResult = [
|
||||
{
|
||||
noteId: 1,
|
||||
fields: {
|
||||
Word: { value: '警告' },
|
||||
'Word Reading': { value: 'けいこく' },
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
await manager.refresh(true);
|
||||
|
||||
// Reading-only match stays available for kana subtitle text…
|
||||
assert.equal(manager.isKnownWord('けいこく'), true);
|
||||
// …but a kanji token's reading (渓谷/けいこく) must not borrow 警告's.
|
||||
assert.equal(manager.isKnownWord('けいこく', undefined, { allowReadingOnlyMatch: false }), false);
|
||||
// Mined word texts still match regardless of the flag.
|
||||
assert.equal(manager.isKnownWord('警告', undefined, { allowReadingOnlyMatch: false }), true);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('KnownWordCacheManager does not match single-kana text by reading alone', async () => {
|
||||
const config: AnkiConnectConfig = {
|
||||
fields: {
|
||||
word: 'Word',
|
||||
},
|
||||
knownWords: {
|
||||
highlightEnabled: true,
|
||||
},
|
||||
};
|
||||
const { manager, clientState, cleanup } = createKnownWordCacheHarness(config);
|
||||
|
||||
try {
|
||||
clientState.findNotesResult = [1, 2];
|
||||
clientState.notesInfoResult = [
|
||||
{
|
||||
noteId: 1,
|
||||
fields: {
|
||||
Word: { value: '夜' },
|
||||
'Word Reading': { value: 'よ' },
|
||||
},
|
||||
},
|
||||
{
|
||||
noteId: 2,
|
||||
fields: {
|
||||
Word: { value: 'え' },
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
await manager.refresh(true);
|
||||
|
||||
// よ must not count as known just because 夜 is read よ.
|
||||
assert.equal(manager.isKnownWord('よ'), false);
|
||||
assert.equal(manager.isKnownWord('ヨ'), false);
|
||||
assert.equal(manager.isKnownWord('夜'), true);
|
||||
assert.equal(manager.isKnownWord('夜', 'よ'), true);
|
||||
// A literal single-kana word entry still matches via the word map.
|
||||
assert.equal(manager.isKnownWord('え'), true);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('KnownWordCacheManager probes reading fields even with per-deck word fields configured', async () => {
|
||||
const config: AnkiConnectConfig = {
|
||||
fields: {
|
||||
|
||||
@@ -142,7 +142,11 @@ export class KnownWordCacheManager {
|
||||
);
|
||||
}
|
||||
|
||||
isKnownWord(text: string, reading?: string): boolean {
|
||||
isKnownWord(
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
): boolean {
|
||||
if (!this.isKnownWordCacheEnabled()) {
|
||||
return false;
|
||||
}
|
||||
@@ -163,7 +167,22 @@ export class KnownWordCacheManager {
|
||||
);
|
||||
}
|
||||
|
||||
return this.readingCounts.has(convertKatakanaToHiragana(normalized));
|
||||
// Callers that look up a kanji token's reading (not subtitle text) must
|
||||
// opt out of the reading-only fallback: readingCounts holds readings of
|
||||
// every note including kanji words, so 渓谷's けいこく would match a
|
||||
// mined 警告/けいこく.
|
||||
if (options?.allowReadingOnlyMatch === false) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Reading-only fallback, except for single-kana text: particles and
|
||||
// interjections (よ, ね, え…) would otherwise borrow the reading of an
|
||||
// unrelated note (夜「よ」, 絵「え」) and count as known.
|
||||
const hiragana = convertKatakanaToHiragana(normalized);
|
||||
if ([...hiragana].length === 1) {
|
||||
return false;
|
||||
}
|
||||
return this.readingCounts.has(hiragana);
|
||||
}
|
||||
|
||||
refresh(force = false): Promise<void> {
|
||||
|
||||
@@ -301,6 +301,7 @@ function createMockTracker(
|
||||
{ epochDay: Math.floor(Date.now() / 86_400_000) - 1, totalActiveMin: 30 },
|
||||
{ epochDay: Math.floor(Date.now() / 86_400_000), totalActiveMin: 45 },
|
||||
],
|
||||
ensureAnimeCoverArt: async () => false,
|
||||
getAnimeCoverArt: async (animeId: number) =>
|
||||
animeId === 1
|
||||
? {
|
||||
@@ -520,6 +521,44 @@ describe('stats server API routes', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('GET /api/stats/sessions enriches known-word metrics from a v3 reading-aware cache', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
const cachePath = path.join(dir, 'known-words.json');
|
||||
fs.writeFileSync(
|
||||
cachePath,
|
||||
JSON.stringify({
|
||||
version: 3,
|
||||
refreshedAtMs: 1,
|
||||
scope: 'deck:test',
|
||||
notes: {
|
||||
'101': [{ word: 'する', reading: 'する' }],
|
||||
'102': [{ word: '猫', reading: null }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const app = createStatsApp(
|
||||
createMockTracker({
|
||||
getSessionWordsByLine: async (sessionId: number) =>
|
||||
sessionId === 1
|
||||
? [
|
||||
{ lineIndex: 1, headword: 'する', occurrenceCount: 2 },
|
||||
{ lineIndex: 2, headword: '未知', occurrenceCount: 1 },
|
||||
]
|
||||
: [],
|
||||
}),
|
||||
{ knownWordCachePath: cachePath },
|
||||
);
|
||||
|
||||
const res = await app.request('/api/stats/sessions?limit=5');
|
||||
assert.equal(res.status, 200);
|
||||
const body = await res.json();
|
||||
const first = body[0];
|
||||
assert.equal(first.knownWordsSeen, 2);
|
||||
assert.equal(first.knownWordRate, 66.7);
|
||||
});
|
||||
});
|
||||
|
||||
it('GET /api/stats/sessions/:id/events forwards event type filters to the tracker', async () => {
|
||||
let seenSessionId = 0;
|
||||
let seenLimit = 0;
|
||||
@@ -994,8 +1033,9 @@ describe('stats server API routes', () => {
|
||||
assert.equal(res.status, 404);
|
||||
});
|
||||
|
||||
it('POST /api/stats/covers batches stored cover art without fetching missing art', async () => {
|
||||
it('POST /api/stats/covers batches stored cover art and backfills missing anime art in the background', async () => {
|
||||
let ensureCoverArtCalls = 0;
|
||||
const ensureAnimeCoverArtCalls: number[] = [];
|
||||
const app = createStatsApp(
|
||||
createMockTracker({
|
||||
getCoverArt: async (videoId: number) =>
|
||||
@@ -1015,6 +1055,10 @@ describe('stats server API routes', () => {
|
||||
ensureCoverArtCalls += 1;
|
||||
return true;
|
||||
},
|
||||
ensureAnimeCoverArt: async (animeId: number) => {
|
||||
ensureAnimeCoverArtCalls.push(animeId);
|
||||
return false;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
@@ -1042,6 +1086,68 @@ describe('stats server API routes', () => {
|
||||
},
|
||||
});
|
||||
assert.equal(ensureCoverArtCalls, 0);
|
||||
assert.deepEqual(ensureAnimeCoverArtCalls, [99999]);
|
||||
});
|
||||
|
||||
it('POST /api/stats/covers limits concurrent missing anime cover backfills', async () => {
|
||||
let activeBackfills = 0;
|
||||
let maxActiveBackfills = 0;
|
||||
const pendingBackfills: Array<() => void> = [];
|
||||
const app = createStatsApp(
|
||||
createMockTracker({
|
||||
getAnimeCoverArt: async () => null,
|
||||
ensureAnimeCoverArt: async () => {
|
||||
activeBackfills += 1;
|
||||
maxActiveBackfills = Math.max(maxActiveBackfills, activeBackfills);
|
||||
await new Promise<void>((resolve) => {
|
||||
pendingBackfills.push(resolve);
|
||||
});
|
||||
activeBackfills -= 1;
|
||||
return false;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const res = await app.request('/api/stats/covers', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ animeIds: [101, 102, 103, 104, 105] }),
|
||||
});
|
||||
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(maxActiveBackfills, 3);
|
||||
for (const resolveBackfill of pendingBackfills) {
|
||||
resolveBackfill();
|
||||
}
|
||||
});
|
||||
|
||||
it('GET /api/stats/anime/:animeId/cover fetches missing art before serving', async () => {
|
||||
let fetched = false;
|
||||
const app = createStatsApp(
|
||||
createMockTracker({
|
||||
getAnimeCoverArt: async () =>
|
||||
fetched
|
||||
? {
|
||||
videoId: 1,
|
||||
anilistId: 21858,
|
||||
coverUrl: 'https://example.com/cover.jpg',
|
||||
coverBlob: Buffer.from([0xff, 0xd8, 0xff, 0xd9]),
|
||||
titleRomaji: 'Little Witch Academia',
|
||||
titleEnglish: 'Little Witch Academia',
|
||||
episodesTotal: 25,
|
||||
fetchedAtMs: Date.now(),
|
||||
}
|
||||
: null,
|
||||
ensureAnimeCoverArt: async () => {
|
||||
fetched = true;
|
||||
return true;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const res = await app.request('/api/stats/anime/1/cover');
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(res.headers.get('content-type'), 'image/jpeg');
|
||||
});
|
||||
|
||||
it('GET /api/stats/anime/:animeId/words returns top words for an anime', async () => {
|
||||
|
||||
@@ -4041,3 +4041,91 @@ test('markActiveVideoWatched returns false when no active session', async () =>
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('handleMediaChange prefetches cover art at session start', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor({ dbPath });
|
||||
|
||||
const fetchedVideoIds: number[] = [];
|
||||
tracker.setCoverArtFetcher({
|
||||
fetchIfMissing: async (_db, videoId) => {
|
||||
fetchedVideoIds.push(videoId);
|
||||
return false;
|
||||
},
|
||||
});
|
||||
|
||||
tracker.handleMediaChange('/tmp/Little Witch Academia S02E05.mkv', 'Episode 5');
|
||||
await waitForPendingAnimeMetadata(tracker);
|
||||
await waitForCondition(() => fetchedVideoIds.length > 0);
|
||||
|
||||
const privateApi = tracker as unknown as {
|
||||
sessionState: { videoId: number } | null;
|
||||
};
|
||||
assert.deepEqual(fetchedVideoIds, [privateApi.sessionState?.videoId]);
|
||||
} finally {
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('ensureAnimeCoverArt fetches art via the latest video of the anime', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor({ dbPath });
|
||||
const privateApi = tracker as unknown as { db: DatabaseSync };
|
||||
|
||||
privateApi.db.exec(`
|
||||
INSERT INTO imm_anime (
|
||||
anime_id,
|
||||
normalized_title_key,
|
||||
canonical_title,
|
||||
CREATED_DATE,
|
||||
LAST_UPDATE_DATE
|
||||
) VALUES (
|
||||
1,
|
||||
'little witch academia',
|
||||
'Little Witch Academia',
|
||||
1000,
|
||||
1000
|
||||
);
|
||||
INSERT INTO imm_videos (
|
||||
video_id,
|
||||
video_key,
|
||||
canonical_title,
|
||||
source_type,
|
||||
duration_ms,
|
||||
anime_id,
|
||||
CREATED_DATE,
|
||||
LAST_UPDATE_DATE
|
||||
) VALUES
|
||||
(1, 'local:/tmp/lwa-1.mkv', 'Little Witch Academia S01E01', 1, 0, 1, 1000, 1000),
|
||||
(2, 'local:/tmp/lwa-2.mkv', 'Little Witch Academia S01E02', 1, 0, 1, 1000, 1000);
|
||||
`);
|
||||
|
||||
const fetchedVideoIds: number[] = [];
|
||||
tracker.setCoverArtFetcher({
|
||||
fetchIfMissing: async (_db, videoId) => {
|
||||
fetchedVideoIds.push(videoId);
|
||||
return false;
|
||||
},
|
||||
});
|
||||
|
||||
const result = await tracker.ensureAnimeCoverArt(1);
|
||||
assert.equal(result, false);
|
||||
assert.deepEqual(fetchedVideoIds, [2]);
|
||||
|
||||
const missing = await tracker.ensureAnimeCoverArt(999);
|
||||
assert.equal(missing, false);
|
||||
assert.deepEqual(fetchedVideoIds, [2]);
|
||||
} finally {
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -854,6 +854,22 @@ export class ImmersionTrackerService {
|
||||
this.coverArtFetcher = fetcher;
|
||||
}
|
||||
|
||||
async ensureAnimeCoverArt(animeId: number): Promise<boolean> {
|
||||
const existing = await this.getAnimeCoverArt(animeId);
|
||||
if (existing?.coverBlob) {
|
||||
return true;
|
||||
}
|
||||
const row = this.db
|
||||
.prepare(
|
||||
'SELECT video_id AS videoId FROM imm_videos WHERE anime_id = ? ORDER BY video_id DESC LIMIT 1',
|
||||
)
|
||||
.get(animeId) as { videoId: number } | undefined;
|
||||
if (!row?.videoId) {
|
||||
return false;
|
||||
}
|
||||
return this.ensureCoverArt(row.videoId);
|
||||
}
|
||||
|
||||
async ensureCoverArt(videoId: number): Promise<boolean> {
|
||||
const existing = await this.getCoverArt(videoId);
|
||||
if (existing?.coverBlob) {
|
||||
@@ -879,8 +895,10 @@ export class ImmersionTrackerService {
|
||||
}
|
||||
|
||||
const fetchPromise = (async () => {
|
||||
const detail = getMediaDetail(this.db, videoId);
|
||||
const canonicalTitle = detail?.canonicalTitle?.trim();
|
||||
const titleRow = this.db
|
||||
.prepare('SELECT canonical_title AS canonicalTitle FROM imm_videos WHERE video_id = ?')
|
||||
.get(videoId) as { canonicalTitle: string | null } | undefined;
|
||||
const canonicalTitle = titleRow?.canonicalTitle?.trim();
|
||||
if (!canonicalTitle) {
|
||||
return false;
|
||||
}
|
||||
@@ -1342,6 +1360,9 @@ export class ImmersionTrackerService {
|
||||
} else if (!this.hasJellyfinMetadata(sessionInfo.videoId)) {
|
||||
this.captureAnimeMetadataAsync(sessionInfo.videoId, normalizedPath, normalizedTitle || null);
|
||||
}
|
||||
if (!youtubeVideoId) {
|
||||
this.prefetchCoverArtAsync(sessionInfo.videoId);
|
||||
}
|
||||
this.captureVideoMetadataAsync(sessionInfo.videoId, sourceType, normalizedPath);
|
||||
}
|
||||
|
||||
@@ -1924,6 +1945,24 @@ export class ImmersionTrackerService {
|
||||
});
|
||||
}
|
||||
|
||||
// Fetch cover art eagerly at session start (after anime metadata parsing
|
||||
// settles) so new series show art on the stats timeline without requiring a
|
||||
// visit to the series detail page first.
|
||||
private prefetchCoverArtAsync(videoId: number): void {
|
||||
const pendingMetadata = this.pendingAnimeMetadataUpdates.get(videoId);
|
||||
void (async () => {
|
||||
try {
|
||||
await pendingMetadata;
|
||||
if (this.isDestroyed) {
|
||||
return;
|
||||
}
|
||||
await this.ensureCoverArt(videoId);
|
||||
} catch (error) {
|
||||
this.logger.warn('Unable to prefetch cover art', (error as Error).message);
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
private updateVideoTitleForActiveSession(canonicalTitle: string): void {
|
||||
if (!this.sessionState) return;
|
||||
updateVideoTitleRecord(this.db, this.sessionState.videoId, canonicalTitle);
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
import type { Hono } from 'hono';
|
||||
import type { ImmersionTrackerService } from './immersion-tracker-service.js';
|
||||
|
||||
type StatsCoverImagePayload = {
|
||||
contentType: string;
|
||||
dataUrl: string;
|
||||
} | null;
|
||||
|
||||
type StatsCoverBatchBody = {
|
||||
animeIds?: unknown;
|
||||
videoIds?: unknown;
|
||||
};
|
||||
|
||||
const MAX_BACKGROUND_ANIME_COVER_FETCHES = 3;
|
||||
|
||||
function parseIntQuery(raw: string | undefined, fallback: number, maxLimit?: number): number {
|
||||
if (raw === undefined) return fallback;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n < 0) {
|
||||
return fallback;
|
||||
}
|
||||
const parsed = Math.floor(n);
|
||||
return maxLimit === undefined ? parsed : Math.min(parsed, maxLimit);
|
||||
}
|
||||
|
||||
function parsePositiveIdList(raw: unknown, maxItems = 100): number[] {
|
||||
if (!Array.isArray(raw)) return [];
|
||||
|
||||
const ids = new Set<number>();
|
||||
for (const rawId of raw) {
|
||||
const id = typeof rawId === 'number' ? rawId : typeof rawId === 'string' ? Number(rawId) : NaN;
|
||||
if (Number.isFinite(id) && id > 0) {
|
||||
ids.add(Math.floor(id));
|
||||
if (ids.size >= maxItems) break;
|
||||
}
|
||||
}
|
||||
|
||||
return Array.from(ids).sort((a, b) => a - b);
|
||||
}
|
||||
|
||||
function coverImagePayload(
|
||||
art: { coverBlob?: Uint8Array | null } | null | undefined,
|
||||
): StatsCoverImagePayload {
|
||||
if (!art?.coverBlob) return null;
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
const contentType = detectImageContentType(bytes);
|
||||
return {
|
||||
contentType,
|
||||
dataUrl: `data:${contentType};base64,${Buffer.from(bytes).toString('base64')}`,
|
||||
};
|
||||
}
|
||||
|
||||
function detectImageContentType(bytes: Uint8Array): string {
|
||||
if (
|
||||
bytes.length >= 8 &&
|
||||
bytes[0] === 0x89 &&
|
||||
bytes[1] === 0x50 &&
|
||||
bytes[2] === 0x4e &&
|
||||
bytes[3] === 0x47
|
||||
) {
|
||||
return 'image/png';
|
||||
}
|
||||
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) {
|
||||
return 'image/jpeg';
|
||||
}
|
||||
if (
|
||||
bytes.length >= 12 &&
|
||||
bytes[0] === 0x52 &&
|
||||
bytes[1] === 0x49 &&
|
||||
bytes[2] === 0x46 &&
|
||||
bytes[3] === 0x46 &&
|
||||
bytes[8] === 0x57 &&
|
||||
bytes[9] === 0x45 &&
|
||||
bytes[10] === 0x42 &&
|
||||
bytes[11] === 0x50
|
||||
) {
|
||||
return 'image/webp';
|
||||
}
|
||||
return 'application/octet-stream';
|
||||
}
|
||||
|
||||
function createLimitedTaskRunner(maxConcurrentTasks: number): (task: () => Promise<void>) => void {
|
||||
const queue: Array<() => Promise<void>> = [];
|
||||
let activeTasks = 0;
|
||||
|
||||
const drain = (): void => {
|
||||
while (activeTasks < maxConcurrentTasks && queue.length > 0) {
|
||||
const task = queue.shift();
|
||||
if (!task) return;
|
||||
activeTasks += 1;
|
||||
void task()
|
||||
.catch(() => {})
|
||||
.finally(() => {
|
||||
activeTasks -= 1;
|
||||
drain();
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
return (task: () => Promise<void>): void => {
|
||||
queue.push(task);
|
||||
drain();
|
||||
};
|
||||
}
|
||||
|
||||
export function registerStatsCoverRoutes(app: Hono, tracker: ImmersionTrackerService): void {
|
||||
const enqueueAnimeCoverBackfill = createLimitedTaskRunner(MAX_BACKGROUND_ANIME_COVER_FETCHES);
|
||||
|
||||
app.post('/api/stats/covers', async (c) => {
|
||||
const body = (await c.req.json().catch(() => null)) as StatsCoverBatchBody | null;
|
||||
const animeIds = parsePositiveIdList(body?.animeIds);
|
||||
const videoIds = parsePositiveIdList(body?.videoIds);
|
||||
const anime: Record<number, StatsCoverImagePayload> = {};
|
||||
const media: Record<number, StatsCoverImagePayload> = {};
|
||||
|
||||
await Promise.all(
|
||||
animeIds.map(async (animeId) => {
|
||||
const art = await tracker.getAnimeCoverArt(animeId);
|
||||
if (!art?.coverBlob) {
|
||||
enqueueAnimeCoverBackfill(async () => {
|
||||
await tracker.ensureAnimeCoverArt(animeId);
|
||||
});
|
||||
}
|
||||
anime[animeId] = coverImagePayload(art);
|
||||
}),
|
||||
);
|
||||
await Promise.all(
|
||||
videoIds.map(async (videoId) => {
|
||||
media[videoId] = coverImagePayload(await tracker.getCoverArt(videoId));
|
||||
}),
|
||||
);
|
||||
|
||||
return c.json({ anime, media });
|
||||
});
|
||||
|
||||
app.get('/api/stats/anime/:animeId/cover', async (c) => {
|
||||
const animeId = parseIntQuery(c.req.param('animeId'), 0);
|
||||
if (animeId <= 0) return c.body(null, 404);
|
||||
let art = await tracker.getAnimeCoverArt(animeId);
|
||||
if (!art?.coverBlob) {
|
||||
await tracker.ensureAnimeCoverArt(animeId);
|
||||
art = await tracker.getAnimeCoverArt(animeId);
|
||||
}
|
||||
if (!art?.coverBlob) return c.body(null, 404);
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
return new Response(bytes, {
|
||||
headers: {
|
||||
'Content-Type': detectImageContentType(bytes),
|
||||
'Cache-Control': 'public, max-age=86400',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
app.get('/api/stats/media/:videoId/cover', async (c) => {
|
||||
const videoId = parseIntQuery(c.req.param('videoId'), 0);
|
||||
if (videoId <= 0) return c.body(null, 404);
|
||||
let art = await tracker.getCoverArt(videoId);
|
||||
if (!art?.coverBlob) {
|
||||
await tracker.ensureCoverArt(videoId);
|
||||
art = await tracker.getCoverArt(videoId);
|
||||
}
|
||||
if (!art?.coverBlob) return c.body(null, 404);
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
return new Response(bytes, {
|
||||
headers: {
|
||||
'Content-Type': detectImageContentType(bytes),
|
||||
'Cache-Control': 'public, max-age=604800',
|
||||
},
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
} from '../../anki-field-config.js';
|
||||
import { resolveAnimatedImageLeadInSeconds } from '../../anki-integration/animated-image-sync.js';
|
||||
import type { AnilistRateLimiter } from './anilist/rate-limiter.js';
|
||||
import { registerStatsCoverRoutes } from './stats-cover-routes.js';
|
||||
import {
|
||||
resolveRetimedSecondarySubtitleTextFromSidecar,
|
||||
resolveSecondarySubtitleTextFromSidecar,
|
||||
@@ -51,16 +52,6 @@ type StatsExcludedWordPayload = {
|
||||
reading: string;
|
||||
};
|
||||
|
||||
type StatsCoverImagePayload = {
|
||||
contentType: string;
|
||||
dataUrl: string;
|
||||
} | null;
|
||||
|
||||
type StatsCoverBatchBody = {
|
||||
animeIds?: unknown;
|
||||
videoIds?: unknown;
|
||||
};
|
||||
|
||||
function parseIntQuery(raw: string | undefined, fallback: number, maxLimit?: number): number {
|
||||
if (raw === undefined) return fallback;
|
||||
const n = Number(raw);
|
||||
@@ -113,62 +104,6 @@ function parseExcludedWordsBody(body: unknown): StatsExcludedWordPayload[] | nul
|
||||
return words;
|
||||
}
|
||||
|
||||
function parsePositiveIdList(raw: unknown, maxItems = 100): number[] {
|
||||
if (!Array.isArray(raw)) return [];
|
||||
|
||||
const ids = new Set<number>();
|
||||
for (const rawId of raw) {
|
||||
const id = typeof rawId === 'number' ? rawId : typeof rawId === 'string' ? Number(rawId) : NaN;
|
||||
if (Number.isFinite(id) && id > 0) {
|
||||
ids.add(Math.floor(id));
|
||||
if (ids.size >= maxItems) break;
|
||||
}
|
||||
}
|
||||
|
||||
return Array.from(ids).sort((a, b) => a - b);
|
||||
}
|
||||
|
||||
function coverImagePayload(
|
||||
art: { coverBlob?: Uint8Array | null } | null | undefined,
|
||||
): StatsCoverImagePayload {
|
||||
if (!art?.coverBlob) return null;
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
const contentType = detectImageContentType(bytes);
|
||||
return {
|
||||
contentType,
|
||||
dataUrl: `data:${contentType};base64,${Buffer.from(bytes).toString('base64')}`,
|
||||
};
|
||||
}
|
||||
|
||||
function detectImageContentType(bytes: Uint8Array): string {
|
||||
if (
|
||||
bytes.length >= 8 &&
|
||||
bytes[0] === 0x89 &&
|
||||
bytes[1] === 0x50 &&
|
||||
bytes[2] === 0x4e &&
|
||||
bytes[3] === 0x47
|
||||
) {
|
||||
return 'image/png';
|
||||
}
|
||||
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) {
|
||||
return 'image/jpeg';
|
||||
}
|
||||
if (
|
||||
bytes.length >= 12 &&
|
||||
bytes[0] === 0x52 &&
|
||||
bytes[1] === 0x49 &&
|
||||
bytes[2] === 0x46 &&
|
||||
bytes[3] === 0x46 &&
|
||||
bytes[8] === 0x57 &&
|
||||
bytes[9] === 0x45 &&
|
||||
bytes[10] === 0x42 &&
|
||||
bytes[11] === 0x50
|
||||
) {
|
||||
return 'image/webp';
|
||||
}
|
||||
return 'application/octet-stream';
|
||||
}
|
||||
|
||||
function resolveStatsNoteFieldName(
|
||||
noteInfo: StatsServerNoteInfo,
|
||||
...preferredNames: (string | undefined)[]
|
||||
@@ -326,10 +261,25 @@ function loadKnownWordsSet(cachePath: string | undefined): Set<string> | null {
|
||||
const raw = JSON.parse(readFileSync(cachePath, 'utf-8')) as {
|
||||
version?: number;
|
||||
words?: string[];
|
||||
notes?: Record<string, Array<{ word?: unknown; reading?: unknown }>>;
|
||||
};
|
||||
if ((raw.version === 1 || raw.version === 2) && Array.isArray(raw.words)) {
|
||||
return new Set(raw.words);
|
||||
}
|
||||
// v3 stores reading-aware entries per note; stats rows only carry
|
||||
// headwords, so flatten to a word set (reading-agnostic, fail-open).
|
||||
if (raw.version === 3 && raw.notes && typeof raw.notes === 'object') {
|
||||
const words = new Set<string>();
|
||||
for (const entries of Object.values(raw.notes)) {
|
||||
if (!Array.isArray(entries)) continue;
|
||||
for (const entry of entries) {
|
||||
if (entry && typeof entry.word === 'string' && entry.word) {
|
||||
words.add(entry.word);
|
||||
}
|
||||
}
|
||||
}
|
||||
return words;
|
||||
}
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
@@ -1017,58 +967,7 @@ export function createStatsApp(
|
||||
return c.json({ ok: true });
|
||||
});
|
||||
|
||||
app.post('/api/stats/covers', async (c) => {
|
||||
const body = (await c.req.json().catch(() => null)) as StatsCoverBatchBody | null;
|
||||
const animeIds = parsePositiveIdList(body?.animeIds);
|
||||
const videoIds = parsePositiveIdList(body?.videoIds);
|
||||
const anime: Record<number, StatsCoverImagePayload> = {};
|
||||
const media: Record<number, StatsCoverImagePayload> = {};
|
||||
|
||||
await Promise.all(
|
||||
animeIds.map(async (animeId) => {
|
||||
anime[animeId] = coverImagePayload(await tracker.getAnimeCoverArt(animeId));
|
||||
}),
|
||||
);
|
||||
await Promise.all(
|
||||
videoIds.map(async (videoId) => {
|
||||
media[videoId] = coverImagePayload(await tracker.getCoverArt(videoId));
|
||||
}),
|
||||
);
|
||||
|
||||
return c.json({ anime, media });
|
||||
});
|
||||
|
||||
app.get('/api/stats/anime/:animeId/cover', async (c) => {
|
||||
const animeId = parseIntQuery(c.req.param('animeId'), 0);
|
||||
if (animeId <= 0) return c.body(null, 404);
|
||||
const art = await tracker.getAnimeCoverArt(animeId);
|
||||
if (!art?.coverBlob) return c.body(null, 404);
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
return new Response(bytes, {
|
||||
headers: {
|
||||
'Content-Type': detectImageContentType(bytes),
|
||||
'Cache-Control': 'public, max-age=86400',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
app.get('/api/stats/media/:videoId/cover', async (c) => {
|
||||
const videoId = parseIntQuery(c.req.param('videoId'), 0);
|
||||
if (videoId <= 0) return c.body(null, 404);
|
||||
let art = await tracker.getCoverArt(videoId);
|
||||
if (!art?.coverBlob) {
|
||||
await tracker.ensureCoverArt(videoId);
|
||||
art = await tracker.getCoverArt(videoId);
|
||||
}
|
||||
if (!art?.coverBlob) return c.body(null, 404);
|
||||
const bytes = new Uint8Array(art.coverBlob);
|
||||
return new Response(bytes, {
|
||||
headers: {
|
||||
'Content-Type': detectImageContentType(bytes),
|
||||
'Cache-Control': 'public, max-age=604800',
|
||||
},
|
||||
});
|
||||
});
|
||||
registerStatsCoverRoutes(app, tracker);
|
||||
|
||||
app.get('/api/stats/episode/:videoId/detail', async (c) => {
|
||||
const videoId = parseIntQuery(c.req.param('videoId'), 0);
|
||||
|
||||
@@ -28,6 +28,7 @@ interface YomitanTokenInput {
|
||||
frequencyRank?: number;
|
||||
isNameMatch?: boolean;
|
||||
wordClasses?: string[];
|
||||
isUnparsedRun?: boolean;
|
||||
}
|
||||
|
||||
function makeDepsFromYomitanTokens(
|
||||
@@ -60,6 +61,7 @@ function makeDepsFromYomitanTokens(
|
||||
isNameMatch: token.isNameMatch ?? false,
|
||||
frequencyRank: token.frequencyRank,
|
||||
wordClasses: token.wordClasses,
|
||||
isUnparsedRun: token.isUnparsedRun,
|
||||
};
|
||||
});
|
||||
},
|
||||
@@ -3486,8 +3488,8 @@ test('tokenizeSubtitle keeps known-word highlight for exact non-independent kanj
|
||||
assert.equal(result.tokens?.[1]?.surface, '点');
|
||||
assert.equal(result.tokens?.[1]?.isKnown, true);
|
||||
assert.equal(result.tokens?.[1]?.isNPlusOneTarget, false);
|
||||
assert.equal(result.tokens?.[1]?.frequencyRank, undefined);
|
||||
assert.equal(result.tokens?.[1]?.jlptLevel, undefined);
|
||||
assert.equal(result.tokens?.[1]?.frequencyRank, 1384);
|
||||
assert.equal(result.tokens?.[1]?.jlptLevel, 'N3');
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle keeps mecab-tagged interjections tokenized while clearing annotation metadata', async () => {
|
||||
@@ -4223,6 +4225,38 @@ test('tokenizeSubtitle clears all annotations for explanatory pondering endings'
|
||||
);
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle ignores unparsed-run tokens for annotations and N+1', async () => {
|
||||
// もう いるぅ~!: the ぅ~ elongation has no Yomitan dictionary entry; it must
|
||||
// not become the sole N+1 candidate or receive frequency/JLPT annotations.
|
||||
const result = await tokenizeSubtitle(
|
||||
'もう いるぅ~!',
|
||||
makeDepsFromYomitanTokens(
|
||||
[
|
||||
{ surface: 'もう', reading: 'もう', headword: 'もう' },
|
||||
{ surface: 'いる', reading: 'いる', headword: 'いる' },
|
||||
{ surface: 'ぅ~', reading: '', headword: 'ぅ~', isUnparsedRun: true, frequencyRank: 999 },
|
||||
],
|
||||
{
|
||||
getFrequencyDictionaryEnabled: () => true,
|
||||
getJlptLevel: (text) => (text === 'ぅ~' ? 'N5' : null),
|
||||
isKnownWord: (text) => text === 'もう' || text === 'いる',
|
||||
getMinSentenceWordsForNPlusOne: () => 2,
|
||||
tokenizeWithMecab: async () => null,
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
const filler = result.tokens?.find((token) => token.surface === 'ぅ~');
|
||||
assert.ok(filler);
|
||||
assert.equal(filler?.isNPlusOneTarget, false);
|
||||
assert.equal(filler?.frequencyRank, undefined);
|
||||
assert.equal(filler?.jlptLevel, undefined);
|
||||
assert.equal(
|
||||
result.tokens?.some((token) => token.isNPlusOneTarget),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle keeps frequency for content-led merged token with trailing colloquial suffixes', async () => {
|
||||
const result = await tokenizeSubtitle(
|
||||
'張り切ってんじゃ',
|
||||
|
||||
@@ -33,6 +33,15 @@ type MecabTokenEnrichmentFn = (
|
||||
mecabTokens: MergedToken[] | null,
|
||||
) => Promise<MergedToken[]>;
|
||||
|
||||
// allowReadingOnlyMatch: false suppresses the cache's reading-only index for
|
||||
// lookups that pass a kanji token's reading as the text (see
|
||||
// computeTokenKnownStatus in annotation-stage).
|
||||
export type KnownWordLookupFn = (
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
) => boolean;
|
||||
|
||||
export interface TokenizerServiceDeps {
|
||||
getYomitanExt: () => Extension | null;
|
||||
getYomitanSession?: () => Session | null;
|
||||
@@ -42,7 +51,7 @@ export interface TokenizerServiceDeps {
|
||||
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
|
||||
getYomitanParserInitPromise: () => Promise<boolean> | null;
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: KnownWordLookupFn;
|
||||
getKnownWordMatchMode: () => NPlusOneMatchMode;
|
||||
getKnownWordsEnabled?: () => boolean;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
@@ -77,7 +86,7 @@ export interface TokenizerDepsRuntimeOptions {
|
||||
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
|
||||
getYomitanParserInitPromise: () => Promise<boolean> | null;
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: KnownWordLookupFn;
|
||||
getKnownWordMatchMode: () => NPlusOneMatchMode;
|
||||
getKnownWordsEnabled?: () => boolean;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
@@ -129,7 +138,7 @@ const INVISIBLE_SEPARATOR_PATTERN = /[\u200b\u2060\ufeff]/g;
|
||||
function getKnownWordLookup(
|
||||
deps: TokenizerServiceDeps,
|
||||
options: TokenizerAnnotationOptions,
|
||||
): (text: string, reading?: string) => boolean {
|
||||
): KnownWordLookupFn {
|
||||
if (!options.knownWordsEnabled && !options.nPlusOneEnabled) {
|
||||
return () => false;
|
||||
}
|
||||
@@ -734,6 +743,7 @@ async function parseWithYomitanInternalParser(
|
||||
isNPlusOneTarget: false,
|
||||
isNameMatch: token.isNameMatch ?? false,
|
||||
frequencyRank: token.frequencyRank,
|
||||
isUnparsedRun: token.isUnparsedRun === true ? true : undefined,
|
||||
};
|
||||
}),
|
||||
);
|
||||
|
||||
@@ -208,6 +208,48 @@ test('annotateTokens hides known-word marks while still using known words for N+
|
||||
assert.equal(result[2]?.isNPlusOneTarget, true);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations excludes unparsed-run tokens', () => {
|
||||
// 戻 from 「…とこ戻ろ…」: Yomitan had no dictionary entry, headword falls back
|
||||
// to the surface. Without the flag the token passes every other filter.
|
||||
const unflagged = makeToken({ surface: '戻', headword: '戻', reading: '' });
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(unflagged), false);
|
||||
|
||||
const flagged = makeToken({ surface: '戻', headword: '戻', reading: '', isUnparsedRun: true });
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(flagged), true);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(flagged), true);
|
||||
});
|
||||
|
||||
test('annotateTokens ignores unparsed-run tokens for annotations and N+1', () => {
|
||||
// もう いるぅ~!-style line: the elongation run is the only unknown token and
|
||||
// used to become the sole N+1 candidate despite having no dictionary entry.
|
||||
const tokens = [
|
||||
makeToken({ surface: 'みんな', headword: '皆', reading: 'みんな', startPos: 0, endPos: 3 }),
|
||||
makeToken({ surface: 'とこ', headword: '所', reading: 'とこ', startPos: 3, endPos: 5 }),
|
||||
makeToken({
|
||||
surface: '戻',
|
||||
headword: '戻',
|
||||
reading: '',
|
||||
startPos: 5,
|
||||
endPos: 6,
|
||||
isUnparsedRun: true,
|
||||
frequencyRank: 12,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
isKnownWord: (text) => text === '皆' || text === '所',
|
||||
getJlptLevel: (text) => (text === '戻' ? 'N5' : null),
|
||||
}),
|
||||
{ minSentenceWordsForNPlusOne: 2 },
|
||||
);
|
||||
|
||||
assert.equal(result[2]?.isNPlusOneTarget, false);
|
||||
assert.equal(result[2]?.jlptLevel, undefined);
|
||||
assert.equal(result[2]?.frequencyRank, undefined);
|
||||
});
|
||||
|
||||
test('annotateTokens falls back to reading for known-word matches when headword lookup misses', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
@@ -231,6 +273,29 @@ test('annotateTokens falls back to reading for known-word matches when headword
|
||||
assert.equal(result[0]?.frequencyRank, 1895);
|
||||
});
|
||||
|
||||
test('annotateTokens reading fallback does not match kanji tokens sharing a mined reading', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '渓谷',
|
||||
headword: '渓谷',
|
||||
reading: 'けいこく',
|
||||
endPos: 2,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
// Mimics the cache with a mined 警告/けいこく: けいこく matches through
|
||||
// the reading-only index unless the lookup opts out of it.
|
||||
isKnownWord: (text, _reading, options) =>
|
||||
text === '警告' || (options?.allowReadingOnlyMatch !== false && text === 'けいこく'),
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, false);
|
||||
});
|
||||
|
||||
test('annotateTokens ignores partial furigana readings for known-word fallback', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
@@ -575,7 +640,9 @@ test('shouldExcludeTokenFromSubtitleAnnotations keeps lexical tokens outside exp
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations still excludes lexical non-independent kanji nouns from non-known annotations', () => {
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations keeps lexical non-independent kanji nouns', () => {
|
||||
// Yomitan segments 以外/日/方 as standalone vocabulary tokens; MeCab's
|
||||
// 非自立 tag must only suppress kana grammar nouns (こと, もの, とき).
|
||||
const token = makeToken({
|
||||
surface: '以外',
|
||||
headword: '以外',
|
||||
@@ -586,6 +653,21 @@ test('shouldExcludeTokenFromSubtitleAnnotations still excludes lexical non-indep
|
||||
pos3: '副詞可能',
|
||||
});
|
||||
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), false);
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations still excludes kana non-independent nouns', () => {
|
||||
const token = makeToken({
|
||||
surface: 'こと',
|
||||
headword: 'こと',
|
||||
reading: 'コト',
|
||||
partOfSpeech: PartOfSpeech.noun,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '一般',
|
||||
});
|
||||
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), true);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), true);
|
||||
});
|
||||
@@ -728,15 +810,6 @@ test('shouldExcludeTokenFromVocabularyPersistence excludes common frequency stop
|
||||
pos2: '代名詞|副助詞/並立助詞/終助詞',
|
||||
pos3: '一般|*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: '確かに',
|
||||
headword: '確かに',
|
||||
reading: 'たしかに',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞|助詞',
|
||||
pos2: '形容動詞語幹|副詞化',
|
||||
pos3: '*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: 'あなた',
|
||||
headword: '貴方',
|
||||
@@ -753,6 +826,34 @@ test('shouldExcludeTokenFromVocabularyPersistence excludes common frequency stop
|
||||
}
|
||||
});
|
||||
|
||||
test('content adverbs are not excluded from annotations or vocabulary persistence', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '確かに',
|
||||
headword: '確かに',
|
||||
reading: 'たしかに',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞|助詞',
|
||||
pos2: '形容動詞語幹|副詞化',
|
||||
pos3: '*',
|
||||
}),
|
||||
makeToken({
|
||||
surface: 'やはり',
|
||||
headword: 'やはり',
|
||||
reading: 'ヤハリ',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '副詞',
|
||||
pos2: '一般',
|
||||
pos3: '*',
|
||||
}),
|
||||
];
|
||||
|
||||
for (const token of tokens) {
|
||||
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(token), false, token.surface);
|
||||
assert.equal(shouldExcludeTokenFromVocabularyPersistence(token), false, token.surface);
|
||||
}
|
||||
});
|
||||
|
||||
test('shouldExcludeTokenFromSubtitleAnnotations excludes standalone して grammar helper fragments', () => {
|
||||
const token = makeToken({
|
||||
surface: 'して',
|
||||
@@ -1402,8 +1503,8 @@ test('annotateTokens keeps known-word status for non-independent kanji noun toke
|
||||
|
||||
assert.equal(result[0]?.isKnown, true);
|
||||
assert.equal(result[0]?.isNPlusOneTarget, false);
|
||||
assert.equal(result[0]?.frequencyRank, undefined);
|
||||
assert.equal(result[0]?.jlptLevel, undefined);
|
||||
assert.equal(result[0]?.frequencyRank, 1384);
|
||||
assert.equal(result[0]?.jlptLevel, 'N3');
|
||||
});
|
||||
|
||||
test('annotateTokens keeps known-word status for lexical non-independent kanji nouns', () => {
|
||||
@@ -1431,23 +1532,54 @@ test('annotateTokens keeps known-word status for lexical non-independent kanji n
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, true);
|
||||
assert.equal(result[0]?.frequencyRank, undefined);
|
||||
assert.equal(result[0]?.frequencyRank, 437);
|
||||
assert.equal(result[0]?.isNPlusOneTarget, false);
|
||||
});
|
||||
|
||||
test('annotateTokens clears all annotations for non-independent kanji noun tokens under unified gate', () => {
|
||||
test('annotateTokens keeps frequency for unknown non-independent kanji noun tokens', () => {
|
||||
// 日 in いい日だったな: MeCab tags it 名詞/非自立 but Yomitan segments it as
|
||||
// a standalone vocabulary token, so frequency highlighting must survive.
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: '者',
|
||||
reading: 'もの',
|
||||
headword: '者',
|
||||
surface: '日',
|
||||
reading: 'ひ',
|
||||
headword: '日',
|
||||
partOfSpeech: PartOfSpeech.noun,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '副詞可能',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
frequencyRank: 718,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
getJlptLevel: (text) => (text === '日' ? 'N4' : null),
|
||||
}),
|
||||
{ minSentenceWordsForNPlusOne: 1 },
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, false);
|
||||
assert.equal(result[0]?.frequencyRank, 718);
|
||||
assert.equal(result[0]?.jlptLevel, 'N4');
|
||||
});
|
||||
|
||||
test('annotateTokens still clears annotations for kana non-independent noun tokens', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: 'こと',
|
||||
reading: 'こと',
|
||||
headword: 'こと',
|
||||
partOfSpeech: PartOfSpeech.other,
|
||||
pos1: '名詞',
|
||||
pos2: '非自立',
|
||||
pos3: '一般',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 475,
|
||||
endPos: 2,
|
||||
frequencyRank: 96,
|
||||
}),
|
||||
];
|
||||
|
||||
@@ -1722,6 +1854,36 @@ test('annotateTokens keeps known status while clearing other annotations for sta
|
||||
}
|
||||
});
|
||||
|
||||
test('annotateTokens excludes standalone noun-suffix tokens from annotations while keeping cache-backed known status', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
surface: 'さん',
|
||||
headword: 'さん',
|
||||
reading: 'サン',
|
||||
partOfSpeech: PartOfSpeech.noun,
|
||||
pos1: '名詞',
|
||||
pos2: '接尾',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
frequencyRank: 33,
|
||||
}),
|
||||
];
|
||||
|
||||
const result = annotateTokens(
|
||||
tokens,
|
||||
makeDeps({
|
||||
isKnownWord: (text) => text === 'さん',
|
||||
getJlptLevel: (text) => (text === 'さん' ? 'N5' : null),
|
||||
}),
|
||||
{ minSentenceWordsForNPlusOne: 1 },
|
||||
);
|
||||
|
||||
assert.equal(result[0]?.isKnown, true);
|
||||
assert.equal(result[0]?.isNPlusOneTarget, false);
|
||||
assert.equal(result[0]?.frequencyRank, undefined);
|
||||
assert.equal(result[0]?.jlptLevel, undefined);
|
||||
});
|
||||
|
||||
test('annotateTokens keeps known status while clearing other annotations for auxiliary-only te-kureru helper spans', () => {
|
||||
const tokens = [
|
||||
makeToken({
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
import { JlptLevel, MergedToken, NPlusOneMatchMode, PartOfSpeech } from '../../../types';
|
||||
import { shouldIgnoreJlptByTerm, shouldIgnoreJlptForMecabPos1 } from '../jlpt-token-filter';
|
||||
import {
|
||||
isKanjiNonIndependentNounToken,
|
||||
shouldExcludeTokenFromSubtitleAnnotations as sharedShouldExcludeTokenFromSubtitleAnnotations,
|
||||
stripSubtitleAnnotationMetadata as sharedStripSubtitleAnnotationMetadata,
|
||||
} from './subtitle-annotation-filter';
|
||||
@@ -25,7 +26,11 @@ const jlptLevelLookupCaches = new WeakMap<
|
||||
>();
|
||||
|
||||
export interface AnnotationStageDeps {
|
||||
isKnownWord: (text: string, reading?: string) => boolean;
|
||||
isKnownWord: (
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
) => boolean;
|
||||
knownWordMatchMode: NPlusOneMatchMode;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
}
|
||||
@@ -94,23 +99,6 @@ function normalizePos2Tag(pos2: string | undefined): string {
|
||||
return typeof pos2 === 'string' ? pos2.trim() : '';
|
||||
}
|
||||
|
||||
function hasKanjiChar(text: string): boolean {
|
||||
for (const char of text) {
|
||||
const code = char.codePointAt(0);
|
||||
if (code === undefined) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
(code >= 0x3400 && code <= 0x4dbf) ||
|
||||
(code >= 0x4e00 && code <= 0x9fff) ||
|
||||
(code >= 0xf900 && code <= 0xfaff)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function isExcludedComponent(
|
||||
pos1: string | undefined,
|
||||
pos2: string | undefined,
|
||||
@@ -283,34 +271,6 @@ function isFrequencyExcludedByPos(
|
||||
);
|
||||
}
|
||||
|
||||
function shouldKeepFrequencyForNonIndependentKanjiNoun(
|
||||
token: MergedToken,
|
||||
pos1Exclusions: ReadonlySet<string>,
|
||||
): boolean {
|
||||
if (pos1Exclusions.has('名詞')) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const rank =
|
||||
typeof token.frequencyRank === 'number' && Number.isFinite(token.frequencyRank)
|
||||
? Math.max(1, Math.floor(token.frequencyRank))
|
||||
: null;
|
||||
if (rank === null) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const pos1Parts = splitNormalizedTagParts(normalizePos1Tag(token.pos1));
|
||||
const pos2Parts = splitNormalizedTagParts(normalizePos2Tag(token.pos2));
|
||||
if (pos1Parts.length !== 1 || pos2Parts.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
if (pos1Parts[0] !== '名詞' || pos2Parts[0] !== '非自立') {
|
||||
return false;
|
||||
}
|
||||
|
||||
return hasKanjiChar(token.surface) || hasKanjiChar(token.headword);
|
||||
}
|
||||
|
||||
export function shouldExcludeTokenFromVocabularyPersistence(
|
||||
token: MergedToken,
|
||||
options: Pick<AnnotationStageOptions, 'pos1Exclusions' | 'pos2Exclusions'> = {},
|
||||
@@ -320,7 +280,8 @@ export function shouldExcludeTokenFromVocabularyPersistence(
|
||||
|
||||
return (
|
||||
sharedShouldExcludeTokenFromSubtitleAnnotations(token, { pos1Exclusions, pos2Exclusions }) ||
|
||||
isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions)
|
||||
(isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions) &&
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions))
|
||||
);
|
||||
}
|
||||
|
||||
@@ -697,7 +658,7 @@ function resolveKnownWordReadingForMatch(
|
||||
|
||||
function computeTokenKnownStatus(
|
||||
token: MergedToken,
|
||||
isKnownWord: (text: string, reading?: string) => boolean,
|
||||
isKnownWord: AnnotationStageDeps['isKnownWord'],
|
||||
knownWordMatchMode: NPlusOneMatchMode,
|
||||
): boolean {
|
||||
const matchText = resolveKnownWordText(token.surface, token.headword, knownWordMatchMode);
|
||||
@@ -711,7 +672,14 @@ function computeTokenKnownStatus(
|
||||
return false;
|
||||
}
|
||||
|
||||
return fallbackReading !== matchText.trim() && isKnownWord(fallbackReading);
|
||||
// This fallback covers words mined in kana (token 大体, mined word だいたい),
|
||||
// so the reading must only match mined word texts — the cache's reading-only
|
||||
// index would let any kanji token match an unrelated note that shares its
|
||||
// reading (渓谷/けいこく vs a mined 警告/けいこく).
|
||||
return (
|
||||
fallbackReading !== matchText.trim() &&
|
||||
isKnownWord(fallbackReading, undefined, { allowReadingOnlyMatch: false })
|
||||
);
|
||||
}
|
||||
|
||||
function filterTokenFrequencyRank(
|
||||
@@ -721,7 +689,7 @@ function filterTokenFrequencyRank(
|
||||
): number | undefined {
|
||||
if (
|
||||
isFrequencyExcludedByPos(token, pos1Exclusions, pos2Exclusions) &&
|
||||
!shouldKeepFrequencyForNonIndependentKanjiNoun(token, pos1Exclusions)
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
@@ -146,13 +146,17 @@ test('emits unparsed non-caption text as a token with surface headword', () => {
|
||||
|
||||
const tokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
assert.deepEqual(
|
||||
tokens?.map((token) => ({ surface: token.surface, headword: token.headword })),
|
||||
tokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
headword: token.headword,
|
||||
isUnparsedRun: token.isUnparsedRun ?? false,
|
||||
})),
|
||||
[
|
||||
{ surface: 'みんな', headword: '皆' },
|
||||
{ surface: 'の', headword: 'の' },
|
||||
{ surface: 'とこ', headword: '所' },
|
||||
{ surface: '戻', headword: '戻' },
|
||||
{ surface: 'ろ', headword: '櫓' },
|
||||
{ surface: 'みんな', headword: '皆', isUnparsedRun: false },
|
||||
{ surface: 'の', headword: 'の', isUnparsedRun: false },
|
||||
{ surface: 'とこ', headword: '所', isUnparsedRun: false },
|
||||
{ surface: '戻', headword: '戻', isUnparsedRun: true },
|
||||
{ surface: 'ろ', headword: '櫓', isUnparsedRun: false },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
@@ -239,6 +239,7 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
headword: string,
|
||||
start: number,
|
||||
end: number,
|
||||
isUnparsedRun = false,
|
||||
): void => {
|
||||
tokens.push({
|
||||
surface,
|
||||
@@ -254,6 +255,7 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
const matchText = resolveKnownWordText(surface, headword, knownWordMatchMode);
|
||||
return matchText ? isKnownWord(matchText) : false;
|
||||
})(),
|
||||
...(isUnparsedRun ? { isUnparsedRun: true } : {}),
|
||||
});
|
||||
};
|
||||
|
||||
@@ -275,10 +277,11 @@ export function mapYomitanParseResultItemToMergedTokens(
|
||||
previousToken.reading += combinedReading;
|
||||
previousToken.endPos = end;
|
||||
} else if (shouldEmitUnparsedRunAsToken(combinedSurface)) {
|
||||
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional). Keep it
|
||||
// as a token with its surface as headword so it stays hoverable and counts in
|
||||
// the n+1 math — matching what the embedded Yomitan actually returns.
|
||||
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end);
|
||||
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional, ぅ~
|
||||
// elongations). Keep it as a token with its surface as headword so it stays
|
||||
// hoverable, but flag it so annotation/N+1/vocab logic ignores it — there is
|
||||
// no dictionary entry behind it.
|
||||
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end, true);
|
||||
}
|
||||
} else {
|
||||
hasDictionaryMatch = true;
|
||||
|
||||
@@ -47,7 +47,6 @@ export const SUBTITLE_ANNOTATION_EXCLUDED_TERMS = new Set([
|
||||
'へえ',
|
||||
'ふう',
|
||||
'ほう',
|
||||
'やはり',
|
||||
'何か',
|
||||
'何だ',
|
||||
'何も',
|
||||
@@ -55,7 +54,6 @@ export const SUBTITLE_ANNOTATION_EXCLUDED_TERMS = new Set([
|
||||
'有る',
|
||||
'在る',
|
||||
'様',
|
||||
'確かに',
|
||||
'誰も',
|
||||
'貴方',
|
||||
'もんか',
|
||||
@@ -139,6 +137,46 @@ function resolvePos2Exclusions(options: SubtitleAnnotationFilterOptions = {}): R
|
||||
return resolveAnnotationPos2ExclusionSet(DEFAULT_ANNOTATION_POS2_EXCLUSION_CONFIG);
|
||||
}
|
||||
|
||||
function hasKanjiChar(text: string): boolean {
|
||||
for (const char of text) {
|
||||
const code = char.codePointAt(0);
|
||||
if (code === undefined) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
(code >= 0x3400 && code <= 0x4dbf) ||
|
||||
(code >= 0x4e00 && code <= 0x9fff) ||
|
||||
(code >= 0xf900 && code <= 0xfaff)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// Kanji-bearing non-independent nouns (日, 方, 上, …) are real vocabulary that
|
||||
// Yomitan segments as standalone tokens; MeCab's 非自立 tag exists to suppress
|
||||
// kana grammar nouns (こと, もの, とき) and must not hide these.
|
||||
export function isKanjiNonIndependentNounToken(
|
||||
token: MergedToken,
|
||||
pos1Exclusions: ReadonlySet<string>,
|
||||
): boolean {
|
||||
if (pos1Exclusions.has('名詞')) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const pos1Parts = splitNormalizedTagParts(normalizePosTag(token.pos1));
|
||||
const pos2Parts = splitNormalizedTagParts(normalizePosTag(token.pos2));
|
||||
if (pos1Parts.length !== 1 || pos2Parts.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
if (pos1Parts[0] !== '名詞' || pos2Parts[0] !== '非自立') {
|
||||
return false;
|
||||
}
|
||||
|
||||
return hasKanjiChar(token.surface) || hasKanjiChar(token.headword);
|
||||
}
|
||||
|
||||
function normalizeKana(text: string): string {
|
||||
const raw = text.trim();
|
||||
if (!raw) {
|
||||
@@ -436,6 +474,13 @@ export function shouldExcludeTokenFromSubtitleAnnotations(
|
||||
token: MergedToken,
|
||||
options: SubtitleAnnotationFilterOptions = {},
|
||||
): boolean {
|
||||
// No Yomitan dictionary entry backs this token (ぅ~ elongations, truncated
|
||||
// inflections) — it exists only to stay hoverable and must never receive
|
||||
// annotations or count in the N+1 math.
|
||||
if (token.isUnparsedRun === true) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const pos1Exclusions = resolvePos1Exclusions(options);
|
||||
const pos2Exclusions = resolvePos2Exclusions(options);
|
||||
const normalizedPos1 = normalizePosTag(token.pos1);
|
||||
@@ -447,7 +492,10 @@ export function shouldExcludeTokenFromSubtitleAnnotations(
|
||||
return true;
|
||||
}
|
||||
|
||||
if (isExcludedByTagSet(normalizedPos2, pos2Exclusions)) {
|
||||
if (
|
||||
isExcludedByTagSet(normalizedPos2, pos2Exclusions) &&
|
||||
!isKanjiNonIndependentNounToken(token, pos1Exclusions)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
@@ -820,6 +820,135 @@ test('requestYomitanScanTokens keeps scanner metadata when parse spans agree', a
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens keeps scanner metadata for matching spans when parse segmentation has filler chunks', async () => {
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
if (script.includes('parseText')) {
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: 'や',
|
||||
reading: '',
|
||||
headwords: [[{ term: 'や' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'ほ',
|
||||
reading: '',
|
||||
headwords: [[{ term: '帆' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'っ ',
|
||||
reading: '',
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'ミナト',
|
||||
reading: '',
|
||||
headwords: [[{ term: 'ミナト' }]],
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
// The termsFind scanner skips the unmatched っ+space chunk, so its spans
|
||||
// do not line up 1:1 with the parseText segmentation above.
|
||||
return [
|
||||
{
|
||||
surface: 'や',
|
||||
reading: 'や',
|
||||
headword: 'や',
|
||||
headwordReading: 'や',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 57,
|
||||
},
|
||||
{
|
||||
surface: 'ほ',
|
||||
reading: 'ほ',
|
||||
headword: '帆',
|
||||
headwordReading: 'ほ',
|
||||
startPos: 1,
|
||||
endPos: 2,
|
||||
frequencyRank: 15414,
|
||||
},
|
||||
{
|
||||
surface: 'ミナト',
|
||||
reading: 'ミナト',
|
||||
headword: 'ミナト',
|
||||
headwordReading: 'みなと',
|
||||
startPos: 4,
|
||||
endPos: 7,
|
||||
isNameMatch: true,
|
||||
frequencyRank: 75133,
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const result = await requestYomitanScanTokens('やほっ ミナト', deps, {
|
||||
error: () => undefined,
|
||||
});
|
||||
|
||||
assert.deepEqual(result, [
|
||||
{
|
||||
surface: 'や',
|
||||
reading: 'や',
|
||||
headword: 'や',
|
||||
headwordReading: 'や',
|
||||
startPos: 0,
|
||||
endPos: 1,
|
||||
frequencyRank: 57,
|
||||
},
|
||||
{
|
||||
surface: 'ほ',
|
||||
reading: 'ほ',
|
||||
headword: '帆',
|
||||
headwordReading: 'ほ',
|
||||
startPos: 1,
|
||||
endPos: 2,
|
||||
frequencyRank: 15414,
|
||||
},
|
||||
{
|
||||
surface: 'っ ',
|
||||
reading: '',
|
||||
headword: 'っ ',
|
||||
startPos: 2,
|
||||
endPos: 4,
|
||||
isUnparsedRun: true,
|
||||
},
|
||||
{
|
||||
surface: 'ミナト',
|
||||
reading: 'ミナト',
|
||||
headword: 'ミナト',
|
||||
headwordReading: 'みなと',
|
||||
startPos: 4,
|
||||
endPos: 7,
|
||||
isNameMatch: true,
|
||||
frequencyRank: 75133,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens falls back to left-to-right termsFind scanning', async () => {
|
||||
const scripts: string[] = [];
|
||||
const deps = createDeps(async (script) => {
|
||||
@@ -1761,6 +1890,364 @@ test('requestYomitanScanTokens accepts SubMiner character entries with structure
|
||||
assert.equal((result as Array<{ isNameMatch?: boolean }>)[0]?.isNameMatch, true);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens greedily tokenizes character names before longer generic matches', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'美姫とヨータ',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
|
||||
|
||||
const nameEntry = (term: string, reading: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [
|
||||
{
|
||||
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
},
|
||||
],
|
||||
});
|
||||
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
|
||||
});
|
||||
|
||||
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
|
||||
if (action !== 'termsFind') {
|
||||
throw new Error(`unexpected action: ${action}`);
|
||||
}
|
||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||
if (text.startsWith('美姫')) {
|
||||
return { originalTextLength: 2, dictionaryEntries: [nameEntry('美姫', 'みき')] };
|
||||
}
|
||||
if (text.startsWith('とヨータ')) {
|
||||
// Greedy generic match: とヨー normalizes to とよう (渡洋). Without the
|
||||
// name pre-pass this consumes the ヨ of ヨータ.
|
||||
return {
|
||||
originalTextLength: 3,
|
||||
dictionaryEntries: [jmdictEntry('渡洋', 'とよう', 'とヨー'), jmdictEntry('と', 'と', 'と')],
|
||||
};
|
||||
}
|
||||
if (text.startsWith('ヨータ')) {
|
||||
return { originalTextLength: 3, dictionaryEntries: [nameEntry('ヨータ', 'よーた')] };
|
||||
}
|
||||
if (text === 'と') {
|
||||
return { originalTextLength: 1, dictionaryEntries: [jmdictEntry('と', 'と', 'と')] };
|
||||
}
|
||||
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||
});
|
||||
|
||||
assert.equal(Array.isArray(result), true);
|
||||
assert.deepEqual(
|
||||
(result as Array<Record<string, unknown>>).map(
|
||||
({ surface, headword, startPos, endPos, isNameMatch }) => ({
|
||||
surface,
|
||||
headword,
|
||||
startPos,
|
||||
endPos,
|
||||
isNameMatch,
|
||||
}),
|
||||
),
|
||||
[
|
||||
{ surface: '美姫', headword: '美姫', startPos: 0, endPos: 2, isNameMatch: true },
|
||||
{ surface: 'と', headword: 'と', startPos: 2, endPos: 3, isNameMatch: false },
|
||||
{ surface: 'ヨータ', headword: 'ヨータ', startPos: 3, endPos: 6, isNameMatch: true },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens lets a longer generic word beat a shorter name at the same position', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'空気変わって',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
|
||||
|
||||
const nameEntry = (term: string, reading: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [
|
||||
{
|
||||
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
|
||||
},
|
||||
],
|
||||
});
|
||||
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
|
||||
headwords: [
|
||||
{
|
||||
term,
|
||||
reading,
|
||||
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
|
||||
},
|
||||
],
|
||||
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
|
||||
});
|
||||
|
||||
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
|
||||
if (action !== 'termsFind') {
|
||||
throw new Error(`unexpected action: ${action}`);
|
||||
}
|
||||
const text = (params as { text?: string } | undefined)?.text ?? '';
|
||||
if (text.startsWith('空気')) {
|
||||
// A character named 空 matches here, but the generic 空気 is longer and
|
||||
// must win the position.
|
||||
return {
|
||||
originalTextLength: 2,
|
||||
dictionaryEntries: [nameEntry('空', 'くう'), jmdictEntry('空気', 'くうき', '空気')],
|
||||
};
|
||||
}
|
||||
if (text.startsWith('変わって')) {
|
||||
return {
|
||||
originalTextLength: 4,
|
||||
dictionaryEntries: [jmdictEntry('変わる', 'かわる', '変わって')],
|
||||
};
|
||||
}
|
||||
return { originalTextLength: 0, dictionaryEntries: [] };
|
||||
});
|
||||
|
||||
assert.equal(Array.isArray(result), true);
|
||||
assert.deepEqual(
|
||||
(result as Array<Record<string, unknown>>).map(
|
||||
({ surface, headword, startPos, endPos, isNameMatch }) => ({
|
||||
surface,
|
||||
headword,
|
||||
startPos,
|
||||
endPos,
|
||||
isNameMatch,
|
||||
}),
|
||||
),
|
||||
[
|
||||
{ surface: '空気', headword: '空気', startPos: 0, endPos: 2, isNameMatch: false },
|
||||
{ surface: '変わって', headword: '変わる', startPos: 2, endPos: 6, isNameMatch: false },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens skips greedy name scan without an enabled character dictionary', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('termsFind')) {
|
||||
scannerScript = script;
|
||||
return [];
|
||||
}
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [{ name: 'JMdict', enabled: true }],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
await requestYomitanScanTokens(
|
||||
'アクア',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.match(scannerScript, /const greedyNameScanEnabled = false;/);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens replaces parseText segmentation where greedy name tokens re-segment', async () => {
|
||||
const deps = createDeps(async (script) => {
|
||||
if (script.includes('optionsGetFull')) {
|
||||
return {
|
||||
profileCurrent: 0,
|
||||
profiles: [
|
||||
{
|
||||
options: {
|
||||
scanning: { length: 40 },
|
||||
dictionaries: [
|
||||
{ name: 'JMdict', enabled: true },
|
||||
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
if (script.includes('parseText')) {
|
||||
// parseText walks greedily too, so it merges と with ヨー into 渡洋 and
|
||||
// strands the タ.
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: '美姫',
|
||||
reading: 'みき',
|
||||
headwords: [[{ term: '美姫' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'とヨー',
|
||||
reading: 'とよう',
|
||||
headwords: [[{ term: '渡洋' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'タ',
|
||||
reading: '',
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
return [
|
||||
{
|
||||
surface: '美姫',
|
||||
reading: 'みき',
|
||||
headword: '美姫',
|
||||
headwordReading: 'みき',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
isNameMatch: true,
|
||||
},
|
||||
{
|
||||
surface: 'と',
|
||||
reading: 'と',
|
||||
headword: 'と',
|
||||
headwordReading: 'と',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
isNameMatch: false,
|
||||
},
|
||||
{
|
||||
surface: 'ヨータ',
|
||||
reading: 'ヨータ',
|
||||
headword: 'ヨータ',
|
||||
headwordReading: 'よーた',
|
||||
startPos: 3,
|
||||
endPos: 6,
|
||||
isNameMatch: true,
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const result = await requestYomitanScanTokens(
|
||||
'美姫とヨータ',
|
||||
deps,
|
||||
{ error: () => undefined },
|
||||
{ includeNameMatchMetadata: true },
|
||||
);
|
||||
|
||||
assert.deepEqual(result, [
|
||||
{
|
||||
surface: '美姫',
|
||||
reading: 'みき',
|
||||
headword: '美姫',
|
||||
headwordReading: 'みき',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
isNameMatch: true,
|
||||
},
|
||||
{
|
||||
surface: 'と',
|
||||
reading: 'と',
|
||||
headword: 'と',
|
||||
headwordReading: 'と',
|
||||
startPos: 2,
|
||||
endPos: 3,
|
||||
isNameMatch: false,
|
||||
},
|
||||
{
|
||||
surface: 'ヨータ',
|
||||
reading: 'ヨータ',
|
||||
headword: 'ヨータ',
|
||||
headwordReading: 'よーた',
|
||||
startPos: 3,
|
||||
endPos: 6,
|
||||
isNameMatch: true,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('requestYomitanScanTokens preserves matched headword word classes', async () => {
|
||||
let scannerScript = '';
|
||||
const deps = createDeps(async (script) => {
|
||||
|
||||
@@ -57,6 +57,7 @@ export interface YomitanScanToken {
|
||||
isNameMatch?: boolean;
|
||||
frequencyRank?: number;
|
||||
wordClasses?: string[];
|
||||
isUnparsedRun?: boolean;
|
||||
}
|
||||
|
||||
interface YomitanProfileMetadata {
|
||||
@@ -73,6 +74,7 @@ export interface YomitanAddNoteResult {
|
||||
}
|
||||
|
||||
const DEFAULT_YOMITAN_SCAN_LENGTH = 40;
|
||||
const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
const yomitanProfileMetadataByWindow = new WeakMap<BrowserWindow, YomitanProfileMetadata>();
|
||||
const yomitanProfileDiagnosticsLoggedByWindow = new WeakSet<BrowserWindow>();
|
||||
const yomitanFrequencyCacheByWindow = new WeakMap<
|
||||
@@ -105,20 +107,89 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
);
|
||||
}
|
||||
|
||||
function hasSameTokenSpans(left: YomitanScanToken[], right: YomitanScanToken[]): boolean {
|
||||
if (left.length !== right.length) {
|
||||
return false;
|
||||
function scanTokenSpanKey(token: YomitanScanToken): string {
|
||||
return `${token.startPos}:${token.endPos}:${token.surface}`;
|
||||
}
|
||||
|
||||
// Maps a parse-selected token to the scanner-token shape carried out of the
|
||||
// parser runtime. Shared by both selectYomitanParseTokens fallback paths so the
|
||||
// projected fields stay in sync as the shape changes.
|
||||
function toYomitanScanToken(token: {
|
||||
surface: string;
|
||||
reading: string;
|
||||
headword: string;
|
||||
startPos: number;
|
||||
endPos: number;
|
||||
isUnparsedRun?: boolean;
|
||||
}): YomitanScanToken {
|
||||
return {
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
...(token.isUnparsedRun === true ? { isUnparsedRun: true } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
// parseText segmentation is authoritative (it emits filler chunks for text the
|
||||
// termsFind scanner skips), but only the termsFind scanner carries annotation
|
||||
// metadata (isNameMatch, frequencyRank, headwordReading, wordClasses). Graft
|
||||
// scanner tokens onto the parseText segmentation per matching span so one
|
||||
// unmatched chunk degrades only itself instead of dropping the whole line's
|
||||
// metadata.
|
||||
//
|
||||
// Exception: character-name tokens. The greedy name scan can re-segment text
|
||||
// around a name (e.g. とヨータ → と + ヨータ instead of とヨー + タ), so
|
||||
// parseText segmentation cannot be authoritative there. Each name span is
|
||||
// expanded until it aligns with token boundaries in both segmentations, then
|
||||
// the parse tokens inside are replaced with the scanner tokens.
|
||||
function mergeScannerTokensIntoParseTokens(
|
||||
parseScanTokens: YomitanScanToken[],
|
||||
scannerTokens: YomitanScanToken[],
|
||||
): YomitanScanToken[] {
|
||||
const scannerTokensBySpan = new Map<string, YomitanScanToken>();
|
||||
for (const token of scannerTokens) {
|
||||
scannerTokensBySpan.set(scanTokenSpanKey(token), token);
|
||||
}
|
||||
const graftedTokens = parseScanTokens.map(
|
||||
(token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token,
|
||||
);
|
||||
|
||||
const nameTokens = scannerTokens.filter((token) => token.isNameMatch === true);
|
||||
if (nameTokens.length === 0) {
|
||||
return graftedTokens;
|
||||
}
|
||||
|
||||
return left.every((token, index) => {
|
||||
const other = right[index];
|
||||
return (
|
||||
other !== undefined &&
|
||||
token.surface === other.surface &&
|
||||
token.startPos === other.startPos &&
|
||||
token.endPos === other.endPos
|
||||
);
|
||||
});
|
||||
const regions = nameTokens.map((token) => ({ start: token.startPos, end: token.endPos }));
|
||||
const allTokens = [...parseScanTokens, ...scannerTokens];
|
||||
let expanded = true;
|
||||
while (expanded) {
|
||||
expanded = false;
|
||||
for (const region of regions) {
|
||||
for (const token of allTokens) {
|
||||
const overlaps = token.startPos < region.end && token.endPos > region.start;
|
||||
const extendsBeyond = token.startPos < region.start || token.endPos > region.end;
|
||||
if (overlaps && extendsBeyond) {
|
||||
region.start = Math.min(region.start, token.startPos);
|
||||
region.end = Math.max(region.end, token.endPos);
|
||||
expanded = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isInsideNameRegion = (token: YomitanScanToken): boolean =>
|
||||
regions.some((region) => token.startPos >= region.start && token.endPos <= region.end);
|
||||
|
||||
const merged = graftedTokens.filter((token) => !isInsideNameRegion(token));
|
||||
for (const token of scannerTokens) {
|
||||
if (isInsideNameRegion(token)) {
|
||||
merged.push(token);
|
||||
}
|
||||
}
|
||||
merged.sort((a, b) => a.startPos - b.startPos || a.endPos - b.endPos);
|
||||
return merged;
|
||||
}
|
||||
|
||||
function makeTermReadingCacheKey(term: string, reading: string | null): string {
|
||||
@@ -1104,8 +1175,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
function normalizeWordClasses(headword) {
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
@@ -1146,7 +1216,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith("SubMiner Character Dictionary"));
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
@@ -1154,7 +1224,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/SubMiner Character Dictionary[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
@@ -1214,7 +1284,43 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
@@ -1261,6 +1367,7 @@ function buildYomitanScanningScript(
|
||||
profileIndex: number,
|
||||
scanLength: number,
|
||||
includeNameMatchMetadata: boolean,
|
||||
greedyNameScanEnabled: boolean,
|
||||
currentCharacterDictionaryMediaId: number | null,
|
||||
dictionaryPriorityByName: Record<string, number>,
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>,
|
||||
@@ -1287,6 +1394,7 @@ function buildYomitanScanningScript(
|
||||
});
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const includeNameMatchMetadata = ${includeNameMatchMetadata ? 'true' : 'false'};
|
||||
const greedyNameScanEnabled = ${greedyNameScanEnabled ? 'true' : 'false'};
|
||||
const currentCharacterDictionaryMediaId = ${
|
||||
currentCharacterDictionaryMediaId !== null
|
||||
? String(currentCharacterDictionaryMediaId)
|
||||
@@ -1297,26 +1405,17 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
const text = ${JSON.stringify(text)};
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const termsFindCache = new Map();
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const cacheKey = position + ":" + windowLength;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached) { return cached; }
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const result = await invoke("termsFind", { text: substring, details, optionsContext: { index: ${profileIndex} } });
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = substring.substring(0, originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
termsFindCache.set(cacheKey, result);
|
||||
return result;
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
@@ -1325,7 +1424,7 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + originalTextLength,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
@@ -1335,17 +1434,93 @@ ${YOMITAN_SCANNING_HELPERS}
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return { token: tokenPayload, matchedLength: originalTextLength };
|
||||
return tokenPayload;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, ${scanLength});
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + ${scanLength});
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
while (i < text.length) {
|
||||
let attempt = await findTokenAt(i, ${scanLength});
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
// Cap the window at the next reserved name span so a generic match
|
||||
// cannot consume into it.
|
||||
const windowLength = nextNameToken ? Math.min(${scanLength}, nextNameToken.startPos - i) : ${scanLength};
|
||||
let attempt = await findTokenAt(i, windowLength);
|
||||
// Yomitan text normalization can consume characters (whitespace,
|
||||
// punctuation) beyond the matched term, leaving no headword whose
|
||||
// source equals the consumed text. Retry with shorter windows so a
|
||||
// valid prefix term (e.g. a character name before a paren) still
|
||||
// tokenizes instead of the position being skipped.
|
||||
let retryLength = Math.min(attempt.matchedLength, ${scanLength}) - 1;
|
||||
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
const retry = await findTokenAt(i, retryLength);
|
||||
if (retry.token) {
|
||||
@@ -1482,18 +1657,17 @@ export async function requestYomitanScanTokens(
|
||||
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedParseTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens =
|
||||
selectedParseTokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
})) ?? null;
|
||||
const parseScanTokens = selectedParseTokens?.map(toYomitanScanToken) ?? null;
|
||||
|
||||
const metadata = await requestYomitanProfileMetadata(parserWindow, logger);
|
||||
const profileIndex = metadata?.profileIndex ?? 0;
|
||||
const scanLength = metadata?.scanLength ?? DEFAULT_YOMITAN_SCAN_LENGTH;
|
||||
const includeNameMatchMetadata = options?.includeNameMatchMetadata === true;
|
||||
const greedyNameScanEnabled =
|
||||
includeNameMatchMetadata &&
|
||||
(metadata?.dictionaries ?? []).some((name) =>
|
||||
name.startsWith(CHARACTER_DICTIONARY_TITLE_PREFIX),
|
||||
);
|
||||
|
||||
try {
|
||||
const rawResult = await parserWindow.webContents.executeJavaScript(
|
||||
@@ -1501,7 +1675,8 @@ export async function requestYomitanScanTokens(
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
options?.includeNameMatchMetadata === true,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
typeof options?.currentCharacterDictionaryMediaId === 'number' &&
|
||||
Number.isFinite(options.currentCharacterDictionaryMediaId) &&
|
||||
options.currentCharacterDictionaryMediaId > 0
|
||||
@@ -1514,21 +1689,13 @@ export async function requestYomitanScanTokens(
|
||||
);
|
||||
if (isScanTokenArray(rawResult)) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return hasSameTokenSpans(parseScanTokens, rawResult) ? rawResult : parseScanTokens;
|
||||
return mergeScannerTokensIntoParseTokens(parseScanTokens, rawResult);
|
||||
}
|
||||
return rawResult;
|
||||
}
|
||||
if (Array.isArray(rawResult)) {
|
||||
const selectedTokens = selectYomitanParseTokens(rawResult, () => false, 'headword');
|
||||
return (
|
||||
selectedTokens?.map((token) => ({
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
})) ?? null
|
||||
);
|
||||
return selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
}
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
|
||||
@@ -4652,7 +4652,8 @@ const {
|
||||
setYomitanParserInitPromise: (promise) => {
|
||||
appState.yomitanParserInitPromise = promise;
|
||||
},
|
||||
isKnownWord: (text, reading) => Boolean(appState.ankiIntegration?.isKnownWord(text, reading)),
|
||||
isKnownWord: (text, reading, options) =>
|
||||
Boolean(appState.ankiIntegration?.isKnownWord(text, reading, options)),
|
||||
recordLookup: (hit) => {
|
||||
ensureImmersionTrackerStarted();
|
||||
appState.immersionTracker?.recordLookup(hit);
|
||||
|
||||
@@ -37,8 +37,8 @@ export function createBuildTokenizerDepsMainHandler(deps: TokenizerMainDeps) {
|
||||
getYomitanParserInitPromise: () => deps.getYomitanParserInitPromise(),
|
||||
setYomitanParserInitPromise: (promise: Promise<boolean> | null) =>
|
||||
deps.setYomitanParserInitPromise(promise),
|
||||
isKnownWord: (text: string, reading?: string) => {
|
||||
const hit = deps.isKnownWord(text, reading);
|
||||
isKnownWord: (text, reading, options) => {
|
||||
const hit = deps.isKnownWord(text, reading, options);
|
||||
deps.recordLookup(hit);
|
||||
return hit;
|
||||
},
|
||||
|
||||
@@ -3,6 +3,7 @@ import { normalizePos1ExclusionList } from './token-pos1-exclusions';
|
||||
|
||||
export const DEFAULT_ANNOTATION_POS2_EXCLUSION_DEFAULTS = Object.freeze([
|
||||
'非自立',
|
||||
'接尾',
|
||||
]) as readonly string[];
|
||||
|
||||
export const DEFAULT_ANNOTATION_POS2_EXCLUSION_CONFIG: ResolvedTokenPos2ExclusionConfig = {
|
||||
|
||||
@@ -40,6 +40,12 @@ export interface MergedToken {
|
||||
isMerged: boolean;
|
||||
isKnown: boolean;
|
||||
isNPlusOneTarget: boolean;
|
||||
/**
|
||||
* Text Yomitan had no dictionary entry for (e.g. ぅ~ elongation runs,
|
||||
* truncated inflections). Kept as a token so it stays hoverable, but
|
||||
* ignored by annotation, N+1, and vocabulary-stats logic.
|
||||
*/
|
||||
isUnparsedRun?: boolean;
|
||||
isNameMatch?: boolean;
|
||||
characterImage?: CharacterNameImage;
|
||||
jlptLevel?: JlptLevel;
|
||||
|
||||
|
Before Width: | Height: | Size: 6.5 KiB After Width: | Height: | Size: 6.2 KiB |