Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
695613b1e7
|
||
|
|
20797772f8 | ||
|
|
72c312810a
|
||
|
|
b61f050e78
|
@@ -15,7 +15,7 @@ Integrates Yomitan and mpv - on-screen lookups, mine to Anki, and track immersio
|
||||
[](https://www.gnu.org/licenses/gpl-3.0)
|
||||
[](https://www.typescriptlang.org)
|
||||
|
||||
[](https://github.com/user-attachments/assets/89e61895-e2b7-4b47-8d50-a35afe4132b2)
|
||||
[](https://github.com/user-attachments/assets/7abab8a9-4e4e-4f06-9f3c-9783e15a3807)
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
Before Width: | Height: | Size: 114 KiB |
|
Before Width: | Height: | Size: 23 MiB |
|
Before Width: | Height: | Size: 303 KiB |
|
Before Width: | Height: | Size: 3.0 MiB After Width: | Height: | Size: 3.8 MiB |
@@ -0,0 +1,5 @@
|
||||
type: fixed
|
||||
area: character dictionary
|
||||
|
||||
- Reuse character dictionaries after MeCab completes without finding any name splits instead of regenerating character data and portraits on every launch.
|
||||
- Restore inline character portraits when a cached portrait index finishes loading after subtitles have already been tokenized.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: docs
|
||||
area: documentation
|
||||
|
||||
- Hid the unfinished feature demos page from the documentation sidebar while keeping its direct URL available.
|
||||
@@ -5,7 +5,7 @@ Short recordings of SubMiner's key features and integrations from real playback
|
||||
<script setup>
|
||||
import { withBase } from 'vitepress';
|
||||
|
||||
const v = '20260301-1';
|
||||
const v = '20260819-1';
|
||||
</script>
|
||||
|
||||
## Anki Card Mining & Enrichment
|
||||
|
||||
@@ -88,7 +88,7 @@ features:
|
||||
<script setup>
|
||||
import { withBase } from 'vitepress';
|
||||
|
||||
const demoAssetVersion = '20260223-2';
|
||||
const demoAssetVersion = '20260819-1';
|
||||
</script>
|
||||
|
||||
<div class="landing-shell">
|
||||
|
||||
|
Before Width: | Height: | Size: 56 KiB After Width: | Height: | Size: 160 KiB |
|
Before Width: | Height: | Size: 23 MiB |
|
Before Width: | Height: | Size: 303 KiB |
|
Before Width: | Height: | Size: 523 KiB |
|
Before Width: | Height: | Size: 3.0 MiB After Width: | Height: | Size: 3.8 MiB |
@@ -2,7 +2,7 @@
|
||||
"name": "subminer",
|
||||
"productName": "SubMiner",
|
||||
"desktopName": "SubMiner.desktop",
|
||||
"version": "0.19.4-beta.1",
|
||||
"version": "0.19.4-beta.2",
|
||||
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
|
||||
"packageManager": "bun@1.3.5",
|
||||
"main": "dist/main-entry.js",
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
> This is a prerelease build for testing. Stable changelog and docs-site updates remain pending until the final stable release.
|
||||
|
||||
<!-- prerelease-base-version: 0.19.4 -->
|
||||
|
||||
## Highlights
|
||||
### Added
|
||||
|
||||
- Library Merge & Reassignment
|
||||
- Duplicate library entries for the same show can now be merged: pick entries in "Select" mode and use "Merge Selected" to combine sessions, mined cards, and watch time onto one card.
|
||||
- Episodes can be moved to a different library entry with a per-episode "→" button, fixing cases where a stray filename split off its own entry; manual assignments now survive later filename parsing, Jellyfin refreshes, and season repair.
|
||||
- Exact AniList matches with compatible seasons now merge automatically, and likely (fuzzy) matches surface as a dismissible "Possible duplicate" suggestion instead of merging silently.
|
||||
|
||||
- Duplicate Line Cleanup
|
||||
- The Vocabulary tab's new **Duplicates** button scans a chosen time window for the repeated-line bursts described under Fixed below and collapses each burst to a single line once you confirm it; a matching `subminer stats cleanup --duplicate-lines` command (with `--dry-run` and `--lookback-days <n>`) is available from the terminal.
|
||||
- Only the affected subtitle lines and the vocabulary counts they inflated are touched; watch time and lines-seen totals are left as recorded.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Subtitle Duplication from Karaoke & Animated Signs
|
||||
- Typeset ASS karaoke and animated signs no longer flood the overlay, subtitle sidebar, immersion history, mined cards, or stats with repeated glyph fragments or per-frame duplicates; the complete authored line is recovered instead, without merging genuinely repeated dialogue or separately positioned signs.
|
||||
- The secondary overlay now shares the same deduplication logic as the primary overlay, so layered animation text no longer appears multiple times there or in what gets mined.
|
||||
- Vocabulary stats no longer count every animation frame of a karaoke opening as a separate line, which previously could push an OP lyric to the top of "Top Repeated Words."
|
||||
|
||||
- Anki Media Generation
|
||||
- Sentence-audio generation no longer times out on slow network-mounted video files with many subtitle and font streams, and a failed extraction now reports a clear error instead of a raw `ENOENT`.
|
||||
- Mined audio and animated AVIF clips now capture the subtitle line you actually mined, instead of whatever line happened to be on screen once slow audio extraction finished.
|
||||
|
||||
- Character Dictionary Performance & Notifications
|
||||
- Character dictionary generation, merged rebuilds, and imports no longer freeze the app on large dictionaries, and cached results are reused across launches instead of regenerating character data and portraits every time.
|
||||
- Desktop progress notifications, including on Linux AppImage installs, now update in place instead of flickering closed and reopening.
|
||||
|
||||
- Overlay Reliability
|
||||
- Overlay modals (settings, stats, etc.) now open promptly on the first shortcut press and appear above fullscreen mpv on macOS instead of switching Spaces or opening off-screen.
|
||||
- The overlay no longer gets stuck on "Overlay loading" indefinitely if mpv's connection stalls; it now retries and shows an actionable error after 30 seconds.
|
||||
- Fixed native Wayland drag-and-drop from file managers like Thunar, so subtitle and video files dropped on the overlay reach mpv.
|
||||
- Fixed system-wide mouse lag on Windows caused by the overlay's click-through handling and repeated mpv window lookups.
|
||||
|
||||
- Stats Dashboard
|
||||
- Deletes, library merges, video moves, and AniList reassignments no longer freeze the stats dashboard or rebuild lifetime totals from scratch; large deletes that used to take minutes now finish in milliseconds.
|
||||
- Vocabulary totals and charts now count all tracked vocabulary instead of just the first page, new-word history uses corrected daily rollups, calendar labels respect time zones west of UTC, and vocabulary cards refresh automatically after editing the word exclusion list.
|
||||
|
||||
- Linux Launcher Thumbnails
|
||||
- Fixed missing MKV thumbnails in the Linux rofi picker when the system thumbnailer only registers legacy Matroska MIME aliases.
|
||||
|
||||
## What's Changed
|
||||
|
||||
- feat(stats): add library entry merge and episode move by @ksyasuda in #190
|
||||
- fix(stats): stop counting duplicate typeset subtitle lines by @ksyasuda in #191
|
||||
- fix(media): tolerate slow MKV audio extraction by @ksyasuda in #195
|
||||
- fix(stats): subtract lifetime totals incrementally on delete by @ksyasuda in #196
|
||||
- fix(anki): snapshot mining media clip timing by @ksyasuda in #197
|
||||
- fix(notifications): replace Linux progress updates in place by @ksyasuda in #198
|
||||
- fix(overlay): support native Wayland file drag-and-drop by @ksyasuda in #199
|
||||
- fix(overlay): keep macOS modal windows on fullscreen Spaces by @ksyasuda in #200
|
||||
- fix(overlay): prevent Windows mouse lag during click-through tracking by @ksyasuda in #201
|
||||
- fix(stats): report complete vocabulary totals and new-word history by @ksyasuda in #202
|
||||
- fix(mpv): recover from stalled IPC connects by @ksyasuda in #204
|
||||
- fix(dictionary): prevent freezes and restore AppImage notifications by @ksyasuda in #205
|
||||
- fix(subtitles): recover canonical lines from ASS animation by @ksyasuda in #207
|
||||
- fix(overlay): deduplicate secondary subtitle rendering by @ksyasuda in #208
|
||||
- fix(launcher): restore Matroska thumbnails in Linux rofi picker by @ksyasuda in #210
|
||||
- fix(character-dictionary): cache completed MeCab refreshes by @ksyasuda in #212
|
||||
|
||||
## Installation
|
||||
|
||||
See the README and docs/installation guide for full setup steps.
|
||||
|
||||
## Assets
|
||||
|
||||
- Linux: `SubMiner.AppImage`
|
||||
- macOS: `SubMiner-*.dmg` and `SubMiner-*.zip`
|
||||
- Windows: `SubMiner-*.exe` and `SubMiner-*-win.zip`
|
||||
- Optional extras: `subminer-assets.tar.gz` and the `subminer` launcher
|
||||
|
||||
Note: the `subminer` wrapper script uses Bun (`#!/usr/bin/env bun`), so `bun` must be installed and on `PATH`.
|
||||
@@ -19,7 +19,8 @@ Options:
|
||||
-w, --webp Generate animated WebP preview
|
||||
|
||||
Encoding profile:
|
||||
- Crop: 1920x1080 at x=760 y=200
|
||||
- Crop: mpv region at 1920x1080, x=760 y=205 on a 3440x1440 canvas
|
||||
- Output size: 1920x1080
|
||||
- MP4: H.264 + AAC
|
||||
- WebM: AV1/VP9 + Opus at 30 fps
|
||||
USAGE
|
||||
@@ -148,7 +149,8 @@ pick_webp_encoder() {
|
||||
return 1
|
||||
}
|
||||
|
||||
crop_vf="crop=1920:1080:760:205"
|
||||
# OBS may resize the 3440x1440 canvas, so scale the mpv bounds with the input.
|
||||
crop_vf="crop=1920*iw/3440:1080*ih/1440:760*iw/3440:205*ih/1440,scale=1920:1080:flags=lanczos"
|
||||
webm_vf="${crop_vf},fps=30"
|
||||
|
||||
echo "Generating MP4: $mp4_out"
|
||||
|
||||
@@ -40,7 +40,7 @@ function toBashPath(filePath: string): string {
|
||||
return `${drive.toUpperCase()}:/${rest}`;
|
||||
}
|
||||
|
||||
test('mkv-to-readme-video accepts libwebp_anim when libwebp is unavailable', () => {
|
||||
test('mkv-to-readme-video builds every output with the scaled mpv crop', () => {
|
||||
withTempDir((root) => {
|
||||
const binDir = path.join(root, 'bin');
|
||||
const inputPath = path.join(root, 'sample.mkv');
|
||||
@@ -104,5 +104,9 @@ touch "$output"
|
||||
|
||||
const ffmpegLog = fs.readFileSync(ffmpegLogPath, 'utf8');
|
||||
assert.match(ffmpegLog, /-c:v libwebp_anim/);
|
||||
const scaledCropUses = ffmpegLog.match(
|
||||
/-vf crop=1920\*iw\/3440:1080\*ih\/1440:760\*iw\/3440:205\*ih\/1440,scale=1920:1080:flags=lanczos/g,
|
||||
);
|
||||
assert.equal(scaledCropUses?.length, 4);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2623,6 +2623,12 @@ const characterDictionaryAutoSyncRuntime = createCharacterDictionaryAutoSyncRunt
|
||||
const characterDictionaryImageLookup = createCharacterDictionaryImageLookup({
|
||||
userDataPath: USER_DATA_PATH,
|
||||
getCurrentMediaId: () => characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
onIndexReady: () => refreshCurrentSubtitleAnnotations(),
|
||||
onIndexReadyError: (error) =>
|
||||
logger.warn(
|
||||
'Failed to refresh subtitle annotations after character portrait index became ready.',
|
||||
error,
|
||||
),
|
||||
});
|
||||
|
||||
// Lets the Yomitan scan runtime skip name lookups at positions where no
|
||||
@@ -4047,7 +4053,7 @@ const recordTrackedCardsMined = (count: number, noteIds?: number[]): void => {
|
||||
ensureImmersionTrackerStarted();
|
||||
appState.immersionTracker?.recordCardsMined(count, noteIds);
|
||||
};
|
||||
const refreshCurrentSubtitleAfterKnownWordUpdate = (): void => {
|
||||
function refreshCurrentSubtitleAnnotations(): void {
|
||||
const hasCurrentSubtitle = appState.currentSubText.trim().length > 0;
|
||||
if (hasCurrentSubtitle) {
|
||||
subtitlePrefetchService?.pause();
|
||||
@@ -4058,7 +4064,7 @@ const refreshCurrentSubtitleAfterKnownWordUpdate = (): void => {
|
||||
// Idle controller: no settle is coming to release the pause above.
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
};
|
||||
}
|
||||
let hasAttemptedImmersionTrackerStartup = false;
|
||||
const ensureImmersionTrackerStarted = (): void => {
|
||||
if (hasAttemptedImmersionTrackerStartup || appState.immersionTracker) {
|
||||
@@ -5081,9 +5087,7 @@ function initializeOverlayRuntime(): void {
|
||||
overlayModalRuntime.primeModalWindow();
|
||||
}
|
||||
appState.ankiIntegration?.setRecordCardsMinedCallback(recordTrackedCardsMined);
|
||||
appState.ankiIntegration?.setKnownWordCacheUpdatedCallback(
|
||||
refreshCurrentSubtitleAfterKnownWordUpdate,
|
||||
);
|
||||
appState.ankiIntegration?.setKnownWordCacheUpdatedCallback(refreshCurrentSubtitleAnnotations);
|
||||
appState.ankiIntegration?.setSubtitleMiningContextConsumer(consumePendingSubtitleMiningContext);
|
||||
syncOverlayMpvSubtitleSuppression();
|
||||
}
|
||||
@@ -5876,7 +5880,7 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
appState.ankiIntegration = integration;
|
||||
appState.ankiIntegration?.setRecordCardsMinedCallback(recordTrackedCardsMined);
|
||||
appState.ankiIntegration?.setKnownWordCacheUpdatedCallback(
|
||||
refreshCurrentSubtitleAfterKnownWordUpdate,
|
||||
refreshCurrentSubtitleAnnotations,
|
||||
);
|
||||
appState.ankiIntegration?.setSubtitleMiningContextConsumer(
|
||||
consumePendingSubtitleMiningContext,
|
||||
|
||||
@@ -450,7 +450,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
}
|
||||
|
||||
const nameSplitTokenizerAvailable = isNameSplitTokenizerAvailable();
|
||||
const resolvedNameSplits = nameSplitTokenizerAvailable
|
||||
const nameSplitResolution = nameSplitTokenizerAvailable
|
||||
? await resolveJapaneseNameSplits(
|
||||
characters,
|
||||
deps.tokenizeJapaneseName!,
|
||||
@@ -466,8 +466,8 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
},
|
||||
)
|
||||
: undefined;
|
||||
const nameSplitSource =
|
||||
resolvedNameSplits && resolvedNameSplits.size > 0 ? 'mecab' : 'heuristic';
|
||||
const resolvedNameSplits = nameSplitResolution?.splits;
|
||||
const nameSplitSource = nameSplitResolution?.kind === 'complete' ? 'mecab' : 'heuristic';
|
||||
|
||||
progress?.onGenerateProgress?.({
|
||||
mediaId,
|
||||
|
||||
@@ -198,6 +198,66 @@ test('createCharacterDictionaryImageLookup can scope duplicate names to the curr
|
||||
assert.equal(scoped.alt, 'Kazuma');
|
||||
});
|
||||
|
||||
test('createCharacterDictionaryImageLookup reports and retries a failed index-ready callback', async () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
mediaId: 21858,
|
||||
mediaTitle: 'Little Witch Academia',
|
||||
entryCount: 1,
|
||||
updatedAt: 1_700_000_000_000,
|
||||
termEntries: [
|
||||
[
|
||||
'ダイアナ',
|
||||
'だいあな',
|
||||
'name primary',
|
||||
'',
|
||||
75,
|
||||
[
|
||||
{
|
||||
type: 'structured-content',
|
||||
content: {
|
||||
tag: 'img',
|
||||
path: 'img/m21858-c81709.png',
|
||||
alt: 'ダイアナ・キャベンディッシュ',
|
||||
},
|
||||
},
|
||||
],
|
||||
0,
|
||||
'',
|
||||
],
|
||||
],
|
||||
images: [{ path: 'img/m21858-c81709.png', dataBase64: PNG_1X1_BASE64 }],
|
||||
};
|
||||
await writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
const callbackError = new Error('annotation refresh failed');
|
||||
const reportingError = new Error('error reporter failed');
|
||||
let readyCount = 0;
|
||||
const reportedErrors: unknown[] = [];
|
||||
const lookup = createCharacterDictionaryImageLookup({
|
||||
outputDir,
|
||||
onIndexReady: () => {
|
||||
readyCount += 1;
|
||||
if (readyCount === 1) {
|
||||
throw callbackError;
|
||||
}
|
||||
},
|
||||
onIndexReadyError: (error) => {
|
||||
reportedErrors.push(error);
|
||||
throw reportingError;
|
||||
},
|
||||
});
|
||||
|
||||
assert.equal(lookup.get('ダイアナ', snapshot.mediaId), null);
|
||||
await waitForRefresh(() => (reportedErrors.length === 1 ? true : null));
|
||||
assert.ok(lookup.get('ダイアナ', snapshot.mediaId));
|
||||
|
||||
assert.equal(readyCount, 2);
|
||||
assert.deepEqual(reportedErrors, [callbackError]);
|
||||
lookup.get('ダイアナ', snapshot.mediaId);
|
||||
assert.equal(readyCount, 2);
|
||||
});
|
||||
|
||||
test('createCharacterDictionaryImageLookup does not fall back globally on scoped miss', async () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
|
||||
@@ -218,6 +218,8 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
userDataPath?: string;
|
||||
outputDir?: string;
|
||||
getCurrentMediaId?: () => number | null | undefined;
|
||||
onIndexReady?: () => void;
|
||||
onIndexReadyError?: (error: unknown) => void;
|
||||
}): {
|
||||
get: (term: string, mediaId?: number | null) => CharacterNameImage | null;
|
||||
invalidate: () => void;
|
||||
@@ -229,6 +231,24 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
let index = new Map<string, CharacterNameImage>();
|
||||
let indexByMediaId = new Map<number, Map<string, CharacterNameImage>>();
|
||||
let refreshInFlight = false;
|
||||
let indexReadyDeliveryPending = false;
|
||||
|
||||
function deliverIndexReadyIfPending(): void {
|
||||
if (!indexReadyDeliveryPending || !deps.onIndexReady) {
|
||||
return;
|
||||
}
|
||||
indexReadyDeliveryPending = false;
|
||||
try {
|
||||
deps.onIndexReady();
|
||||
} catch (error) {
|
||||
indexReadyDeliveryPending = true;
|
||||
try {
|
||||
deps.onIndexReadyError?.(error);
|
||||
} catch {
|
||||
// Error reporting must not reject the detached index refresh task.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Rebuilding means re-reading every cached snapshot (potentially GBs of JSON), which used to run
|
||||
// synchronously inside a lookup and froze the whole app right after a snapshot changed. Lookups
|
||||
@@ -241,6 +261,7 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
signature = '';
|
||||
return;
|
||||
}
|
||||
deliverIndexReadyIfPending();
|
||||
const nextSignature = getSnapshotDirectorySignature(outputDir);
|
||||
if (nextSignature === signature || refreshInFlight) {
|
||||
return;
|
||||
@@ -262,6 +283,8 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
index = nextIndex;
|
||||
indexByMediaId = nextIndexByMediaId;
|
||||
signature = nextSignature;
|
||||
indexReadyDeliveryPending = deps.onIndexReady !== undefined;
|
||||
deliverIndexReadyIfPending();
|
||||
} finally {
|
||||
refreshInFlight = false;
|
||||
}
|
||||
|
||||
@@ -43,7 +43,8 @@ test('resolveJapaneseNameSplits splits a single-kanji surname via person-name PO
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(splits.get('東紫乃'), { family: '東', given: '紫乃' });
|
||||
assert.equal(splits.kind, 'complete');
|
||||
assert.deepEqual(splits.splits.get('東紫乃'), { family: '東', given: '紫乃' });
|
||||
});
|
||||
|
||||
test('resolveJapaneseNameSplits corrects a hint-length-misleading surname boundary', async () => {
|
||||
@@ -64,7 +65,8 @@ test('resolveJapaneseNameSplits corrects a hint-length-misleading surname bounda
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(splits.get('渡辺真奈美'), { family: '渡辺', given: '真奈美' });
|
||||
assert.equal(splits.kind, 'complete');
|
||||
assert.deepEqual(splits.splits.get('渡辺真奈美'), { family: '渡辺', given: '真奈美' });
|
||||
});
|
||||
|
||||
test('resolveJapaneseNameSplits falls back to hint readings when POS tags are generic', async () => {
|
||||
@@ -85,7 +87,8 @@ test('resolveJapaneseNameSplits falls back to hint readings when POS tags are ge
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(splits.get('鈴木みゆ'), { family: '鈴木', given: 'みゆ' });
|
||||
assert.equal(splits.kind, 'complete');
|
||||
assert.deepEqual(splits.splits.get('鈴木みゆ'), { family: '鈴木', given: 'みゆ' });
|
||||
});
|
||||
|
||||
test('resolveJapaneseNameSplits skips names whose tokens do not reconstruct the name', async () => {
|
||||
@@ -96,7 +99,8 @@ test('resolveJapaneseNameSplits skips names whose tokens do not reconstruct the
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(splits.size, 0);
|
||||
assert.equal(splits.kind, 'complete');
|
||||
assert.equal(splits.splits.size, 0);
|
||||
});
|
||||
|
||||
test('resolveJapaneseNameSplits skips ambiguous or untagged segmentations', async () => {
|
||||
@@ -117,7 +121,8 @@ test('resolveJapaneseNameSplits skips ambiguous or untagged segmentations', asyn
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(splits.size, 0);
|
||||
assert.equal(splits.kind, 'complete');
|
||||
assert.equal(splits.splits.size, 0);
|
||||
});
|
||||
|
||||
test('resolveJapaneseNameSplits survives tokenizer failures', async () => {
|
||||
@@ -130,7 +135,8 @@ test('resolveJapaneseNameSplits survives tokenizer failures', async () => {
|
||||
(message) => warnings.push(message),
|
||||
);
|
||||
|
||||
assert.equal(splits.size, 0);
|
||||
assert.equal(splits.kind, 'incomplete');
|
||||
assert.equal(splits.splits.size, 0);
|
||||
assert.equal(warnings.length, 1);
|
||||
assert.match(warnings[0]!, /mecab unavailable/);
|
||||
});
|
||||
|
||||
@@ -7,6 +7,10 @@ import type {
|
||||
ResolvedNameSplit,
|
||||
} from './types';
|
||||
|
||||
export type JapaneseNameSplitResolution =
|
||||
| { kind: 'complete'; splits: Map<string, ResolvedNameSplit> }
|
||||
| { kind: 'incomplete'; splits: Map<string, ResolvedNameSplit> };
|
||||
|
||||
const NAME_SEPARATOR_PATTERN = /[\s ・・·•]/;
|
||||
|
||||
function joinSurfaces(tokens: NameSplitToken[]): string {
|
||||
@@ -87,8 +91,9 @@ export async function resolveJapaneseNameSplits(
|
||||
tokenize: NameSplitTokenizer,
|
||||
logWarn?: (message: string) => void,
|
||||
onCharacterResolved?: (completed: number, total: number) => void,
|
||||
): Promise<Map<string, ResolvedNameSplit>> {
|
||||
): Promise<JapaneseNameSplitResolution> {
|
||||
const splits = new Map<string, ResolvedNameSplit>();
|
||||
let tokenizerFailed = false;
|
||||
let resolvedCharacters = 0;
|
||||
for (const character of characters) {
|
||||
const familyHintReading = buildReadingFromHint(character.lastNameHint?.trim() || '');
|
||||
@@ -99,12 +104,17 @@ export async function resolveJapaneseNameSplits(
|
||||
try {
|
||||
tokens = await tokenize(name);
|
||||
} catch (err) {
|
||||
tokenizerFailed = true;
|
||||
logWarn?.(
|
||||
`[dictionary] name split tokenization failed for "${name}": ${(err as Error).message}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (!tokens || tokens.length < 2 || joinSurfaces(tokens) !== name) continue;
|
||||
if (!tokens) {
|
||||
tokenizerFailed = true;
|
||||
continue;
|
||||
}
|
||||
if (tokens.length < 2 || joinSurfaces(tokens) !== name) continue;
|
||||
const splitIndex =
|
||||
splitIndexFromPersonNamePos(tokens) ??
|
||||
splitIndexFromHintReadings(tokens, familyHintReading, givenHintReading);
|
||||
@@ -118,5 +128,5 @@ export async function resolveJapaneseNameSplits(
|
||||
resolvedCharacters += 1;
|
||||
onCharacterResolved?.(resolvedCharacters, characters.length);
|
||||
}
|
||||
return splits;
|
||||
return tokenizerFailed ? { kind: 'incomplete', splits } : { kind: 'complete', splits };
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ import test from 'node:test';
|
||||
import { createCharacterDictionaryRuntimeService } from '../character-dictionary-runtime';
|
||||
import { getSnapshotPath, writeSnapshot } from './cache';
|
||||
import { CHARACTER_DICTIONARY_FORMAT_VERSION } from './constants';
|
||||
import type { CharacterDictionarySnapshot } from './types';
|
||||
import type { CharacterDictionarySnapshot, NameSplitTokenizer } from './types';
|
||||
|
||||
const GRAPHQL_URL = 'https://graphql.anilist.co';
|
||||
const PNG_1X1 = Buffer.from(
|
||||
@@ -121,7 +121,12 @@ test('generateForCurrentMedia refreshes same-version snapshots missing images wh
|
||||
}
|
||||
});
|
||||
|
||||
test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable', async () => {
|
||||
async function runNameSplitRefreshScenario(tokenizeJapaneseName: NameSplitTokenizer): Promise<{
|
||||
characterPageRequests: number;
|
||||
firstResultFromCache: boolean;
|
||||
refreshedNameSplitSource: CharacterDictionarySnapshot['nameSplitSource'];
|
||||
secondResultFromCache: boolean;
|
||||
}> {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
@@ -172,7 +177,6 @@ test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable'
|
||||
}) as typeof globalThis.fetch;
|
||||
|
||||
try {
|
||||
let tokenizerCalls = 0;
|
||||
const runtime = createCharacterDictionaryRuntimeService({
|
||||
userDataPath,
|
||||
getCurrentMediaPath: () => '/tmp/eminence-s01e05.mkv',
|
||||
@@ -185,29 +189,54 @@ test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable'
|
||||
source: 'fallback',
|
||||
}),
|
||||
getNameMatchImagesEnabled: () => false,
|
||||
tokenizeJapaneseName: async () => {
|
||||
tokenizerCalls += 1;
|
||||
return null;
|
||||
},
|
||||
tokenizeJapaneseName,
|
||||
getJapaneseNameTokenizerAvailable: () => true,
|
||||
now: () => 1_700_000_000_500,
|
||||
});
|
||||
|
||||
const result = await runtime.generateForCurrentMedia();
|
||||
const firstResult = await runtime.generateForCurrentMedia();
|
||||
const refreshedSnapshot = JSON.parse(
|
||||
fs.readFileSync(getSnapshotPath(outputDir, 130298), 'utf8'),
|
||||
) as CharacterDictionarySnapshot;
|
||||
const secondResult = await runtime.generateForCurrentMedia();
|
||||
|
||||
assert.equal(result.fromCache, false);
|
||||
assert.equal(refreshedSnapshot.nameSplitSource, 'heuristic');
|
||||
|
||||
const retriedResult = await runtime.generateForCurrentMedia();
|
||||
assert.equal(retriedResult.fromCache, false);
|
||||
assert.equal(characterPageRequests, 2);
|
||||
assert.equal(tokenizerCalls, 2);
|
||||
return {
|
||||
characterPageRequests,
|
||||
firstResultFromCache: firstResult.fromCache,
|
||||
refreshedNameSplitSource: refreshedSnapshot.nameSplitSource,
|
||||
secondResultFromCache: secondResult.fromCache,
|
||||
};
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
}
|
||||
|
||||
test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable', async () => {
|
||||
let tokenizerCalls = 0;
|
||||
const result = await runNameSplitRefreshScenario(async () => {
|
||||
tokenizerCalls += 1;
|
||||
return null;
|
||||
});
|
||||
|
||||
assert.equal(result.firstResultFromCache, false);
|
||||
assert.equal(result.refreshedNameSplitSource, 'heuristic');
|
||||
assert.equal(result.secondResultFromCache, false);
|
||||
assert.equal(result.characterPageRequests, 2);
|
||||
assert.equal(tokenizerCalls, 2);
|
||||
});
|
||||
|
||||
test('generateForCurrentMedia caches completed MeCab refreshes with no resolved splits', async () => {
|
||||
let tokenizerCalls = 0;
|
||||
const result = await runNameSplitRefreshScenario(async () => {
|
||||
tokenizerCalls += 1;
|
||||
return [];
|
||||
});
|
||||
|
||||
assert.equal(result.firstResultFromCache, false);
|
||||
assert.equal(result.refreshedNameSplitSource, 'mecab');
|
||||
assert.equal(result.secondResultFromCache, true);
|
||||
assert.equal(result.characterPageRequests, 1);
|
||||
assert.equal(tokenizerCalls, 1);
|
||||
});
|
||||
|
||||
test('generateForCurrentMedia keeps mecab-split snapshots when MeCab is available', async () => {
|
||||
|
||||
@@ -482,10 +482,10 @@ test('Linux visible overlay recreation avoids display fallback before tracked ge
|
||||
assert.doesNotMatch(actionBlock, /setOverlayWindowBounds\(getCurrentOverlayGeometry\(\)\)/);
|
||||
});
|
||||
|
||||
test('known-word updates invalidate prefetched tokenizations before refreshing current subtitle', () => {
|
||||
test('subtitle annotation updates invalidate prefetched tokenizations before refreshing current subtitle', () => {
|
||||
const source = readMainSource();
|
||||
const actionBlock = source.match(
|
||||
/const refreshCurrentSubtitleAfterKnownWordUpdate = \(\): void => \{(?<body>[\s\S]*?)\n\};/,
|
||||
/function refreshCurrentSubtitleAnnotations\(\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(actionBlock);
|
||||
@@ -503,6 +503,20 @@ test('known-word updates invalidate prefetched tokenizations before refreshing c
|
||||
);
|
||||
});
|
||||
|
||||
test('character portrait index readiness refreshes cached subtitle annotations', () => {
|
||||
const source = readMainSource();
|
||||
const lookupDeps = source.match(
|
||||
/const characterDictionaryImageLookup = createCharacterDictionaryImageLookup\(\{(?<body>[\s\S]*?)\n\}\);/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(lookupDeps);
|
||||
assert.match(lookupDeps, /onIndexReady: \(\) => refreshCurrentSubtitleAnnotations\(\),/);
|
||||
assert.match(
|
||||
lookupDeps,
|
||||
/onIndexReadyError: \(error\) =>[\s\S]*?logger\.warn\([\s\S]*?character portrait index became ready\.[\s\S]*?error,/,
|
||||
);
|
||||
});
|
||||
|
||||
test('subtitle processing controller resumes prefetch on settle, not on its emits', () => {
|
||||
const source = readMainSource();
|
||||
const depsBlock = source.match(
|
||||
|
||||