mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-18 00:21:41 -07:00
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5af519dc6c
|
||
|
|
7046a4451f
|
||
|
|
39286c2c55
|
@@ -0,0 +1,5 @@
|
||||
type: fixed
|
||||
area: subtitles
|
||||
|
||||
- Typeset ASS karaoke and animated signs no longer flood the primary overlay, subtitle sidebar, immersion history, or sentence mining with repeated glyph fragments or full-line color phases. Matching timed comments and full-line boundary events recover the complete authored line without merging ordinary repeated dialogue or separately positioned signs, and dialogue spoken while a song's animation is on screen is kept intact instead of being replaced by the lyric. Entrance and exit frames that run past the authored line timing still resolve to the clean line during lyric transitions, and dialogue spoken while a song's animation is on screen enters immersion and subtitle history without the fragment lines beside it.
|
||||
- The secondary subtitle overlay drops layered duplicate lines from animated tracks, so a short stack of repeated words collapses to its distinct lines even when the full karaoke heuristic does not apply.
|
||||
@@ -1,5 +0,0 @@
|
||||
type: fixed
|
||||
area: dictionary
|
||||
|
||||
- Character dictionary generation, merged rebuilds, and imports no longer freeze the app (and trigger the compositor's "application not responding" dialog) on large dictionaries; snapshot reads/writes, archive building, and the character image/name lookup caches now do their heavy work off the UI's critical path.
|
||||
- Desktop progress notifications now update in place on Linux AppImage installs too: the AppImage's bundled libraries broke the system notify-send helper, which silently forced the flickering close-and-reopen notification fallback.
|
||||
@@ -138,6 +138,8 @@ Karaoke openings and animated signs are authored as one subtitle event per anima
|
||||
|
||||
Recording now collapses those runs as they happen, matching what the subtitle sidebar shows:
|
||||
|
||||
- When a typeset ASS file stores a clean lyric or sign in a timed authoring comment, or in full-line events surrounding generated fragments, the matching complete line is recorded once. The repeated glyph or clip-animation frames are not recorded. Dialogue spoken while such an animation is on screen records as itself, without the fragment lines beside it.
|
||||
- When karaoke styling redraws the same complete lyric across consecutive color or highlight phases, those phases are combined into one line with their full timing. Repeated ordinary dialogue remains separate.
|
||||
- When the active subtitle source has been parsed, its cue list has already had duplicate events and animation bursts merged. A line landing inside a surviving cue but after that cue's start is a frame the sidebar merged away, and is not recorded.
|
||||
- When no parsed cue covers the live timing, including while a subtitle source is changing or shifted, the strict metadata-free rule applies: a run of identical, contiguous lines each shorter than 0.1s stops being recorded after a few frames. Runs are tracked per line of text, so dual-line karaoke (a kanji and a romaji line frame-flipped together) collapses both lines. Ordinary repeated dialogue, and lines held for a normal beat, always record.
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ When SubMiner parses the active subtitle source into a cue list, the sidebar bec
|
||||
- Clicking any cue seeks mpv to that timestamp.
|
||||
- The sidebar stays synchronized with the overlay - media transitions and subtitle source changes update both simultaneously.
|
||||
|
||||
For typeset ASS karaoke and animated signs, SubMiner collapses generated animation frames and repeated full-line color phases before they reach the sidebar. It recovers a clean complete line from a matching timed authoring comment or from full-line events surrounding generated fragments. Ordinary ASS comments, editor notes, alternate lines, repeated dialogue, and separately positioned signs remain distinct.
|
||||
|
||||
The sidebar only appears when a parsed cue list is available. External subtitle sources that SubMiner cannot parse (for example, embedded ASS tracks rendered directly by mpv) will not populate the sidebar.
|
||||
|
||||
## Layout Modes
|
||||
|
||||
@@ -70,18 +70,25 @@ interface SubtitleCue {
|
||||
startTime: number; // seconds
|
||||
endTime: number; // seconds
|
||||
text: string; // plain text, decoded from the source format
|
||||
source?: 'canonical-ass'; // recovered authored text for generated ASS animation
|
||||
animationStartTime?: number; // full generated-frame envelope; entrance/exit frames
|
||||
animationEndTime?: number; // run past the authored timing, live matching uses this
|
||||
}
|
||||
```
|
||||
|
||||
**Supported formats:**
|
||||
|
||||
- SRT/VTT: Regex-based parsing of timing lines + text content between timing blocks.
|
||||
- ASS: Parse `[Events]` section, extract `Dialogue:` lines, read the field order from the `Format:` row, and take everything after the Text field index as the text (Text can itself contain commas).
|
||||
- ASS: Parse the `[Events]` section, read the field order from the `Format:` row, and extract timed `Dialogue:` lines. Timed `Comment:` lines are normally ignored, but can supply canonical authored text when they match a nearby generated animation from the same style and actor. Text can itself contain commas.
|
||||
|
||||
**ASS decoding.** The parser is where ASS text is decoded, once, via `assToPlainText()` in `src/core/services/ass-text.ts`. That decoder mirrors mpv's `ass_to_plaintext` so a cue read from a file reads identically to the same line arriving live on `sub-text`: `{...}` override blocks are markup, `\pN … \p0` vector drawing runs are dropped rather than shown as text, `\N`/`\n`/`\h` are the only escapes (`\{`, `\}` and `\\` are not), and an unclosed `{` is rendered verbatim. Every layer downstream — renderer, timing tracker, tokenizer, tokenization cache keys — receives plain text and uses `normalizePlainSubtitleText()` for whitespace only, so nothing decodes the same string twice and one authored line always maps to one cache key.
|
||||
|
||||
**Duplicate collapsing.** Typeset scripts emit one `Dialogue:` event per animation frame, plus layered copies of the same line. The parser collapses identical text over an identical span unconditionally, and collapses contiguous same-text runs of at least three events when the run looks like an animation. For ASS that means shared style and actor plus authoring evidence: a temporal tag (`\t`, `\move`, `\k`/`\kf`/`\ko`/`\K`, or anything wrapped in `\t(...)`), an animated `Effect` column (`Karaoke`, `Banner`, `Scroll`), or override values that change across the run. Static tags shared by every event (`\pos`, an identical `\clip`) are not evidence. SRT/VTT carry no such metadata, so there collapsing needs at least five contiguous events all under 0.1s — the frame timing left behind by ASS-to-SRT conversion. The parser keeps this authoring metadata (style, actor, layer, `Effect`, parsed override commands, source order) private; `parseSubtitleCues()` returns only `SubtitleCue`.
|
||||
|
||||
ASS scripts can also redraw one complete lyric for two or more long color/highlight phases. Those flush-timed phases collapse separately from short animation frames when they share text, style, actor, and layer and carry direct animation evidence, such as temporal tags or changing non-spatial overrides. Spatial command changes do not prove a phase, so separately positioned signs remain distinct.
|
||||
|
||||
**Canonical animation recovery.** Some ASS producers keep the readable lyric or sign as a timed `Comment:` and generate hundreds of `Dialogue:` frames containing repeated glyphs or changing clip regions. Others retain the complete line as brief `Dialogue:` events around the generated fragments. A complete event is promoted only when nearby dialogue from the same style and actor forms a proven animation cluster and reconstructs its entire text in source order. The generated frames are then replaced by one cue marked `source: 'canonical-ass'`. This source marker lets the live primary-subtitle path prefer the clean authored text and timing for display, sidebar history, immersion recording, and mining, while unmatched editor notes and alternative translations remain ignored.
|
||||
|
||||
#### Prefetch Service Lifecycle
|
||||
|
||||
1. **Activation trigger:** When a subtitle track is activated (or changes), check if it's external via MPV's `track-list` property. If `external === true`, read the file via `external-filename` using the existing `loadSubtitleSourceText` infrastructure.
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
import { Database } from '../sqlite.js';
|
||||
import type { DatabaseSync } from '../sqlite.js';
|
||||
@@ -21,17 +18,6 @@ interface SeedLine {
|
||||
createdMs?: number;
|
||||
}
|
||||
|
||||
function makeDbPath(): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-duplicate-line-test-'));
|
||||
return path.join(dir, 'immersion.sqlite');
|
||||
}
|
||||
|
||||
function cleanupDbPath(dbPath: string): void {
|
||||
const dir = path.dirname(dbPath);
|
||||
if (!fs.existsSync(dir)) return;
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
/** One episode, two sessions of it, and one word occurrence per seeded line. */
|
||||
function seed(db: DatabaseSync, lines: SeedLine[]): void {
|
||||
db.exec(`
|
||||
@@ -82,12 +68,16 @@ function seed(db: DatabaseSync, lines: SeedLine[]): void {
|
||||
`);
|
||||
}
|
||||
|
||||
function createDb(lines: SeedLine[]): { db: DatabaseSync; dbPath: string } {
|
||||
const dbPath = makeDbPath();
|
||||
const db = new Database(dbPath);
|
||||
/**
|
||||
* These tests exercise the cleanup SQL, not durability. A fresh on-disk database per
|
||||
* test pays a schema-creation fsync that is cheap on a local NVMe but slow enough on CI
|
||||
* runners to blow the 5s per-test timeout, so the database stays in memory.
|
||||
*/
|
||||
function createDb(lines: SeedLine[]): { db: DatabaseSync } {
|
||||
const db = new Database(':memory:');
|
||||
ensureSchema(db);
|
||||
seed(db, lines);
|
||||
return { db, dbPath };
|
||||
return { db };
|
||||
}
|
||||
|
||||
/** A typeset line mpv reported once per animation frame. */
|
||||
@@ -119,7 +109,7 @@ function wordFrequency(db: DatabaseSync): number {
|
||||
}
|
||||
|
||||
test('a karaoke burst collapses to one line and gives back its word counts', () => {
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 40, 40),
|
||||
{ session: 1, text: 'おはよう', startMs: 20_000, endMs: 22_000 },
|
||||
]);
|
||||
@@ -148,7 +138,6 @@ test('a karaoke burst collapses to one line and gives back its word counts', ()
|
||||
assert.equal(summary.samples[0]!.videoTitle, 'Ep 1');
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -160,7 +149,7 @@ test('ordinary repeated dialogue survives', () => {
|
||||
startMs: 5_000 + index * 800,
|
||||
endMs: 5_000 + (index + 1) * 800,
|
||||
}));
|
||||
const { db, dbPath } = createDb(lines);
|
||||
const { db } = createDb(lines);
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -171,14 +160,13 @@ test('ordinary repeated dialogue survives', () => {
|
||||
assert.equal(wordFrequency(db), 6);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a long run of quarter-second frames is still a burst', () => {
|
||||
// Between the timing-only bound (0.1s) and the animation-frame bound (0.3s): heavier
|
||||
// typesetting lands here, and the run length is what makes it conclusive.
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -189,12 +177,11 @@ test('a long run of quarter-second frames is still a burst', () => {
|
||||
assert.equal(wordFrequency(db), 1);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a qualifying short-frame burst may end with one long hold frame', () => {
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 8, 40),
|
||||
{ session: 1, text: '飛び上がる', startMs: 10_320, endMs: 12_320 },
|
||||
]);
|
||||
@@ -208,12 +195,11 @@ test('a qualifying short-frame burst may end with one long hold frame', () => {
|
||||
assert.equal(wordFrequency(db), 1);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a long event before the final frame prevents burst cleanup', () => {
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 5, 40),
|
||||
{ session: 1, text: '飛び上がる', startMs: 10_200, endMs: 12_200 },
|
||||
{ session: 1, text: '飛び上がる', startMs: 12_200, endMs: 12_240 },
|
||||
@@ -226,12 +212,11 @@ test('a long event before the final frame prevents burst cleanup', () => {
|
||||
assert.equal(countLines(db), 7);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a run of frames longer than the animation bound survives', () => {
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 400));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 400));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -240,7 +225,6 @@ test('a run of frames longer than the animation bound survives', () => {
|
||||
assert.equal(countLines(db), 6);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -248,7 +232,7 @@ test('the four-frame residue the live gate stores is cleaned up', () => {
|
||||
// The streaming gate records the first four frames of a burst before the run is long
|
||||
// enough to recognise. Four contiguous identical events under the strict timing-only
|
||||
// bound are that residue, and no real dialogue.
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 40));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 40));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -259,14 +243,13 @@ test('the four-frame residue the live gate stores is cleaned up', () => {
|
||||
assert.equal(wordFrequency(db), 1);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a four-frame run above the strict frame bound survives', () => {
|
||||
// Long enough per event to be plausible dialogue; only a five-event run may use the
|
||||
// looser animation-frame bound.
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 250));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 250));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -275,14 +258,13 @@ test('a four-frame run above the strict frame bound survives', () => {
|
||||
assert.equal(countLines(db), 4);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('an explicit minRunLength raises the bar', () => {
|
||||
// Five quarter-second frames qualify under the defaults; a cautious run asking for six
|
||||
// leaves them alone. Above the strict bound, so the residue rule stays out of it.
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 5, 250));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 5, 250));
|
||||
|
||||
try {
|
||||
const preview = cleanupDuplicateSubtitleLines(db, { dryRun: true });
|
||||
@@ -293,12 +275,11 @@ test('an explicit minRunLength raises the bar', () => {
|
||||
assert.equal(countLines(db), 5);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('an explicit maxFrameSeconds tightens the frame bound', () => {
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db, { maxFrameSeconds: 0.2 });
|
||||
@@ -307,13 +288,12 @@ test('an explicit maxFrameSeconds tightens the frame bound', () => {
|
||||
assert.equal(countLines(db), 6);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a non-finite maxFrameSeconds falls back to the default bound', () => {
|
||||
// Six normal-beat lines: Infinity must not turn every event into a "short frame".
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 800));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 800));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db, { maxFrameSeconds: Infinity });
|
||||
@@ -322,12 +302,11 @@ test('a non-finite maxFrameSeconds falls back to the default bound', () => {
|
||||
assert.equal(countLines(db), 6);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('sampleLimit zero removes bursts but reports no samples', () => {
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db, { sampleLimit: 0 });
|
||||
@@ -337,12 +316,11 @@ test('sampleLimit zero removes bursts but reports no samples', () => {
|
||||
assert.equal(countLines(db), 1);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a short run below every threshold survives', () => {
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 3, 40));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 3, 40));
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -351,7 +329,6 @@ test('a short run below every threshold survives', () => {
|
||||
assert.equal(countLines(db), 3);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -361,7 +338,7 @@ test('interleaved dual-line karaoke collapses each line to one row', () => {
|
||||
const kanji = karaokeFrames(1, '飛び上がる', 10_000, 20, 60);
|
||||
const romaji = karaokeFrames(1, 'tobiagaru', 10_001, 20, 60);
|
||||
const interleaved = [...kanji, ...romaji].sort((a, b) => a.startMs - b.startMs);
|
||||
const { db, dbPath } = createDb(interleaved);
|
||||
const { db } = createDb(interleaved);
|
||||
|
||||
try {
|
||||
const summary = cleanupDuplicateSubtitleLines(db);
|
||||
@@ -372,12 +349,11 @@ test('interleaved dual-line karaoke collapses each line to one row', () => {
|
||||
assert.equal(wordFrequency(db), 2);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('the same line in a rewatch session is never merged into the first watch', () => {
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40),
|
||||
...karaokeFrames(2, '飛び上がる', 10_000, 6, 40),
|
||||
]);
|
||||
@@ -392,12 +368,11 @@ test('the same line in a rewatch session is never merged into the first watch',
|
||||
assert.equal(wordFrequency(db), 2);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a gap between runs splits them', () => {
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40),
|
||||
...karaokeFrames(1, '飛び上がる', 60_000, 6, 40),
|
||||
]);
|
||||
@@ -409,12 +384,11 @@ test('a gap between runs splits them', () => {
|
||||
assert.equal(countLines(db), 2);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('a dry run reports what an apply would do and writes nothing', () => {
|
||||
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
|
||||
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
|
||||
|
||||
try {
|
||||
const preview = cleanupDuplicateSubtitleLines(db, { dryRun: true });
|
||||
@@ -430,14 +404,13 @@ test('a dry run reports what an apply would do and writes nothing', () => {
|
||||
assert.equal(countLines(db), 1);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('the lookback window leaves older bursts alone', () => {
|
||||
const recentMs = BASE_MS;
|
||||
const oldMs = BASE_MS - 40 * DAY_MS;
|
||||
const { db, dbPath } = createDb([
|
||||
const { db } = createDb([
|
||||
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40).map((line) => ({
|
||||
...line,
|
||||
createdMs: oldMs,
|
||||
@@ -462,6 +435,5 @@ test('the lookback window leaves older bursts alone', () => {
|
||||
} finally {
|
||||
globalThis.__subminerTestNowMs = undefined;
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -125,6 +125,49 @@ test('mineSentenceCard creates sentence card from mpv subtitle state', async ()
|
||||
]);
|
||||
});
|
||||
|
||||
test('mineSentenceCard prefers a canonical primary subtitle snapshot', async () => {
|
||||
const created: Array<{
|
||||
sentence: string;
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
secondarySub?: string;
|
||||
}> = [];
|
||||
|
||||
await mineSentenceCard({
|
||||
ankiIntegration: {
|
||||
updateLastAddedFromClipboard: async () => {},
|
||||
triggerFieldGroupingForLastAddedCard: async () => {},
|
||||
markLastCardAsAudioCard: async () => {},
|
||||
createSentenceCard: async (sentence, startTime, endTime, secondarySub) => {
|
||||
created.push({ sentence, startTime, endTime, secondarySub });
|
||||
return true;
|
||||
},
|
||||
},
|
||||
mpvClient: {
|
||||
connected: true,
|
||||
currentSubText: '今今今手手手',
|
||||
currentSubStart: 11.4,
|
||||
currentSubEnd: 11.8,
|
||||
currentSecondarySubText: 'English subtitle',
|
||||
},
|
||||
primarySubtitle: {
|
||||
text: '今 手にある物差しでは',
|
||||
startTime: 11.13,
|
||||
endTime: 13.83,
|
||||
},
|
||||
showMpvOsd: () => {},
|
||||
});
|
||||
|
||||
assert.deepEqual(created, [
|
||||
{
|
||||
sentence: '今 手にある物差しでは',
|
||||
startTime: 11.13,
|
||||
endTime: 13.83,
|
||||
secondarySub: 'English subtitle',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('mineSentenceCard refreshes secondary subtitle text before creating card', async () => {
|
||||
const created: Array<{ sentence: string; secondarySub?: string }> = [];
|
||||
const requestedProperties: string[] = [];
|
||||
|
||||
@@ -131,8 +131,8 @@ function normalizeSecondarySubText(text: unknown, primaryText: string): string |
|
||||
|
||||
async function getCurrentSecondarySubTextForSentenceCard(
|
||||
mpvClient: MpvClientLike,
|
||||
primaryText: string,
|
||||
): Promise<string | undefined> {
|
||||
const primaryText = mpvClient.currentSubText;
|
||||
if (mpvClient.requestProperty) {
|
||||
try {
|
||||
const latestSecondaryText = await mpvClient.requestProperty('secondary-sub-text');
|
||||
@@ -175,6 +175,7 @@ export async function markLastCardAsAudioCard(deps: {
|
||||
export async function mineSentenceCard(deps: {
|
||||
ankiIntegration: AnkiIntegrationLike | null;
|
||||
mpvClient: MpvClientLike | null;
|
||||
primarySubtitle?: Pick<SubtitleMiningContext, 'text' | 'startTime' | 'endTime'>;
|
||||
showMpvOsd: (text: string) => void;
|
||||
}): Promise<boolean> {
|
||||
const anki = requireAnkiIntegration(deps.ankiIntegration, deps.showMpvOsd);
|
||||
@@ -185,16 +186,17 @@ export async function mineSentenceCard(deps: {
|
||||
deps.showMpvOsd('MPV not connected');
|
||||
return false;
|
||||
}
|
||||
if (!mpvClient.currentSubText) {
|
||||
const primaryText = deps.primarySubtitle?.text ?? mpvClient.currentSubText;
|
||||
if (!primaryText) {
|
||||
deps.showMpvOsd('No current subtitle');
|
||||
return false;
|
||||
}
|
||||
|
||||
const secondarySubText = await getCurrentSecondarySubTextForSentenceCard(mpvClient);
|
||||
const secondarySubText = await getCurrentSecondarySubTextForSentenceCard(mpvClient, primaryText);
|
||||
return await anki.createSentenceCard(
|
||||
mpvClient.currentSubText,
|
||||
mpvClient.currentSubStart,
|
||||
mpvClient.currentSubEnd,
|
||||
primaryText,
|
||||
deps.primarySubtitle?.startTime ?? mpvClient.currentSubStart,
|
||||
deps.primarySubtitle?.endTime ?? mpvClient.currentSubEnd,
|
||||
secondarySubText,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -27,17 +27,188 @@ function cueKey(cue: SubtitleCue): string {
|
||||
|
||||
/**
|
||||
* Identical text over an identical span is redundant however it was authored -- most
|
||||
* often a layered ASS event stacking a shadow copy under the visible one.
|
||||
* often a layered ASS event stacking a shadow copy under the visible one. When one of
|
||||
* the duplicates is a recovered canonical cue, that copy survives: dropping it would
|
||||
* strip the `source` marker and animation envelope the live overlay substitutes on.
|
||||
*/
|
||||
function collapseExactDuplicates(cues: AnnotatedSubtitleCue[]): AnnotatedSubtitleCue[] {
|
||||
const seen = new Set<string>();
|
||||
return cues.filter((cue) => {
|
||||
const survivorByKey = new Map<string, AnnotatedSubtitleCue>();
|
||||
const keysInOrder: string[] = [];
|
||||
for (const cue of cues) {
|
||||
const key = cueKey(cue);
|
||||
if (seen.has(key)) {
|
||||
return false;
|
||||
const existing = survivorByKey.get(key);
|
||||
if (!existing) {
|
||||
survivorByKey.set(key, cue);
|
||||
keysInOrder.push(key);
|
||||
} else if (!existing.source && cue.source) {
|
||||
survivorByKey.set(key, cue);
|
||||
}
|
||||
seen.add(key);
|
||||
}
|
||||
return keysInOrder.map((key) => survivorByKey.get(key)!);
|
||||
}
|
||||
|
||||
const SPATIAL_ASS_OVERRIDE_COMMANDS = new Set([
|
||||
'a',
|
||||
'an',
|
||||
'clip',
|
||||
'iclip',
|
||||
'move',
|
||||
'org',
|
||||
'pbo',
|
||||
'pos',
|
||||
'q',
|
||||
]);
|
||||
|
||||
interface RepeatedPhaseRun {
|
||||
cues: AnnotatedSubtitleCue[];
|
||||
indices: number[];
|
||||
}
|
||||
|
||||
// A changing override signature alone is weak: two ordinary repeats restyled with
|
||||
// different colors look identical to a phase pair. Real phase redraws carry a styling
|
||||
// stack over a full lyric line, and they exist to move a color/highlight boundary
|
||||
// *within* the line -- so every event also has an override block after visible text
|
||||
// began. An ordinary restyled repeat carries only a leading block and stays separate.
|
||||
const MIN_PHASE_EVIDENCE_OVERRIDES = 2;
|
||||
const MIN_PHASE_TEXT_LENGTH = 4;
|
||||
|
||||
function hasMidLineOverrideBlock(rawText: string): boolean {
|
||||
let sawVisibleText = false;
|
||||
for (let i = 0; i < rawText.length; i += 1) {
|
||||
if (rawText[i] === '{') {
|
||||
const close = rawText.indexOf('}', i);
|
||||
if (close === -1) {
|
||||
// Unclosed brace renders as literal text; nothing after it is markup.
|
||||
return false;
|
||||
}
|
||||
if (sawVisibleText) {
|
||||
return true;
|
||||
}
|
||||
i = close;
|
||||
} else if (!/\s/.test(rawText[i]!)) {
|
||||
sawVisibleText = true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function assStyleKey(cue: AnnotatedSubtitleCue): string {
|
||||
return `${cue.style}\0${cue.name}\0${cue.layer}`;
|
||||
}
|
||||
|
||||
function spatialOverrideSignature(cue: AnnotatedSubtitleCue): string {
|
||||
return cue.overrides
|
||||
.filter((command) => SPATIAL_ASS_OVERRIDE_COMMANDS.has(command.name.toLowerCase()))
|
||||
.map((command) => `${command.name.toLowerCase()}(${command.args})`)
|
||||
.join('|');
|
||||
}
|
||||
|
||||
function hasStableSpatialOverrides(run: readonly AnnotatedSubtitleCue[]): boolean {
|
||||
const firstSignature = spatialOverrideSignature(run[0]!);
|
||||
return run.every((cue) => spatialOverrideSignature(cue) === firstSignature);
|
||||
}
|
||||
|
||||
function hasDirectPhaseEvidence(run: readonly AnnotatedSubtitleCue[]): boolean {
|
||||
// Phases redraw one authored line in place. Whatever the animation evidence, a run
|
||||
// whose spatial placement changes is separate authored occurrences -- two flush
|
||||
// same-text `\move` signs at different coordinates must never merge.
|
||||
if (!hasStableSpatialOverrides(run)) {
|
||||
return false;
|
||||
}
|
||||
if (run.every((cue) => hasAssTemporalOverride(cue.overrides))) {
|
||||
return true;
|
||||
}
|
||||
if (run.every((cue) => isAnimatedAssEffectKind(cue.effectKind))) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const [first] = run;
|
||||
return (
|
||||
first!.text.replace(/\s+/gu, '').length >= MIN_PHASE_TEXT_LENGTH &&
|
||||
run.every(
|
||||
(cue) =>
|
||||
cue.overrides.length >= MIN_PHASE_EVIDENCE_OVERRIDES &&
|
||||
hasMidLineOverrideBlock(cue.rawText),
|
||||
) &&
|
||||
run.some((cue) => cue.overrideSignature !== first!.overrideSignature)
|
||||
);
|
||||
}
|
||||
|
||||
function collectRepeatedPhaseRuns(cues: AnnotatedSubtitleCue[]): RepeatedPhaseRun[] {
|
||||
const runs: RepeatedPhaseRun[] = [];
|
||||
let start = 0;
|
||||
|
||||
while (start < cues.length) {
|
||||
const first = cues[start]!;
|
||||
const styleKey = assStyleKey(first);
|
||||
let end = start;
|
||||
|
||||
while (end + 1 < cues.length) {
|
||||
const current = cues[end]!;
|
||||
const next = cues[end + 1]!;
|
||||
const isFlush =
|
||||
Math.abs(next.startTime - current.endTime) <= DUPLICATE_CUE_GAP_TOLERANCE_SECONDS;
|
||||
if (
|
||||
first.source === 'canonical-ass' ||
|
||||
next.source === 'canonical-ass' ||
|
||||
next.text !== first.text ||
|
||||
assStyleKey(next) !== styleKey ||
|
||||
!isFlush
|
||||
) {
|
||||
break;
|
||||
}
|
||||
end += 1;
|
||||
}
|
||||
|
||||
if (end > start) {
|
||||
const indices = Array.from({ length: end - start + 1 }, (_, offset) => start + offset);
|
||||
runs.push({
|
||||
cues: indices.map((index) => cues[index]!),
|
||||
indices,
|
||||
});
|
||||
}
|
||||
start = end + 1;
|
||||
}
|
||||
|
||||
return runs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Some karaoke scripts redraw one complete lyric for each color/highlight phase. These
|
||||
* events last far longer than animation frames, but are still one sidebar/history line.
|
||||
* The events must prove themselves through direct animation metadata or changing
|
||||
* non-spatial overrides. Plain repeated dialogue and separately positioned signs stay
|
||||
* intact.
|
||||
*/
|
||||
function collapseAnimatedStylePhases(cues: AnnotatedSubtitleCue[]): AnnotatedSubtitleCue[] {
|
||||
const runs = collectRepeatedPhaseRuns(cues);
|
||||
if (runs.length === 0) {
|
||||
return cues;
|
||||
}
|
||||
|
||||
const dropped = new Set<number>();
|
||||
const extendedEnd = new Map<number, number>();
|
||||
for (const run of runs) {
|
||||
if (!hasDirectPhaseEvidence(run.cues)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const [firstIndex, ...remainingIndices] = run.indices;
|
||||
for (const index of remainingIndices) {
|
||||
dropped.add(index);
|
||||
}
|
||||
extendedEnd.set(firstIndex!, Math.max(...run.cues.map((cue) => cue.endTime)));
|
||||
}
|
||||
|
||||
if (dropped.size === 0) {
|
||||
return cues;
|
||||
}
|
||||
return cues.flatMap((cue, index) => {
|
||||
if (dropped.has(index)) {
|
||||
return [];
|
||||
}
|
||||
const endTime = extendedEnd.get(index);
|
||||
return endTime !== undefined ? [{ ...cue, endTime }] : [cue];
|
||||
});
|
||||
}
|
||||
|
||||
@@ -176,5 +347,8 @@ export function mergeDuplicateCues(
|
||||
cues: AnnotatedSubtitleCue[],
|
||||
format: SubtitleSourceFormat,
|
||||
): AnnotatedSubtitleCue[] {
|
||||
return collapseAnimationBursts(collapseExactDuplicates(cues), format);
|
||||
const exactDeduplicated = collapseExactDuplicates(cues);
|
||||
const phaseDeduplicated =
|
||||
format === 'ass' ? collapseAnimatedStylePhases(exactDeduplicated) : exactDeduplicated;
|
||||
return collapseAnimationBursts(phaseDeduplicated, format);
|
||||
}
|
||||
|
||||
@@ -327,6 +327,122 @@ test('parseSubtitleCues collapses per-frame karaoke duplicates into one cue', ()
|
||||
assert.equal(cues[0]!.text, '過ぎ去ってしまう瞬間を');
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses long full-line color phases', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:03:49.75,0:03:51.21,OPJP,,0,0,0,,{\\blur0.6\\c&H312D38&\\4c&HFFFFFF&}ちゃんと目を{\\4c&HD590FF&}合わせてよ',
|
||||
'Dialogue: 1,0:03:51.21,0:03:52.25,OPJP,,0,0,0,,{\\blur0.6\\4c&H312D38&\\c&HFFFFFF&}ちゃんと目を{\\4c&HD590FF&}合わせてよ',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{
|
||||
startTime: 229.75,
|
||||
endTime: 232.25,
|
||||
text: 'ちゃんと目を合わせてよ',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps ordinary repeated dialogue separate', () => {
|
||||
// A single restyle tag on a repeated line is how ordinary dialogue gets decorated;
|
||||
// it is not phase evidence, whatever the line length.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:00:01.00,0:00:02.00,OPJP,,0,0,0,,{\\c&H111111&}歌詞',
|
||||
'Dialogue: 1,0:00:02.00,0:00:03.00,OPJP,,0,0,0,,{\\c&H222222&}歌詞',
|
||||
'Dialogue: 1,0:00:04.00,0:00:05.00,OPJP,,0,0,0,,{\\c&H333333&}別の歌詞',
|
||||
'Dialogue: 1,0:00:05.00,0:00:06.00,OPJP,,0,0,0,,{\\c&H444444&}別の歌詞',
|
||||
'Dialogue: 8,0:00:07.00,0:00:08.00,Text - JP,,0,0,0,,えっ?',
|
||||
'Dialogue: 8,0:00:08.00,0:00:09.00,Text - JP,,0,0,0,,えっ?',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{ startTime: 1, endTime: 2, text: '歌詞' },
|
||||
{ startTime: 2, endTime: 3, text: '歌詞' },
|
||||
{ startTime: 4, endTime: 5, text: '別の歌詞' },
|
||||
{ startTime: 5, endTime: 6, text: '別の歌詞' },
|
||||
{ startTime: 7, endTime: 8, text: 'えっ?' },
|
||||
{ startTime: 8, endTime: 9, text: 'えっ?' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps separately positioned temporal signs separate', () => {
|
||||
// Two flush signs with the same text but different \move paths are separate authored
|
||||
// occurrences, not phases of one redraw: temporal evidence alone must not merge them.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:02.00,Sign,,0,0,0,,{\\move(100,100,200,100)}立入禁止',
|
||||
'Dialogue: 0,0:00:02.00,0:00:03.00,Sign,,0,0,0,,{\\move(500,400,600,400)}立入禁止',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{ startTime: 1, endTime: 2, text: '立入禁止' },
|
||||
{ startTime: 2, endTime: 3, text: '立入禁止' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps richly styled ordinary repeats separate', () => {
|
||||
// Blur plus a changing color is still an ordinary restyle. Phase redraws are
|
||||
// recognized by the color/highlight boundary moving *within* the line, which these
|
||||
// leading-block-only events do not have.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:02.00,Dial,,0,0,0,,{\\blur0.4\\c&H111111&}待ってよ',
|
||||
'Dialogue: 0,0:00:02.00,0:00:03.00,Dial,,0,0,0,,{\\blur0.4\\c&H222222&}待ってよ',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{ startTime: 1, endTime: 2, text: '待ってよ' },
|
||||
{ startTime: 2, endTime: 3, text: '待ってよ' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps canonical metadata when an identical plain cue exists', () => {
|
||||
// A plain dialogue line can share exact timing and text with a recovered canonical
|
||||
// cue from another style. The canonical copy must win the exact-duplicate collapse,
|
||||
// or the live overlay loses the marker it substitutes on.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:05.00,0:00:08.00,Plain,,0,0,0,,ライン',
|
||||
'Comment: 0,0:00:05.00,0:00:08.00,OP,,0,0,0,,ライン',
|
||||
'Dialogue: 0,0:00:05.00,0:00:05.04,OP,,0,0,0,,{\\pos(1,1)\\clip(m 1 1)}ライン',
|
||||
'Dialogue: 0,0:00:05.04,0:00:05.08,OP,,0,0,0,,{\\pos(1,1)\\clip(m 2 2)}ライン',
|
||||
'Dialogue: 0,0:00:05.08,0:00:08.00,OP,,0,0,0,,{\\pos(1,1)\\clip(m 3 3)}ライン',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{
|
||||
startTime: 5,
|
||||
endTime: 8,
|
||||
text: 'ライン',
|
||||
source: 'canonical-ass',
|
||||
animationStartTime: 5,
|
||||
animationEndTime: 8,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps short styled repeats separate even with richer styling', () => {
|
||||
// Two ordinary えっ lines restyled with different colors are two utterances, not two
|
||||
// phases of one lyric: short text never satisfies the changing-override evidence path.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:02.00,Dial,,0,0,0,,{\\blur0.4\\c&H111111&}えっ',
|
||||
'Dialogue: 0,0:00:02.00,0:00:03.00,Dial,,0,0,0,,{\\blur0.4\\c&H222222&}えっ',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{ startTime: 1, endTime: 2, text: 'えっ' },
|
||||
{ startTime: 2, endTime: 3, text: 'えっ' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps back-to-back plain dialogue repeats separate', () => {
|
||||
// Several characters greeting in turn: distinct utterances that happen to abut.
|
||||
const content = [
|
||||
@@ -357,6 +473,194 @@ test('parseSubtitleCues collapses exact duplicate cues even without effect tags'
|
||||
assert.equal(cues.length, 1);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues replaces generated glyph animation with its timed canonical comment', () => {
|
||||
// Aegisub automation commonly keeps the authored lyric as a Comment and emits
|
||||
// multiple moving Dialogue layers for every glyph. This mirrors the MyGO ED script:
|
||||
// three entrance copies followed by three exit copies for each character.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Comment: 0,0:00:01.20,0:00:03.80,ED_JP,,0,0,0,,{\\fad(480,480)}今 手にある',
|
||||
'Dialogue: 0,0:00:00.80,0:00:01.50,ED_JP,,0,0,0,,{\\move(10,20,100,200)\\t(0,600,\\fscx100)}今',
|
||||
'Dialogue: 0,0:00:00.80,0:00:01.50,ED_JP,,0,0,0,,{\\move(30,40,100,200)\\t(0,600,\\fscx100)}今',
|
||||
'Dialogue: 0,0:00:00.80,0:00:01.50,ED_JP,,0,0,0,,{\\move(50,60,100,200)\\t(0,600,\\fscx100)}今',
|
||||
'Dialogue: 1,0:00:01.40,0:00:04.20,ED_JP,,0,0,0,,{\\move(100,200,20,30)\\t(2000,2600,\\blur20)}今',
|
||||
'Dialogue: 1,0:00:01.40,0:00:04.20,ED_JP,,0,0,0,,{\\move(100,200,40,50)\\t(2000,2600,\\blur20)}今',
|
||||
'Dialogue: 1,0:00:01.40,0:00:04.20,ED_JP,,0,0,0,,{\\move(100,200,60,70)\\t(2000,2600,\\blur20)}今',
|
||||
'Dialogue: 0,0:00:00.86,0:00:01.56,ED_JP,,0,0,0,,{\\move(10,20,140,200)\\t(0,600,\\fscx100)}手',
|
||||
'Dialogue: 0,0:00:00.86,0:00:01.56,ED_JP,,0,0,0,,{\\move(30,40,140,200)\\t(0,600,\\fscx100)}手',
|
||||
'Dialogue: 0,0:00:00.86,0:00:01.56,ED_JP,,0,0,0,,{\\move(50,60,140,200)\\t(0,600,\\fscx100)}手',
|
||||
'Dialogue: 1,0:00:01.46,0:00:04.26,ED_JP,,0,0,0,,{\\move(140,200,20,30)\\t(2000,2600,\\blur20)}手',
|
||||
'Dialogue: 1,0:00:01.46,0:00:04.26,ED_JP,,0,0,0,,{\\move(140,200,40,50)\\t(2000,2600,\\blur20)}手',
|
||||
'Dialogue: 1,0:00:01.46,0:00:04.26,ED_JP,,0,0,0,,{\\move(140,200,60,70)\\t(2000,2600,\\blur20)}手',
|
||||
'Dialogue: 0,0:00:00.92,0:00:01.62,ED_JP,,0,0,0,,{\\move(10,20,180,200)\\t(0,600,\\fscx100)}にある',
|
||||
'Dialogue: 0,0:00:00.92,0:00:01.62,ED_JP,,0,0,0,,{\\move(30,40,180,200)\\t(0,600,\\fscx100)}にある',
|
||||
'Dialogue: 0,0:00:00.92,0:00:01.62,ED_JP,,0,0,0,,{\\move(50,60,180,200)\\t(0,600,\\fscx100)}にある',
|
||||
'Dialogue: 1,0:00:01.52,0:00:04.32,ED_JP,,0,0,0,,{\\move(180,200,20,30)\\t(2000,2600,\\blur20)}にある',
|
||||
'Dialogue: 1,0:00:01.52,0:00:04.32,ED_JP,,0,0,0,,{\\move(180,200,40,50)\\t(2000,2600,\\blur20)}にある',
|
||||
'Dialogue: 1,0:00:01.52,0:00:04.32,ED_JP,,0,0,0,,{\\move(180,200,60,70)\\t(2000,2600,\\blur20)}にある',
|
||||
'Dialogue: 0,0:00:06.00,0:00:08.00,Dial_JP,,0,0,0,,普通の会話',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.deepEqual(cues, [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
// Entrance frames start before and exit frames end after the authored timing.
|
||||
animationStartTime: 0.8,
|
||||
animationEndTime: 4.32,
|
||||
},
|
||||
{ startTime: 6, endTime: 8, text: '普通の会話' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues recovers a full Dialogue line surrounding generated fragments', () => {
|
||||
// Some scripts do not retain the authored line as a Comment. Instead, brief entrance
|
||||
// and exit events contain the complete line around a long run of generated syllables.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:00:01.00,0:00:01.15,ED Romaji,,0,0,0,fx,{\\move(100,40,60,40)}toki yo ugokidase',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(0,300,\\c&HFFFFFF&)}to',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(300,500,\\c&HFFFFFF&)}ki',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(500,700,\\c&HFFFFFF&)}yo',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(700,900,\\c&HFFFFFF&)}u',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(900,1100,\\c&HFFFFFF&)}go',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(1100,1300,\\c&HFFFFFF&)}ki',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(1300,1500,\\c&HFFFFFF&)}da',
|
||||
'Dialogue: 1,0:00:01.15,0:00:03.00,ED Romaji,,0,0,0,fx,{\\t(1500,1800,\\c&HFFFFFF&)}se',
|
||||
'Dialogue: 1,0:00:03.00,0:00:03.15,ED Romaji,,0,0,0,fx,{\\move(60,40,20,40)}toki yo ugokidase',
|
||||
'Dialogue: 0,0:00:06.00,0:00:08.00,Default,,0,0,0,,Ordinary dialogue',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{
|
||||
startTime: 1,
|
||||
endTime: 3.15,
|
||||
text: 'toki yo ugokidase',
|
||||
source: 'canonical-ass',
|
||||
animationStartTime: 1,
|
||||
animationEndTime: 3.15,
|
||||
},
|
||||
{ startTime: 6, endTime: 8, text: 'Ordinary dialogue' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues does not promote a short animated fragment as a complete line', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(100,100)\\t(0,100,\\fscx120)}my',
|
||||
'Dialogue: 2,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(100,100)\\t(0,100,\\fscx120)}my',
|
||||
'Dialogue: 1,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(100,100)\\t(0,100,\\fscx120)}m',
|
||||
'Dialogue: 2,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(100,100)\\t(0,100,\\fscx120)}m',
|
||||
'Dialogue: 1,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(120,100)\\t(20,120,\\fscx120)}y',
|
||||
'Dialogue: 2,0:00:01.00,0:00:03.00,OP English,,0,0,0,,{\\pos(120,100)\\t(20,120,\\fscx120)}y',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(
|
||||
cues.some((cue) => cue.source === 'canonical-ass'),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues ignores timed comments without a matching animated dialogue cluster', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Comment: 0,0:00:01.00,0:00:03.00,Dial_JP,,0,0,0,,編集メモ',
|
||||
'Comment: 0,0:00:04.00,0:00:06.00,Dial_JP,,0,0,0,,別案の字幕',
|
||||
'Dialogue: 0,0:00:01.00,0:00:03.00,Dial_JP,,0,0,0,,通常の字幕',
|
||||
'Dialogue: 0,0:00:04.00,0:00:06.00,Dial_JP,,0,0,0,,別案の字幕',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.deepEqual(cues, [
|
||||
{ startTime: 1, endTime: 3, text: '通常の字幕' },
|
||||
{ startTime: 4, endTime: 6, text: '別案の字幕' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseAssCues returns recovered canonical cues in chronological order', () => {
|
||||
// Recovery appends recovered cues after surviving dialogue; the bare parseAssCues
|
||||
// export must still come back time-ordered.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:06.00,0:00:08.00,Dial,,0,0,0,,あとのセリフ',
|
||||
'Comment: 0,0:00:01.20,0:00:03.80,OP,,0,0,0,,雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.24,OP,,0,0,0,,{\\pos(1,1)\\clip(m 1 1)}雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.24,0:00:01.28,OP,,0,0,0,,{\\pos(1,1)\\clip(m 2 2)}雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.28,0:00:03.80,OP,,0,0,0,,{\\pos(1,1)\\clip(m 3 3)}雨が上がっても',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(
|
||||
parseAssCues(content).map((cue) => cue.startTime),
|
||||
[1.2, 6],
|
||||
);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues withdraws a recovery whose owner is claimed by a later candidate', () => {
|
||||
// The exit boundary event appears first in the file and recovers a canonical cue from
|
||||
// its own small cluster. The entrance candidate then proves that exit event was a
|
||||
// generated frame of the full animation; the earlier recovery is a duplicate of the
|
||||
// same authored line and must not survive alongside it.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:00:14.00,0:00:14.20,ED,,0,0,0,,{\\move(100,200,20,30)}ABCDEFGH',
|
||||
'Dialogue: 1,0:00:13.50,0:00:14.50,ED,,0,0,0,,{\\t(0,300,\\c&HFFFFFF&)}ABC',
|
||||
'Dialogue: 1,0:00:13.50,0:00:14.50,ED,,0,0,0,,{\\t(300,600,\\c&HFFFFFF&)}DEF',
|
||||
'Dialogue: 1,0:00:13.50,0:00:14.50,ED,,0,0,0,,{\\t(600,900,\\c&HFFFFFF&)}GH',
|
||||
'Dialogue: 0,0:00:10.00,0:00:10.20,ED,,0,0,0,,{\\move(10,20,100,200)}ABCDEFGH',
|
||||
'Dialogue: 0,0:00:10.00,0:00:12.00,ED,,0,0,0,,{\\t(0,300,\\fscx100)}ABC',
|
||||
'Dialogue: 0,0:00:10.00,0:00:12.00,ED,,0,0,0,,{\\t(300,600,\\fscx100)}DEF',
|
||||
'Dialogue: 0,0:00:10.00,0:00:13.40,ED,,0,0,0,,{\\t(600,900,\\fscx100)}GH',
|
||||
].join('\n');
|
||||
|
||||
assert.deepEqual(parseSubtitleCues(content, 'test.ass'), [
|
||||
{
|
||||
startTime: 10,
|
||||
endTime: 14.2,
|
||||
text: 'ABCDEFGH',
|
||||
source: 'canonical-ass',
|
||||
animationStartTime: 10,
|
||||
animationEndTime: 14.2,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues recovers canonical comments from generated clip frames', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Comment: 0,0:00:01.00,0:00:03.00,OP_JP,,0,0,0,,雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.04,OP_JP,,0,0,0,,{\\pos(960,1068)\\clip(m 1 1)}雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.04,0:00:01.08,OP_JP,,0,0,0,,{\\pos(960,1068)\\clip(m 2 2)}雨が上がっても',
|
||||
'Dialogue: 0,0:00:01.08,0:00:03.00,OP_JP,,0,0,0,,{\\pos(960,1068)\\clip(m 3 3)}雨が上がっても',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.deepEqual(cues, [
|
||||
{
|
||||
startTime: 1,
|
||||
endTime: 3,
|
||||
text: '雨が上がっても',
|
||||
source: 'canonical-ass',
|
||||
animationStartTime: 1,
|
||||
animationEndTime: 3,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses tag-less animation frames in converted SRT', () => {
|
||||
// ASS -> SRT conversion drops override tags, so only the ~0.04s frame timing remains.
|
||||
const lines = ['1', '00:00:07,870 --> 00:00:07,910', 'Kaguya Wants to be Confessed to', ''];
|
||||
|
||||
@@ -6,12 +6,21 @@ import {
|
||||
type AssEffectKind,
|
||||
type AssOverrideCommand,
|
||||
} from './ass-text';
|
||||
import { mergeDuplicateCues } from './subtitle-cue-dedup';
|
||||
import { hasAssAnimationEvidence, mergeDuplicateCues } from './subtitle-cue-dedup';
|
||||
|
||||
export interface SubtitleCue {
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
text: string;
|
||||
/** A complete authored line recovered from matching generated ASS animation events. */
|
||||
source?: 'canonical-ass';
|
||||
/**
|
||||
* Full span of the generated animation events a canonical cue replaced. Entrance and
|
||||
* exit frames routinely run past the authored `startTime`/`endTime`, so live-text
|
||||
* matching must use this envelope while display and history keep the authored timing.
|
||||
*/
|
||||
animationStartTime?: number;
|
||||
animationEndTime?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -19,7 +28,8 @@ export interface SubtitleCue {
|
||||
* Deduplication needs the authoring context -- which style the line belongs to, which
|
||||
* override commands it carries, whether the `Effect` column was set -- to tell a karaoke
|
||||
* burst apart from two characters saying the same word in turn. None of it is meaningful
|
||||
* outside the parser, so the public API stays `{startTime, endTime, text}`.
|
||||
* outside the parser, so the public API exposes only timing, text, and the optional
|
||||
* canonical-source marker used by live subtitle consumers.
|
||||
*/
|
||||
export interface AnnotatedSubtitleCue extends SubtitleCue {
|
||||
/** Text exactly as authored, override blocks and all. */
|
||||
@@ -70,7 +80,11 @@ function sanitizeSubtitleCueText(text: string): string {
|
||||
}
|
||||
|
||||
function toPublicCues(cues: AnnotatedSubtitleCue[]): SubtitleCue[] {
|
||||
return cues.map(({ startTime, endTime, text }) => ({ startTime, endTime, text }));
|
||||
return cues.map(({ startTime, endTime, text, source, animationStartTime, animationEndTime }) =>
|
||||
source
|
||||
? { startTime, endTime, text, source, animationStartTime, animationEndTime }
|
||||
: { startTime, endTime, text },
|
||||
);
|
||||
}
|
||||
|
||||
function parseAnnotatedSrtCues(content: string): AnnotatedSubtitleCue[] {
|
||||
@@ -138,7 +152,13 @@ export function parseSrtCues(content: string): SubtitleCue[] {
|
||||
const ASS_TIMING_PATTERN = /^(\d+):(\d{2}):(\d{2})\.(\d{1,2})$/;
|
||||
const ASS_FORMAT_PREFIX = 'Format:';
|
||||
const ASS_DIALOGUE_PREFIX = 'Dialogue:';
|
||||
const ASS_COMMENT_PREFIX = 'Comment:';
|
||||
const ASS_NAME_FIELD_ALIASES = ['name', 'actor'];
|
||||
const CANONICAL_MATCH_MARGIN_SECONDS = 1;
|
||||
const MIN_CANONICAL_ANIMATION_EVENTS = 3;
|
||||
// A tiny animated fragment can itself be composed from still smaller glyph events. It is
|
||||
// not enough evidence that the fragment represents an authored line boundary.
|
||||
const MIN_CANONICAL_DIALOGUE_TEXT_LENGTH = 4;
|
||||
|
||||
function parseAssTimestamp(raw: string): number | null {
|
||||
const match = ASS_TIMING_PATTERN.exec(raw.trim());
|
||||
@@ -166,10 +186,333 @@ function findFieldIndex(formatFields: string[], aliases: string[]): number {
|
||||
return -1;
|
||||
}
|
||||
|
||||
function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
interface ParsedAssEvents {
|
||||
dialogue: AnnotatedSubtitleCue[];
|
||||
comments: AnnotatedSubtitleCue[];
|
||||
}
|
||||
|
||||
// Every candidate line re-reads the compacted text of each event in its window, so on
|
||||
// fragment-heavy scripts the same event compacts thousands of times without this cache.
|
||||
const compactMatchTextCache = new WeakMap<AnnotatedSubtitleCue, string>();
|
||||
|
||||
function compactAssMatchText(text: string): string {
|
||||
return text.replace(/\s+/gu, '');
|
||||
}
|
||||
|
||||
function compactCueMatchText(cue: AnnotatedSubtitleCue): string {
|
||||
let compact = compactMatchTextCache.get(cue);
|
||||
if (compact === undefined) {
|
||||
compact = compactAssMatchText(cue.text);
|
||||
compactMatchTextCache.set(cue, compact);
|
||||
}
|
||||
return compact;
|
||||
}
|
||||
|
||||
function assEventGroupKey(cue: AnnotatedSubtitleCue): string {
|
||||
return `${cue.style}\0${cue.name}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Windowed lookup over one style/name group. Every candidate line queries its time
|
||||
* neighborhood, and fragment-heavy scripts put thousands of candidates in one group, so
|
||||
* a linear rescan per candidate is quadratic in practice. Events are sorted by start
|
||||
* once; `prefixMaxEnd` lets the backward walk stop as soon as no earlier event can still
|
||||
* reach the window.
|
||||
*/
|
||||
interface AssEventGroupIndex {
|
||||
byStart: AnnotatedSubtitleCue[];
|
||||
prefixMaxEnd: number[];
|
||||
}
|
||||
|
||||
function buildAssEventGroupIndex(events: readonly AnnotatedSubtitleCue[]): AssEventGroupIndex {
|
||||
const byStart = [...events].sort((a, b) => a.startTime - b.startTime || a.order - b.order);
|
||||
const prefixMaxEnd: number[] = [];
|
||||
let maxEnd = -Infinity;
|
||||
for (const event of byStart) {
|
||||
maxEnd = Math.max(maxEnd, event.endTime);
|
||||
prefixMaxEnd.push(maxEnd);
|
||||
}
|
||||
return { byStart, prefixMaxEnd };
|
||||
}
|
||||
|
||||
/** Group events overlapping `[startTime, endTime]`, returned in source order. */
|
||||
function eventsOverlappingWindow(
|
||||
index: AssEventGroupIndex,
|
||||
startTime: number,
|
||||
endTime: number,
|
||||
): AnnotatedSubtitleCue[] {
|
||||
const { byStart, prefixMaxEnd } = index;
|
||||
let low = 0;
|
||||
let high = byStart.length;
|
||||
while (low < high) {
|
||||
const mid = (low + high) >>> 1;
|
||||
if (byStart[mid]!.startTime <= endTime) {
|
||||
low = mid + 1;
|
||||
} else {
|
||||
high = mid;
|
||||
}
|
||||
}
|
||||
const matches: AnnotatedSubtitleCue[] = [];
|
||||
for (let i = low - 1; i >= 0 && prefixMaxEnd[i]! >= startTime; i -= 1) {
|
||||
if (byStart[i]!.endTime >= startTime) {
|
||||
matches.push(byStart[i]!);
|
||||
}
|
||||
}
|
||||
return matches.sort((a, b) => a.order - b.order);
|
||||
}
|
||||
|
||||
interface FragmentGroup {
|
||||
text: string;
|
||||
events: AnnotatedSubtitleCue[];
|
||||
}
|
||||
|
||||
function fragmentPlacementAnchors(event: AnnotatedSubtitleCue): Set<string> {
|
||||
const anchors = new Set<string>();
|
||||
for (const command of event.overrides) {
|
||||
const name = command.name.toLowerCase();
|
||||
const args = command.args.split(',').map((value) => value.trim());
|
||||
if (name === 'pos' && args.length >= 2) {
|
||||
anchors.add(`pos:${args[0]},${args[1]}`);
|
||||
} else if (name === 'move' && args.length >= 4) {
|
||||
anchors.add(`move:${args[0]},${args[1]}`);
|
||||
anchors.add(`move:${args[2]},${args[3]}`);
|
||||
}
|
||||
}
|
||||
return anchors;
|
||||
}
|
||||
|
||||
function isRepeatedFragmentCopy(
|
||||
previous: AnnotatedSubtitleCue,
|
||||
current: AnnotatedSubtitleCue,
|
||||
): boolean {
|
||||
const previousAnchors = fragmentPlacementAnchors(previous);
|
||||
if ([...fragmentPlacementAnchors(current)].some((anchor) => previousAnchors.has(anchor))) {
|
||||
return true;
|
||||
}
|
||||
return (
|
||||
previous.startTime === current.startTime &&
|
||||
previous.endTime === current.endTime &&
|
||||
previous.overrideSignature === current.overrideSignature
|
||||
);
|
||||
}
|
||||
|
||||
function groupConsecutiveAssFragments(events: readonly AnnotatedSubtitleCue[]): FragmentGroup[] {
|
||||
const groups: FragmentGroup[] = [];
|
||||
for (const event of events) {
|
||||
const text = compactCueMatchText(event);
|
||||
if (!text) {
|
||||
continue;
|
||||
}
|
||||
const previous = groups.at(-1);
|
||||
if (
|
||||
previous?.text === text &&
|
||||
previous.events.some((previousEvent) => isRepeatedFragmentCopy(previousEvent, event))
|
||||
) {
|
||||
previous.events.push(event);
|
||||
} else {
|
||||
groups.push({ text, events: [event] });
|
||||
}
|
||||
}
|
||||
return groups;
|
||||
}
|
||||
|
||||
function findCanonicalFragmentEvents(
|
||||
events: readonly AnnotatedSubtitleCue[],
|
||||
canonicalText: string,
|
||||
): AnnotatedSubtitleCue[] {
|
||||
const groups = groupConsecutiveAssFragments(events);
|
||||
const matches = new Set<AnnotatedSubtitleCue>();
|
||||
|
||||
for (let start = 0; start < groups.length; start += 1) {
|
||||
let combined = '';
|
||||
for (let end = start; end < groups.length; end += 1) {
|
||||
const group = groups[end]!;
|
||||
// A complete rendered copy cannot prove that the neighboring events are its
|
||||
// fragments. Exact full-line animation is handled separately for comments.
|
||||
if (group.text.length >= canonicalText.length) {
|
||||
break;
|
||||
}
|
||||
const next = combined + group.text;
|
||||
if (!canonicalText.startsWith(next)) {
|
||||
break;
|
||||
}
|
||||
combined = next;
|
||||
if (combined !== canonicalText) {
|
||||
continue;
|
||||
}
|
||||
for (let index = start; index <= end; index += 1) {
|
||||
for (const event of groups[index]!.events) {
|
||||
matches.add(event);
|
||||
}
|
||||
}
|
||||
start = end;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return [...matches];
|
||||
}
|
||||
|
||||
function matchingAssAnimationEvents(options: {
|
||||
candidate: AnnotatedSubtitleCue;
|
||||
group: AssEventGroupIndex;
|
||||
allowFullLineFrames: boolean;
|
||||
}): AnnotatedSubtitleCue[] {
|
||||
const canonicalText = compactCueMatchText(options.candidate);
|
||||
// The group index already restricts to the candidate's style and name.
|
||||
const nearby = eventsOverlappingWindow(
|
||||
options.group,
|
||||
options.candidate.startTime - CANONICAL_MATCH_MARGIN_SECONDS,
|
||||
options.candidate.endTime + CANONICAL_MATCH_MARGIN_SECONDS,
|
||||
);
|
||||
const fragments = findCanonicalFragmentEvents(nearby, canonicalText);
|
||||
if (fragments.length >= MIN_CANONICAL_ANIMATION_EVENTS && hasAssAnimationEvidence(fragments)) {
|
||||
return fragments;
|
||||
}
|
||||
|
||||
if (!options.allowFullLineFrames) {
|
||||
return [];
|
||||
}
|
||||
const fullLineFrames = nearby.filter((cue) => compactCueMatchText(cue) === canonicalText);
|
||||
return fullLineFrames.length >= MIN_CANONICAL_ANIMATION_EVENTS &&
|
||||
hasAssAnimationEvidence(fullLineFrames)
|
||||
? fullLineFrames
|
||||
: [];
|
||||
}
|
||||
|
||||
// Reductions rather than `Math.min(...events)`: one generated line can carry an
|
||||
// unbounded number of events, and spreading them all as arguments risks the engine's
|
||||
// argument-count limit.
|
||||
function earliestStartTime(events: readonly AnnotatedSubtitleCue[], seed = Infinity): number {
|
||||
return events.reduce((earliest, event) => Math.min(earliest, event.startTime), seed);
|
||||
}
|
||||
|
||||
function latestEndTime(events: readonly AnnotatedSubtitleCue[], seed = -Infinity): number {
|
||||
return events.reduce((latest, event) => Math.max(latest, event.endTime), seed);
|
||||
}
|
||||
|
||||
function includeCanonicalBoundaryEvents(options: {
|
||||
candidate: AnnotatedSubtitleCue;
|
||||
group: AssEventGroupIndex;
|
||||
animationEvents: readonly AnnotatedSubtitleCue[];
|
||||
}): AnnotatedSubtitleCue[] {
|
||||
const canonicalText = compactCueMatchText(options.candidate);
|
||||
const startTime = earliestStartTime(options.animationEvents);
|
||||
const endTime = latestEndTime(options.animationEvents);
|
||||
return eventsOverlappingWindow(
|
||||
options.group,
|
||||
startTime - CANONICAL_MATCH_MARGIN_SECONDS,
|
||||
endTime + CANONICAL_MATCH_MARGIN_SECONDS,
|
||||
).filter((cue) => compactCueMatchText(cue) === canonicalText);
|
||||
}
|
||||
|
||||
function recoverCanonicalAssEvents({
|
||||
dialogue,
|
||||
comments,
|
||||
}: ParsedAssEvents): AnnotatedSubtitleCue[] {
|
||||
const recovered: AnnotatedSubtitleCue[] = [];
|
||||
const suppressed = new Set<AnnotatedSubtitleCue>();
|
||||
// A recovery is only as good as its owning event. When a later candidate proves that
|
||||
// an earlier candidate was itself a generated frame of its animation, the earlier
|
||||
// recovery is a duplicate of the same authored line and must be withdrawn.
|
||||
const recoveredByOwner = new Map<AnnotatedSubtitleCue, AnnotatedSubtitleCue>();
|
||||
const withdrawn = new Set<AnnotatedSubtitleCue>();
|
||||
const eventsByGroup = new Map<string, AnnotatedSubtitleCue[]>();
|
||||
for (const cue of dialogue) {
|
||||
const key = assEventGroupKey(cue);
|
||||
const group = eventsByGroup.get(key);
|
||||
if (group) {
|
||||
group.push(cue);
|
||||
} else {
|
||||
eventsByGroup.set(key, [cue]);
|
||||
}
|
||||
}
|
||||
const indexByGroup = new Map<string, AssEventGroupIndex>();
|
||||
for (const [key, events] of eventsByGroup) {
|
||||
indexByGroup.set(key, buildAssEventGroupIndex(events));
|
||||
}
|
||||
const emptyGroupIndex: AssEventGroupIndex = { byStart: [], prefixMaxEnd: [] };
|
||||
const candidates = [
|
||||
...comments.map((cue) => ({ cue, kind: 'comment' as const })),
|
||||
...dialogue
|
||||
.filter(
|
||||
(cue) =>
|
||||
compactCueMatchText(cue).length >= MIN_CANONICAL_DIALOGUE_TEXT_LENGTH &&
|
||||
hasAssAnimationEvidence([cue]),
|
||||
)
|
||||
.sort((left, right) => right.text.length - left.text.length || left.order - right.order)
|
||||
.map((cue) => ({ cue, kind: 'dialogue' as const })),
|
||||
];
|
||||
|
||||
for (const { cue: candidate, kind } of candidates) {
|
||||
if (candidate.endTime <= candidate.startTime || suppressed.has(candidate)) {
|
||||
continue;
|
||||
}
|
||||
const canonicalText = compactCueMatchText(candidate);
|
||||
if (!canonicalText) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const group = indexByGroup.get(assEventGroupKey(candidate)) ?? emptyGroupIndex;
|
||||
const animationEvents = matchingAssAnimationEvents({
|
||||
candidate,
|
||||
group,
|
||||
allowFullLineFrames: kind === 'comment',
|
||||
});
|
||||
if (animationEvents.length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const boundaryEvents = includeCanonicalBoundaryEvents({
|
||||
candidate,
|
||||
group,
|
||||
animationEvents,
|
||||
});
|
||||
const generatedEvents = [...new Set([...animationEvents, ...boundaryEvents])];
|
||||
const animationStartTime = earliestStartTime(generatedEvents, candidate.startTime);
|
||||
const animationEndTime = latestEndTime(generatedEvents, candidate.endTime);
|
||||
const startTime = kind === 'comment' ? candidate.startTime : animationStartTime;
|
||||
const endTime = kind === 'comment' ? candidate.endTime : animationEndTime;
|
||||
const recoveredCue: AnnotatedSubtitleCue = {
|
||||
...candidate,
|
||||
startTime,
|
||||
endTime,
|
||||
animationStartTime,
|
||||
animationEndTime,
|
||||
source: 'canonical-ass',
|
||||
};
|
||||
recovered.push(recoveredCue);
|
||||
recoveredByOwner.set(candidate, recoveredCue);
|
||||
for (const event of generatedEvents) {
|
||||
suppressed.add(event);
|
||||
if (event === candidate) {
|
||||
continue;
|
||||
}
|
||||
const priorRecovery = recoveredByOwner.get(event);
|
||||
if (priorRecovery) {
|
||||
// No text is lost by withdrawing: a fragment claim means the withdrawn line is
|
||||
// a contiguous piece of this candidate's text, and a boundary claim means the
|
||||
// texts are equal, so the surviving canonical cue always contains it.
|
||||
withdrawn.add(priorRecovery);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const survivingRecovered = recovered.filter((cue) => !withdrawn.has(cue));
|
||||
if (survivingRecovered.length === 0) {
|
||||
return dialogue;
|
||||
}
|
||||
return [...dialogue.filter((cue) => !suppressed.has(cue)), ...survivingRecovered].sort(
|
||||
(a, b) => a.startTime - b.startTime || a.endTime - b.endTime || a.order - b.order,
|
||||
);
|
||||
}
|
||||
|
||||
function parseAnnotatedAssEvents(content: string): ParsedAssEvents {
|
||||
const cues: AnnotatedSubtitleCue[] = [];
|
||||
const comments: AnnotatedSubtitleCue[] = [];
|
||||
const lines = content.split(/\r?\n/);
|
||||
let inEventsSection = false;
|
||||
let eventOrder = 0;
|
||||
const fieldIndex = {
|
||||
start: -1,
|
||||
end: -1,
|
||||
@@ -222,7 +565,12 @@ function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!trimmed.startsWith(ASS_DIALOGUE_PREFIX)) {
|
||||
const eventPrefix = trimmed.startsWith(ASS_DIALOGUE_PREFIX)
|
||||
? ASS_DIALOGUE_PREFIX
|
||||
: trimmed.startsWith(ASS_COMMENT_PREFIX)
|
||||
? ASS_COMMENT_PREFIX
|
||||
: null;
|
||||
if (!eventPrefix) {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -230,7 +578,7 @@ function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
continue;
|
||||
}
|
||||
|
||||
const fields = trimmed.slice(ASS_DIALOGUE_PREFIX.length).split(',');
|
||||
const fields = trimmed.slice(eventPrefix.length).split(',');
|
||||
if (
|
||||
fieldIndex.start >= fields.length ||
|
||||
fieldIndex.end >= fields.length ||
|
||||
@@ -254,7 +602,7 @@ function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
const effect = readField(fields, fieldIndex.effect);
|
||||
const layer = Number(readField(fields, fieldIndex.layer));
|
||||
const overrides = collectAssOverrideCommands(rawText);
|
||||
cues.push({
|
||||
const cue: AnnotatedSubtitleCue = {
|
||||
startTime,
|
||||
endTime,
|
||||
text,
|
||||
@@ -266,11 +614,21 @@ function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
effectKind: parseAssEffectField(effect),
|
||||
overrides,
|
||||
overrideSignature: assOverrideSignature(overrides),
|
||||
order: cues.length,
|
||||
});
|
||||
order: eventOrder,
|
||||
};
|
||||
eventOrder += 1;
|
||||
if (eventPrefix === ASS_COMMENT_PREFIX) {
|
||||
comments.push(cue);
|
||||
} else {
|
||||
cues.push(cue);
|
||||
}
|
||||
}
|
||||
|
||||
return cues;
|
||||
return { dialogue: cues, comments };
|
||||
}
|
||||
|
||||
function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
return recoverCanonicalAssEvents(parseAnnotatedAssEvents(content));
|
||||
}
|
||||
|
||||
export function parseAssCues(content: string): SubtitleCue[] {
|
||||
|
||||
@@ -1,22 +1,6 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import {
|
||||
buildNotifySendEnv,
|
||||
createNotifySendReplacer,
|
||||
resolveDefaultNotificationIconPath,
|
||||
} from './notification';
|
||||
|
||||
test('notify-send child environment drops the AppImage library-path override', () => {
|
||||
const env = buildNotifySendEnv({
|
||||
LD_LIBRARY_PATH: '/tmp/.mount_SubMinXXXXXX/usr/lib',
|
||||
DBUS_SESSION_BUS_ADDRESS: 'unix:path=/run/user/1000/bus',
|
||||
HOME: '/home/user',
|
||||
});
|
||||
|
||||
assert.equal(env.LD_LIBRARY_PATH, undefined);
|
||||
assert.equal(env.DBUS_SESSION_BUS_ADDRESS, 'unix:path=/run/user/1000/bus');
|
||||
assert.equal(env.HOME, '/home/user');
|
||||
});
|
||||
import { createNotifySendReplacer, resolveDefaultNotificationIconPath } from './notification';
|
||||
|
||||
test('default notification icon resolves packaged SubMiner asset when no per-notification icon is provided', () => {
|
||||
const path = resolveDefaultNotificationIconPath({
|
||||
|
||||
@@ -203,19 +203,8 @@ export function createNotifySendReplacer(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Electron AppImages export `LD_LIBRARY_PATH=<mount>/usr/lib`, whose bundled libnotify predates the
|
||||
* symbols the system notify-send links against, so an inherited environment kills the child with a
|
||||
* symbol lookup error before it can send anything. A system binary resolves its own libraries fine,
|
||||
* so the override is dropped entirely rather than filtered.
|
||||
*/
|
||||
export function buildNotifySendEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
|
||||
const { LD_LIBRARY_PATH: _dropped, ...rest } = env;
|
||||
return rest;
|
||||
}
|
||||
|
||||
const showLinuxReplaceableNotification = createNotifySendReplacer((args, callback) =>
|
||||
execFile('notify-send', args, { timeout: 5_000, env: buildNotifySendEnv() }, (error, stdout) =>
|
||||
execFile('notify-send', args, { timeout: 5_000 }, (error, stdout) =>
|
||||
callback(error, stdout ?? ''),
|
||||
),
|
||||
);
|
||||
|
||||
+37
-5
@@ -235,6 +235,7 @@ import {
|
||||
createCycleSecondarySubModeRuntimeHandler,
|
||||
} from './main/runtime/domains/mpv';
|
||||
import { buildSubtitleTrackDiagnostics } from './main/runtime/mpv-track-diagnostics';
|
||||
import { resolveCanonicalPrimarySubtitle } from './main/runtime/primary-subtitle-text';
|
||||
import {
|
||||
createBuildCopyCurrentSubtitleMainDepsHandler,
|
||||
createBuildHandleMineSentenceDigitMainDepsHandler,
|
||||
@@ -1806,10 +1807,42 @@ async function openYoutubeTrackPickerFromPlayback(): Promise<void> {
|
||||
let appTray: Tray | null = null;
|
||||
let tokenizeSubtitleDeferred: ((text: string) => Promise<SubtitleData>) | null = null;
|
||||
function withCurrentSubtitleTiming(payload: SubtitleData): SubtitleData {
|
||||
const canonical = resolveCanonicalPrimarySubtitle({
|
||||
liveText: payload.text,
|
||||
currentTimeSec: Number(appState.mpvClient?.currentTimePos),
|
||||
cues: appState.activeParsedSubtitleCues,
|
||||
});
|
||||
return {
|
||||
...payload,
|
||||
startTime: appState.mpvClient?.currentSubStart ?? null,
|
||||
endTime: appState.mpvClient?.currentSubEnd ?? null,
|
||||
startTime: canonical?.startTime ?? appState.mpvClient?.currentSubStart ?? null,
|
||||
endTime: canonical?.endTime ?? appState.mpvClient?.currentSubEnd ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function captureCurrentPrimarySubtitleMiningContext(): SubtitleMiningContext | null {
|
||||
const canonical = resolveCanonicalPrimarySubtitle({
|
||||
liveText: appState.mpvClient?.currentSubText ?? '',
|
||||
currentTimeSec: Number(appState.mpvClient?.currentTimePos),
|
||||
cues: appState.activeParsedSubtitleCues,
|
||||
});
|
||||
// Same validity bar as the live capture path: an unusable canonical span must fall
|
||||
// back rather than hand mining an empty line or an inverted range.
|
||||
const canonicalText = canonical?.text.trim();
|
||||
if (
|
||||
!canonical ||
|
||||
!canonicalText ||
|
||||
!Number.isFinite(canonical.startTime) ||
|
||||
!Number.isFinite(canonical.endTime) ||
|
||||
canonical.endTime <= canonical.startTime
|
||||
) {
|
||||
return captureLiveSubtitleMiningContext(appState.mpvClient);
|
||||
}
|
||||
return {
|
||||
source: 'overlay',
|
||||
text: canonicalText,
|
||||
startTime: canonical.startTime,
|
||||
endTime: canonical.endTime,
|
||||
capturedAtMs: Date.now(),
|
||||
};
|
||||
}
|
||||
function emitSubtitlePayload(payload: SubtitleData, options?: { resumePrefetch?: boolean }): void {
|
||||
@@ -5231,6 +5264,7 @@ const markLastCardAsAudioCardHandler = createMarkLastCardAsAudioCardHandler(
|
||||
const buildMineSentenceCardMainDepsHandler = createBuildMineSentenceCardMainDepsHandler({
|
||||
getAnkiIntegration: () => appState.ankiIntegration,
|
||||
getMpvClient: () => appState.mpvClient,
|
||||
getPrimarySubtitle: () => captureCurrentPrimarySubtitleMiningContext(),
|
||||
showMpvOsd: (text) => overlayNotificationsRuntime.showConfiguredStatusNotification(text),
|
||||
mineSentenceCardCore,
|
||||
recordCardsMined: (count, noteIds) => {
|
||||
@@ -5544,9 +5578,7 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
// live mpv sub timings at lookup time so media generation clips the mined line even
|
||||
// when extraction finishes long after playback has moved on.
|
||||
recordSubtitleMiningContext: (context) =>
|
||||
recordSubtitleMiningContext(
|
||||
context ?? captureLiveSubtitleMiningContext(appState.mpvClient),
|
||||
),
|
||||
recordSubtitleMiningContext(context ?? captureCurrentPrimarySubtitleMiningContext()),
|
||||
quitApp: () => requestAppQuit(),
|
||||
toggleVisibleOverlay: () => toggleVisibleOverlay(),
|
||||
tokenizeCurrentSubtitle: async () => {
|
||||
|
||||
@@ -244,13 +244,13 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
};
|
||||
};
|
||||
|
||||
const findCachedSnapshotForSeriesKey = async (
|
||||
const findCachedSnapshotForSeriesKey = (
|
||||
seriesKey: string,
|
||||
fallbackSeriesKey?: string,
|
||||
): Promise<CharacterDictionarySnapshot | null> => {
|
||||
): CharacterDictionarySnapshot | null => {
|
||||
const acceptedKeys = new Set([seriesKey, fallbackSeriesKey].filter(Boolean));
|
||||
return (
|
||||
(await readCachedSnapshots(outputDir)).find((snapshot) => {
|
||||
readCachedSnapshots(outputDir).find((snapshot) => {
|
||||
const snapshotSeriesKey = buildCharacterDictionarySeriesKey({
|
||||
mediaPath: null,
|
||||
mediaTitle: snapshot.mediaTitle,
|
||||
@@ -293,9 +293,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
|
||||
const cachedResolution = readCachedMediaResolution(outputDir, seriesKey);
|
||||
if (cachedResolution) {
|
||||
const cachedSnapshot = await readSnapshot(
|
||||
getSnapshotPath(outputDir, cachedResolution.mediaId),
|
||||
);
|
||||
const cachedSnapshot = readSnapshot(getSnapshotPath(outputDir, cachedResolution.mediaId));
|
||||
if (cachedSnapshot) {
|
||||
deps.logInfo?.(
|
||||
`[dictionary] cached AniList match: ${cachedSnapshot.mediaTitle} -> AniList ${cachedSnapshot.mediaId}`,
|
||||
@@ -307,7 +305,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
}
|
||||
}
|
||||
|
||||
const cachedSnapshot = await findCachedSnapshotForSeriesKey(seriesKey, unscopedSeriesKey);
|
||||
const cachedSnapshot = findCachedSnapshotForSeriesKey(seriesKey, unscopedSeriesKey);
|
||||
if (cachedSnapshot) {
|
||||
writeCachedMediaResolution(outputDir, {
|
||||
seriesKey,
|
||||
@@ -350,7 +348,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
progress?: CharacterDictionarySnapshotProgressCallbacks,
|
||||
): Promise<CharacterDictionarySnapshotResult> => {
|
||||
const snapshotPath = getSnapshotPath(outputDir, mediaId);
|
||||
const cachedSnapshot = await readSnapshot(snapshotPath);
|
||||
const cachedSnapshot = readSnapshot(snapshotPath);
|
||||
const refreshReason = cachedSnapshot ? getCachedSnapshotRefreshReason(cachedSnapshot) : null;
|
||||
if (cachedSnapshot && refreshReason === null) {
|
||||
deps.logInfo?.(`[dictionary] snapshot hit for AniList ${mediaId}`);
|
||||
@@ -487,7 +485,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
resolvedNameSplits,
|
||||
nameSplitSource,
|
||||
);
|
||||
await writeSnapshot(snapshotPath, snapshot);
|
||||
writeSnapshot(snapshotPath, snapshot);
|
||||
deps.logInfo?.(
|
||||
`[dictionary] stored snapshot for AniList ${mediaId}: ${snapshot.entryCount} terms`,
|
||||
);
|
||||
@@ -528,22 +526,19 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
const snapshotResults = await Promise.all(
|
||||
normalizedMediaIds.map((mediaId) => getOrCreateSnapshot(mediaId)),
|
||||
);
|
||||
// Sequential on purpose: each snapshot parse is a chunk of main-thread work, so reading them
|
||||
// one at a time keeps the event loop breathing between files.
|
||||
const snapshots: CharacterDictionarySnapshot[] = [];
|
||||
for (const { mediaId } of snapshotResults) {
|
||||
const snapshot = await readSnapshot(getSnapshotPath(outputDir, mediaId));
|
||||
const snapshots = snapshotResults.map(({ mediaId }) => {
|
||||
const snapshot = readSnapshot(getSnapshotPath(outputDir, mediaId));
|
||||
if (!snapshot) {
|
||||
throw new Error(`Missing character dictionary snapshot for AniList ${mediaId}.`);
|
||||
}
|
||||
snapshots.push(snapshot);
|
||||
}
|
||||
return snapshot;
|
||||
});
|
||||
const revision = buildMergedRevision(normalizedMediaIds, snapshots);
|
||||
const description =
|
||||
snapshots.length === 1
|
||||
? `Character names from ${snapshots[0]!.mediaTitle}`
|
||||
: `Character names from ${snapshots.length} recent anime`;
|
||||
const { zipPath, entryCount } = await buildDictionaryZip(
|
||||
const { zipPath, entryCount } = buildDictionaryZip(
|
||||
getMergedZipPath(outputDir),
|
||||
CHARACTER_DICTIONARY_MERGED_TITLE,
|
||||
description,
|
||||
@@ -638,7 +633,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
resolvedMedia.title,
|
||||
waitForAniListRequestSlot,
|
||||
);
|
||||
const storedSnapshot = await readSnapshot(getSnapshotPath(outputDir, resolvedMedia.id));
|
||||
const storedSnapshot = readSnapshot(getSnapshotPath(outputDir, resolvedMedia.id));
|
||||
if (!storedSnapshot) {
|
||||
throw new Error(`Snapshot missing after generation for AniList ${resolvedMedia.id}.`);
|
||||
}
|
||||
@@ -647,7 +642,7 @@ export function createCharacterDictionaryRuntimeService(deps: CharacterDictionar
|
||||
const description = `Character names from ${storedSnapshot.mediaTitle} [AniList media ID ${resolvedMedia.id}]`;
|
||||
const zipPath = path.join(outputDir, `anilist-${resolvedMedia.id}.zip`);
|
||||
deps.logInfo?.(`[dictionary] building ZIP for AniList ${resolvedMedia.id}`);
|
||||
await buildDictionaryZip(
|
||||
buildDictionaryZip(
|
||||
zipPath,
|
||||
dictionaryTitle,
|
||||
description,
|
||||
|
||||
@@ -3,7 +3,6 @@ import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import test from 'node:test';
|
||||
import { isDeepStrictEqual } from 'node:util';
|
||||
import { getSnapshotPath, readSnapshot, writeSnapshot } from './cache';
|
||||
import { CHARACTER_DICTIONARY_FORMAT_VERSION } from './constants';
|
||||
import type { CharacterDictionarySnapshot } from './types';
|
||||
@@ -30,72 +29,17 @@ function createSnapshot(): CharacterDictionarySnapshot {
|
||||
};
|
||||
}
|
||||
|
||||
test('writeSnapshot persists and readSnapshot restores current-format snapshots', async () => {
|
||||
test('writeSnapshot persists and readSnapshot restores current-format snapshots', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshotPath = getSnapshotPath(outputDir, 130298);
|
||||
const snapshot = createSnapshot();
|
||||
|
||||
await writeSnapshot(snapshotPath, snapshot);
|
||||
writeSnapshot(snapshotPath, snapshot);
|
||||
|
||||
assert.deepEqual(await readSnapshot(snapshotPath), { ...snapshot, nameSplitSource: 'heuristic' });
|
||||
assert.deepEqual(readSnapshot(snapshotPath), { ...snapshot, nameSplitSource: 'heuristic' });
|
||||
});
|
||||
|
||||
// A manual generate and an auto-sync can both land on the same media, so two writes for one
|
||||
// snapshot can overlap. They must not stream into a shared temp file and interleave into a
|
||||
// half-and-half snapshot.
|
||||
test('concurrent writeSnapshot calls for the same media leave one complete snapshot', async () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshotPath = getSnapshotPath(outputDir, 130298);
|
||||
const base = createSnapshot();
|
||||
// Distinct titles, lengths, and term text so the surviving file can be pinned to exactly one
|
||||
// writer rather than merely "a snapshot that parses". A shared temp file is caught by the
|
||||
// losing writers failing to rename; interleaved content is only caught when the timing happens
|
||||
// to leave a mix, which is why the assertion checks identity rather than shape.
|
||||
const variants: CharacterDictionarySnapshot[] = ['alpha', 'beta', 'gamma'].map((label, index) => {
|
||||
const entryCount = 400 + index * 100;
|
||||
return {
|
||||
...base,
|
||||
mediaTitle: `${base.mediaTitle} ${label}`,
|
||||
entryCount,
|
||||
termEntries: Array.from({ length: entryCount }, (_entry, entryIndex) => [
|
||||
`${label}${entryIndex}`,
|
||||
'なまえ',
|
||||
'name primary',
|
||||
'',
|
||||
75,
|
||||
[`${label} character ${entryIndex} `.repeat(600)],
|
||||
0,
|
||||
'',
|
||||
]) as CharacterDictionarySnapshot['termEntries'],
|
||||
};
|
||||
});
|
||||
|
||||
await Promise.all(variants.map((variant) => writeSnapshot(snapshotPath, variant)));
|
||||
|
||||
const restored = await readSnapshot(snapshotPath);
|
||||
const expected = variants.map((variant) => ({
|
||||
...variant,
|
||||
nameSplitSource: 'heuristic' as const,
|
||||
}));
|
||||
const matches = expected.filter((candidate) => isDeepStrictEqual(restored, candidate));
|
||||
assert.equal(
|
||||
matches.length,
|
||||
1,
|
||||
`expected exactly one writer's complete snapshot to survive, got ${
|
||||
restored === null
|
||||
? 'an unreadable file'
|
||||
: `entryCount=${restored.entryCount}, terms=${restored.termEntries.length}, title=${restored.mediaTitle}`
|
||||
}`,
|
||||
);
|
||||
|
||||
// Every writer cleaned up after itself, so no temp files are left behind.
|
||||
const leftovers = fs
|
||||
.readdirSync(path.dirname(snapshotPath))
|
||||
.filter((name) => name.includes('.tmp-'));
|
||||
assert.deepEqual(leftovers, []);
|
||||
});
|
||||
|
||||
test('readSnapshot preserves the mecab name-split source and defaults missing values to heuristic', async () => {
|
||||
test('readSnapshot preserves the mecab name-split source and defaults missing values to heuristic', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshotPath = getSnapshotPath(outputDir, 130298);
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
@@ -103,12 +47,12 @@ test('readSnapshot preserves the mecab name-split source and defaults missing va
|
||||
nameSplitSource: 'mecab',
|
||||
};
|
||||
|
||||
await writeSnapshot(snapshotPath, snapshot);
|
||||
writeSnapshot(snapshotPath, snapshot);
|
||||
|
||||
assert.equal((await readSnapshot(snapshotPath))?.nameSplitSource, 'mecab');
|
||||
assert.equal(readSnapshot(snapshotPath)?.nameSplitSource, 'mecab');
|
||||
});
|
||||
|
||||
test('readSnapshot ignores snapshots written with an older format version', async () => {
|
||||
test('readSnapshot ignores snapshots written with an older format version', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshotPath = getSnapshotPath(outputDir, 130298);
|
||||
const staleSnapshot = {
|
||||
@@ -119,10 +63,10 @@ test('readSnapshot ignores snapshots written with an older format version', asyn
|
||||
fs.mkdirSync(path.dirname(snapshotPath), { recursive: true });
|
||||
fs.writeFileSync(snapshotPath, JSON.stringify(staleSnapshot), 'utf8');
|
||||
|
||||
assert.equal(await readSnapshot(snapshotPath), null);
|
||||
assert.equal(readSnapshot(snapshotPath), null);
|
||||
});
|
||||
|
||||
test('readSnapshot ignores v15 snapshots with stale romanized character-name entries', async () => {
|
||||
test('readSnapshot ignores v15 snapshots with stale romanized character-name entries', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshotPath = getSnapshotPath(outputDir, 130298);
|
||||
const staleSnapshot = {
|
||||
@@ -134,5 +78,5 @@ test('readSnapshot ignores v15 snapshots with stale romanized character-name ent
|
||||
fs.mkdirSync(path.dirname(snapshotPath), { recursive: true });
|
||||
fs.writeFileSync(snapshotPath, JSON.stringify(staleSnapshot), 'utf8');
|
||||
|
||||
assert.equal(await readSnapshot(snapshotPath), null);
|
||||
assert.equal(readSnapshot(snapshotPath), null);
|
||||
});
|
||||
|
||||
@@ -102,42 +102,24 @@ export function writeCachedMediaResolution(
|
||||
writeMediaResolutionEntries(outputDir, [...remaining, normalized]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshots for long series run to hundreds of MB each, so everything here reads them off the main
|
||||
* thread's critical path: file IO is async and only the unavoidable JSON.parse runs on the loop,
|
||||
* one file at a time. Reading the whole directory synchronously used to block the process for
|
||||
* multiple seconds, long enough for the compositor to declare the app unresponsive mid-playback.
|
||||
*/
|
||||
export async function readCachedSnapshots(
|
||||
outputDir: string,
|
||||
): Promise<CharacterDictionarySnapshot[]> {
|
||||
export function readCachedSnapshots(outputDir: string): CharacterDictionarySnapshot[] {
|
||||
let entries: fs.Dirent[] = [];
|
||||
try {
|
||||
entries = await fs.promises.readdir(getSnapshotsDir(outputDir), { withFileTypes: true });
|
||||
entries = fs.readdirSync(getSnapshotsDir(outputDir), { withFileTypes: true });
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
|
||||
const names = entries
|
||||
return entries
|
||||
.filter((entry) => entry.isFile() && /^anilist-\d+\.json$/.test(entry.name))
|
||||
.map((entry) => entry.name)
|
||||
.sort((left, right) => left.localeCompare(right));
|
||||
|
||||
const snapshots: CharacterDictionarySnapshot[] = [];
|
||||
for (const name of names) {
|
||||
const snapshot = await readSnapshot(path.join(getSnapshotsDir(outputDir), name));
|
||||
if (snapshot) {
|
||||
snapshots.push(snapshot);
|
||||
}
|
||||
}
|
||||
return snapshots;
|
||||
.sort((left, right) => left.name.localeCompare(right.name))
|
||||
.map((entry) => readSnapshot(path.join(getSnapshotsDir(outputDir), entry.name)))
|
||||
.filter((snapshot): snapshot is CharacterDictionarySnapshot => snapshot !== null);
|
||||
}
|
||||
|
||||
export async function readSnapshot(
|
||||
snapshotPath: string,
|
||||
): Promise<CharacterDictionarySnapshot | null> {
|
||||
export function readSnapshot(snapshotPath: string): CharacterDictionarySnapshot | null {
|
||||
try {
|
||||
const raw = await fs.promises.readFile(snapshotPath, 'utf8');
|
||||
const raw = fs.readFileSync(snapshotPath, 'utf8');
|
||||
const parsed = JSON.parse(raw) as Partial<CharacterDictionarySnapshot>;
|
||||
if (!parsed || typeof parsed !== 'object') {
|
||||
return null;
|
||||
@@ -168,64 +150,9 @@ export async function readSnapshot(
|
||||
}
|
||||
}
|
||||
|
||||
// Flushing in a few-MB batches keeps each stringify-and-write slice short; a single
|
||||
// JSON.stringify of a large snapshot blocks the event loop for seconds.
|
||||
const SNAPSHOT_WRITE_FLUSH_BYTES = 4 * 1024 * 1024;
|
||||
|
||||
// Distinguishes concurrent writes of the same snapshot within one process; the pid alone only
|
||||
// separates processes, so two overlapping writers would otherwise stream into the same temp file.
|
||||
let snapshotWriteSequence = 0;
|
||||
|
||||
/**
|
||||
* Streams the snapshot to disk piece by piece instead of stringifying it in one shot, then renames
|
||||
* the finished file into place so a crash mid-write (or two concurrent writers for the same media)
|
||||
* can never leave a torn file where a snapshot used to be.
|
||||
*/
|
||||
export async function writeSnapshot(
|
||||
snapshotPath: string,
|
||||
snapshot: CharacterDictionarySnapshot,
|
||||
): Promise<void> {
|
||||
export function writeSnapshot(snapshotPath: string, snapshot: CharacterDictionarySnapshot): void {
|
||||
ensureDir(path.dirname(snapshotPath));
|
||||
snapshotWriteSequence += 1;
|
||||
const tempPath = `${snapshotPath}.tmp-${process.pid}-${snapshotWriteSequence}`;
|
||||
const handle = await fs.promises.open(tempPath, 'w');
|
||||
try {
|
||||
let buffered: string[] = [];
|
||||
let bufferedBytes = 0;
|
||||
const push = async (chunk: string): Promise<void> => {
|
||||
buffered.push(chunk);
|
||||
bufferedBytes += chunk.length;
|
||||
if (bufferedBytes >= SNAPSHOT_WRITE_FLUSH_BYTES) {
|
||||
const joined = buffered.join('');
|
||||
buffered = [];
|
||||
bufferedBytes = 0;
|
||||
await handle.write(joined, null, 'utf8');
|
||||
}
|
||||
};
|
||||
const writeArray = async (key: string, items: readonly unknown[]): Promise<void> => {
|
||||
await push(`,${JSON.stringify(key)}:[`);
|
||||
for (let i = 0; i < items.length; i += 1) {
|
||||
await push(`${i > 0 ? ',' : ''}${JSON.stringify(items[i])}`);
|
||||
}
|
||||
await push(']');
|
||||
};
|
||||
|
||||
const { termEntries, images, ...scalars } = snapshot;
|
||||
const head = JSON.stringify(scalars);
|
||||
await push(head.slice(0, -1));
|
||||
await writeArray('termEntries', termEntries);
|
||||
await writeArray('images', images);
|
||||
await push('}');
|
||||
if (buffered.length > 0) {
|
||||
await handle.write(buffered.join(''), null, 'utf8');
|
||||
}
|
||||
} catch (error) {
|
||||
await handle.close();
|
||||
await fs.promises.rm(tempPath, { force: true });
|
||||
throw error;
|
||||
}
|
||||
await handle.close();
|
||||
await fs.promises.rename(tempPath, snapshotPath);
|
||||
fs.writeFileSync(snapshotPath, JSON.stringify(snapshot, null, 2), 'utf8');
|
||||
}
|
||||
|
||||
export function buildMergedRevision(
|
||||
|
||||
@@ -11,22 +11,6 @@ import {
|
||||
} from './image-lookup';
|
||||
import type { CharacterDictionarySnapshot } from './types';
|
||||
|
||||
// Lookup indexes rebuild in the background while gets serve stale data, so tests poll until the
|
||||
// refresh they triggered has landed.
|
||||
async function waitForRefresh<T>(probe: () => T | null | undefined): Promise<T> {
|
||||
const deadline = Date.now() + 5000;
|
||||
for (;;) {
|
||||
const value = probe();
|
||||
if (value !== null && value !== undefined) {
|
||||
return value;
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
throw new Error('timed out waiting for background snapshot refresh');
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
}
|
||||
|
||||
const PNG_1X1_BASE64 =
|
||||
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+nmX8AAAAASUVORK5CYII=';
|
||||
|
||||
@@ -34,7 +18,7 @@ function makeTempDir(): string {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-character-image-lookup-'));
|
||||
}
|
||||
|
||||
test('buildCharacterNameImageIndexFromSnapshots maps name terms to character portrait data URLs', async () => {
|
||||
test('buildCharacterNameImageIndexFromSnapshots maps name terms to character portrait data URLs', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
@@ -91,9 +75,9 @@ test('buildCharacterNameImageIndexFromSnapshots maps name terms to character por
|
||||
{ path: 'img/m130298-va456.png', dataBase64: 'BBBB' },
|
||||
],
|
||||
};
|
||||
await writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
|
||||
const index = await buildCharacterNameImageIndexFromSnapshots(outputDir);
|
||||
const index = buildCharacterNameImageIndexFromSnapshots(outputDir);
|
||||
|
||||
assert.deepEqual(index.get('アレクシア'), {
|
||||
src: 'data:image/png;base64,AAAA',
|
||||
@@ -101,7 +85,7 @@ test('buildCharacterNameImageIndexFromSnapshots maps name terms to character por
|
||||
});
|
||||
});
|
||||
|
||||
test('buildCharacterNameImageIndexFromSnapshots sniffs image MIME from bytes before path extension', async () => {
|
||||
test('buildCharacterNameImageIndexFromSnapshots sniffs image MIME from bytes before path extension', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
@@ -132,14 +116,14 @@ test('buildCharacterNameImageIndexFromSnapshots sniffs image MIME from bytes bef
|
||||
],
|
||||
images: [{ path: 'img/m130298-c123.jpg', dataBase64: PNG_1X1_BASE64 }],
|
||||
};
|
||||
await writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
|
||||
const index = await buildCharacterNameImageIndexFromSnapshots(outputDir);
|
||||
const index = buildCharacterNameImageIndexFromSnapshots(outputDir);
|
||||
|
||||
assert.equal(index.get('アレクシア')?.src, `data:image/png;base64,${PNG_1X1_BASE64}`);
|
||||
});
|
||||
|
||||
test('createCharacterDictionaryImageLookup can scope duplicate names to the current media', async () => {
|
||||
test('createCharacterDictionaryImageLookup can scope duplicate names to the current media', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const towerSnapshot: CharacterDictionarySnapshot = {
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
@@ -189,16 +173,15 @@ test('createCharacterDictionaryImageLookup can scope duplicate names to the curr
|
||||
],
|
||||
images: [{ path: 'img/m21202-c2.png', dataBase64: 'KONOSUBA' }],
|
||||
};
|
||||
await writeSnapshot(getSnapshotPath(outputDir, towerSnapshot.mediaId), towerSnapshot);
|
||||
await writeSnapshot(getSnapshotPath(outputDir, konosubaSnapshot.mediaId), konosubaSnapshot);
|
||||
writeSnapshot(getSnapshotPath(outputDir, towerSnapshot.mediaId), towerSnapshot);
|
||||
writeSnapshot(getSnapshotPath(outputDir, konosubaSnapshot.mediaId), konosubaSnapshot);
|
||||
|
||||
const lookup = createCharacterDictionaryImageLookup({ outputDir });
|
||||
|
||||
const scoped = await waitForRefresh(() => lookup.get('カズ', 21202));
|
||||
assert.equal(scoped.alt, 'Kazuma');
|
||||
assert.equal(lookup.get('カズ', 21202)?.alt, 'Kazuma');
|
||||
});
|
||||
|
||||
test('createCharacterDictionaryImageLookup does not fall back globally on scoped miss', async () => {
|
||||
test('createCharacterDictionaryImageLookup does not fall back globally on scoped miss', () => {
|
||||
const outputDir = makeTempDir();
|
||||
const snapshot: CharacterDictionarySnapshot = {
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
@@ -225,11 +208,10 @@ test('createCharacterDictionaryImageLookup does not fall back globally on scoped
|
||||
],
|
||||
images: [{ path: 'img/m115230-c1.png', dataBase64: 'TOWER' }],
|
||||
};
|
||||
await writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
writeSnapshot(getSnapshotPath(outputDir, snapshot.mediaId), snapshot);
|
||||
|
||||
const lookup = createCharacterDictionaryImageLookup({ outputDir });
|
||||
|
||||
const unscoped = await waitForRefresh(() => lookup.get('カズ'));
|
||||
assert.equal(unscoped.alt, 'Kaz');
|
||||
assert.equal(lookup.get('カズ', 21202), null);
|
||||
assert.equal(lookup.get('カズ')?.alt, 'Kaz');
|
||||
});
|
||||
|
||||
@@ -204,11 +204,11 @@ function getSnapshotDirectorySignature(outputDir: string): string {
|
||||
return parts.sort().join('|');
|
||||
}
|
||||
|
||||
export async function buildCharacterNameImageIndexFromSnapshots(
|
||||
export function buildCharacterNameImageIndexFromSnapshots(
|
||||
outputDir: string,
|
||||
): Promise<Map<string, CharacterNameImage>> {
|
||||
): Map<string, CharacterNameImage> {
|
||||
const index = new Map<string, CharacterNameImage>();
|
||||
for (const snapshot of await readCachedSnapshots(outputDir)) {
|
||||
for (const snapshot of readCachedSnapshots(outputDir)) {
|
||||
appendSnapshotImages(index, snapshot);
|
||||
}
|
||||
return index;
|
||||
@@ -228,12 +228,7 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
let signature: string | null = null;
|
||||
let index = new Map<string, CharacterNameImage>();
|
||||
let indexByMediaId = new Map<number, Map<string, CharacterNameImage>>();
|
||||
let refreshInFlight = false;
|
||||
|
||||
// Rebuilding means re-reading every cached snapshot (potentially GBs of JSON), which used to run
|
||||
// synchronously inside a lookup and froze the whole app right after a snapshot changed. Lookups
|
||||
// now serve the previous index while a single background rebuild catches up; the swap is atomic
|
||||
// and the signature only advances once the rebuild it belongs to has landed.
|
||||
function refreshIfNeeded(): void {
|
||||
if (!outputDir) {
|
||||
index = new Map<string, CharacterNameImage>();
|
||||
@@ -242,30 +237,20 @@ export function createCharacterDictionaryImageLookup(deps: {
|
||||
return;
|
||||
}
|
||||
const nextSignature = getSnapshotDirectorySignature(outputDir);
|
||||
if (nextSignature === signature || refreshInFlight) {
|
||||
if (nextSignature === signature) {
|
||||
return;
|
||||
}
|
||||
refreshInFlight = true;
|
||||
void (async () => {
|
||||
try {
|
||||
const snapshots = await readCachedSnapshots(outputDir);
|
||||
const nextIndex = new Map<string, CharacterNameImage>();
|
||||
const nextIndexByMediaId = new Map<number, Map<string, CharacterNameImage>>();
|
||||
for (const snapshot of snapshots) {
|
||||
appendSnapshotImages(nextIndex, snapshot);
|
||||
const mediaIndex = new Map<string, CharacterNameImage>();
|
||||
appendSnapshotImages(mediaIndex, snapshot);
|
||||
if (mediaIndex.size > 0) {
|
||||
nextIndexByMediaId.set(snapshot.mediaId, mediaIndex);
|
||||
}
|
||||
}
|
||||
index = nextIndex;
|
||||
indexByMediaId = nextIndexByMediaId;
|
||||
signature = nextSignature;
|
||||
} finally {
|
||||
refreshInFlight = false;
|
||||
signature = nextSignature;
|
||||
index = new Map<string, CharacterNameImage>();
|
||||
indexByMediaId = new Map<number, Map<string, CharacterNameImage>>();
|
||||
for (const snapshot of readCachedSnapshots(outputDir)) {
|
||||
appendSnapshotImages(index, snapshot);
|
||||
const mediaIndex = new Map<string, CharacterNameImage>();
|
||||
appendSnapshotImages(mediaIndex, snapshot);
|
||||
if (mediaIndex.size > 0) {
|
||||
indexByMediaId.set(snapshot.mediaId, mediaIndex);
|
||||
}
|
||||
})();
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
|
||||
@@ -32,33 +32,17 @@ function writeSnapshot(outputDir: string, mediaId: number, entries: Array<[strin
|
||||
);
|
||||
}
|
||||
|
||||
async function withTempDir<T>(run: (dir: string) => Promise<T> | T): Promise<T> {
|
||||
function withTempDir<T>(run: (dir: string) => T): T {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-name-candidates-'));
|
||||
try {
|
||||
return await run(dir);
|
||||
return run(dir);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
// The snapshot index rebuilds in the background while lookups serve stale data, so tests poll the
|
||||
// probe until the refresh they triggered has landed.
|
||||
async function waitForRefresh<T>(probe: () => T | null | undefined): Promise<T> {
|
||||
const deadline = Date.now() + 5000;
|
||||
for (;;) {
|
||||
const value = probe();
|
||||
if (value !== null && value !== undefined) {
|
||||
return value;
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
throw new Error('timed out waiting for background snapshot refresh');
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
}
|
||||
|
||||
test('collects terms and readings for the current media', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('collects terms and readings for the current media', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['湊', 'みなと'],
|
||||
@@ -69,16 +53,17 @@ test('collects terms and readings for the current media', async () => {
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const candidates = await waitForRefresh(() => lookup.get());
|
||||
const candidates = lookup.get();
|
||||
|
||||
assert.ok(candidates);
|
||||
assert.deepEqual([...candidates.forms].sort(), ['みなと', 'ミナト', '湊'].sort());
|
||||
// Deduplicated: both entries share the みなと reading.
|
||||
assert.equal(candidates.forms.length, 3);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null without a media scope so the scanner stays exhaustive', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('returns null without a media scope so the scanner stays exhaustive', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
@@ -86,14 +71,12 @@ test('returns null without a media scope so the scanner stays exhaustive', async
|
||||
getCurrentMediaId: () => null,
|
||||
});
|
||||
|
||||
// The explicitly-scoped probe proves the index has loaded before the unscoped case is judged.
|
||||
await waitForRefresh(() => lookup.get(1));
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null for a media with no cached snapshot', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('returns null for a media with no cached snapshot', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
@@ -101,31 +84,29 @@ test('returns null for a media with no cached snapshot', async () => {
|
||||
getCurrentMediaId: () => 999,
|
||||
});
|
||||
|
||||
await waitForRefresh(() => lookup.get(1));
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('key changes when the snapshot content changes', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('key changes when the snapshot content changes', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const first = await waitForRefresh(() => lookup.get());
|
||||
const first = lookup.get();
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
lookup.invalidate();
|
||||
const second = await waitForRefresh(() => {
|
||||
const candidates = lookup.get();
|
||||
return candidates && candidates.forms.length === 4 ? candidates : null;
|
||||
});
|
||||
const second = lookup.get();
|
||||
|
||||
assert.ok(first && second);
|
||||
assert.notEqual(first.key, second.key);
|
||||
assert.equal(second.forms.length, 4);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -133,8 +114,8 @@ test('key changes when the snapshot content changes', async () => {
|
||||
// directory every call. Asserted behaviorally: an unannounced on-disk change is
|
||||
// invisible until the recheck interval elapses, which can only be true if the
|
||||
// filesystem is not consulted per lookup.
|
||||
test('does not re-read the snapshot directory on every lookup', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('does not re-read the snapshot directory on every lookup', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
@@ -143,7 +124,6 @@ test('does not re-read the snapshot directory on every lookup', async () => {
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
await waitForRefresh(() => lookup.get());
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
@@ -155,16 +135,12 @@ test('does not re-read the snapshot directory on every lookup', async () => {
|
||||
assert.equal(lookup.get()?.forms.length, 2, 'expected the cached list within the interval');
|
||||
|
||||
nowMs += 10_000;
|
||||
const refreshed = await waitForRefresh(() => {
|
||||
const candidates = lookup.get();
|
||||
return candidates && candidates.forms.length === 4 ? candidates : null;
|
||||
});
|
||||
assert.equal(refreshed.forms.length, 4, 'expected a refresh past the interval');
|
||||
assert.equal(lookup.get()?.forms.length, 4, 'expected a refresh past the interval');
|
||||
});
|
||||
});
|
||||
|
||||
test('invalidate picks up a snapshot change on the next refresh', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
test('invalidate picks up a snapshot change immediately', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
@@ -173,7 +149,6 @@ test('invalidate picks up a snapshot change on the next refresh', async () => {
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
await waitForRefresh(() => lookup.get());
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
@@ -183,10 +158,6 @@ test('invalidate picks up a snapshot change on the next refresh', async () => {
|
||||
nowMs += 1;
|
||||
lookup.invalidate();
|
||||
|
||||
const refreshed = await waitForRefresh(() => {
|
||||
const candidates = lookup.get();
|
||||
return candidates && candidates.forms.length === 4 ? candidates : null;
|
||||
});
|
||||
assert.equal(refreshed.forms.length, 4);
|
||||
assert.equal(lookup.get()?.forms.length, 4);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -98,12 +98,7 @@ export function createCharacterNameCandidateLookup(deps: {
|
||||
let signature: string | null = null;
|
||||
let lastSignatureCheckAtMs = 0;
|
||||
let formsByMediaId = new Map<number, string[]>();
|
||||
let refreshInFlight = false;
|
||||
|
||||
// Same stale-while-revalidate shape as the image lookup: the rebuild re-reads every cached
|
||||
// snapshot, so it runs in the background while lookups keep serving the previous forms. The
|
||||
// signature only advances once its rebuild has landed, so a failed or superseded rebuild is
|
||||
// retried on the next signature check.
|
||||
function refreshIfNeeded(): void {
|
||||
if (!outputDir) {
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
@@ -119,26 +114,17 @@ export function createCharacterNameCandidateLookup(deps: {
|
||||
}
|
||||
lastSignatureCheckAtMs = nowMs;
|
||||
const nextSignature = getSnapshotDirectorySignature(outputDir);
|
||||
if (nextSignature === signature || refreshInFlight) {
|
||||
if (nextSignature === signature) {
|
||||
return;
|
||||
}
|
||||
refreshInFlight = true;
|
||||
void (async () => {
|
||||
try {
|
||||
const snapshots = await readCachedSnapshots(outputDir);
|
||||
const nextFormsByMediaId = new Map<number, string[]>();
|
||||
for (const snapshot of snapshots) {
|
||||
const forms = collectSnapshotNameForms(snapshot);
|
||||
if (forms.length > 0) {
|
||||
nextFormsByMediaId.set(snapshot.mediaId, forms);
|
||||
}
|
||||
}
|
||||
formsByMediaId = nextFormsByMediaId;
|
||||
signature = nextSignature;
|
||||
} finally {
|
||||
refreshInFlight = false;
|
||||
signature = nextSignature;
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
for (const snapshot of readCachedSnapshots(outputDir)) {
|
||||
const forms = collectSnapshotNameForms(snapshot);
|
||||
if (forms.length > 0) {
|
||||
formsByMediaId.set(snapshot.mediaId, forms);
|
||||
}
|
||||
})();
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
|
||||
@@ -34,7 +34,7 @@ function createSnapshotWithoutImages(): CharacterDictionarySnapshot {
|
||||
test('generateForCurrentMedia refreshes same-version snapshots missing images when inline images are enabled', async () => {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), createSnapshotWithoutImages());
|
||||
writeSnapshot(getSnapshotPath(outputDir, 130298), createSnapshotWithoutImages());
|
||||
const originalFetch = globalThis.fetch;
|
||||
const fetchUrls: string[] = [];
|
||||
|
||||
@@ -124,7 +124,7 @@ test('generateForCurrentMedia refreshes same-version snapshots missing images wh
|
||||
test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable', async () => {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
...createSnapshotWithoutImages(),
|
||||
nameSplitSource: 'heuristic',
|
||||
});
|
||||
@@ -213,7 +213,7 @@ test('generateForCurrentMedia keeps failed MeCab name split refreshes retryable'
|
||||
test('generateForCurrentMedia keeps mecab-split snapshots when MeCab is available', async () => {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
...createSnapshotWithoutImages(),
|
||||
nameSplitSource: 'mecab',
|
||||
});
|
||||
@@ -253,7 +253,7 @@ test('generateForCurrentMedia keeps mecab-split snapshots when MeCab is availabl
|
||||
test('generateForCurrentMedia keeps heuristic-split snapshots while MeCab is unavailable', async () => {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
writeSnapshot(getSnapshotPath(outputDir, 130298), {
|
||||
...createSnapshotWithoutImages(),
|
||||
nameSplitSource: 'heuristic',
|
||||
});
|
||||
@@ -293,7 +293,7 @@ test('generateForCurrentMedia keeps heuristic-split snapshots while MeCab is una
|
||||
test('generateForCurrentMedia keeps same-version snapshots without images when inline images are disabled', async () => {
|
||||
const userDataPath = makeTempDir();
|
||||
const outputDir = path.join(userDataPath, 'character-dictionaries');
|
||||
await writeSnapshot(getSnapshotPath(outputDir, 130298), createSnapshotWithoutImages());
|
||||
writeSnapshot(getSnapshotPath(outputDir, 130298), createSnapshotWithoutImages());
|
||||
const originalFetch = globalThis.fetch;
|
||||
|
||||
globalThis.fetch = (async (input: string | URL | Request) => {
|
||||
|
||||
@@ -42,7 +42,7 @@ function readStoredZipEntries(zipPath: string): Map<string, Buffer> {
|
||||
return entries;
|
||||
}
|
||||
|
||||
test('buildDictionaryZip writes a valid stored zip without fs.writeFileSync', async () => {
|
||||
test('buildDictionaryZip writes a valid stored zip without fs.writeFileSync', () => {
|
||||
const tempDir = makeTempDir();
|
||||
const outputPath = path.join(tempDir, 'dictionary.zip');
|
||||
const termEntries: CharacterDictionaryTermEntry[] = [
|
||||
@@ -62,7 +62,7 @@ test('buildDictionaryZip writes a valid stored zip without fs.writeFileSync', as
|
||||
);
|
||||
}) as typeof Buffer.concat;
|
||||
|
||||
const result = await buildDictionaryZip(
|
||||
const result = buildDictionaryZip(
|
||||
outputPath,
|
||||
'Dictionary Title',
|
||||
'Dictionary Description',
|
||||
@@ -106,11 +106,11 @@ test('buildDictionaryZip writes a valid stored zip without fs.writeFileSync', as
|
||||
}
|
||||
});
|
||||
|
||||
test('readDictionaryZipRevision reads the built revision and rejects foreign archives', async () => {
|
||||
test('readDictionaryZipRevision reads the built revision and rejects foreign archives', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const zipPath = path.join(dir, 'merged.zip');
|
||||
await buildDictionaryZip(
|
||||
buildDictionaryZip(
|
||||
zipPath,
|
||||
'SubMiner Character Dictionary',
|
||||
'Character names',
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import * as path from 'path';
|
||||
import { readStoredZipFirstFile, writeStoredZipAsync } from '../../shared/stored-zip';
|
||||
import { readStoredZipFirstFile, writeStoredZip } from '../../shared/stored-zip';
|
||||
import { ensureDir } from './fs-utils';
|
||||
import type { CharacterDictionarySnapshotImage, CharacterDictionaryTermEntry } from './types';
|
||||
|
||||
@@ -48,14 +48,14 @@ export function readDictionaryZipRevision(zipPath: string): string | null {
|
||||
}
|
||||
}
|
||||
|
||||
export async function buildDictionaryZip(
|
||||
export function buildDictionaryZip(
|
||||
outputPath: string,
|
||||
dictionaryTitle: string,
|
||||
description: string,
|
||||
revision: string,
|
||||
termEntries: CharacterDictionaryTermEntry[],
|
||||
images: CharacterDictionarySnapshotImage[],
|
||||
): Promise<{ zipPath: string; entryCount: number }> {
|
||||
): { zipPath: string; entryCount: number } {
|
||||
ensureDir(path.dirname(outputPath));
|
||||
|
||||
function* zipFiles(): Iterable<{ name: string; data: Buffer }> {
|
||||
@@ -78,11 +78,7 @@ export async function buildDictionaryZip(
|
||||
};
|
||||
}
|
||||
|
||||
// Each bank is stringified in one shot, so the bank size sets the longest single block in the
|
||||
// build. 10k entries measured ~38MB and ~135ms per bank on a real merged dictionary; 2k keeps
|
||||
// every bank under the archive writer's yield budget at ~27ms. Yomitan reads any number of
|
||||
// term_bank_N.json files, so this only changes how the terms are split across them.
|
||||
const entriesPerBank = 2_000;
|
||||
const entriesPerBank = 10_000;
|
||||
for (let i = 0; i < termEntries.length; i += entriesPerBank) {
|
||||
yield {
|
||||
name: `term_bank_${Math.floor(i / entriesPerBank) + 1}.json`,
|
||||
@@ -91,6 +87,6 @@ export async function buildDictionaryZip(
|
||||
}
|
||||
}
|
||||
|
||||
await writeStoredZipAsync(outputPath, zipFiles());
|
||||
writeStoredZip(outputPath, zipFiles());
|
||||
return { zipPath: outputPath, entryCount: termEntries.length };
|
||||
}
|
||||
|
||||
@@ -64,11 +64,17 @@ test('anki action main deps builders map callbacks', async () => {
|
||||
const mine = createBuildMineSentenceCardMainDepsHandler({
|
||||
getAnkiIntegration: () => ({ enabled: true }),
|
||||
getMpvClient: () => ({ connected: true }),
|
||||
getPrimarySubtitle: () => ({ text: '正式な字幕', startTime: 1, endTime: 3 }),
|
||||
showMpvOsd: (text) => calls.push(`mine:${text}`),
|
||||
mineSentenceCardCore: async () => true,
|
||||
recordCardsMined: (count) => calls.push(`cards:${count}`),
|
||||
})();
|
||||
assert.deepEqual(mine.getMpvClient(), { connected: true });
|
||||
assert.deepEqual(mine.getPrimarySubtitle?.(), {
|
||||
text: '正式な字幕',
|
||||
startTime: 1,
|
||||
endTime: 3,
|
||||
});
|
||||
mine.showMpvOsd('m');
|
||||
await mine.mineSentenceCardCore({
|
||||
ankiIntegration: { enabled: true },
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { createRefreshKnownWordCacheHandler } from './anki-actions';
|
||||
import type { createRefreshKnownWordCacheHandler, PrimarySubtitle } from './anki-actions';
|
||||
|
||||
type RefreshKnownWordCacheMainDeps = Parameters<typeof createRefreshKnownWordCacheHandler>[0];
|
||||
|
||||
@@ -72,10 +72,12 @@ export function createBuildMarkLastCardAsAudioCardMainDepsHandler<TAnki>(deps: {
|
||||
export function createBuildMineSentenceCardMainDepsHandler<TAnki, TMpv>(deps: {
|
||||
getAnkiIntegration: () => TAnki;
|
||||
getMpvClient: () => TMpv;
|
||||
getPrimarySubtitle?: () => PrimarySubtitle | null;
|
||||
showMpvOsd: (text: string) => void;
|
||||
mineSentenceCardCore: (options: {
|
||||
ankiIntegration: TAnki;
|
||||
mpvClient: TMpv;
|
||||
primarySubtitle?: PrimarySubtitle;
|
||||
showMpvOsd: (text: string) => void;
|
||||
}) => Promise<boolean>;
|
||||
recordCardsMined: (count: number, noteIds?: number[]) => void;
|
||||
@@ -83,10 +85,14 @@ export function createBuildMineSentenceCardMainDepsHandler<TAnki, TMpv>(deps: {
|
||||
return () => ({
|
||||
getAnkiIntegration: () => deps.getAnkiIntegration(),
|
||||
getMpvClient: () => deps.getMpvClient(),
|
||||
...(deps.getPrimarySubtitle
|
||||
? { getPrimarySubtitle: () => deps.getPrimarySubtitle?.() ?? null }
|
||||
: {}),
|
||||
showMpvOsd: (text: string) => deps.showMpvOsd(text),
|
||||
mineSentenceCardCore: (options: {
|
||||
ankiIntegration: TAnki;
|
||||
mpvClient: TMpv;
|
||||
primarySubtitle?: PrimarySubtitle;
|
||||
showMpvOsd: (text: string) => void;
|
||||
}) => deps.mineSentenceCardCore(options),
|
||||
recordCardsMined: (count: number, noteIds?: number[]) => deps.recordCardsMined(count, noteIds),
|
||||
|
||||
@@ -87,3 +87,20 @@ test('mine sentence handler records mined cards only when core returns true', as
|
||||
await mineSentenceCard();
|
||||
assert.deepEqual(calls, ['osd:mine', 'osd:mine', 'cards:1']);
|
||||
});
|
||||
|
||||
test('mine sentence handler forwards the canonical primary subtitle snapshot', async () => {
|
||||
const primarySubtitle = { text: '正式な字幕', startTime: 1, endTime: 3 };
|
||||
const mineSentenceCard = createMineSentenceCardHandler({
|
||||
getAnkiIntegration: () => ({}),
|
||||
getMpvClient: () => ({}),
|
||||
getPrimarySubtitle: () => primarySubtitle,
|
||||
showMpvOsd: () => {},
|
||||
mineSentenceCardCore: async (options) => {
|
||||
assert.equal(options.primarySubtitle, primarySubtitle);
|
||||
return true;
|
||||
},
|
||||
recordCardsMined: () => {},
|
||||
});
|
||||
|
||||
await mineSentenceCard();
|
||||
});
|
||||
|
||||
@@ -2,6 +2,12 @@ type AnkiIntegrationLike = {
|
||||
refreshKnownWordCache: () => Promise<void>;
|
||||
};
|
||||
|
||||
export type PrimarySubtitle = {
|
||||
text: string;
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
};
|
||||
|
||||
export function createUpdateLastCardFromClipboardHandler<TAnki>(deps: {
|
||||
getAnkiIntegration: () => TAnki;
|
||||
readClipboardText: () => string;
|
||||
@@ -69,18 +75,22 @@ export function createMarkLastCardAsAudioCardHandler<TAnki>(deps: {
|
||||
export function createMineSentenceCardHandler<TAnki, TMpv>(deps: {
|
||||
getAnkiIntegration: () => TAnki;
|
||||
getMpvClient: () => TMpv;
|
||||
getPrimarySubtitle?: () => PrimarySubtitle | null;
|
||||
showMpvOsd: (text: string) => void;
|
||||
mineSentenceCardCore: (options: {
|
||||
ankiIntegration: TAnki;
|
||||
mpvClient: TMpv;
|
||||
primarySubtitle?: PrimarySubtitle;
|
||||
showMpvOsd: (text: string) => void;
|
||||
}) => Promise<boolean>;
|
||||
recordCardsMined: (count: number, noteIds?: number[]) => void;
|
||||
}) {
|
||||
return async (): Promise<void> => {
|
||||
const primarySubtitle = deps.getPrimarySubtitle?.();
|
||||
const created = await deps.mineSentenceCardCore({
|
||||
ankiIntegration: deps.getAnkiIntegration(),
|
||||
mpvClient: deps.getMpvClient(),
|
||||
...(primarySubtitle ? { primarySubtitle } : {}),
|
||||
showMpvOsd: deps.showMpvOsd,
|
||||
});
|
||||
if (created) {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { SubtitleCue, SubtitleData } from '../../types';
|
||||
import { selectAutoplayStartupCue } from './autoplay-subtitle-primer';
|
||||
import { primeVisibleOverlaySubtitleFromMpv } from './current-subtitle-snapshot';
|
||||
import { resolvePrimarySubtitleText } from './primary-subtitle-text';
|
||||
import { resolveSubtitleSourcePath } from './subtitle-prefetch-source';
|
||||
|
||||
const AUTOPLAY_SUBTITLE_PRIME_LOOKAHEAD_SECONDS = 2;
|
||||
@@ -141,6 +142,16 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
return true;
|
||||
}
|
||||
|
||||
function resolveLivePrimarySubtitleText(text: string): string {
|
||||
const client = deps.getMpvClient();
|
||||
const currentTimeSec = Number(client?.currentTimePos ?? deps.getLastObservedTimePos());
|
||||
return resolvePrimarySubtitleText({
|
||||
liveText: text,
|
||||
currentTimeSec,
|
||||
cues: deps.getActiveParsedSubtitleCues(),
|
||||
});
|
||||
}
|
||||
|
||||
async function primeCurrentSubtitleForAutoplay(mediaPath: string): Promise<void> {
|
||||
const client = deps.getMpvClient();
|
||||
if (!client?.connected || !isCurrentAutoplayMediaPath(mediaPath)) {
|
||||
@@ -155,7 +166,8 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
);
|
||||
return null;
|
||||
});
|
||||
const text = typeof subTextRaw === 'string' ? subTextRaw : '';
|
||||
const liveText = typeof subTextRaw === 'string' ? subTextRaw : '';
|
||||
const text = resolveLivePrimarySubtitleText(liveText);
|
||||
if (emitAutoplayPrimedSubtitle(mediaPath, text)) {
|
||||
return;
|
||||
}
|
||||
@@ -175,6 +187,7 @@ export function createAutoplaySubtitlePrimingRuntime(deps: AutoplaySubtitlePrimi
|
||||
async function primeCurrentSubtitleForVisibleOverlay(): Promise<void> {
|
||||
await primeVisibleOverlaySubtitleFromMpv({
|
||||
getMpvClient: () => deps.getMpvClient(),
|
||||
resolvePrimarySubtitleText: (text) => resolveLivePrimarySubtitleText(text),
|
||||
setCurrentSubText: (text) => {
|
||||
deps.setCurrentSubText(text);
|
||||
},
|
||||
|
||||
@@ -46,6 +46,7 @@ export async function resolveCurrentSubtitleForRenderer(deps: {
|
||||
export async function primeVisibleOverlaySubtitleFromMpv(deps: {
|
||||
getMpvClient: () => CurrentSubtitleMpvClient | null;
|
||||
setCurrentSubText: (text: string) => void;
|
||||
resolvePrimarySubtitleText?: (text: string) => string;
|
||||
getCurrentSubtitleData: () => SubtitleData | null;
|
||||
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
||||
onSubtitleChange: (text: string) => void;
|
||||
@@ -73,7 +74,8 @@ export async function primeVisibleOverlaySubtitleFromMpv(deps: {
|
||||
return;
|
||||
}
|
||||
|
||||
const text = typeof subTextRaw === 'string' ? subTextRaw : '';
|
||||
const liveText = typeof subTextRaw === 'string' ? subTextRaw : '';
|
||||
const text = deps.resolvePrimarySubtitleText?.(liveText) ?? liveText;
|
||||
deps.setCurrentSubText(text);
|
||||
|
||||
const primeSecondarySubtitle = async (): Promise<void> => {
|
||||
|
||||
@@ -26,6 +26,29 @@ test('subtitle change handler updates state and forwards uncached text without r
|
||||
assert.deepEqual(calls, ['set:line', 'process:line', 'presence']);
|
||||
});
|
||||
|
||||
test('subtitle change handler consistently forwards resolved canonical text', () => {
|
||||
const calls: string[] = [];
|
||||
const handler = createHandleMpvSubtitleChangeHandler({
|
||||
resolveSubtitleText: () => '今 手にある物差しでは',
|
||||
setCurrentSubText: (text) => calls.push(`set:${text}`),
|
||||
getImmediateSubtitlePayload: (text) => {
|
||||
calls.push(`lookup:${text}`);
|
||||
return null;
|
||||
},
|
||||
broadcastSubtitle: () => {},
|
||||
onSubtitleChange: (text) => calls.push(`process:${text}`),
|
||||
refreshDiscordPresence: () => {},
|
||||
});
|
||||
|
||||
handler({ text: '今今今手手手ににに' });
|
||||
|
||||
assert.deepEqual(calls, [
|
||||
'set:今 手にある物差しでは',
|
||||
'lookup:今 手にある物差しでは',
|
||||
'process:今 手にある物差しでは',
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle change handler clears immediately for empty subtitle text', () => {
|
||||
const calls: string[] = [];
|
||||
const handler = createHandleMpvSubtitleChangeHandler({
|
||||
|
||||
@@ -4,7 +4,8 @@ type AnilistPostWatchRunOptions = {
|
||||
watchedSeconds?: number;
|
||||
};
|
||||
|
||||
const SEEK_LIKE_TIME_DELTA_SECONDS = 2.5;
|
||||
/** Jump size that marks a time-pos change as a seek rather than normal playback. */
|
||||
export const SEEK_LIKE_TIME_DELTA_SECONDS = 2.5;
|
||||
|
||||
function isSeekLikeTimeChange(previousTime: number | null, nextTime: number): boolean {
|
||||
if (previousTime === null || !Number.isFinite(previousTime) || !Number.isFinite(nextTime)) {
|
||||
@@ -14,6 +15,7 @@ function isSeekLikeTimeChange(previousTime: number | null, nextTime: number): bo
|
||||
}
|
||||
|
||||
export function createHandleMpvSubtitleChangeHandler(deps: {
|
||||
resolveSubtitleText?: (text: string) => string;
|
||||
setCurrentSubText: (text: string) => void;
|
||||
getImmediateSubtitlePayload?: (text: string) => SubtitleData | null;
|
||||
emitImmediateSubtitle?: (payload: SubtitleData) => void;
|
||||
@@ -22,7 +24,8 @@ export function createHandleMpvSubtitleChangeHandler(deps: {
|
||||
refreshDiscordPresence: () => void;
|
||||
logDebug?: (message: string) => void;
|
||||
}) {
|
||||
return ({ text }: { text: string }): void => {
|
||||
return ({ text: liveText }: { text: string }): void => {
|
||||
const text = deps.resolveSubtitleText?.(liveText) ?? liveText;
|
||||
deps.setCurrentSubText(text);
|
||||
const immediatePayload = deps.getImmediateSubtitlePayload?.(text) ?? null;
|
||||
if (immediatePayload) {
|
||||
|
||||
@@ -43,6 +43,7 @@ export function createBindMpvMainEventHandlersHandler(deps: {
|
||||
logSubtitleTimingError: (message: string, error: unknown) => void;
|
||||
|
||||
setCurrentSubText: (text: string) => void;
|
||||
resolveSubtitleText?: (text: string) => string;
|
||||
getImmediateSubtitlePayload?: (text: string) => SubtitleData | null;
|
||||
emitImmediateSubtitle?: (payload: SubtitleData) => void;
|
||||
broadcastSubtitle: (payload: SubtitleData) => void;
|
||||
@@ -117,6 +118,7 @@ export function createBindMpvMainEventHandlersHandler(deps: {
|
||||
logError: (message, error) => deps.logSubtitleTimingError(message, error),
|
||||
});
|
||||
const handleMpvSubtitleChange = createHandleMpvSubtitleChangeHandler({
|
||||
resolveSubtitleText: deps.resolveSubtitleText,
|
||||
setCurrentSubText: (text) => deps.setCurrentSubText(text),
|
||||
getImmediateSubtitlePayload: (text) => deps.getImmediateSubtitlePayload?.(text) ?? null,
|
||||
emitImmediateSubtitle: deps.emitImmediateSubtitle
|
||||
|
||||
@@ -387,3 +387,170 @@ test('subtitle-track transitions ignore stale parsed cues until replacement cues
|
||||
handlers.recordImmersionSubtitleLine('飛び上がる', 20.04, 20.08);
|
||||
assert.deepEqual(recordedStarts.slice(-1), [20]);
|
||||
});
|
||||
|
||||
test('canonical ASS cues replace live glyph spam for display, history, and immersion', () => {
|
||||
const immersion: Array<{ text: string; start: number; end: number }> = [];
|
||||
const timing: Array<{ text: string; start: number; end: number }> = [];
|
||||
const handlers = createBuildBindMpvMainEventHandlersMainDepsHandler({
|
||||
appState: {
|
||||
initialArgs: null,
|
||||
overlayRuntimeInitialized: true,
|
||||
mpvClient: { currentTimePos: 2 },
|
||||
immersionTracker: {
|
||||
recordSubtitleLine: (text: string, start: number, end: number) =>
|
||||
immersion.push({ text, start, end }),
|
||||
},
|
||||
subtitleTimingTracker: {
|
||||
recordSubtitle: (text: string, start: number, end: number) =>
|
||||
timing.push({ text, start, end }),
|
||||
},
|
||||
activeParsedSubtitleCues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある物差しでは',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
{
|
||||
startTime: 3,
|
||||
endTime: 6,
|
||||
text: '飛び越えてみたくて',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
currentMediaPath: '/video.mkv',
|
||||
currentSubText: '',
|
||||
currentSubAssText: '',
|
||||
playbackPaused: null,
|
||||
previousSecondarySubVisibility: false,
|
||||
},
|
||||
getQuitOnDisconnectArmed: () => false,
|
||||
scheduleQuitCheck: () => {},
|
||||
quitApp: () => {},
|
||||
reportJellyfinRemoteStopped: () => {},
|
||||
syncOverlayMpvSubtitleSuppression: () => {},
|
||||
maybeRunAnilistPostWatchUpdate: async () => {},
|
||||
logSubtitleTimingError: () => {},
|
||||
broadcastToOverlayWindows: () => {},
|
||||
onSubtitleChange: () => {},
|
||||
ensureImmersionTrackerInitialized: () => {},
|
||||
updateCurrentMediaPath: () => {},
|
||||
restoreMpvSubVisibility: () => {},
|
||||
getCurrentAnilistMediaKey: () => null,
|
||||
resetAnilistMediaTracking: () => {},
|
||||
maybeProbeAnilistDuration: () => {},
|
||||
ensureAnilistMediaGuess: () => {},
|
||||
syncImmersionMediaState: () => {},
|
||||
updateCurrentMediaTitle: () => {},
|
||||
resetAnilistMediaGuessState: () => {},
|
||||
reportJellyfinRemoteProgress: () => {},
|
||||
updateSubtitleRenderMetrics: () => {},
|
||||
refreshDiscordPresence: () => {},
|
||||
})();
|
||||
|
||||
assert.equal(handlers.resolveSubtitleText?.('今\n今\n今\n手\n手\n手'), '今 手にある物差しでは');
|
||||
handlers.recordImmersionSubtitleLine('今', 0.8, 1.5);
|
||||
handlers.recordImmersionSubtitleLine('手', 0.86, 1.56);
|
||||
handlers.recordSubtitleTiming('今', 0.8, 1.5);
|
||||
|
||||
assert.deepEqual(immersion, [{ text: '今 手にある物差しでは', start: 1.2, end: 3.8 }]);
|
||||
assert.deepEqual(timing, [{ text: '今 手にある物差しでは', start: 1.2, end: 3.8 }]);
|
||||
|
||||
// Concurrent dialogue during the song is not part of the animation: it must be
|
||||
// recorded as itself -- without the fragment lines beside it -- and must not cause
|
||||
// the song line to be recorded again when the animation frames resume.
|
||||
assert.equal(handlers.resolveSubtitleText?.('普通のセリフ\n今\n手'), '普通のセリフ\n今\n手');
|
||||
handlers.recordImmersionSubtitleLine('普通のセリフ\n今\n手', 1.9, 3.2);
|
||||
handlers.recordImmersionSubtitleLine('にある', 2.1, 2.9);
|
||||
handlers.recordSubtitleTiming('次のセリフ', 3.9, 5.0);
|
||||
|
||||
assert.deepEqual(immersion.slice(1), [{ text: '普通のセリフ', start: 1.9, end: 3.2 }]);
|
||||
assert.deepEqual(timing.slice(1), [{ text: '次のセリフ', start: 3.9, end: 5 }]);
|
||||
|
||||
// Overlapping canonical lines resolve as shifting subsets (A, then A+B, then A).
|
||||
// Every recorded cue is remembered, so each authored line still records exactly once.
|
||||
handlers.recordImmersionSubtitleLine('飛び越えて', 3.2, 3.4);
|
||||
handlers.recordImmersionSubtitleLine('手にある', 3.5, 3.7);
|
||||
handlers.recordSubtitleTiming('飛び越えて', 3.2, 3.4);
|
||||
handlers.recordSubtitleTiming('手にある', 3.5, 3.7);
|
||||
|
||||
assert.deepEqual(immersion.slice(2), [{ text: '飛び越えてみたくて', start: 3, end: 6 }]);
|
||||
assert.deepEqual(timing.slice(2), [{ text: '飛び越えてみたくて', start: 3, end: 6 }]);
|
||||
|
||||
// A backward seek means the user is rewatching: the timing history (a viewing log)
|
||||
// records the revisited line again, while immersion stays once-per-media.
|
||||
handlers.onTimePosUpdate?.(30);
|
||||
handlers.onTimePosUpdate?.(2);
|
||||
handlers.recordSubtitleTiming('今', 0.8, 1.5);
|
||||
handlers.recordImmersionSubtitleLine('今', 0.8, 1.5);
|
||||
|
||||
assert.deepEqual(timing.slice(3), [{ text: '今 手にある物差しでは', start: 1.2, end: 3.8 }]);
|
||||
assert.equal(immersion.length, 3);
|
||||
|
||||
// A jump of exactly the seek threshold counts as a seek, matching the time-pos
|
||||
// handler's own `>=` boundary.
|
||||
handlers.onTimePosUpdate?.(4.5);
|
||||
handlers.onTimePosUpdate?.(2);
|
||||
handlers.recordSubtitleTiming('今', 0.8, 1.5);
|
||||
|
||||
assert.deepEqual(timing.slice(4), [{ text: '今 手にある物差しでは', start: 1.2, end: 3.8 }]);
|
||||
});
|
||||
|
||||
test('subtitle-track changes stop stale canonical cues from substituting immediately', () => {
|
||||
const appState = {
|
||||
initialArgs: null,
|
||||
overlayRuntimeInitialized: true,
|
||||
mpvClient: { currentTimePos: 2 },
|
||||
immersionTracker: { recordSubtitleLine: () => {} },
|
||||
subtitleTimingTracker: { recordSubtitle: () => {} },
|
||||
activeParsedSubtitleCues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある物差しでは',
|
||||
source: 'canonical-ass' as const,
|
||||
},
|
||||
] as Array<{ startTime: number; endTime: number; text: string; source?: 'canonical-ass' }>,
|
||||
activeParsedSubtitleSource: 'track-a.ass' as string | null,
|
||||
currentMediaPath: '/video.mkv',
|
||||
currentSubText: '',
|
||||
currentSubAssText: '',
|
||||
playbackPaused: null,
|
||||
previousSecondarySubVisibility: false,
|
||||
};
|
||||
const handlers = createBuildBindMpvMainEventHandlersMainDepsHandler({
|
||||
appState,
|
||||
getQuitOnDisconnectArmed: () => false,
|
||||
scheduleQuitCheck: () => {},
|
||||
quitApp: () => {},
|
||||
reportJellyfinRemoteStopped: () => {},
|
||||
syncOverlayMpvSubtitleSuppression: () => {},
|
||||
maybeRunAnilistPostWatchUpdate: async () => {},
|
||||
logSubtitleTimingError: () => {},
|
||||
broadcastToOverlayWindows: () => {},
|
||||
onSubtitleChange: () => {},
|
||||
ensureImmersionTrackerInitialized: () => {},
|
||||
updateCurrentMediaPath: () => {},
|
||||
restoreMpvSubVisibility: () => {},
|
||||
getCurrentAnilistMediaKey: () => null,
|
||||
resetAnilistMediaTracking: () => {},
|
||||
maybeProbeAnilistDuration: () => {},
|
||||
ensureAnilistMediaGuess: () => {},
|
||||
syncImmersionMediaState: () => {},
|
||||
updateCurrentMediaTitle: () => {},
|
||||
resetAnilistMediaGuessState: () => {},
|
||||
reportJellyfinRemoteProgress: () => {},
|
||||
updateSubtitleRenderMetrics: () => {},
|
||||
refreshDiscordPresence: () => {},
|
||||
})();
|
||||
|
||||
assert.equal(handlers.resolveSubtitleText?.('今\n手にある'), '今 手にある物差しでは');
|
||||
|
||||
// The new track's cues arrive only after an async re-parse; until then, the old
|
||||
// track's canonical lyric must not replace the new track's live text.
|
||||
handlers.onSubtitleTrackChange?.(2);
|
||||
|
||||
assert.deepEqual(appState.activeParsedSubtitleCues, []);
|
||||
assert.equal(appState.activeParsedSubtitleSource, null);
|
||||
assert.equal(handlers.resolveSubtitleText?.('今\n手にある'), '今\n手にある');
|
||||
});
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
import { createSubtitleLineDedupGate } from '../../core/services/subtitle-line-dedup-gate';
|
||||
import type { MergedToken, SubtitleCue, SubtitleData } from '../../types';
|
||||
import { SEEK_LIKE_TIME_DELTA_SECONDS } from './mpv-main-event-actions';
|
||||
import {
|
||||
resolveCanonicalPrimarySubtitle,
|
||||
resolvePrimarySubtitleText,
|
||||
stripCanonicalFragmentLines,
|
||||
} from './primary-subtitle-text';
|
||||
|
||||
type AnilistPostWatchRunOptions = {
|
||||
watchedSeconds?: number;
|
||||
@@ -36,6 +42,8 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
recordSubtitle?: (text: string, start: number, end: number, secondaryText?: string) => void;
|
||||
} | null;
|
||||
activeParsedSubtitleCues?: SubtitleCue[] | null;
|
||||
/** Cache key of the source the cues were parsed from; cleared with the cues. */
|
||||
activeParsedSubtitleSource?: string | null;
|
||||
currentMediaPath?: string | null;
|
||||
currentSubText: string;
|
||||
currentSubAssText: string;
|
||||
@@ -93,6 +101,38 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
const immersionLineDedupGate = createSubtitleLineDedupGate({
|
||||
getParsedCues: () => deps.appState.activeParsedSubtitleCues,
|
||||
});
|
||||
// One seen-set per consumer: canonical cues overlap, so live samples resolve to
|
||||
// shifting subsets (A, then A+B, then B). Remembering every recorded cue -- not just
|
||||
// the previous sample -- keeps each authored line recorded exactly once per source.
|
||||
const recordedImmersionCanonicalKeys = new Set<string>();
|
||||
const recordedTimingCanonicalKeys = new Set<string>();
|
||||
// Bumped on track/media changes so an immersion record whose tokenization resolves
|
||||
// after the change is dropped instead of landing in the next session.
|
||||
let subtitleSessionEpoch = 0;
|
||||
let lastTimePosForTimingReset: number | null = null;
|
||||
const canonicalCueKey = (cue: SubtitleCue): string =>
|
||||
`${cue.startTime}|${cue.endTime}|${cue.text}`;
|
||||
const resetSubtitleDeduplication = (): void => {
|
||||
immersionLineDedupGate.reset();
|
||||
recordedImmersionCanonicalKeys.clear();
|
||||
recordedTimingCanonicalKeys.clear();
|
||||
subtitleSessionEpoch += 1;
|
||||
lastTimePosForTimingReset = null;
|
||||
};
|
||||
const resolveCanonicalSample = (liveText: string, startSec: number) =>
|
||||
resolveCanonicalPrimarySubtitle({
|
||||
liveText,
|
||||
currentTimeSec: startSec,
|
||||
cues: deps.appState.activeParsedSubtitleCues,
|
||||
});
|
||||
// When substitution declined because dialogue shares the screen with a song, record
|
||||
// the dialogue alone rather than the combined dialogue-plus-fragments stack.
|
||||
const stripFragmentsForRecording = (liveText: string, startSec: number) =>
|
||||
stripCanonicalFragmentLines({
|
||||
liveText,
|
||||
currentTimeSec: startSec,
|
||||
cues: deps.appState.activeParsedSubtitleCues,
|
||||
});
|
||||
const hasInitialPlaybackQuitOnDisconnectArg = (): boolean =>
|
||||
Boolean(
|
||||
deps.appState.initialArgs?.managedPlayback ||
|
||||
@@ -111,45 +151,99 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
scheduleQuitCheck: (callback: () => void) => deps.scheduleQuitCheck(callback),
|
||||
isMpvConnected: () => Boolean(deps.appState.mpvClient?.connected),
|
||||
quitApp: () => deps.quitApp(),
|
||||
resolveSubtitleText: (liveText: string) =>
|
||||
resolvePrimarySubtitleText({
|
||||
liveText,
|
||||
currentTimeSec: Number(deps.appState.mpvClient?.currentTimePos),
|
||||
cues: deps.appState.activeParsedSubtitleCues,
|
||||
}),
|
||||
recordImmersionSubtitleLine: (text: string, start: number, end: number) => {
|
||||
deps.ensureImmersionTrackerInitialized();
|
||||
const tracker = deps.appState.immersionTracker;
|
||||
if (!tracker?.recordSubtitleLine) {
|
||||
return;
|
||||
}
|
||||
const recordLine = (lineText: string, startSec: number, endSec: number): void => {
|
||||
const secondaryText = deps.appState.mpvClient?.currentSecondarySubText || null;
|
||||
const cachedTokens =
|
||||
deps.appState.currentSubtitleData?.text === lineText
|
||||
? deps.appState.currentSubtitleData.tokens
|
||||
: null;
|
||||
if (cachedTokens) {
|
||||
tracker.recordSubtitleLine?.(lineText, startSec, endSec, cachedTokens, secondaryText);
|
||||
return;
|
||||
}
|
||||
if (!deps.tokenizeSubtitleForImmersion) {
|
||||
tracker.recordSubtitleLine?.(lineText, startSec, endSec, null, secondaryText);
|
||||
return;
|
||||
}
|
||||
const epochAtRecord = subtitleSessionEpoch;
|
||||
void deps
|
||||
.tokenizeSubtitleForImmersion(lineText)
|
||||
.then((payload) => {
|
||||
if (subtitleSessionEpoch !== epochAtRecord) {
|
||||
return;
|
||||
}
|
||||
tracker.recordSubtitleLine?.(
|
||||
lineText,
|
||||
startSec,
|
||||
endSec,
|
||||
payload?.tokens ?? null,
|
||||
secondaryText,
|
||||
);
|
||||
})
|
||||
.catch(() => {
|
||||
if (subtitleSessionEpoch !== epochAtRecord) {
|
||||
return;
|
||||
}
|
||||
tracker.recordSubtitleLine?.(lineText, startSec, endSec, null, secondaryText);
|
||||
});
|
||||
};
|
||||
const canonical = resolveCanonicalSample(text, start);
|
||||
if (canonical) {
|
||||
for (const cue of canonical.cues) {
|
||||
const key = canonicalCueKey(cue);
|
||||
if (recordedImmersionCanonicalKeys.has(key)) {
|
||||
continue;
|
||||
}
|
||||
recordedImmersionCanonicalKeys.add(key);
|
||||
recordLine(cue.text, cue.startTime, cue.endTime);
|
||||
}
|
||||
return;
|
||||
}
|
||||
text = stripFragmentsForRecording(text, start);
|
||||
if (!immersionLineDedupGate.shouldRecord({ text, startSec: start, endSec: end })) {
|
||||
return;
|
||||
}
|
||||
const secondaryText = deps.appState.mpvClient?.currentSecondarySubText || null;
|
||||
const cachedTokens =
|
||||
deps.appState.currentSubtitleData?.text === text
|
||||
? deps.appState.currentSubtitleData.tokens
|
||||
: null;
|
||||
if (cachedTokens) {
|
||||
tracker.recordSubtitleLine(text, start, end, cachedTokens, secondaryText);
|
||||
return;
|
||||
}
|
||||
if (!deps.tokenizeSubtitleForImmersion) {
|
||||
tracker.recordSubtitleLine(text, start, end, null, secondaryText);
|
||||
return;
|
||||
}
|
||||
void deps
|
||||
.tokenizeSubtitleForImmersion(text)
|
||||
.then((payload) => {
|
||||
tracker.recordSubtitleLine?.(text, start, end, payload?.tokens ?? null, secondaryText);
|
||||
})
|
||||
.catch(() => {
|
||||
tracker.recordSubtitleLine?.(text, start, end, null, secondaryText);
|
||||
});
|
||||
recordLine(text, start, end);
|
||||
},
|
||||
hasSubtitleTimingTracker: () => Boolean(deps.appState.subtitleTimingTracker),
|
||||
recordSubtitleTiming: (text: string, start: number, end: number) =>
|
||||
deps.appState.subtitleTimingTracker?.recordSubtitle?.(
|
||||
text,
|
||||
start,
|
||||
end,
|
||||
deps.appState.mpvClient?.currentSecondarySubText || undefined,
|
||||
),
|
||||
recordSubtitleTiming: (text: string, start: number, end: number) => {
|
||||
const secondaryText = deps.appState.mpvClient?.currentSecondarySubText || undefined;
|
||||
const canonical = resolveCanonicalSample(text, start);
|
||||
if (!canonical) {
|
||||
deps.appState.subtitleTimingTracker?.recordSubtitle?.(
|
||||
stripFragmentsForRecording(text, start),
|
||||
start,
|
||||
end,
|
||||
secondaryText,
|
||||
);
|
||||
return;
|
||||
}
|
||||
for (const cue of canonical.cues) {
|
||||
const key = canonicalCueKey(cue);
|
||||
if (recordedTimingCanonicalKeys.has(key)) {
|
||||
continue;
|
||||
}
|
||||
recordedTimingCanonicalKeys.add(key);
|
||||
deps.appState.subtitleTimingTracker?.recordSubtitle?.(
|
||||
cue.text,
|
||||
cue.startTime,
|
||||
cue.endTime,
|
||||
secondaryText,
|
||||
);
|
||||
}
|
||||
},
|
||||
maybeRunAnilistPostWatchUpdate: (options?: AnilistPostWatchRunOptions) =>
|
||||
deps.maybeRunAnilistPostWatchUpdate(options),
|
||||
logSubtitleTimingError: (message: string, error: unknown) =>
|
||||
@@ -170,7 +264,14 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
? (message: string) => deps.logSubtitleProcessingDebug!(message)
|
||||
: undefined,
|
||||
onSubtitleTrackChange: (sid: number | null) => {
|
||||
immersionLineDedupGate.reset();
|
||||
resetSubtitleDeduplication();
|
||||
// The replacement track's cues arrive only after an async re-read and re-parse.
|
||||
// Clearing synchronously keeps the previous track's canonical cues from
|
||||
// substituting into, or recording against, the new track's live text. The source
|
||||
// key is cleared with the cues so cue-list consumers (the sidebar snapshot)
|
||||
// re-parse on demand instead of trusting the stale pairing.
|
||||
deps.appState.activeParsedSubtitleCues = [];
|
||||
deps.appState.activeParsedSubtitleSource = null;
|
||||
deps.onSubtitleTrackChange?.(sid);
|
||||
},
|
||||
onSubtitleTrackListChange: deps.onSubtitleTrackListChange
|
||||
@@ -185,7 +286,7 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
broadcastSecondarySubtitle: (text: string) =>
|
||||
deps.broadcastToOverlayWindows('secondary-subtitle:set', text),
|
||||
updateCurrentMediaPath: (path: string) => {
|
||||
immersionLineDedupGate.reset();
|
||||
resetSubtitleDeduplication();
|
||||
deps.updateCurrentMediaPath(path);
|
||||
},
|
||||
restoreMpvSubVisibility: () => deps.restoreMpvSubVisibility(),
|
||||
@@ -217,9 +318,22 @@ export function createBuildBindMpvMainEventHandlersMainDepsHandler(deps: {
|
||||
},
|
||||
reportJellyfinRemoteProgress: (forceImmediate: boolean) =>
|
||||
deps.reportJellyfinRemoteProgress(forceImmediate),
|
||||
onTimePosUpdate: deps.onTimePosUpdate
|
||||
? (time: number) => deps.onTimePosUpdate!(time)
|
||||
: undefined,
|
||||
onTimePosUpdate: (time: number) => {
|
||||
// Timing history is a viewing log: after a real backward seek, a rewatched
|
||||
// canonical line should enter it again. Immersion stats keep their
|
||||
// once-per-media deduplication and are not reset here.
|
||||
if (
|
||||
Number.isFinite(time) &&
|
||||
lastTimePosForTimingReset !== null &&
|
||||
time <= lastTimePosForTimingReset - SEEK_LIKE_TIME_DELTA_SECONDS
|
||||
) {
|
||||
recordedTimingCanonicalKeys.clear();
|
||||
}
|
||||
if (Number.isFinite(time)) {
|
||||
lastTimePosForTimingReset = time;
|
||||
}
|
||||
deps.onTimePosUpdate?.(time);
|
||||
},
|
||||
onFullscreenChange: deps.onFullscreenChange
|
||||
? (fullscreen: boolean) => deps.onFullscreenChange!(fullscreen)
|
||||
: undefined,
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import {
|
||||
resolveCanonicalPrimarySubtitle,
|
||||
resolvePrimarySubtitleText,
|
||||
stripCanonicalFragmentLines,
|
||||
} from './primary-subtitle-text';
|
||||
|
||||
test('resolvePrimarySubtitleText prefers an active canonical cue over flattened mpv glyphs', () => {
|
||||
// mpv renders each simultaneously active ASS event on its own sub-text line.
|
||||
const text = resolvePrimarySubtitleText({
|
||||
liveText: '今\n今\n今\n手\n手\n手\nにある\nにある\nにある',
|
||||
currentTimeSec: 2,
|
||||
cues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(text, '今 手にある');
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText preserves live text outside canonical cue timing', () => {
|
||||
const text = resolvePrimarySubtitleText({
|
||||
liveText: '通常の会話',
|
||||
currentTimeSec: 8,
|
||||
cues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(text, '通常の会話');
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText keeps concurrent dialogue that is not part of the animation', () => {
|
||||
// An insert song's canonical window can overlap real dialogue on the same track.
|
||||
const text = resolvePrimarySubtitleText({
|
||||
liveText: '普通のセリフ\n今\n手にある',
|
||||
currentTimeSec: 2,
|
||||
cues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(text, '普通のセリフ\n今\n手にある');
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText keeps a fresh line starting just after the animation ended', () => {
|
||||
const text = resolvePrimarySubtitleText({
|
||||
liveText: '次のセリフ',
|
||||
currentTimeSec: 4.1,
|
||||
cues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(text, '次のセリフ');
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText survives overlapping frames of consecutive karaoke lines', () => {
|
||||
// Near a line boundary the previous line's exit frames and the next line's entrance
|
||||
// frames render together; neither line alone explains every live segment.
|
||||
const cues = [
|
||||
{ startTime: 1.2, endTime: 3.8, text: '今 手にある', source: 'canonical-ass' as const },
|
||||
{ startTime: 3.8, endTime: 6.4, text: '物差しでは', source: 'canonical-ass' as const },
|
||||
];
|
||||
|
||||
assert.equal(
|
||||
resolvePrimarySubtitleText({
|
||||
liveText: '手にある\n物差し\nでは',
|
||||
currentTimeSec: 3.6,
|
||||
cues,
|
||||
}),
|
||||
'今 手にある',
|
||||
);
|
||||
assert.equal(
|
||||
resolvePrimarySubtitleText({
|
||||
liveText: '手にある\n物差し\nでは',
|
||||
currentTimeSec: 3.9,
|
||||
cues,
|
||||
}),
|
||||
'物差しでは',
|
||||
);
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText combines simultaneous canonical cues in source order', () => {
|
||||
const text = resolvePrimarySubtitleText({
|
||||
liveText: 'fir\nst\nsecond',
|
||||
currentTimeSec: 2,
|
||||
cues: [
|
||||
{ startTime: 1, endTime: 3, text: 'first', source: 'canonical-ass' },
|
||||
{ startTime: 1.5, endTime: 2.5, text: 'second', source: 'canonical-ass' },
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(text, 'first\nsecond');
|
||||
});
|
||||
|
||||
test('resolveCanonicalPrimarySubtitle covers a nearby generated animation edge', () => {
|
||||
const cue = {
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass' as const,
|
||||
};
|
||||
const resolved = resolveCanonicalPrimarySubtitle({
|
||||
liveText: '今\n手にある',
|
||||
currentTimeSec: 0.8,
|
||||
cues: [cue],
|
||||
});
|
||||
|
||||
assert.deepEqual(resolved, {
|
||||
text: '今 手にある',
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
cues: [cue],
|
||||
});
|
||||
});
|
||||
|
||||
test('resolveCanonicalPrimarySubtitle covers exit frames that outlive the authored timing', () => {
|
||||
// Real generated animations keep exit fragments on screen well past the authored
|
||||
// comment window; the recorded animation envelope is what makes them resolvable.
|
||||
const cue = {
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass' as const,
|
||||
animationStartTime: 0.8,
|
||||
animationEndTime: 5.6,
|
||||
};
|
||||
const resolved = resolveCanonicalPrimarySubtitle({
|
||||
liveText: '今\n手にある',
|
||||
currentTimeSec: 5.4,
|
||||
cues: [cue],
|
||||
});
|
||||
|
||||
assert.deepEqual(resolved, {
|
||||
text: '今 手にある',
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
cues: [cue],
|
||||
});
|
||||
});
|
||||
|
||||
test('resolvePrimarySubtitleText handles late exit frames overlapping the next active line', () => {
|
||||
// The previous line's exit fragments can persist more than a second into the next
|
||||
// authored line. The next line supplies the text; the previous line's envelope
|
||||
// explains its lingering fragments.
|
||||
const cues = [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass' as const,
|
||||
animationStartTime: 0.8,
|
||||
animationEndTime: 5.6,
|
||||
},
|
||||
{
|
||||
startTime: 3.8,
|
||||
endTime: 6.4,
|
||||
text: '物差しでは',
|
||||
source: 'canonical-ass' as const,
|
||||
animationStartTime: 3.4,
|
||||
animationEndTime: 7.0,
|
||||
},
|
||||
];
|
||||
|
||||
assert.equal(
|
||||
resolvePrimarySubtitleText({
|
||||
liveText: '手にある\n手にある\n物差し\nでは',
|
||||
currentTimeSec: 5.2,
|
||||
cues,
|
||||
}),
|
||||
'物差しでは',
|
||||
);
|
||||
});
|
||||
|
||||
test('resolveCanonicalPrimarySubtitle rejects unrelated live text at the animation edge', () => {
|
||||
const resolved = resolveCanonicalPrimarySubtitle({
|
||||
liveText: '次のセリフ',
|
||||
currentTimeSec: 4.1,
|
||||
cues: [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
assert.equal(resolved, null);
|
||||
});
|
||||
|
||||
test('stripCanonicalFragmentLines drops fragment lines but keeps concurrent dialogue', () => {
|
||||
const cues = [
|
||||
{
|
||||
startTime: 1.2,
|
||||
endTime: 3.8,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass' as const,
|
||||
},
|
||||
];
|
||||
|
||||
assert.equal(
|
||||
stripCanonicalFragmentLines({
|
||||
liveText: '普通のセリフ\n今\n手にある',
|
||||
currentTimeSec: 2,
|
||||
cues,
|
||||
}),
|
||||
'普通のセリフ',
|
||||
);
|
||||
// No canonical cue nearby: nothing to strip.
|
||||
assert.equal(
|
||||
stripCanonicalFragmentLines({ liveText: '普通のセリフ\n今', currentTimeSec: 30, cues }),
|
||||
'普通のセリフ\n今',
|
||||
);
|
||||
// Everything matched (defensive): return the input rather than empty text.
|
||||
assert.equal(
|
||||
stripCanonicalFragmentLines({ liveText: '今\n手にある', currentTimeSec: 2, cues }),
|
||||
'今\n手にある',
|
||||
);
|
||||
});
|
||||
|
||||
test('resolveCanonicalPrimarySubtitle picks the cue its fragments spell, not the nearest', () => {
|
||||
// In the gap between two authored spans, the next line sits closer in time while only
|
||||
// the previous line's exit fragments are on screen: the fragments decide.
|
||||
const cues = [
|
||||
{
|
||||
startTime: 1,
|
||||
endTime: 3,
|
||||
text: '今 手にある',
|
||||
source: 'canonical-ass' as const,
|
||||
animationStartTime: 0.6,
|
||||
animationEndTime: 3.9,
|
||||
},
|
||||
{
|
||||
startTime: 4,
|
||||
endTime: 6,
|
||||
text: '物差しでは',
|
||||
source: 'canonical-ass' as const,
|
||||
animationStartTime: 3.5,
|
||||
animationEndTime: 6.4,
|
||||
},
|
||||
];
|
||||
|
||||
assert.equal(
|
||||
resolveCanonicalPrimarySubtitle({ liveText: '手にある', currentTimeSec: 3.8, cues })?.text,
|
||||
'今 手にある',
|
||||
);
|
||||
// Fragments of both lines in the gap: both envelopes cover the moment (distance 0),
|
||||
// and the earlier line wins the tie while it is still animating out.
|
||||
assert.equal(
|
||||
resolveCanonicalPrimarySubtitle({ liveText: '手にある\n物差し', currentTimeSec: 3.8, cues })
|
||||
?.text,
|
||||
'今 手にある',
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,160 @@
|
||||
import type { SubtitleCue } from '../../types';
|
||||
|
||||
// Slack on top of each cue's recorded animation envelope, for time-pos observation
|
||||
// staleness and small user sub-delay offsets. The envelope itself covers how far
|
||||
// entrance/exit frames actually run past the authored timing.
|
||||
const CANONICAL_ANIMATION_EDGE_TOLERANCE_SECONDS = 1;
|
||||
|
||||
export interface ResolvedPrimarySubtitle {
|
||||
text: string;
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
/** The canonical cues behind `text`, for consumers that record lines individually. */
|
||||
cues: SubtitleCue[];
|
||||
}
|
||||
|
||||
function animationSpan(cue: SubtitleCue): { start: number; end: number } {
|
||||
return {
|
||||
start: cue.animationStartTime ?? cue.startTime,
|
||||
end: cue.animationEndTime ?? cue.endTime,
|
||||
};
|
||||
}
|
||||
|
||||
function nearbyCanonicalCues(
|
||||
cues: readonly SubtitleCue[] | null | undefined,
|
||||
currentTimeSec: number,
|
||||
): SubtitleCue[] {
|
||||
return (cues ?? []).filter((cue) => {
|
||||
if (cue.source !== 'canonical-ass') {
|
||||
return false;
|
||||
}
|
||||
const span = animationSpan(cue);
|
||||
return (
|
||||
span.end >= currentTimeSec - CANONICAL_ANIMATION_EDGE_TOLERANCE_SECONDS &&
|
||||
span.start <= currentTimeSec + CANONICAL_ANIMATION_EDGE_TOLERANCE_SECONDS
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
function compactWhitespace(text: string): string {
|
||||
return text.replace(/\s+/gu, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* mpv's `sub-text` renders each simultaneously active ASS event on its own line, so
|
||||
* while a generated animation plays every live line is a contiguous piece of the
|
||||
* authored text. A line that is not -- concurrent dialogue during an insert song, or a
|
||||
* fresh line starting just after the animation ended -- proves the live text is not this
|
||||
* animation, and substituting the canonical line would swallow real dialogue.
|
||||
*/
|
||||
function liveTextIsFromCues(liveText: string, cues: readonly SubtitleCue[]): boolean {
|
||||
const compactCues = cues.map((cue) => compactWhitespace(cue.text));
|
||||
const segments = liveText.split('\n').map(compactWhitespace).filter(Boolean);
|
||||
return (
|
||||
segments.length > 0 &&
|
||||
segments.every((segment) => compactCues.some((cueText) => cueText.includes(segment)))
|
||||
);
|
||||
}
|
||||
|
||||
export function resolveCanonicalPrimarySubtitle(options: {
|
||||
liveText: string;
|
||||
currentTimeSec: number;
|
||||
cues: readonly SubtitleCue[] | null | undefined;
|
||||
}): ResolvedPrimarySubtitle | null {
|
||||
if (!Number.isFinite(options.currentTimeSec)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Consecutive karaoke lines overlap: one line's exit frames are still on screen while
|
||||
// the next line's entrance frames appear. The fragment check therefore runs against
|
||||
// every canonical cue whose animation envelope reaches the current time, while only
|
||||
// the active (or single nearest) cue supplies the displayed text.
|
||||
const nearby = nearbyCanonicalCues(options.cues, options.currentTimeSec);
|
||||
const active = nearby.filter(
|
||||
(cue) => cue.startTime <= options.currentTimeSec && cue.endTime > options.currentTimeSec,
|
||||
);
|
||||
const liveSegments = options.liveText.split('\n').map(compactWhitespace).filter(Boolean);
|
||||
const selected =
|
||||
active.length > 0
|
||||
? active
|
||||
: nearby
|
||||
// Between authored spans, proximity alone can pick the wrong neighbor: the
|
||||
// next line can sit closer while only the previous line's exit fragments are
|
||||
// on screen. Only cues that explain at least one live line may be selected.
|
||||
.filter((cue) => {
|
||||
const cueText = compactWhitespace(cue.text);
|
||||
return liveSegments.some((segment) => cueText.includes(segment));
|
||||
})
|
||||
.map((cue) => {
|
||||
const span = animationSpan(cue);
|
||||
const distance =
|
||||
options.currentTimeSec < span.start
|
||||
? span.start - options.currentTimeSec
|
||||
: Math.max(0, options.currentTimeSec - span.end);
|
||||
return { cue, distance };
|
||||
})
|
||||
.sort((a, b) => a.distance - b.distance || a.cue.startTime - b.cue.startTime)
|
||||
.slice(0, 1)
|
||||
.map(({ cue }) => cue);
|
||||
if (selected.length === 0 || !liveTextIsFromCues(options.liveText, nearby)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const texts: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const cue of selected) {
|
||||
if (!seen.has(cue.text)) {
|
||||
seen.add(cue.text);
|
||||
texts.push(cue.text);
|
||||
}
|
||||
}
|
||||
return {
|
||||
text: texts.join('\n'),
|
||||
startTime: Math.min(...selected.map((cue) => cue.startTime)),
|
||||
endTime: Math.max(...selected.map((cue) => cue.endTime)),
|
||||
cues: selected,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Live text with generated-animation fragment lines removed. Recording paths use this
|
||||
* when full canonical substitution declined -- concurrent dialogue during an insert
|
||||
* song: the dialogue is worth recording, the glyph fragments beside it are not. Returns
|
||||
* the input unchanged when no canonical cue is near or nothing non-fragment remains.
|
||||
*/
|
||||
export function stripCanonicalFragmentLines(options: {
|
||||
liveText: string;
|
||||
currentTimeSec: number;
|
||||
cues: readonly SubtitleCue[] | null | undefined;
|
||||
}): string {
|
||||
if (!Number.isFinite(options.currentTimeSec)) {
|
||||
return options.liveText;
|
||||
}
|
||||
const nearby = nearbyCanonicalCues(options.cues, options.currentTimeSec);
|
||||
if (nearby.length === 0) {
|
||||
return options.liveText;
|
||||
}
|
||||
const compactCues = nearby.map((cue) => compactWhitespace(cue.text));
|
||||
const kept = options.liveText.split('\n').filter((line) => {
|
||||
const compact = compactWhitespace(line);
|
||||
return compact && !compactCues.some((cueText) => cueText.includes(compact));
|
||||
});
|
||||
return kept.length > 0 ? kept.join('\n') : options.liveText;
|
||||
}
|
||||
|
||||
export function resolvePrimarySubtitleText(options: {
|
||||
liveText: string;
|
||||
currentTimeSec: number;
|
||||
cues: readonly SubtitleCue[] | null | undefined;
|
||||
}): string {
|
||||
if (!options.liveText.trim()) {
|
||||
return options.liveText;
|
||||
}
|
||||
return (
|
||||
resolveCanonicalPrimarySubtitle({
|
||||
liveText: options.liveText,
|
||||
currentTimeSec: options.currentTimeSec,
|
||||
cues: options.cues,
|
||||
})?.text ?? options.liveText
|
||||
);
|
||||
}
|
||||
@@ -1424,6 +1424,19 @@ test('subtitle annotation CSS underlines JLPT tokens without changing token colo
|
||||
);
|
||||
});
|
||||
|
||||
test('prepareSecondarySubtitleLines drops layered duplicate lines in short stacks', () => {
|
||||
// A word-level animation stacks one event per layer copy; the stack is too short for
|
||||
// the karaoke heuristic but the duplicates are still never distinct content.
|
||||
assert.deepEqual(prepareSecondarySubtitleLines('Your\\NYour\\NYour\\NYour\\Nmosaic'), [
|
||||
'Your',
|
||||
'mosaic',
|
||||
]);
|
||||
assert.deepEqual(prepareSecondarySubtitleLines('One line\\NAnother line'), [
|
||||
'One line',
|
||||
'Another line',
|
||||
]);
|
||||
});
|
||||
|
||||
test('prepareSecondarySubtitleLines collapses karaoke syllable spam into one deduped line', () => {
|
||||
// Karaoke-typeset OP/ED: one ASS event per syllable, duplicated across layers,
|
||||
// joined with \N by mpv's secondary-sub-text.
|
||||
|
||||
@@ -677,10 +677,12 @@ export function prepareSecondarySubtitleLines(text: string): string[] {
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0);
|
||||
if (!isKaraokeLikeLineSet(lines)) {
|
||||
return lines;
|
||||
}
|
||||
|
||||
// Identical lines in one render are layered copies of the same event (animation
|
||||
// scripts stack several per glyph), never distinct content -- always drop them, so a
|
||||
// short stack like "Your ×4 / mosaic" collapses without needing the karaoke
|
||||
// heuristic. Karaoke-likeness is still judged on the raw stack, where the layered
|
||||
// repetition is the signal.
|
||||
const seen = new Set<string>();
|
||||
const unique: string[] = [];
|
||||
for (const line of lines) {
|
||||
@@ -688,6 +690,10 @@ export function prepareSecondarySubtitleLines(text: string): string[] {
|
||||
seen.add(line);
|
||||
unique.push(line);
|
||||
}
|
||||
if (!isKaraokeLikeLineSet(lines)) {
|
||||
return unique;
|
||||
}
|
||||
|
||||
return [unique.join(' ')];
|
||||
}
|
||||
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
import { readStoredZipFirstFile, writeStoredZip, writeStoredZipAsync } from './stored-zip';
|
||||
|
||||
function makeTempDir(): string {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-stored-zip-'));
|
||||
}
|
||||
|
||||
function readEntries(zipPath: string): Map<string, Buffer> {
|
||||
const archive = fs.readFileSync(zipPath);
|
||||
const entries = new Map<string, Buffer>();
|
||||
let cursor = 0;
|
||||
|
||||
while (cursor + 4 <= archive.length) {
|
||||
const signature = archive.readUInt32LE(cursor);
|
||||
if (signature === 0x02014b50 || signature === 0x06054b50) {
|
||||
break;
|
||||
}
|
||||
assert.equal(signature, 0x04034b50, `unexpected local file header at offset ${cursor}`);
|
||||
const size = archive.readUInt32LE(cursor + 18);
|
||||
const nameLength = archive.readUInt16LE(cursor + 26);
|
||||
const extraLength = archive.readUInt16LE(cursor + 28);
|
||||
const nameStart = cursor + 30;
|
||||
const dataStart = nameStart + nameLength + extraLength;
|
||||
entries.set(
|
||||
archive.subarray(nameStart, nameStart + nameLength).toString('utf8'),
|
||||
Buffer.from(archive.subarray(dataStart, dataStart + size)),
|
||||
);
|
||||
cursor = dataStart + size;
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
// The async writer yields on a byte budget between entries, so an entry that is itself larger than
|
||||
// that budget is the case where the accounting could drift: the offsets, CRCs, and central
|
||||
// directory all have to come out identical to the synchronous writer.
|
||||
test('writeStoredZipAsync writes a correct archive when one entry exceeds the yield budget', async () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
// Comfortably past the writer's 8MB yield budget.
|
||||
const oversized = Buffer.alloc(10 * 1024 * 1024);
|
||||
for (let i = 0; i < oversized.length; i += 1) {
|
||||
oversized[i] = i % 251;
|
||||
}
|
||||
const files = [
|
||||
{ name: 'index.json', data: Buffer.from('{"revision":"rev-1"}', 'utf8') },
|
||||
{ name: 'big.bin', data: oversized },
|
||||
{ name: 'after.txt', data: Buffer.from('written after the oversized entry', 'utf8') },
|
||||
];
|
||||
|
||||
const asyncPath = path.join(dir, 'async.zip');
|
||||
const syncPath = path.join(dir, 'sync.zip');
|
||||
const asyncResult = await writeStoredZipAsync(asyncPath, files);
|
||||
const syncResult = writeStoredZip(syncPath, files);
|
||||
|
||||
assert.equal(asyncResult.entryCount, 3);
|
||||
assert.deepEqual(asyncResult, syncResult);
|
||||
// Byte-identical to the synchronous writer: yielding mid-archive changed no offset or CRC.
|
||||
assert.ok(fs.readFileSync(asyncPath).equals(fs.readFileSync(syncPath)));
|
||||
|
||||
const entries = readEntries(asyncPath);
|
||||
assert.deepEqual([...entries.keys()], ['index.json', 'big.bin', 'after.txt']);
|
||||
assert.ok(entries.get('big.bin')!.equals(oversized));
|
||||
assert.equal(entries.get('after.txt')!.toString('utf8'), 'written after the oversized entry');
|
||||
// The trailing records still parse, which is what proves the archive is complete.
|
||||
assert.equal(readStoredZipFirstFile(asyncPath)?.name, 'index.json');
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
+46
-102
@@ -1,5 +1,4 @@
|
||||
import * as fs from 'fs';
|
||||
import * as zlib from 'zlib';
|
||||
|
||||
type ZipEntry = {
|
||||
name: string;
|
||||
@@ -37,17 +36,10 @@ const CRC32_TABLE = (() => {
|
||||
return table;
|
||||
})();
|
||||
|
||||
// Native CRC32 (Node >= 20.15) runs at native throughput, which matters for the multi-hundred-MB
|
||||
// dictionary archives; the table loop stays as a fallback for runtimes without it.
|
||||
const nativeCrc32 = (zlib as { crc32?: (data: Uint8Array, value?: number) => number }).crc32;
|
||||
|
||||
function crc32(data: Buffer): number {
|
||||
if (typeof nativeCrc32 === 'function') {
|
||||
return nativeCrc32(data) >>> 0;
|
||||
}
|
||||
let crc = 0xffffffff;
|
||||
for (let i = 0; i < data.length; i += 1) {
|
||||
crc = CRC32_TABLE[(crc ^ data[i]!) & 0xff]! ^ (crc >>> 8);
|
||||
for (const byte of data) {
|
||||
crc = CRC32_TABLE[(crc ^ byte) & 0xff]! ^ (crc >>> 8);
|
||||
}
|
||||
return (crc ^ 0xffffffff) >>> 0;
|
||||
}
|
||||
@@ -302,107 +294,59 @@ function writeBuffer(fd: number, buffer: Buffer): void {
|
||||
}
|
||||
}
|
||||
|
||||
type ZipWriteState = {
|
||||
entries: ZipEntry[];
|
||||
offset: number;
|
||||
};
|
||||
|
||||
/** Appends one stored entry (local header + data) and returns the bytes written. */
|
||||
function appendStoredZipFile(fd: number, state: ZipWriteState, file: StoredZipFile): number {
|
||||
const fileName = Buffer.from(file.name, 'utf8');
|
||||
const fileSize = file.data.length;
|
||||
if (fileName.length > ZIP32_MAX_UINT16) {
|
||||
throw new RangeError(`ZIP entry name too long: ${file.name}`);
|
||||
}
|
||||
if (fileSize > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError(`ZIP entry too large for ZIP32: ${file.name}`);
|
||||
}
|
||||
if (state.offset > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
const fileCrc32 = crc32(file.data);
|
||||
const localHeader = createLocalFileHeader(fileName, fileCrc32, fileSize);
|
||||
const nextOffset = state.offset + localHeader.length + fileSize;
|
||||
if (nextOffset > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
writeBuffer(fd, localHeader);
|
||||
writeBuffer(fd, file.data);
|
||||
state.entries.push({
|
||||
name: file.name,
|
||||
crc32: fileCrc32,
|
||||
size: fileSize,
|
||||
localHeaderOffset: state.offset,
|
||||
});
|
||||
const written = nextOffset - state.offset;
|
||||
state.offset = nextOffset;
|
||||
return written;
|
||||
}
|
||||
|
||||
function finishStoredZip(fd: number, state: ZipWriteState): void {
|
||||
const centralStart = state.offset;
|
||||
if (centralStart > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
for (const entry of state.entries) {
|
||||
const centralHeader = createCentralDirectoryHeader(entry);
|
||||
writeBuffer(fd, centralHeader);
|
||||
state.offset += centralHeader.length;
|
||||
}
|
||||
|
||||
const centralSize = state.offset - centralStart;
|
||||
writeBuffer(fd, createEndOfCentralDirectory(state.entries.length, centralSize, centralStart));
|
||||
}
|
||||
|
||||
export function writeStoredZip(
|
||||
outputPath: string,
|
||||
files: Iterable<StoredZipFile>,
|
||||
): { entryCount: number } {
|
||||
const state: ZipWriteState = { entries: [], offset: 0 };
|
||||
const entries: ZipEntry[] = [];
|
||||
let offset = 0;
|
||||
const fd = fs.openSync(outputPath, 'w');
|
||||
|
||||
try {
|
||||
for (const file of files) {
|
||||
appendStoredZipFile(fd, state, file);
|
||||
}
|
||||
finishStoredZip(fd, state);
|
||||
} catch (error) {
|
||||
fs.closeSync(fd);
|
||||
fs.rmSync(outputPath, { force: true });
|
||||
throw error;
|
||||
}
|
||||
|
||||
fs.closeSync(fd);
|
||||
return { entryCount: state.entries.length };
|
||||
}
|
||||
|
||||
// Yielding roughly every 8MB keeps individual event-loop blocks in the low tens of milliseconds
|
||||
// while adding a negligible number of macrotask hops even for the largest merged dictionary.
|
||||
const ASYNC_ZIP_YIELD_BYTE_BUDGET = 8 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Same archive as {@link writeStoredZip}, written without starving the event loop: entry
|
||||
* generation, CRC, and writes proceed in byte-budgeted slices with a macrotask yield in between.
|
||||
* Multi-hundred-MB dictionary archives previously blocked the main process long enough for the
|
||||
* compositor to declare the app unresponsive.
|
||||
*/
|
||||
export async function writeStoredZipAsync(
|
||||
outputPath: string,
|
||||
files: Iterable<StoredZipFile>,
|
||||
): Promise<{ entryCount: number }> {
|
||||
const state: ZipWriteState = { entries: [], offset: 0 };
|
||||
const fd = fs.openSync(outputPath, 'w');
|
||||
|
||||
try {
|
||||
let bytesSinceYield = 0;
|
||||
for (const file of files) {
|
||||
bytesSinceYield += appendStoredZipFile(fd, state, file);
|
||||
if (bytesSinceYield >= ASYNC_ZIP_YIELD_BYTE_BUDGET) {
|
||||
bytesSinceYield = 0;
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
const fileName = Buffer.from(file.name, 'utf8');
|
||||
const fileSize = file.data.length;
|
||||
if (fileName.length > ZIP32_MAX_UINT16) {
|
||||
throw new RangeError(`ZIP entry name too long: ${file.name}`);
|
||||
}
|
||||
if (fileSize > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError(`ZIP entry too large for ZIP32: ${file.name}`);
|
||||
}
|
||||
if (offset > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
const fileCrc32 = crc32(file.data);
|
||||
const localHeader = createLocalFileHeader(fileName, fileCrc32, fileSize);
|
||||
const nextOffset = offset + localHeader.length + fileSize;
|
||||
if (nextOffset > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
writeBuffer(fd, localHeader);
|
||||
writeBuffer(fd, file.data);
|
||||
entries.push({
|
||||
name: file.name,
|
||||
crc32: fileCrc32,
|
||||
size: fileSize,
|
||||
localHeaderOffset: offset,
|
||||
});
|
||||
if (nextOffset > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
offset = nextOffset;
|
||||
}
|
||||
finishStoredZip(fd, state);
|
||||
|
||||
const centralStart = offset;
|
||||
if (centralStart > ZIP32_MAX_UINT32) {
|
||||
throw new RangeError('Archive exceeds ZIP32 limits (Zip64 not implemented)');
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const centralHeader = createCentralDirectoryHeader(entry);
|
||||
writeBuffer(fd, centralHeader);
|
||||
offset += centralHeader.length;
|
||||
}
|
||||
|
||||
const centralSize = offset - centralStart;
|
||||
writeBuffer(fd, createEndOfCentralDirectory(entries.length, centralSize, centralStart));
|
||||
} catch (error) {
|
||||
fs.closeSync(fd);
|
||||
fs.rmSync(outputPath, { force: true });
|
||||
@@ -410,5 +354,5 @@ export async function writeStoredZipAsync(
|
||||
}
|
||||
|
||||
fs.closeSync(fd);
|
||||
return { entryCount: state.entries.length };
|
||||
return { entryCount: entries.length };
|
||||
}
|
||||
|
||||
@@ -98,10 +98,10 @@ export function DuplicateLineCleanup({ onClose, onCleaned }: DuplicateLineCleanu
|
||||
|
||||
<div className="space-y-4 px-5 py-4">
|
||||
<p className="text-xs leading-relaxed text-ctp-subtext0">
|
||||
Karaoke openings and animated signs are typeset as one subtitle event per animation
|
||||
frame, and older versions counted every frame as a line. This collapses those runs back
|
||||
to one line and drops the word and kanji counts they added. Repeated dialogue is left
|
||||
alone.
|
||||
Typeset subtitles — karaoke openings, animated signs — are authored as one event per
|
||||
animation frame, and older versions counted every frame as its own line. This finds
|
||||
those runs and collapses each one back to a single line, giving back the word and kanji
|
||||
counts they inflated. Ordinary repeated dialogue is left alone.
|
||||
</p>
|
||||
|
||||
<div>
|
||||
@@ -192,8 +192,8 @@ export function DuplicateLineCleanup({ onClose, onCleaned }: DuplicateLineCleanu
|
||||
</button>
|
||||
</div>
|
||||
<p className="text-[11px] text-ctp-overlay1">
|
||||
Scan first: cleanup deletes rows and can't be undone. Watch time and lines-seen totals
|
||||
stay as they are.
|
||||
Scan first: cleanup removes rows and cannot be undone. Session watch time and lines-seen
|
||||
totals are left untouched.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user