Compare commits

...

4 Commits

Author SHA1 Message Date
sudacode 772635ab96 fix(stats): harden delete maintenance queue and shutdown handling
- Flush pending writes before locking during maintenance
- Handle scheduler failures and worker shutdown races
2026-08-12 00:49:40 -07:00
sudacode 675cb33519 fix(stats): prevent delete maintenance from freezing the UI
- Serialize and coalesce delete requests through a dedicated scheduler
- Chunk large SQLite ID lists to stay below variable limits
2026-08-11 23:11:45 -07:00
sudacode 320db591ae fix(stats): batch deletes off the main thread
- Keep stats and playback responsive during delete maintenance
- Serialize concurrent deletes and rebuild summaries once
2026-08-11 22:23:38 -07:00
sudacode 7b0fbdf254 fix(subtitles): collapse duplicate ASS events and decode text once (#186) 2026-08-10 22:21:44 -07:00
27 changed files with 2871 additions and 139 deletions
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: stats
- Kept the stats page and active video player responsive during deletes, and batched concurrent session, episode, and library deletes into one transaction and summary rebuild.
+5
View File
@@ -0,0 +1,5 @@
type: fixed
area: subtitles
- Heavily typeset ASS scripts (karaoke OP/ED, sign work) no longer fill the subtitle sidebar with garbage. Vector drawing runs (`\p1``\p0`) are no longer shown as subtitle text (e.g. `m 20 0 b 10 0 0 10 0 20 …`), and duplicate events in a parsed subtitle file collapse into one cue: identical text over an identical span (layered "shadow" copies), and per-frame animation bursts. An ASS burst has to prove itself with authoring evidence — a temporal tag (`\t`, `\move`, karaoke timing), an animated `Effect` column, or override values that change from event to event — plus one shared style and actor, so three rapid `えっ` reactions from three characters, or a sign repeated with the same static `\clip`, stay separate. SRT and VTT carry no such metadata, so there the run has to be at least five contiguous events all shorter than 0.1s, which is where ASS-to-SRT conversion leaves karaoke frames.
- Subtitle text is now decoded from ASS exactly once, where it enters the app, and matches how mpv renders the same line (including `\N`, `\n`, `\h`, unclosed `{`, and the fact that `\{` is not an escape). Renderer, timing tracker, tokenizer and the tokenization cache take that decoded text as-is instead of each re-deriving it, so one authored line can no longer produce two different cache keys, and a cue that normalizes to nothing is no longer stored as subtitle text or cached under an empty key.
@@ -64,18 +64,23 @@ External subtitle files only (SRT, VTT, ASS). Embedded subtitle tracks are out o
A cue parser extracts both timing and text content from subtitle files for prefetching. A cue parser extracts both timing and text content from subtitle files for prefetching.
**Parsed cue structure:** **Parsed cue structure:**
```typescript ```typescript
interface SubtitleCue { interface SubtitleCue {
startTime: number; // seconds startTime: number; // seconds
endTime: number; // seconds endTime: number; // seconds
text: string; // raw subtitle text text: string; // plain text, decoded from the source format
} }
``` ```
**Supported formats:** **Supported formats:**
- SRT/VTT: Regex-based parsing of timing lines + text content between timing blocks. - SRT/VTT: Regex-based parsing of timing lines + text content between timing blocks.
- ASS: Parse `[Events]` section, extract `Dialogue:` lines, split on the first 9 commas only (ASS v4+ has 10 fields; the last field is Text which can itself contain commas). Strip ASS override tags (`{\...}`) from the text before storing. - ASS: Parse `[Events]` section, extract `Dialogue:` lines, read the field order from the `Format:` row, and take everything after the Text field index as the text (Text can itself contain commas).
ASS text fields contain inline override tags like `{\b1}`, `{\an8}`, `{\fad(200,300)}`. The cue parser strips these during extraction so the tokenizer receives clean text.
**ASS decoding.** The parser is where ASS text is decoded, once, via `assToPlainText()` in `src/core/services/ass-text.ts`. That decoder mirrors mpv's `ass_to_plaintext` so a cue read from a file reads identically to the same line arriving live on `sub-text`: `{...}` override blocks are markup, `\pN … \p0` vector drawing runs are dropped rather than shown as text, `\N`/`\n`/`\h` are the only escapes (`\{`, `\}` and `\\` are not), and an unclosed `{` is rendered verbatim. Every layer downstream — renderer, timing tracker, tokenizer, tokenization cache keys — receives plain text and uses `normalizePlainSubtitleText()` for whitespace only, so nothing decodes the same string twice and one authored line always maps to one cache key.
**Duplicate collapsing.** Typeset scripts emit one `Dialogue:` event per animation frame, plus layered copies of the same line. The parser collapses identical text over an identical span unconditionally, and collapses contiguous same-text runs of at least three events when the run looks like an animation. For ASS that means shared style and actor plus authoring evidence: a temporal tag (`\t`, `\move`, `\k`/`\kf`/`\ko`/`\K`, or anything wrapped in `\t(...)`), an animated `Effect` column (`Karaoke`, `Banner`, `Scroll`), or override values that change across the run. Static tags shared by every event (`\pos`, an identical `\clip`) are not evidence. SRT/VTT carry no such metadata, so there collapsing needs at least five contiguous events all under 0.1s — the frame timing left behind by ASS-to-SRT conversion. The parser keeps this authoring metadata (style, actor, layer, `Effect`, parsed override commands, source order) private; `parseSubtitleCues()` returns only `SubtitleCue`.
#### Prefetch Service Lifecycle #### Prefetch Service Lifecycle
@@ -153,6 +158,7 @@ tokens (already have frequencyRank values from parser-level applyFrequencyRanks)
### Dependency Analysis ### Dependency Analysis
All annotations either depend on MeCab POS data or benefit from running after it: All annotations either depend on MeCab POS data or benefit from running after it:
- **Known word marking:** Needs base tokens (surface/headword). No POS dependency, but no reason to run separately. - **Known word marking:** Needs base tokens (surface/headword). No POS dependency, but no reason to run separately.
- **Frequency filtering:** Uses `pos1Exclusions` and `pos2Exclusions` to clear frequency ranks on excluded tokens (particles, noise). Depends on MeCab POS data. - **Frequency filtering:** Uses `pos1Exclusions` and `pos2Exclusions` to clear frequency ranks on excluded tokens (particles, noise). Depends on MeCab POS data.
- **JLPT marking:** Uses `shouldIgnoreJlptForMecabPos1` to filter. Depends on MeCab POS data. - **JLPT marking:** Uses `shouldIgnoreJlptForMecabPos1` to filter. Depends on MeCab POS data.
@@ -169,18 +175,14 @@ function annotateTokens(tokens, deps, options): MergedToken[] {
// Single pass: known word + frequency filtering + JLPT computed together // Single pass: known word + frequency filtering + JLPT computed together
const annotated = tokens.map((token) => { const annotated = tokens.map((token) => {
const isKnown = nPlusOneEnabled const isKnown = nPlusOneEnabled ? token.isKnown || computeIsKnown(token, deps) : false;
? token.isKnown || computeIsKnown(token, deps)
: false;
// Filter frequency rank using POS exclusions (rank values already set at parser level) // Filter frequency rank using POS exclusions (rank values already set at parser level)
const frequencyRank = frequencyEnabled const frequencyRank = frequencyEnabled
? filterFrequencyRank(token, pos1Exclusions, pos2Exclusions) ? filterFrequencyRank(token, pos1Exclusions, pos2Exclusions)
: undefined; : undefined;
const jlptLevel = jlptEnabled const jlptLevel = jlptEnabled ? computeJlptLevel(token, deps.getJlptLevel) : undefined;
? computeJlptLevel(token, deps.getJlptLevel)
: undefined;
return { ...token, isKnown, frequencyRank, jlptLevel }; return { ...token, isKnown, frequencyRank, jlptLevel };
}); });
@@ -221,6 +223,7 @@ Replace `document.createElement('span')` calls in the renderer with `templateSpa
### Current Behavior ### Current Behavior
In `renderWithTokens` (`subtitle-render.ts`), each render cycle: In `renderWithTokens` (`subtitle-render.ts`), each render cycle:
1. Clears DOM with `innerHTML = ''` 1. Clears DOM with `innerHTML = ''`
2. Creates a `DocumentFragment` 2. Creates a `DocumentFragment`
3. Calls `document.createElement('span')` for each token (~10-15 per subtitle) 3. Calls `document.createElement('span')` for each token (~10-15 per subtitle)
@@ -257,7 +260,7 @@ Full recycling (collecting old nodes, clearing attributes, reusing them) require
## Combined Impact Summary ## Combined Impact Summary
| Scenario | Before | After | Improvement | | Scenario | Before | After | Improvement |
|----------|--------|-------|-------------| | --------------------------------- | ---------- | ---------- | ----------- |
| Normal playback (prefetch-warmed) | ~200-320ms | ~30-50ms | ~80-85% | | Normal playback (prefetch-warmed) | ~200-320ms | ~30-50ms | ~80-85% |
| Cache hit (repeated subtitle) | ~72ms | ~55-65ms | ~10-20% | | Cache hit (repeated subtitle) | ~72ms | ~55-65ms | ~10-20% |
| Cache miss (immediate seek) | ~200-320ms | ~150-260ms | ~20-25% | | Cache miss (immediate seek) | ~200-320ms | ~150-260ms | ~20-25% |
@@ -267,16 +270,19 @@ Full recycling (collecting old nodes, clearing attributes, reusing them) require
## Files Summary ## Files Summary
### New Files ### New Files
- `src/core/services/subtitle-prefetch.ts` - `src/core/services/subtitle-prefetch.ts`
- `src/core/services/subtitle-cue-parser.ts` - `src/core/services/subtitle-cue-parser.ts`
### Modified Files ### Modified Files
- `src/core/services/subtitle-processing-controller.ts` (expose `preCacheTokenization`) - `src/core/services/subtitle-processing-controller.ts` (expose `preCacheTokenization`)
- `src/core/services/tokenizer/annotation-stage.ts` (batched single-pass) - `src/core/services/tokenizer/annotation-stage.ts` (batched single-pass)
- `src/renderer/subtitle-render.ts` (template cloneNode) - `src/renderer/subtitle-render.ts` (template cloneNode)
- `src/main.ts` (wire up prefetch service) - `src/main.ts` (wire up prefetch service)
### Test Files ### Test Files
- New tests for subtitle cue parser (SRT, VTT, ASS formats) - New tests for subtitle cue parser (SRT, VTT, ASS formats)
- New tests for subtitle prefetch service (priority window, seek, pause/resume) - New tests for subtitle prefetch service (priority window, seek, pause/resume)
- Updated tests for annotation stage (same behavior, new implementation) - Updated tests for annotation stage (same behavior, new implementation)
+1
View File
@@ -25,6 +25,7 @@ Read when: you need to find the owner module for a behavior or test surface
- Anki workflow: `src/anki-integration/`, `src/core/services/anki-jimaku*.ts` - Anki workflow: `src/anki-integration/`, `src/core/services/anki-jimaku*.ts`
- Immersion tracking: `src/core/services/immersion-tracker/` - Immersion tracking: `src/core/services/immersion-tracker/`
Includes stats storage/query schema such as `imm_videos`, `imm_media_art`, and `imm_youtube_videos` for per-video and YouTube-specific library metadata. Includes stats storage/query schema such as `imm_videos`, `imm_media_art`, and `imm_youtube_videos` for per-video and YouTube-specific library metadata.
`delete-maintenance-scheduler.ts` coalesces and serializes stats deletes; expensive deletion and summary rebuilds run in `delete-maintenance-worker-thread.ts` while the tracker queues playback writes. Each batch uses one transaction, lexical update, rollup refresh, and lifetime rebuild.
- AniList tracking + character dictionary: `src/core/services/anilist/`, `src/main/runtime/composers/anilist-*`, `src/main/character-dictionary-runtime.ts`, `src/main/character-dictionary-runtime/` - AniList tracking + character dictionary: `src/core/services/anilist/`, `src/main/runtime/composers/anilist-*`, `src/main/character-dictionary-runtime.ts`, `src/main/character-dictionary-runtime/`
- Jellyfin integration: `src/core/services/jellyfin*.ts`, `src/main/runtime/composers/jellyfin-*` - Jellyfin integration: `src/core/services/jellyfin*.ts`, `src/main/runtime/composers/jellyfin-*`
- Window trackers: `src/window-trackers/` - Window trackers: `src/window-trackers/`
+195
View File
@@ -0,0 +1,195 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
assOverrideSignature,
assToPlainText,
collectAssOverrideCommands,
extractAssOverrideBlocks,
hasAssTemporalOverride,
isAnimatedAssEffectKind,
isAssTemporalCommand,
normalizePlainSubtitleText,
parseAssEffectField,
} from './ass-text';
test('assToPlainText drops vector drawing runs', () => {
assert.equal(
assToPlainText(
'{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
),
'',
);
});
test('assToPlainText keeps text around drawing runs on the same event', () => {
assert.equal(
assToPlainText('{\\p1}m 0 0 l 10 10{\\p0}本文{\\p1}m 5 5 l 6 6{\\p0}続き'),
'本文続き',
);
});
test('assToPlainText leaves \\pos alone when no drawing mode is active', () => {
assert.equal(assToPlainText('{\\pos(960,1068)\\bord3}位置指定'), '位置指定');
});
test('assToPlainText does not read \\pos as a drawing tag', () => {
assert.equal(assToPlainText('{\\p1\\pos(1,2)}m 0 0 l 5 5'), '');
});
test('assToPlainText resolves line-break and space escapes', () => {
assert.equal(assToPlainText('一行目\\N二行目'), '一行目\n二行目');
assert.equal(assToPlainText('一行目\\n二行目'), '一行目\n二行目');
assert.equal(assToPlainText('一行目\\N二行目', ' '), '一行目 二行目');
assert.equal(assToPlainText('間\\h隔'), '間 隔');
});
test('assToPlainText matches mpv on brace and backslash sequences', () => {
// mpv has no `\{` / `\}` / `\\` escapes: the backslashes are literal text and the
// braces still open and close an override block.
assert.equal(assToPlainText('\\{注\\}'), '\\');
assert.equal(assToPlainText('\\\\N'), '\\\n');
});
test('assToPlainText renders an unclosed override block verbatim', () => {
// mpv shows the stray brace; guessing where the block ended can eat a whole line.
assert.equal(assToPlainText('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
});
test('assToPlainText is idempotent', () => {
const samples = [
'{\\an5\\p1}m 0 0 l 5 5{\\p0}本文',
'\\{注\\}',
'\\\\N',
'本文{\\pos(1,2)',
'一行目\\N二行目\\h終わり',
];
for (const sample of samples) {
const once = assToPlainText(sample);
assert.equal(assToPlainText(once), once, sample);
}
});
test('assToPlainText normalizes CRLF before converting', () => {
assert.equal(assToPlainText('一行目\r\n二行目'), '一行目\n二行目');
});
test('normalizePlainSubtitleText settles whitespace without decoding ASS', () => {
// A brace reaching this layer is literal text mpv chose to show, not markup.
assert.equal(normalizePlainSubtitleText('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
assert.equal(normalizePlainSubtitleText('一行目\\N二行目'), '一行目\n二行目');
assert.equal(
normalizePlainSubtitleText('一行目\\N二行目', { collapseLineBreaks: true }),
'一行目 二行目',
);
assert.equal(normalizePlainSubtitleText(' 余白 ', { trim: false }), ' 余白 ');
});
test('normalizePlainSubtitleText is idempotent', () => {
for (const sample of ['一行目\\N二行目', '間\\h隔', '本文{\\pos(1,2)', ' 余白 ']) {
const once = normalizePlainSubtitleText(sample);
assert.equal(normalizePlainSubtitleText(once), once, sample);
}
});
test('extractAssOverrideBlocks returns block contents', () => {
assert.deepEqual(extractAssOverrideBlocks('{\\an8}上{\\fad(200,200)}下'), [
'\\an8',
'\\fad(200,200)',
]);
assert.deepEqual(extractAssOverrideBlocks('括弧なし'), []);
});
test('collectAssOverrideCommands captures names and arguments from blocks only', () => {
const commands = collectAssOverrideCommands('{\\pos(1,2)\\1c&HFFFFFF&\\kf30}歌詞');
assert.deepEqual(commands, [
{ name: 'pos', args: '1,2', animated: false },
{ name: '1c', args: '&HFFFFFF&', animated: false },
{ name: 'kf', args: '30', animated: false },
]);
// A `\pos(...)` sitting in visible text is not typesetting markup.
assert.deepEqual(collectAssOverrideCommands('\\pos(730,1042) と書いてある'), []);
});
test('collectAssOverrideCommands marks tags animated by a wrapping \\t', () => {
const commands = collectAssOverrideCommands('{\\clip(0,0,10,10)\\t(0,500,\\frz30)}文字');
assert.deepEqual(
commands.map((command) => [command.name, command.animated]),
[
['clip', false],
['t', false],
['frz', true],
],
);
assert.equal(hasAssTemporalOverride(commands), true);
});
test('collectAssOverrideCommands stops descending into deeply nested \\t tags', () => {
// Nested far past the recursion cap. Uncapped, this recurses once per level, and a
// pathological line (real files reach one or two levels) overflows the stack.
const nesting = 32;
const block = `{${'\\t(0,500,'.repeat(nesting)}\\frz30${')'.repeat(nesting)}}文字`;
const commands = collectAssOverrideCommands(block);
// The outer `\t` plus one per allowed recursion level, and nothing from below the cap.
assert.equal(commands.length, 9);
assert.deepEqual(new Set(commands.map((command) => command.name)), new Set(['t']));
assert.equal(hasAssTemporalOverride(commands), true);
});
test('hasAssTemporalOverride ignores static placement and shape tags', () => {
assert.equal(
hasAssTemporalOverride(collectAssOverrideCommands('{\\pos(1,2)\\clip(m 1 1)\\blur2}文字')),
false,
);
assert.equal(hasAssTemporalOverride(collectAssOverrideCommands('{\\move(1,2,3,4)}文字')), true);
});
test('isAssTemporalCommand covers only intrinsically animated tags', () => {
for (const command of ['t', 'move', 'k', 'kf', 'ko', 'K']) {
assert.equal(isAssTemporalCommand(command), true, command);
}
for (const command of ['clip', 'iclip', 'frz', 'fscx', 'blur', 'be', 'pos', 'fad']) {
assert.equal(isAssTemporalCommand(command), false, command);
}
});
test('assOverrideSignature distinguishes events by their override values', () => {
const first = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 1 1)}歌詞'));
const second = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 2 2)}歌詞'));
const repeat = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 1 1)}別の行'));
assert.notEqual(first, second);
assert.equal(first, repeat);
});
test('parseAssEffectField classifies the event-level Effect column', () => {
assert.equal(parseAssEffectField(''), 'none');
assert.equal(parseAssEffectField(' '), 'none');
assert.equal(parseAssEffectField('Banner;20;1;0'), 'banner');
assert.equal(parseAssEffectField('Scroll up;0;0;30;10'), 'scroll');
assert.equal(parseAssEffectField('Scroll down;0;0;30;10'), 'scroll');
assert.equal(parseAssEffectField('Karaoke'), 'karaoke');
assert.equal(parseAssEffectField('fx-template'), 'other');
});
test('parseAssEffectField matches stock effect names exactly', () => {
// Custom effect names that merely start with a stock name are not stock effects.
assert.equal(parseAssEffectField('scrolling-credit'), 'other');
assert.equal(parseAssEffectField('bannerfx;1'), 'other');
assert.equal(parseAssEffectField('karaoke-template'), 'other');
assert.equal(parseAssEffectField('Scroll'), 'other');
});
test('isAnimatedAssEffectKind covers the stock animated effects only', () => {
assert.equal(isAnimatedAssEffectKind('karaoke'), true);
assert.equal(isAnimatedAssEffectKind('banner'), true);
assert.equal(isAnimatedAssEffectKind('scroll'), true);
// Typesetting groups put static template names in this column too.
assert.equal(isAnimatedAssEffectKind('other'), false);
assert.equal(isAnimatedAssEffectKind('none'), false);
});
+280
View File
@@ -0,0 +1,280 @@
/*
* ASS/SSA text handling, split into two deliberately distinct contracts:
*
* assToPlainText() raw ASS event text -> plain text. Ingestion only.
* normalizePlainSubtitleText() already-decoded text -> display/lookup form.
*
* Subtitle text is decoded from ASS exactly once, at the point it enters the app: the
* file cue parser does it for sidecar/embedded scripts, and mpv does it for live text
* (`sub-text` is already run through mpv's own `ass_to_plaintext`). Everything
* downstream -- renderer, timing tracker, tokenizer, tokenization cache keys -- gets
* plain text and only normalizes whitespace, so no layer decodes the same string twice.
*
* assToPlainText mirrors mpv's `ass_to_plaintext` rather than inventing its own rules,
* so a cue parsed from a file reads the same as the same line arriving live:
* - `{...}` override blocks are markup
* - `\pN ... \p0` runs are vector paths, not dialogue
* - `\N`, `\n` and `\h` are the only escapes; `\{`, `\}` and `\\` are NOT escapes,
* so `\{注\}` decodes to a lone backslash exactly as mpv renders it
* - an unclosed `{` is rendered verbatim instead of swallowing the rest of the line
* Because the decoder never emits an escape or a closed brace, running it twice is a
* no-op -- but downstream code should still use normalizePlainSubtitleText.
*/
/** What `\N` and `\n` become. */
export type AssLineBreak = '\n' | ' ';
// `\p<n>` with n > 0 switches libass into vector-drawing mode: everything until the
// next `\p0` is a path (`m 20 0 b 10 0 ...`), not dialogue. The negative lookahead keeps
// `\pos(...)` from being read as a drawing tag.
const ASS_DRAWING_SCALE_PATTERN = /\\p(?![a-zA-Z])(\d*)/g;
function readDrawingScale(block: string): number | null {
ASS_DRAWING_SCALE_PATTERN.lastIndex = 0;
let scale: number | null = null;
let match: RegExpExecArray | null;
// Drawing mode is whatever the last `\p` tag in this block set it to.
while ((match = ASS_DRAWING_SCALE_PATTERN.exec(block)) !== null) {
scale = match[1] ? Number(match[1]) : 0;
}
return scale;
}
/** Resolve `\N`, `\n` and `\h`. The only text-level escapes libass recognises. */
function resolveWhitespaceEscapes(text: string, lineBreak: AssLineBreak): string {
return text.replace(/\\([Nnh])/g, (_match, escaped: string) =>
escaped === 'h' ? ' ' : lineBreak,
);
}
/** Strip `{...}` override blocks and the drawing runs they enable. */
function stripAssMarkup(raw: string): string {
let out = '';
let cursor = 0;
let drawing = false;
while (cursor < raw.length) {
if (raw[cursor] !== '{') {
if (!drawing) {
out += raw[cursor];
}
cursor += 1;
continue;
}
const close = raw.indexOf('}', cursor + 1);
if (close === -1) {
// mpv shows an unclosed `{` and everything after it. Guessing where the block was
// meant to end can eat a whole line of dialogue.
if (!drawing) {
out += raw.slice(cursor);
}
break;
}
const scale = readDrawingScale(raw.slice(cursor, close + 1));
if (scale !== null) {
drawing = scale > 0;
}
cursor = close + 1;
}
return out;
}
/**
* Decode a raw ASS/SSA event text field. Call this once, where the text enters the app;
* downstream layers take the result as plain text.
*/
export function assToPlainText(text: string, lineBreak: AssLineBreak = '\n'): string {
if (!text) return '';
return resolveWhitespaceEscapes(stripAssMarkup(text.replace(/\r\n/g, '\n')), lineBreak);
}
export interface NormalizePlainSubtitleTextOptions {
/** Fold every line break into a single space. */
collapseLineBreaks?: boolean;
trim?: boolean;
}
/**
* Whitespace normalization for text that has already been decoded -- by mpv for live
* subtitles, by the cue parser for files. Override blocks and drawing runs are none of
* this function's business; a `{` that reaches here is literal text mpv chose to show.
*
* `\N`/`\n`/`\h` are still folded, because subtitle sources outside the ASS path (asbplayer
* and other websocket clients) forward them raw and the display layer has to cope.
*/
export function normalizePlainSubtitleText(
text: string,
options: NormalizePlainSubtitleTextOptions = {},
): string {
if (!text) return '';
const { collapseLineBreaks = false, trim = true } = options;
let normalized = resolveWhitespaceEscapes(
text.replace(/\r\n/g, '\n'),
collapseLineBreaks ? ' ' : '\n',
);
if (collapseLineBreaks) {
normalized = normalized.replace(/\n/g, ' ').replace(/\s+/g, ' ');
}
return trim ? normalized.trim() : normalized;
}
/** The contents of each `{...}` block, without the braces. */
export function extractAssOverrideBlocks(text: string): string[] {
const blocks: string[] = [];
let cursor = 0;
while (cursor < text.length) {
const open = text.indexOf('{', cursor);
if (open === -1) {
break;
}
const close = text.indexOf('}', open + 1);
if (close === -1) {
break;
}
blocks.push(text.slice(open + 1, close));
cursor = close + 1;
}
return blocks;
}
export interface AssOverrideCommand {
/** Tag name without the backslash, e.g. `pos`, `kf`, `1c`. */
name: string;
/** Everything the tag was given, e.g. `960,1068` for `\pos(960,1068)`. */
args: string;
/** Nested inside a `\t(...)` argument, so its value is animated over the event. */
animated: boolean;
}
const ASS_OVERRIDE_NAME_PATTERN = /[1-4]?[a-zA-Z]+/y;
function readCommandArgs(block: string, start: number): { args: string; next: number } {
if (block[start] === '(') {
let depth = 0;
for (let i = start; i < block.length; i += 1) {
if (block[i] === '(') depth += 1;
else if (block[i] === ')') {
depth -= 1;
if (depth === 0) {
return { args: block.slice(start + 1, i), next: i + 1 };
}
}
}
return { args: block.slice(start + 1), next: block.length };
}
const nextTag = block.indexOf('\\', start);
const end = nextTag === -1 ? block.length : nextTag;
return { args: block.slice(start, end), next: end };
}
// `\t(...)` can wrap another `\t(...)`, and nothing in the format stops an author (or a
// malformed file) from nesting them thousands deep. Real typesetting never goes past one
// or two levels, so stop recursing well before the call stack is at risk.
const MAX_ANIMATION_NESTING_DEPTH = 8;
function parseOverrideBlock(
block: string,
animated: boolean,
into: AssOverrideCommand[],
depth = 0,
): void {
let cursor = 0;
while (cursor < block.length) {
if (block[cursor] !== '\\') {
cursor += 1;
continue;
}
ASS_OVERRIDE_NAME_PATTERN.lastIndex = cursor + 1;
const nameMatch = ASS_OVERRIDE_NAME_PATTERN.exec(block);
if (!nameMatch) {
cursor += 1;
continue;
}
const name = nameMatch[0];
const { args, next } = readCommandArgs(block, cursor + 1 + name.length);
into.push({ name, args: args.trim(), animated });
// `\t(0,500,\frz30)` animates whatever it wraps, so record the inner tags too.
if (name === 't' && args.includes('\\') && depth < MAX_ANIMATION_NESTING_DEPTH) {
parseOverrideBlock(args, true, into, depth + 1);
}
cursor = next;
}
}
/**
* Override commands with their arguments, in source order. Only `{...}` blocks are
* inspected, so a `\pos(...)` sitting in visible text is never mistaken for markup.
*/
export function collectAssOverrideCommands(text: string): AssOverrideCommand[] {
const commands: AssOverrideCommand[] = [];
for (const block of extractAssOverrideBlocks(text)) {
parseOverrideBlock(block, false, commands);
}
return commands;
}
// Tags that are animated by definition: `\t` interpolates, `\move` travels, and the
// karaoke tags advance a highlight across the event's own duration. Everything else --
// `\pos`, `\clip`, `\frz`, `\blur`, `\fad` -- is a static value for the event, so its
// presence says nothing about whether neighbouring events form one animation.
const ASS_TEMPORAL_COMMANDS = new Set(['t', 'move', 'k', 'kf', 'ko', 'K']);
export function isAssTemporalCommand(name: string): boolean {
return ASS_TEMPORAL_COMMANDS.has(name);
}
/** True when the event animates on its own, or animates a static tag through `\t(...)`. */
export function hasAssTemporalOverride(commands: readonly AssOverrideCommand[]): boolean {
return commands.some((command) => command.animated || isAssTemporalCommand(command.name));
}
/**
* Canonical form of an event's override values, for comparing consecutive events. Two
* events with the same signature were typeset identically, so neither is a frame of an
* animation the other belongs to.
*/
export function assOverrideSignature(commands: readonly AssOverrideCommand[]): string {
return commands.map((command) => `${command.name}(${command.args})`).join('|');
}
export type AssEffectKind = 'none' | 'banner' | 'scroll' | 'karaoke' | 'other';
// The stock effects, matched exactly. Typesetting groups put their own template names in
// this column -- `scrolling-credit` is a static sign, not libass's `Scroll up` -- so a
// prefix match would hand out animation evidence to arbitrary custom effects.
const STOCK_ASS_EFFECTS = new Map<string, AssEffectKind>([
['banner', 'banner'],
['scroll up', 'scroll'],
['scroll down', 'scroll'],
['karaoke', 'karaoke'],
]);
/**
* The event-level `Effect` column. The stock values (`Banner;...`, `Scroll up;...`,
* `Scroll down;...`, `Karaoke`) all animate; anything else is a custom name and lands in
* `other`.
*/
export function parseAssEffectField(raw: string): AssEffectKind {
const value = raw.trim().toLowerCase();
if (!value) return 'none';
const name = value.split(';', 1)[0]!.trim();
return STOCK_ASS_EFFECTS.get(name) ?? 'other';
}
const ANIMATED_ASS_EFFECT_KINDS = new Set<AssEffectKind>(['banner', 'scroll', 'karaoke']);
export function isAnimatedAssEffectKind(kind: AssEffectKind): boolean {
return ANIMATED_ASS_EFFECT_KINDS.has(kind);
}
@@ -1414,6 +1414,353 @@ test('deleteSession ignores the currently active session and keeps new writes fl
} }
}); });
test('deleteSession yields the main event loop while delete maintenance is pending', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const deleteGate: { release?: () => void } = {};
let deleteRunnerCalled = false;
let bufferedWritesAtDeleteStart = -1;
try {
const Ctor = await loadTrackerCtor();
const createdTracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async () => {
deleteRunnerCalled = true;
bufferedWritesAtDeleteStart = (tracker as unknown as { queue: unknown[] }).queue.length;
await new Promise<void>((resolve) => {
deleteGate.release = resolve;
});
},
},
);
tracker = createdTracker;
createdTracker.handleMediaChange('/tmp/delete-yield-first.mkv', 'Delete Yield First');
createdTracker.handleMediaChange('/tmp/delete-yield-active.mkv', 'Delete Yield Active');
const privateApi = createdTracker as unknown as {
db: DatabaseSync;
queue: unknown[];
flushNow: () => void;
};
const sessionId = (
privateApi.db
.prepare(
`SELECT session_id AS sessionId
FROM imm_sessions
WHERE ended_at_ms IS NOT NULL
ORDER BY session_id
LIMIT 1`,
)
.get() as { sessionId: number } | null
)?.sessionId;
assert.ok(sessionId);
const deletePromise = createdTracker.deleteSession(sessionId);
let timerAdvanced = false;
setTimeout(() => {
timerAdvanced = true;
}, 0);
await waitForCondition(() => deleteRunnerCalled);
assert.equal(deleteRunnerCalled, true, 'delete should be dispatched to the maintenance runner');
assert.equal(
bufferedWritesAtDeleteStart,
0,
'writes buffered before delete should flush first',
);
await waitForCondition(() => timerAdvanced);
createdTracker.recordSubtitleLine('queued during delete', 0, 1);
privateApi.flushNow();
assert.ok(privateApi.queue.length > 0, 'tracking writes should wait for delete maintenance');
assert.ok(deleteGate.release);
deleteGate.release();
await deletePromise;
} finally {
deleteGate.release?.();
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('delete maintenance flushes the entire write queue before locking writes', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const deleteGate: { release?: () => void } = {};
let queuedWritesAtDeleteStart = -1;
let writeLockedAtDeleteStart = false;
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async () => {
const privateApi = tracker as unknown as {
queue: unknown[];
writeLock: { locked: boolean };
};
queuedWritesAtDeleteStart = privateApi.queue.length;
writeLockedAtDeleteStart = privateApi.writeLock.locked;
await new Promise<void>((resolve) => {
deleteGate.release = resolve;
});
},
},
);
const privateApi = tracker as unknown as {
batchSize: number;
flushNow: () => void;
queue: unknown[];
};
privateApi.batchSize = 1;
privateApi.queue.push({}, {}, {});
privateApi.flushNow = () => {
privateApi.queue.shift();
};
const deletePromise = tracker.deleteSession(101);
await waitForCondition(() => deleteGate.release !== undefined);
assert.equal(queuedWritesAtDeleteStart, 0);
assert.equal(writeLockedAtDeleteStart, true);
deleteGate.release?.();
await deletePromise;
} finally {
deleteGate.release?.();
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('delete maintenance tasks stay serialized under concurrent requests', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const releases: Array<() => void> = [];
let activeTasks = 0;
let maxActiveTasks = 0;
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async () => {
activeTasks += 1;
maxActiveTasks = Math.max(maxActiveTasks, activeTasks);
await new Promise<void>((resolve) => {
releases.push(resolve);
});
activeTasks -= 1;
},
},
);
const firstDelete = tracker.deleteSession(101);
await waitForCondition(() => releases.length === 1);
assert.equal(maxActiveTasks, 1);
const secondDelete = tracker.deleteSession(102);
releases[0]?.();
await waitForCondition(() => releases.length === 2);
assert.equal(maxActiveTasks, 1);
releases[1]?.();
await Promise.all([firstDelete, secondDelete]);
} finally {
for (const release of releases) release();
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('concurrent delete requests share one maintenance worker batch', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const tasks: unknown[] = [];
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async (_path, task) => {
tasks.push(task);
},
},
);
const firstDelete = tracker.deleteSession(201);
const secondDelete = tracker.deleteSessions([202, 203]);
const thirdDelete = tracker.deleteVideo(204);
await Promise.all([firstDelete, secondDelete, thirdDelete]);
assert.equal(tasks.length, 1, 'concurrent deletes should use one maintenance pass');
assert.deepEqual(tasks[0], {
kind: 'batch',
tasks: [
{ kind: 'session', sessionId: 201 },
{ kind: 'sessions', sessionIds: [202, 203] },
{ kind: 'video', videoId: 204 },
],
});
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('destroy rejects delete requests waiting behind active maintenance', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let releaseFirstTask: () => void = () => {};
try {
const Ctor = await loadTrackerCtor();
let markFirstTaskStarted: () => void = () => {};
const firstTaskStarted = new Promise<void>((resolve) => {
markFirstTaskStarted = resolve;
});
tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async () => {
markFirstTaskStarted();
await new Promise<void>((resolve) => {
releaseFirstTask = resolve;
});
},
},
);
const firstDelete = tracker.deleteSession(301);
await firstTaskStarted;
const queuedDelete = tracker.deleteSession(302);
tracker.destroy();
const queuedOutcome = await Promise.race([
queuedDelete.then(
() => 'resolved',
(error: unknown) =>
error instanceof Error && /shutting down/.test(error.message)
? 'rejected'
: 'wrong-error',
),
new Promise<'pending'>((resolve) => setTimeout(() => resolve('pending'), 25)),
]);
assert.equal(queuedOutcome, 'rejected');
releaseFirstTask();
await firstDelete;
} finally {
releaseFirstTask();
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('delete requested after destroy rejects without running maintenance', async () => {
const dbPath = makeDbPath();
let maintenanceCalls = 0;
const Ctor = await loadTrackerCtor();
const tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async () => {
maintenanceCalls += 1;
},
},
);
tracker.destroy();
await assert.rejects(tracker.deleteSession(303), /shutting down/);
assert.equal(maintenanceCalls, 0);
cleanupDbPath(dbPath);
});
test('deleteSessions skips maintenance when no sessions are deletable', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const tasks: unknown[] = [];
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async (_path, task) => {
tasks.push(task);
},
},
);
await tracker.deleteSessions([]);
assert.deepEqual(tasks, []);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('queued video delete is skipped when that video becomes active before dispatch', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const tasks: Array<{ kind: string }> = [];
let releaseFirstTask: () => void = () => {};
try {
const Ctor = await loadTrackerCtor();
const createdTracker = new Ctor(
{ dbPath },
{
runDeleteMaintenanceTask: async (_path, task) => {
tasks.push(task);
if (tasks.length === 1) {
await new Promise<void>((resolve) => {
releaseFirstTask = resolve;
});
}
},
},
);
tracker = createdTracker;
createdTracker.handleMediaChange('/tmp/delete-race-target.mkv', 'Delete Race Target');
createdTracker.handleMediaChange('/tmp/delete-race-other.mkv', 'Delete Race Other');
const privateApi = createdTracker as unknown as { db: DatabaseSync };
const targetVideoId = (
privateApi.db
.prepare(`SELECT video_id AS videoId FROM imm_videos WHERE video_key LIKE '%target.mkv'`)
.get() as { videoId: number } | null
)?.videoId;
assert.ok(targetVideoId);
const firstDelete = createdTracker.deleteSession(999_001);
await waitForCondition(() => tasks.length === 1);
const queuedVideoDelete = createdTracker.deleteVideo(targetVideoId);
createdTracker.handleMediaChange('/tmp/delete-race-target.mkv', 'Delete Race Target');
releaseFirstTask();
await Promise.all([firstDelete, queuedVideoDelete]);
assert.deepEqual(
tasks.map((task) => task.kind),
['session'],
);
} finally {
releaseFirstTask();
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('deleteVideo ignores the currently active video and keeps new writes flushable', async () => { test('deleteVideo ignores the currently active video and keeps new writes flushable', async () => {
const dbPath = makeDbPath(); const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null; let tracker: ImmersionTrackerService | null = null;
+65 -15
View File
@@ -83,14 +83,15 @@ import {
} from './immersion-tracker/query-library'; } from './immersion-tracker/query-library';
import { import {
cleanupVocabularyStats, cleanupVocabularyStats,
deleteAnime as deleteAnimeQuery,
deleteSession as deleteSessionQuery,
deleteSessions as deleteSessionsQuery,
deleteVideo as deleteVideoQuery,
getVideoDurationMs, getVideoDurationMs,
markVideoWatched, markVideoWatched,
upsertCoverArt, upsertCoverArt,
} from './immersion-tracker/query-maintenance'; } from './immersion-tracker/query-maintenance';
import {
DeleteMaintenanceWorkerRuntime,
type RunDeleteMaintenanceTask,
} from './immersion-tracker/delete-maintenance-worker-runtime';
import { DeleteMaintenanceScheduler } from './immersion-tracker/delete-maintenance-scheduler';
import { repairJellyfinStreamVideoLinks } from './immersion-tracker/jellyfin-link-repair'; import { repairJellyfinStreamVideoLinks } from './immersion-tracker/jellyfin-link-repair';
import { import {
repairLegacySeasonlessAnimeRows, repairLegacySeasonlessAnimeRows,
@@ -182,6 +183,7 @@ const YOUTUBE_SCREENSHOT_MAX_SECONDS = 120;
const YOUTUBE_OEMBED_ENDPOINT = 'https://www.youtube.com/oembed'; const YOUTUBE_OEMBED_ENDPOINT = 'https://www.youtube.com/oembed';
const YOUTUBE_ID_PATTERN = /^[A-Za-z0-9_-]{6,}$/; const YOUTUBE_ID_PATTERN = /^[A-Za-z0-9_-]{6,}$/;
const YOUTUBE_METADATA_REFRESH_MS = 24 * 60 * 60 * 1000; const YOUTUBE_METADATA_REFRESH_MS = 24 * 60 * 60 * 1000;
const DELETE_MAINTENANCE_BATCH_WINDOW_MS = 10;
function isValidYouTubeVideoId(value: string | null): boolean { function isValidYouTubeVideoId(value: string | null): boolean {
return Boolean(value && YOUTUBE_ID_PATTERN.test(value)); return Boolean(value && YOUTUBE_ID_PATTERN.test(value));
@@ -385,6 +387,8 @@ export class ImmersionTrackerService {
private readonly vacuumIntervalMs: number; private readonly vacuumIntervalMs: number;
private readonly dbPath: string; private readonly dbPath: string;
private readonly writeLock = { locked: false }; private readonly writeLock = { locked: false };
private readonly destroyDeleteMaintenanceRunner: () => void;
private readonly deleteMaintenanceScheduler: DeleteMaintenanceScheduler;
private flushTimer: ReturnType<typeof setTimeout> | null = null; private flushTimer: ReturnType<typeof setTimeout> | null = null;
private maintenanceTimer: ReturnType<typeof setInterval> | null = null; private maintenanceTimer: ReturnType<typeof setInterval> | null = null;
private flushScheduled = false; private flushScheduled = false;
@@ -406,9 +410,38 @@ export class ImmersionTrackerService {
| ((row: LegacyVocabularyPosRow) => Promise<LegacyVocabularyPosResolution | null>) | ((row: LegacyVocabularyPosRow) => Promise<LegacyVocabularyPosResolution | null>)
| undefined; | undefined;
constructor(options: ImmersionTrackerOptions) { constructor(
options: ImmersionTrackerOptions,
dependencies: {
runDeleteMaintenanceTask?: RunDeleteMaintenanceTask;
destroyDeleteMaintenanceRunner?: () => void;
} = {},
) {
this.dbPath = options.dbPath; this.dbPath = options.dbPath;
this.resolveLegacyVocabularyPos = options.resolveLegacyVocabularyPos; this.resolveLegacyVocabularyPos = options.resolveLegacyVocabularyPos;
let runDeleteMaintenanceTask: RunDeleteMaintenanceTask;
if (dependencies.runDeleteMaintenanceTask) {
runDeleteMaintenanceTask = dependencies.runDeleteMaintenanceTask;
this.destroyDeleteMaintenanceRunner =
dependencies.destroyDeleteMaintenanceRunner ?? (() => {});
} else {
const deleteMaintenanceRuntime = new DeleteMaintenanceWorkerRuntime();
runDeleteMaintenanceTask = (dbPath, task) => deleteMaintenanceRuntime.run(dbPath, task);
this.destroyDeleteMaintenanceRunner = () => deleteMaintenanceRuntime.destroy();
}
this.deleteMaintenanceScheduler = new DeleteMaintenanceScheduler({
batchWindowMs: DELETE_MAINTENANCE_BATCH_WINDOW_MS,
runTask: (task) => runDeleteMaintenanceTask(this.dbPath, task),
onBusy: () => {
this.flushTelemetry(true);
while (this.queue.length > 0) this.flushNow();
this.writeLock.locked = true;
},
onIdle: () => {
this.writeLock.locked = false;
if (!this.isDestroyed && this.queue.length > 0) this.scheduleFlush(0);
},
});
const parentDir = path.dirname(this.dbPath); const parentDir = path.dirname(this.dbPath);
if (!fs.existsSync(parentDir)) { if (!fs.existsSync(parentDir)) {
fs.mkdirSync(parentDir, { recursive: true }); fs.mkdirSync(parentDir, { recursive: true });
@@ -512,6 +545,8 @@ export class ImmersionTrackerService {
} }
this.finalizeActiveSession(); this.finalizeActiveSession();
this.isDestroyed = true; this.isDestroyed = true;
this.deleteMaintenanceScheduler.destroy();
this.destroyDeleteMaintenanceRunner();
this.db.close(); this.db.close();
} }
@@ -709,10 +744,11 @@ export class ImmersionTrackerService {
this.logger.warn(`Ignoring delete request for active immersion session ${sessionId}`); this.logger.warn(`Ignoring delete request for active immersion session ${sessionId}`);
return; return;
} }
deleteSessionQuery(this.db, sessionId); await this.enqueueDeleteMaintenanceTask(() => ({ kind: 'session', sessionId }));
} }
async deleteSessions(sessionIds: number[]): Promise<void> { async deleteSessions(sessionIds: number[]): Promise<void> {
await this.enqueueDeleteMaintenanceTask(() => {
const activeSessionId = this.sessionState?.sessionId; const activeSessionId = this.sessionState?.sessionId;
const deletableSessionIds = const deletableSessionIds =
activeSessionId === undefined activeSessionId === undefined
@@ -723,21 +759,25 @@ export class ImmersionTrackerService {
`Ignoring bulk delete request for active immersion session ${activeSessionId}`, `Ignoring bulk delete request for active immersion session ${activeSessionId}`,
); );
} }
deleteSessionsQuery(this.db, deletableSessionIds); if (deletableSessionIds.length === 0) return null;
return { kind: 'sessions', sessionIds: deletableSessionIds };
});
} }
async deleteVideo(videoId: number): Promise<void> { async deleteVideo(videoId: number): Promise<void> {
await this.enqueueDeleteMaintenanceTask(() => {
if (this.sessionState?.videoId === videoId) { if (this.sessionState?.videoId === videoId) {
this.logger.warn(`Ignoring delete request for active immersion video ${videoId}`); this.logger.warn(`Ignoring delete request for active immersion video ${videoId}`);
return; return null;
} }
deleteVideoQuery(this.db, videoId); return { kind: 'video', videoId };
});
} }
async deleteAnime(animeId: number): Promise<void> { async deleteAnime(animeId: number): Promise<void> {
// The active video's anime link is assigned asynchronously after the title await this.enqueueDeleteMaintenanceTask(async () => {
// is parsed, so a guard reading imm_videos too early sees a null and lets // Resolve this at dispatch time because another queued delete can leave
// the delete through — then the late update recreates the anime row. // enough time for playback to switch to an episode of this anime.
const pendingVideoId = this.sessionState?.videoId; const pendingVideoId = this.sessionState?.videoId;
if (pendingVideoId !== undefined) { if (pendingVideoId !== undefined) {
await this.pendingAnimeMetadataUpdates.get(pendingVideoId); await this.pendingAnimeMetadataUpdates.get(pendingVideoId);
@@ -750,10 +790,20 @@ export class ImmersionTrackerService {
.get(activeVideoId) as { anime_id: number | null } | null; .get(activeVideoId) as { anime_id: number | null } | null;
if (activeAnime?.anime_id === animeId) { if (activeAnime?.anime_id === animeId) {
this.logger.warn(`Ignoring delete request for active immersion anime ${animeId}`); this.logger.warn(`Ignoring delete request for active immersion anime ${animeId}`);
return; return null;
} }
} }
deleteAnimeQuery(this.db, animeId); return { kind: 'anime', animeId };
});
}
private enqueueDeleteMaintenanceTask(
resolveTask: Parameters<DeleteMaintenanceScheduler['enqueue']>[0],
): Promise<void> {
if (this.isDestroyed) {
return Promise.reject(new Error('Immersion tracker is shutting down'));
}
return this.deleteMaintenanceScheduler.enqueue(resolveTask);
} }
async reassignAnimeAnilist( async reassignAnimeAnilist(
@@ -1811,7 +1861,7 @@ export class ImmersionTrackerService {
} }
private runMaintenance(): void { private runMaintenance(): void {
if (this.isDestroyed) return; if (this.isDestroyed || this.writeLock.locked) return;
try { try {
this.flushTelemetry(true); this.flushTelemetry(true);
this.flushNow(); this.flushNow();
@@ -50,6 +50,7 @@ import {
updateAnimeAnilistInfo, updateAnimeAnilistInfo,
upsertCoverArt, upsertCoverArt,
} from '../query-maintenance.js'; } from '../query-maintenance.js';
import { deleteMaintenanceBatch } from '../query-delete-maintenance.js';
import { getLocalEpochDay } from '../query-shared.js'; import { getLocalEpochDay } from '../query-shared.js';
import { EVENT_CARD_MINED, EVENT_SUBTITLE_LINE, SOURCE_TYPE_LOCAL } from '../types.js'; import { EVENT_CARD_MINED, EVENT_SUBTITLE_LINE, SOURCE_TYPE_LOCAL } from '../types.js';
@@ -985,3 +986,197 @@ test('split maintenance helpers delete multiple sessions and whole videos with d
cleanupDbPath(dbPath); cleanupDbPath(dbPath);
} }
}); });
test('delete maintenance batch preserves retained data across overlapping session, video, and anime targets', () => {
const { db, dbPath, stmts } = createDb();
try {
const retainedAnimeId = getOrCreateAnimeRecord(db, {
parsedTitle: 'Retained Anime',
canonicalTitle: 'Retained Anime',
anilistId: null,
titleRomaji: null,
titleEnglish: null,
titleNative: null,
metadataJson: null,
});
const deletedAnimeId = getOrCreateAnimeRecord(db, {
parsedTitle: 'Deleted Anime',
canonicalTitle: 'Deleted Anime',
anilistId: null,
titleRomaji: null,
titleEnglish: null,
titleNative: null,
metadataJson: null,
});
const retainedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-retain.mkv', {
canonicalTitle: 'Batch Retain',
sourcePath: '/tmp/batch-retain.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
const deletedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-video.mkv', {
canonicalTitle: 'Batch Video',
sourcePath: '/tmp/batch-video.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
const animeVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-anime.mkv', {
canonicalTitle: 'Batch Anime',
sourcePath: '/tmp/batch-anime.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
for (const [videoId, animeId, episode] of [
[retainedVideoId, retainedAnimeId, 1],
[deletedVideoId, retainedAnimeId, 2],
[animeVideoId, deletedAnimeId, 1],
] as const) {
linkVideoToAnimeRecord(db, videoId, {
animeId,
parsedBasename: `batch-${episode}.mkv`,
parsedTitle: animeId === retainedAnimeId ? 'Retained Anime' : 'Deleted Anime',
parsedSeason: 1,
parsedEpisode: episode,
parserSource: 'test',
parserConfidence: 1,
parseMetadataJson: null,
});
}
const startedAtMs = 1_700_000_000_000;
const deletedSessionId = startSessionRecord(db, retainedVideoId, startedAtMs).sessionId;
const retainedSessionId = startSessionRecord(
db,
retainedVideoId,
startedAtMs + 1_000,
).sessionId;
const videoSessionId = startSessionRecord(db, deletedVideoId, startedAtMs + 2_000).sessionId;
const animeSessionId = startSessionRecord(db, animeVideoId, startedAtMs + 3_000).sessionId;
for (const [sessionId, sessionStartedAtMs] of [
[deletedSessionId, startedAtMs],
[retainedSessionId, startedAtMs + 1_000],
[videoSessionId, startedAtMs + 2_000],
[animeSessionId, startedAtMs + 3_000],
] as const) {
finalizeSessionMetrics(db, sessionId, sessionStartedAtMs);
}
for (const [index, sessionId, videoId, animeId] of [
[1, deletedSessionId, retainedVideoId, retainedAnimeId],
[2, retainedSessionId, retainedVideoId, retainedAnimeId],
[3, videoSessionId, deletedVideoId, retainedAnimeId],
[4, animeSessionId, animeVideoId, deletedAnimeId],
] as const) {
insertWordOccurrence(db, stmts, {
sessionId,
videoId,
animeId,
lineIndex: index,
text: '猫日',
word: { headword: '猫', word: '猫', reading: 'ねこ' },
});
insertKanjiOccurrence(db, stmts, {
sessionId,
videoId,
animeId,
lineIndex: index + 10,
text: '猫日',
kanji: '日',
});
}
const rollupDay = getLocalEpochDay(db, startedAtMs);
const rollupMonth = (
db
.prepare(
`SELECT CAST(strftime('%Y%m', CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER) AS rollupMonth`,
)
.get(startedAtMs) as { rollupMonth: number }
).rollupMonth;
for (const videoId of [retainedVideoId, deletedVideoId, animeVideoId]) {
db.prepare(
`INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, 99, 99, 99, 99, 99, ?, ?)`,
).run(rollupDay, videoId, startedAtMs, startedAtMs);
db.prepare(
`INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, 99, 99, 99, 99, 99, ?, ?)`,
).run(rollupMonth, videoId, startedAtMs, startedAtMs);
}
deleteMaintenanceBatch(db, [
{ kind: 'session', sessionId: deletedSessionId },
{ kind: 'session', sessionId: videoSessionId },
{ kind: 'video', videoId: deletedVideoId },
{ kind: 'video', videoId: animeVideoId },
{ kind: 'anime', animeId: deletedAnimeId },
]);
assert.deepEqual(db.prepare('SELECT session_id FROM imm_sessions').all(), [
{ session_id: retainedSessionId },
]);
assert.deepEqual(db.prepare('SELECT video_id FROM imm_videos').all(), [
{ video_id: retainedVideoId },
]);
assert.deepEqual(db.prepare('SELECT anime_id FROM imm_anime').all(), [
{ anime_id: retainedAnimeId },
]);
assert.equal(
(
db.prepare(`SELECT frequency FROM imm_words WHERE headword = '猫'`).get() as {
frequency: number;
}
).frequency,
1,
);
assert.equal(
(
db.prepare(`SELECT frequency FROM imm_kanji WHERE kanji = '日'`).get() as {
frequency: number;
}
).frequency,
1,
);
assert.deepEqual(
db.prepare('SELECT video_id, total_sessions FROM imm_daily_rollups').all() as Array<{
video_id: number;
total_sessions: number;
}>,
[{ video_id: retainedVideoId, total_sessions: 1 }],
);
assert.deepEqual(
db.prepare('SELECT video_id, total_sessions FROM imm_monthly_rollups').all() as Array<{
video_id: number;
total_sessions: number;
}>,
[{ video_id: retainedVideoId, total_sessions: 1 }],
);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('delete maintenance batch chunks id lists below the SQLite variable limit', () => {
const { db, dbPath } = createDb();
try {
const ids = Array.from({ length: 32_767 }, (_, index) => index + 1);
assert.doesNotThrow(() => {
deleteMaintenanceBatch(db, [
{ kind: 'sessions', sessionIds: ids },
...ids.map((videoId) => ({ kind: 'video' as const, videoId })),
...ids.map((animeId) => ({ kind: 'anime' as const, animeId })),
]);
});
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
@@ -0,0 +1,160 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { DeleteMaintenanceScheduler } from './delete-maintenance-scheduler';
import type { DeleteMaintenanceTask } from './delete-maintenance';
test('scheduler batches same-turn requests and balances busy state', async () => {
const tasks: DeleteMaintenanceTask[] = [];
const states: string[] = [];
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async (task) => {
tasks.push(task);
},
onBusy: () => states.push('busy'),
onIdle: () => states.push('idle'),
});
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
const second = scheduler.enqueue(() => ({ kind: 'sessions', sessionIds: [2, 3] }));
const third = scheduler.enqueue(() => null);
await Promise.all([first, second, third]);
assert.deepEqual(tasks, [
{
kind: 'batch',
tasks: [
{ kind: 'session', sessionId: 1 },
{ kind: 'sessions', sessionIds: [2, 3] },
],
},
]);
assert.deepEqual(states, ['busy', 'idle']);
});
test('scheduler rejects enqueue after destruction without entering busy state', async () => {
let busyCalls = 0;
let runCalls = 0;
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async () => {
runCalls += 1;
},
onBusy: () => {
busyCalls += 1;
},
onIdle: () => {},
});
scheduler.destroy();
await assert.rejects(
scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 })),
/shutting down/,
);
assert.equal(busyCalls, 0);
assert.equal(runCalls, 0);
});
test('scheduler rejects every request in a batch when the maintenance task fails', async () => {
const failure = new Error('maintenance failed');
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async () => {
throw failure;
},
onBusy: () => {},
onIdle: () => {},
});
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
const second = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
const results = await Promise.allSettled([first, second]);
assert.deepEqual(
results.map((result) => (result.status === 'rejected' ? result.reason : null)),
[failure, failure],
);
});
test('scheduler rejects only the request whose task resolution fails', async () => {
const failure = new Error('resolution failed');
const tasks: DeleteMaintenanceTask[] = [];
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async (task) => {
tasks.push(task);
},
onBusy: () => {},
onIdle: () => {},
});
const failed = scheduler.enqueue(() => {
throw failure;
});
const succeeded = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
const results = await Promise.allSettled([failed, succeeded]);
assert.equal(results[0]?.status, 'rejected');
assert.equal(results[0]?.status === 'rejected' ? results[0].reason : null, failure);
assert.equal(results[1]?.status, 'fulfilled');
assert.deepEqual(tasks, [{ kind: 'session', sessionId: 2 }]);
});
test('scheduler does not schedule another drain when the queue is empty', async () => {
const originalSetTimeout = globalThis.setTimeout;
let timerCalls = 0;
globalThis.setTimeout = ((handler: TimerHandler, timeout?: number, ...args: unknown[]) => {
timerCalls += 1;
return originalSetTimeout(handler, timeout, ...args);
}) as typeof setTimeout;
try {
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async () => {},
onBusy: () => {},
onIdle: () => {},
});
await scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
assert.equal(timerCalls, 1);
} finally {
globalThis.setTimeout = originalSetTimeout;
}
});
test('scheduler serializes batches and rejects requests queued at destruction', async () => {
const releases: Array<() => void> = [];
let activeTasks = 0;
let maxActiveTasks = 0;
const scheduler = new DeleteMaintenanceScheduler({
batchWindowMs: 0,
runTask: async () => {
activeTasks += 1;
maxActiveTasks = Math.max(maxActiveTasks, activeTasks);
await new Promise<void>((resolve) => releases.push(resolve));
activeTasks -= 1;
},
onBusy: () => {},
onIdle: () => {},
});
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
const maxPollAttempts = 100;
let pollAttempts = 0;
while (releases.length === 0 && pollAttempts < maxPollAttempts) {
pollAttempts += 1;
await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
assert.ok(
releases.length > 0,
`runTask did not produce a release after ${maxPollAttempts} polling attempts`,
);
const queued = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
scheduler.destroy();
await assert.rejects(queued, /shutting down/);
releases[0]?.();
await first;
assert.equal(maxActiveTasks, 1);
});
@@ -0,0 +1,105 @@
import type { DeleteMaintenanceOperation, DeleteMaintenanceTask } from './delete-maintenance';
type ResolveDeleteMaintenanceOperation = () =>
| DeleteMaintenanceOperation
| null
| Promise<DeleteMaintenanceOperation | null>;
interface PendingDeleteMaintenanceRequest {
resolveTask: ResolveDeleteMaintenanceOperation;
resolve: () => void;
reject: (error: unknown) => void;
}
interface DeleteMaintenanceSchedulerOptions {
batchWindowMs: number;
runTask: (task: DeleteMaintenanceTask) => Promise<void>;
onBusy: () => void;
onIdle: () => void;
}
export class DeleteMaintenanceScheduler {
private readonly pendingRequests: PendingDeleteMaintenanceRequest[] = [];
private running = false;
private drainTimer: ReturnType<typeof setTimeout> | null = null;
private pendingTaskCount = 0;
private destroyed = false;
constructor(private readonly options: DeleteMaintenanceSchedulerOptions) {}
enqueue(resolveTask: ResolveDeleteMaintenanceOperation): Promise<void> {
if (this.destroyed) {
return Promise.reject(new Error('Immersion tracker is shutting down'));
}
if (this.pendingTaskCount === 0) this.options.onBusy();
this.pendingTaskCount += 1;
const result = new Promise<void>((resolve, reject) => {
this.pendingRequests.push({ resolveTask, resolve, reject });
this.scheduleDrain();
});
return result.finally(() => {
this.pendingTaskCount -= 1;
if (this.pendingTaskCount === 0) this.options.onIdle();
});
}
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
if (this.drainTimer) {
clearTimeout(this.drainTimer);
this.drainTimer = null;
}
const error = new Error('Immersion tracker is shutting down');
for (const request of this.pendingRequests.splice(0)) request.reject(error);
}
private scheduleDrain(): void {
if (this.destroyed || this.running || this.drainTimer || this.pendingRequests.length === 0) {
return;
}
this.drainTimer = setTimeout(() => {
this.drainTimer = null;
void this.drain();
}, this.options.batchWindowMs);
}
private async drain(): Promise<void> {
if (this.running || this.pendingRequests.length === 0) return;
this.running = true;
const requests = this.pendingRequests.splice(0);
const runnable: Array<{
request: PendingDeleteMaintenanceRequest;
task: DeleteMaintenanceOperation;
}> = [];
for (const request of requests) {
try {
const task = await request.resolveTask();
if (task) runnable.push({ request, task });
else request.resolve();
} catch (error) {
request.reject(error);
}
}
if (runnable.length > 0) {
const task: DeleteMaintenanceTask =
runnable.length === 1
? runnable[0]!.task
: { kind: 'batch', tasks: runnable.map((entry) => entry.task) };
try {
await this.options.runTask(task);
for (const { request } of runnable) request.resolve();
} catch (error) {
for (const { request } of runnable) request.reject(error);
}
}
this.running = false;
this.scheduleDrain();
}
}
@@ -0,0 +1,239 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import {
DeleteMaintenanceWorkerRuntime,
resolveDeleteMaintenanceWorkerPath,
} from './delete-maintenance-worker-runtime';
import { executeDeleteMaintenanceTask } from './delete-maintenance';
import { startSessionRecord } from './session';
import { Database } from './sqlite';
import { applyPragmas, ensureSchema, getOrCreateVideoRecord } from './storage';
type FakeWorkerListener = (value: never) => void;
function createFakeWorker() {
const listeners = new Map<string, FakeWorkerListener>();
const terminationState = { calls: 0 };
const worker = {
once(event: string, listener: FakeWorkerListener) {
listeners.set(event, listener);
return this;
},
terminate: async () => {
terminationState.calls += 1;
return 0;
},
};
return { worker, listeners, terminationState };
}
type FakeWorker = ReturnType<typeof createFakeWorker>['worker'];
test('a delete batch rebuilds lifetime summaries once', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-delete-batch-test-'));
const dbPath = path.join(tempDir, 'immersion.sqlite');
let db = new Database(dbPath);
try {
applyPragmas(db);
ensureSchema(db);
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-delete.mkv', {
canonicalTitle: 'Batch Delete',
sourcePath: '/tmp/batch-delete.mkv',
sourceUrl: null,
sourceType: 1,
});
const firstSessionId = startSessionRecord(db, videoId, 1_000).sessionId;
const secondSessionId = startSessionRecord(db, videoId, 2_000).sessionId;
const deletedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-delete-video.mkv', {
canonicalTitle: 'Batch Delete Video',
sourcePath: '/tmp/batch-delete-video.mkv',
sourceUrl: null,
sourceType: 1,
});
startSessionRecord(db, deletedVideoId, 3_000);
db.exec(`
CREATE TABLE delete_rebuild_audit (id INTEGER PRIMARY KEY);
CREATE TRIGGER count_delete_lifetime_rebuild
AFTER UPDATE OF last_rebuilt_ms ON imm_lifetime_global
BEGIN
INSERT INTO delete_rebuild_audit (id) VALUES (NULL);
END;
`);
db.close();
executeDeleteMaintenanceTask(dbPath, {
kind: 'batch',
tasks: [
{ kind: 'session', sessionId: firstSessionId },
{ kind: 'video', videoId: deletedVideoId },
],
});
db = new Database(dbPath);
const audit = db.prepare('SELECT COUNT(*) AS total FROM delete_rebuild_audit').get() as {
total: number;
};
const retainedSession = db
.prepare('SELECT session_id AS sessionId FROM imm_sessions WHERE video_id = ?')
.get(videoId) as { sessionId: number } | null;
const deletedVideo = db
.prepare('SELECT video_id AS videoId FROM imm_videos WHERE video_id = ?')
.get(deletedVideoId) as { videoId: number } | null;
assert.equal(retainedSession?.sessionId, secondSessionId);
assert.equal(deletedVideo, undefined);
assert.equal(
audit.total,
2,
'one rebuild performs exactly its reset and final global summary writes',
);
} finally {
try {
db.close();
} catch {
// The setup connection closes before maintenance runs.
}
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
test(
'compiled delete worker removes data through its separate database connection',
{ skip: resolveDeleteMaintenanceWorkerPath() === null },
async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-delete-worker-test-'));
const dbPath = path.join(tempDir, 'immersion.sqlite');
const runtime = new DeleteMaintenanceWorkerRuntime();
let db = new Database(dbPath);
try {
applyPragmas(db);
ensureSchema(db);
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/worker-delete.mkv', {
canonicalTitle: 'Worker Delete',
sourcePath: '/tmp/worker-delete.mkv',
sourceUrl: null,
sourceType: 1,
});
const firstSessionId = startSessionRecord(db, videoId, 1_000).sessionId;
const secondSessionId = startSessionRecord(db, videoId, 2_000).sessionId;
db.close();
await runtime.run(dbPath, {
kind: 'batch',
tasks: [
{ kind: 'session', sessionId: firstSessionId },
{ kind: 'session', sessionId: secondSessionId },
],
});
db = new Database(dbPath);
const row = db
.prepare('SELECT COUNT(*) AS total FROM imm_sessions WHERE video_id = ?')
.get(videoId) as { total: number };
assert.equal(row.total, 0);
} finally {
runtime.destroy();
try {
db.close();
} catch {
// The setup connection is already closed before the worker starts.
}
fs.rmSync(tempDir, { recursive: true, force: true });
}
},
);
test('worker runtime warns before falling back when no emitted worker is available', async () => {
const warnings: unknown[][] = [];
const fallbackTasks: unknown[] = [];
const runtime = new DeleteMaintenanceWorkerRuntime({
resolveWorkerPath: () => null,
warn: (...args) => warnings.push(args),
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
});
await runtime.run('/tmp/fallback.sqlite', { kind: 'session', sessionId: 1 });
assert.equal(warnings.length, 1);
assert.match(String(warnings[0]?.[0]), /worker unavailable/i);
assert.deepEqual(fallbackTasks, [{ kind: 'session', sessionId: 1 }]);
});
test('worker runtime terminates a worker after successful settlement', async () => {
const { worker, listeners, terminationState } = createFakeWorker();
const runtime = new DeleteMaintenanceWorkerRuntime({
resolveWorkerPath: () => '/tmp/delete-worker.js',
createWorker: async () => worker,
});
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
await new Promise<void>((resolve) => setTimeout(resolve, 0));
listeners.get('message')?.({ ok: true } as never);
await result;
assert.equal(terminationState.calls, 1);
});
test('worker runtime terminates a worker after failed settlement', async () => {
const { worker, listeners, terminationState } = createFakeWorker();
const runtime = new DeleteMaintenanceWorkerRuntime({
resolveWorkerPath: () => '/tmp/delete-worker.js',
createWorker: async () => worker,
});
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
await new Promise<void>((resolve) => setTimeout(resolve, 0));
listeners.get('error')?.(new Error('worker failed') as never);
await assert.rejects(result, /worker failed/);
assert.equal(terminationState.calls, 1);
});
test('worker runtime terminates a worker created after shutdown begins', async () => {
const { worker, listeners, terminationState } = createFakeWorker();
const createGate: { resolve?: (worker: FakeWorker) => void } = {};
const fallbackTasks: unknown[] = [];
const runtime = new DeleteMaintenanceWorkerRuntime({
resolveWorkerPath: () => '/tmp/delete-worker.js',
createWorker: () =>
new Promise((resolve) => {
createGate.resolve = resolve;
}),
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
});
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
await new Promise<void>((resolve) => setTimeout(resolve, 0));
runtime.destroy();
createGate.resolve?.(worker);
await assert.rejects(result, /shut down/);
assert.equal(terminationState.calls, 1);
assert.equal(listeners.size, 0);
assert.deepEqual(fallbackTasks, []);
});
test('worker runtime does not fall back when worker creation fails during shutdown', async () => {
const createGate: { reject?: (error: Error) => void } = {};
const fallbackTasks: unknown[] = [];
const runtime = new DeleteMaintenanceWorkerRuntime({
resolveWorkerPath: () => '/tmp/delete-worker.js',
createWorker: () =>
new Promise((_resolve, reject) => {
createGate.reject = reject;
}),
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
});
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
await new Promise<void>((resolve) => setTimeout(resolve, 0));
runtime.destroy();
createGate.reject?.(new Error('creation failed'));
await assert.rejects(result, /shut down/);
assert.deepEqual(fallbackTasks, []);
});
@@ -0,0 +1,121 @@
import fs from 'node:fs';
import path from 'node:path';
import { createLogger } from '../../../logger';
import { executeDeleteMaintenanceTask, type DeleteMaintenanceTask } from './delete-maintenance';
interface DeleteMaintenanceWorkerResponse {
ok?: unknown;
error?: unknown;
}
export type RunDeleteMaintenanceTask = (
dbPath: string,
task: DeleteMaintenanceTask,
) => Promise<void>;
interface DeleteMaintenanceWorkerHandle {
once(event: 'message', listener: (message: DeleteMaintenanceWorkerResponse) => void): this;
once(event: 'error', listener: (error: Error) => void): this;
once(event: 'exit', listener: (code: number) => void): this;
terminate(): Promise<number>;
}
interface DeleteMaintenanceWorkerRuntimeOptions {
resolveWorkerPath?: () => string | null;
createWorker?: (
workerPath: string,
workerData: { dbPath: string; task: DeleteMaintenanceTask },
) => Promise<DeleteMaintenanceWorkerHandle>;
executeFallback?: typeof executeDeleteMaintenanceTask;
warn?: (message: string, ...meta: unknown[]) => void;
}
export function resolveDeleteMaintenanceWorkerPath(): string | null {
const workerPath = path.join(__dirname, 'delete-maintenance-worker-thread.js');
return fs.existsSync(workerPath) ? workerPath : null;
}
const logger = createLogger('main:immersion-tracker:delete-worker');
export class DeleteMaintenanceWorkerRuntime {
private readonly activeWorkers = new Set<DeleteMaintenanceWorkerHandle>();
private destroyed = false;
constructor(private readonly options: DeleteMaintenanceWorkerRuntimeOptions = {}) {}
async run(dbPath: string, task: DeleteMaintenanceTask): Promise<void> {
if (this.destroyed) {
throw new Error('Delete maintenance worker is shut down');
}
let worker: DeleteMaintenanceWorkerHandle;
try {
const workerPath = (this.options.resolveWorkerPath ?? resolveDeleteMaintenanceWorkerPath)();
if (!workerPath) throw new Error('Emitted delete-maintenance worker module was not found');
const createWorker =
this.options.createWorker ??
(async (resolvedPath, workerData) => {
const { Worker } = await import('node:worker_threads');
return new Worker(resolvedPath, { workerData });
});
worker = await createWorker(workerPath, { dbPath, task });
} catch (error) {
if (this.destroyed) {
throw new Error('Delete maintenance worker is shut down');
}
(this.options.warn ?? logger.warn)(
'Delete maintenance worker unavailable; running maintenance on the current thread',
error,
);
(this.options.executeFallback ?? executeDeleteMaintenanceTask)(dbPath, task);
return;
}
if (this.destroyed) {
await worker.terminate().catch(() => undefined);
throw new Error('Delete maintenance worker is shut down');
}
await new Promise<void>((resolve, reject) => {
let settled = false;
this.activeWorkers.add(worker);
const settle = (error?: Error) => {
if (settled) return;
settled = true;
this.activeWorkers.delete(worker);
if (error) reject(error);
else resolve();
void worker.terminate();
};
worker.once('message', (message: DeleteMaintenanceWorkerResponse) => {
if (message.ok === true) {
settle();
return;
}
const detail = typeof message.error === 'string' ? message.error : 'unknown worker error';
settle(new Error(`Delete maintenance failed: ${detail}`));
});
worker.once('error', (error) => settle(error));
worker.once('exit', (code) => {
settle(
new Error(
code === 0
? 'Delete maintenance worker exited without a response'
: `Delete maintenance worker exited with code ${code}`,
),
);
});
});
}
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
for (const worker of this.activeWorkers) {
void worker.terminate();
}
this.activeWorkers.clear();
}
}
@@ -0,0 +1,22 @@
import { parentPort, workerData } from 'node:worker_threads';
import { executeDeleteMaintenanceTask, type DeleteMaintenanceTask } from './delete-maintenance';
interface DeleteMaintenanceWorkerData {
dbPath: string;
task: DeleteMaintenanceTask;
}
if (!parentPort) {
throw new Error('delete maintenance worker missing parent port');
}
const port = parentPort;
const request = workerData as DeleteMaintenanceWorkerData;
try {
executeDeleteMaintenanceTask(request.dbPath, request.task);
port.postMessage({ ok: true });
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
port.postMessage({ ok: false, error: message });
}
@@ -0,0 +1,47 @@
import { Database } from './sqlite';
import { applyPragmas } from './storage';
import { deleteAnime, deleteSession, deleteSessions, deleteVideo } from './query-maintenance';
import {
deleteMaintenanceBatch,
type DeleteMaintenanceOperation,
} from './query-delete-maintenance';
export type { DeleteMaintenanceOperation } from './query-delete-maintenance';
export type DeleteMaintenanceTask =
| DeleteMaintenanceOperation
| { kind: 'batch'; tasks: DeleteMaintenanceOperation[] };
function executeDeleteMaintenanceOperation(
db: InstanceType<typeof Database>,
task: DeleteMaintenanceOperation,
): void {
switch (task.kind) {
case 'session':
deleteSession(db, task.sessionId);
return;
case 'sessions':
deleteSessions(db, task.sessionIds);
return;
case 'video':
deleteVideo(db, task.videoId);
return;
case 'anime':
deleteAnime(db, task.animeId);
return;
}
}
export function executeDeleteMaintenanceTask(dbPath: string, task: DeleteMaintenanceTask): void {
const db = new Database(dbPath);
try {
applyPragmas(db);
if (task.kind === 'batch') {
deleteMaintenanceBatch(db, task.tasks);
return;
}
executeDeleteMaintenanceOperation(db, task);
} finally {
db.close();
}
}
@@ -0,0 +1,196 @@
import type { DatabaseSync } from './sqlite';
import { rebuildLifetimeSummariesInTransaction } from './lifetime';
import { getRollupGroupsForSessions, refreshRollupsForGroupsInTransaction } from './maintenance';
import {
applyLexicalRemovals,
cleanupUnusedCoverArtBlobHash,
deleteSessionsByIds,
forEachIdChunk,
makePlaceholders,
planLexicalRemovalsForSessions,
SQLITE_ID_CHUNK_SIZE,
type LexicalRemovalPlan,
} from './query-shared';
export type DeleteMaintenanceOperation =
| { kind: 'session'; sessionId: number }
| { kind: 'sessions'; sessionIds: number[] }
| { kind: 'video'; videoId: number }
| { kind: 'anime'; animeId: number };
function addOperationTargets(
operations: DeleteMaintenanceOperation[],
sessionIds: Set<number>,
videoIds: Set<number>,
animeIds: Set<number>,
): void {
for (const operation of operations) {
switch (operation.kind) {
case 'session':
sessionIds.add(operation.sessionId);
break;
case 'sessions':
for (const sessionId of operation.sessionIds) sessionIds.add(sessionId);
break;
case 'video':
videoIds.add(operation.videoId);
break;
case 'anime':
animeIds.add(operation.animeId);
break;
}
}
}
function selectIds(
db: DatabaseSync,
buildSql: (placeholders: string) => string,
params: number[],
column: string,
): number[] {
if (params.length === 0) return [];
const ids: number[] = [];
forEachIdChunk(params, (chunk) => {
const rows = db.prepare(buildSql(makePlaceholders(chunk))).all(...chunk) as Array<
Record<string, number>
>;
for (const row of rows) ids.push(row[column]!);
});
return ids;
}
function planLexicalRemovalsInChunks(db: DatabaseSync, sessionIds: number[]): LexicalRemovalPlan {
const combined: LexicalRemovalPlan = { words: [], kanji: [] };
const merge = (target: LexicalRemovalPlan['words'], source: LexicalRemovalPlan['words']) => {
const byId = new Map(target.map((entry) => [entry.id, entry]));
for (const entry of source) {
const existing = byId.get(entry.id);
if (!existing) {
const added = { ...entry };
target.push(added);
byId.set(entry.id, added);
continue;
}
existing.removedFrequency += entry.removedFrequency;
if (
entry.removedFirstSeenMs !== null &&
(existing.removedFirstSeenMs === null ||
entry.removedFirstSeenMs < existing.removedFirstSeenMs)
) {
existing.removedFirstSeenMs = entry.removedFirstSeenMs;
}
if (
entry.removedLastSeenMs !== null &&
(existing.removedLastSeenMs === null ||
entry.removedLastSeenMs > existing.removedLastSeenMs)
) {
existing.removedLastSeenMs = entry.removedLastSeenMs;
}
}
};
forEachIdChunk(sessionIds, (chunk) => {
const plan = planLexicalRemovalsForSessions(db, chunk);
merge(combined.words, plan.words);
merge(combined.kanji, plan.kanji);
});
return combined;
}
export function deleteMaintenanceBatch(
db: DatabaseSync,
operations: DeleteMaintenanceOperation[],
): void {
if (operations.length === 0) return;
db.exec('BEGIN IMMEDIATE');
try {
const sessionIds = new Set<number>();
const videoIds = new Set<number>();
const animeIds = new Set<number>();
addOperationTargets(operations, sessionIds, videoIds, animeIds);
const animeIdList = [...animeIds];
for (const videoId of selectIds(
db,
(placeholders) => `SELECT video_id FROM imm_videos WHERE anime_id IN (${placeholders})`,
animeIdList,
'video_id',
)) {
videoIds.add(videoId);
}
const videoIdList = [...videoIds];
for (const sessionId of selectIds(
db,
(placeholders) => `SELECT session_id FROM imm_sessions WHERE video_id IN (${placeholders})`,
videoIdList,
'session_id',
)) {
sessionIds.add(sessionId);
}
const sessionIdList = [...sessionIds];
const lexicalRemovals = planLexicalRemovalsInChunks(db, sessionIdList);
const affectedRollupGroups = sessionIdList
.flatMap((_, index) =>
index % SQLITE_ID_CHUNK_SIZE === 0
? getRollupGroupsForSessions(db, sessionIdList.slice(index, index + SQLITE_ID_CHUNK_SIZE))
: [],
)
.filter((group) => !videoIds.has(group.videoId));
const coverBlobHashes = new Set<string>();
if (videoIdList.length > 0) {
forEachIdChunk(videoIdList, (chunk) => {
const placeholders = makePlaceholders(chunk);
const artRows = db
.prepare(
`SELECT cover_blob_hash AS coverBlobHash
FROM imm_media_art
WHERE video_id IN (${placeholders}) AND cover_blob_hash IS NOT NULL`,
)
.all(...chunk) as Array<{ coverBlobHash: string }>;
for (const row of artRows) coverBlobHashes.add(row.coverBlobHash);
});
deleteSessionsByIds(db, sessionIdList);
forEachIdChunk(videoIdList, (chunk) => {
const placeholders = makePlaceholders(chunk);
db.prepare(`DELETE FROM imm_subtitle_lines WHERE video_id IN (${placeholders})`).run(
...chunk,
);
db.prepare(`DELETE FROM imm_daily_rollups WHERE video_id IN (${placeholders})`).run(
...chunk,
);
db.prepare(`DELETE FROM imm_monthly_rollups WHERE video_id IN (${placeholders})`).run(
...chunk,
);
db.prepare(`DELETE FROM imm_media_art WHERE video_id IN (${placeholders})`).run(...chunk);
db.prepare(`DELETE FROM imm_videos WHERE video_id IN (${placeholders})`).run(...chunk);
});
} else {
deleteSessionsByIds(db, sessionIdList);
}
for (const coverBlobHash of coverBlobHashes) {
cleanupUnusedCoverArtBlobHash(db, coverBlobHash);
}
if (animeIdList.length > 0) {
forEachIdChunk(animeIdList, (chunk) => {
const placeholders = makePlaceholders(chunk);
db.prepare(`DELETE FROM imm_lifetime_anime WHERE anime_id IN (${placeholders})`).run(
...chunk,
);
db.prepare(`DELETE FROM imm_anime WHERE anime_id IN (${placeholders})`).run(...chunk);
});
}
applyLexicalRemovals(db, lexicalRemovals);
rebuildLifetimeSummariesInTransaction(db);
refreshRollupsForGroupsInTransaction(db, affectedRollupGroups);
db.exec('COMMIT');
} catch (error) {
db.exec('ROLLBACK');
throw error;
}
}
@@ -80,6 +80,14 @@ export function makePlaceholders(values: number[]): string {
return values.map(() => '?').join(','); return values.map(() => '?').join(',');
} }
export const SQLITE_ID_CHUNK_SIZE = 1_000;
export function forEachIdChunk(ids: number[], callback: (chunk: number[]) => void): void {
for (let start = 0; start < ids.length; start += SQLITE_ID_CHUNK_SIZE) {
callback(ids.slice(start, start + SQLITE_ID_CHUNK_SIZE));
}
}
export function resolvedCoverBlobExpr(mediaAlias: string, blobStoreAlias: string): string { export function resolvedCoverBlobExpr(mediaAlias: string, blobStoreAlias: string): string {
return `COALESCE(${blobStoreAlias}.cover_blob, CASE WHEN ${mediaAlias}.cover_blob_hash IS NULL THEN ${mediaAlias}.cover_blob ELSE NULL END)`; return `COALESCE(${blobStoreAlias}.cover_blob, CASE WHEN ${mediaAlias}.cover_blob_hash IS NULL THEN ${mediaAlias}.cover_blob ELSE NULL END)`;
} }
@@ -490,17 +498,19 @@ export function deleteSessionsByIds(db: DatabaseSync, sessionIds: number[]): voi
return; return;
} }
const placeholders = makePlaceholders(sessionIds); forEachIdChunk(sessionIds, (chunk) => {
const placeholders = makePlaceholders(chunk);
db.prepare(`DELETE FROM imm_subtitle_lines WHERE session_id IN (${placeholders})`).run( db.prepare(`DELETE FROM imm_subtitle_lines WHERE session_id IN (${placeholders})`).run(
...sessionIds, ...chunk,
); );
db.prepare(`DELETE FROM imm_session_telemetry WHERE session_id IN (${placeholders})`).run( db.prepare(`DELETE FROM imm_session_telemetry WHERE session_id IN (${placeholders})`).run(
...sessionIds, ...chunk,
); );
db.prepare(`DELETE FROM imm_session_events WHERE session_id IN (${placeholders})`).run( db.prepare(`DELETE FROM imm_session_events WHERE session_id IN (${placeholders})`).run(
...sessionIds, ...chunk,
); );
db.prepare(`DELETE FROM imm_sessions WHERE session_id IN (${placeholders})`).run(...sessionIds); db.prepare(`DELETE FROM imm_sessions WHERE session_id IN (${placeholders})`).run(...chunk);
});
} }
export function toDbMs(ms: number | bigint): bigint { export function toDbMs(ms: number | bigint): bigint {
+191
View File
@@ -0,0 +1,191 @@
/*
* Duplicate/animation-burst collapsing for parsed subtitle cues.
*
* Split out of the cue parser so the parsing rules and the "is this run one animation?"
* heuristics can be read -- and tested -- on their own. The parser owns the cue shape;
* this module only decides which cues survive.
*/
import { hasAssTemporalOverride, isAnimatedAssEffectKind } from './ass-text';
import type {
AnnotatedSubtitleCue,
SubtitleCue,
SubtitleSourceFormat,
} from './subtitle-cue-parser';
// Back-to-back frames of the same animation are authored flush against each other; a
// tiny tolerance absorbs the centisecond rounding of the ASS timestamp format.
const DUPLICATE_CUE_GAP_TOLERANCE_SECONDS = 0.05;
// A burst is a *sequence*. Two adjacent events are two events, not an animation --
// characters do repeat each other, and a repeated line can legitimately be short.
const MIN_BURST_EVENTS = 3;
// Real dialogue holds on screen for about a second, so a run with a couple of much
// shorter events among them looks like frames. Used only alongside authoring evidence.
const ANIMATION_FRAME_MAX_SECONDS = 0.3;
// A karaoke run usually ends on a long "hold" frame, so not every event is short.
const MIN_TAGGED_BURST_FRAMES = 2;
// SRT and VTT carry no authoring metadata at all, so timing is the only signal available
// -- which makes it the easiest one to get wrong. ASS->SRT conversion leaves frames at
// ~0.04s, well under any real utterance, and a burst leaves many of them behind. Both
// bounds are deliberately far stricter than the ASS path: a run of ordinary short lines
// (`えっ` traded between characters) must not clear them.
const TIMING_ONLY_FRAME_MAX_SECONDS = 0.1;
const MIN_TIMING_ONLY_FRAMES = 5;
function cueKey(cue: SubtitleCue): string {
return `${cue.startTime}|${cue.endTime}|${cue.text}`;
}
/**
* Identical text over an identical span is redundant however it was authored -- most
* often a layered ASS event stacking a shadow copy under the visible one.
*/
function collapseExactDuplicates(cues: AnnotatedSubtitleCue[]): AnnotatedSubtitleCue[] {
const seen = new Set<string>();
return cues.filter((cue) => {
const key = cueKey(cue);
if (seen.has(key)) {
return false;
}
seen.add(key);
return true;
});
}
function countFramesShorterThan(run: AnnotatedSubtitleCue[], maxSeconds: number): number {
return run.filter((cue) => cue.endTime - cue.startTime < maxSeconds).length;
}
/**
* Evidence that a run of ASS events is one animation rather than several authored lines.
* A static tag says nothing on its own -- three events sharing one `\clip(...)` are three
* signs -- so the tag has to be temporal by nature (`\t`, `\move`, karaoke timing, or
* anything wrapped in `\t(...)`), an animated `Effect` column, or a value that actually
* changes from event to event, which is how per-frame typesetting is authored.
*/
export function hasAssAnimationEvidence(run: AnnotatedSubtitleCue[]): boolean {
if (run.every((cue) => hasAssTemporalOverride(cue.overrides))) {
return true;
}
if (run.every((cue) => isAnimatedAssEffectKind(cue.effectKind))) {
return true;
}
const [first] = run;
const everyEventTypeset = run.every((cue) => cue.overrides.length > 0);
const signatureChanges = run.some((cue) => cue.overrideSignature !== first!.overrideSignature);
return everyEventTypeset && signatureChanges;
}
export function isAnimationBurst(
run: AnnotatedSubtitleCue[],
format: SubtitleSourceFormat,
): boolean {
if (run.length < MIN_BURST_EVENTS) {
return false;
}
if (format === 'srt') {
return (
run.length >= MIN_TIMING_ONLY_FRAMES &&
countFramesShorterThan(run, TIMING_ONLY_FRAME_MAX_SECONDS) === run.length
);
}
if (countFramesShorterThan(run, ANIMATION_FRAME_MAX_SECONDS) < MIN_TAGGED_BURST_FRAMES) {
return false;
}
// One animation belongs to one styled, one named source line. Two characters trading
// the same short word are two styles or two actors, and never merge.
const [first] = run;
if (run.some((cue) => cue.style !== first!.style || cue.name !== first!.name)) {
return false;
}
return hasAssAnimationEvidence(run);
}
/**
* Karaoke and sign typesetting emits one Dialogue event per animation frame, all carrying
* the same visible text over a contiguous span. Collapse each such run into a single cue.
*
* Only runs that look like animation collapse. Two ordinary lines that happen to repeat
* -- several characters each saying `おはよう` in turn, a positioned sign redrawn with a
* different fade -- stay separate, because merging them would destroy real mineable lines.
*/
function collapseAnimationBursts(
cues: AnnotatedSubtitleCue[],
format: SubtitleSourceFormat,
): AnnotatedSubtitleCue[] {
const indicesByText = new Map<string, number[]>();
cues.forEach((cue, index) => {
const bucket = indicesByText.get(cue.text);
if (bucket) {
bucket.push(index);
} else {
indicesByText.set(cue.text, [index]);
}
});
const dropped = new Set<number>();
const extendedEnd = new Map<number, number>();
for (const indices of indicesByText.values()) {
if (indices.length < MIN_BURST_EVENTS) {
continue;
}
let runStart = 0;
while (runStart < indices.length) {
let runEnd = runStart;
let chainEnd = cues[indices[runStart]!]!.endTime;
while (runEnd + 1 < indices.length) {
const next = cues[indices[runEnd + 1]!]!;
if (next.startTime > chainEnd + DUPLICATE_CUE_GAP_TOLERANCE_SECONDS) {
break;
}
chainEnd = Math.max(chainEnd, next.endTime);
runEnd += 1;
}
const run = indices.slice(runStart, runEnd + 1).map((index) => cues[index]!);
if (isAnimationBurst(run, format)) {
for (let i = runStart + 1; i <= runEnd; i += 1) {
dropped.add(indices[i]!);
}
extendedEnd.set(indices[runStart]!, chainEnd);
}
runStart = runEnd + 1;
}
}
if (dropped.size === 0) {
return cues;
}
const merged: AnnotatedSubtitleCue[] = [];
cues.forEach((cue, index) => {
if (dropped.has(index)) {
return;
}
const end = extendedEnd.get(index);
merged.push(end !== undefined && end > cue.endTime ? { ...cue, endTime: end } : cue);
});
return merged;
}
/**
* Collapse redundant cues. Input must already be sorted by non-decreasing `startTime`,
* ties broken by `endTime` then source `order` -- burst detection chains events by
* comparing each one against the running end of the events before it, so an unsorted
* list breaks runs apart and leaves the frames behind.
*/
export function mergeDuplicateCues(
cues: AnnotatedSubtitleCue[],
format: SubtitleSourceFormat,
): AnnotatedSubtitleCue[] {
return collapseAnimationBursts(collapseExactDuplicates(cues), format);
}
+393 -3
View File
@@ -91,6 +91,17 @@ test('parseSrtCues skips malformed timing lines gracefully', () => {
assert.equal(cues[0]!.text, '有効'); assert.equal(cues[0]!.text, '有効');
}); });
test('parseSubtitleCues strips complete brace blocks from SRT and VTT text', () => {
const content = ['1', '00:00:01,000 --> 00:00:02,000', '彼は{謎}と言った', ''].join('\n');
for (const filename of ['test.srt', 'test.vtt']) {
const cues = parseSubtitleCues(content, filename);
assert.equal(cues.length, 1, filename);
assert.equal(cues[0]!.text, '彼はと言った', filename);
}
});
test('parseAssCues parses basic ASS dialogue lines', () => { test('parseAssCues parses basic ASS dialogue lines', () => {
const content = [ const content = [
'[Script Info]', '[Script Info]',
@@ -137,7 +148,9 @@ test('parseAssCues handles text containing commas', () => {
assert.equal(cues[0]!.text, 'はい、そうです、ね'); assert.equal(cues[0]!.text, 'はい、そうです、ね');
}); });
test('parseAssCues handles \\N line breaks', () => { test('parseAssCues decodes \\N line breaks into real newlines', () => {
// ASS is decoded once, here at ingestion, so cue text matches what mpv hands over for
// the same line played live.
const content = [ const content = [
'[Events]', '[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text', 'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
@@ -146,7 +159,7 @@ test('parseAssCues handles \\N line breaks', () => {
const cues = parseAssCues(content); const cues = parseAssCues(content);
assert.equal(cues[0]!.text, '一行目\\N二行目'); assert.equal(cues[0]!.text, '一行目\n二行目');
}); });
test('parseAssCues strips HTML-like markup while preserving ASS line breaks', () => { test('parseAssCues strips HTML-like markup while preserving ASS line breaks', () => {
@@ -158,7 +171,46 @@ test('parseAssCues strips HTML-like markup while preserving ASS line breaks', ()
const cues = parseAssCues(content); const cues = parseAssCues(content);
assert.equal(cues[0]!.text, '一行目\\N二行目'); assert.equal(cues[0]!.text, '一行目\n二行目');
});
test('parseAssCues drops vector drawing runs enabled by \\p', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 1,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
'Dialogue: 0,0:00:05.00,0:00:08.00,Default,,0,0,0,,これは字幕',
].join('\n');
const cues = parseAssCues(content);
assert.equal(cues.length, 1);
assert.equal(cues[0]!.text, 'これは字幕');
});
test('parseAssCues keeps text that follows a \\p0 reset on the same line', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\p1}m 0 0 l 10 10{\\p0}本文{\\p1}m 5 5 l 6 6{\\p0}続き',
].join('\n');
const cues = parseAssCues(content);
assert.equal(cues.length, 1);
assert.equal(cues[0]!.text, '本文続き');
});
test('parseAssCues leaves \\pos untouched when no drawing mode is active', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\pos(960,1068)\\bord3}位置指定',
].join('\n');
const cues = parseAssCues(content);
assert.equal(cues[0]!.text, '位置指定');
}); });
test('parseAssCues returns empty for content without Events section', () => { test('parseAssCues returns empty for content without Events section', () => {
@@ -258,6 +310,344 @@ test('parseSubtitleCues returns cues sorted by start time', () => {
assert.equal(cues[1]!.text, '二番目'); assert.equal(cues[1]!.text, '二番目');
}); });
test('parseSubtitleCues collapses per-frame karaoke duplicates into one cue', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}過ぎ去ってしまう瞬間を',
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}過ぎ去ってしまう瞬間を',
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,,{\\clip(m 3 3)}過ぎ去ってしまう瞬間を',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 1);
assert.equal(cues[0]!.startTime, 1.0);
assert.equal(cues[0]!.endTime, 3.55);
assert.equal(cues[0]!.text, '過ぎ去ってしまう瞬間を');
});
test('parseSubtitleCues keeps back-to-back plain dialogue repeats separate', () => {
// Several characters greeting in turn: distinct utterances that happen to abut.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:04:05.67,0:04:06.82,Dial_JP,,0,0,0,,おはよう',
'Dialogue: 0,0:04:06.82,0:04:07.56,Dial_JP,,0,0,0,,おはよう',
'Dialogue: 0,0:04:07.56,0:04:08.78,Dial_JP,,0,0,0,,おはよう',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
assert.equal(cues[0]!.endTime, 246.82);
assert.equal(cues[2]!.startTime, 247.56);
});
test('parseSubtitleCues collapses exact duplicate cues even without effect tags', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,重なった行',
'Dialogue: 1,0:00:01.00,0:00:04.00,Default,,0,0,0,,重なった行',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 1);
});
test('parseSubtitleCues collapses tag-less animation frames in converted SRT', () => {
// ASS -> SRT conversion drops override tags, so only the ~0.04s frame timing remains.
const lines = ['1', '00:00:07,870 --> 00:00:07,910', 'Kaguya Wants to be Confessed to', ''];
for (let i = 1; i < 8; i++) {
const start = 7910 + (i - 1) * 40;
const end = start + 40;
const at = (ms: number) =>
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
lines.push(String(i + 1), `${at(start)} --> ${at(end)}`, 'Kaguya Wants to be Confessed to', '');
}
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
assert.equal(cues.length, 1);
assert.equal(cues[0]!.startTime, 7.87);
});
test('parseSubtitleCues keeps identical lines that recur far apart', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:02.00,Default,,0,0,0,,なんで',
'Dialogue: 0,0:05:00.00,0:05:01.00,Default,,0,0,0,,なんで',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 2);
assert.equal(cues[0]!.startTime, 1.0);
assert.equal(cues[1]!.startTime, 300.0);
});
test('parseSubtitleCues keeps two positioned signs that repeat the same text', () => {
// Both carry override tags, but `\pos` and `\fad` are static placement, not animation.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:01:00.00,0:01:03.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(200,200)}第一話',
'Dialogue: 0,0:01:03.00,0:01:06.00,Sign,,0,0,0,,{\\pos(960,900)\\fad(200,200)}第一話',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 2);
assert.equal(cues[1]!.startTime, 63.0);
});
test('parseSubtitleCues keeps a run of ordinary positioned lines separate', () => {
// Three events is a sequence, but none of them runs at animation-frame speed.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:01:00.00,0:01:02.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
'Dialogue: 0,0:01:02.00,0:01:04.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
'Dialogue: 0,0:01:04.00,0:01:06.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues keeps a short repeated SRT pair without burst evidence', () => {
const content = [
'1',
'00:00:01,000 --> 00:00:01,200',
'えっ',
'',
'2',
'00:00:01,200 --> 00:00:01,400',
'えっ',
'',
].join('\n');
const cues = parseSubtitleCues(content, 'test.srt');
assert.equal(cues.length, 2);
});
test('parseSubtitleCues collapses a burst marked only by the Effect column', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,Karaoke,歌詞',
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,Karaoke,歌詞',
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,Karaoke,歌詞',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 1);
assert.equal(cues[0]!.endTime, 3.55);
});
test('parseSubtitleCues keeps a second karaoke burst that starts after a gap', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}リフレイン',
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}リフレイン',
'Dialogue: 0,0:00:01.09,0:00:03.00,OP_JP,,0,0,0,,{\\clip(m 3 3)}リフレイン',
'Dialogue: 0,0:00:20.00,0:00:20.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}リフレイン',
'Dialogue: 0,0:00:20.05,0:00:20.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}リフレイン',
'Dialogue: 0,0:00:20.09,0:00:22.00,OP_JP,,0,0,0,,{\\clip(m 3 3)}リフレイン',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 2);
assert.equal(cues[0]!.endTime, 3.0);
assert.equal(cues[1]!.startTime, 20.0);
assert.equal(cues[1]!.endTime, 22.0);
});
test('parseSubtitleCues does not merge a burst into unrelated dialogue between frames', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}歌詞',
'Dialogue: 0,0:00:01.02,0:00:03.00,Dial_JP,,0,0,0,,別のセリフ',
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}歌詞',
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,,{\\clip(m 3 3)}歌詞',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 2);
assert.deepEqual(
cues.map((cue) => cue.text),
['歌詞', '別のセリフ'],
);
assert.equal(cues[0]!.endTime, 3.55);
});
test('parseSubtitleCues keeps rapid ASS lines from different actors separate', () => {
// Three 200ms `えっ` reactions traded between characters. Fast, adjacent and identical,
// but authored as three lines: different styles and different actors.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_A,アリス,0,0,0,,えっ',
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_B,ボブ,0,0,0,,えっ',
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_C,キャロル,0,0,0,,えっ',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues reads the speaker column when it is spelled Actor', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Actor, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_JP,アリス,0,0,0,,えっ',
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_JP,ボブ,0,0,0,,えっ',
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_JP,キャロル,0,0,0,,えっ',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues does not treat a custom Effect name as animation', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,scrolling-credit,制作',
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,scrolling-credit,制作',
'Dialogue: 0,0:00:01.40,0:00:01.60,Sign,,0,0,0,scrolling-credit,制作',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues keeps rapid ASS lines that share a style but not an actor', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_JP,アリス,0,0,0,,えっ',
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_JP,ボブ,0,0,0,,えっ',
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_JP,キャロル,0,0,0,,えっ',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues keeps untagged rapid ASS repeats separate', () => {
// No overrides at all: timing-only evidence is an SRT/VTT fallback and must not apply
// to ASS, where the absence of typesetting is itself evidence of plain dialogue.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.05,Dial_JP,,0,0,0,,えっ',
'Dialogue: 0,0:00:01.05,0:00:01.10,Dial_JP,,0,0,0,,えっ',
'Dialogue: 0,0:00:01.10,0:00:01.15,Dial_JP,,0,0,0,,えっ',
'Dialogue: 0,0:00:01.15,0:00:01.20,Dial_JP,,0,0,0,,えっ',
'Dialogue: 0,0:00:01.20,0:00:01.25,Dial_JP,,0,0,0,,えっ',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 5);
});
test('parseSubtitleCues keeps repeated signs sharing one static clip', () => {
// `\clip` is a static shape for the event. Three events with the identical clip were
// typeset the same way, so none of them is a frame of the others.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
'Dialogue: 0,0:00:01.40,0:00:01.60,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 3);
});
test('parseSubtitleCues collapses a sign animated through \\t', () => {
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
'Dialogue: 0,0:00:01.40,0:00:03.00,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
].join('\n');
const cues = parseSubtitleCues(content, 'test.ass');
assert.equal(cues.length, 1);
assert.equal(cues[0]!.endTime, 3.0);
});
test('parseSubtitleCues keeps a short repeated SRT run above the frame threshold', () => {
// Five contiguous 200ms cues: a sequence, but nowhere near animation-frame speed.
const lines: string[] = [];
for (let i = 0; i < 5; i++) {
const start = 1000 + i * 200;
const at = (ms: number) =>
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
lines.push(String(i + 1), `${at(start)} --> ${at(start + 200)}`, 'えっ', '');
}
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
assert.equal(cues.length, 5);
});
test('parseSubtitleCues keeps a short SRT frame run below the minimum length', () => {
// Four 40ms frames: frame-speed, but too few to tell an animation from an artefact.
const lines: string[] = [];
for (let i = 0; i < 4; i++) {
const start = 7870 + i * 40;
const at = (ms: number) =>
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
lines.push(String(i + 1), `${at(start)} --> ${at(start + 40)}`, 'タイトル', '');
}
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
assert.equal(cues.length, 4);
});
test('parseSubtitleCues applies ASS burst rules to ASS content behind an .srt filename', () => {
// The extension lies, so the SRT parser finds nothing and the content-sniffing fallback
// takes over -- which has to carry the `ass` source format with it, or the far stricter
// timing-only thresholds would let this karaoke burst through as three cues.
const content = [
'[Events]',
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
'Dialogue: 0,0:00:01.00,0:00:01.20,Karaoke,,0,0,0,,{\\k20}歌詞',
'Dialogue: 0,0:00:01.20,0:00:01.40,Karaoke,,0,0,0,,{\\k20}歌詞',
'Dialogue: 0,0:00:01.40,0:00:03.00,Karaoke,,0,0,0,,{\\k20}歌詞',
].join('\n');
const cues = parseSubtitleCues(content, 'test.srt');
assert.equal(cues.length, 1);
assert.equal(cues[0]!.startTime, 1.0);
assert.equal(cues[0]!.endTime, 3.0);
assert.equal(cues[0]!.text, '歌詞');
});
test('parseSubtitleCues detects subtitle formats from remote URLs', () => { test('parseSubtitleCues detects subtitle formats from remote URLs', () => {
const assContent = [ const assContent = [
'[Events]', '[Events]',
+160 -34
View File
@@ -1,9 +1,46 @@
import {
assOverrideSignature,
assToPlainText,
collectAssOverrideCommands,
parseAssEffectField,
type AssEffectKind,
type AssOverrideCommand,
} from './ass-text';
import { mergeDuplicateCues } from './subtitle-cue-dedup';
export interface SubtitleCue { export interface SubtitleCue {
startTime: number; startTime: number;
endTime: number; endTime: number;
text: string; text: string;
} }
/**
* Everything the parser knows about a source event, shared only with the dedup engine.
* Deduplication needs the authoring context -- which style the line belongs to, which
* override commands it carries, whether the `Effect` column was set -- to tell a karaoke
* burst apart from two characters saying the same word in turn. None of it is meaningful
* outside the parser, so the public API stays `{startTime, endTime, text}`.
*/
export interface AnnotatedSubtitleCue extends SubtitleCue {
/** Text exactly as authored, override blocks and all. */
rawText: string;
style: string;
layer: number;
/** ASS `Name`/`Actor` column. */
name: string;
/** ASS `Effect` column, verbatim. */
effect: string;
effectKind: AssEffectKind;
/** Override commands found in `{...}` blocks, with their arguments. */
overrides: readonly AssOverrideCommand[];
/** Canonical form of `overrides`, for spotting values that change across a run. */
overrideSignature: string;
/** Position in the source file, so sorting by time stays deterministic across layers. */
order: number;
}
export type SubtitleSourceFormat = 'ass' | 'srt';
const HTML_SUBTITLE_TAG_PATTERN = /<\/?[A-Za-z][^>\n]*>/g; const HTML_SUBTITLE_TAG_PATTERN = /<\/?[A-Za-z][^>\n]*>/g;
const SRT_TIMING_PATTERN = const SRT_TIMING_PATTERN =
@@ -23,12 +60,21 @@ function parseTimestamp(
); );
} }
/**
* The single ASS decode for the file path: cues leave the parser as plain text with real
* line breaks, matching what mpv hands over for the same line played live. No layer
* downstream decodes ASS again.
*/
function sanitizeSubtitleCueText(text: string): string { function sanitizeSubtitleCueText(text: string): string {
return text.replace(ASS_OVERRIDE_TAG_PATTERN, '').replace(HTML_SUBTITLE_TAG_PATTERN, '').trim(); return assToPlainText(text, '\n').replace(HTML_SUBTITLE_TAG_PATTERN, '').trim();
} }
export function parseSrtCues(content: string): SubtitleCue[] { function toPublicCues(cues: AnnotatedSubtitleCue[]): SubtitleCue[] {
const cues: SubtitleCue[] = []; return cues.map(({ startTime, endTime, text }) => ({ startTime, endTime, text }));
}
function parseAnnotatedSrtCues(content: string): AnnotatedSubtitleCue[] {
const cues: AnnotatedSubtitleCue[] = [];
const lines = content.split(/\r?\n/); const lines = content.split(/\r?\n/);
let i = 0; let i = 0;
@@ -60,20 +106,39 @@ export function parseSrtCues(content: string): SubtitleCue[] {
i += 1; i += 1;
} }
const text = sanitizeSubtitleCueText(textLines.join('\n')); const rawText = textLines.join('\n');
const text = sanitizeSubtitleCueText(rawText);
if (text) { if (text) {
cues.push({ startTime, endTime, text }); cues.push({
startTime,
endTime,
text,
rawText,
style: '',
layer: 0,
name: '',
effect: '',
effectKind: 'none',
// SRT and VTT carry no authoring metadata, and the dedup engine never reads
// overrides for those formats -- collecting them would be parsing for nobody.
overrides: [],
overrideSignature: '',
order: cues.length,
});
} }
} }
return cues; return cues;
} }
const ASS_OVERRIDE_TAG_PATTERN = /\{[^}]*\}/g; export function parseSrtCues(content: string): SubtitleCue[] {
return toPublicCues(parseAnnotatedSrtCues(content));
}
const ASS_TIMING_PATTERN = /^(\d+):(\d{2}):(\d{2})\.(\d{1,2})$/; const ASS_TIMING_PATTERN = /^(\d+):(\d{2}):(\d{2})\.(\d{1,2})$/;
const ASS_FORMAT_PREFIX = 'Format:'; const ASS_FORMAT_PREFIX = 'Format:';
const ASS_DIALOGUE_PREFIX = 'Dialogue:'; const ASS_DIALOGUE_PREFIX = 'Dialogue:';
const ASS_NAME_FIELD_ALIASES = ['name', 'actor'];
function parseAssTimestamp(raw: string): number | null { function parseAssTimestamp(raw: string): number | null {
const match = ASS_TIMING_PATTERN.exec(raw.trim()); const match = ASS_TIMING_PATTERN.exec(raw.trim());
@@ -87,13 +152,43 @@ function parseAssTimestamp(raw: string): number | null {
return hours * 3600 + minutes * 60 + seconds + centiseconds / 100; return hours * 3600 + minutes * 60 + seconds + centiseconds / 100;
} }
export function parseAssCues(content: string): SubtitleCue[] { function readField(fields: string[], index: number): string {
const cues: SubtitleCue[] = []; return index >= 0 && index < fields.length ? fields[index]!.trim() : '';
}
function findFieldIndex(formatFields: string[], aliases: string[]): number {
for (const alias of aliases) {
const index = formatFields.indexOf(alias);
if (index >= 0) {
return index;
}
}
return -1;
}
function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
const cues: AnnotatedSubtitleCue[] = [];
const lines = content.split(/\r?\n/); const lines = content.split(/\r?\n/);
let inEventsSection = false; let inEventsSection = false;
let startFieldIndex = -1; const fieldIndex = {
let endFieldIndex = -1; start: -1,
let textFieldIndex = -1; end: -1,
text: -1,
style: -1,
layer: -1,
name: -1,
effect: -1,
};
const resetFieldIndex = () => {
fieldIndex.start = -1;
fieldIndex.end = -1;
fieldIndex.text = -1;
fieldIndex.style = -1;
fieldIndex.layer = -1;
fieldIndex.name = -1;
fieldIndex.effect = -1;
};
for (const line of lines) { for (const line of lines) {
const trimmed = line.trim(); const trimmed = line.trim();
@@ -101,9 +196,7 @@ export function parseAssCues(content: string): SubtitleCue[] {
if (trimmed.startsWith('[') && trimmed.endsWith(']')) { if (trimmed.startsWith('[') && trimmed.endsWith(']')) {
inEventsSection = trimmed.toLowerCase() === '[events]'; inEventsSection = trimmed.toLowerCase() === '[events]';
if (!inEventsSection) { if (!inEventsSection) {
startFieldIndex = -1; resetFieldIndex();
endFieldIndex = -1;
textFieldIndex = -1;
} }
continue; continue;
} }
@@ -117,9 +210,15 @@ export function parseAssCues(content: string): SubtitleCue[] {
.slice(ASS_FORMAT_PREFIX.length) .slice(ASS_FORMAT_PREFIX.length)
.split(',') .split(',')
.map((field) => field.trim().toLowerCase()); .map((field) => field.trim().toLowerCase());
startFieldIndex = formatFields.indexOf('start'); fieldIndex.start = formatFields.indexOf('start');
endFieldIndex = formatFields.indexOf('end'); fieldIndex.end = formatFields.indexOf('end');
textFieldIndex = formatFields.indexOf('text'); fieldIndex.text = formatFields.indexOf('text');
fieldIndex.style = formatFields.indexOf('style');
fieldIndex.layer = formatFields.indexOf('layer');
// Aegisub writes the speaker column as `Actor`; the v4+ spec calls it `Name`.
// Missing it costs the burst check its speaker guard, so both spellings count.
fieldIndex.name = findFieldIndex(formatFields, ASS_NAME_FIELD_ALIASES);
fieldIndex.effect = formatFields.indexOf('effect');
continue; continue;
} }
@@ -127,34 +226,57 @@ export function parseAssCues(content: string): SubtitleCue[] {
continue; continue;
} }
if (startFieldIndex < 0 || endFieldIndex < 0 || textFieldIndex < 0) { if (fieldIndex.start < 0 || fieldIndex.end < 0 || fieldIndex.text < 0) {
continue; continue;
} }
const fields = trimmed.slice(ASS_DIALOGUE_PREFIX.length).split(','); const fields = trimmed.slice(ASS_DIALOGUE_PREFIX.length).split(',');
if ( if (
startFieldIndex >= fields.length || fieldIndex.start >= fields.length ||
endFieldIndex >= fields.length || fieldIndex.end >= fields.length ||
textFieldIndex >= fields.length fieldIndex.text >= fields.length
) { ) {
continue; continue;
} }
const startTime = parseAssTimestamp(fields[startFieldIndex]!); const startTime = parseAssTimestamp(fields[fieldIndex.start]!);
const endTime = parseAssTimestamp(fields[endFieldIndex]!); const endTime = parseAssTimestamp(fields[fieldIndex.end]!);
if (startTime === null || endTime === null) { if (startTime === null || endTime === null) {
continue; continue;
} }
const text = sanitizeSubtitleCueText(fields.slice(textFieldIndex).join(',')); const rawText = fields.slice(fieldIndex.text).join(',');
if (text) { const text = sanitizeSubtitleCueText(rawText);
cues.push({ startTime, endTime, text }); if (!text) {
continue;
} }
const effect = readField(fields, fieldIndex.effect);
const layer = Number(readField(fields, fieldIndex.layer));
const overrides = collectAssOverrideCommands(rawText);
cues.push({
startTime,
endTime,
text,
rawText,
style: readField(fields, fieldIndex.style),
layer: Number.isFinite(layer) ? layer : 0,
name: readField(fields, fieldIndex.name),
effect,
effectKind: parseAssEffectField(effect),
overrides,
overrideSignature: assOverrideSignature(overrides),
order: cues.length,
});
} }
return cues; return cues;
} }
export function parseAssCues(content: string): SubtitleCue[] {
return toPublicCues(parseAnnotatedAssCues(content));
}
function detectSubtitleFormat(source: string): 'srt' | 'vtt' | 'ass' | 'ssa' | null { function detectSubtitleFormat(source: string): 'srt' | 'vtt' | 'ass' | 'ssa' | null {
const [normalizedSource = source] = const [normalizedSource = source] =
(() => { (() => {
@@ -173,27 +295,31 @@ function detectSubtitleFormat(source: string): 'srt' | 'vtt' | 'ass' | 'ssa' | n
export function parseSubtitleCues(content: string, filename: string): SubtitleCue[] { export function parseSubtitleCues(content: string, filename: string): SubtitleCue[] {
const format = detectSubtitleFormat(filename); const format = detectSubtitleFormat(filename);
let cues: SubtitleCue[]; let cues: AnnotatedSubtitleCue[];
let sourceFormat: SubtitleSourceFormat = 'srt';
switch (format) { switch (format) {
case 'srt': case 'srt':
case 'vtt': case 'vtt':
cues = parseSrtCues(content); cues = parseAnnotatedSrtCues(content);
break; break;
case 'ass': case 'ass':
case 'ssa': case 'ssa':
cues = parseAssCues(content); cues = parseAnnotatedAssCues(content);
sourceFormat = 'ass';
break; break;
default: default:
cues = []; cues = [];
} }
if (cues.length === 0) { if (cues.length === 0) {
const assCues = parseAssCues(content); const assCues = parseAnnotatedAssCues(content);
const srtCues = parseSrtCues(content); const srtCues = parseAnnotatedSrtCues(content);
cues = assCues.length >= srtCues.length ? assCues : srtCues; const preferAss = assCues.length >= srtCues.length;
cues = preferAss ? assCues : srtCues;
sourceFormat = preferAss && assCues.length > 0 ? 'ass' : 'srt';
} }
cues.sort((a, b) => a.startTime - b.startTime); cues.sort((a, b) => a.startTime - b.startTime || a.endTime - b.endTime || a.order - b.order);
return cues; return toPublicCues(mergeDuplicateCues(cues, sourceFormat));
} }
@@ -115,6 +115,21 @@ test('subtitle processing does not emit plain payload for cached lines', async (
assert.deepEqual(emitted, [{ text: '字幕', tokens: [] }]); assert.deepEqual(emitted, [{ text: '字幕', tokens: [] }]);
}); });
test('text that normalizes to nothing is never cached', () => {
const controller = createSubtitleProcessingController({
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
emitSubtitle: () => {},
});
// Two different inputs both reduce to an empty key; sharing one entry would serve the
// first one's tokens for the second.
controller.preCacheTokenization(' ', { text: ' ', tokens: [] });
assert.equal(controller.hasCachedSubtitle(' '), false);
assert.equal(controller.hasCachedSubtitle('\\n'), false);
assert.equal(controller.consumeCachedSubtitle('\\n'), null);
});
test('subtitle processing shows plain line while tokenization is still pending', async () => { test('subtitle processing shows plain line while tokenization is still pending', async () => {
const emitted: SubtitleData[] = []; const emitted: SubtitleData[] = [];
let resolveTokenization: ((value: SubtitleData | null) => void) | undefined; let resolveTokenization: ((value: SubtitleData | null) => void) | undefined;
@@ -1,4 +1,5 @@
import type { SubtitleData } from '../../types'; import type { SubtitleData } from '../../types';
import { normalizePlainSubtitleText } from './ass-text';
export interface SubtitleProcessingControllerDeps { export interface SubtitleProcessingControllerDeps {
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>; tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
@@ -47,8 +48,15 @@ export interface SubtitleProcessingController {
hasCachedSubtitle: (text: string) => boolean; hasCachedSubtitle: (text: string) => boolean;
} }
/**
* Prefetched cues and live mpv text are both already decoded from ASS, so the key only
* has to settle whitespace for one authored line to resolve to one entry.
*
* An empty key is not a line: it is whatever normalization reduced to nothing. Callers
* must skip the cache for it rather than let every such input share one entry.
*/
export function normalizeSubtitleCacheKey(text: string): string { export function normalizeSubtitleCacheKey(text: string): string {
return text.replace(/\r\n/g, '\n').replace(/\\N/g, '\n').replace(/\\n/g, '\n').trim(); return normalizePlainSubtitleText(text);
} }
export function createSubtitleProcessingController( export function createSubtitleProcessingController(
@@ -72,6 +80,9 @@ export function createSubtitleProcessingController(
const getCachedTokenization = (text: string): SubtitleData | null => { const getCachedTokenization = (text: string): SubtitleData | null => {
const cacheKey = normalizeSubtitleCacheKey(text); const cacheKey = normalizeSubtitleCacheKey(text);
if (!cacheKey) {
return null;
}
const cached = tokenizationCache.get(cacheKey); const cached = tokenizationCache.get(cacheKey);
if (!cached) { if (!cached) {
return null; return null;
@@ -83,7 +94,11 @@ export function createSubtitleProcessingController(
}; };
const setCachedTokenization = (text: string, payload: SubtitleData): void => { const setCachedTokenization = (text: string, payload: SubtitleData): void => {
tokenizationCache.set(normalizeSubtitleCacheKey(text), payload); const cacheKey = normalizeSubtitleCacheKey(text);
if (!cacheKey) {
return;
}
tokenizationCache.set(cacheKey, payload);
while (tokenizationCache.size > SUBTITLE_TOKENIZATION_CACHE_LIMIT) { while (tokenizationCache.size > SUBTITLE_TOKENIZATION_CACHE_LIMIT) {
const firstKey = tokenizationCache.keys().next().value; const firstKey = tokenizationCache.keys().next().value;
if (firstKey !== undefined) { if (firstKey !== undefined) {
@@ -254,7 +269,8 @@ export function createSubtitleProcessingController(
return cached; return cached;
}, },
hasCachedSubtitle: (text: string) => { hasCachedSubtitle: (text: string) => {
return tokenizationCache.has(normalizeSubtitleCacheKey(text)); const cacheKey = normalizeSubtitleCacheKey(text);
return cacheKey.length > 0 && tokenizationCache.has(cacheKey);
}, },
}; };
} }
+4 -2
View File
@@ -1651,9 +1651,11 @@ test('tokenizeSubtitle clears JLPT level from standalone Yomitan particle token'
assert.equal(result.tokens?.[0]?.jlptLevel, undefined); assert.equal(result.tokens?.[0]?.jlptLevel, undefined);
}); });
test('tokenizeSubtitle returns null tokens for empty normalized text', async () => { test('tokenizeSubtitle returns the normalized text when it comes out empty', async () => {
// Handing back the original would push whatever normalization dropped into app state
// as if it were subtitle text.
const result = await tokenizeSubtitle(' \\n ', makeDeps()); const result = await tokenizeSubtitle(' \\n ', makeDeps());
assert.deepEqual(result, { text: ' \\n ', tokens: null }); assert.deepEqual(result, { text: '', tokens: null });
}); });
test('tokenizeSubtitle normalizes newlines before Yomitan parse request', async () => { test('tokenizeSubtitle normalizes newlines before Yomitan parse request', async () => {
+7 -6
View File
@@ -27,6 +27,7 @@ import {
} from './tokenizer/yomitan-parser-runtime'; } from './tokenizer/yomitan-parser-runtime';
import type { YomitanTermFrequency } from './tokenizer/yomitan-parser-runtime'; import type { YomitanTermFrequency } from './tokenizer/yomitan-parser-runtime';
import { isKanaChar } from './tokenizer/token-classification'; import { isKanaChar } from './tokenizer/token-classification';
import { normalizePlainSubtitleText } from './ass-text';
const logger = createLogger('main:tokenizer'); const logger = createLogger('main:tokenizer');
@@ -886,14 +887,14 @@ export async function tokenizeSubtitle(
text: string, text: string,
deps: TokenizerServiceDeps, deps: TokenizerServiceDeps,
): Promise<SubtitleData> { ): Promise<SubtitleData> {
const displayText = text const displayText = normalizePlainSubtitleText(text);
.replace(/\r\n/g, '\n')
.replace(/\\N/g, '\n')
.replace(/\\n/g, '\n')
.trim();
// ASS decoding already happened upstream (cue parser for files, mpv for live text), so
// all this drops is whitespace -- but a whitespace-only line still normalizes to empty.
// Return the normalized form anyway: handing back the original would put a blank line
// into application state as if it were subtitle text.
if (!displayText) { if (!displayText) {
return { text, tokens: null }; return { text: displayText, tokens: null };
} }
const tokenizeText = displayText const tokenizeText = displayText
+18
View File
@@ -1004,6 +1004,24 @@ test('normalizeSubtitle collapses explicit line breaks when collapseLineBreaks i
); );
}); });
test('normalizeSubtitle leaves already-decoded text alone', () => {
// Primary subtitle text is decoded from ASS once, upstream: by mpv for live lines and
// by the cue parser for prefetched ones. A brace that survives that is literal text.
assert.equal(normalizeSubtitle('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
assert.equal(normalizeSubtitle(' 余白 ', false), ' 余白 ');
});
test('prepareSecondarySubtitleLines drops ASS vector drawing runs', () => {
assert.deepEqual(
prepareSecondarySubtitleLines(
'{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
),
[],
);
assert.deepEqual(prepareSecondarySubtitleLines('{\\p1}m 0 0 l 10 10{\\p0}本文'), ['本文']);
assert.deepEqual(prepareSecondarySubtitleLines('{\\pos(960,1068)\\bord3}位置指定'), ['位置指定']);
});
test('shouldRenderTokenizedSubtitle enables token rendering when tokens exist', () => { test('shouldRenderTokenizedSubtitle enables token rendering when tokens exist', () => {
assert.equal(shouldRenderTokenizedSubtitle(5), true); assert.equal(shouldRenderTokenizedSubtitle(5), true);
assert.equal(shouldRenderTokenizedSubtitle(0), false); assert.equal(shouldRenderTokenizedSubtitle(0), false);
+8 -11
View File
@@ -5,6 +5,7 @@ import type {
SubtitleData, SubtitleData,
SubtitleRendererStyleConfig, SubtitleRendererStyleConfig,
} from '../types'; } from '../types';
import { assToPlainText, normalizePlainSubtitleText } from '../core/services/ass-text.js';
import type { RendererContext } from './context'; import type { RendererContext } from './context';
import { PRIMARY_SUB_VISIBLE_ON_YOMITAN_POPUP_CLASS } from './yomitan-popup.js'; import { PRIMARY_SUB_VISIBLE_ON_YOMITAN_POPUP_CLASS } from './yomitan-popup.js';
@@ -42,17 +43,10 @@ function isWhitespaceOnly(value: string): boolean {
return value.trim().length === 0; return value.trim().length === 0;
} }
// Text reaching the overlay has already been decoded from ASS -- by mpv for live lines,
// by the cue parser for prefetched ones -- so this only settles line breaks.
export function normalizeSubtitle(text: string, trim = true, collapseLineBreaks = false): string { export function normalizeSubtitle(text: string, trim = true, collapseLineBreaks = false): string {
if (!text) return ''; return normalizePlainSubtitleText(text, { trim, collapseLineBreaks });
let normalized = text.replace(/\\N/g, '\n').replace(/\\n/g, '\n');
normalized = normalized.replace(/\{[^}]*\}/g, '');
if (collapseLineBreaks) {
normalized = normalized.replace(/\n/g, ' ');
normalized = normalized.replace(/\s+/g, ' ');
}
return trim ? normalized.trim() : normalized;
} }
const HEX_COLOR_PATTERN = /^#(?:[0-9a-fA-F]{3}|[0-9a-fA-F]{4}|[0-9a-fA-F]{6}|[0-9a-fA-F]{8})$/; const HEX_COLOR_PATTERN = /^#(?:[0-9a-fA-F]{3}|[0-9a-fA-F]{4}|[0-9a-fA-F]{6}|[0-9a-fA-F]{8})$/;
@@ -672,7 +666,10 @@ function isKaraokeLikeLineSet(lines: string[]): boolean {
} }
export function prepareSecondarySubtitleLines(text: string): string[] { export function prepareSecondarySubtitleLines(text: string): string[] {
const normalized = normalizeSubtitle(text, true, false); // The one display-side ASS decode: secondary text also reaches the overlay from
// websocket clients that forward their source line untouched, so unlike the primary
// path it cannot assume mpv already decoded it.
const normalized = assToPlainText(text).trim();
if (!normalized) return []; if (!normalized) return [];
+6 -13
View File
@@ -16,6 +16,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>. * along with this program. If not, see <https://www.gnu.org/licenses/>.
*/ */
import { normalizePlainSubtitleText } from './core/services/ass-text';
interface TimingEntry { interface TimingEntry {
startTime: number; startTime: number;
endTime: number; endTime: number;
@@ -191,23 +193,14 @@ export class SubtitleTimingTracker {
return costs[shorter.length] || 0; return costs[shorter.length] || 0;
} }
// Both sides take text mpv has already decoded from ASS; only whitespace differs
// between the lookup key (single line) and the display form (line breaks kept).
private normalizeText(text: string): string { private normalizeText(text: string): string {
return text return normalizePlainSubtitleText(text, { collapseLineBreaks: true });
.replace(/\\N/g, ' ')
.replace(/\\n/g, ' ')
.replace(/\n/g, ' ')
.replace(/{[^}]*}/g, '')
.replace(/\s+/g, ' ')
.trim();
} }
private prepareDisplayText(text: string): string { private prepareDisplayText(text: string): string {
// Convert ASS/SSA newlines to real newlines, strip tags return normalizePlainSubtitleText(text);
return text
.replace(/\\N/g, '\n')
.replace(/\\n/g, '\n')
.replace(/{[^}]*}/g, '')
.trim();
} }
private startCleanup(): void { private startCleanup(): void {