mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-13 01:55:50 -07:00
Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
772635ab96
|
|||
|
675cb33519
|
|||
|
320db591ae
|
|||
| 7b0fbdf254 |
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Kept the stats page and active video player responsive during deletes, and batched concurrent session, episode, and library deletes into one transaction and summary rebuild.
|
||||
@@ -0,0 +1,5 @@
|
||||
type: fixed
|
||||
area: subtitles
|
||||
|
||||
- Heavily typeset ASS scripts (karaoke OP/ED, sign work) no longer fill the subtitle sidebar with garbage. Vector drawing runs (`\p1` … `\p0`) are no longer shown as subtitle text (e.g. `m 20 0 b 10 0 0 10 0 20 …`), and duplicate events in a parsed subtitle file collapse into one cue: identical text over an identical span (layered "shadow" copies), and per-frame animation bursts. An ASS burst has to prove itself with authoring evidence — a temporal tag (`\t`, `\move`, karaoke timing), an animated `Effect` column, or override values that change from event to event — plus one shared style and actor, so three rapid `えっ` reactions from three characters, or a sign repeated with the same static `\clip`, stay separate. SRT and VTT carry no such metadata, so there the run has to be at least five contiguous events all shorter than 0.1s, which is where ASS-to-SRT conversion leaves karaoke frames.
|
||||
- Subtitle text is now decoded from ASS exactly once, where it enters the app, and matches how mpv renders the same line (including `\N`, `\n`, `\h`, unclosed `{`, and the fact that `\{` is not an escape). Renderer, timing tracker, tokenizer and the tokenization cache take that decoded text as-is instead of each re-deriving it, so one authored line can no longer produce two different cache keys, and a cue that normalizes to nothing is no longer stored as subtitle text or cached under an empty key.
|
||||
@@ -64,18 +64,23 @@ External subtitle files only (SRT, VTT, ASS). Embedded subtitle tracks are out o
|
||||
A cue parser extracts both timing and text content from subtitle files for prefetching.
|
||||
|
||||
**Parsed cue structure:**
|
||||
|
||||
```typescript
|
||||
interface SubtitleCue {
|
||||
startTime: number; // seconds
|
||||
endTime: number; // seconds
|
||||
text: string; // raw subtitle text
|
||||
text: string; // plain text, decoded from the source format
|
||||
}
|
||||
```
|
||||
|
||||
**Supported formats:**
|
||||
|
||||
- SRT/VTT: Regex-based parsing of timing lines + text content between timing blocks.
|
||||
- ASS: Parse `[Events]` section, extract `Dialogue:` lines, split on the first 9 commas only (ASS v4+ has 10 fields; the last field is Text which can itself contain commas). Strip ASS override tags (`{\...}`) from the text before storing.
|
||||
ASS text fields contain inline override tags like `{\b1}`, `{\an8}`, `{\fad(200,300)}`. The cue parser strips these during extraction so the tokenizer receives clean text.
|
||||
- ASS: Parse `[Events]` section, extract `Dialogue:` lines, read the field order from the `Format:` row, and take everything after the Text field index as the text (Text can itself contain commas).
|
||||
|
||||
**ASS decoding.** The parser is where ASS text is decoded, once, via `assToPlainText()` in `src/core/services/ass-text.ts`. That decoder mirrors mpv's `ass_to_plaintext` so a cue read from a file reads identically to the same line arriving live on `sub-text`: `{...}` override blocks are markup, `\pN … \p0` vector drawing runs are dropped rather than shown as text, `\N`/`\n`/`\h` are the only escapes (`\{`, `\}` and `\\` are not), and an unclosed `{` is rendered verbatim. Every layer downstream — renderer, timing tracker, tokenizer, tokenization cache keys — receives plain text and uses `normalizePlainSubtitleText()` for whitespace only, so nothing decodes the same string twice and one authored line always maps to one cache key.
|
||||
|
||||
**Duplicate collapsing.** Typeset scripts emit one `Dialogue:` event per animation frame, plus layered copies of the same line. The parser collapses identical text over an identical span unconditionally, and collapses contiguous same-text runs of at least three events when the run looks like an animation. For ASS that means shared style and actor plus authoring evidence: a temporal tag (`\t`, `\move`, `\k`/`\kf`/`\ko`/`\K`, or anything wrapped in `\t(...)`), an animated `Effect` column (`Karaoke`, `Banner`, `Scroll`), or override values that change across the run. Static tags shared by every event (`\pos`, an identical `\clip`) are not evidence. SRT/VTT carry no such metadata, so there collapsing needs at least five contiguous events all under 0.1s — the frame timing left behind by ASS-to-SRT conversion. The parser keeps this authoring metadata (style, actor, layer, `Effect`, parsed override commands, source order) private; `parseSubtitleCues()` returns only `SubtitleCue`.
|
||||
|
||||
#### Prefetch Service Lifecycle
|
||||
|
||||
@@ -153,6 +158,7 @@ tokens (already have frequencyRank values from parser-level applyFrequencyRanks)
|
||||
### Dependency Analysis
|
||||
|
||||
All annotations either depend on MeCab POS data or benefit from running after it:
|
||||
|
||||
- **Known word marking:** Needs base tokens (surface/headword). No POS dependency, but no reason to run separately.
|
||||
- **Frequency filtering:** Uses `pos1Exclusions` and `pos2Exclusions` to clear frequency ranks on excluded tokens (particles, noise). Depends on MeCab POS data.
|
||||
- **JLPT marking:** Uses `shouldIgnoreJlptForMecabPos1` to filter. Depends on MeCab POS data.
|
||||
@@ -169,18 +175,14 @@ function annotateTokens(tokens, deps, options): MergedToken[] {
|
||||
|
||||
// Single pass: known word + frequency filtering + JLPT computed together
|
||||
const annotated = tokens.map((token) => {
|
||||
const isKnown = nPlusOneEnabled
|
||||
? token.isKnown || computeIsKnown(token, deps)
|
||||
: false;
|
||||
const isKnown = nPlusOneEnabled ? token.isKnown || computeIsKnown(token, deps) : false;
|
||||
|
||||
// Filter frequency rank using POS exclusions (rank values already set at parser level)
|
||||
const frequencyRank = frequencyEnabled
|
||||
? filterFrequencyRank(token, pos1Exclusions, pos2Exclusions)
|
||||
: undefined;
|
||||
|
||||
const jlptLevel = jlptEnabled
|
||||
? computeJlptLevel(token, deps.getJlptLevel)
|
||||
: undefined;
|
||||
const jlptLevel = jlptEnabled ? computeJlptLevel(token, deps.getJlptLevel) : undefined;
|
||||
|
||||
return { ...token, isKnown, frequencyRank, jlptLevel };
|
||||
});
|
||||
@@ -221,6 +223,7 @@ Replace `document.createElement('span')` calls in the renderer with `templateSpa
|
||||
### Current Behavior
|
||||
|
||||
In `renderWithTokens` (`subtitle-render.ts`), each render cycle:
|
||||
|
||||
1. Clears DOM with `innerHTML = ''`
|
||||
2. Creates a `DocumentFragment`
|
||||
3. Calls `document.createElement('span')` for each token (~10-15 per subtitle)
|
||||
@@ -257,7 +260,7 @@ Full recycling (collecting old nodes, clearing attributes, reusing them) require
|
||||
## Combined Impact Summary
|
||||
|
||||
| Scenario | Before | After | Improvement |
|
||||
|----------|--------|-------|-------------|
|
||||
| --------------------------------- | ---------- | ---------- | ----------- |
|
||||
| Normal playback (prefetch-warmed) | ~200-320ms | ~30-50ms | ~80-85% |
|
||||
| Cache hit (repeated subtitle) | ~72ms | ~55-65ms | ~10-20% |
|
||||
| Cache miss (immediate seek) | ~200-320ms | ~150-260ms | ~20-25% |
|
||||
@@ -267,16 +270,19 @@ Full recycling (collecting old nodes, clearing attributes, reusing them) require
|
||||
## Files Summary
|
||||
|
||||
### New Files
|
||||
|
||||
- `src/core/services/subtitle-prefetch.ts`
|
||||
- `src/core/services/subtitle-cue-parser.ts`
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/core/services/subtitle-processing-controller.ts` (expose `preCacheTokenization`)
|
||||
- `src/core/services/tokenizer/annotation-stage.ts` (batched single-pass)
|
||||
- `src/renderer/subtitle-render.ts` (template cloneNode)
|
||||
- `src/main.ts` (wire up prefetch service)
|
||||
|
||||
### Test Files
|
||||
|
||||
- New tests for subtitle cue parser (SRT, VTT, ASS formats)
|
||||
- New tests for subtitle prefetch service (priority window, seek, pause/resume)
|
||||
- Updated tests for annotation stage (same behavior, new implementation)
|
||||
|
||||
@@ -25,6 +25,7 @@ Read when: you need to find the owner module for a behavior or test surface
|
||||
- Anki workflow: `src/anki-integration/`, `src/core/services/anki-jimaku*.ts`
|
||||
- Immersion tracking: `src/core/services/immersion-tracker/`
|
||||
Includes stats storage/query schema such as `imm_videos`, `imm_media_art`, and `imm_youtube_videos` for per-video and YouTube-specific library metadata.
|
||||
`delete-maintenance-scheduler.ts` coalesces and serializes stats deletes; expensive deletion and summary rebuilds run in `delete-maintenance-worker-thread.ts` while the tracker queues playback writes. Each batch uses one transaction, lexical update, rollup refresh, and lifetime rebuild.
|
||||
- AniList tracking + character dictionary: `src/core/services/anilist/`, `src/main/runtime/composers/anilist-*`, `src/main/character-dictionary-runtime.ts`, `src/main/character-dictionary-runtime/`
|
||||
- Jellyfin integration: `src/core/services/jellyfin*.ts`, `src/main/runtime/composers/jellyfin-*`
|
||||
- Window trackers: `src/window-trackers/`
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {
|
||||
assOverrideSignature,
|
||||
assToPlainText,
|
||||
collectAssOverrideCommands,
|
||||
extractAssOverrideBlocks,
|
||||
hasAssTemporalOverride,
|
||||
isAnimatedAssEffectKind,
|
||||
isAssTemporalCommand,
|
||||
normalizePlainSubtitleText,
|
||||
parseAssEffectField,
|
||||
} from './ass-text';
|
||||
|
||||
test('assToPlainText drops vector drawing runs', () => {
|
||||
assert.equal(
|
||||
assToPlainText(
|
||||
'{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
|
||||
),
|
||||
'',
|
||||
);
|
||||
});
|
||||
|
||||
test('assToPlainText keeps text around drawing runs on the same event', () => {
|
||||
assert.equal(
|
||||
assToPlainText('{\\p1}m 0 0 l 10 10{\\p0}本文{\\p1}m 5 5 l 6 6{\\p0}続き'),
|
||||
'本文続き',
|
||||
);
|
||||
});
|
||||
|
||||
test('assToPlainText leaves \\pos alone when no drawing mode is active', () => {
|
||||
assert.equal(assToPlainText('{\\pos(960,1068)\\bord3}位置指定'), '位置指定');
|
||||
});
|
||||
|
||||
test('assToPlainText does not read \\pos as a drawing tag', () => {
|
||||
assert.equal(assToPlainText('{\\p1\\pos(1,2)}m 0 0 l 5 5'), '');
|
||||
});
|
||||
|
||||
test('assToPlainText resolves line-break and space escapes', () => {
|
||||
assert.equal(assToPlainText('一行目\\N二行目'), '一行目\n二行目');
|
||||
assert.equal(assToPlainText('一行目\\n二行目'), '一行目\n二行目');
|
||||
assert.equal(assToPlainText('一行目\\N二行目', ' '), '一行目 二行目');
|
||||
assert.equal(assToPlainText('間\\h隔'), '間 隔');
|
||||
});
|
||||
|
||||
test('assToPlainText matches mpv on brace and backslash sequences', () => {
|
||||
// mpv has no `\{` / `\}` / `\\` escapes: the backslashes are literal text and the
|
||||
// braces still open and close an override block.
|
||||
assert.equal(assToPlainText('\\{注\\}'), '\\');
|
||||
assert.equal(assToPlainText('\\\\N'), '\\\n');
|
||||
});
|
||||
|
||||
test('assToPlainText renders an unclosed override block verbatim', () => {
|
||||
// mpv shows the stray brace; guessing where the block ended can eat a whole line.
|
||||
assert.equal(assToPlainText('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
|
||||
});
|
||||
|
||||
test('assToPlainText is idempotent', () => {
|
||||
const samples = [
|
||||
'{\\an5\\p1}m 0 0 l 5 5{\\p0}本文',
|
||||
'\\{注\\}',
|
||||
'\\\\N',
|
||||
'本文{\\pos(1,2)',
|
||||
'一行目\\N二行目\\h終わり',
|
||||
];
|
||||
|
||||
for (const sample of samples) {
|
||||
const once = assToPlainText(sample);
|
||||
assert.equal(assToPlainText(once), once, sample);
|
||||
}
|
||||
});
|
||||
|
||||
test('assToPlainText normalizes CRLF before converting', () => {
|
||||
assert.equal(assToPlainText('一行目\r\n二行目'), '一行目\n二行目');
|
||||
});
|
||||
|
||||
test('normalizePlainSubtitleText settles whitespace without decoding ASS', () => {
|
||||
// A brace reaching this layer is literal text mpv chose to show, not markup.
|
||||
assert.equal(normalizePlainSubtitleText('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
|
||||
assert.equal(normalizePlainSubtitleText('一行目\\N二行目'), '一行目\n二行目');
|
||||
assert.equal(
|
||||
normalizePlainSubtitleText('一行目\\N二行目', { collapseLineBreaks: true }),
|
||||
'一行目 二行目',
|
||||
);
|
||||
assert.equal(normalizePlainSubtitleText(' 余白 ', { trim: false }), ' 余白 ');
|
||||
});
|
||||
|
||||
test('normalizePlainSubtitleText is idempotent', () => {
|
||||
for (const sample of ['一行目\\N二行目', '間\\h隔', '本文{\\pos(1,2)', ' 余白 ']) {
|
||||
const once = normalizePlainSubtitleText(sample);
|
||||
assert.equal(normalizePlainSubtitleText(once), once, sample);
|
||||
}
|
||||
});
|
||||
|
||||
test('extractAssOverrideBlocks returns block contents', () => {
|
||||
assert.deepEqual(extractAssOverrideBlocks('{\\an8}上{\\fad(200,200)}下'), [
|
||||
'\\an8',
|
||||
'\\fad(200,200)',
|
||||
]);
|
||||
assert.deepEqual(extractAssOverrideBlocks('括弧なし'), []);
|
||||
});
|
||||
|
||||
test('collectAssOverrideCommands captures names and arguments from blocks only', () => {
|
||||
const commands = collectAssOverrideCommands('{\\pos(1,2)\\1c&HFFFFFF&\\kf30}歌詞');
|
||||
|
||||
assert.deepEqual(commands, [
|
||||
{ name: 'pos', args: '1,2', animated: false },
|
||||
{ name: '1c', args: '&HFFFFFF&', animated: false },
|
||||
{ name: 'kf', args: '30', animated: false },
|
||||
]);
|
||||
|
||||
// A `\pos(...)` sitting in visible text is not typesetting markup.
|
||||
assert.deepEqual(collectAssOverrideCommands('\\pos(730,1042) と書いてある'), []);
|
||||
});
|
||||
|
||||
test('collectAssOverrideCommands marks tags animated by a wrapping \\t', () => {
|
||||
const commands = collectAssOverrideCommands('{\\clip(0,0,10,10)\\t(0,500,\\frz30)}文字');
|
||||
|
||||
assert.deepEqual(
|
||||
commands.map((command) => [command.name, command.animated]),
|
||||
[
|
||||
['clip', false],
|
||||
['t', false],
|
||||
['frz', true],
|
||||
],
|
||||
);
|
||||
assert.equal(hasAssTemporalOverride(commands), true);
|
||||
});
|
||||
|
||||
test('collectAssOverrideCommands stops descending into deeply nested \\t tags', () => {
|
||||
// Nested far past the recursion cap. Uncapped, this recurses once per level, and a
|
||||
// pathological line (real files reach one or two levels) overflows the stack.
|
||||
const nesting = 32;
|
||||
const block = `{${'\\t(0,500,'.repeat(nesting)}\\frz30${')'.repeat(nesting)}}文字`;
|
||||
|
||||
const commands = collectAssOverrideCommands(block);
|
||||
|
||||
// The outer `\t` plus one per allowed recursion level, and nothing from below the cap.
|
||||
assert.equal(commands.length, 9);
|
||||
assert.deepEqual(new Set(commands.map((command) => command.name)), new Set(['t']));
|
||||
assert.equal(hasAssTemporalOverride(commands), true);
|
||||
});
|
||||
|
||||
test('hasAssTemporalOverride ignores static placement and shape tags', () => {
|
||||
assert.equal(
|
||||
hasAssTemporalOverride(collectAssOverrideCommands('{\\pos(1,2)\\clip(m 1 1)\\blur2}文字')),
|
||||
false,
|
||||
);
|
||||
assert.equal(hasAssTemporalOverride(collectAssOverrideCommands('{\\move(1,2,3,4)}文字')), true);
|
||||
});
|
||||
|
||||
test('isAssTemporalCommand covers only intrinsically animated tags', () => {
|
||||
for (const command of ['t', 'move', 'k', 'kf', 'ko', 'K']) {
|
||||
assert.equal(isAssTemporalCommand(command), true, command);
|
||||
}
|
||||
for (const command of ['clip', 'iclip', 'frz', 'fscx', 'blur', 'be', 'pos', 'fad']) {
|
||||
assert.equal(isAssTemporalCommand(command), false, command);
|
||||
}
|
||||
});
|
||||
|
||||
test('assOverrideSignature distinguishes events by their override values', () => {
|
||||
const first = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 1 1)}歌詞'));
|
||||
const second = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 2 2)}歌詞'));
|
||||
const repeat = assOverrideSignature(collectAssOverrideCommands('{\\clip(m 1 1)}別の行'));
|
||||
|
||||
assert.notEqual(first, second);
|
||||
assert.equal(first, repeat);
|
||||
});
|
||||
|
||||
test('parseAssEffectField classifies the event-level Effect column', () => {
|
||||
assert.equal(parseAssEffectField(''), 'none');
|
||||
assert.equal(parseAssEffectField(' '), 'none');
|
||||
assert.equal(parseAssEffectField('Banner;20;1;0'), 'banner');
|
||||
assert.equal(parseAssEffectField('Scroll up;0;0;30;10'), 'scroll');
|
||||
assert.equal(parseAssEffectField('Scroll down;0;0;30;10'), 'scroll');
|
||||
assert.equal(parseAssEffectField('Karaoke'), 'karaoke');
|
||||
assert.equal(parseAssEffectField('fx-template'), 'other');
|
||||
});
|
||||
|
||||
test('parseAssEffectField matches stock effect names exactly', () => {
|
||||
// Custom effect names that merely start with a stock name are not stock effects.
|
||||
assert.equal(parseAssEffectField('scrolling-credit'), 'other');
|
||||
assert.equal(parseAssEffectField('bannerfx;1'), 'other');
|
||||
assert.equal(parseAssEffectField('karaoke-template'), 'other');
|
||||
assert.equal(parseAssEffectField('Scroll'), 'other');
|
||||
});
|
||||
|
||||
test('isAnimatedAssEffectKind covers the stock animated effects only', () => {
|
||||
assert.equal(isAnimatedAssEffectKind('karaoke'), true);
|
||||
assert.equal(isAnimatedAssEffectKind('banner'), true);
|
||||
assert.equal(isAnimatedAssEffectKind('scroll'), true);
|
||||
// Typesetting groups put static template names in this column too.
|
||||
assert.equal(isAnimatedAssEffectKind('other'), false);
|
||||
assert.equal(isAnimatedAssEffectKind('none'), false);
|
||||
});
|
||||
@@ -0,0 +1,280 @@
|
||||
/*
|
||||
* ASS/SSA text handling, split into two deliberately distinct contracts:
|
||||
*
|
||||
* assToPlainText() raw ASS event text -> plain text. Ingestion only.
|
||||
* normalizePlainSubtitleText() already-decoded text -> display/lookup form.
|
||||
*
|
||||
* Subtitle text is decoded from ASS exactly once, at the point it enters the app: the
|
||||
* file cue parser does it for sidecar/embedded scripts, and mpv does it for live text
|
||||
* (`sub-text` is already run through mpv's own `ass_to_plaintext`). Everything
|
||||
* downstream -- renderer, timing tracker, tokenizer, tokenization cache keys -- gets
|
||||
* plain text and only normalizes whitespace, so no layer decodes the same string twice.
|
||||
*
|
||||
* assToPlainText mirrors mpv's `ass_to_plaintext` rather than inventing its own rules,
|
||||
* so a cue parsed from a file reads the same as the same line arriving live:
|
||||
* - `{...}` override blocks are markup
|
||||
* - `\pN ... \p0` runs are vector paths, not dialogue
|
||||
* - `\N`, `\n` and `\h` are the only escapes; `\{`, `\}` and `\\` are NOT escapes,
|
||||
* so `\{注\}` decodes to a lone backslash exactly as mpv renders it
|
||||
* - an unclosed `{` is rendered verbatim instead of swallowing the rest of the line
|
||||
* Because the decoder never emits an escape or a closed brace, running it twice is a
|
||||
* no-op -- but downstream code should still use normalizePlainSubtitleText.
|
||||
*/
|
||||
|
||||
/** What `\N` and `\n` become. */
|
||||
export type AssLineBreak = '\n' | ' ';
|
||||
|
||||
// `\p<n>` with n > 0 switches libass into vector-drawing mode: everything until the
|
||||
// next `\p0` is a path (`m 20 0 b 10 0 ...`), not dialogue. The negative lookahead keeps
|
||||
// `\pos(...)` from being read as a drawing tag.
|
||||
const ASS_DRAWING_SCALE_PATTERN = /\\p(?![a-zA-Z])(\d*)/g;
|
||||
|
||||
function readDrawingScale(block: string): number | null {
|
||||
ASS_DRAWING_SCALE_PATTERN.lastIndex = 0;
|
||||
let scale: number | null = null;
|
||||
let match: RegExpExecArray | null;
|
||||
// Drawing mode is whatever the last `\p` tag in this block set it to.
|
||||
while ((match = ASS_DRAWING_SCALE_PATTERN.exec(block)) !== null) {
|
||||
scale = match[1] ? Number(match[1]) : 0;
|
||||
}
|
||||
return scale;
|
||||
}
|
||||
|
||||
/** Resolve `\N`, `\n` and `\h`. The only text-level escapes libass recognises. */
|
||||
function resolveWhitespaceEscapes(text: string, lineBreak: AssLineBreak): string {
|
||||
return text.replace(/\\([Nnh])/g, (_match, escaped: string) =>
|
||||
escaped === 'h' ? ' ' : lineBreak,
|
||||
);
|
||||
}
|
||||
|
||||
/** Strip `{...}` override blocks and the drawing runs they enable. */
|
||||
function stripAssMarkup(raw: string): string {
|
||||
let out = '';
|
||||
let cursor = 0;
|
||||
let drawing = false;
|
||||
|
||||
while (cursor < raw.length) {
|
||||
if (raw[cursor] !== '{') {
|
||||
if (!drawing) {
|
||||
out += raw[cursor];
|
||||
}
|
||||
cursor += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const close = raw.indexOf('}', cursor + 1);
|
||||
if (close === -1) {
|
||||
// mpv shows an unclosed `{` and everything after it. Guessing where the block was
|
||||
// meant to end can eat a whole line of dialogue.
|
||||
if (!drawing) {
|
||||
out += raw.slice(cursor);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
const scale = readDrawingScale(raw.slice(cursor, close + 1));
|
||||
if (scale !== null) {
|
||||
drawing = scale > 0;
|
||||
}
|
||||
cursor = close + 1;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode a raw ASS/SSA event text field. Call this once, where the text enters the app;
|
||||
* downstream layers take the result as plain text.
|
||||
*/
|
||||
export function assToPlainText(text: string, lineBreak: AssLineBreak = '\n'): string {
|
||||
if (!text) return '';
|
||||
return resolveWhitespaceEscapes(stripAssMarkup(text.replace(/\r\n/g, '\n')), lineBreak);
|
||||
}
|
||||
|
||||
export interface NormalizePlainSubtitleTextOptions {
|
||||
/** Fold every line break into a single space. */
|
||||
collapseLineBreaks?: boolean;
|
||||
trim?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whitespace normalization for text that has already been decoded -- by mpv for live
|
||||
* subtitles, by the cue parser for files. Override blocks and drawing runs are none of
|
||||
* this function's business; a `{` that reaches here is literal text mpv chose to show.
|
||||
*
|
||||
* `\N`/`\n`/`\h` are still folded, because subtitle sources outside the ASS path (asbplayer
|
||||
* and other websocket clients) forward them raw and the display layer has to cope.
|
||||
*/
|
||||
export function normalizePlainSubtitleText(
|
||||
text: string,
|
||||
options: NormalizePlainSubtitleTextOptions = {},
|
||||
): string {
|
||||
if (!text) return '';
|
||||
const { collapseLineBreaks = false, trim = true } = options;
|
||||
|
||||
let normalized = resolveWhitespaceEscapes(
|
||||
text.replace(/\r\n/g, '\n'),
|
||||
collapseLineBreaks ? ' ' : '\n',
|
||||
);
|
||||
if (collapseLineBreaks) {
|
||||
normalized = normalized.replace(/\n/g, ' ').replace(/\s+/g, ' ');
|
||||
}
|
||||
|
||||
return trim ? normalized.trim() : normalized;
|
||||
}
|
||||
|
||||
/** The contents of each `{...}` block, without the braces. */
|
||||
export function extractAssOverrideBlocks(text: string): string[] {
|
||||
const blocks: string[] = [];
|
||||
let cursor = 0;
|
||||
|
||||
while (cursor < text.length) {
|
||||
const open = text.indexOf('{', cursor);
|
||||
if (open === -1) {
|
||||
break;
|
||||
}
|
||||
const close = text.indexOf('}', open + 1);
|
||||
if (close === -1) {
|
||||
break;
|
||||
}
|
||||
blocks.push(text.slice(open + 1, close));
|
||||
cursor = close + 1;
|
||||
}
|
||||
|
||||
return blocks;
|
||||
}
|
||||
|
||||
export interface AssOverrideCommand {
|
||||
/** Tag name without the backslash, e.g. `pos`, `kf`, `1c`. */
|
||||
name: string;
|
||||
/** Everything the tag was given, e.g. `960,1068` for `\pos(960,1068)`. */
|
||||
args: string;
|
||||
/** Nested inside a `\t(...)` argument, so its value is animated over the event. */
|
||||
animated: boolean;
|
||||
}
|
||||
|
||||
const ASS_OVERRIDE_NAME_PATTERN = /[1-4]?[a-zA-Z]+/y;
|
||||
|
||||
function readCommandArgs(block: string, start: number): { args: string; next: number } {
|
||||
if (block[start] === '(') {
|
||||
let depth = 0;
|
||||
for (let i = start; i < block.length; i += 1) {
|
||||
if (block[i] === '(') depth += 1;
|
||||
else if (block[i] === ')') {
|
||||
depth -= 1;
|
||||
if (depth === 0) {
|
||||
return { args: block.slice(start + 1, i), next: i + 1 };
|
||||
}
|
||||
}
|
||||
}
|
||||
return { args: block.slice(start + 1), next: block.length };
|
||||
}
|
||||
|
||||
const nextTag = block.indexOf('\\', start);
|
||||
const end = nextTag === -1 ? block.length : nextTag;
|
||||
return { args: block.slice(start, end), next: end };
|
||||
}
|
||||
|
||||
// `\t(...)` can wrap another `\t(...)`, and nothing in the format stops an author (or a
|
||||
// malformed file) from nesting them thousands deep. Real typesetting never goes past one
|
||||
// or two levels, so stop recursing well before the call stack is at risk.
|
||||
const MAX_ANIMATION_NESTING_DEPTH = 8;
|
||||
|
||||
function parseOverrideBlock(
|
||||
block: string,
|
||||
animated: boolean,
|
||||
into: AssOverrideCommand[],
|
||||
depth = 0,
|
||||
): void {
|
||||
let cursor = 0;
|
||||
|
||||
while (cursor < block.length) {
|
||||
if (block[cursor] !== '\\') {
|
||||
cursor += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
ASS_OVERRIDE_NAME_PATTERN.lastIndex = cursor + 1;
|
||||
const nameMatch = ASS_OVERRIDE_NAME_PATTERN.exec(block);
|
||||
if (!nameMatch) {
|
||||
cursor += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const name = nameMatch[0];
|
||||
const { args, next } = readCommandArgs(block, cursor + 1 + name.length);
|
||||
into.push({ name, args: args.trim(), animated });
|
||||
// `\t(0,500,\frz30)` animates whatever it wraps, so record the inner tags too.
|
||||
if (name === 't' && args.includes('\\') && depth < MAX_ANIMATION_NESTING_DEPTH) {
|
||||
parseOverrideBlock(args, true, into, depth + 1);
|
||||
}
|
||||
cursor = next;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Override commands with their arguments, in source order. Only `{...}` blocks are
|
||||
* inspected, so a `\pos(...)` sitting in visible text is never mistaken for markup.
|
||||
*/
|
||||
export function collectAssOverrideCommands(text: string): AssOverrideCommand[] {
|
||||
const commands: AssOverrideCommand[] = [];
|
||||
for (const block of extractAssOverrideBlocks(text)) {
|
||||
parseOverrideBlock(block, false, commands);
|
||||
}
|
||||
return commands;
|
||||
}
|
||||
|
||||
// Tags that are animated by definition: `\t` interpolates, `\move` travels, and the
|
||||
// karaoke tags advance a highlight across the event's own duration. Everything else --
|
||||
// `\pos`, `\clip`, `\frz`, `\blur`, `\fad` -- is a static value for the event, so its
|
||||
// presence says nothing about whether neighbouring events form one animation.
|
||||
const ASS_TEMPORAL_COMMANDS = new Set(['t', 'move', 'k', 'kf', 'ko', 'K']);
|
||||
|
||||
export function isAssTemporalCommand(name: string): boolean {
|
||||
return ASS_TEMPORAL_COMMANDS.has(name);
|
||||
}
|
||||
|
||||
/** True when the event animates on its own, or animates a static tag through `\t(...)`. */
|
||||
export function hasAssTemporalOverride(commands: readonly AssOverrideCommand[]): boolean {
|
||||
return commands.some((command) => command.animated || isAssTemporalCommand(command.name));
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonical form of an event's override values, for comparing consecutive events. Two
|
||||
* events with the same signature were typeset identically, so neither is a frame of an
|
||||
* animation the other belongs to.
|
||||
*/
|
||||
export function assOverrideSignature(commands: readonly AssOverrideCommand[]): string {
|
||||
return commands.map((command) => `${command.name}(${command.args})`).join('|');
|
||||
}
|
||||
|
||||
export type AssEffectKind = 'none' | 'banner' | 'scroll' | 'karaoke' | 'other';
|
||||
|
||||
// The stock effects, matched exactly. Typesetting groups put their own template names in
|
||||
// this column -- `scrolling-credit` is a static sign, not libass's `Scroll up` -- so a
|
||||
// prefix match would hand out animation evidence to arbitrary custom effects.
|
||||
const STOCK_ASS_EFFECTS = new Map<string, AssEffectKind>([
|
||||
['banner', 'banner'],
|
||||
['scroll up', 'scroll'],
|
||||
['scroll down', 'scroll'],
|
||||
['karaoke', 'karaoke'],
|
||||
]);
|
||||
|
||||
/**
|
||||
* The event-level `Effect` column. The stock values (`Banner;...`, `Scroll up;...`,
|
||||
* `Scroll down;...`, `Karaoke`) all animate; anything else is a custom name and lands in
|
||||
* `other`.
|
||||
*/
|
||||
export function parseAssEffectField(raw: string): AssEffectKind {
|
||||
const value = raw.trim().toLowerCase();
|
||||
if (!value) return 'none';
|
||||
|
||||
const name = value.split(';', 1)[0]!.trim();
|
||||
return STOCK_ASS_EFFECTS.get(name) ?? 'other';
|
||||
}
|
||||
|
||||
const ANIMATED_ASS_EFFECT_KINDS = new Set<AssEffectKind>(['banner', 'scroll', 'karaoke']);
|
||||
|
||||
export function isAnimatedAssEffectKind(kind: AssEffectKind): boolean {
|
||||
return ANIMATED_ASS_EFFECT_KINDS.has(kind);
|
||||
}
|
||||
@@ -1414,6 +1414,353 @@ test('deleteSession ignores the currently active session and keeps new writes fl
|
||||
}
|
||||
});
|
||||
|
||||
test('deleteSession yields the main event loop while delete maintenance is pending', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const deleteGate: { release?: () => void } = {};
|
||||
let deleteRunnerCalled = false;
|
||||
let bufferedWritesAtDeleteStart = -1;
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
const createdTracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async () => {
|
||||
deleteRunnerCalled = true;
|
||||
bufferedWritesAtDeleteStart = (tracker as unknown as { queue: unknown[] }).queue.length;
|
||||
await new Promise<void>((resolve) => {
|
||||
deleteGate.release = resolve;
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
tracker = createdTracker;
|
||||
createdTracker.handleMediaChange('/tmp/delete-yield-first.mkv', 'Delete Yield First');
|
||||
createdTracker.handleMediaChange('/tmp/delete-yield-active.mkv', 'Delete Yield Active');
|
||||
|
||||
const privateApi = createdTracker as unknown as {
|
||||
db: DatabaseSync;
|
||||
queue: unknown[];
|
||||
flushNow: () => void;
|
||||
};
|
||||
const sessionId = (
|
||||
privateApi.db
|
||||
.prepare(
|
||||
`SELECT session_id AS sessionId
|
||||
FROM imm_sessions
|
||||
WHERE ended_at_ms IS NOT NULL
|
||||
ORDER BY session_id
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get() as { sessionId: number } | null
|
||||
)?.sessionId;
|
||||
assert.ok(sessionId);
|
||||
|
||||
const deletePromise = createdTracker.deleteSession(sessionId);
|
||||
let timerAdvanced = false;
|
||||
setTimeout(() => {
|
||||
timerAdvanced = true;
|
||||
}, 0);
|
||||
|
||||
await waitForCondition(() => deleteRunnerCalled);
|
||||
assert.equal(deleteRunnerCalled, true, 'delete should be dispatched to the maintenance runner');
|
||||
assert.equal(
|
||||
bufferedWritesAtDeleteStart,
|
||||
0,
|
||||
'writes buffered before delete should flush first',
|
||||
);
|
||||
await waitForCondition(() => timerAdvanced);
|
||||
|
||||
createdTracker.recordSubtitleLine('queued during delete', 0, 1);
|
||||
privateApi.flushNow();
|
||||
assert.ok(privateApi.queue.length > 0, 'tracking writes should wait for delete maintenance');
|
||||
|
||||
assert.ok(deleteGate.release);
|
||||
deleteGate.release();
|
||||
await deletePromise;
|
||||
} finally {
|
||||
deleteGate.release?.();
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('delete maintenance flushes the entire write queue before locking writes', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const deleteGate: { release?: () => void } = {};
|
||||
let queuedWritesAtDeleteStart = -1;
|
||||
let writeLockedAtDeleteStart = false;
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async () => {
|
||||
const privateApi = tracker as unknown as {
|
||||
queue: unknown[];
|
||||
writeLock: { locked: boolean };
|
||||
};
|
||||
queuedWritesAtDeleteStart = privateApi.queue.length;
|
||||
writeLockedAtDeleteStart = privateApi.writeLock.locked;
|
||||
await new Promise<void>((resolve) => {
|
||||
deleteGate.release = resolve;
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
const privateApi = tracker as unknown as {
|
||||
batchSize: number;
|
||||
flushNow: () => void;
|
||||
queue: unknown[];
|
||||
};
|
||||
privateApi.batchSize = 1;
|
||||
privateApi.queue.push({}, {}, {});
|
||||
privateApi.flushNow = () => {
|
||||
privateApi.queue.shift();
|
||||
};
|
||||
|
||||
const deletePromise = tracker.deleteSession(101);
|
||||
await waitForCondition(() => deleteGate.release !== undefined);
|
||||
|
||||
assert.equal(queuedWritesAtDeleteStart, 0);
|
||||
assert.equal(writeLockedAtDeleteStart, true);
|
||||
|
||||
deleteGate.release?.();
|
||||
await deletePromise;
|
||||
} finally {
|
||||
deleteGate.release?.();
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('delete maintenance tasks stay serialized under concurrent requests', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const releases: Array<() => void> = [];
|
||||
let activeTasks = 0;
|
||||
let maxActiveTasks = 0;
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async () => {
|
||||
activeTasks += 1;
|
||||
maxActiveTasks = Math.max(maxActiveTasks, activeTasks);
|
||||
await new Promise<void>((resolve) => {
|
||||
releases.push(resolve);
|
||||
});
|
||||
activeTasks -= 1;
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
const firstDelete = tracker.deleteSession(101);
|
||||
await waitForCondition(() => releases.length === 1);
|
||||
assert.equal(maxActiveTasks, 1);
|
||||
|
||||
const secondDelete = tracker.deleteSession(102);
|
||||
|
||||
releases[0]?.();
|
||||
await waitForCondition(() => releases.length === 2);
|
||||
assert.equal(maxActiveTasks, 1);
|
||||
|
||||
releases[1]?.();
|
||||
await Promise.all([firstDelete, secondDelete]);
|
||||
} finally {
|
||||
for (const release of releases) release();
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('concurrent delete requests share one maintenance worker batch', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const tasks: unknown[] = [];
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async (_path, task) => {
|
||||
tasks.push(task);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
const firstDelete = tracker.deleteSession(201);
|
||||
const secondDelete = tracker.deleteSessions([202, 203]);
|
||||
const thirdDelete = tracker.deleteVideo(204);
|
||||
await Promise.all([firstDelete, secondDelete, thirdDelete]);
|
||||
|
||||
assert.equal(tasks.length, 1, 'concurrent deletes should use one maintenance pass');
|
||||
assert.deepEqual(tasks[0], {
|
||||
kind: 'batch',
|
||||
tasks: [
|
||||
{ kind: 'session', sessionId: 201 },
|
||||
{ kind: 'sessions', sessionIds: [202, 203] },
|
||||
{ kind: 'video', videoId: 204 },
|
||||
],
|
||||
});
|
||||
} finally {
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('destroy rejects delete requests waiting behind active maintenance', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
let releaseFirstTask: () => void = () => {};
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
let markFirstTaskStarted: () => void = () => {};
|
||||
const firstTaskStarted = new Promise<void>((resolve) => {
|
||||
markFirstTaskStarted = resolve;
|
||||
});
|
||||
tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async () => {
|
||||
markFirstTaskStarted();
|
||||
await new Promise<void>((resolve) => {
|
||||
releaseFirstTask = resolve;
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
const firstDelete = tracker.deleteSession(301);
|
||||
await firstTaskStarted;
|
||||
const queuedDelete = tracker.deleteSession(302);
|
||||
tracker.destroy();
|
||||
|
||||
const queuedOutcome = await Promise.race([
|
||||
queuedDelete.then(
|
||||
() => 'resolved',
|
||||
(error: unknown) =>
|
||||
error instanceof Error && /shutting down/.test(error.message)
|
||||
? 'rejected'
|
||||
: 'wrong-error',
|
||||
),
|
||||
new Promise<'pending'>((resolve) => setTimeout(() => resolve('pending'), 25)),
|
||||
]);
|
||||
assert.equal(queuedOutcome, 'rejected');
|
||||
releaseFirstTask();
|
||||
await firstDelete;
|
||||
} finally {
|
||||
releaseFirstTask();
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('delete requested after destroy rejects without running maintenance', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let maintenanceCalls = 0;
|
||||
const Ctor = await loadTrackerCtor();
|
||||
const tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async () => {
|
||||
maintenanceCalls += 1;
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
tracker.destroy();
|
||||
|
||||
await assert.rejects(tracker.deleteSession(303), /shutting down/);
|
||||
assert.equal(maintenanceCalls, 0);
|
||||
cleanupDbPath(dbPath);
|
||||
});
|
||||
|
||||
test('deleteSessions skips maintenance when no sessions are deletable', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const tasks: unknown[] = [];
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
tracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async (_path, task) => {
|
||||
tasks.push(task);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
await tracker.deleteSessions([]);
|
||||
|
||||
assert.deepEqual(tasks, []);
|
||||
} finally {
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('queued video delete is skipped when that video becomes active before dispatch', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
const tasks: Array<{ kind: string }> = [];
|
||||
let releaseFirstTask: () => void = () => {};
|
||||
|
||||
try {
|
||||
const Ctor = await loadTrackerCtor();
|
||||
const createdTracker = new Ctor(
|
||||
{ dbPath },
|
||||
{
|
||||
runDeleteMaintenanceTask: async (_path, task) => {
|
||||
tasks.push(task);
|
||||
if (tasks.length === 1) {
|
||||
await new Promise<void>((resolve) => {
|
||||
releaseFirstTask = resolve;
|
||||
});
|
||||
}
|
||||
},
|
||||
},
|
||||
);
|
||||
tracker = createdTracker;
|
||||
createdTracker.handleMediaChange('/tmp/delete-race-target.mkv', 'Delete Race Target');
|
||||
createdTracker.handleMediaChange('/tmp/delete-race-other.mkv', 'Delete Race Other');
|
||||
|
||||
const privateApi = createdTracker as unknown as { db: DatabaseSync };
|
||||
const targetVideoId = (
|
||||
privateApi.db
|
||||
.prepare(`SELECT video_id AS videoId FROM imm_videos WHERE video_key LIKE '%target.mkv'`)
|
||||
.get() as { videoId: number } | null
|
||||
)?.videoId;
|
||||
assert.ok(targetVideoId);
|
||||
|
||||
const firstDelete = createdTracker.deleteSession(999_001);
|
||||
await waitForCondition(() => tasks.length === 1);
|
||||
|
||||
const queuedVideoDelete = createdTracker.deleteVideo(targetVideoId);
|
||||
createdTracker.handleMediaChange('/tmp/delete-race-target.mkv', 'Delete Race Target');
|
||||
releaseFirstTask();
|
||||
await Promise.all([firstDelete, queuedVideoDelete]);
|
||||
|
||||
assert.deepEqual(
|
||||
tasks.map((task) => task.kind),
|
||||
['session'],
|
||||
);
|
||||
} finally {
|
||||
releaseFirstTask();
|
||||
tracker?.destroy();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('deleteVideo ignores the currently active video and keeps new writes flushable', async () => {
|
||||
const dbPath = makeDbPath();
|
||||
let tracker: ImmersionTrackerService | null = null;
|
||||
|
||||
@@ -83,14 +83,15 @@ import {
|
||||
} from './immersion-tracker/query-library';
|
||||
import {
|
||||
cleanupVocabularyStats,
|
||||
deleteAnime as deleteAnimeQuery,
|
||||
deleteSession as deleteSessionQuery,
|
||||
deleteSessions as deleteSessionsQuery,
|
||||
deleteVideo as deleteVideoQuery,
|
||||
getVideoDurationMs,
|
||||
markVideoWatched,
|
||||
upsertCoverArt,
|
||||
} from './immersion-tracker/query-maintenance';
|
||||
import {
|
||||
DeleteMaintenanceWorkerRuntime,
|
||||
type RunDeleteMaintenanceTask,
|
||||
} from './immersion-tracker/delete-maintenance-worker-runtime';
|
||||
import { DeleteMaintenanceScheduler } from './immersion-tracker/delete-maintenance-scheduler';
|
||||
import { repairJellyfinStreamVideoLinks } from './immersion-tracker/jellyfin-link-repair';
|
||||
import {
|
||||
repairLegacySeasonlessAnimeRows,
|
||||
@@ -182,6 +183,7 @@ const YOUTUBE_SCREENSHOT_MAX_SECONDS = 120;
|
||||
const YOUTUBE_OEMBED_ENDPOINT = 'https://www.youtube.com/oembed';
|
||||
const YOUTUBE_ID_PATTERN = /^[A-Za-z0-9_-]{6,}$/;
|
||||
const YOUTUBE_METADATA_REFRESH_MS = 24 * 60 * 60 * 1000;
|
||||
const DELETE_MAINTENANCE_BATCH_WINDOW_MS = 10;
|
||||
|
||||
function isValidYouTubeVideoId(value: string | null): boolean {
|
||||
return Boolean(value && YOUTUBE_ID_PATTERN.test(value));
|
||||
@@ -385,6 +387,8 @@ export class ImmersionTrackerService {
|
||||
private readonly vacuumIntervalMs: number;
|
||||
private readonly dbPath: string;
|
||||
private readonly writeLock = { locked: false };
|
||||
private readonly destroyDeleteMaintenanceRunner: () => void;
|
||||
private readonly deleteMaintenanceScheduler: DeleteMaintenanceScheduler;
|
||||
private flushTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
private maintenanceTimer: ReturnType<typeof setInterval> | null = null;
|
||||
private flushScheduled = false;
|
||||
@@ -406,9 +410,38 @@ export class ImmersionTrackerService {
|
||||
| ((row: LegacyVocabularyPosRow) => Promise<LegacyVocabularyPosResolution | null>)
|
||||
| undefined;
|
||||
|
||||
constructor(options: ImmersionTrackerOptions) {
|
||||
constructor(
|
||||
options: ImmersionTrackerOptions,
|
||||
dependencies: {
|
||||
runDeleteMaintenanceTask?: RunDeleteMaintenanceTask;
|
||||
destroyDeleteMaintenanceRunner?: () => void;
|
||||
} = {},
|
||||
) {
|
||||
this.dbPath = options.dbPath;
|
||||
this.resolveLegacyVocabularyPos = options.resolveLegacyVocabularyPos;
|
||||
let runDeleteMaintenanceTask: RunDeleteMaintenanceTask;
|
||||
if (dependencies.runDeleteMaintenanceTask) {
|
||||
runDeleteMaintenanceTask = dependencies.runDeleteMaintenanceTask;
|
||||
this.destroyDeleteMaintenanceRunner =
|
||||
dependencies.destroyDeleteMaintenanceRunner ?? (() => {});
|
||||
} else {
|
||||
const deleteMaintenanceRuntime = new DeleteMaintenanceWorkerRuntime();
|
||||
runDeleteMaintenanceTask = (dbPath, task) => deleteMaintenanceRuntime.run(dbPath, task);
|
||||
this.destroyDeleteMaintenanceRunner = () => deleteMaintenanceRuntime.destroy();
|
||||
}
|
||||
this.deleteMaintenanceScheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: DELETE_MAINTENANCE_BATCH_WINDOW_MS,
|
||||
runTask: (task) => runDeleteMaintenanceTask(this.dbPath, task),
|
||||
onBusy: () => {
|
||||
this.flushTelemetry(true);
|
||||
while (this.queue.length > 0) this.flushNow();
|
||||
this.writeLock.locked = true;
|
||||
},
|
||||
onIdle: () => {
|
||||
this.writeLock.locked = false;
|
||||
if (!this.isDestroyed && this.queue.length > 0) this.scheduleFlush(0);
|
||||
},
|
||||
});
|
||||
const parentDir = path.dirname(this.dbPath);
|
||||
if (!fs.existsSync(parentDir)) {
|
||||
fs.mkdirSync(parentDir, { recursive: true });
|
||||
@@ -512,6 +545,8 @@ export class ImmersionTrackerService {
|
||||
}
|
||||
this.finalizeActiveSession();
|
||||
this.isDestroyed = true;
|
||||
this.deleteMaintenanceScheduler.destroy();
|
||||
this.destroyDeleteMaintenanceRunner();
|
||||
this.db.close();
|
||||
}
|
||||
|
||||
@@ -709,10 +744,11 @@ export class ImmersionTrackerService {
|
||||
this.logger.warn(`Ignoring delete request for active immersion session ${sessionId}`);
|
||||
return;
|
||||
}
|
||||
deleteSessionQuery(this.db, sessionId);
|
||||
await this.enqueueDeleteMaintenanceTask(() => ({ kind: 'session', sessionId }));
|
||||
}
|
||||
|
||||
async deleteSessions(sessionIds: number[]): Promise<void> {
|
||||
await this.enqueueDeleteMaintenanceTask(() => {
|
||||
const activeSessionId = this.sessionState?.sessionId;
|
||||
const deletableSessionIds =
|
||||
activeSessionId === undefined
|
||||
@@ -723,21 +759,25 @@ export class ImmersionTrackerService {
|
||||
`Ignoring bulk delete request for active immersion session ${activeSessionId}`,
|
||||
);
|
||||
}
|
||||
deleteSessionsQuery(this.db, deletableSessionIds);
|
||||
if (deletableSessionIds.length === 0) return null;
|
||||
return { kind: 'sessions', sessionIds: deletableSessionIds };
|
||||
});
|
||||
}
|
||||
|
||||
async deleteVideo(videoId: number): Promise<void> {
|
||||
await this.enqueueDeleteMaintenanceTask(() => {
|
||||
if (this.sessionState?.videoId === videoId) {
|
||||
this.logger.warn(`Ignoring delete request for active immersion video ${videoId}`);
|
||||
return;
|
||||
return null;
|
||||
}
|
||||
deleteVideoQuery(this.db, videoId);
|
||||
return { kind: 'video', videoId };
|
||||
});
|
||||
}
|
||||
|
||||
async deleteAnime(animeId: number): Promise<void> {
|
||||
// The active video's anime link is assigned asynchronously after the title
|
||||
// is parsed, so a guard reading imm_videos too early sees a null and lets
|
||||
// the delete through — then the late update recreates the anime row.
|
||||
await this.enqueueDeleteMaintenanceTask(async () => {
|
||||
// Resolve this at dispatch time because another queued delete can leave
|
||||
// enough time for playback to switch to an episode of this anime.
|
||||
const pendingVideoId = this.sessionState?.videoId;
|
||||
if (pendingVideoId !== undefined) {
|
||||
await this.pendingAnimeMetadataUpdates.get(pendingVideoId);
|
||||
@@ -750,10 +790,20 @@ export class ImmersionTrackerService {
|
||||
.get(activeVideoId) as { anime_id: number | null } | null;
|
||||
if (activeAnime?.anime_id === animeId) {
|
||||
this.logger.warn(`Ignoring delete request for active immersion anime ${animeId}`);
|
||||
return;
|
||||
return null;
|
||||
}
|
||||
}
|
||||
deleteAnimeQuery(this.db, animeId);
|
||||
return { kind: 'anime', animeId };
|
||||
});
|
||||
}
|
||||
|
||||
private enqueueDeleteMaintenanceTask(
|
||||
resolveTask: Parameters<DeleteMaintenanceScheduler['enqueue']>[0],
|
||||
): Promise<void> {
|
||||
if (this.isDestroyed) {
|
||||
return Promise.reject(new Error('Immersion tracker is shutting down'));
|
||||
}
|
||||
return this.deleteMaintenanceScheduler.enqueue(resolveTask);
|
||||
}
|
||||
|
||||
async reassignAnimeAnilist(
|
||||
@@ -1811,7 +1861,7 @@ export class ImmersionTrackerService {
|
||||
}
|
||||
|
||||
private runMaintenance(): void {
|
||||
if (this.isDestroyed) return;
|
||||
if (this.isDestroyed || this.writeLock.locked) return;
|
||||
try {
|
||||
this.flushTelemetry(true);
|
||||
this.flushNow();
|
||||
|
||||
@@ -50,6 +50,7 @@ import {
|
||||
updateAnimeAnilistInfo,
|
||||
upsertCoverArt,
|
||||
} from '../query-maintenance.js';
|
||||
import { deleteMaintenanceBatch } from '../query-delete-maintenance.js';
|
||||
import { getLocalEpochDay } from '../query-shared.js';
|
||||
import { EVENT_CARD_MINED, EVENT_SUBTITLE_LINE, SOURCE_TYPE_LOCAL } from '../types.js';
|
||||
|
||||
@@ -985,3 +986,197 @@ test('split maintenance helpers delete multiple sessions and whole videos with d
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('delete maintenance batch preserves retained data across overlapping session, video, and anime targets', () => {
|
||||
const { db, dbPath, stmts } = createDb();
|
||||
|
||||
try {
|
||||
const retainedAnimeId = getOrCreateAnimeRecord(db, {
|
||||
parsedTitle: 'Retained Anime',
|
||||
canonicalTitle: 'Retained Anime',
|
||||
anilistId: null,
|
||||
titleRomaji: null,
|
||||
titleEnglish: null,
|
||||
titleNative: null,
|
||||
metadataJson: null,
|
||||
});
|
||||
const deletedAnimeId = getOrCreateAnimeRecord(db, {
|
||||
parsedTitle: 'Deleted Anime',
|
||||
canonicalTitle: 'Deleted Anime',
|
||||
anilistId: null,
|
||||
titleRomaji: null,
|
||||
titleEnglish: null,
|
||||
titleNative: null,
|
||||
metadataJson: null,
|
||||
});
|
||||
const retainedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-retain.mkv', {
|
||||
canonicalTitle: 'Batch Retain',
|
||||
sourcePath: '/tmp/batch-retain.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: SOURCE_TYPE_LOCAL,
|
||||
});
|
||||
const deletedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-video.mkv', {
|
||||
canonicalTitle: 'Batch Video',
|
||||
sourcePath: '/tmp/batch-video.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: SOURCE_TYPE_LOCAL,
|
||||
});
|
||||
const animeVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-anime.mkv', {
|
||||
canonicalTitle: 'Batch Anime',
|
||||
sourcePath: '/tmp/batch-anime.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: SOURCE_TYPE_LOCAL,
|
||||
});
|
||||
for (const [videoId, animeId, episode] of [
|
||||
[retainedVideoId, retainedAnimeId, 1],
|
||||
[deletedVideoId, retainedAnimeId, 2],
|
||||
[animeVideoId, deletedAnimeId, 1],
|
||||
] as const) {
|
||||
linkVideoToAnimeRecord(db, videoId, {
|
||||
animeId,
|
||||
parsedBasename: `batch-${episode}.mkv`,
|
||||
parsedTitle: animeId === retainedAnimeId ? 'Retained Anime' : 'Deleted Anime',
|
||||
parsedSeason: 1,
|
||||
parsedEpisode: episode,
|
||||
parserSource: 'test',
|
||||
parserConfidence: 1,
|
||||
parseMetadataJson: null,
|
||||
});
|
||||
}
|
||||
|
||||
const startedAtMs = 1_700_000_000_000;
|
||||
const deletedSessionId = startSessionRecord(db, retainedVideoId, startedAtMs).sessionId;
|
||||
const retainedSessionId = startSessionRecord(
|
||||
db,
|
||||
retainedVideoId,
|
||||
startedAtMs + 1_000,
|
||||
).sessionId;
|
||||
const videoSessionId = startSessionRecord(db, deletedVideoId, startedAtMs + 2_000).sessionId;
|
||||
const animeSessionId = startSessionRecord(db, animeVideoId, startedAtMs + 3_000).sessionId;
|
||||
for (const [sessionId, sessionStartedAtMs] of [
|
||||
[deletedSessionId, startedAtMs],
|
||||
[retainedSessionId, startedAtMs + 1_000],
|
||||
[videoSessionId, startedAtMs + 2_000],
|
||||
[animeSessionId, startedAtMs + 3_000],
|
||||
] as const) {
|
||||
finalizeSessionMetrics(db, sessionId, sessionStartedAtMs);
|
||||
}
|
||||
|
||||
for (const [index, sessionId, videoId, animeId] of [
|
||||
[1, deletedSessionId, retainedVideoId, retainedAnimeId],
|
||||
[2, retainedSessionId, retainedVideoId, retainedAnimeId],
|
||||
[3, videoSessionId, deletedVideoId, retainedAnimeId],
|
||||
[4, animeSessionId, animeVideoId, deletedAnimeId],
|
||||
] as const) {
|
||||
insertWordOccurrence(db, stmts, {
|
||||
sessionId,
|
||||
videoId,
|
||||
animeId,
|
||||
lineIndex: index,
|
||||
text: '猫日',
|
||||
word: { headword: '猫', word: '猫', reading: 'ねこ' },
|
||||
});
|
||||
insertKanjiOccurrence(db, stmts, {
|
||||
sessionId,
|
||||
videoId,
|
||||
animeId,
|
||||
lineIndex: index + 10,
|
||||
text: '猫日',
|
||||
kanji: '日',
|
||||
});
|
||||
}
|
||||
|
||||
const rollupDay = getLocalEpochDay(db, startedAtMs);
|
||||
const rollupMonth = (
|
||||
db
|
||||
.prepare(
|
||||
`SELECT CAST(strftime('%Y%m', CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER) AS rollupMonth`,
|
||||
)
|
||||
.get(startedAtMs) as { rollupMonth: number }
|
||||
).rollupMonth;
|
||||
for (const videoId of [retainedVideoId, deletedVideoId, animeVideoId]) {
|
||||
db.prepare(
|
||||
`INSERT INTO imm_daily_rollups (
|
||||
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, 99, 99, 99, 99, 99, ?, ?)`,
|
||||
).run(rollupDay, videoId, startedAtMs, startedAtMs);
|
||||
db.prepare(
|
||||
`INSERT INTO imm_monthly_rollups (
|
||||
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, 99, 99, 99, 99, 99, ?, ?)`,
|
||||
).run(rollupMonth, videoId, startedAtMs, startedAtMs);
|
||||
}
|
||||
|
||||
deleteMaintenanceBatch(db, [
|
||||
{ kind: 'session', sessionId: deletedSessionId },
|
||||
{ kind: 'session', sessionId: videoSessionId },
|
||||
{ kind: 'video', videoId: deletedVideoId },
|
||||
{ kind: 'video', videoId: animeVideoId },
|
||||
{ kind: 'anime', animeId: deletedAnimeId },
|
||||
]);
|
||||
|
||||
assert.deepEqual(db.prepare('SELECT session_id FROM imm_sessions').all(), [
|
||||
{ session_id: retainedSessionId },
|
||||
]);
|
||||
assert.deepEqual(db.prepare('SELECT video_id FROM imm_videos').all(), [
|
||||
{ video_id: retainedVideoId },
|
||||
]);
|
||||
assert.deepEqual(db.prepare('SELECT anime_id FROM imm_anime').all(), [
|
||||
{ anime_id: retainedAnimeId },
|
||||
]);
|
||||
assert.equal(
|
||||
(
|
||||
db.prepare(`SELECT frequency FROM imm_words WHERE headword = '猫'`).get() as {
|
||||
frequency: number;
|
||||
}
|
||||
).frequency,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
(
|
||||
db.prepare(`SELECT frequency FROM imm_kanji WHERE kanji = '日'`).get() as {
|
||||
frequency: number;
|
||||
}
|
||||
).frequency,
|
||||
1,
|
||||
);
|
||||
assert.deepEqual(
|
||||
db.prepare('SELECT video_id, total_sessions FROM imm_daily_rollups').all() as Array<{
|
||||
video_id: number;
|
||||
total_sessions: number;
|
||||
}>,
|
||||
[{ video_id: retainedVideoId, total_sessions: 1 }],
|
||||
);
|
||||
assert.deepEqual(
|
||||
db.prepare('SELECT video_id, total_sessions FROM imm_monthly_rollups').all() as Array<{
|
||||
video_id: number;
|
||||
total_sessions: number;
|
||||
}>,
|
||||
[{ video_id: retainedVideoId, total_sessions: 1 }],
|
||||
);
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('delete maintenance batch chunks id lists below the SQLite variable limit', () => {
|
||||
const { db, dbPath } = createDb();
|
||||
|
||||
try {
|
||||
const ids = Array.from({ length: 32_767 }, (_, index) => index + 1);
|
||||
|
||||
assert.doesNotThrow(() => {
|
||||
deleteMaintenanceBatch(db, [
|
||||
{ kind: 'sessions', sessionIds: ids },
|
||||
...ids.map((videoId) => ({ kind: 'video' as const, videoId })),
|
||||
...ids.map((animeId) => ({ kind: 'anime' as const, animeId })),
|
||||
]);
|
||||
});
|
||||
} finally {
|
||||
db.close();
|
||||
cleanupDbPath(dbPath);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { DeleteMaintenanceScheduler } from './delete-maintenance-scheduler';
|
||||
import type { DeleteMaintenanceTask } from './delete-maintenance';
|
||||
|
||||
test('scheduler batches same-turn requests and balances busy state', async () => {
|
||||
const tasks: DeleteMaintenanceTask[] = [];
|
||||
const states: string[] = [];
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async (task) => {
|
||||
tasks.push(task);
|
||||
},
|
||||
onBusy: () => states.push('busy'),
|
||||
onIdle: () => states.push('idle'),
|
||||
});
|
||||
|
||||
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
|
||||
const second = scheduler.enqueue(() => ({ kind: 'sessions', sessionIds: [2, 3] }));
|
||||
const third = scheduler.enqueue(() => null);
|
||||
await Promise.all([first, second, third]);
|
||||
|
||||
assert.deepEqual(tasks, [
|
||||
{
|
||||
kind: 'batch',
|
||||
tasks: [
|
||||
{ kind: 'session', sessionId: 1 },
|
||||
{ kind: 'sessions', sessionIds: [2, 3] },
|
||||
],
|
||||
},
|
||||
]);
|
||||
assert.deepEqual(states, ['busy', 'idle']);
|
||||
});
|
||||
|
||||
test('scheduler rejects enqueue after destruction without entering busy state', async () => {
|
||||
let busyCalls = 0;
|
||||
let runCalls = 0;
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async () => {
|
||||
runCalls += 1;
|
||||
},
|
||||
onBusy: () => {
|
||||
busyCalls += 1;
|
||||
},
|
||||
onIdle: () => {},
|
||||
});
|
||||
scheduler.destroy();
|
||||
|
||||
await assert.rejects(
|
||||
scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 })),
|
||||
/shutting down/,
|
||||
);
|
||||
assert.equal(busyCalls, 0);
|
||||
assert.equal(runCalls, 0);
|
||||
});
|
||||
|
||||
test('scheduler rejects every request in a batch when the maintenance task fails', async () => {
|
||||
const failure = new Error('maintenance failed');
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async () => {
|
||||
throw failure;
|
||||
},
|
||||
onBusy: () => {},
|
||||
onIdle: () => {},
|
||||
});
|
||||
|
||||
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
|
||||
const second = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
|
||||
|
||||
const results = await Promise.allSettled([first, second]);
|
||||
assert.deepEqual(
|
||||
results.map((result) => (result.status === 'rejected' ? result.reason : null)),
|
||||
[failure, failure],
|
||||
);
|
||||
});
|
||||
|
||||
test('scheduler rejects only the request whose task resolution fails', async () => {
|
||||
const failure = new Error('resolution failed');
|
||||
const tasks: DeleteMaintenanceTask[] = [];
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async (task) => {
|
||||
tasks.push(task);
|
||||
},
|
||||
onBusy: () => {},
|
||||
onIdle: () => {},
|
||||
});
|
||||
|
||||
const failed = scheduler.enqueue(() => {
|
||||
throw failure;
|
||||
});
|
||||
const succeeded = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
|
||||
|
||||
const results = await Promise.allSettled([failed, succeeded]);
|
||||
assert.equal(results[0]?.status, 'rejected');
|
||||
assert.equal(results[0]?.status === 'rejected' ? results[0].reason : null, failure);
|
||||
assert.equal(results[1]?.status, 'fulfilled');
|
||||
assert.deepEqual(tasks, [{ kind: 'session', sessionId: 2 }]);
|
||||
});
|
||||
|
||||
test('scheduler does not schedule another drain when the queue is empty', async () => {
|
||||
const originalSetTimeout = globalThis.setTimeout;
|
||||
let timerCalls = 0;
|
||||
globalThis.setTimeout = ((handler: TimerHandler, timeout?: number, ...args: unknown[]) => {
|
||||
timerCalls += 1;
|
||||
return originalSetTimeout(handler, timeout, ...args);
|
||||
}) as typeof setTimeout;
|
||||
|
||||
try {
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async () => {},
|
||||
onBusy: () => {},
|
||||
onIdle: () => {},
|
||||
});
|
||||
|
||||
await scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
|
||||
assert.equal(timerCalls, 1);
|
||||
} finally {
|
||||
globalThis.setTimeout = originalSetTimeout;
|
||||
}
|
||||
});
|
||||
|
||||
test('scheduler serializes batches and rejects requests queued at destruction', async () => {
|
||||
const releases: Array<() => void> = [];
|
||||
let activeTasks = 0;
|
||||
let maxActiveTasks = 0;
|
||||
const scheduler = new DeleteMaintenanceScheduler({
|
||||
batchWindowMs: 0,
|
||||
runTask: async () => {
|
||||
activeTasks += 1;
|
||||
maxActiveTasks = Math.max(maxActiveTasks, activeTasks);
|
||||
await new Promise<void>((resolve) => releases.push(resolve));
|
||||
activeTasks -= 1;
|
||||
},
|
||||
onBusy: () => {},
|
||||
onIdle: () => {},
|
||||
});
|
||||
|
||||
const first = scheduler.enqueue(() => ({ kind: 'session', sessionId: 1 }));
|
||||
const maxPollAttempts = 100;
|
||||
let pollAttempts = 0;
|
||||
while (releases.length === 0 && pollAttempts < maxPollAttempts) {
|
||||
pollAttempts += 1;
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
assert.ok(
|
||||
releases.length > 0,
|
||||
`runTask did not produce a release after ${maxPollAttempts} polling attempts`,
|
||||
);
|
||||
const queued = scheduler.enqueue(() => ({ kind: 'session', sessionId: 2 }));
|
||||
scheduler.destroy();
|
||||
|
||||
await assert.rejects(queued, /shutting down/);
|
||||
releases[0]?.();
|
||||
await first;
|
||||
assert.equal(maxActiveTasks, 1);
|
||||
});
|
||||
@@ -0,0 +1,105 @@
|
||||
import type { DeleteMaintenanceOperation, DeleteMaintenanceTask } from './delete-maintenance';
|
||||
|
||||
type ResolveDeleteMaintenanceOperation = () =>
|
||||
| DeleteMaintenanceOperation
|
||||
| null
|
||||
| Promise<DeleteMaintenanceOperation | null>;
|
||||
|
||||
interface PendingDeleteMaintenanceRequest {
|
||||
resolveTask: ResolveDeleteMaintenanceOperation;
|
||||
resolve: () => void;
|
||||
reject: (error: unknown) => void;
|
||||
}
|
||||
|
||||
interface DeleteMaintenanceSchedulerOptions {
|
||||
batchWindowMs: number;
|
||||
runTask: (task: DeleteMaintenanceTask) => Promise<void>;
|
||||
onBusy: () => void;
|
||||
onIdle: () => void;
|
||||
}
|
||||
|
||||
export class DeleteMaintenanceScheduler {
|
||||
private readonly pendingRequests: PendingDeleteMaintenanceRequest[] = [];
|
||||
private running = false;
|
||||
private drainTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
private pendingTaskCount = 0;
|
||||
private destroyed = false;
|
||||
|
||||
constructor(private readonly options: DeleteMaintenanceSchedulerOptions) {}
|
||||
|
||||
enqueue(resolveTask: ResolveDeleteMaintenanceOperation): Promise<void> {
|
||||
if (this.destroyed) {
|
||||
return Promise.reject(new Error('Immersion tracker is shutting down'));
|
||||
}
|
||||
|
||||
if (this.pendingTaskCount === 0) this.options.onBusy();
|
||||
this.pendingTaskCount += 1;
|
||||
|
||||
const result = new Promise<void>((resolve, reject) => {
|
||||
this.pendingRequests.push({ resolveTask, resolve, reject });
|
||||
this.scheduleDrain();
|
||||
});
|
||||
|
||||
return result.finally(() => {
|
||||
this.pendingTaskCount -= 1;
|
||||
if (this.pendingTaskCount === 0) this.options.onIdle();
|
||||
});
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
this.destroyed = true;
|
||||
if (this.drainTimer) {
|
||||
clearTimeout(this.drainTimer);
|
||||
this.drainTimer = null;
|
||||
}
|
||||
const error = new Error('Immersion tracker is shutting down');
|
||||
for (const request of this.pendingRequests.splice(0)) request.reject(error);
|
||||
}
|
||||
|
||||
private scheduleDrain(): void {
|
||||
if (this.destroyed || this.running || this.drainTimer || this.pendingRequests.length === 0) {
|
||||
return;
|
||||
}
|
||||
this.drainTimer = setTimeout(() => {
|
||||
this.drainTimer = null;
|
||||
void this.drain();
|
||||
}, this.options.batchWindowMs);
|
||||
}
|
||||
|
||||
private async drain(): Promise<void> {
|
||||
if (this.running || this.pendingRequests.length === 0) return;
|
||||
this.running = true;
|
||||
const requests = this.pendingRequests.splice(0);
|
||||
const runnable: Array<{
|
||||
request: PendingDeleteMaintenanceRequest;
|
||||
task: DeleteMaintenanceOperation;
|
||||
}> = [];
|
||||
|
||||
for (const request of requests) {
|
||||
try {
|
||||
const task = await request.resolveTask();
|
||||
if (task) runnable.push({ request, task });
|
||||
else request.resolve();
|
||||
} catch (error) {
|
||||
request.reject(error);
|
||||
}
|
||||
}
|
||||
|
||||
if (runnable.length > 0) {
|
||||
const task: DeleteMaintenanceTask =
|
||||
runnable.length === 1
|
||||
? runnable[0]!.task
|
||||
: { kind: 'batch', tasks: runnable.map((entry) => entry.task) };
|
||||
try {
|
||||
await this.options.runTask(task);
|
||||
for (const { request } of runnable) request.resolve();
|
||||
} catch (error) {
|
||||
for (const { request } of runnable) request.reject(error);
|
||||
}
|
||||
}
|
||||
|
||||
this.running = false;
|
||||
this.scheduleDrain();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
import {
|
||||
DeleteMaintenanceWorkerRuntime,
|
||||
resolveDeleteMaintenanceWorkerPath,
|
||||
} from './delete-maintenance-worker-runtime';
|
||||
import { executeDeleteMaintenanceTask } from './delete-maintenance';
|
||||
import { startSessionRecord } from './session';
|
||||
import { Database } from './sqlite';
|
||||
import { applyPragmas, ensureSchema, getOrCreateVideoRecord } from './storage';
|
||||
|
||||
type FakeWorkerListener = (value: never) => void;
|
||||
|
||||
function createFakeWorker() {
|
||||
const listeners = new Map<string, FakeWorkerListener>();
|
||||
const terminationState = { calls: 0 };
|
||||
const worker = {
|
||||
once(event: string, listener: FakeWorkerListener) {
|
||||
listeners.set(event, listener);
|
||||
return this;
|
||||
},
|
||||
terminate: async () => {
|
||||
terminationState.calls += 1;
|
||||
return 0;
|
||||
},
|
||||
};
|
||||
return { worker, listeners, terminationState };
|
||||
}
|
||||
|
||||
type FakeWorker = ReturnType<typeof createFakeWorker>['worker'];
|
||||
|
||||
test('a delete batch rebuilds lifetime summaries once', () => {
|
||||
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-delete-batch-test-'));
|
||||
const dbPath = path.join(tempDir, 'immersion.sqlite');
|
||||
let db = new Database(dbPath);
|
||||
|
||||
try {
|
||||
applyPragmas(db);
|
||||
ensureSchema(db);
|
||||
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-delete.mkv', {
|
||||
canonicalTitle: 'Batch Delete',
|
||||
sourcePath: '/tmp/batch-delete.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: 1,
|
||||
});
|
||||
const firstSessionId = startSessionRecord(db, videoId, 1_000).sessionId;
|
||||
const secondSessionId = startSessionRecord(db, videoId, 2_000).sessionId;
|
||||
const deletedVideoId = getOrCreateVideoRecord(db, 'local:/tmp/batch-delete-video.mkv', {
|
||||
canonicalTitle: 'Batch Delete Video',
|
||||
sourcePath: '/tmp/batch-delete-video.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: 1,
|
||||
});
|
||||
startSessionRecord(db, deletedVideoId, 3_000);
|
||||
db.exec(`
|
||||
CREATE TABLE delete_rebuild_audit (id INTEGER PRIMARY KEY);
|
||||
CREATE TRIGGER count_delete_lifetime_rebuild
|
||||
AFTER UPDATE OF last_rebuilt_ms ON imm_lifetime_global
|
||||
BEGIN
|
||||
INSERT INTO delete_rebuild_audit (id) VALUES (NULL);
|
||||
END;
|
||||
`);
|
||||
db.close();
|
||||
|
||||
executeDeleteMaintenanceTask(dbPath, {
|
||||
kind: 'batch',
|
||||
tasks: [
|
||||
{ kind: 'session', sessionId: firstSessionId },
|
||||
{ kind: 'video', videoId: deletedVideoId },
|
||||
],
|
||||
});
|
||||
|
||||
db = new Database(dbPath);
|
||||
const audit = db.prepare('SELECT COUNT(*) AS total FROM delete_rebuild_audit').get() as {
|
||||
total: number;
|
||||
};
|
||||
const retainedSession = db
|
||||
.prepare('SELECT session_id AS sessionId FROM imm_sessions WHERE video_id = ?')
|
||||
.get(videoId) as { sessionId: number } | null;
|
||||
const deletedVideo = db
|
||||
.prepare('SELECT video_id AS videoId FROM imm_videos WHERE video_id = ?')
|
||||
.get(deletedVideoId) as { videoId: number } | null;
|
||||
assert.equal(retainedSession?.sessionId, secondSessionId);
|
||||
assert.equal(deletedVideo, undefined);
|
||||
assert.equal(
|
||||
audit.total,
|
||||
2,
|
||||
'one rebuild performs exactly its reset and final global summary writes',
|
||||
);
|
||||
} finally {
|
||||
try {
|
||||
db.close();
|
||||
} catch {
|
||||
// The setup connection closes before maintenance runs.
|
||||
}
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test(
|
||||
'compiled delete worker removes data through its separate database connection',
|
||||
{ skip: resolveDeleteMaintenanceWorkerPath() === null },
|
||||
async () => {
|
||||
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-delete-worker-test-'));
|
||||
const dbPath = path.join(tempDir, 'immersion.sqlite');
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime();
|
||||
let db = new Database(dbPath);
|
||||
|
||||
try {
|
||||
applyPragmas(db);
|
||||
ensureSchema(db);
|
||||
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/worker-delete.mkv', {
|
||||
canonicalTitle: 'Worker Delete',
|
||||
sourcePath: '/tmp/worker-delete.mkv',
|
||||
sourceUrl: null,
|
||||
sourceType: 1,
|
||||
});
|
||||
const firstSessionId = startSessionRecord(db, videoId, 1_000).sessionId;
|
||||
const secondSessionId = startSessionRecord(db, videoId, 2_000).sessionId;
|
||||
db.close();
|
||||
|
||||
await runtime.run(dbPath, {
|
||||
kind: 'batch',
|
||||
tasks: [
|
||||
{ kind: 'session', sessionId: firstSessionId },
|
||||
{ kind: 'session', sessionId: secondSessionId },
|
||||
],
|
||||
});
|
||||
|
||||
db = new Database(dbPath);
|
||||
const row = db
|
||||
.prepare('SELECT COUNT(*) AS total FROM imm_sessions WHERE video_id = ?')
|
||||
.get(videoId) as { total: number };
|
||||
assert.equal(row.total, 0);
|
||||
} finally {
|
||||
runtime.destroy();
|
||||
try {
|
||||
db.close();
|
||||
} catch {
|
||||
// The setup connection is already closed before the worker starts.
|
||||
}
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test('worker runtime warns before falling back when no emitted worker is available', async () => {
|
||||
const warnings: unknown[][] = [];
|
||||
const fallbackTasks: unknown[] = [];
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime({
|
||||
resolveWorkerPath: () => null,
|
||||
warn: (...args) => warnings.push(args),
|
||||
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
|
||||
});
|
||||
|
||||
await runtime.run('/tmp/fallback.sqlite', { kind: 'session', sessionId: 1 });
|
||||
|
||||
assert.equal(warnings.length, 1);
|
||||
assert.match(String(warnings[0]?.[0]), /worker unavailable/i);
|
||||
assert.deepEqual(fallbackTasks, [{ kind: 'session', sessionId: 1 }]);
|
||||
});
|
||||
|
||||
test('worker runtime terminates a worker after successful settlement', async () => {
|
||||
const { worker, listeners, terminationState } = createFakeWorker();
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime({
|
||||
resolveWorkerPath: () => '/tmp/delete-worker.js',
|
||||
createWorker: async () => worker,
|
||||
});
|
||||
|
||||
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
listeners.get('message')?.({ ok: true } as never);
|
||||
await result;
|
||||
|
||||
assert.equal(terminationState.calls, 1);
|
||||
});
|
||||
|
||||
test('worker runtime terminates a worker after failed settlement', async () => {
|
||||
const { worker, listeners, terminationState } = createFakeWorker();
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime({
|
||||
resolveWorkerPath: () => '/tmp/delete-worker.js',
|
||||
createWorker: async () => worker,
|
||||
});
|
||||
|
||||
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
listeners.get('error')?.(new Error('worker failed') as never);
|
||||
|
||||
await assert.rejects(result, /worker failed/);
|
||||
assert.equal(terminationState.calls, 1);
|
||||
});
|
||||
|
||||
test('worker runtime terminates a worker created after shutdown begins', async () => {
|
||||
const { worker, listeners, terminationState } = createFakeWorker();
|
||||
const createGate: { resolve?: (worker: FakeWorker) => void } = {};
|
||||
const fallbackTasks: unknown[] = [];
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime({
|
||||
resolveWorkerPath: () => '/tmp/delete-worker.js',
|
||||
createWorker: () =>
|
||||
new Promise((resolve) => {
|
||||
createGate.resolve = resolve;
|
||||
}),
|
||||
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
|
||||
});
|
||||
|
||||
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
runtime.destroy();
|
||||
createGate.resolve?.(worker);
|
||||
|
||||
await assert.rejects(result, /shut down/);
|
||||
assert.equal(terminationState.calls, 1);
|
||||
assert.equal(listeners.size, 0);
|
||||
assert.deepEqual(fallbackTasks, []);
|
||||
});
|
||||
|
||||
test('worker runtime does not fall back when worker creation fails during shutdown', async () => {
|
||||
const createGate: { reject?: (error: Error) => void } = {};
|
||||
const fallbackTasks: unknown[] = [];
|
||||
const runtime = new DeleteMaintenanceWorkerRuntime({
|
||||
resolveWorkerPath: () => '/tmp/delete-worker.js',
|
||||
createWorker: () =>
|
||||
new Promise((_resolve, reject) => {
|
||||
createGate.reject = reject;
|
||||
}),
|
||||
executeFallback: (_dbPath, task) => fallbackTasks.push(task),
|
||||
});
|
||||
|
||||
const result = runtime.run('/tmp/test.sqlite', { kind: 'session', sessionId: 1 });
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
runtime.destroy();
|
||||
createGate.reject?.(new Error('creation failed'));
|
||||
|
||||
await assert.rejects(result, /shut down/);
|
||||
assert.deepEqual(fallbackTasks, []);
|
||||
});
|
||||
@@ -0,0 +1,121 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { createLogger } from '../../../logger';
|
||||
import { executeDeleteMaintenanceTask, type DeleteMaintenanceTask } from './delete-maintenance';
|
||||
|
||||
interface DeleteMaintenanceWorkerResponse {
|
||||
ok?: unknown;
|
||||
error?: unknown;
|
||||
}
|
||||
|
||||
export type RunDeleteMaintenanceTask = (
|
||||
dbPath: string,
|
||||
task: DeleteMaintenanceTask,
|
||||
) => Promise<void>;
|
||||
|
||||
interface DeleteMaintenanceWorkerHandle {
|
||||
once(event: 'message', listener: (message: DeleteMaintenanceWorkerResponse) => void): this;
|
||||
once(event: 'error', listener: (error: Error) => void): this;
|
||||
once(event: 'exit', listener: (code: number) => void): this;
|
||||
terminate(): Promise<number>;
|
||||
}
|
||||
|
||||
interface DeleteMaintenanceWorkerRuntimeOptions {
|
||||
resolveWorkerPath?: () => string | null;
|
||||
createWorker?: (
|
||||
workerPath: string,
|
||||
workerData: { dbPath: string; task: DeleteMaintenanceTask },
|
||||
) => Promise<DeleteMaintenanceWorkerHandle>;
|
||||
executeFallback?: typeof executeDeleteMaintenanceTask;
|
||||
warn?: (message: string, ...meta: unknown[]) => void;
|
||||
}
|
||||
|
||||
export function resolveDeleteMaintenanceWorkerPath(): string | null {
|
||||
const workerPath = path.join(__dirname, 'delete-maintenance-worker-thread.js');
|
||||
return fs.existsSync(workerPath) ? workerPath : null;
|
||||
}
|
||||
|
||||
const logger = createLogger('main:immersion-tracker:delete-worker');
|
||||
|
||||
export class DeleteMaintenanceWorkerRuntime {
|
||||
private readonly activeWorkers = new Set<DeleteMaintenanceWorkerHandle>();
|
||||
private destroyed = false;
|
||||
|
||||
constructor(private readonly options: DeleteMaintenanceWorkerRuntimeOptions = {}) {}
|
||||
|
||||
async run(dbPath: string, task: DeleteMaintenanceTask): Promise<void> {
|
||||
if (this.destroyed) {
|
||||
throw new Error('Delete maintenance worker is shut down');
|
||||
}
|
||||
|
||||
let worker: DeleteMaintenanceWorkerHandle;
|
||||
try {
|
||||
const workerPath = (this.options.resolveWorkerPath ?? resolveDeleteMaintenanceWorkerPath)();
|
||||
if (!workerPath) throw new Error('Emitted delete-maintenance worker module was not found');
|
||||
const createWorker =
|
||||
this.options.createWorker ??
|
||||
(async (resolvedPath, workerData) => {
|
||||
const { Worker } = await import('node:worker_threads');
|
||||
return new Worker(resolvedPath, { workerData });
|
||||
});
|
||||
worker = await createWorker(workerPath, { dbPath, task });
|
||||
} catch (error) {
|
||||
if (this.destroyed) {
|
||||
throw new Error('Delete maintenance worker is shut down');
|
||||
}
|
||||
(this.options.warn ?? logger.warn)(
|
||||
'Delete maintenance worker unavailable; running maintenance on the current thread',
|
||||
error,
|
||||
);
|
||||
(this.options.executeFallback ?? executeDeleteMaintenanceTask)(dbPath, task);
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.destroyed) {
|
||||
await worker.terminate().catch(() => undefined);
|
||||
throw new Error('Delete maintenance worker is shut down');
|
||||
}
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
let settled = false;
|
||||
this.activeWorkers.add(worker);
|
||||
|
||||
const settle = (error?: Error) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
this.activeWorkers.delete(worker);
|
||||
if (error) reject(error);
|
||||
else resolve();
|
||||
void worker.terminate();
|
||||
};
|
||||
|
||||
worker.once('message', (message: DeleteMaintenanceWorkerResponse) => {
|
||||
if (message.ok === true) {
|
||||
settle();
|
||||
return;
|
||||
}
|
||||
const detail = typeof message.error === 'string' ? message.error : 'unknown worker error';
|
||||
settle(new Error(`Delete maintenance failed: ${detail}`));
|
||||
});
|
||||
worker.once('error', (error) => settle(error));
|
||||
worker.once('exit', (code) => {
|
||||
settle(
|
||||
new Error(
|
||||
code === 0
|
||||
? 'Delete maintenance worker exited without a response'
|
||||
: `Delete maintenance worker exited with code ${code}`,
|
||||
),
|
||||
);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
this.destroyed = true;
|
||||
for (const worker of this.activeWorkers) {
|
||||
void worker.terminate();
|
||||
}
|
||||
this.activeWorkers.clear();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { parentPort, workerData } from 'node:worker_threads';
|
||||
import { executeDeleteMaintenanceTask, type DeleteMaintenanceTask } from './delete-maintenance';
|
||||
|
||||
interface DeleteMaintenanceWorkerData {
|
||||
dbPath: string;
|
||||
task: DeleteMaintenanceTask;
|
||||
}
|
||||
|
||||
if (!parentPort) {
|
||||
throw new Error('delete maintenance worker missing parent port');
|
||||
}
|
||||
|
||||
const port = parentPort;
|
||||
const request = workerData as DeleteMaintenanceWorkerData;
|
||||
|
||||
try {
|
||||
executeDeleteMaintenanceTask(request.dbPath, request.task);
|
||||
port.postMessage({ ok: true });
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
port.postMessage({ ok: false, error: message });
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
import { Database } from './sqlite';
|
||||
import { applyPragmas } from './storage';
|
||||
import { deleteAnime, deleteSession, deleteSessions, deleteVideo } from './query-maintenance';
|
||||
import {
|
||||
deleteMaintenanceBatch,
|
||||
type DeleteMaintenanceOperation,
|
||||
} from './query-delete-maintenance';
|
||||
|
||||
export type { DeleteMaintenanceOperation } from './query-delete-maintenance';
|
||||
|
||||
export type DeleteMaintenanceTask =
|
||||
| DeleteMaintenanceOperation
|
||||
| { kind: 'batch'; tasks: DeleteMaintenanceOperation[] };
|
||||
|
||||
function executeDeleteMaintenanceOperation(
|
||||
db: InstanceType<typeof Database>,
|
||||
task: DeleteMaintenanceOperation,
|
||||
): void {
|
||||
switch (task.kind) {
|
||||
case 'session':
|
||||
deleteSession(db, task.sessionId);
|
||||
return;
|
||||
case 'sessions':
|
||||
deleteSessions(db, task.sessionIds);
|
||||
return;
|
||||
case 'video':
|
||||
deleteVideo(db, task.videoId);
|
||||
return;
|
||||
case 'anime':
|
||||
deleteAnime(db, task.animeId);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
export function executeDeleteMaintenanceTask(dbPath: string, task: DeleteMaintenanceTask): void {
|
||||
const db = new Database(dbPath);
|
||||
try {
|
||||
applyPragmas(db);
|
||||
if (task.kind === 'batch') {
|
||||
deleteMaintenanceBatch(db, task.tasks);
|
||||
return;
|
||||
}
|
||||
executeDeleteMaintenanceOperation(db, task);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
import type { DatabaseSync } from './sqlite';
|
||||
import { rebuildLifetimeSummariesInTransaction } from './lifetime';
|
||||
import { getRollupGroupsForSessions, refreshRollupsForGroupsInTransaction } from './maintenance';
|
||||
import {
|
||||
applyLexicalRemovals,
|
||||
cleanupUnusedCoverArtBlobHash,
|
||||
deleteSessionsByIds,
|
||||
forEachIdChunk,
|
||||
makePlaceholders,
|
||||
planLexicalRemovalsForSessions,
|
||||
SQLITE_ID_CHUNK_SIZE,
|
||||
type LexicalRemovalPlan,
|
||||
} from './query-shared';
|
||||
|
||||
export type DeleteMaintenanceOperation =
|
||||
| { kind: 'session'; sessionId: number }
|
||||
| { kind: 'sessions'; sessionIds: number[] }
|
||||
| { kind: 'video'; videoId: number }
|
||||
| { kind: 'anime'; animeId: number };
|
||||
|
||||
function addOperationTargets(
|
||||
operations: DeleteMaintenanceOperation[],
|
||||
sessionIds: Set<number>,
|
||||
videoIds: Set<number>,
|
||||
animeIds: Set<number>,
|
||||
): void {
|
||||
for (const operation of operations) {
|
||||
switch (operation.kind) {
|
||||
case 'session':
|
||||
sessionIds.add(operation.sessionId);
|
||||
break;
|
||||
case 'sessions':
|
||||
for (const sessionId of operation.sessionIds) sessionIds.add(sessionId);
|
||||
break;
|
||||
case 'video':
|
||||
videoIds.add(operation.videoId);
|
||||
break;
|
||||
case 'anime':
|
||||
animeIds.add(operation.animeId);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function selectIds(
|
||||
db: DatabaseSync,
|
||||
buildSql: (placeholders: string) => string,
|
||||
params: number[],
|
||||
column: string,
|
||||
): number[] {
|
||||
if (params.length === 0) return [];
|
||||
const ids: number[] = [];
|
||||
forEachIdChunk(params, (chunk) => {
|
||||
const rows = db.prepare(buildSql(makePlaceholders(chunk))).all(...chunk) as Array<
|
||||
Record<string, number>
|
||||
>;
|
||||
for (const row of rows) ids.push(row[column]!);
|
||||
});
|
||||
return ids;
|
||||
}
|
||||
|
||||
function planLexicalRemovalsInChunks(db: DatabaseSync, sessionIds: number[]): LexicalRemovalPlan {
|
||||
const combined: LexicalRemovalPlan = { words: [], kanji: [] };
|
||||
const merge = (target: LexicalRemovalPlan['words'], source: LexicalRemovalPlan['words']) => {
|
||||
const byId = new Map(target.map((entry) => [entry.id, entry]));
|
||||
for (const entry of source) {
|
||||
const existing = byId.get(entry.id);
|
||||
if (!existing) {
|
||||
const added = { ...entry };
|
||||
target.push(added);
|
||||
byId.set(entry.id, added);
|
||||
continue;
|
||||
}
|
||||
existing.removedFrequency += entry.removedFrequency;
|
||||
if (
|
||||
entry.removedFirstSeenMs !== null &&
|
||||
(existing.removedFirstSeenMs === null ||
|
||||
entry.removedFirstSeenMs < existing.removedFirstSeenMs)
|
||||
) {
|
||||
existing.removedFirstSeenMs = entry.removedFirstSeenMs;
|
||||
}
|
||||
if (
|
||||
entry.removedLastSeenMs !== null &&
|
||||
(existing.removedLastSeenMs === null ||
|
||||
entry.removedLastSeenMs > existing.removedLastSeenMs)
|
||||
) {
|
||||
existing.removedLastSeenMs = entry.removedLastSeenMs;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
forEachIdChunk(sessionIds, (chunk) => {
|
||||
const plan = planLexicalRemovalsForSessions(db, chunk);
|
||||
merge(combined.words, plan.words);
|
||||
merge(combined.kanji, plan.kanji);
|
||||
});
|
||||
return combined;
|
||||
}
|
||||
|
||||
export function deleteMaintenanceBatch(
|
||||
db: DatabaseSync,
|
||||
operations: DeleteMaintenanceOperation[],
|
||||
): void {
|
||||
if (operations.length === 0) return;
|
||||
|
||||
db.exec('BEGIN IMMEDIATE');
|
||||
try {
|
||||
const sessionIds = new Set<number>();
|
||||
const videoIds = new Set<number>();
|
||||
const animeIds = new Set<number>();
|
||||
addOperationTargets(operations, sessionIds, videoIds, animeIds);
|
||||
|
||||
const animeIdList = [...animeIds];
|
||||
for (const videoId of selectIds(
|
||||
db,
|
||||
(placeholders) => `SELECT video_id FROM imm_videos WHERE anime_id IN (${placeholders})`,
|
||||
animeIdList,
|
||||
'video_id',
|
||||
)) {
|
||||
videoIds.add(videoId);
|
||||
}
|
||||
|
||||
const videoIdList = [...videoIds];
|
||||
for (const sessionId of selectIds(
|
||||
db,
|
||||
(placeholders) => `SELECT session_id FROM imm_sessions WHERE video_id IN (${placeholders})`,
|
||||
videoIdList,
|
||||
'session_id',
|
||||
)) {
|
||||
sessionIds.add(sessionId);
|
||||
}
|
||||
|
||||
const sessionIdList = [...sessionIds];
|
||||
const lexicalRemovals = planLexicalRemovalsInChunks(db, sessionIdList);
|
||||
const affectedRollupGroups = sessionIdList
|
||||
.flatMap((_, index) =>
|
||||
index % SQLITE_ID_CHUNK_SIZE === 0
|
||||
? getRollupGroupsForSessions(db, sessionIdList.slice(index, index + SQLITE_ID_CHUNK_SIZE))
|
||||
: [],
|
||||
)
|
||||
.filter((group) => !videoIds.has(group.videoId));
|
||||
const coverBlobHashes = new Set<string>();
|
||||
if (videoIdList.length > 0) {
|
||||
forEachIdChunk(videoIdList, (chunk) => {
|
||||
const placeholders = makePlaceholders(chunk);
|
||||
const artRows = db
|
||||
.prepare(
|
||||
`SELECT cover_blob_hash AS coverBlobHash
|
||||
FROM imm_media_art
|
||||
WHERE video_id IN (${placeholders}) AND cover_blob_hash IS NOT NULL`,
|
||||
)
|
||||
.all(...chunk) as Array<{ coverBlobHash: string }>;
|
||||
for (const row of artRows) coverBlobHashes.add(row.coverBlobHash);
|
||||
});
|
||||
|
||||
deleteSessionsByIds(db, sessionIdList);
|
||||
forEachIdChunk(videoIdList, (chunk) => {
|
||||
const placeholders = makePlaceholders(chunk);
|
||||
db.prepare(`DELETE FROM imm_subtitle_lines WHERE video_id IN (${placeholders})`).run(
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_daily_rollups WHERE video_id IN (${placeholders})`).run(
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_monthly_rollups WHERE video_id IN (${placeholders})`).run(
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_media_art WHERE video_id IN (${placeholders})`).run(...chunk);
|
||||
db.prepare(`DELETE FROM imm_videos WHERE video_id IN (${placeholders})`).run(...chunk);
|
||||
});
|
||||
} else {
|
||||
deleteSessionsByIds(db, sessionIdList);
|
||||
}
|
||||
|
||||
for (const coverBlobHash of coverBlobHashes) {
|
||||
cleanupUnusedCoverArtBlobHash(db, coverBlobHash);
|
||||
}
|
||||
if (animeIdList.length > 0) {
|
||||
forEachIdChunk(animeIdList, (chunk) => {
|
||||
const placeholders = makePlaceholders(chunk);
|
||||
db.prepare(`DELETE FROM imm_lifetime_anime WHERE anime_id IN (${placeholders})`).run(
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_anime WHERE anime_id IN (${placeholders})`).run(...chunk);
|
||||
});
|
||||
}
|
||||
|
||||
applyLexicalRemovals(db, lexicalRemovals);
|
||||
rebuildLifetimeSummariesInTransaction(db);
|
||||
refreshRollupsForGroupsInTransaction(db, affectedRollupGroups);
|
||||
db.exec('COMMIT');
|
||||
} catch (error) {
|
||||
db.exec('ROLLBACK');
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -80,6 +80,14 @@ export function makePlaceholders(values: number[]): string {
|
||||
return values.map(() => '?').join(',');
|
||||
}
|
||||
|
||||
export const SQLITE_ID_CHUNK_SIZE = 1_000;
|
||||
|
||||
export function forEachIdChunk(ids: number[], callback: (chunk: number[]) => void): void {
|
||||
for (let start = 0; start < ids.length; start += SQLITE_ID_CHUNK_SIZE) {
|
||||
callback(ids.slice(start, start + SQLITE_ID_CHUNK_SIZE));
|
||||
}
|
||||
}
|
||||
|
||||
export function resolvedCoverBlobExpr(mediaAlias: string, blobStoreAlias: string): string {
|
||||
return `COALESCE(${blobStoreAlias}.cover_blob, CASE WHEN ${mediaAlias}.cover_blob_hash IS NULL THEN ${mediaAlias}.cover_blob ELSE NULL END)`;
|
||||
}
|
||||
@@ -490,17 +498,19 @@ export function deleteSessionsByIds(db: DatabaseSync, sessionIds: number[]): voi
|
||||
return;
|
||||
}
|
||||
|
||||
const placeholders = makePlaceholders(sessionIds);
|
||||
forEachIdChunk(sessionIds, (chunk) => {
|
||||
const placeholders = makePlaceholders(chunk);
|
||||
db.prepare(`DELETE FROM imm_subtitle_lines WHERE session_id IN (${placeholders})`).run(
|
||||
...sessionIds,
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_session_telemetry WHERE session_id IN (${placeholders})`).run(
|
||||
...sessionIds,
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_session_events WHERE session_id IN (${placeholders})`).run(
|
||||
...sessionIds,
|
||||
...chunk,
|
||||
);
|
||||
db.prepare(`DELETE FROM imm_sessions WHERE session_id IN (${placeholders})`).run(...sessionIds);
|
||||
db.prepare(`DELETE FROM imm_sessions WHERE session_id IN (${placeholders})`).run(...chunk);
|
||||
});
|
||||
}
|
||||
|
||||
export function toDbMs(ms: number | bigint): bigint {
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
/*
|
||||
* Duplicate/animation-burst collapsing for parsed subtitle cues.
|
||||
*
|
||||
* Split out of the cue parser so the parsing rules and the "is this run one animation?"
|
||||
* heuristics can be read -- and tested -- on their own. The parser owns the cue shape;
|
||||
* this module only decides which cues survive.
|
||||
*/
|
||||
|
||||
import { hasAssTemporalOverride, isAnimatedAssEffectKind } from './ass-text';
|
||||
import type {
|
||||
AnnotatedSubtitleCue,
|
||||
SubtitleCue,
|
||||
SubtitleSourceFormat,
|
||||
} from './subtitle-cue-parser';
|
||||
|
||||
// Back-to-back frames of the same animation are authored flush against each other; a
|
||||
// tiny tolerance absorbs the centisecond rounding of the ASS timestamp format.
|
||||
const DUPLICATE_CUE_GAP_TOLERANCE_SECONDS = 0.05;
|
||||
// A burst is a *sequence*. Two adjacent events are two events, not an animation --
|
||||
// characters do repeat each other, and a repeated line can legitimately be short.
|
||||
const MIN_BURST_EVENTS = 3;
|
||||
// Real dialogue holds on screen for about a second, so a run with a couple of much
|
||||
// shorter events among them looks like frames. Used only alongside authoring evidence.
|
||||
const ANIMATION_FRAME_MAX_SECONDS = 0.3;
|
||||
// A karaoke run usually ends on a long "hold" frame, so not every event is short.
|
||||
const MIN_TAGGED_BURST_FRAMES = 2;
|
||||
// SRT and VTT carry no authoring metadata at all, so timing is the only signal available
|
||||
// -- which makes it the easiest one to get wrong. ASS->SRT conversion leaves frames at
|
||||
// ~0.04s, well under any real utterance, and a burst leaves many of them behind. Both
|
||||
// bounds are deliberately far stricter than the ASS path: a run of ordinary short lines
|
||||
// (`えっ` traded between characters) must not clear them.
|
||||
const TIMING_ONLY_FRAME_MAX_SECONDS = 0.1;
|
||||
const MIN_TIMING_ONLY_FRAMES = 5;
|
||||
|
||||
function cueKey(cue: SubtitleCue): string {
|
||||
return `${cue.startTime}|${cue.endTime}|${cue.text}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Identical text over an identical span is redundant however it was authored -- most
|
||||
* often a layered ASS event stacking a shadow copy under the visible one.
|
||||
*/
|
||||
function collapseExactDuplicates(cues: AnnotatedSubtitleCue[]): AnnotatedSubtitleCue[] {
|
||||
const seen = new Set<string>();
|
||||
return cues.filter((cue) => {
|
||||
const key = cueKey(cue);
|
||||
if (seen.has(key)) {
|
||||
return false;
|
||||
}
|
||||
seen.add(key);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
function countFramesShorterThan(run: AnnotatedSubtitleCue[], maxSeconds: number): number {
|
||||
return run.filter((cue) => cue.endTime - cue.startTime < maxSeconds).length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Evidence that a run of ASS events is one animation rather than several authored lines.
|
||||
* A static tag says nothing on its own -- three events sharing one `\clip(...)` are three
|
||||
* signs -- so the tag has to be temporal by nature (`\t`, `\move`, karaoke timing, or
|
||||
* anything wrapped in `\t(...)`), an animated `Effect` column, or a value that actually
|
||||
* changes from event to event, which is how per-frame typesetting is authored.
|
||||
*/
|
||||
export function hasAssAnimationEvidence(run: AnnotatedSubtitleCue[]): boolean {
|
||||
if (run.every((cue) => hasAssTemporalOverride(cue.overrides))) {
|
||||
return true;
|
||||
}
|
||||
if (run.every((cue) => isAnimatedAssEffectKind(cue.effectKind))) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const [first] = run;
|
||||
const everyEventTypeset = run.every((cue) => cue.overrides.length > 0);
|
||||
const signatureChanges = run.some((cue) => cue.overrideSignature !== first!.overrideSignature);
|
||||
return everyEventTypeset && signatureChanges;
|
||||
}
|
||||
|
||||
export function isAnimationBurst(
|
||||
run: AnnotatedSubtitleCue[],
|
||||
format: SubtitleSourceFormat,
|
||||
): boolean {
|
||||
if (run.length < MIN_BURST_EVENTS) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (format === 'srt') {
|
||||
return (
|
||||
run.length >= MIN_TIMING_ONLY_FRAMES &&
|
||||
countFramesShorterThan(run, TIMING_ONLY_FRAME_MAX_SECONDS) === run.length
|
||||
);
|
||||
}
|
||||
|
||||
if (countFramesShorterThan(run, ANIMATION_FRAME_MAX_SECONDS) < MIN_TAGGED_BURST_FRAMES) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// One animation belongs to one styled, one named source line. Two characters trading
|
||||
// the same short word are two styles or two actors, and never merge.
|
||||
const [first] = run;
|
||||
if (run.some((cue) => cue.style !== first!.style || cue.name !== first!.name)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return hasAssAnimationEvidence(run);
|
||||
}
|
||||
|
||||
/**
|
||||
* Karaoke and sign typesetting emits one Dialogue event per animation frame, all carrying
|
||||
* the same visible text over a contiguous span. Collapse each such run into a single cue.
|
||||
*
|
||||
* Only runs that look like animation collapse. Two ordinary lines that happen to repeat
|
||||
* -- several characters each saying `おはよう` in turn, a positioned sign redrawn with a
|
||||
* different fade -- stay separate, because merging them would destroy real mineable lines.
|
||||
*/
|
||||
function collapseAnimationBursts(
|
||||
cues: AnnotatedSubtitleCue[],
|
||||
format: SubtitleSourceFormat,
|
||||
): AnnotatedSubtitleCue[] {
|
||||
const indicesByText = new Map<string, number[]>();
|
||||
cues.forEach((cue, index) => {
|
||||
const bucket = indicesByText.get(cue.text);
|
||||
if (bucket) {
|
||||
bucket.push(index);
|
||||
} else {
|
||||
indicesByText.set(cue.text, [index]);
|
||||
}
|
||||
});
|
||||
|
||||
const dropped = new Set<number>();
|
||||
const extendedEnd = new Map<number, number>();
|
||||
|
||||
for (const indices of indicesByText.values()) {
|
||||
if (indices.length < MIN_BURST_EVENTS) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let runStart = 0;
|
||||
while (runStart < indices.length) {
|
||||
let runEnd = runStart;
|
||||
let chainEnd = cues[indices[runStart]!]!.endTime;
|
||||
|
||||
while (runEnd + 1 < indices.length) {
|
||||
const next = cues[indices[runEnd + 1]!]!;
|
||||
if (next.startTime > chainEnd + DUPLICATE_CUE_GAP_TOLERANCE_SECONDS) {
|
||||
break;
|
||||
}
|
||||
chainEnd = Math.max(chainEnd, next.endTime);
|
||||
runEnd += 1;
|
||||
}
|
||||
|
||||
const run = indices.slice(runStart, runEnd + 1).map((index) => cues[index]!);
|
||||
if (isAnimationBurst(run, format)) {
|
||||
for (let i = runStart + 1; i <= runEnd; i += 1) {
|
||||
dropped.add(indices[i]!);
|
||||
}
|
||||
extendedEnd.set(indices[runStart]!, chainEnd);
|
||||
}
|
||||
|
||||
runStart = runEnd + 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (dropped.size === 0) {
|
||||
return cues;
|
||||
}
|
||||
|
||||
const merged: AnnotatedSubtitleCue[] = [];
|
||||
cues.forEach((cue, index) => {
|
||||
if (dropped.has(index)) {
|
||||
return;
|
||||
}
|
||||
const end = extendedEnd.get(index);
|
||||
merged.push(end !== undefined && end > cue.endTime ? { ...cue, endTime: end } : cue);
|
||||
});
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapse redundant cues. Input must already be sorted by non-decreasing `startTime`,
|
||||
* ties broken by `endTime` then source `order` -- burst detection chains events by
|
||||
* comparing each one against the running end of the events before it, so an unsorted
|
||||
* list breaks runs apart and leaves the frames behind.
|
||||
*/
|
||||
export function mergeDuplicateCues(
|
||||
cues: AnnotatedSubtitleCue[],
|
||||
format: SubtitleSourceFormat,
|
||||
): AnnotatedSubtitleCue[] {
|
||||
return collapseAnimationBursts(collapseExactDuplicates(cues), format);
|
||||
}
|
||||
@@ -91,6 +91,17 @@ test('parseSrtCues skips malformed timing lines gracefully', () => {
|
||||
assert.equal(cues[0]!.text, '有効');
|
||||
});
|
||||
|
||||
test('parseSubtitleCues strips complete brace blocks from SRT and VTT text', () => {
|
||||
const content = ['1', '00:00:01,000 --> 00:00:02,000', '彼は{謎}と言った', ''].join('\n');
|
||||
|
||||
for (const filename of ['test.srt', 'test.vtt']) {
|
||||
const cues = parseSubtitleCues(content, filename);
|
||||
|
||||
assert.equal(cues.length, 1, filename);
|
||||
assert.equal(cues[0]!.text, '彼はと言った', filename);
|
||||
}
|
||||
});
|
||||
|
||||
test('parseAssCues parses basic ASS dialogue lines', () => {
|
||||
const content = [
|
||||
'[Script Info]',
|
||||
@@ -137,7 +148,9 @@ test('parseAssCues handles text containing commas', () => {
|
||||
assert.equal(cues[0]!.text, 'はい、そうです、ね');
|
||||
});
|
||||
|
||||
test('parseAssCues handles \\N line breaks', () => {
|
||||
test('parseAssCues decodes \\N line breaks into real newlines', () => {
|
||||
// ASS is decoded once, here at ingestion, so cue text matches what mpv hands over for
|
||||
// the same line played live.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
@@ -146,7 +159,7 @@ test('parseAssCues handles \\N line breaks', () => {
|
||||
|
||||
const cues = parseAssCues(content);
|
||||
|
||||
assert.equal(cues[0]!.text, '一行目\\N二行目');
|
||||
assert.equal(cues[0]!.text, '一行目\n二行目');
|
||||
});
|
||||
|
||||
test('parseAssCues strips HTML-like markup while preserving ASS line breaks', () => {
|
||||
@@ -158,7 +171,46 @@ test('parseAssCues strips HTML-like markup while preserving ASS line breaks', ()
|
||||
|
||||
const cues = parseAssCues(content);
|
||||
|
||||
assert.equal(cues[0]!.text, '一行目\\N二行目');
|
||||
assert.equal(cues[0]!.text, '一行目\n二行目');
|
||||
});
|
||||
|
||||
test('parseAssCues drops vector drawing runs enabled by \\p', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 1,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
|
||||
'Dialogue: 0,0:00:05.00,0:00:08.00,Default,,0,0,0,,これは字幕',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseAssCues(content);
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.text, 'これは字幕');
|
||||
});
|
||||
|
||||
test('parseAssCues keeps text that follows a \\p0 reset on the same line', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\p1}m 0 0 l 10 10{\\p0}本文{\\p1}m 5 5 l 6 6{\\p0}続き',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseAssCues(content);
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.text, '本文続き');
|
||||
});
|
||||
|
||||
test('parseAssCues leaves \\pos untouched when no drawing mode is active', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,{\\pos(960,1068)\\bord3}位置指定',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseAssCues(content);
|
||||
|
||||
assert.equal(cues[0]!.text, '位置指定');
|
||||
});
|
||||
|
||||
test('parseAssCues returns empty for content without Events section', () => {
|
||||
@@ -258,6 +310,344 @@ test('parseSubtitleCues returns cues sorted by start time', () => {
|
||||
assert.equal(cues[1]!.text, '二番目');
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses per-frame karaoke duplicates into one cue', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}過ぎ去ってしまう瞬間を',
|
||||
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}過ぎ去ってしまう瞬間を',
|
||||
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,,{\\clip(m 3 3)}過ぎ去ってしまう瞬間を',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.startTime, 1.0);
|
||||
assert.equal(cues[0]!.endTime, 3.55);
|
||||
assert.equal(cues[0]!.text, '過ぎ去ってしまう瞬間を');
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps back-to-back plain dialogue repeats separate', () => {
|
||||
// Several characters greeting in turn: distinct utterances that happen to abut.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:04:05.67,0:04:06.82,Dial_JP,,0,0,0,,おはよう',
|
||||
'Dialogue: 0,0:04:06.82,0:04:07.56,Dial_JP,,0,0,0,,おはよう',
|
||||
'Dialogue: 0,0:04:07.56,0:04:08.78,Dial_JP,,0,0,0,,おはよう',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
assert.equal(cues[0]!.endTime, 246.82);
|
||||
assert.equal(cues[2]!.startTime, 247.56);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses exact duplicate cues even without effect tags', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:04.00,Default,,0,0,0,,重なった行',
|
||||
'Dialogue: 1,0:00:01.00,0:00:04.00,Default,,0,0,0,,重なった行',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses tag-less animation frames in converted SRT', () => {
|
||||
// ASS -> SRT conversion drops override tags, so only the ~0.04s frame timing remains.
|
||||
const lines = ['1', '00:00:07,870 --> 00:00:07,910', 'Kaguya Wants to be Confessed to', ''];
|
||||
for (let i = 1; i < 8; i++) {
|
||||
const start = 7910 + (i - 1) * 40;
|
||||
const end = start + 40;
|
||||
const at = (ms: number) =>
|
||||
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
|
||||
lines.push(String(i + 1), `${at(start)} --> ${at(end)}`, 'Kaguya Wants to be Confessed to', '');
|
||||
}
|
||||
|
||||
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.startTime, 7.87);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps identical lines that recur far apart', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:02.00,Default,,0,0,0,,なんで',
|
||||
'Dialogue: 0,0:05:00.00,0:05:01.00,Default,,0,0,0,,なんで',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 2);
|
||||
assert.equal(cues[0]!.startTime, 1.0);
|
||||
assert.equal(cues[1]!.startTime, 300.0);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps two positioned signs that repeat the same text', () => {
|
||||
// Both carry override tags, but `\pos` and `\fad` are static placement, not animation.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:01:00.00,0:01:03.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(200,200)}第一話',
|
||||
'Dialogue: 0,0:01:03.00,0:01:06.00,Sign,,0,0,0,,{\\pos(960,900)\\fad(200,200)}第一話',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 2);
|
||||
assert.equal(cues[1]!.startTime, 63.0);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps a run of ordinary positioned lines separate', () => {
|
||||
// Three events is a sequence, but none of them runs at animation-frame speed.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:01:00.00,0:01:02.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
|
||||
'Dialogue: 0,0:01:02.00,0:01:04.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
|
||||
'Dialogue: 0,0:01:04.00,0:01:06.00,Sign,,0,0,0,,{\\pos(960,120)\\fad(100,100)}止まれ',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps a short repeated SRT pair without burst evidence', () => {
|
||||
const content = [
|
||||
'1',
|
||||
'00:00:01,000 --> 00:00:01,200',
|
||||
'えっ',
|
||||
'',
|
||||
'2',
|
||||
'00:00:01,200 --> 00:00:01,400',
|
||||
'えっ',
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.srt');
|
||||
|
||||
assert.equal(cues.length, 2);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses a burst marked only by the Effect column', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,Karaoke,歌詞',
|
||||
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,Karaoke,歌詞',
|
||||
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,Karaoke,歌詞',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.endTime, 3.55);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps a second karaoke burst that starts after a gap', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}リフレイン',
|
||||
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}リフレイン',
|
||||
'Dialogue: 0,0:00:01.09,0:00:03.00,OP_JP,,0,0,0,,{\\clip(m 3 3)}リフレイン',
|
||||
'Dialogue: 0,0:00:20.00,0:00:20.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}リフレイン',
|
||||
'Dialogue: 0,0:00:20.05,0:00:20.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}リフレイン',
|
||||
'Dialogue: 0,0:00:20.09,0:00:22.00,OP_JP,,0,0,0,,{\\clip(m 3 3)}リフレイン',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 2);
|
||||
assert.equal(cues[0]!.endTime, 3.0);
|
||||
assert.equal(cues[1]!.startTime, 20.0);
|
||||
assert.equal(cues[1]!.endTime, 22.0);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues does not merge a burst into unrelated dialogue between frames', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.05,OP_JP,,0,0,0,,{\\clip(m 1 1)}歌詞',
|
||||
'Dialogue: 0,0:00:01.02,0:00:03.00,Dial_JP,,0,0,0,,別のセリフ',
|
||||
'Dialogue: 0,0:00:01.05,0:00:01.09,OP_JP,,0,0,0,,{\\clip(m 2 2)}歌詞',
|
||||
'Dialogue: 0,0:00:01.09,0:00:03.55,OP_JP,,0,0,0,,{\\clip(m 3 3)}歌詞',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 2);
|
||||
assert.deepEqual(
|
||||
cues.map((cue) => cue.text),
|
||||
['歌詞', '別のセリフ'],
|
||||
);
|
||||
assert.equal(cues[0]!.endTime, 3.55);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps rapid ASS lines from different actors separate', () => {
|
||||
// Three 200ms `えっ` reactions traded between characters. Fast, adjacent and identical,
|
||||
// but authored as three lines: different styles and different actors.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_A,アリス,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_B,ボブ,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_C,キャロル,0,0,0,,えっ',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues reads the speaker column when it is spelled Actor', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Actor, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_JP,アリス,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_JP,ボブ,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_JP,キャロル,0,0,0,,えっ',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues does not treat a custom Effect name as animation', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,scrolling-credit,制作',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,scrolling-credit,制作',
|
||||
'Dialogue: 0,0:00:01.40,0:00:01.60,Sign,,0,0,0,scrolling-credit,制作',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps rapid ASS lines that share a style but not an actor', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Dial_JP,アリス,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Dial_JP,ボブ,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.40,0:00:01.60,Dial_JP,キャロル,0,0,0,,えっ',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps untagged rapid ASS repeats separate', () => {
|
||||
// No overrides at all: timing-only evidence is an SRT/VTT fallback and must not apply
|
||||
// to ASS, where the absence of typesetting is itself evidence of plain dialogue.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.05,Dial_JP,,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.05,0:00:01.10,Dial_JP,,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.10,0:00:01.15,Dial_JP,,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.15,0:00:01.20,Dial_JP,,0,0,0,,えっ',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.25,Dial_JP,,0,0,0,,えっ',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 5);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps repeated signs sharing one static clip', () => {
|
||||
// `\clip` is a static shape for the event. Three events with the identical clip were
|
||||
// typeset the same way, so none of them is a frame of the others.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
|
||||
'Dialogue: 0,0:00:01.40,0:00:01.60,Sign,,0,0,0,,{\\clip(0,0,100,100)}注意',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 3);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues collapses a sign animated through \\t', () => {
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
|
||||
'Dialogue: 0,0:00:01.40,0:00:03.00,Sign,,0,0,0,,{\\pos(10,10)\\t(0,200,\\frz30)}回る',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.ass');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.endTime, 3.0);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps a short repeated SRT run above the frame threshold', () => {
|
||||
// Five contiguous 200ms cues: a sequence, but nowhere near animation-frame speed.
|
||||
const lines: string[] = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const start = 1000 + i * 200;
|
||||
const at = (ms: number) =>
|
||||
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
|
||||
lines.push(String(i + 1), `${at(start)} --> ${at(start + 200)}`, 'えっ', '');
|
||||
}
|
||||
|
||||
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
|
||||
|
||||
assert.equal(cues.length, 5);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues keeps a short SRT frame run below the minimum length', () => {
|
||||
// Four 40ms frames: frame-speed, but too few to tell an animation from an artefact.
|
||||
const lines: string[] = [];
|
||||
for (let i = 0; i < 4; i++) {
|
||||
const start = 7870 + i * 40;
|
||||
const at = (ms: number) =>
|
||||
`00:00:0${Math.floor(ms / 1000)},${String(ms % 1000).padStart(3, '0')}`;
|
||||
lines.push(String(i + 1), `${at(start)} --> ${at(start + 40)}`, 'タイトル', '');
|
||||
}
|
||||
|
||||
const cues = parseSubtitleCues(lines.join('\n'), 'test.srt');
|
||||
|
||||
assert.equal(cues.length, 4);
|
||||
});
|
||||
|
||||
test('parseSubtitleCues applies ASS burst rules to ASS content behind an .srt filename', () => {
|
||||
// The extension lies, so the SRT parser finds nothing and the content-sniffing fallback
|
||||
// takes over -- which has to carry the `ass` source format with it, or the far stricter
|
||||
// timing-only thresholds would let this karaoke burst through as three cues.
|
||||
const content = [
|
||||
'[Events]',
|
||||
'Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text',
|
||||
'Dialogue: 0,0:00:01.00,0:00:01.20,Karaoke,,0,0,0,,{\\k20}歌詞',
|
||||
'Dialogue: 0,0:00:01.20,0:00:01.40,Karaoke,,0,0,0,,{\\k20}歌詞',
|
||||
'Dialogue: 0,0:00:01.40,0:00:03.00,Karaoke,,0,0,0,,{\\k20}歌詞',
|
||||
].join('\n');
|
||||
|
||||
const cues = parseSubtitleCues(content, 'test.srt');
|
||||
|
||||
assert.equal(cues.length, 1);
|
||||
assert.equal(cues[0]!.startTime, 1.0);
|
||||
assert.equal(cues[0]!.endTime, 3.0);
|
||||
assert.equal(cues[0]!.text, '歌詞');
|
||||
});
|
||||
|
||||
test('parseSubtitleCues detects subtitle formats from remote URLs', () => {
|
||||
const assContent = [
|
||||
'[Events]',
|
||||
|
||||
@@ -1,9 +1,46 @@
|
||||
import {
|
||||
assOverrideSignature,
|
||||
assToPlainText,
|
||||
collectAssOverrideCommands,
|
||||
parseAssEffectField,
|
||||
type AssEffectKind,
|
||||
type AssOverrideCommand,
|
||||
} from './ass-text';
|
||||
import { mergeDuplicateCues } from './subtitle-cue-dedup';
|
||||
|
||||
export interface SubtitleCue {
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
text: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Everything the parser knows about a source event, shared only with the dedup engine.
|
||||
* Deduplication needs the authoring context -- which style the line belongs to, which
|
||||
* override commands it carries, whether the `Effect` column was set -- to tell a karaoke
|
||||
* burst apart from two characters saying the same word in turn. None of it is meaningful
|
||||
* outside the parser, so the public API stays `{startTime, endTime, text}`.
|
||||
*/
|
||||
export interface AnnotatedSubtitleCue extends SubtitleCue {
|
||||
/** Text exactly as authored, override blocks and all. */
|
||||
rawText: string;
|
||||
style: string;
|
||||
layer: number;
|
||||
/** ASS `Name`/`Actor` column. */
|
||||
name: string;
|
||||
/** ASS `Effect` column, verbatim. */
|
||||
effect: string;
|
||||
effectKind: AssEffectKind;
|
||||
/** Override commands found in `{...}` blocks, with their arguments. */
|
||||
overrides: readonly AssOverrideCommand[];
|
||||
/** Canonical form of `overrides`, for spotting values that change across a run. */
|
||||
overrideSignature: string;
|
||||
/** Position in the source file, so sorting by time stays deterministic across layers. */
|
||||
order: number;
|
||||
}
|
||||
|
||||
export type SubtitleSourceFormat = 'ass' | 'srt';
|
||||
|
||||
const HTML_SUBTITLE_TAG_PATTERN = /<\/?[A-Za-z][^>\n]*>/g;
|
||||
|
||||
const SRT_TIMING_PATTERN =
|
||||
@@ -23,12 +60,21 @@ function parseTimestamp(
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The single ASS decode for the file path: cues leave the parser as plain text with real
|
||||
* line breaks, matching what mpv hands over for the same line played live. No layer
|
||||
* downstream decodes ASS again.
|
||||
*/
|
||||
function sanitizeSubtitleCueText(text: string): string {
|
||||
return text.replace(ASS_OVERRIDE_TAG_PATTERN, '').replace(HTML_SUBTITLE_TAG_PATTERN, '').trim();
|
||||
return assToPlainText(text, '\n').replace(HTML_SUBTITLE_TAG_PATTERN, '').trim();
|
||||
}
|
||||
|
||||
export function parseSrtCues(content: string): SubtitleCue[] {
|
||||
const cues: SubtitleCue[] = [];
|
||||
function toPublicCues(cues: AnnotatedSubtitleCue[]): SubtitleCue[] {
|
||||
return cues.map(({ startTime, endTime, text }) => ({ startTime, endTime, text }));
|
||||
}
|
||||
|
||||
function parseAnnotatedSrtCues(content: string): AnnotatedSubtitleCue[] {
|
||||
const cues: AnnotatedSubtitleCue[] = [];
|
||||
const lines = content.split(/\r?\n/);
|
||||
let i = 0;
|
||||
|
||||
@@ -60,20 +106,39 @@ export function parseSrtCues(content: string): SubtitleCue[] {
|
||||
i += 1;
|
||||
}
|
||||
|
||||
const text = sanitizeSubtitleCueText(textLines.join('\n'));
|
||||
const rawText = textLines.join('\n');
|
||||
const text = sanitizeSubtitleCueText(rawText);
|
||||
if (text) {
|
||||
cues.push({ startTime, endTime, text });
|
||||
cues.push({
|
||||
startTime,
|
||||
endTime,
|
||||
text,
|
||||
rawText,
|
||||
style: '',
|
||||
layer: 0,
|
||||
name: '',
|
||||
effect: '',
|
||||
effectKind: 'none',
|
||||
// SRT and VTT carry no authoring metadata, and the dedup engine never reads
|
||||
// overrides for those formats -- collecting them would be parsing for nobody.
|
||||
overrides: [],
|
||||
overrideSignature: '',
|
||||
order: cues.length,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return cues;
|
||||
}
|
||||
|
||||
const ASS_OVERRIDE_TAG_PATTERN = /\{[^}]*\}/g;
|
||||
export function parseSrtCues(content: string): SubtitleCue[] {
|
||||
return toPublicCues(parseAnnotatedSrtCues(content));
|
||||
}
|
||||
|
||||
const ASS_TIMING_PATTERN = /^(\d+):(\d{2}):(\d{2})\.(\d{1,2})$/;
|
||||
const ASS_FORMAT_PREFIX = 'Format:';
|
||||
const ASS_DIALOGUE_PREFIX = 'Dialogue:';
|
||||
const ASS_NAME_FIELD_ALIASES = ['name', 'actor'];
|
||||
|
||||
function parseAssTimestamp(raw: string): number | null {
|
||||
const match = ASS_TIMING_PATTERN.exec(raw.trim());
|
||||
@@ -87,13 +152,43 @@ function parseAssTimestamp(raw: string): number | null {
|
||||
return hours * 3600 + minutes * 60 + seconds + centiseconds / 100;
|
||||
}
|
||||
|
||||
export function parseAssCues(content: string): SubtitleCue[] {
|
||||
const cues: SubtitleCue[] = [];
|
||||
function readField(fields: string[], index: number): string {
|
||||
return index >= 0 && index < fields.length ? fields[index]!.trim() : '';
|
||||
}
|
||||
|
||||
function findFieldIndex(formatFields: string[], aliases: string[]): number {
|
||||
for (const alias of aliases) {
|
||||
const index = formatFields.indexOf(alias);
|
||||
if (index >= 0) {
|
||||
return index;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
|
||||
const cues: AnnotatedSubtitleCue[] = [];
|
||||
const lines = content.split(/\r?\n/);
|
||||
let inEventsSection = false;
|
||||
let startFieldIndex = -1;
|
||||
let endFieldIndex = -1;
|
||||
let textFieldIndex = -1;
|
||||
const fieldIndex = {
|
||||
start: -1,
|
||||
end: -1,
|
||||
text: -1,
|
||||
style: -1,
|
||||
layer: -1,
|
||||
name: -1,
|
||||
effect: -1,
|
||||
};
|
||||
|
||||
const resetFieldIndex = () => {
|
||||
fieldIndex.start = -1;
|
||||
fieldIndex.end = -1;
|
||||
fieldIndex.text = -1;
|
||||
fieldIndex.style = -1;
|
||||
fieldIndex.layer = -1;
|
||||
fieldIndex.name = -1;
|
||||
fieldIndex.effect = -1;
|
||||
};
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
@@ -101,9 +196,7 @@ export function parseAssCues(content: string): SubtitleCue[] {
|
||||
if (trimmed.startsWith('[') && trimmed.endsWith(']')) {
|
||||
inEventsSection = trimmed.toLowerCase() === '[events]';
|
||||
if (!inEventsSection) {
|
||||
startFieldIndex = -1;
|
||||
endFieldIndex = -1;
|
||||
textFieldIndex = -1;
|
||||
resetFieldIndex();
|
||||
}
|
||||
continue;
|
||||
}
|
||||
@@ -117,9 +210,15 @@ export function parseAssCues(content: string): SubtitleCue[] {
|
||||
.slice(ASS_FORMAT_PREFIX.length)
|
||||
.split(',')
|
||||
.map((field) => field.trim().toLowerCase());
|
||||
startFieldIndex = formatFields.indexOf('start');
|
||||
endFieldIndex = formatFields.indexOf('end');
|
||||
textFieldIndex = formatFields.indexOf('text');
|
||||
fieldIndex.start = formatFields.indexOf('start');
|
||||
fieldIndex.end = formatFields.indexOf('end');
|
||||
fieldIndex.text = formatFields.indexOf('text');
|
||||
fieldIndex.style = formatFields.indexOf('style');
|
||||
fieldIndex.layer = formatFields.indexOf('layer');
|
||||
// Aegisub writes the speaker column as `Actor`; the v4+ spec calls it `Name`.
|
||||
// Missing it costs the burst check its speaker guard, so both spellings count.
|
||||
fieldIndex.name = findFieldIndex(formatFields, ASS_NAME_FIELD_ALIASES);
|
||||
fieldIndex.effect = formatFields.indexOf('effect');
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -127,34 +226,57 @@ export function parseAssCues(content: string): SubtitleCue[] {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (startFieldIndex < 0 || endFieldIndex < 0 || textFieldIndex < 0) {
|
||||
if (fieldIndex.start < 0 || fieldIndex.end < 0 || fieldIndex.text < 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const fields = trimmed.slice(ASS_DIALOGUE_PREFIX.length).split(',');
|
||||
if (
|
||||
startFieldIndex >= fields.length ||
|
||||
endFieldIndex >= fields.length ||
|
||||
textFieldIndex >= fields.length
|
||||
fieldIndex.start >= fields.length ||
|
||||
fieldIndex.end >= fields.length ||
|
||||
fieldIndex.text >= fields.length
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const startTime = parseAssTimestamp(fields[startFieldIndex]!);
|
||||
const endTime = parseAssTimestamp(fields[endFieldIndex]!);
|
||||
const startTime = parseAssTimestamp(fields[fieldIndex.start]!);
|
||||
const endTime = parseAssTimestamp(fields[fieldIndex.end]!);
|
||||
if (startTime === null || endTime === null) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const text = sanitizeSubtitleCueText(fields.slice(textFieldIndex).join(','));
|
||||
if (text) {
|
||||
cues.push({ startTime, endTime, text });
|
||||
const rawText = fields.slice(fieldIndex.text).join(',');
|
||||
const text = sanitizeSubtitleCueText(rawText);
|
||||
if (!text) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const effect = readField(fields, fieldIndex.effect);
|
||||
const layer = Number(readField(fields, fieldIndex.layer));
|
||||
const overrides = collectAssOverrideCommands(rawText);
|
||||
cues.push({
|
||||
startTime,
|
||||
endTime,
|
||||
text,
|
||||
rawText,
|
||||
style: readField(fields, fieldIndex.style),
|
||||
layer: Number.isFinite(layer) ? layer : 0,
|
||||
name: readField(fields, fieldIndex.name),
|
||||
effect,
|
||||
effectKind: parseAssEffectField(effect),
|
||||
overrides,
|
||||
overrideSignature: assOverrideSignature(overrides),
|
||||
order: cues.length,
|
||||
});
|
||||
}
|
||||
|
||||
return cues;
|
||||
}
|
||||
|
||||
export function parseAssCues(content: string): SubtitleCue[] {
|
||||
return toPublicCues(parseAnnotatedAssCues(content));
|
||||
}
|
||||
|
||||
function detectSubtitleFormat(source: string): 'srt' | 'vtt' | 'ass' | 'ssa' | null {
|
||||
const [normalizedSource = source] =
|
||||
(() => {
|
||||
@@ -173,27 +295,31 @@ function detectSubtitleFormat(source: string): 'srt' | 'vtt' | 'ass' | 'ssa' | n
|
||||
|
||||
export function parseSubtitleCues(content: string, filename: string): SubtitleCue[] {
|
||||
const format = detectSubtitleFormat(filename);
|
||||
let cues: SubtitleCue[];
|
||||
let cues: AnnotatedSubtitleCue[];
|
||||
let sourceFormat: SubtitleSourceFormat = 'srt';
|
||||
|
||||
switch (format) {
|
||||
case 'srt':
|
||||
case 'vtt':
|
||||
cues = parseSrtCues(content);
|
||||
cues = parseAnnotatedSrtCues(content);
|
||||
break;
|
||||
case 'ass':
|
||||
case 'ssa':
|
||||
cues = parseAssCues(content);
|
||||
cues = parseAnnotatedAssCues(content);
|
||||
sourceFormat = 'ass';
|
||||
break;
|
||||
default:
|
||||
cues = [];
|
||||
}
|
||||
|
||||
if (cues.length === 0) {
|
||||
const assCues = parseAssCues(content);
|
||||
const srtCues = parseSrtCues(content);
|
||||
cues = assCues.length >= srtCues.length ? assCues : srtCues;
|
||||
const assCues = parseAnnotatedAssCues(content);
|
||||
const srtCues = parseAnnotatedSrtCues(content);
|
||||
const preferAss = assCues.length >= srtCues.length;
|
||||
cues = preferAss ? assCues : srtCues;
|
||||
sourceFormat = preferAss && assCues.length > 0 ? 'ass' : 'srt';
|
||||
}
|
||||
|
||||
cues.sort((a, b) => a.startTime - b.startTime);
|
||||
return cues;
|
||||
cues.sort((a, b) => a.startTime - b.startTime || a.endTime - b.endTime || a.order - b.order);
|
||||
return toPublicCues(mergeDuplicateCues(cues, sourceFormat));
|
||||
}
|
||||
|
||||
@@ -115,6 +115,21 @@ test('subtitle processing does not emit plain payload for cached lines', async (
|
||||
assert.deepEqual(emitted, [{ text: '字幕', tokens: [] }]);
|
||||
});
|
||||
|
||||
test('text that normalizes to nothing is never cached', () => {
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: () => {},
|
||||
});
|
||||
|
||||
// Two different inputs both reduce to an empty key; sharing one entry would serve the
|
||||
// first one's tokens for the second.
|
||||
controller.preCacheTokenization(' ', { text: ' ', tokens: [] });
|
||||
|
||||
assert.equal(controller.hasCachedSubtitle(' '), false);
|
||||
assert.equal(controller.hasCachedSubtitle('\\n'), false);
|
||||
assert.equal(controller.consumeCachedSubtitle('\\n'), null);
|
||||
});
|
||||
|
||||
test('subtitle processing shows plain line while tokenization is still pending', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveTokenization: ((value: SubtitleData | null) => void) | undefined;
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { SubtitleData } from '../../types';
|
||||
import { normalizePlainSubtitleText } from './ass-text';
|
||||
|
||||
export interface SubtitleProcessingControllerDeps {
|
||||
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
||||
@@ -47,8 +48,15 @@ export interface SubtitleProcessingController {
|
||||
hasCachedSubtitle: (text: string) => boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefetched cues and live mpv text are both already decoded from ASS, so the key only
|
||||
* has to settle whitespace for one authored line to resolve to one entry.
|
||||
*
|
||||
* An empty key is not a line: it is whatever normalization reduced to nothing. Callers
|
||||
* must skip the cache for it rather than let every such input share one entry.
|
||||
*/
|
||||
export function normalizeSubtitleCacheKey(text: string): string {
|
||||
return text.replace(/\r\n/g, '\n').replace(/\\N/g, '\n').replace(/\\n/g, '\n').trim();
|
||||
return normalizePlainSubtitleText(text);
|
||||
}
|
||||
|
||||
export function createSubtitleProcessingController(
|
||||
@@ -72,6 +80,9 @@ export function createSubtitleProcessingController(
|
||||
|
||||
const getCachedTokenization = (text: string): SubtitleData | null => {
|
||||
const cacheKey = normalizeSubtitleCacheKey(text);
|
||||
if (!cacheKey) {
|
||||
return null;
|
||||
}
|
||||
const cached = tokenizationCache.get(cacheKey);
|
||||
if (!cached) {
|
||||
return null;
|
||||
@@ -83,7 +94,11 @@ export function createSubtitleProcessingController(
|
||||
};
|
||||
|
||||
const setCachedTokenization = (text: string, payload: SubtitleData): void => {
|
||||
tokenizationCache.set(normalizeSubtitleCacheKey(text), payload);
|
||||
const cacheKey = normalizeSubtitleCacheKey(text);
|
||||
if (!cacheKey) {
|
||||
return;
|
||||
}
|
||||
tokenizationCache.set(cacheKey, payload);
|
||||
while (tokenizationCache.size > SUBTITLE_TOKENIZATION_CACHE_LIMIT) {
|
||||
const firstKey = tokenizationCache.keys().next().value;
|
||||
if (firstKey !== undefined) {
|
||||
@@ -254,7 +269,8 @@ export function createSubtitleProcessingController(
|
||||
return cached;
|
||||
},
|
||||
hasCachedSubtitle: (text: string) => {
|
||||
return tokenizationCache.has(normalizeSubtitleCacheKey(text));
|
||||
const cacheKey = normalizeSubtitleCacheKey(text);
|
||||
return cacheKey.length > 0 && tokenizationCache.has(cacheKey);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1651,9 +1651,11 @@ test('tokenizeSubtitle clears JLPT level from standalone Yomitan particle token'
|
||||
assert.equal(result.tokens?.[0]?.jlptLevel, undefined);
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle returns null tokens for empty normalized text', async () => {
|
||||
test('tokenizeSubtitle returns the normalized text when it comes out empty', async () => {
|
||||
// Handing back the original would push whatever normalization dropped into app state
|
||||
// as if it were subtitle text.
|
||||
const result = await tokenizeSubtitle(' \\n ', makeDeps());
|
||||
assert.deepEqual(result, { text: ' \\n ', tokens: null });
|
||||
assert.deepEqual(result, { text: '', tokens: null });
|
||||
});
|
||||
|
||||
test('tokenizeSubtitle normalizes newlines before Yomitan parse request', async () => {
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
} from './tokenizer/yomitan-parser-runtime';
|
||||
import type { YomitanTermFrequency } from './tokenizer/yomitan-parser-runtime';
|
||||
import { isKanaChar } from './tokenizer/token-classification';
|
||||
import { normalizePlainSubtitleText } from './ass-text';
|
||||
|
||||
const logger = createLogger('main:tokenizer');
|
||||
|
||||
@@ -886,14 +887,14 @@ export async function tokenizeSubtitle(
|
||||
text: string,
|
||||
deps: TokenizerServiceDeps,
|
||||
): Promise<SubtitleData> {
|
||||
const displayText = text
|
||||
.replace(/\r\n/g, '\n')
|
||||
.replace(/\\N/g, '\n')
|
||||
.replace(/\\n/g, '\n')
|
||||
.trim();
|
||||
const displayText = normalizePlainSubtitleText(text);
|
||||
|
||||
// ASS decoding already happened upstream (cue parser for files, mpv for live text), so
|
||||
// all this drops is whitespace -- but a whitespace-only line still normalizes to empty.
|
||||
// Return the normalized form anyway: handing back the original would put a blank line
|
||||
// into application state as if it were subtitle text.
|
||||
if (!displayText) {
|
||||
return { text, tokens: null };
|
||||
return { text: displayText, tokens: null };
|
||||
}
|
||||
|
||||
const tokenizeText = displayText
|
||||
|
||||
@@ -1004,6 +1004,24 @@ test('normalizeSubtitle collapses explicit line breaks when collapseLineBreaks i
|
||||
);
|
||||
});
|
||||
|
||||
test('normalizeSubtitle leaves already-decoded text alone', () => {
|
||||
// Primary subtitle text is decoded from ASS once, upstream: by mpv for live lines and
|
||||
// by the cue parser for prefetched ones. A brace that survives that is literal text.
|
||||
assert.equal(normalizeSubtitle('本文{\\pos(1,2)'), '本文{\\pos(1,2)');
|
||||
assert.equal(normalizeSubtitle(' 余白 ', false), ' 余白 ');
|
||||
});
|
||||
|
||||
test('prepareSecondarySubtitleLines drops ASS vector drawing runs', () => {
|
||||
assert.deepEqual(
|
||||
prepareSecondarySubtitleLines(
|
||||
'{\\an5\\pos(730,1042)\\p1\\blur1}m 20 0 b 10 0 0 10 0 20 b 0 31 10 40 20 40 {\\p0}',
|
||||
),
|
||||
[],
|
||||
);
|
||||
assert.deepEqual(prepareSecondarySubtitleLines('{\\p1}m 0 0 l 10 10{\\p0}本文'), ['本文']);
|
||||
assert.deepEqual(prepareSecondarySubtitleLines('{\\pos(960,1068)\\bord3}位置指定'), ['位置指定']);
|
||||
});
|
||||
|
||||
test('shouldRenderTokenizedSubtitle enables token rendering when tokens exist', () => {
|
||||
assert.equal(shouldRenderTokenizedSubtitle(5), true);
|
||||
assert.equal(shouldRenderTokenizedSubtitle(0), false);
|
||||
|
||||
@@ -5,6 +5,7 @@ import type {
|
||||
SubtitleData,
|
||||
SubtitleRendererStyleConfig,
|
||||
} from '../types';
|
||||
import { assToPlainText, normalizePlainSubtitleText } from '../core/services/ass-text.js';
|
||||
import type { RendererContext } from './context';
|
||||
import { PRIMARY_SUB_VISIBLE_ON_YOMITAN_POPUP_CLASS } from './yomitan-popup.js';
|
||||
|
||||
@@ -42,17 +43,10 @@ function isWhitespaceOnly(value: string): boolean {
|
||||
return value.trim().length === 0;
|
||||
}
|
||||
|
||||
// Text reaching the overlay has already been decoded from ASS -- by mpv for live lines,
|
||||
// by the cue parser for prefetched ones -- so this only settles line breaks.
|
||||
export function normalizeSubtitle(text: string, trim = true, collapseLineBreaks = false): string {
|
||||
if (!text) return '';
|
||||
|
||||
let normalized = text.replace(/\\N/g, '\n').replace(/\\n/g, '\n');
|
||||
normalized = normalized.replace(/\{[^}]*\}/g, '');
|
||||
if (collapseLineBreaks) {
|
||||
normalized = normalized.replace(/\n/g, ' ');
|
||||
normalized = normalized.replace(/\s+/g, ' ');
|
||||
}
|
||||
|
||||
return trim ? normalized.trim() : normalized;
|
||||
return normalizePlainSubtitleText(text, { trim, collapseLineBreaks });
|
||||
}
|
||||
|
||||
const HEX_COLOR_PATTERN = /^#(?:[0-9a-fA-F]{3}|[0-9a-fA-F]{4}|[0-9a-fA-F]{6}|[0-9a-fA-F]{8})$/;
|
||||
@@ -672,7 +666,10 @@ function isKaraokeLikeLineSet(lines: string[]): boolean {
|
||||
}
|
||||
|
||||
export function prepareSecondarySubtitleLines(text: string): string[] {
|
||||
const normalized = normalizeSubtitle(text, true, false);
|
||||
// The one display-side ASS decode: secondary text also reaches the overlay from
|
||||
// websocket clients that forward their source line untouched, so unlike the primary
|
||||
// path it cannot assume mpv already decoded it.
|
||||
const normalized = assToPlainText(text).trim();
|
||||
|
||||
if (!normalized) return [];
|
||||
|
||||
|
||||
@@ -16,6 +16,8 @@
|
||||
* along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
import { normalizePlainSubtitleText } from './core/services/ass-text';
|
||||
|
||||
interface TimingEntry {
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
@@ -191,23 +193,14 @@ export class SubtitleTimingTracker {
|
||||
return costs[shorter.length] || 0;
|
||||
}
|
||||
|
||||
// Both sides take text mpv has already decoded from ASS; only whitespace differs
|
||||
// between the lookup key (single line) and the display form (line breaks kept).
|
||||
private normalizeText(text: string): string {
|
||||
return text
|
||||
.replace(/\\N/g, ' ')
|
||||
.replace(/\\n/g, ' ')
|
||||
.replace(/\n/g, ' ')
|
||||
.replace(/{[^}]*}/g, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
return normalizePlainSubtitleText(text, { collapseLineBreaks: true });
|
||||
}
|
||||
|
||||
private prepareDisplayText(text: string): string {
|
||||
// Convert ASS/SSA newlines to real newlines, strip tags
|
||||
return text
|
||||
.replace(/\\N/g, '\n')
|
||||
.replace(/\\n/g, '\n')
|
||||
.replace(/{[^}]*}/g, '')
|
||||
.trim();
|
||||
return normalizePlainSubtitleText(text);
|
||||
}
|
||||
|
||||
private startCleanup(): void {
|
||||
|
||||
Reference in New Issue
Block a user