fix(youtube): keep auto captions timed and paged correctly

- Page oversized rolling captions within YouTube’s row and column limits
- Preserve explicit durations for sound cues

changes/youtube-auto-caption-duration.md

@@ -1,4 +1,4 @@
 type: fixed
 area: youtube
 
-- YouTube auto-generated captions no longer disappear mid-speech: long caption rows carry a placeholder 3-second duration in YouTube's TimedText data, so converted cues now stay on screen until the next caption event, matching YouTube's own display timing.
+- YouTube auto-generated captions now follow their intended timing and two-row roll-up layout: long speech is paged instead of covering the video with a wall of text, while explicitly timed sound cues such as `[音楽]` no longer cover later dialogue.

release/release-notes.md

@@ -1,78 +0,0 @@
-## Highlights
-### Added
-- **Library Duplicate & Misfiled Episode Tools**
-  - Merge duplicate show cards from the Library grid: select cards and use "Merge Selected" to combine sessions, mined cards, and watch time onto one entry while keeping remembered title aliases.
-  - Reassign a misfiled episode to the correct show with the "→" button on an episode row; the fix survives later filename parsing, Jellyfin refreshes, and season repair.
-  - Exact AniList matches merge automatically, while likely (fuzzy) matches surface as a dismissible "Possible duplicate" suggestion instead of merging without confirmation.
-- **Stats Duplicate-Line Cleanup Tool**
-  - The Vocabulary tab's new Duplicates button scans a chosen time window for old karaoke/animation duplicate bursts and collapses each one to a single line after you confirm, without touching watch time or lines-seen totals.
-  - The same cleanup is available from the terminal via `subminer stats cleanup --duplicate-lines`, with `--dry-run` and `--lookback-days` options.
-
-### Changed
-- **Prerelease Notes "Changes Since" Section**
-  - Prerelease release notes now open with a "Changes since" section listing only what changed since the previous beta/RC of the same version, shown above the full cumulative highlights.
-
-### Fixed
-- **Subtitle Deduplication & Karaoke Reconstruction**
-  - Typeset ASS karaoke and animated signs are reconstructed into their authored line and shown once, instead of flooding the overlay, subtitle sidebar, immersion history, sentence mining, and stats with per-frame glyph fragments and repeated lyric bursts (a lyric could previously pin itself to the top of "Top Repeated Words").
-  - The same deduplication now applies consistently everywhere, including embedded subtitles extracted from network-mounted (SMB/NFS) media and the secondary subtitle overlay, while ordinary repeated dialogue, signs, and rewatches remain unaffected.
-  - Secondary subtitle overlays no longer clip long lines after about four rows, and no longer show scattered-letter or duplicated text while embedded subtitles are still being extracted.
-- **Character Dictionary Reliability & Notifications**
-  - Character dictionary generation, rebuilds, and imports no longer freeze the app or trigger "not responding" dialogs on large dictionaries; the heavy work now runs off the main UI thread.
-  - Dictionaries are reused instead of being regenerated on every launch when no name splits were found, and portraits reappear correctly once the cached portrait index finishes loading.
-  - Linux desktop progress notifications, including on AppImage installs, now update in place instead of flickering closed and reopening.
-- **Overlay Startup Reliability**
-  - The overlay no longer gets stuck on an endless "Overlay loading" screen when mpv's connection stalls at startup; connections now time out and retry, and a clear error appears if content still isn't ready after 30 seconds.
-- **Overlay Modal Windows (macOS & Windows)**
-  - Modal windows such as Settings prewarm so shortcuts open them promptly on first press.
-  - On Windows, the hidden modal renderer now refreshes between sessions so later modals stay interactive.
-  - On macOS, reused modals and the stats window open above fullscreen mpv on the correct Space instead of jumping to another desktop; the overlay-attach helper also now supports macOS 12.0+, fixing "Overlay loading" getting stuck on older macOS versions.
-- **Windows Mouse Lag**
-  - Fixed system-wide mouse lag while SubMiner is running: the overlay no longer installs a global mouse hook, and the mpv window tracker no longer blocks the app with repeated command-line lookups.
-- **Linux Overlay & Launcher Fixes**
-  - Native Wayland drag-and-drop from file managers such as Thunar now works, so subtitle and video files dropped on the overlay reach mpv.
-  - Fixed missing MKV thumbnails in the rofi file picker on systems that only advertise legacy Matroska MIME aliases.
-- **Sentence Mining Audio & Clip Accuracy**
-  - Sentence-audio generation no longer times out on slow network-mounted MKV files with many subtitle/font streams; probing is now bounded with a two-minute extraction budget and a clear error instead of a raw failure.
-  - Mined audio and animated clips now capture the exact subtitle line that was mined, instead of whatever line was on screen after audio extraction finished, fixing too-short or misaligned clips.
-- **Stats Reliability & Performance**
-  - Fixed transient database-lock errors when multiple stats workers wrote at once.
-  - Stats deletes, library merges, video moves, and AniList reassignments no longer freeze the dashboard or rebuild lifetime totals from scratch, so they're fast and preserve lifetime totals older than the recent session-retention window; session deletes on large databases dropped from minutes to milliseconds.
-- **Vocabulary Tab Accuracy**
-  - Vocabulary totals and charts now count all tracked vocabulary instead of only the first page, with new-word history rebuilt from corrected daily rollups to match.
-  - Calendar charts keep the correct local date in time zones west of UTC, and vocabulary cards/charts now refresh automatically and retry after the word exclusion list changes.
-
-## What's Changed
-
-- feat(stats): add library entry merge and episode move by @ksyasuda in #190
-- fix(stats): stop counting duplicate typeset subtitle lines by @ksyasuda in #191
-- fix(media): tolerate slow MKV audio extraction by @ksyasuda in #195
-- fix(stats): subtract lifetime totals incrementally on delete by @ksyasuda in #196
-- fix(anki): snapshot mining media clip timing by @ksyasuda in #197
-- fix(notifications): replace Linux progress updates in place by @ksyasuda in #198
-- fix(overlay): support native Wayland file drag-and-drop by @ksyasuda in #199
-- fix(overlay): keep macOS modal windows on fullscreen Spaces by @ksyasuda in #200
-- fix(overlay): prevent Windows mouse lag during click-through tracking by @ksyasuda in #201
-- fix(stats): report complete vocabulary totals and new-word history by @ksyasuda in #202
-- fix(mpv): recover from stalled IPC connects by @ksyasuda in #204
-- fix(dictionary): prevent freezes and restore AppImage notifications by @ksyasuda in #205
-- fix(subtitles): recover canonical lines from ASS animation by @ksyasuda in #207
-- fix(overlay): deduplicate secondary subtitle rendering by @ksyasuda in #208
-- fix(launcher): restore Matroska thumbnails in Linux rofi picker by @ksyasuda in #210
-- fix(character-dictionary): cache completed MeCab refreshes by @ksyasuda in #212
-- fix(subtitles): improve secondary subtitle extraction and display by @ksyasuda in #215
-- feat(release): track prerelease deltas and validate committed notes by @ksyasuda in #216
-- fix(subtitles): recover positioned ASS word spacing and drop control debris by @ksyasuda in #217
-
-## Installation
-
-See the README and docs/installation guide for full setup steps.
-
-## Assets
-
-- Linux: `SubMiner.AppImage`
-- macOS: `SubMiner-*.dmg` and `SubMiner-*.zip`
-- Windows: `SubMiner-*.exe` and `SubMiner-*-win.zip`
-- Optional extras: `subminer-assets.tar.gz` and the `subminer` launcher
-
-Note: the `subminer` wrapper script uses Bun (`#!/usr/bin/env bun`), so `bun` must be installed and on `PATH`.

src/core/services/youtube/timedtext.test.ts

@@ -69,6 +69,88 @@ test('convertYoutubeTimedTextToVtt extends rolling captions to the next window e
   );
 });
 
+test('convertYoutubeTimedTextToVtt pages oversized two-row rolling captions', () => {
+  const text =
+    'あの西に結構こう山田がスーパーアプローチしてるんだけど西気づかないからちょっとこっちも気づかない感じでこう接してあげようかなて思ってんだけどあの唇巻き込んじゃうしあの思ってることも全部縁に出ちゃって自分であちゃったって言っちゃうタイプなんで結構なんかこうドライなんだけどそこがおもろいよねみたいな';
+  const result = convertYoutubeTimedTextToVtt(
+    [
+      '<timedtext format="3">',
+      '<head>',
+      '<ws id="1" mh="2" ju="0" sd="3"/>',
+      '<wp id="1" ap="6" ah="20" av="100" rc="2" cc="40"/>',
+      '</head>',
+      '<body>',
+      '<w t="0" id="1" wp="1" ws="1"/>',
+      `<p t="60440" d="3000" w="1"><s ac="0">${text}</s></p>`,
+      '<p t="72695" w="1" a="1">\n</p>',
+      '</body>',
+      '</timedtext>',
+    ].join('\n'),
+  );
+
+  const cues = result
+    .trim()
+    .split(/\n\n/)
+    .filter((block) => block.includes('-->'));
+  const cueText = cues.map((cue) => cue.split('\n').slice(1).join('\n'));
+
+  assert.equal(cues.length, 2);
+  assert.deepEqual(
+    cues.map((cue) => cue.split('\n')[0]),
+    ['00:01:00.440 --> 00:01:07.064', '00:01:07.064 --> 00:01:12.695'],
+  );
+  assert.ok(cueText.every((page) => [...page].length <= 80));
+  assert.equal(cueText.join(''), text);
+});
+
+test('convertYoutubeTimedTextToVtt leaves pop-on captions intact', () => {
+  const result = convertYoutubeTimedTextToVtt(
+    [
+      '<timedtext format="3">',
+      '<head>',
+      '<ws id="1" mh="0"/>',
+      '<wp id="1" rc="2" cc="4"/>',
+      '</head>',
+      '<body>',
+      '<w t="0" id="1" wp="1" ws="1"/>',
+      '<p t="1000" d="3000" w="1">abcdefghijklmnopqrst</p>',
+      '</body>',
+      '</timedtext>',
+    ].join('\n'),
+  );
+
+  assert.equal(
+    result,
+    ['WEBVTT', '', '00:00:01.000 --> 00:00:04.000', 'abcdefghijklmnopqrst', ''].join('\n'),
+  );
+});
+
+test('convertYoutubeTimedTextToVtt keeps explicit sound-cue durations in rolling documents', () => {
+  const result = convertYoutubeTimedTextToVtt(
+    [
+      '<timedtext><body>',
+      '<p t="20305" d="2020" w="1">[音楽]</p>',
+      '<p t="26269" w="1" a="1">\n</p>',
+      '<p t="26279" d="3000" w="1"><s ac="0">じゃあ、君からお願いします。</s></p>',
+      '</body></timedtext>',
+    ].join('\n'),
+  );
+
+  assert.equal(
+    result,
+    [
+      'WEBVTT',
+      '',
+      '00:00:20.305 --> 00:00:22.325',
+      '[音楽]',
+      '',
+      '00:00:26.279 --> 00:00:29.279',
+      'じゃあ、君からお願いします。',
+      '',
+    ].join('\n'),
+  );
+});
+
 test('normalizeYoutubeAutoVtt strips cumulative rolling-caption prefixes', () => {
   const result = normalizeYoutubeAutoVtt(
     [

src/core/services/youtube/timedtext.ts

@@ -2,19 +2,30 @@ interface YoutubeTimedTextRow {
   startMs: number;
   durationMs: number;
   text: string;
+  rollingWindow: YoutubeRollingWindow | null;
+}
+
+interface YoutubeRollingWindow {
+  rowCount: number;
+  columnCount: number;
+}
+
+interface YoutubeTimedTextWindowDefinitions {
+  rollingStyleIds: Set<string>;
+  positions: Map<string, YoutubeRollingWindow>;
+  windows: Map<string, YoutubeRollingWindow>;
 }
 
 interface YoutubeTimedTextDocument {
   rows: YoutubeTimedTextRow[];
   // Start times of every <p> event, including empty window-append fillers.
-  // In the rolling auto-caption format YouTube displays each caption until the
-  // next window event and the row's own duration is often a 3000ms placeholder,
-  // so these timestamps are the only reliable source for cue end times.
+  // Rolling speech rows with a 3000ms placeholder display until the next event.
   eventStartsMs: number[];
   hasRollingWindowEvents: boolean;
 }
 
 const YOUTUBE_TIMEDTEXT_EXTENSIONS = new Set(['srv1', 'srv2', 'srv3', 'ytsrv3']);
+const YOUTUBE_ROLLING_PLACEHOLDER_DURATION_MS = 3_000;
 
 function decodeNumericEntity(match: string, codePoint: number): string {
   if (
@@ -49,10 +60,81 @@ function parseAttributeMap(raw: string): Map<string, string> {
   return attrs;
 }
 
+function parsePositiveInteger(value: string | undefined): number | null {
+  if (value === undefined) {
+    return null;
+  }
+  const parsed = Number(value);
+  return Number.isSafeInteger(parsed) && parsed > 0 ? parsed : null;
+}
+
+function extractYoutubeTimedTextWindowDefinitions(xml: string): YoutubeTimedTextWindowDefinitions {
+  const rollingStyleIds = new Set<string>();
+  for (const match of xml.matchAll(/<ws\b([^>]*)\/?\s*>/g)) {
+    const attrs = parseAttributeMap(match[1] ?? '');
+    const id = attrs.get('id');
+    if (id !== undefined && attrs.get('mh') === '2') {
+      rollingStyleIds.add(id);
+    }
+  }
+
+  const positions = new Map<string, YoutubeRollingWindow>();
+  for (const match of xml.matchAll(/<wp\b([^>]*)\/?\s*>/g)) {
+    const attrs = parseAttributeMap(match[1] ?? '');
+    const id = attrs.get('id');
+    const rowCount = parsePositiveInteger(attrs.get('rc'));
+    const columnCount = parsePositiveInteger(attrs.get('cc'));
+    if (id !== undefined && rowCount !== null && columnCount !== null) {
+      positions.set(id, { rowCount, columnCount });
+    }
+  }
+
+  const windows = new Map<string, YoutubeRollingWindow>();
+  for (const match of xml.matchAll(/<w\b([^>]*)\/?\s*>/g)) {
+    const attrs = parseAttributeMap(match[1] ?? '');
+    const id = attrs.get('id');
+    const styleId = attrs.get('ws');
+    const positionId = attrs.get('wp');
+    const position = positionId === undefined ? undefined : positions.get(positionId);
+    if (
+      id !== undefined &&
+      styleId !== undefined &&
+      rollingStyleIds.has(styleId) &&
+      position !== undefined
+    ) {
+      windows.set(id, position);
+    }
+  }
+
+  return { rollingStyleIds, positions, windows };
+}
+
+function resolveRollingWindow(
+  attrs: Map<string, string>,
+  definitions: YoutubeTimedTextWindowDefinitions,
+): YoutubeRollingWindow | null {
+  const windowId = attrs.get('w');
+  if (windowId !== undefined) {
+    return definitions.windows.get(windowId) ?? null;
+  }
+
+  const styleId = attrs.get('ws');
+  const positionId = attrs.get('wp');
+  if (
+    styleId === undefined ||
+    positionId === undefined ||
+    !definitions.rollingStyleIds.has(styleId)
+  ) {
+    return null;
+  }
+  return definitions.positions.get(positionId) ?? null;
+}
+
 function extractYoutubeTimedTextDocument(xml: string): YoutubeTimedTextDocument {
   const rows: YoutubeTimedTextRow[] = [];
   const eventStartsMs: number[] = [];
   let hasRollingWindowEvents = false;
+  const windowDefinitions = extractYoutubeTimedTextWindowDefinitions(xml);
 
   for (const match of xml.matchAll(/<p\b([^>]*)>([\s\S]*?)<\/p>/g)) {
     const attrs = parseAttributeMap(match[1] ?? '');
@@ -76,7 +158,12 @@ function extractYoutubeTimedTextDocument(xml: string): YoutubeTimedTextDocument
       continue;
     }
 
-    rows.push({ startMs, durationMs, text });
+    rows.push({
+      startMs,
+      durationMs,
+      text,
+      rollingWindow: resolveRollingWindow(attrs, windowDefinitions),
+    });
   }
 
   eventStartsMs.sort((a, b) => a - b);
@@ -101,6 +188,79 @@ function formatVttTimestamp(ms: number): string {
   return `${String(hours).padStart(2, '0')}:${String(minutes).padStart(2, '0')}:${String(seconds).padStart(2, '0')}.${String(millis).padStart(3, '0')}`;
 }
 
+const ROLLING_PAGE_BREAK_PATTERN = /[\s、。！？!?]/u;
+
+// VTT cannot carry SRV3's row and column limits. Page only roll-up windows so
+// the overlay keeps their bounded presentation without changing authored cues.
+function splitRollingCaptionIntoPages(text: string, rollingWindow: YoutubeRollingWindow): string[] {
+  const pageCapacity = rollingWindow.rowCount * rollingWindow.columnCount;
+  const characters = [...text];
+  if (
+    !Number.isSafeInteger(pageCapacity) ||
+    pageCapacity <= 0 ||
+    characters.length <= pageCapacity
+  ) {
+    return [text];
+  }
+
+  const pages: string[] = [];
+  let pageStart = 0;
+  while (pageStart < characters.length) {
+    let pageEnd = Math.min(pageStart + pageCapacity, characters.length);
+    if (pageEnd < characters.length) {
+      const earliestNaturalBreak = pageStart + Math.ceil(pageCapacity * 0.6);
+      for (let index = pageEnd - 1; index >= earliestNaturalBreak; index -= 1) {
+        if (ROLLING_PAGE_BREAK_PATTERN.test(characters[index]!)) {
+          pageEnd = index + 1;
+          break;
+        }
+      }
+    }
+    pages.push(characters.slice(pageStart, pageEnd).join(''));
+    pageStart = pageEnd;
+  }
+  return pages;
+}
+
+interface TimedCaptionPage {
+  startMs: number;
+  endMs: number;
+  text: string;
+}
+
+function timeCaptionPages(input: {
+  text: string;
+  pages: string[];
+  startMs: number;
+  endMs: number;
+}): TimedCaptionPage[] {
+  const durationMs = input.endMs - input.startMs;
+  if (input.pages.length === 1 || durationMs < input.pages.length) {
+    return [{ startMs: input.startMs, endMs: input.endMs, text: input.text }];
+  }
+
+  const totalCharacters = [...input.text].length;
+  const timedPages: TimedCaptionPage[] = [];
+  let consumedCharacters = 0;
+  let pageStartMs = input.startMs;
+  // Automatic captions often omit span offsets, so distribute the known cue
+  // duration by page length while guaranteeing every page at least one ms.
+  for (let index = 0; index < input.pages.length; index += 1) {
+    const page = input.pages[index]!;
+    consumedCharacters += [...page].length;
+    const remainingPages = input.pages.length - index - 1;
+    const proportionalEndMs =
+      input.startMs + Math.round((durationMs * consumedCharacters) / totalCharacters);
+    const pageEndMs =
+      remainingPages === 0
+        ? input.endMs
+        : Math.min(Math.max(proportionalEndMs, pageStartMs + 1), input.endMs - remainingPages);
+    timedPages.push({ startMs: pageStartMs, endMs: pageEndMs, text: page });
+    pageStartMs = pageEndMs;
+  }
+  return timedPages;
+}
+
 export function isYoutubeTimedTextExtension(value: string | undefined): boolean {
   if (!value) {
     return false;
@@ -120,11 +280,12 @@ export function convertYoutubeTimedTextToVtt(xml: string): string {
     const row = rows[index]!;
     const nextRow = rows[index + 1];
     const unclampedEnd = row.startMs + row.durationMs;
-    // Rolling auto captions display until the next window event; the row's own
-    // duration is frequently a 3000ms placeholder that cuts long lines short.
-    const nextEventStart = hasRollingWindowEvents
-      ? findNextEventStartMs(eventStartsMs, row.startMs)
-      : undefined;
+    // YouTube uses exactly 3000ms as a placeholder for rolling speech rows.
+    // Other durations are explicit, including short sound cues such as [音楽].
+    const nextEventStart =
+      hasRollingWindowEvents && row.durationMs === YOUTUBE_ROLLING_PLACEHOLDER_DURATION_MS
+        ? findNextEventStartMs(eventStartsMs, row.startMs)
+        : undefined;
     const clampedEnd =
       nextEventStart !== undefined
         ? nextEventStart
@@ -143,9 +304,19 @@ export function convertYoutubeTimedTextToVtt(xml: string): string {
     if (!text) {
       continue;
     }
-    blocks.push(
-      `${formatVttTimestamp(row.startMs)} --> ${formatVttTimestamp(clampedEnd)}\n${text}`,
-    );
+    const pages = row.rollingWindow
+      ? splitRollingCaptionIntoPages(text, row.rollingWindow)
+      : [text];
+    for (const page of timeCaptionPages({
+      text,
+      pages,
+      startMs: row.startMs,
+      endMs: clampedEnd,
+    })) {
+      blocks.push(
+        `${formatVttTimestamp(page.startMs)} --> ${formatVttTimestamp(page.endMs)}\n${page.text}`,
+      );
+    }
   }
 
   return `WEBVTT\n\n${blocks.join('\n\n')}\n`;