feat(subtitles): generate Jellyfin subtitles and widen halfwidth kana

- Whisper subtitle generation now works for Jellyfin playback. It reads the item's original file through toJellyfinOriginalStreamUrl and writes to a reused cache folder. For transcoded playback it uses Jellyfin's audio index
- generateJapaneseSubtitles accepts http(s) streams when an output path is set; subprocess errors redact URL tokens
- Add widenHalfwidthKana/narrowFullwidthKana and widen subtitle text at ingest (mpv sub-text, parsed cues, live primary text), so halfwidth and fullwidth duplicates merge into one cue
- Stats queries widen older stored lines, sentence search matches katakana in either width, and stats mining widens sentence/word
- Update Jellyfin, subtitle generation, and architecture docs; add changelog fragments
This commit is contained in:
sudacode committed 2026-10-13 22:30:50 -07:00
commit f08d9d1d53
30 files changed
+580 -51

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
type: added
area: subtitles
- Japanese subtitle generation now works while playing a Jellyfin episode. SubMiner reads the original file from the server, saves the subtitles to a cache folder, and loads them into mpv. Direct-play and transcoded playback are both supported.
+5
View File
@@ -0,0 +1,5 @@
type: fixed
area: stats
- Subtitle lines recorded before halfwidth katakana widening now show fullwidth katakana in the stats dashboard (word/kanji occurrences, sentence search), and cards mined from stats history get the fullwidth sentence and word.
- Sentence search matches katakana in either width, so searching アンタ also finds older lines stored as アンタ.
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Halfwidth katakana in subtitles (common in broadcast-derived Japanese subs, e.g. アンタ, ダキニ) now displays as normal fullwidth katakana, so lines no longer mix squashed and normal-width text. Mined sentences and stats get the fullwidth form too.
+2
View File
@@ -23,6 +23,8 @@ SubMiner starts mpv if it is not already running. Pause, seek, stop, and track c
SubMiner selects a Japanese subtitle track automatically and resets mpv's subtitle delay to zero. It direct-plays files when it can and asks Jellyfin to transcode the rest. SubMiner selects a Japanese subtitle track automatically and resets mpv's subtitle delay to zero. It direct-plays files when it can and asks Jellyfin to transcode the rest.
If an episode has no Japanese subtitles, you can [generate them with Whisper](/subtitle-generation#generating-from-the-overlay) while it plays.
On Windows, casting finds mpv through `mpv.executablePath`, then `SUBMINER_MPV_PATH`, then `PATH`. An invalid `mpv.executablePath` stops mpv from starting. On Windows, casting finds mpv through `mpv.executablePath`, then `SUBMINER_MPV_PATH`, then `PATH`. An invalid `mpv.executablePath` stops mpv from starting.
## Playing from the terminal ## Playing from the terminal
+4 -2
View File
@@ -16,7 +16,7 @@ The modal's **Local tools** section lists anything missing. After you install a
## Generating from the overlay ## Generating from the overlay
1. Open a local video in mpv and select its Japanese audio track, or play a YouTube video. 1. Open a local video in mpv and select its Japanese audio track, or play a Jellyfin or YouTube video.
2. Press `Ctrl+Shift+G`. If the subtitle sidebar is empty, its **Generate Japanese subtitles** button opens the same modal. 2. Press `Ctrl+Shift+G`. If the subtitle sidebar is empty, its **Generate Japanese subtitles** button opens the same modal.
3. Pick a model and download it if needed. 3. Pick a model and download it if needed.
4. Optionally check **Focus on spoken dialogue** (see below). 4. Optionally check **Focus on spoken dialogue** (see below).
@@ -26,6 +26,8 @@ The modal shows progress. **Cancel** stops the job. Closing the modal lets the j
SubMiner saves `<video>.ja.generated.srt` next to the video and adds a number if that name is taken. If the same file is still playing, it loads the subtitles and resets the subtitle delay. SubMiner saves `<video>.ja.generated.srt` next to the video and adds a number if that name is taken. If the same file is still playing, it loads the subtitles and resets the subtitle delay.
For a Jellyfin video, SubMiner reads the original file from your server and saves the subtitles to `~/.cache/subminer/generated-subs/`, since a stream has no folder of its own. Each Jellyfin generation replaces the previous file there. Playback keeps going while it works. If Jellyfin is transcoding, SubMiner transcribes the audio track that playback started with.
For a YouTube video, SubMiner downloads its audio first and pauses the video until the subtitles load. The subtitles are temporary, and the audio is deleted when generation ends or you switch videos. See [YouTube integration](/youtube-integration#generate-subtitles-with-whisper). For a YouTube video, SubMiner downloads its audio first and pauses the video until the subtitles load. The subtitles are temporary, and the audio is deleted when generation ends or you switch videos. See [YouTube integration](/youtube-integration#generate-subtitles-with-whisper).
Change the shortcut with `shortcuts.openSubtitleGeneration`. Change the shortcut with `shortcuts.openSubtitleGeneration`.
@@ -46,7 +48,7 @@ subminer generate-subs episode.mkv --model-path /path/to/ggml-small.bin
| `--audio-stream <index>` | Pick an audio stream by its absolute FFmpeg index | | `--audio-stream <index>` | Pick an audio stream by its absolute FFmpeg index |
| `--output <path>` | Write to this SRT path. Existing files are never overwritten. | | `--output <path>` | Write to this SRT path. Existing files are never overwritten. |
With a file argument, SubMiner uses the audio stream tagged Japanese, or the first stream. `Ctrl+C` cancels. With a file argument, SubMiner uses the audio stream tagged Japanese, or the first stream. `Ctrl+C` cancels. The launcher only works with local files. Use the overlay for Jellyfin and YouTube.
## Choosing a model ## Choosing a model
+1 -1
View File
@@ -32,7 +32,7 @@ Update checks and startup launcher migration share a serialized update-state sto
- `src/main/boot/` owns boot-phase assembly seams so `src/main.ts` can stay focused on lifecycle coordination and startup-path selection. - `src/main/boot/` owns boot-phase assembly seams so `src/main.ts` can stay focused on lifecycle coordination and startup-path selection.
- `src/main/runtime/linux-overlay-mode-runtime.ts` owns Linux fullscreen mode state and window replacement. App cleanup cancels pending replacements; `main.ts` supplies window creation and subtitle refresh hooks. - `src/main/runtime/linux-overlay-mode-runtime.ts` owns Linux fullscreen mode state and window replacement. App cleanup cancels pending replacements; `main.ts` supplies window creation and subtitle refresh hooks.
- `src/core/services/` owns focused runtime services plus pure or side-effect-bounded logic. - `src/core/services/` owns focused runtime services plus pure or side-effect-bounded logic.
- `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. Optional dialogue mode retains both Silero-detected speech and other audible sections, omits confidently silent gaps, decodes passages independently, and restores original media timing. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same local media; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative. - `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. Optional dialogue mode retains both Silero-detected speech and other audible sections, omits confidently silent gaps, decodes passages independently, and restores original media timing. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same media. For Jellyfin playback it reads the item's original-file stream (`toJellyfinOriginalStreamUrl`) into one reused output folder, where each run best-effort deletes earlier files and writes a uniquely named SRT; transcoded playback uses Jellyfin's audio index and only external references, since mpv's track indexes describe the transcode. Subprocess errors redact URL tokens; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative.
- Subtitle model recommendations use bounded `nvidia-smi` and Whisper CUDA discovery probes in `subtitle-generation-acceleration.ts`. The overlay runtime caches results by executable path for 30 seconds and exposes acceleration status through the existing status IPC. Recommendations do not alter model selection or transcription arguments. - Subtitle model recommendations use bounded `nvidia-smi` and Whisper CUDA discovery probes in `subtitle-generation-acceleration.ts`. The overlay runtime caches results by executable path for 30 seconds and exposes acceleration status through the existing status IPC. Recommendations do not alter model selection or transcription arguments.
- `subtitle-generation-reference.ts` ranks mpv's loaded text subtitle tracks, excludes signs/songs and forced references, and extracts timing hints with FFmpeg. The overlay and launcher snapshot references only for matching media, including active subtitle delays. Hints guide long-passage cuts with or without VAD; they never limit audio coverage or replace Whisper timestamps. - `subtitle-generation-reference.ts` ranks mpv's loaded text subtitle tracks, excludes signs/songs and forced references, and extracts timing hints with FFmpeg. The overlay and launcher snapshot references only for matching media, including active subtitle delays. Hints guide long-passage cuts with or without VAD; they never limit audio coverage or replace Whisper timestamps.
- `src/main/runtime/subtitle-selection.ts` reads and validates mpv subtitle tracks and applies primary/secondary selections. Its opt-in session shortcut opens the shared overlay modal window, with renderer focus and subtitle suppression handled by the modal registry. - `src/main/runtime/subtitle-selection.ts` reads and validates mpv subtitle tracks and applies primary/secondary selections. Its opt-in session shortcut opens the shared overlay modal window, with renderer focus and subtitle suppression handled by the modal registry.
@@ -1935,6 +1935,27 @@ describe('stats server API routes', () => {
}); });
}); });
it('POST /api/stats/mine-card widens halfwidth katakana from older stats lines', async () => {
await withMiningEnv({}, async (env) => {
const app = createStatsApp(createMockTracker(), {
addYomitanNote: async () => 777,
ankiConnectConfig: ankiConfig(env, { fields: { sentence: 'Sentence' } }),
});
const { status, body } = await mineCard(app, 'word', {
sourcePath: env.sourcePath,
sentence: 'アンタがアータルたん?',
word: 'アンタ',
});
assert.equal(status, 200, JSON.stringify(body));
assert.equal(
sentNote(env.requests, 'updateNoteFields').fields.Sentence,
'<b>アンタ</b>がアータルたん?',
);
});
});
it('POST /api/stats/mine-card leaves word card selection text to Yomitan glossary', async () => { it('POST /api/stats/mine-card leaves word card selection text to Yomitan glossary', async () => {
await withMiningEnv({}, async (env) => { await withMiningEnv({}, async (env) => {
const app = createStatsApp(createMockTracker(), { const app = createStatsApp(createMockTracker(), {
@@ -3936,6 +3936,83 @@ test('getWordOccurrences maps a normalized word back to anime, video, and subtit
} }
}); });
test('getWordOccurrences widens halfwidth katakana in lines recorded before ingest widening', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
try {
ensureSchema(db);
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/kamikatsu-s01e08.mkv', {
canonicalTitle: 'Episode 8',
sourcePath: '/tmp/KamiKatsu S01E08.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
const { sessionId } = startSessionRecord(db, videoId, 1_000_000);
const wordResult = db
.prepare(
`INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3, first_seen, last_seen, frequency
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run('アンタ', 'アンタ', 'あんた', 'noun', '名詞', '代名詞', '', 1_000, 1_000, 1);
const lineResult = db
.prepare(
`INSERT INTO imm_subtitle_lines (
session_id, event_id, video_id, anime_id, line_index, segment_start_ms, segment_end_ms, text, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run(sessionId, null, videoId, null, 1, 0, 1000, 'アンタがアータルたん?', 1_000, 1_000);
db.prepare(
`INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count)
VALUES (?, ?, ?)`,
).run(Number(lineResult.lastInsertRowid), Number(wordResult.lastInsertRowid), 1);
const rows = getWordOccurrences(db, 'アンタ', 'アンタ', 'あんた', 10);
assert.deepEqual(
rows.map((row) => row.text),
['アンタがアータルたん?'],
);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('searchSubtitleSentences matches halfwidth and fullwidth katakana in either direction', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
try {
ensureSchema(db);
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/kamikatsu-s01e08.mkv', {
canonicalTitle: 'Episode 8',
sourcePath: '/tmp/KamiKatsu S01E08.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
const { sessionId } = startSessionRecord(db, videoId, 1_000_000);
const insertLine = db.prepare(
`INSERT INTO imm_subtitle_lines (
session_id, event_id, video_id, anime_id, line_index, segment_start_ms, segment_end_ms, text, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
);
insertLine.run(sessionId, null, videoId, null, 1, 0, 1000, 'アンタがアータルたん?', 1_000, 1_000);
insertLine.run(sessionId, null, videoId, null, 2, 1000, 2000, 'アンタ、ダキニ', 2_000, 2_000);
for (const query of ['アンタ', 'アンタ']) {
assert.deepEqual(
searchSubtitleSentences(db, query, 10).map((row) => row.text),
['アンタ、ダキニ', 'アンタがアータルたん?'],
);
}
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('searchSubtitleSentences searches known subtitle lines and returns media context', () => { test('searchSubtitleSentences searches known subtitle lines and returns media context', () => {
const dbPath = makeDbPath(); const dbPath = makeDbPath();
const db = openTestDb(dbPath); const db = openTestDb(dbPath);
@@ -18,6 +18,7 @@ import type {
} from './types'; } from './types';
import { fromDbTimestamp, toDbTimestamp } from './query-shared'; import { fromDbTimestamp, toDbTimestamp } from './query-shared';
import { nowMs } from './time'; import { nowMs } from './time';
import { narrowFullwidthKana, widenHalfwidthKana } from '../../utils/halfwidth-kana';
import { import {
areLexicalDailyRollupsReady, areLexicalDailyRollupsReady,
getLexicalDailyRollups, getLexicalDailyRollups,
@@ -382,6 +383,13 @@ export function getKanjiStats(db: DatabaseSync, limit = 100): KanjiStatsRow[] {
return stmt.all(limit) as KanjiStatsRow[]; return stmt.all(limit) as KanjiStatsRow[];
} }
// Lines recorded before subtitle text was widened at ingest can still hold halfwidth
// katakana; widen them on the way out so the dashboard and cards mined from it match
// newer rows. Display-only: word identity columns stay as stored since they are keys.
function widenLineText<T extends { text: string }>(row: T): T {
return { ...row, text: widenHalfwidthKana(row.text) };
}
export function getWordOccurrences( export function getWordOccurrences(
db: DatabaseSync, db: DatabaseSync,
headword: string, headword: string,
@@ -390,7 +398,7 @@ export function getWordOccurrences(
limit = 100, limit = 100,
offset = 0, offset = 0,
): WordOccurrenceRow[] { ): WordOccurrenceRow[] {
return db const rows = db
.prepare( .prepare(
` `
SELECT SELECT
@@ -418,6 +426,7 @@ export function getWordOccurrences(
`, `,
) )
.all(headword, word, reading, limit, offset) as unknown as WordOccurrenceRow[]; .all(headword, word, reading, limit, offset) as unknown as WordOccurrenceRow[];
return rows.map(widenLineText);
} }
export function getKanjiOccurrences( export function getKanjiOccurrences(
@@ -426,7 +435,7 @@ export function getKanjiOccurrences(
limit = 100, limit = 100,
offset = 0, offset = 0,
): KanjiOccurrenceRow[] { ): KanjiOccurrenceRow[] {
return db const rows = db
.prepare( .prepare(
` `
SELECT SELECT
@@ -454,6 +463,7 @@ export function getKanjiOccurrences(
`, `,
) )
.all(kanji, limit, offset) as unknown as KanjiOccurrenceRow[]; .all(kanji, limit, offset) as unknown as KanjiOccurrenceRow[];
return rows.map(widenLineText);
} }
export function searchSubtitleSentences( export function searchSubtitleSentences(
@@ -470,6 +480,14 @@ export function searchSubtitleSentences(
const params: string[] = []; const params: string[] = [];
for (const term of terms) { for (const term of terms) {
const likeTerm = `%${escapeLikeTerm(term)}%`; const likeTerm = `%${escapeLikeTerm(term)}%`;
// Lines recorded before ingest widening can hold halfwidth katakana, so subtitle
// text is matched against the term's widened and narrowed forms as well.
const widenedTerm = widenHalfwidthKana(term);
const textLikeTerms = uniqueNonEmptyTerms([
term,
widenedTerm,
narrowFullwidthKana(widenedTerm),
]).map((variant) => `%${escapeLikeTerm(variant)}%`);
const headwords = getHeadwordCandidatesForSentenceSearchTerm(term, options); const headwords = getHeadwordCandidatesForSentenceSearchTerm(term, options);
const headwordClause = const headwordClause =
headwords.length > 0 headwords.length > 0
@@ -485,16 +503,16 @@ export function searchSubtitleSentences(
: ''; : '';
clauses.push(` clauses.push(`
( (
l.text LIKE ? ESCAPE '\\' ${textLikeTerms.map(() => "l.text LIKE ? ESCAPE '\\'").join(' OR ')}
OR v.canonical_title LIKE ? ESCAPE '\\' OR v.canonical_title LIKE ? ESCAPE '\\'
OR COALESCE(a.canonical_title, '') LIKE ? ESCAPE '\\' OR COALESCE(a.canonical_title, '') LIKE ? ESCAPE '\\'
${headwordClause} ${headwordClause}
) )
`); `);
params.push(likeTerm, likeTerm, likeTerm, ...headwords); params.push(...textLikeTerms, likeTerm, likeTerm, ...headwords);
} }
return db const rows = db
.prepare( .prepare(
` `
SELECT SELECT
@@ -518,6 +536,7 @@ export function searchSubtitleSentences(
`, `,
) )
.all(...params, resolvedLimit) as unknown as SentenceSearchResultRow[]; .all(...params, resolvedLimit) as unknown as SentenceSearchResultRow[];
return rows.map(widenLineText);
} }
export function getSessionEvents( export function getSessionEvents(
+1
View File
@@ -130,6 +130,7 @@ export {
listSubtitleTracks as listJellyfinSubtitleTracksRuntime, listSubtitleTracks as listJellyfinSubtitleTracksRuntime,
resolvePlaybackPlan as resolveJellyfinPlaybackPlanRuntime, resolvePlaybackPlan as resolveJellyfinPlaybackPlanRuntime,
ticksToSeconds as jellyfinTicksToSecondsRuntime, ticksToSeconds as jellyfinTicksToSecondsRuntime,
toJellyfinOriginalStreamUrl,
} from './jellyfin'; } from './jellyfin';
export { JellyfinRemoteSessionService } from './jellyfin-remote'; export { JellyfinRemoteSessionService } from './jellyfin-remote';
export { export {
+24
View File
@@ -7,6 +7,7 @@ import {
listSubtitleTracks, listSubtitleTracks,
resolvePlaybackPlan, resolvePlaybackPlan,
ticksToSeconds, ticksToSeconds,
toJellyfinOriginalStreamUrl,
} from './jellyfin'; } from './jellyfin';
const clientInfo = { const clientInfo = {
@@ -862,3 +863,26 @@ test('resolvePlaybackPlan replaces a legacy api_key on the server transcoding ur
globalThis.fetch = originalFetch; globalThis.fetch = originalFetch;
} }
}); });
test('toJellyfinOriginalStreamUrl maps direct and transcoded playback to the original file', () => {
assert.equal(
toJellyfinOriginalStreamUrl(
'https://media.example/jf/videos/item-1/master.m3u8?ApiKey=token&MediaSourceId=src&VideoCodec=h264&StartTimeTicks=9',
'item-1',
),
'https://media.example/jf/Videos/item-1/stream?static=true&ApiKey=token&MediaSourceId=src',
);
assert.equal(
toJellyfinOriginalStreamUrl(
'http://jf.local:8096/Videos/item-1/stream?static=true&api_key=legacy&AudioStreamIndex=2',
'item-1',
),
'http://jf.local:8096/Videos/item-1/stream?static=true&ApiKey=legacy',
);
assert.equal(
toJellyfinOriginalStreamUrl('https://jf/Videos/other/stream?ApiKey=token', 'item-1'),
null,
);
assert.equal(toJellyfinOriginalStreamUrl('https://jf/Videos/item-1/stream', 'item-1'), null);
assert.equal(toJellyfinOriginalStreamUrl('/media/item-1.mkv', 'item-1'), null);
});
+23
View File
@@ -254,6 +254,29 @@ function createDirectPlayUrl(
return `${session.serverUrl}/Videos/${itemId}/stream?${query.toString()}`; return `${session.serverUrl}/Videos/${itemId}/stream?${query.toString()}`;
} }
/**
* Maps a playing stream URL of `itemId`, direct or transcoded, to its original file. Returns null
* when the URL is not that item's stream or carries no access token.
*/
export function toJellyfinOriginalStreamUrl(playbackUrl: string, itemId: string): string | null {
let url: URL;
try {
url = new URL(playbackUrl);
} catch {
return null;
}
const at = url.pathname.toLowerCase().indexOf(`/videos/${itemId.toLowerCase()}/`);
if (!/^https?:$/.test(url.protocol) || !itemId || at < 0) return null;
const param = (pattern: RegExp) =>
[...url.searchParams].find(([key, value]) => pattern.test(key) && value)?.[1];
const accessToken = param(/^api_?key$/i);
if (!accessToken) return null;
const query = new URLSearchParams({ static: 'true', ApiKey: accessToken });
const mediaSourceId = param(/^mediasourceid$/i);
if (mediaSourceId) query.set('MediaSourceId', mediaSourceId);
return `${url.origin}${url.pathname.slice(0, at)}/Videos/${itemId}/stream?${query.toString()}`;
}
function createTranscodeUrl( function createTranscodeUrl(
session: JellyfinAuthSession, session: JellyfinAuthSession,
itemId: string, itemId: string,
+12
View File
@@ -131,6 +131,18 @@ test('dispatchMpvProtocolMessage emits subtitle text on property change', async
assert.deepEqual(state.events, [{ text: '字幕', isOverlayVisible: false }]); assert.deepEqual(state.events, [{ text: '字幕', isOverlayVisible: false }]);
}); });
test('dispatchMpvProtocolMessage widens halfwidth katakana in subtitle text', async () => {
const { deps, state } = createDeps();
await dispatchMpvProtocolMessage(
{ event: 'property-change', name: 'sub-text', data: 'アンタがアータルたん?' },
deps,
);
assert.equal(state.subText, 'アンタがアータルたん?');
assert.deepEqual(state.events, [{ text: 'アンタがアータルたん?', isOverlayVisible: false }]);
});
test('dispatchMpvProtocolMessage emits subtitle track changes', async () => { test('dispatchMpvProtocolMessage emits subtitle track changes', async () => {
const { deps, state } = createDeps({ const { deps, state } = createDeps({
emitSubtitleTrackChange: (payload) => state.events.push(payload), emitSubtitleTrackChange: (payload) => state.events.push(payload),
+3 -2
View File
@@ -1,5 +1,6 @@
import { MpvSubtitleRenderMetrics } from '../../types'; import { MpvSubtitleRenderMetrics } from '../../types';
import { sanitizeMediaTitle } from '../../shared/media-identity'; import { sanitizeMediaTitle } from '../../shared/media-identity';
import { widenHalfwidthKana } from '../utils/halfwidth-kana';
export type MpvMessage = { export type MpvMessage = {
event?: string; event?: string;
@@ -244,7 +245,7 @@ export async function dispatchMpvProtocolMessage(
): Promise<void> { ): Promise<void> {
if (msg.event === 'property-change') { if (msg.event === 'property-change') {
if (msg.name === 'sub-text') { if (msg.name === 'sub-text') {
const nextSubText = (msg.data as string) || ''; const nextSubText = widenHalfwidthKana((msg.data as string) || '');
const overlayVisible = deps.isVisibleOverlayVisible(); const overlayVisible = deps.isVisibleOverlayVisible();
deps.emitSubtitleChange({ deps.emitSubtitleChange({
text: nextSubText, text: nextSubText,
@@ -458,7 +459,7 @@ export async function dispatchMpvProtocolMessage(
}>, }>,
); );
} else if (msg.request_id === MPV_REQUEST_ID_SUBTEXT) { } else if (msg.request_id === MPV_REQUEST_ID_SUBTEXT) {
const nextSubText = (msg.data as string) || ''; const nextSubText = widenHalfwidthKana((msg.data as string) || '');
deps.setCurrentSubText(nextSubText); deps.setCurrentSubText(nextSubText);
deps.emitSubtitleChange({ deps.emitSubtitleChange({
text: nextSubText, text: nextSubText,
@@ -8,6 +8,7 @@ import { clampMediaEndTime } from '../../../anki-integration/media-duration.js';
import { MediaGenerator } from '../../../media-generator.js'; import { MediaGenerator } from '../../../media-generator.js';
import { statsJson } from '../../../types/stats-http-contract.js'; import { statsJson } from '../../../types/stats-http-contract.js';
import { STATS_MINING_TAG } from '../../../shared/anki-source.js'; import { STATS_MINING_TAG } from '../../../shared/anki-source.js';
import { widenHalfwidthKana } from '../../utils/halfwidth-kana.js';
import { import {
resolveRetimedSecondarySubtitleTextFromSidecar, resolveRetimedSecondarySubtitleTextFromSidecar,
resolveSecondarySubtitleTextFromSidecar, resolveSecondarySubtitleTextFromSidecar,
@@ -36,8 +37,11 @@ export function registerStatsMiningRoutes(app: Hono, options?: StatsMiningRouteO
const sourcePath = typeof body?.sourcePath === 'string' ? body.sourcePath.trim() : ''; const sourcePath = typeof body?.sourcePath === 'string' ? body.sourcePath.trim() : '';
const startMs = typeof body?.startMs === 'number' ? body.startMs : NaN; const startMs = typeof body?.startMs === 'number' ? body.startMs : NaN;
const endMs = typeof body?.endMs === 'number' ? body.endMs : NaN; const endMs = typeof body?.endMs === 'number' ? body.endMs : NaN;
const sentence = typeof body?.sentence === 'string' ? body.sentence.trim() : ''; // Older stats rows can carry halfwidth katakana; cards get the fullwidth form that
const word = typeof body?.word === 'string' ? body.word.trim() : ''; // live mining produces, and the word is widened alongside so highlighting still hits.
const sentence =
typeof body?.sentence === 'string' ? widenHalfwidthKana(body.sentence.trim()) : '';
const word = typeof body?.word === 'string' ? widenHalfwidthKana(body.word.trim()) : '';
const bodySecondaryText = const bodySecondaryText =
typeof body?.secondaryText === 'string' ? body.secondaryText.trim() : ''; typeof body?.secondaryText === 'string' ? body.secondaryText.trim() : '';
const videoTitle = typeof body?.videoTitle === 'string' ? body.videoTitle.trim() : ''; const videoTitle = typeof body?.videoTitle === 'string' ? body.videoTitle.trim() : '';
@@ -55,6 +55,36 @@ test('parseSrtCues strips HTML-like markup while preserving line breaks', () =>
assert.equal(cues[0]!.text, 'これは\nテストです'); assert.equal(cues[0]!.text, 'これは\nテストです');
}); });
test('parseSubtitleCues widens halfwidth katakana in cue text', () => {
const cues = parseSubtitleCues(
'1\n00:00:01,000 --> 00:00:02,000\n(トーカ)ダキニお姉ちゃん\n',
'ep.srt',
);
assert.equal(cues[0]?.text, '(トーカ)ダキニお姉ちゃん');
});
test('parseSubtitleCues merges cues that differ only in halfwidth katakana', () => {
const cues = parseSubtitleCues(
[
'1',
'00:00:01,000 --> 00:00:02,000',
'山ガク',
'',
'2',
'00:00:01,000 --> 00:00:02,000',
'山ガク',
'',
].join('\n'),
'ep.srt',
);
assert.deepEqual(
cues.map((cue) => cue.text),
['山ガク'],
);
});
test('parseSrtCues handles hours in timestamps', () => { test('parseSrtCues handles hours in timestamps', () => {
const content = ['1', '01:30:00,000 --> 01:30:05,000', 'テスト', ''].join('\n'); const content = ['1', '01:30:00,000 --> 01:30:05,000', 'テスト', ''].join('\n');
+9 -2
View File
@@ -9,6 +9,7 @@ import {
type AssOverrideCommand, type AssOverrideCommand,
} from './ass-text'; } from './ass-text';
import { hasAssAnimationEvidence, mergeDuplicateCues } from './subtitle-cue-dedup'; import { hasAssAnimationEvidence, mergeDuplicateCues } from './subtitle-cue-dedup';
import { widenHalfwidthKana } from '../utils/halfwidth-kana';
/** Vertical third of the screen a cue is authored to occupy. */ /** Vertical third of the screen a cue is authored to occupy. */
export type AssVerticalBand = 'top' | 'middle' | 'bottom'; export type AssVerticalBand = 'top' | 'middle' | 'bottom';
@@ -104,7 +105,7 @@ function decodeSubtitleCueText(text: string): string {
} }
function sanitizeSubtitleCueText(text: string): string { function sanitizeSubtitleCueText(text: string): string {
return decodeSubtitleCueText(text).trim(); return widenHalfwidthKana(decodeSubtitleCueText(text)).trim();
} }
function sanitizeAssCueText(text: string): string { function sanitizeAssCueText(text: string): string {
@@ -2655,10 +2656,16 @@ function parseAnnotatedAssCues(content: string): AnnotatedSubtitleCue[] {
removeAssFontTextureEvents(parseAnnotatedAssEvents(content, placement)), removeAssFontTextureEvents(parseAnnotatedAssEvents(content, placement)),
placement.playResY, placement.playResY,
); );
return mergeAssCaptionRows( const cues = mergeAssCaptionRows(
recoverFragmentOnlyAssLines(recoverCanonicalAssEvents(events)), recoverFragmentOnlyAssLines(recoverCanonicalAssEvents(events)),
placement.playResY, placement.playResY,
); );
// Widened after reconstruction, which works glyph by glyph, but before duplicate
// merging, so ガク and ガク authored for the same span collapse into one cue.
for (const cue of cues) {
cue.text = widenHalfwidthKana(cue.text);
}
return cues;
} }
export function parseAssCues(content: string): SubtitleCue[] { export function parseAssCues(content: string): SubtitleCue[] {
@@ -3,6 +3,11 @@ import { expandSubtitleGenerationPath } from './subtitle-generation-files';
const OUTPUT_LIMIT = 64 * 1024; const OUTPUT_LIMIT = 64 * 1024;
// ffmpeg echoes input URLs in its errors; streamed media carries access tokens in the query.
function redactUrlCredentials(text: string): string {
return text.replace(/([?&](?:api_?key|token|access_token)=)[^&#\s'"]+/gi, '$1<redacted>');
}
// Keep partial lines between chunks: ffmpeg and whisper both report progress on stderr. // Keep partial lines between chunks: ffmpeg and whisper both report progress on stderr.
export function runSubtitleGenerationProcess(input: { export function runSubtitleGenerationProcess(input: {
command: string; command: string;
@@ -63,7 +68,11 @@ export function runSubtitleGenerationProcess(input: {
cleanup(); cleanup();
if (input.signal?.aborted) reject(new Error('Subtitle generation cancelled.')); if (input.signal?.aborted) reject(new Error('Subtitle generation cancelled.'));
else if (code !== 0) { else if (code !== 0) {
reject(new Error(`${input.command} exited with status ${code}: ${stderr.trim()}`)); reject(
new Error(
`${input.command} exited with status ${code}: ${redactUrlCredentials(stderr.trim())}`,
),
);
} else resolve(stdout); } else resolve(stdout);
}); });
}); });
+27 -1
View File
@@ -378,7 +378,22 @@ test('empty executable paths find tools on PATH and explicit overrides take prec
} }
})); }));
test('generation rejects remote media, missing models, and missing tools before starting a subprocess', () => test('generation reads an HTTP stream as-is and saves to the requested output', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
const mediaPath = 'https://jellyfin.example/Videos/item/stream?static=true&ApiKey=secret';
const outputPath = path.join(directory, 'out', 'stream.ja.srt');
await mkdir(path.dirname(outputPath));
assert.equal(await generateJapaneseSubtitles({ ...input, mediaPath, outputPath }), outputPath);
const calls: string[][] = (await readFile(input.callsPath, 'utf8'))
.trim()
.split('\n')
.map((line) => JSON.parse(line));
assert.equal(calls[0]?.at(-1), mediaPath);
assert.equal(calls[1]?.[calls[1].indexOf('-i') + 1], mediaPath);
}));
test('generation rejects remote media without an output path, missing models, and missing tools before starting a subprocess', () =>
fixture(async (directory) => { fixture(async (directory) => {
const input = await generationFixture(directory); const input = await generationFixture(directory);
await assert.rejects( await assert.rejects(
@@ -473,6 +488,17 @@ test('process cancellation terminates work and bounds diagnostic output', () =>
error.message.length < 66000 && error.message.length < 66000 &&
error.message.includes('status 7'), error.message.includes('status 7'),
); );
const unauthorized = await executable(
directory,
'unauthorized',
"process.stderr.write('https://host/Videos/1/stream?ApiKey=se+cr/et==&static=true: Server returned 401');process.exitCode=1;",
);
await assert.rejects(
runSubtitleGenerationProcess({ command: unauthorized, args: [] }),
(error: unknown) =>
error instanceof Error &&
error.message.includes('?ApiKey=<redacted>&static=true: Server returned 401'),
);
})); }));
test('download uses the pinned model revision and removes files that fail integrity', () => test('download uses the pinned model revision and removes files that fail integrity', () =>
+6 -3
View File
@@ -191,6 +191,7 @@ export async function requireSubtitleGenerationSetup(
export async function generateJapaneseSubtitles(input: { export async function generateJapaneseSubtitles(input: {
config: SubtitleGenerationConfig; config: SubtitleGenerationConfig;
modelDirectory: string; modelDirectory: string;
/** A local file, or an http(s) stream when `outputPath` is set. */
mediaPath: string; mediaPath: string;
audioStreamIndex?: number; audioStreamIndex?: number;
references?: readonly SubtitleGenerationReference[]; references?: readonly SubtitleGenerationReference[];
@@ -201,10 +202,12 @@ export async function generateJapaneseSubtitles(input: {
signal?: AbortSignal; signal?: AbortSignal;
}): Promise<string> { }): Promise<string> {
input.signal?.throwIfAborted(); input.signal?.throwIfAborted();
if (/^[a-z][a-z\d+.-]*:\/\//i.test(input.mediaPath)) // HTTP streams are read by ffmpeg directly but have no directory to save beside.
const remote = /^https?:\/\//i.test(input.mediaPath);
if (remote ? !input.outputPath : /^[a-z][a-z\d+.-]*:\/\//i.test(input.mediaPath))
throw new Error('Subtitle generation requires a local media file.'); throw new Error('Subtitle generation requires a local media file.');
const mediaPath = path.resolve(input.mediaPath); const mediaPath = remote ? input.mediaPath : path.resolve(input.mediaPath);
if (!(await stat(mediaPath)).isFile()) if (!remote && !(await stat(mediaPath)).isFile())
throw new Error('Subtitle generation requires a local media file.'); throw new Error('Subtitle generation requires a local media file.');
if (input.outputPath) await ensureAvailableOutput(path.resolve(input.outputPath)); if (input.outputPath) await ensureAvailableOutput(path.resolve(input.outputPath));
await ensureWritableDirectory( await ensureWritableDirectory(
@@ -1,23 +1,12 @@
import { MergedToken, PartOfSpeech } from '../../../types'; import { MergedToken, PartOfSpeech } from '../../../types';
import { widenHalfwidthKana } from '../../utils/halfwidth-kana';
const KATAKANA_TO_HIRAGANA_OFFSET = 0x60; const KATAKANA_TO_HIRAGANA_OFFSET = 0x60;
const KATAKANA_CODEPOINT_START = 0x30a1; const KATAKANA_CODEPOINT_START = 0x30a1;
const KATAKANA_CODEPOINT_END = 0x30f6; const KATAKANA_CODEPOINT_END = 0x30f6;
// No `u` flag: the range is entirely BMP so it changes nothing here, and
// Bun's unicode-mode matcher mis-handles this class next to certain ligatures.
const HALFWIDTH_KANA_RUN = /[\uff66-\uff9f]+/g;
// NFKC over the halfwidth kana only, never the whole string: it composes the
// voiced pairs (カ + ゙) into single characters so ガク compares equal to ガク
// instead of counting one character longer than the word it spells, but run
// over everything it would also rewrite unrelated text (① → 1, ㍑ → リットル).
function composeHalfwidthKana(text: string): string {
return text.replace(HALFWIDTH_KANA_RUN, (run) => run.normalize('NFKC'));
}
export function normalizeKana(text: string): string { export function normalizeKana(text: string): string {
const raw = composeHalfwidthKana(text).trim(); const raw = widenHalfwidthKana(text).trim();
if (!raw) { if (!raw) {
return ''; return '';
} }
+24
View File
@@ -0,0 +1,24 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { narrowFullwidthKana, widenHalfwidthKana } from './halfwidth-kana';
test('widenHalfwidthKana widens halfwidth katakana and composes voiced marks', () => {
assert.equal(
widenHalfwidthKana('(トーカ)ダキニお姉ちゃん アンタがアータルたん。'),
'(トーカ)ダキニお姉ちゃん アンタがアータルたん。',
);
});
test('widenHalfwidthKana leaves other compatibility characters alone', () => {
assert.equal(widenHalfwidthKana('見て見てっ!①㍑AB'), '見て見てっ!①㍑AB');
});
test('widenHalfwidthKana folds a lone halfwidth voiced mark into the kana before it', () => {
assert.equal(widenHalfwidthKana('山ガク パン'), '山ガク パン');
assert.equal(widenHalfwidthKana('ア゙'), 'ア\u3099');
});
test('narrowFullwidthKana inverts widening and leaves other text alone', () => {
assert.equal(narrowFullwidthKana('ダキニお姉ちゃん アンタ。ヴ'), 'ダキニお姉ちゃん アンタ。ヴ');
assert.equal(narrowFullwidthKana('漢字ひらがな!'), '漢字ひらがな!');
});
+49
View File
@@ -0,0 +1,49 @@
// No `u` flag: the range is entirely BMP so it changes nothing here, and
// Bun's unicode-mode matcher mis-handles this class next to certain ligatures.
// The optional leading character is captured so a run opening with a voiced
// mark (ガ) can fold the mark into the fullwidth kana before it.
const HALFWIDTH_KANA_RUN = /([^。-゚]?)([。-゚]+)/g;
const LEADING_COMBINING_VOICED_MARK = /^[゙゚]/;
/**
* Rewrites halfwidth katakana and its punctuation (アンタ, ダキニ, 。「」、・) to the fullwidth
* forms. Broadcast-derived subtitles write katakana halfwidth to save space; subtitle
* text is normalized with this where it enters the app (live mpv text, parsed cues) so
* display, lookups, mining, and stats all see one form.
*
* NFKC runs over the halfwidth runs only, never the whole string: it composes the voiced
* pairs (カ + ゙) into single characters, but over everything it would also rewrite
* unrelated text (① → 1, ! → !, ㍑ → リットル).
*/
export function widenHalfwidthKana(text: string): string {
return text.replace(HALFWIDTH_KANA_RUN, (_match, previous: string, run: string) => {
const widened = run.normalize('NFKC');
if (!previous || !LEADING_COMBINING_VOICED_MARK.test(widened)) {
return previous + widened;
}
// A lone ゙ widens to a combining mark; NFC composes it with the kana before it
// (カ + ゛ → ガ) and leaves pairs that have no composed form untouched.
return (previous + widened[0]).normalize('NFC') + widened.slice(1);
});
}
// Inverse of widenHalfwidthKana per character, derived from it so the two stay in sync.
// Voiced pairs (ガ) are included because they widen to a single character (ガ).
const HALFWIDTH_BY_FULLWIDTH = new Map<string, string>();
for (let code = 0xff61; code <= 0xff9f; code += 1) {
const halfwidth = String.fromCharCode(code);
for (const form of [halfwidth, `${halfwidth}゙`, `${halfwidth}゚`]) {
const widened = widenHalfwidthKana(form);
if (widened.length === 1 && !HALFWIDTH_BY_FULLWIDTH.has(widened)) {
HALFWIDTH_BY_FULLWIDTH.set(widened, form);
}
}
}
/**
* Rewrites fullwidth katakana and punctuation to halfwidth forms (アンタ → アンタ). Used to
* search text stored before subtitle ingest widened it; display paths should widen instead.
*/
export function narrowFullwidthKana(text: string): string {
return Array.from(text, (char) => HALFWIDTH_BY_FULLWIDTH.get(char) ?? char).join('');
}
+12
View File
@@ -319,6 +319,7 @@ import {
replayCurrentSubtitleRuntime, replayCurrentSubtitleRuntime,
resolveSanitizedSubtitleSeekCommand, resolveSanitizedSubtitleSeekCommand,
resolveJellyfinPlaybackPlanRuntime, resolveJellyfinPlaybackPlanRuntime,
toJellyfinOriginalStreamUrl,
runStartupBootstrapRuntime, runStartupBootstrapRuntime,
saveSubtitlePosition as saveSubtitlePositionCore, saveSubtitlePosition as saveSubtitlePositionCore,
clearYomitanParserCachesForWindow, clearYomitanParserCachesForWindow,
@@ -6924,6 +6925,17 @@ const subtitleGenerationRuntime = createSubtitleGenerationRuntime({
}), }),
resolveYoutubeSourceUrl: (mediaPath) => resolveYoutubeSourceUrl: (mediaPath) =>
youtubePlaybackRuntime.getYoutubeSourceUrlForStream(mediaPath), youtubePlaybackRuntime.getYoutubeSourceUrlForStream(mediaPath),
resolveJellyfinMedia: (mediaPath) => {
const playback = activeJellyfinRemotePlayback;
const sourceUrl = playback && toJellyfinOriginalStreamUrl(mediaPath, playback.itemId);
return sourceUrl
? {
sourceUrl,
transcoded: playback.playMethod === 'Transcode',
audioStreamIndex: playback.audioStreamIndex ?? null,
}
: null;
},
getMpvClient: () => appState.mpvClient, getMpvClient: () => appState.mpvClient,
onProgress: (progress) => { onProgress: (progress) => {
for (const window of [overlayManager.getMainWindow(), overlayManager.getModalWindow()]) { for (const window of [overlayManager.getMainWindow(), overlayManager.getModalWindow()]) {
+5 -1
View File
@@ -3,6 +3,7 @@ import {
removeAssControlDebrisLines, removeAssControlDebrisLines,
removeLiveGlyphFragmentLines, removeLiveGlyphFragmentLines,
} from '../../core/services/ass-text'; } from '../../core/services/ass-text';
import { widenHalfwidthKana } from '../../core/utils/halfwidth-kana';
// Slack on top of each cue's recorded animation envelope, for time-pos observation // Slack on top of each cue's recorded animation envelope, for time-pos observation
// staleness and small user sub-delay offsets. The envelope itself covers how far // staleness and small user sub-delay offsets. The envelope itself covers how far
@@ -26,11 +27,14 @@ function cuesUseAssSyntax(cues: readonly SubtitleCue[] | null | undefined): bool
); );
} }
// Live text also arrives from direct `sub-text` reads that bypass the mpv protocol
// handler, so it is widened here too to match the parsed cues.
function decodedLiveText( function decodedLiveText(
liveText: string, liveText: string,
cues: readonly SubtitleCue[] | null | undefined, cues: readonly SubtitleCue[] | null | undefined,
): string { ): string {
return cuesUseAssSyntax(cues) ? removeAssControlDebrisLines(liveText) : liveText; const text = widenHalfwidthKana(liveText);
return cuesUseAssSyntax(cues) ? removeAssControlDebrisLines(text) : text;
} }
function animationSpan(cue: SubtitleCue): { start: number; end: number } { function animationSpan(cue: SubtitleCue): { start: number; end: number } {
@@ -1,8 +1,10 @@
import assert from 'node:assert/strict'; import assert from 'node:assert/strict';
import path from 'node:path';
import test from 'node:test'; import test from 'node:test';
import { DEFAULT_SUBTITLE_GENERATION_CONFIG } from '../../shared/subtitle-generation'; import { DEFAULT_SUBTITLE_GENERATION_CONFIG } from '../../shared/subtitle-generation';
import { import {
createSubtitleGenerationRuntime, createSubtitleGenerationRuntime,
type SubtitleGenerationJellyfinMedia,
type SubtitleGenerationRuntimeDeps, type SubtitleGenerationRuntimeDeps,
} from './subtitle-generation-runtime'; } from './subtitle-generation-runtime';
@@ -506,3 +508,101 @@ test('switching to another resolved YouTube stream cancels the modal job', async
assert.deepEqual(await job, { ok: false, message: 'Cancelled.' }); assert.deepEqual(await job, { ok: false, message: 'Cancelled.' });
}); });
function jellyfinFixture(
jellyfin: Omit<SubtitleGenerationJellyfinMedia, 'sourceUrl'>,
overrides: Partial<SubtitleGenerationRuntimeDeps> = {},
) {
const playing = 'https://jf.example/Videos/item/master.m3u8?ApiKey=secret';
const subject = fixture({
resolveJellyfinMedia: (mediaPath) =>
mediaPath === playing
? {
sourceUrl: 'https://jf.example/Videos/item/stream?static=true&ApiKey=secret',
...jellyfin,
}
: null,
createStreamOutputDirectory: async () => '/tmp/generated',
...overrides,
});
subject.client.requestProperty = async (name) =>
name === 'path'
? playing
: name === 'media-title'
? 'Show S01E02'
: name === 'track-list'
? [
{ type: 'audio', selected: true, 'ff-index': 0 },
{ type: 'sub', lang: 'eng', title: 'English Full', 'ff-index': 4 },
{
type: 'sub',
lang: 'eng',
title: 'English',
external: true,
'external-filename': '/tmp/jellyfin-subs/track-5.srt',
},
]
: null;
return subject;
}
test('a direct-play Jellyfin stream generates from its original file and shows by title', async () => {
let outputPath = '';
const subject = jellyfinFixture(
{ transcoded: false, audioStreamIndex: 2 },
{
generate: async (input) => {
assert.equal(
input.mediaPath,
'https://jf.example/Videos/item/stream?static=true&ApiKey=secret',
);
outputPath = input.outputPath ?? '';
assert.equal(path.dirname(outputPath), path.join('/tmp', 'generated'));
assert.match(path.basename(outputPath), /^jellyfin-\d+\.ja\.generated\.srt$/);
assert.equal(input.audioStreamIndex, 0);
assert.deepEqual(
input.references?.map((reference) => reference.source.kind),
['embedded', 'external'],
);
return outputPath;
},
},
);
assert.equal((await subject.runtime.getStatus()).mediaPath, 'Jellyfin: Show S01E02');
assert.equal((await subject.runtime.start()).ok, true);
assert.deepEqual(subject.commands[0], [
'sub-add',
outputPath,
'select',
'Generated Japanese',
'ja',
]);
});
test('a transcoded Jellyfin stream uses the Jellyfin audio index and only external references', async () => {
const subject = jellyfinFixture(
{ transcoded: true, audioStreamIndex: 2 },
{
generate: async (input) => {
assert.equal(input.audioStreamIndex, 2);
assert.deepEqual(
input.references?.map((reference) => reference.source.kind),
['external'],
);
return '/tmp/generated/jellyfin.ja.generated.srt';
},
},
);
assert.equal((await subject.runtime.start()).ok, true);
});
test('unrecognized streams cannot generate', async () => {
const subject = jellyfinFixture(
{ transcoded: false, audioStreamIndex: null },
{ resolveJellyfinMedia: () => null },
);
assert.equal((await subject.runtime.getStatus()).mediaPath, null);
const result = await subject.runtime.start();
assert.equal(result.ok, false);
assert.match(result.message, /Jellyfin stream/);
});
+78 -15
View File
@@ -1,3 +1,5 @@
import { mkdir, readdir, rm } from 'node:fs/promises';
import { homedir } from 'node:os';
import path from 'node:path'; import path from 'node:path';
import { readSubtitleGenerationReferences } from '../../core/services/subtitle-generation-reference'; import { readSubtitleGenerationReferences } from '../../core/services/subtitle-generation-reference';
import { detectSubtitleGenerationAcceleration } from '../../core/services/subtitle-generation-acceleration'; import { detectSubtitleGenerationAcceleration } from '../../core/services/subtitle-generation-acceleration';
@@ -29,6 +31,22 @@ interface GenerationMpvClient {
request: (command: unknown[]) => Promise<{ error?: string }>; request: (command: unknown[]) => Promise<{ error?: string }>;
} }
/** A Jellyfin stream playing in mpv, as subtitle generation reads it. */
export interface SubtitleGenerationJellyfinMedia {
/** The item's original file. It carries the access token, so it never reaches status text. */
sourceUrl: string;
/**
* Transcoded playback has its own stream layout, so mpv's track indexes do not address the
* original file. Generation then uses Jellyfin's audio index and skips embedded references.
*/
transcoded: boolean;
audioStreamIndex: number | null;
}
type GenerationMedia =
| { kind: 'local'; path: string }
| { kind: 'jellyfin'; path: string; jellyfin: SubtitleGenerationJellyfinMedia };
export interface SubtitleGenerationRuntimeDeps { export interface SubtitleGenerationRuntimeDeps {
getConfig: () => SubtitleGenerationConfig; getConfig: () => SubtitleGenerationConfig;
getModelDirectory: () => string; getModelDirectory: () => string;
@@ -56,11 +74,27 @@ export interface SubtitleGenerationRuntimeDeps {
* resolved direct stream in mpv, so mpv's path alone does not identify the video. * resolved direct stream in mpv, so mpv's path alone does not identify the video.
*/ */
resolveYoutubeSourceUrl?: (mediaPath: string) => string | null; resolveYoutubeSourceUrl?: (mediaPath: string) => string | null;
/** Identifies mpv's path as Jellyfin playback. Without it, streams cannot generate. */
resolveJellyfinMedia?: (mediaPath: string) => SubtitleGenerationJellyfinMedia | null;
/**
* Directory for subtitles of streamed media, which has no folder of its own. Called once per
* stream generation, so it may delete what an earlier run wrote there.
*/
createStreamOutputDirectory?: () => Promise<string>;
} }
async function currentLocalMedia(client: GenerationMpvClient | null): Promise<string | null> { // One reused folder keeps stream subtitles from piling up: each run deletes earlier files.
if (!client?.connected) return null; // mpv has already parsed a loaded track, and a file still locked (Windows) waits for a later run.
const media = await client.requestProperty('path'); async function reuseStreamOutputDirectory(): Promise<string> {
const directory = path.join(homedir(), '.cache', 'subminer', 'generated-subs');
await mkdir(directory, { recursive: true });
for (const name of await readdir(directory))
await rm(path.join(directory, name), { recursive: true, force: true }).catch(() => undefined);
return directory;
}
// Resolves mpv's `path` value to an absolute local file, or null for URLs.
async function localMediaPath(client: GenerationMpvClient, media: unknown): Promise<string | null> {
if (typeof media !== 'string' || !media || /^[a-z][a-z\d+.-]*:\/\//i.test(media)) return null; if (typeof media !== 'string' || !media || /^[a-z][a-z\d+.-]*:\/\//i.test(media)) return null;
if (path.isAbsolute(media)) return path.normalize(media); if (path.isAbsolute(media)) return path.normalize(media);
const directory = await client.requestProperty('working-directory'); const directory = await client.requestProperty('working-directory');
@@ -139,6 +173,22 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
function youtubeSource(mediaPath: string | null | undefined): string | null | undefined { function youtubeSource(mediaPath: string | null | undefined): string | null | undefined {
return (mediaPath && deps.resolveYoutubeSourceUrl?.(mediaPath)) || mediaPath; return (mediaPath && deps.resolveYoutubeSourceUrl?.(mediaPath)) || mediaPath;
} }
async function currentMedia(client: GenerationMpvClient | null): Promise<GenerationMedia | null> {
if (!client?.connected) return null;
const media = await client.requestProperty('path');
if (typeof media === 'string' && /^https?:\/\//i.test(media)) {
const jellyfin = deps.resolveJellyfinMedia?.(media);
return jellyfin ? { kind: 'jellyfin', path: media, jellyfin } : null;
}
const local = await localMediaPath(client, media);
return local ? { kind: 'local', path: local } : null;
}
// Stream URLs carry access tokens, so Jellyfin media shows by title.
async function describeMedia(client: GenerationMpvClient, media: GenerationMedia) {
if (media.kind === 'local') return media.path;
const title = await client.requestProperty('media-title').catch(() => null);
return `Jellyfin: ${typeof title === 'string' && title.trim() ? title.trim() : 'current item'}`;
}
async function currentYoutubeMedia(client: GenerationMpvClient | null): Promise<string | null> { async function currentYoutubeMedia(client: GenerationMpvClient | null): Promise<string | null> {
if (!client?.connected) return null; if (!client?.connected) return null;
const media = await client.requestProperty('path'); const media = await client.requestProperty('path');
@@ -196,9 +246,10 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
deps.getModelDirectory(), deps.getModelDirectory(),
); );
const client = deps.getMpvClient(); const client = deps.getMpvClient();
const media = client ? await currentMedia(client).catch(() => null) : null;
const mediaPath = const mediaPath =
youtubeJobUrl ?? youtubeJobUrl ??
(await currentLocalMedia(client).catch(() => null)) ?? (client && media ? await describeMedia(client, media) : null) ??
(deps.generateYoutube ? await currentYoutubeMedia(client).catch(() => null) : null); (deps.generateYoutube ? await currentYoutubeMedia(client).catch(() => null) : null);
return { return {
model, model,
@@ -350,21 +401,33 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
const youtubeUrl = deps.generateYoutube ? await currentYoutubeMedia(client) : null; const youtubeUrl = deps.generateYoutube ? await currentYoutubeMedia(client) : null;
if (client && youtubeUrl && deps.generateYoutube) if (client && youtubeUrl && deps.generateYoutube)
return await startYoutube(client, youtubeUrl, config, signal, deps.generateYoutube); return await startYoutube(client, youtubeUrl, config, signal, deps.generateYoutube);
const mediaPath = await currentLocalMedia(client); const media = await currentMedia(client);
if (!client || !mediaPath) if (!client || !media)
throw new Error('Open a local video or audio file, or a YouTube video, in mpv first.'); throw new Error(
const tracks = await client.requestProperty('track-list'); 'Open a local video or audio file, a Jellyfin stream, or a YouTube video in mpv first.',
const audioStreamIndex = selectedAudioIndex(tracks);
const references = await readSubtitleGenerationReferences(tracks, (name) =>
client.requestProperty(name),
); );
if ((await currentLocalMedia(client)) !== mediaPath) const tracks = await client.requestProperty('track-list');
const transcoded = media.kind === 'jellyfin' && media.jellyfin.transcoded;
const audioStreamIndex = transcoded
? (media.jellyfin.audioStreamIndex ?? undefined)
: selectedAudioIndex(tracks);
const references = (
await readSubtitleGenerationReferences(tracks, (name) => client.requestProperty(name))
).filter((reference) => !transcoded || reference.source.kind === 'external');
if ((await currentMedia(client))?.path !== media.path)
throw new Error('The current media changed. Start generation again.'); throw new Error('The current media changed. Start generation again.');
signal.throwIfAborted(); signal.throwIfAborted();
const outputPath = await (deps.generate ?? generateJapaneseSubtitles)({ const outputPath = await (deps.generate ?? generateJapaneseSubtitles)({
config, config,
modelDirectory: deps.getModelDirectory(), modelDirectory: deps.getModelDirectory(),
mediaPath, mediaPath: media.kind === 'jellyfin' ? media.jellyfin.sourceUrl : media.path,
outputPath:
media.kind === 'jellyfin'
? path.join(
await (deps.createStreamOutputDirectory ?? reuseStreamOutputDirectory)(),
`jellyfin-${Date.now()}.ja.generated.srt`,
)
: undefined,
audioStreamIndex, audioStreamIndex,
references, references,
onProgress: report, onProgress: report,
@@ -372,8 +435,8 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
}); });
// Saving succeeds even if playback changes or disconnects during the job. // Saving succeeds even if playback changes or disconnects during the job.
try { try {
const playingMedia = await currentLocalMedia(client); const playingMedia = (await currentMedia(client))?.path;
if (!signal.aborted && deps.getMpvClient() === client && playingMedia === mediaPath) { if (!signal.aborted && deps.getMpvClient() === client && playingMedia === media.path) {
await loadGeneratedSubtitles(client, outputPath); await loadGeneratedSubtitles(client, outputPath);
return { return {
ok: true, ok: true,
+12
View File
@@ -256,6 +256,18 @@ test('youtube flow announces manual picker opening before probing tracks', async
assertNoProblems(harness); assertNoProblems(harness);
}); });
test('youtube flow widens halfwidth katakana in the refreshed subtitle', async () => {
const harness = createFlowHarness({
mpv: { subText: 'アンタがアータルたん?' },
probeTracks: [primaryTrack],
pick: { primaryTrackId: primaryTrack.id, secondaryTrackId: null },
});
await harness.runtime.openManualPicker({ url: 'https://example.com' });
assert.deepEqual(harness.refreshedSubtitles, ['アンタがアータルたん?']);
});
test('youtube flow can open a manual picker session and load the selected subtitles', async () => { test('youtube flow can open a manual picker session and load the selected subtitles', async () => {
const harness = createFlowHarness({ const harness = createFlowHarness({
probeTracks: [primaryTrack, secondaryTrack], probeTracks: [primaryTrack, secondaryTrack],
+2 -1
View File
@@ -21,6 +21,7 @@ import {
} from '../../core/services/youtube/generate'; } from '../../core/services/youtube/generate';
import { resolveSubtitleSourcePath } from './subtitle-prefetch-source'; import { resolveSubtitleSourcePath } from './subtitle-prefetch-source';
import { extractYoutubeVideoId } from './youtube-playback'; import { extractYoutubeVideoId } from './youtube-playback';
import { widenHalfwidthKana } from '../../core/utils/halfwidth-kana';
type YoutubeFlowOpenPicker = (payload: YoutubePickerOpenPayload) => Promise<boolean>; type YoutubeFlowOpenPicker = (payload: YoutubePickerOpenPayload) => Promise<boolean>;
@@ -457,7 +458,7 @@ async function injectDownloadedSubtitles(
const currentSubText = await deps.requestMpvProperty('sub-text').catch(() => null); const currentSubText = await deps.requestMpvProperty('sub-text').catch(() => null);
if (typeof currentSubText === 'string' && currentSubText.trim().length > 0) { if (typeof currentSubText === 'string' && currentSubText.trim().length > 0) {
deps.refreshCurrentSubtitle(currentSubText); deps.refreshCurrentSubtitle(widenHalfwidthKana(currentSubText));
} }
deps.showMpvOsd(secondaryTrack ? 'Primary and secondary subtitles loaded.' : 'Subtitles loaded.'); deps.showMpvOsd(secondaryTrack ? 'Primary and secondary subtitles loaded.' : 'Subtitles loaded.');
+3 -2
View File
@@ -88,7 +88,7 @@ export function createSubtitleGenerationModal(
const vad = snapshot ? describeGenerationVad(snapshot.vad) : null; const vad = snapshot ? describeGenerationVad(snapshot.vad) : null;
const tools = snapshot ? describeGenerationTools(snapshot.tools) : null; const tools = snapshot ? describeGenerationTools(snapshot.tools) : null;
const readyMessage = !snapshot?.mediaPath const readyMessage = !snapshot?.mediaPath
? 'Open a local file or a YouTube video to generate subtitles.' ? 'Open a local file, a Jellyfin stream, or a YouTube video to generate subtitles.'
: !tools?.ready : !tools?.ready
? 'Install the missing tools or set their paths in Settings, then click Check again.' ? 'Install the missing tools or set their paths in Settings, then click Check again.'
: !model?.ready : !model?.ready
@@ -97,7 +97,8 @@ export function createSubtitleGenerationModal(
? 'Download the speech detection model or uncheck Focus on spoken dialogue.' ? 'Download the speech detection model or uncheck Focus on spoken dialogue.'
: 'Ready when you are.'; : 'Ready when you are.';
dom.media.textContent = dom.media.textContent =
snapshot?.mediaPath ?? 'Open a local file or a YouTube video in the player first.'; snapshot?.mediaPath ??
'Open a local file, a Jellyfin stream, or a YouTube video in the player first.';
dom.tools.textContent = tools?.text ?? 'Checking local tools...'; dom.tools.textContent = tools?.text ?? 'Checking local tools...';
dom.model.textContent = model?.text ?? 'Checking local models...'; dom.model.textContent = model?.text ?? 'Checking local models...';
dom.modelPicker.classList.toggle('hidden', !snapshot || Boolean(snapshot.externalModelPath)); dom.modelPicker.classList.toggle('hidden', !snapshot || Boolean(snapshot.externalModelPath));