feat(subtitles): generate subtitles from anime streams

- Support finite HTTP/HTTPS streams with cached generated SRT files
- Document stream requirements and limitations
This commit is contained in:
2026-09-21 21:28:15 -07:00
parent 625f1ad238
commit 2c89bc1754
15 changed files with 369 additions and 24 deletions
+3 -3
View File
@@ -95,8 +95,8 @@ Browse sibling episode files and the active mpv queue in one overlay modal. Open
<td>Search and download Japanese subtitles</td> <td>Search and download Japanese subtitles</td>
</tr> </tr>
<tr> <tr>
<td><b>Local Subtitle Generation</b></td> <td><b>Japanese Subtitle Generation</b></td>
<td>Generate Japanese subtitles from local audio in a standalone modal (<code>Ctrl+Shift+G</code>), the sidebar button, or launcher, with progress and optional managed model downloads. Requires whisper.cpp and FFmpeg. Optional Silero speech detection prioritizes dialogue in separately timed passages. <a href="https://docs.subminer.moe/main/subtitle-generation">Setup guide</a></td> <td>Generate Japanese subtitles from local files or finite anime streams in a standalone modal (<code>Ctrl+Shift+G</code>) or the sidebar button. The launcher supports local files. Transcription runs locally with whisper.cpp and FFmpeg, with progress and optional managed model downloads. Stream subtitles are saved in the application cache. Optional Silero speech detection prioritizes dialogue in separately timed passages. <a href="https://docs.subminer.moe/main/subtitle-generation">Setup guide</a></td>
</tr> </tr>
<tr> <tr>
<td><b>TsukiHime</b></td> <td><b>TsukiHime</b></td>
@@ -277,7 +277,7 @@ SubMiner builds on the work of these open-source projects:
| [Aniyomi](https://github.com/aniyomiorg/aniyomi) | Anime extension API and data model the anime browser targets | | [Aniyomi](https://github.com/aniyomiorg/aniyomi) | Anime extension API and data model the anime browser targets |
| [asbplayer](https://github.com/killergerbah/asbplayer) | Inspiration for subtitle sidebar and logic for YouTube subtitle parsing | | [asbplayer](https://github.com/killergerbah/asbplayer) | Inspiration for subtitle sidebar and logic for YouTube subtitle parsing |
| [Bee's Character Dictionary](https://github.com/bee-san/Japanese_Character_Name_Dictionary) | Character name recognition in subtitles | | [Bee's Character Dictionary](https://github.com/bee-san/Japanese_Character_Name_Dictionary) | Character name recognition in subtitles |
| [Bun](https://github.com/oven-sh/bun) | Bundled runtime for the `subminer` command-line launcher | | [Bun](https://github.com/oven-sh/bun) | Bundled runtime for the `subminer` command-line launcher |
| [GameSentenceMiner](https://github.com/bpwhelan/GameSentenceMiner) | Inspiration for Electron overlay with Yomitan integration | | [GameSentenceMiner](https://github.com/bpwhelan/GameSentenceMiner) | Inspiration for Electron overlay with Yomitan integration |
| [jellyfin-mpv-shim](https://github.com/jellyfin/jellyfin-mpv-shim) | Jellyfin integration | | [jellyfin-mpv-shim](https://github.com/jellyfin/jellyfin-mpv-shim) | Jellyfin integration |
| [Jimaku.cc](https://jimaku.cc) | Japanese subtitle search and downloads | | [Jimaku.cc](https://jimaku.cc) | Japanese subtitle search and downloads |
@@ -0,0 +1,4 @@
type: docs
area: subtitles
- Document anime stream generation, cache locations, full-episode processing, supported audio tracks, and stream access limitations.
@@ -0,0 +1,4 @@
type: added
area: subtitles
- Generate Japanese subtitles from finite HTTP/HTTPS anime episode streams in the existing generation modal. Reuse mpv's supported request headers and the anime proxy, transcribe downloaded audio locally, retain generated SRT files in the application cache, and load them only into the matching stream. Local-file generation and the launcher keep their existing behavior.
+4
View File
@@ -394,6 +394,10 @@ download falls back to its URL so the episode still plays, the format is
detected from the file's own content rather than its URL, and the directory is detected from the file's own content rather than its URL, and the directory is
removed when the next episode starts or the app exits. removed when the next episode starts or the app exits.
### Generating Japanese subtitles
While an episode is playing, press **Ctrl+Shift+G** to [generate Japanese subtitles](/subtitle-generation#generating-for-anime-streams) from its audio. SubMiner fetches the complete episode audio, transcribes it locally with whisper.cpp, and loads the resulting SRT if the same stream is still playing. Generated subtitles stay in the application cache. This requires a finite HTTP/HTTPS stream with an internal audio track; live streams and separately attached audio tracks are not supported.
### Series, season, and episode ### Series, season, and episode
The episode's identity travels with it instead of being guessed back out of the The episode's identity travels with it instead of being guessed back out of the
+13 -3
View File
@@ -1,6 +1,6 @@
# Japanese subtitle generation # Japanese subtitle generation
Generate Japanese SRT subtitles from a local video's audio using [whisper.cpp](https://github.com/ggml-org/whisper.cpp). The launcher and overlay use the same local generation service. Audio stays on your computer. Model downloads require an internet connection; generation with an installed model does not. Generate Japanese SRT subtitles from a local video's audio or an anime stream using [whisper.cpp](https://github.com/ggml-org/whisper.cpp). The launcher and overlay use the same local generation service. Transcription runs on your computer without uploading audio. Model downloads and stream audio extraction require an internet connection; local-file generation with an installed model does not.
## Setup ## Setup
@@ -51,7 +51,7 @@ A configured external Model Path takes precedence and hides the managed model pi
## From the overlay ## From the overlay
1. Open a local video in mpv and select its Japanese audio track. 1. Open a local video or an anime episode stream in mpv and select its Japanese audio track.
2. Press **Ctrl+Shift+G** to open the standalone generation modal. When the subtitle sidebar has no subtitle lines loaded, it also offers a **Generate Japanese subtitles** button. Neither an open sidebar nor an existing subtitle track is required for the shortcut. 2. Press **Ctrl+Shift+G** to open the standalone generation modal. When the subtitle sidebar has no subtitle lines loaded, it also offers a **Generate Japanese subtitles** button. Neither an open sidebar nor an existing subtitle track is required for the shortcut.
3. Choose a model and download it if prompted, or configure your existing model path in Settings and click **Check again**. 3. Choose a model and download it if prompted, or configure your existing model path in Settings and click **Check again**.
4. Optionally check **Focus on spoken dialogue** and click **Download speech detection model** if prompted. 4. Optionally check **Focus on spoken dialogue** and click **Download speech detection model** if prompted.
@@ -63,6 +63,16 @@ The modal adapts to the player window, using a wider layout when space allows an
SubMiner saves `<video>.ja.generated.srt` beside the media, adding a numeric suffix if that name already exists. It selects the generated Japanese subtitle track and resets the subtitle delay when mpv is still playing the same file. If playback changes, the subtitles remain saved and are not attached to the new video. The result includes the saved path even if mpv cannot load it. SubMiner saves `<video>.ja.generated.srt` beside the media, adding a numeric suffix if that name already exists. It selects the generated Japanese subtitle track and resets the subtitle delay when mpv is still playing the same file. If playback changes, the subtitles remain saved and are not attached to the new video. The result includes the saved path even if mpv cannot load it.
### Generating for anime streams
Start the episode from the [anime browser](/anime-browser), then open the generation modal as usual. HTTP and HTTPS episode streams, including HLS through the anime browser's proxy, can use the same models and dialogue options as local files. The selected audio track must be inside the stream; separately attached audio tracks are not supported.
SubMiner fetches the episode's audio into a temporary WAV before transcription. It reuses mpv's supported request headers, referrer, and user agent. The existing media-fetch policy excludes Cookie and Authorization headers, so sources requiring those directly may fail unless the anime bridge handles authentication. Expired URLs or a stopped bridge can also interrupt extraction; reopen the episode and retry.
Generated SRT files are retained under `cache/generated-subtitles` in SubMiner's application data directory, using a URL hash for the filename and a numeric suffix for repeated generation. The result shows the saved path. Temporary audio is removed after success, failure, or cancellation. Playback can continue during generation; the completed SRT is selected only if the same stream is still playing. Cached results are not automatically loaded on a later visit.
This processes the complete episode, so subtitles appear after extraction and transcription finish. Streams must report a finite duration; live streams are not supported. A stream with combined video and audio segments may require downloading those segments even though only audio is retained.
## From the launcher ## From the launcher
```bash ```bash
@@ -77,4 +87,4 @@ With no file argument, the command uses the current local mpv media and its sele
The SRT includes whisper.cpp's timestamps, adjusted for the audio stream's position on the media timeline and, when speech detection is configured, each passage's original start time. No alass step is required to load it. This version uses native Whisper timing; it does not run WhisperX or another forced aligner. Recognition can repeat or invent lines, and timing can be imperfect, especially with music or overlapping speech. Review generated text and audio boundaries when mining. The SRT includes whisper.cpp's timestamps, adjusted for the audio stream's position on the media timeline and, when speech detection is configured, each passage's original start time. No alass step is required to load it. This version uses native Whisper timing; it does not run WhisperX or another forced aligner. Recognition can repeat or invent lines, and timing can be imperfect, especially with music or overlapping speech. Review generated text and audio boundaries when mining.
Generation supports local files and internal audio tracks. Remote URLs, subtitle translation, and transcription of a separately attached mpv audio track are not supported by the modal. Pass a separate local audio file to the launcher if needed. The destination directory needs writable space for subtitles; temporary storage needs enough space for the extracted mono audio. The overlay supports local files and finite HTTP/HTTPS streams with internal audio tracks. The launcher remains local-file only. Subtitle translation and transcription of a separately attached mpv audio track are not supported by the modal. Pass a separate local audio file to the launcher if needed. The destination directory needs writable space for subtitles; temporary storage needs enough space for the extracted mono audio. Cached local subtitle tracks and embedded text subtitles can guide stream transcription; unreadable references fall back to audio timing.
+1 -1
View File
@@ -32,7 +32,7 @@ Update checks and startup launcher migration share a serialized update-state sto
- `src/main/boot/` owns boot-phase assembly seams so `src/main.ts` can stay focused on lifecycle coordination and startup-path selection. - `src/main/boot/` owns boot-phase assembly seams so `src/main.ts` can stay focused on lifecycle coordination and startup-path selection.
- `src/main/runtime/linux-overlay-mode-runtime.ts` owns Linux fullscreen mode state and window replacement. App cleanup cancels pending replacements; `main.ts` supplies window creation and subtitle refresh hooks. - `src/main/runtime/linux-overlay-mode-runtime.ts` owns Linux fullscreen mode state and window replacement. App cleanup cancels pending replacements; `main.ts` supplies window creation and subtitle refresh hooks.
- `src/core/services/` owns focused runtime services plus pure or side-effect-bounded logic. - `src/core/services/` owns focused runtime services plus pure or side-effect-bounded logic.
- `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. Optional dialogue mode retains both Silero-detected speech and other audible sections, omits confidently silent gaps, decodes passages independently, and restores original media timing. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same local media; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative. - `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. The overlay also accepts finite HTTP/HTTPS episode streams, snapshots mpv's supported HTTP headers, and extracts audio through FFmpeg with bounded network reads. Network reference extraction uses the same headers. Generated stream subtitles persist under the injected application cache directory with hashed filenames; temporary audio is removed after the job. The launcher remains local-file only. Optional dialogue mode retains both Silero-detected speech and other audible sections, omits confidently silent gaps, decodes passages independently, and restores original media timing. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same media; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative.
- Subtitle model recommendations use bounded `nvidia-smi` and Whisper CUDA discovery probes in `subtitle-generation-acceleration.ts`. The overlay runtime caches results by executable path for 30 seconds and exposes acceleration status through the existing status IPC. Recommendations do not alter model selection or transcription arguments. - Subtitle model recommendations use bounded `nvidia-smi` and Whisper CUDA discovery probes in `subtitle-generation-acceleration.ts`. The overlay runtime caches results by executable path for 30 seconds and exposes acceleration status through the existing status IPC. Recommendations do not alter model selection or transcription arguments.
- `subtitle-generation-reference.ts` ranks mpv's loaded text subtitle tracks, excludes signs/songs and forced references, and extracts timing hints with FFmpeg. The overlay and launcher snapshot references only for matching media, including active subtitle delays. Hints guide long-passage cuts with or without VAD; they never limit audio coverage or replace Whisper timestamps. - `subtitle-generation-reference.ts` ranks mpv's loaded text subtitle tracks, excludes signs/songs and forced references, and extracts timing hints with FFmpeg. The overlay and launcher snapshot references only for matching media, including active subtitle delays. Hints guide long-passage cuts with or without VAD; they never limit audio coverage or replace Whisper timestamps.
- `src/renderer/` owns overlay rendering and input behavior. - `src/renderer/` owns overlay rendering and input behavior.
@@ -0,0 +1,139 @@
import assert from 'node:assert/strict';
import { createServer } from 'node:http';
import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import test from 'node:test';
import { DEFAULT_SUBTITLE_GENERATION_CONFIG } from '../../shared/subtitle-generation';
import { generateJapaneseSubtitles } from './subtitle-generation';
import { runSubtitleGenerationProcess } from './subtitle-generation-process';
import { resolveSubtitleGenerationTools } from './subtitle-generation-tools';
test(
'real FFmpeg extracts a header-protected HLS episode and cleans up downloaded audio',
{
skip: process.platform === 'win32' ? 'Requires a POSIX executable fixture.' : false,
},
async (t) => {
const tools = await resolveSubtitleGenerationTools(DEFAULT_SUBTITLE_GENERATION_CONFIG);
if (tools.ffmpeg.kind !== 'found' || tools.ffprobe.kind !== 'found') {
t.skip('FFmpeg and ffprobe are required for the network extraction test.');
return;
}
const directory = await mkdtemp(path.join(tmpdir(), 'subminer-network-generation-'));
const requests: string[] = [];
const server = createServer(async (req, res) => {
if (
req.headers.referer !== 'https://anime.example/' ||
req.headers['user-agent'] !== 'SubMiner test'
) {
res.writeHead(403).end();
return;
}
const name = req.url?.slice(1) ?? '';
if (!/^(episode\.m3u8|segment\d+\.ts)$/.test(name)) {
res.writeHead(404).end();
return;
}
requests.push(name);
try {
res.end(await readFile(path.join(directory, name)));
} catch {
res.writeHead(404).end();
}
});
try {
await runSubtitleGenerationProcess({
command: tools.ffmpeg.path,
args: [
'-v',
'error',
'-f',
'lavfi',
'-i',
'sine=frequency=440:duration=3',
'-c:a',
'aac',
'-f',
'hls',
'-hls_time',
'1',
'-hls_playlist_type',
'vod',
'-hls_segment_filename',
path.join(directory, 'segment%d.ts'),
path.join(directory, 'episode.m3u8'),
],
});
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
const address = server.address();
assert.ok(address && typeof address === 'object');
const modelPath = path.join(directory, 'model.bin');
const model = Buffer.alloc(8);
model.writeUInt32LE(0x67676d6c, 0);
model.writeInt32LE(51865, 4);
await writeFile(modelPath, model);
const whisperPath = path.join(directory, 'whisper-test');
// Recognition is deterministic here; probing, HTTP requests and decoding use real FFmpeg.
await writeFile(
whisperPath,
`#!${process.execPath}
const fs = require('node:fs');
const assert = require('node:assert/strict');
const args = process.argv.slice(2);
const wav = args[args.indexOf('-f') + 1];
const bytes = fs.readFileSync(wav);
assert.equal(bytes.toString('ascii', 0, 4), 'RIFF');
assert.ok(bytes.length > 90000);
fs.writeFileSync(${JSON.stringify(path.join(directory, 'audio-path'))}, wav);
fs.writeFileSync(args[args.indexOf('-of') + 1] + '.srt', '1\\n00:00:00,500 --> 00:00:01,500\\nこんにちは\\n');
`,
{ mode: 0o755 },
);
const cacheDirectory = path.join(directory, 'cache');
const input = {
config: {
...DEFAULT_SUBTITLE_GENERATION_CONFIG,
modelPath,
whisperPath,
ffmpegPath: tools.ffmpeg.path,
ffprobePath: tools.ffprobe.path,
},
modelDirectory: directory,
mediaPath: `http://127.0.0.1:${address.port}/episode.m3u8`,
audioStreamIndex: 0,
remote: {
cacheDirectory,
httpHeaders: {
headers: { Referer: 'https://anime.example/' },
userAgent: 'SubMiner test',
},
},
};
const output = await generateJapaneseSubtitles(input);
assert.match(await readFile(output, 'utf8'), /00:00:00,500 --> 00:00:01,500\nこんにちは/);
assert.ok(requests.includes('episode.m3u8'));
assert.ok(requests.includes('segment2.ts'));
const wav = await readFile(path.join(directory, 'audio-path'), 'utf8');
await assert.rejects(readFile(wav), /ENOENT/);
await assert.rejects(
generateJapaneseSubtitles({
...input,
remote: { ...input.remote, httpHeaders: { headers: {}, userAgent: null } },
}),
/403/,
);
assert.deepEqual(await readdir(cacheDirectory), [path.basename(output)]);
} finally {
server.closeAllConnections();
await new Promise<void>((resolve, reject) =>
server.close((error) =>
error && (!('code' in error) || error.code !== 'ERR_SERVER_NOT_RUNNING')
? reject(error)
: resolve(),
),
);
await rm(directory, { recursive: true, force: true });
}
},
);
@@ -4,6 +4,8 @@ import { fileURLToPath } from 'node:url';
import type { SubtitleGenerationProgress } from '../../shared/subtitle-generation'; import type { SubtitleGenerationProgress } from '../../shared/subtitle-generation';
import { parseSrtCues } from './subtitle-cue-parser'; import { parseSrtCues } from './subtitle-cue-parser';
import { runSubtitleGenerationProcess } from './subtitle-generation-process'; import { runSubtitleGenerationProcess } from './subtitle-generation-process';
import type { ResolvedMpvHttpHeaders } from './mpv-http-headers';
import { subtitleGenerationHttpArgs } from './subtitle-generation-source';
export type SubtitleGenerationReference = { export type SubtitleGenerationReference = {
label: string; label: string;
@@ -119,6 +121,7 @@ export async function loadSubtitleGenerationReference(input: {
ffmpegPath: string; ffmpegPath: string;
directory: string; directory: string;
audioOffset: number; audioOffset: number;
httpHeaders?: ResolvedMpvHttpHeaders;
onProgress?: (progress: SubtitleGenerationProgress) => void; onProgress?: (progress: SubtitleGenerationProgress) => void;
signal?: AbortSignal; signal?: AbortSignal;
}): Promise<number[]> { }): Promise<number[]> {
@@ -135,6 +138,7 @@ export async function loadSubtitleGenerationReference(input: {
'-loglevel', '-loglevel',
'error', 'error',
...(embedded ? ['-copyts', '-start_at_zero'] : []), ...(embedded ? ['-copyts', '-start_at_zero'] : []),
...(embedded && input.httpHeaders ? subtitleGenerationHttpArgs(input.httpHeaders) : []),
'-i', '-i',
reference.source.kind === 'external' ? reference.source.path : input.mediaPath, reference.source.kind === 'external' ? reference.source.path : input.mediaPath,
'-map', '-map',
@@ -0,0 +1,17 @@
import { toFfmpegInputHttpArgs, type ResolvedMpvHttpHeaders } from './mpv-http-headers';
export interface SubtitleGenerationRemoteSource {
httpHeaders: ResolvedMpvHttpHeaders;
cacheDirectory: string;
}
/** Apply only to network inputs, including embedded subtitle reference extraction. */
export function subtitleGenerationHttpArgs(headers: ResolvedMpvHttpHeaders): string[] {
return [
'-protocol_whitelist',
'http,https,tcp,tls,crypto',
'-rw_timeout',
'15000000',
...toFfmpegInputHttpArgs(headers),
];
}
@@ -228,6 +228,65 @@ test('explicit audio stream and output path are respected without overwriting ex
); );
})); }));
test('remote generation forwards HTTP options, restores timestamps, and preserves cached subtitles', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
const mediaPath = 'https://anime.example/episode.m3u8?token=private';
const cacheDirectory = path.join(directory, 'cache');
const remote = {
cacheDirectory,
httpHeaders: { headers: { Referer: 'https://anime.example/' }, userAgent: 'SubMiner test' },
};
const first = await generateJapaneseSubtitles({ ...input, mediaPath, remote });
const second = await generateJapaneseSubtitles({ ...input, mediaPath, remote });
assert.equal(path.dirname(first), cacheDirectory);
assert.match(path.basename(first), /^[a-f0-9]{24}\.ja\.generated\.srt$/);
assert.equal(second, first.replace('.srt', '.1.srt'));
assert.match(await readFile(first, 'utf8'), /00:00:03,500 --> 00:00:04,500/);
const calls: string[][] = (await readFile(input.callsPath, 'utf8'))
.trim()
.split('\n')
.map((line) => JSON.parse(line));
for (const args of calls.filter((args) => args.includes(mediaPath))) {
assert.equal(args[args.indexOf('-user_agent') + 1], 'SubMiner test');
assert.equal(args[args.indexOf('-headers') + 1], 'Referer: https://anime.example/\r\n');
assert.ok(args.indexOf('-headers') < args.indexOf(mediaPath));
assert.equal(args[args.indexOf('-protocol_whitelist') + 1], 'http,https,tcp,tls,crypto');
}
assert.deepEqual(
(await readdir(cacheDirectory)).filter((name) => name.startsWith('.')),
[],
);
}));
test('remote generation refuses live streams and unsupported protocols before extraction', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
const remote = {
cacheDirectory: path.join(directory, 'cache'),
httpHeaders: { headers: {}, userAgent: null },
};
await assert.rejects(
generateJapaneseSubtitles({ ...input, mediaPath: 'ftp://anime.example/episode', remote }),
/supported HTTP stream/,
);
const ffprobePath = await executable(
directory,
'live-probe',
"process.stdout.write(JSON.stringify({ streams: [{ index: 0, codec_type: 'audio' }], format: {} }));",
);
await assert.rejects(
generateJapaneseSubtitles({
...input,
config: { ...input.config, ffprobePath },
mediaPath: 'https://anime.example/live.m3u8',
remote,
}),
/finite episode duration/,
);
await assert.rejects(readFile(input.callsPath), /ENOENT/);
}));
test('dialogue generation isolates Whisper state between passages and preserves media timing', () => test('dialogue generation isolates Whisper state between passages and preserves media timing', () =>
fixture(async (directory) => { fixture(async (directory) => {
const input = await generationFixture(directory); const input = await generationFixture(directory);
+34 -7
View File
@@ -1,5 +1,6 @@
import { access, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'; import { access, mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises';
import { constants } from 'node:fs'; import { constants } from 'node:fs';
import { createHash } from 'node:crypto';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import path from 'node:path'; import path from 'node:path';
import type { import type {
@@ -11,6 +12,10 @@ import { runSubtitleGenerationProcess } from './subtitle-generation-process';
import { publishSubtitleGenerationFile } from './subtitle-generation-files'; import { publishSubtitleGenerationFile } from './subtitle-generation-files';
import { formatTimestamp } from './subtitle-generation-srt'; import { formatTimestamp } from './subtitle-generation-srt';
import { transcribeSubtitleDialogue } from './subtitle-generation-dialogue'; import { transcribeSubtitleDialogue } from './subtitle-generation-dialogue';
import {
subtitleGenerationHttpArgs,
type SubtitleGenerationRemoteSource,
} from './subtitle-generation-source';
import { import {
loadSubtitleGenerationReference, loadSubtitleGenerationReference,
type SubtitleGenerationReference, type SubtitleGenerationReference,
@@ -178,21 +183,35 @@ export async function generateJapaneseSubtitles(input: {
modelDirectory: string; modelDirectory: string;
mediaPath: string; mediaPath: string;
audioStreamIndex?: number; audioStreamIndex?: number;
remote?: SubtitleGenerationRemoteSource;
references?: readonly SubtitleGenerationReference[]; references?: readonly SubtitleGenerationReference[];
outputPath?: string; outputPath?: string;
onProgress?: (progress: SubtitleGenerationProgress) => void; onProgress?: (progress: SubtitleGenerationProgress) => void;
signal?: AbortSignal; signal?: AbortSignal;
}): Promise<string> { }): Promise<string> {
input.signal?.throwIfAborted(); input.signal?.throwIfAborted();
if (/^[a-z][a-z\d+.-]*:\/\//i.test(input.mediaPath)) const isRemote = /^[a-z][a-z\d+.-]*:\/\//i.test(input.mediaPath);
throw new Error('Subtitle generation requires a local media file.'); if (isRemote && (!input.remote || !/^https?:\/\//i.test(input.mediaPath)))
const mediaPath = path.resolve(input.mediaPath); throw new Error('Subtitle generation requires a local media file or a supported HTTP stream.');
if (!(await stat(mediaPath)).isFile()) if (!isRemote && input.remote) throw new Error('Expected an HTTP stream for remote generation.');
const mediaPath = isRemote ? new URL(input.mediaPath).href : path.resolve(input.mediaPath);
if (!isRemote && !(await stat(mediaPath)).isFile())
throw new Error('Subtitle generation requires a local media file.'); throw new Error('Subtitle generation requires a local media file.');
// URLs may contain credentials or expiring tokens. Keep them out of cache filenames.
const destinationMediaPath = input.remote
? path.join(
input.remote.cacheDirectory,
createHash('sha256').update(mediaPath).digest('hex').slice(0, 24),
)
: mediaPath;
if (input.remote) await mkdir(input.remote.cacheDirectory, { recursive: true });
if (input.outputPath) await ensureAvailableOutput(path.resolve(input.outputPath)); if (input.outputPath) await ensureAvailableOutput(path.resolve(input.outputPath));
await ensureWritableDirectory( await ensureWritableDirectory(
input.outputPath ? path.dirname(path.resolve(input.outputPath)) : path.dirname(mediaPath), input.outputPath
? path.dirname(path.resolve(input.outputPath))
: path.dirname(destinationMediaPath),
); );
const httpArgs = input.remote ? subtitleGenerationHttpArgs(input.remote.httpHeaders) : [];
const model = await resolveSubtitleGenerationModel(input.config, input.modelDirectory); const model = await resolveSubtitleGenerationModel(input.config, input.modelDirectory);
if (model.kind === 'missing') if (model.kind === 'missing')
throw new Error( throw new Error(
@@ -210,11 +229,16 @@ export async function generateJapaneseSubtitles(input: {
'stream=index,codec_type,start_time,duration:stream_tags=language:format=start_time,duration', 'stream=index,codec_type,start_time,duration:stream_tags=language:format=start_time,duration',
'-of', '-of',
'json', 'json',
...httpArgs,
mediaPath, mediaPath,
], ],
signal: input.signal, signal: input.signal,
}); });
const audio = parseAudioProbe(probe, input.audioStreamIndex); const audio = parseAudioProbe(probe, input.audioStreamIndex);
if (isRemote && (!audio.duration || !Number.isFinite(audio.duration) || audio.duration <= 0))
throw new Error(
'Stream generation requires a finite episode duration. Live streams are not supported.',
);
const temporaryDirectory = await mkdtemp(path.join(tmpdir(), 'subminer-whisper-')); const temporaryDirectory = await mkdtemp(path.join(tmpdir(), 'subminer-whisper-'));
try { try {
const wavPath = path.join(temporaryDirectory, 'audio.wav'); const wavPath = path.join(temporaryDirectory, 'audio.wav');
@@ -227,10 +251,12 @@ export async function generateJapaneseSubtitles(input: {
'-hide_banner', '-hide_banner',
'-loglevel', '-loglevel',
'error', 'error',
...httpArgs,
'-i', '-i',
mediaPath, mediaPath,
'-map', '-map',
`0:${audio.index}`, `0:${audio.index}`,
...(isRemote ? ['-t', String(audio.duration)] : []),
'-vn', '-vn',
'-af', '-af',
'asetpts=PTS-STARTPTS', 'asetpts=PTS-STARTPTS',
@@ -268,6 +294,7 @@ export async function generateJapaneseSubtitles(input: {
ffmpegPath: tools.ffmpeg, ffmpegPath: tools.ffmpeg,
directory: temporaryDirectory, directory: temporaryDirectory,
audioOffset: audio.offset, audioOffset: audio.offset,
httpHeaders: input.remote?.httpHeaders,
onProgress: input.onProgress, onProgress: input.onProgress,
signal: input.signal, signal: input.signal,
}); });
@@ -317,7 +344,7 @@ export async function generateJapaneseSubtitles(input: {
input.onProgress?.({ stage: 'write', message: 'Saving Japanese subtitles...' }); input.onProgress?.({ stage: 'write', message: 'Saving Japanese subtitles...' });
const contents = shiftSubtitleTimestamps(srt, audio.offset); const contents = shiftSubtitleTimestamps(srt, audio.offset);
const outputPath = await writeSubtitles({ const outputPath = await writeSubtitles({
mediaPath, mediaPath: destinationMediaPath,
outputPath: input.outputPath, outputPath: input.outputPath,
contents, contents,
signal: input.signal, signal: input.signal,
+1
View File
@@ -6887,6 +6887,7 @@ function setOverlayVisible(visible: boolean): void {
registerIpcRuntimeHandlers(); registerIpcRuntimeHandlers();
const subtitleGenerationRuntime = createSubtitleGenerationRuntime({ const subtitleGenerationRuntime = createSubtitleGenerationRuntime({
getConfig: () => configService.getConfig().subtitleGeneration, getConfig: () => configService.getConfig().subtitleGeneration,
getCacheDirectory: () => path.join(USER_DATA_PATH, 'cache', 'generated-subtitles'),
getModelDirectory: () => getModelDirectory: () =>
path.join(path.dirname(configService.getConfigPath()), 'models', 'whisper'), path.join(path.dirname(configService.getConfigPath()), 'models', 'whisper'),
getMpvClient: () => appState.mpvClient, getMpvClient: () => appState.mpvClient,
@@ -21,6 +21,7 @@ function fixture(overrides: Partial<SubtitleGenerationRuntimeDeps> = {}) {
const runtime = createSubtitleGenerationRuntime({ const runtime = createSubtitleGenerationRuntime({
getConfig: () => DEFAULT_SUBTITLE_GENERATION_CONFIG, getConfig: () => DEFAULT_SUBTITLE_GENERATION_CONFIG,
getModelDirectory: () => '/models', getModelDirectory: () => '/models',
getCacheDirectory: () => '/cache/generated-subtitles',
getMpvClient: () => client, getMpvClient: () => client,
onProgress: () => {}, onProgress: () => {},
detectAcceleration: async () => ({ kind: 'unavailable' }), detectAcceleration: async () => ({ kind: 'unavailable' }),
@@ -72,6 +73,67 @@ test('generation preserves the output without attaching it to a different video'
assert.deepEqual(subject.commands, []); assert.deepEqual(subject.commands, []);
}); });
test('stream generation snapshots mpv headers and saves in the cache before loading', async () => {
const url = 'http://127.0.0.1:7777/proxy/episode.m3u8';
const subject = fixture({
generate: async (input) => {
assert.equal(input.mediaPath, url);
assert.equal(input.audioStreamIndex, 3);
assert.deepEqual(input.remote, {
cacheDirectory: '/cache/generated-subtitles',
httpHeaders: {
headers: { Referer: 'https://anime.example/', 'X-Stream': 'episode' },
userAgent: 'Anime Player',
},
});
return '/cache/generated-subtitles/episode.ja.generated.srt';
},
});
const request = subject.client.requestProperty;
subject.client.requestProperty = async (name) => {
if (name === 'path') return url;
if (name === 'file-local-options/http-header-fields')
return ['Referer: https://anime.example/', 'X-Stream: episode'];
if (name === 'file-local-options/user-agent') return 'Anime Player';
return request(name);
};
assert.equal((await subject.runtime.getStatus()).mediaPath, url);
assert.equal((await subject.runtime.start()).ok, true);
assert.deepEqual(subject.commands[0], [
'sub-add',
'/cache/generated-subtitles/episode.ja.generated.srt',
'select',
'Generated Japanese',
'ja',
]);
});
test('stream changes during header capture stop generation; changes during transcription keep the saved result', async () => {
for (const duringCapture of [true, false]) {
let current = 'https://anime.example/episode.m3u8';
const next = 'https://anime.example/next.m3u8';
let generated = false;
const subject = fixture({
generate: async () => {
generated = true;
current = next;
return '/cache/generated.srt';
},
});
const request = subject.client.requestProperty;
subject.client.requestProperty = async (name) => {
if (name === 'path') return current;
if (duringCapture && name === 'file-local-options/http-header-fields') current = next;
return request(name);
};
const result = await subject.runtime.start();
assert.equal(result.ok, !duringCapture);
assert.equal(generated, !duringCapture);
assert.match(result.message, /changed/);
assert.deepEqual(subject.commands, []);
}
});
test('generation selects loaded dialogue references and excludes the signs track', async () => { test('generation selects loaded dialogue references and excludes the signs track', async () => {
const subject = fixture({ const subject = fixture({
generate: async (input) => { generate: async (input) => {
@@ -1,4 +1,5 @@
import path from 'node:path'; import path from 'node:path';
import { resolveMpvHttpHeaders } from '../../core/services/mpv-http-headers';
import { readSubtitleGenerationReferences } from '../../core/services/subtitle-generation-reference'; import { readSubtitleGenerationReferences } from '../../core/services/subtitle-generation-reference';
import { detectSubtitleGenerationAcceleration } from '../../core/services/subtitle-generation-acceleration'; import { detectSubtitleGenerationAcceleration } from '../../core/services/subtitle-generation-acceleration';
import { SUBTITLE_GENERATION_VAD_MODEL } from '../../shared/subtitle-generation-vad-model'; import { SUBTITLE_GENERATION_VAD_MODEL } from '../../shared/subtitle-generation-vad-model';
@@ -31,6 +32,7 @@ interface GenerationMpvClient {
export interface SubtitleGenerationRuntimeDeps { export interface SubtitleGenerationRuntimeDeps {
getConfig: () => SubtitleGenerationConfig; getConfig: () => SubtitleGenerationConfig;
getModelDirectory: () => string; getModelDirectory: () => string;
getCacheDirectory: () => string;
getMpvClient: () => GenerationMpvClient | null; getMpvClient: () => GenerationMpvClient | null;
onProgress: (progress: SubtitleGenerationProgress) => void; onProgress: (progress: SubtitleGenerationProgress) => void;
generate?: typeof generateJapaneseSubtitles; generate?: typeof generateJapaneseSubtitles;
@@ -42,10 +44,12 @@ export interface SubtitleGenerationRuntimeDeps {
resolveVadModel?: typeof resolveSubtitleGenerationVadModel; resolveVadModel?: typeof resolveSubtitleGenerationVadModel;
} }
async function currentLocalMedia(client: GenerationMpvClient | null): Promise<string | null> { async function currentMedia(client: GenerationMpvClient | null): Promise<string | null> {
if (!client?.connected) return null; if (!client?.connected) return null;
const media = await client.requestProperty('path'); const media = await client.requestProperty('path');
if (typeof media !== 'string' || !media || /^[a-z][a-z\d+.-]*:\/\//i.test(media)) return null; if (typeof media !== 'string' || !media) return null;
if (/^https?:\/\//i.test(media)) return media;
if (/^[a-z][a-z\d+.-]*:\/\//i.test(media)) return null;
if (path.isAbsolute(media)) return path.normalize(media); if (path.isAbsolute(media)) return path.normalize(media);
const directory = await client.requestProperty('working-directory'); const directory = await client.requestProperty('working-directory');
return typeof directory === 'string' ? path.resolve(directory, media) : null; return typeof directory === 'string' ? path.resolve(directory, media) : null;
@@ -64,7 +68,9 @@ function selectedAudioIndex(tracks: unknown): number {
) )
continue; continue;
if ('external' in track && track.external === true) if ('external' in track && track.external === true)
throw new Error('Select an audio track inside the local video before generating subtitles.'); throw new Error(
'Select an audio track inside the video or stream before generating subtitles.',
);
if ( if (
'ff-index' in track && 'ff-index' in track &&
typeof track['ff-index'] === 'number' && typeof track['ff-index'] === 'number' &&
@@ -155,7 +161,7 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
config, config,
deps.getModelDirectory(), deps.getModelDirectory(),
); );
const mediaPath = await currentLocalMedia(deps.getMpvClient()).catch(() => null); const mediaPath = await currentMedia(deps.getMpvClient()).catch(() => null);
return { return {
model, model,
vad: { vad: {
@@ -236,15 +242,21 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
if (vad.kind === 'invalid') throw new Error(vad.message); if (vad.kind === 'invalid') throw new Error(vad.message);
} }
const client = deps.getMpvClient(); const client = deps.getMpvClient();
const mediaPath = await currentLocalMedia(client); const mediaPath = await currentMedia(client);
if (!client || !mediaPath) if (!client || !mediaPath)
throw new Error('Open a local video or audio file in mpv first.'); throw new Error('Open a local media file or HTTP stream in mpv first.');
const tracks = await client.requestProperty('track-list'); const tracks = await client.requestProperty('track-list');
const audioStreamIndex = selectedAudioIndex(tracks); const audioStreamIndex = selectedAudioIndex(tracks);
const remote = /^https?:\/\//i.test(mediaPath)
? {
httpHeaders: await resolveMpvHttpHeaders(client),
cacheDirectory: deps.getCacheDirectory(),
}
: undefined;
const references = await readSubtitleGenerationReferences(tracks, (name) => const references = await readSubtitleGenerationReferences(tracks, (name) =>
client.requestProperty(name), client.requestProperty(name),
); );
if ((await currentLocalMedia(client)) !== mediaPath) if ((await currentMedia(client)) !== mediaPath)
throw new Error('The current media changed. Start generation again.'); throw new Error('The current media changed. Start generation again.');
signal.throwIfAborted(); signal.throwIfAborted();
const outputPath = await (deps.generate ?? generateJapaneseSubtitles)({ const outputPath = await (deps.generate ?? generateJapaneseSubtitles)({
@@ -252,13 +264,14 @@ export function createSubtitleGenerationRuntime(deps: SubtitleGenerationRuntimeD
modelDirectory: deps.getModelDirectory(), modelDirectory: deps.getModelDirectory(),
mediaPath, mediaPath,
audioStreamIndex, audioStreamIndex,
remote,
references, references,
onProgress: report, onProgress: report,
signal, signal,
}); });
// Saving succeeds even if playback changes or disconnects during the job. // Saving succeeds even if playback changes or disconnects during the job.
try { try {
const playingMedia = await currentLocalMedia(client); const playingMedia = await currentMedia(client);
if (!signal.aborted && deps.getMpvClient() === client && playingMedia === mediaPath) { if (!signal.aborted && deps.getMpvClient() === client && playingMedia === mediaPath) {
const loaded = await client.request([ const loaded = await client.request([
'sub-add', 'sub-add',
+3 -2
View File
@@ -88,7 +88,7 @@ export function createSubtitleGenerationModal(
const vad = snapshot ? describeGenerationVad(snapshot.vad) : null; const vad = snapshot ? describeGenerationVad(snapshot.vad) : null;
const tools = snapshot ? describeGenerationTools(snapshot.tools) : null; const tools = snapshot ? describeGenerationTools(snapshot.tools) : null;
const readyMessage = !snapshot?.mediaPath const readyMessage = !snapshot?.mediaPath
? 'Open local media to generate subtitles.' ? 'Open a local file or anime stream to generate subtitles.'
: !tools?.ready : !tools?.ready
? 'Install the missing tools or set their paths in Settings, then click Check again.' ? 'Install the missing tools or set their paths in Settings, then click Check again.'
: !model?.ready : !model?.ready
@@ -96,7 +96,8 @@ export function createSubtitleGenerationModal(
: !vad?.ready : !vad?.ready
? 'Download the speech detection model or uncheck Focus on spoken dialogue.' ? 'Download the speech detection model or uncheck Focus on spoken dialogue.'
: 'Ready when you are.'; : 'Ready when you are.';
dom.media.textContent = snapshot?.mediaPath ?? 'Open a local media file in the player first.'; dom.media.textContent =
snapshot?.mediaPath ?? 'Open a local file or anime stream in the player first.';
dom.tools.textContent = tools?.text ?? 'Checking local tools...'; dom.tools.textContent = tools?.text ?? 'Checking local tools...';
dom.model.textContent = model?.text ?? 'Checking local models...'; dom.model.textContent = model?.text ?? 'Checking local models...';
dom.modelPicker.classList.toggle('hidden', !snapshot || Boolean(snapshot.externalModelPath)); dom.modelPicker.classList.toggle('hidden', !snapshot || Boolean(snapshot.externalModelPath));