fix(subtitles): retain uncertain audio during dialogue generation

- Preserve audio rejected by VAD so dialogue under music is retained.
- Keep short passages intact and stitch overlapping cues despite punctuation differences.
This commit is contained in:
2026-09-14 19:16:34 -07:00
parent d6cbacf8be
commit 02792232c2
15 changed files with 242 additions and 19 deletions
@@ -0,0 +1,4 @@
type: docs
area: subtitles
- Explain how dialogue generation retains uncertain audible sections, why songs may also be transcribed, and how passage boundaries affect subtitles.
+1 -1
View File
@@ -3,4 +3,4 @@ area: subtitles
- Generate local Japanese SRT subtitles with whisper.cpp from a standalone modal opened with Ctrl+Shift+G, the empty subtitle sidebar's generation button, or `subminer generate-subs`, with shared progress reporting, cancellation, safe output files, and automatic loading into the matching mpv video. The sidebar button hides while subtitle lines are loaded.
- Configure an existing multilingual model in Settings or choose an official multilingual model, including quantized variants, in the modal or launcher. The modal shows download sizes, speed and accuracy guidance, and a recommended starting model before explicitly downloading a verified SubMiner-managed model. Executable paths are optional overrides; empty fields find whisper-cli, ffmpeg, and ffprobe on PATH. The modal's Local tools check and the launcher name any missing executable and its setting before downloading a model or extracting audio, and generation confirms the destination directory grants write and search permissions up front.
- Optionally select Focus on spoken dialogue in the modal and use Download speech detection model to install the separate Silero model with progress and cancellation. The choice lasts for the session; a configured VAD model path sets the default. With the detector executable installed, retain brief utterances and extra audio around speech, split long passages near quiet pauses with overlapping context, and combine duplicate cues by greatest timing overlap. Keep original media timing and separate repeated dialogue without spanning omitted music breaks.
- Optionally select Focus on spoken dialogue in the modal and use Download speech detection model to install the separate Silero model with progress and cancellation. The choice lasts for the session; a configured VAD model path sets the default. Retain uncertain audible sections so VAD rejection does not discard dialogue under music, accepting that songs may also be transcribed. Keep passages intact within Whisper's audio window, split longer passages near quiet pauses with overlapping context, and combine overlapping duplicate cues even when punctuation differs. Preserve original media timing and separate repeated dialogue.
+1 -1
View File
@@ -19,7 +19,7 @@
"threads": 4, // Positive integer CPU thread count for whisper.cpp Japanese transcription.
"ffmpegPath": "", // Optional FFmpeg path override for audio extraction. Leave empty to find ffmpeg on PATH.
"ffprobePath": "", // Optional FFprobe path override for audio tracks and timing. Leave empty to find ffprobe on PATH.
"vadModelPath": "", // Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation from separate speech passages. Leave empty to transcribe the full audio, including songs.
"vadModelPath": "", // Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation while retaining uncertain audible sections, which may include songs. Leave empty to transcribe the full audio.
"vadPath": "" // Optional speech detector executable override. With vadModelPath configured, leave empty to find whisper-vad-speech-segments or vad-speech-segments on PATH.
}, // Generate timed Japanese subtitles from local audio using whisper.cpp.
+1 -1
View File
@@ -19,7 +19,7 @@
"threads": 4, // Positive integer CPU thread count for whisper.cpp Japanese transcription.
"ffmpegPath": "", // Optional FFmpeg path override for audio extraction. Leave empty to find ffmpeg on PATH.
"ffprobePath": "", // Optional FFprobe path override for audio tracks and timing. Leave empty to find ffprobe on PATH.
"vadModelPath": "", // Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation from separate speech passages. Leave empty to transcribe the full audio, including songs.
"vadModelPath": "", // Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation while retaining uncertain audible sections, which may include songs. Leave empty to transcribe the full audio.
"vadPath": "" // Optional speech detector executable override. With vadModelPath configured, leave empty to find whisper-vad-speech-segments or vad-speech-segments on PATH.
}, // Generate timed Japanese subtitles from local audio using whisper.cpp.
+3 -3
View File
@@ -25,11 +25,11 @@ You also need whisper.cpp's [speech segment detector](https://github.com/ggml-or
The checkbox choice lasts for the current SubMiner session, including closing and reopening the modal. To make dialogue mode your default, set `vadModelPath` in Settings to a [Silero GGML VAD model](https://huggingface.co/ggml-org/whisper-vad/tree/main). The modal downloads `ggml-silero-v6.2.0.bin` into the same `models/whisper/` directory as managed Whisper models. An existing configured VAD path takes precedence and checks the box initially. Unchecking it temporarily disables dialogue mode without changing that path. Downloading the model alone does not enable dialogue mode.
With speech detection configured, SubMiner transcribes short speech passages separately and restores each passage's position on the original audio timeline. Detection retains brief utterances and includes extra audio around speech to reduce clipped syllables. If the detector returns a long passage, SubMiner looks for quiet pauses near chunk boundaries. Adjacent chunks overlap slightly to provide context when speech continues through a cut. Matching subtitle cues in that overlap are combined; repeated dialogue at separate times remains separate.
With speech detection configured, SubMiner keeps detected speech and other audible sections for Whisper to evaluate. A low speech score alone does not discard audio, which helps retain dialogue mixed with music. Only confidently silent gaps outside detected speech are omitted, with extra audio retained around each passage to reduce clipped syllables.
SubMiner resets transcription context between passages and limits subtitle cues to the audio supplied for each chunk. A line cannot stretch across an omitted music break. Progress reports completed batches of dialogue passages. These adjustments do not replace Whisper's timestamp estimates or guarantee that every spoken line is recognized.
Passages that fit within Whisper's 30-second audio window stay intact. Longer passages split near quiet pauses, with a small overlap to provide context. Matching overlapping cues are combined even when punctuation differs; repeated dialogue at separate times remains separate. SubMiner resets transcription context between passages, limits subtitle cues to the supplied audio, and restores each passage's position on the original timeline. Progress reports completed batches of passages.
This mode prioritizes spoken dialogue over songs and background sounds. It can miss quiet speech or speech mixed with loud music, and recognition errors are still possible. Uncheck **Focus on spoken dialogue** to return to full-audio transcription for the session, or clear `vadModelPath` to change the default. A selected detector or model that fails stops generation with an error. Existing subtitles are preserved.
This mode favors retaining dialogue over excluding music, so songs and background sounds may also produce subtitles. It can take longer than transcribing only VAD-approved speech. Whisper can still miss or misrecognize dialogue, and its timestamps remain estimates. Uncheck **Focus on spoken dialogue** to return to full-audio transcription for the session, or clear `vadModelPath` to change the default. A selected detector or model that fails stops generation with an error. Existing subtitles are preserved.
## Choosing a model
+1 -1
View File
@@ -31,7 +31,7 @@ Update checks and startup launcher migration share a serialized update-state sto
- `src/main/` owns composition, runtime setup, IPC wiring, and app lifecycle adapters.
- `src/main/boot/` owns boot-phase assembly seams so `src/main.ts` can stay focused on lifecycle coordination and startup-path selection.
- `src/core/services/` owns focused runtime services plus pure or side-effect-bounded logic.
- `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. Optional Silero detection groups short speech passages before transcription, decodes each independently, and restores original media timing without joining omitted gaps. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same local media; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative.
- `src/core/services/subtitle-generation*.ts` shares local whisper.cpp transcription, safe model downloads, and progress between the launcher and Electron. Optional dialogue mode retains both Silero-detected speech and other audible sections, omits confidently silent gaps, decodes passages independently, and restores original media timing. `src/main/runtime/subtitle-generation-runtime.ts` owns the overlay job lifecycle and only loads completed subtitles into the same local media; `src/shared/subtitle-generation*.ts` owns configuration, the multilingual model catalog, and IPC contracts. The overlay runtime retains a session model selection, validates picker requests through IPC, and keeps external model paths authoritative.
- `src/renderer/` owns overlay rendering and input behavior.
- `src/config/` owns config definitions, defaults, loading, and resolution.
- `src/types/` owns shared cross-runtime contracts via domain entrypoints; `src/types.ts` stays a compatibility barrel.
+1 -1
View File
@@ -22,7 +22,7 @@ export function buildSubtitleConfigOptionRegistry(
ffprobePath:
'Optional FFprobe path override for audio tracks and timing. Leave empty to find ffprobe on PATH.',
vadModelPath:
'Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation from separate speech passages. Leave empty to transcribe the full audio, including songs.',
'Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation while retaining uncertain audible sections, which may include songs. Leave empty to transcribe the full audio.',
vadPath:
'Optional speech detector executable override. With vadModelPath configured, leave empty to find whisper-vad-speech-segments or vad-speech-segments on PATH.',
}[key],
@@ -23,6 +23,23 @@ test('uninterrupted speech retains overlapping context without crossing omitted
]);
});
test('speech fitting one Whisper window stays intact instead of cutting a sentence at 20 seconds', () => {
const passage = { startSeconds: 251.71, endSeconds: 275.01 };
assert.deepEqual(splitSpeechPassages([passage], [271.327]), [passage]);
});
test('chunk stitching ignores punctuation differences without merging separate repetitions', () => {
const cues = [{ startTime: 19.7, endTime: 21.2, text: 'ありがとう' }];
appendSpeechChunkCues(cues, [
{ startTime: 19.8, endTime: 21.3, text: 'ありがとう。' },
{ startTime: 22, endTime: 23, text: 'ありがとう!' },
]);
assert.deepEqual(cues, [
{ startTime: 19.7, endTime: 21.3, text: 'ありがとう' },
{ startTime: 22, endTime: 23, text: 'ありがとう!' },
]);
});
test('chunk stitching removes matching overlap cues but retains repeated dialogue', () => {
const cues = [{ startTime: 19.7, endTime: 20.2, text: 'はい' }];
appendSpeechChunkCues(cues, [
@@ -2,14 +2,17 @@ import type { SubtitleCue } from './subtitle-cue-parser';
import { SPEECH_PASSAGE_SECONDS, type SpeechPassage } from './subtitle-generation-speech';
const CHUNK_CONTEXT_SECONDS = 0.25;
const WHISPER_WINDOW_SECONDS = 30;
const PAUSE_SEARCH_SECONDS = 5;
// Prefer a quiet pause near the end of each chunk. Context stays inside detected speech.
// Prefer a quiet pause near the end of each chunk. Context stays inside retained audio.
export function splitSpeechPassages(
passages: readonly SpeechPassage[],
pauses: readonly number[] = [],
): SpeechPassage[] {
return passages.flatMap((passage) => {
if (passage.endSeconds - passage.startSeconds <= WHISPER_WINDOW_SECONDS)
return [{ ...passage }];
const chunks: SpeechPassage[] = [];
let boundary = passage.startSeconds;
while (boundary < passage.endSeconds) {
@@ -43,12 +46,13 @@ export function appendSpeechChunkCues(cues: SubtitleCue[], incoming: readonly Su
const previousCount = cues.length;
const matched = new Set<SubtitleCue>();
for (const cue of incoming) {
const text = cue.text.replace(/\s+/g, '');
const text = cue.text.replace(/[\s\p{P}]+/gu, '');
let duplicate: SubtitleCue | undefined;
let greatestOverlap = 0;
for (const [index, previous] of cues.entries()) {
if (index >= previousCount) break;
if (matched.has(previous) || previous.text.replace(/\s+/g, '') !== text) continue;
if (!text || matched.has(previous) || previous.text.replace(/[\s\p{P}]+/gu, '') !== text)
continue;
const overlap =
Math.min(previous.endTime, cue.endTime) - Math.max(previous.startTime, cue.startTime);
const shorterDuration = Math.min(
@@ -0,0 +1,73 @@
import assert from 'node:assert/strict';
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import test from 'node:test';
import { findAudiblePassages, mergeSpeechPassages } from './subtitle-generation-coverage';
async function analyze(lines: string[], progress = 'out_time_us=20000000\n') {
const directory = await mkdtemp(path.join(tmpdir(), 'subtitle-coverage-test-'));
try {
const ffmpegPath = path.join(directory, 'ffmpeg');
await writeFile(
ffmpegPath,
`#!${process.execPath}
process.stderr.write(${JSON.stringify(lines.join('\n') + '\n')});
process.stdout.write(${JSON.stringify(progress)});
`,
{ mode: 0o755 },
);
return await findAudiblePassages({ ffmpegPath, wavPath: 'audio.wav' });
} finally {
await rm(directory, { recursive: true, force: true });
}
}
test('audible coverage retains the full timeline when there is no confident silence', async () => {
assert.deepEqual(await analyze([]), [{ startSeconds: 0, endSeconds: 20 }]);
});
test('audible coverage omits silence while padding nearby audio without exceeding the timeline', async () => {
assert.deepEqual(
await analyze([
'[silencedetect] silence_start: 0',
'[silencedetect] silence_end: 2 | silence_duration: 2',
'[silencedetect] silence_start: 8',
'[silencedetect] silence_end: 12 | silence_duration: 4',
'[silencedetect] silence_start: 18',
]),
[
{ startSeconds: 1.65, endSeconds: 8.35 },
{ startSeconds: 11.65, endSeconds: 18.35 },
],
);
assert.deepEqual(await analyze(['[silencedetect] silence_end: 10.5 | silence_duration: 0.5']), [
{ startSeconds: 0, endSeconds: 20 },
]);
});
test('entirely silent audio has no audible passages', async () => {
assert.deepEqual(await analyze(['[silencedetect] silence_start: 0']), []);
assert.deepEqual(await analyze(['[silencedetect] silence_end: 20 | silence_duration: 20']), []);
});
test('missing analysis duration fails instead of silently dropping audio', async () => {
await assert.rejects(analyze([], ''), /valid duration/);
});
test('merging coverage preserves quiet VAD speech and does not mutate detector results', () => {
const speech = [{ startSeconds: 10, endSeconds: 11 }];
assert.deepEqual(
mergeSpeechPassages([
...speech,
{ startSeconds: 0, endSeconds: 5 },
{ startSeconds: 4, endSeconds: 8 },
{ startSeconds: 11, endSeconds: 12 },
]),
[
{ startSeconds: 0, endSeconds: 8 },
{ startSeconds: 10, endSeconds: 12 },
],
);
assert.deepEqual(speech, [{ startSeconds: 10, endSeconds: 11 }]);
});
@@ -0,0 +1,78 @@
import { runSubtitleGenerationProcess } from './subtitle-generation-process';
import type { SpeechPassage } from './subtitle-generation-speech';
const AUDIO_PADDING_SECONDS = 0.35;
export function mergeSpeechPassages(passages: readonly SpeechPassage[]): SpeechPassage[] {
const merged: SpeechPassage[] = [];
for (const passage of [...passages].sort((a, b) => a.startSeconds - b.startSeconds)) {
const previous = merged.at(-1);
if (previous && passage.startSeconds <= previous.endSeconds)
previous.endSeconds = Math.max(previous.endSeconds, passage.endSeconds);
else merged.push({ ...passage });
}
return merged;
}
// VAD rejection is not proof of silence. Preserve audible gaps for Whisper to evaluate.
export async function findAudiblePassages(input: {
ffmpegPath: string;
wavPath: string;
signal?: AbortSignal;
}): Promise<SpeechPassage[]> {
const silences: SpeechPassage[] = [];
let duration = 0;
let trailingSilence: number | undefined;
await runSubtitleGenerationProcess({
command: input.ffmpegPath,
args: [
'-nostdin',
'-hide_banner',
'-nostats',
'-i',
input.wavPath,
'-af',
'silencedetect=noise=-50dB:d=0.5',
'-progress',
'pipe:1',
'-f',
'null',
'-',
],
signal: input.signal,
onLine: (line) => {
const progress = /^out_time_us=(\d+)$/.exec(line);
if (progress) duration = Math.max(duration, Number(progress[1]) / 1_000_000);
const start = /silence_start: (\S+)/.exec(line);
if (start && Number.isFinite(Number(start[1]))) trailingSilence = Number(start[1]);
const end = /silence_end: (\S+) \| silence_duration: (\S+)/.exec(line);
if (!end) return;
const endSeconds = Number(end[1]);
const length = Number(end[2]);
if (Number.isFinite(endSeconds) && Number.isFinite(length) && length > 0) {
silences.push({ startSeconds: Math.max(0, endSeconds - length), endSeconds });
trailingSilence = undefined;
}
},
});
if (!Number.isFinite(duration) || duration <= 0)
throw new Error('Audio analysis did not report a valid duration.');
if (trailingSilence !== undefined)
silences.push({ startSeconds: trailingSilence, endSeconds: duration });
const audible: SpeechPassage[] = [];
let cursor = 0;
for (const silence of mergeSpeechPassages(silences)) {
if (cursor >= duration) break;
if (silence.startSeconds > cursor)
audible.push({ startSeconds: cursor, endSeconds: Math.min(duration, silence.startSeconds) });
cursor = Math.max(cursor, silence.endSeconds);
}
if (cursor < duration) audible.push({ startSeconds: cursor, endSeconds: duration });
return mergeSpeechPassages(
audible.map((passage) => ({
startSeconds: Math.max(0, passage.startSeconds - AUDIO_PADDING_SECONDS),
endSeconds: Math.min(duration, passage.endSeconds + AUDIO_PADDING_SECONDS),
})),
);
}
@@ -17,6 +17,7 @@ import {
import type { SubtitleCue } from './subtitle-cue-parser';
import { appendSpeechChunkCues, splitSpeechPassages } from './subtitle-generation-chunks';
import { findSpeechPauses } from './subtitle-generation-pauses';
import { findAudiblePassages, mergeSpeechPassages } from './subtitle-generation-coverage';
const PASSAGES_PER_BATCH = 16;
@@ -61,7 +62,14 @@ export async function transcribeSubtitleDialogue(input: {
segmentLines.push(line);
},
});
const detected = parseSpeechPassages(segmentLines.join('\n'));
const speech = parseSpeechPassages(segmentLines.join('\n'));
input.onProgress?.({ stage: 'transcribe', percent: 0, message: 'Checking audio coverage...' });
const audible = await findAudiblePassages({
ffmpegPath: input.tools.ffmpeg,
wavPath: input.wavPath,
signal: input.signal,
});
const detected = mergeSpeechPassages([...speech, ...audible]);
if (detected.length === 0) throw new Error('No spoken dialogue detected.');
const pauses = detected.some(
(passage) => passage.endSeconds - passage.startSeconds > SPEECH_PASSAGE_SECONDS,
@@ -49,7 +49,7 @@ export function parseSpeechPassages(output: string): SpeechPassage[] {
return grouped;
}
// Clamp to the audio actually supplied to Whisper. A cue cannot cross an omitted music break.
// Clamp to the audio actually supplied to Whisper. A cue cannot cross an omitted gap.
export function speechPassageCues(srt: string, passage: SpeechPassage): SubtitleCue[] {
const duration = passage.endSeconds - passage.startSeconds;
const cues = parseSrtCues(srt);
+43 -4
View File
@@ -54,7 +54,15 @@ async function generationFixture(directory: string) {
const ffmpegPath = await executable(
directory,
'ffmpeg',
`${record}\nrequire('node:fs').writeFileSync(process.argv.at(-1), 'wav'); process.stdout.write('out_time_'); setTimeout(() => process.stdout.write('us=10000000\\nprogress=end\\n'), 10);`,
`${record}
if (process.argv.at(-1) === '-') {
process.stderr.write('[silencedetect] silence_end: 25 | silence_duration: 25\\n');
process.stdout.write('out_time_us=25000000\\nprogress=end\\n');
} else {
require('node:fs').writeFileSync(process.argv.at(-1), 'wav');
process.stdout.write('out_time_');
setTimeout(() => process.stdout.write('us=10000000\\nprogress=end\\n'), 10);
}`,
);
const whisperPath = await executable(
directory,
@@ -217,7 +225,10 @@ process.stdout.write('Detected 1 speech segments:\\nSpeech segment 0: start = 10
directory,
'pause-ffmpeg',
`const args = process.argv.slice(2);
if (args.includes('-af')) {
if (args.includes('silencedetect=noise=-50dB:d=0.5')) {
process.stderr.write('[silencedetect] silence_end: 45 | silence_duration: 45\\n');
process.stdout.write('out_time_us=45000000\\n');
} else if (args.includes('-af')) {
process.stderr.write('[silencedetect] silence_end: 28.1 | silence_duration: 0.2\\n');
} else {
require('node:fs').writeFileSync(args.at(-1), 'wav');
@@ -241,7 +252,7 @@ for (let i = 0; i < args.length; i++) if (args[i] === '-of') {
assert.equal(await readFile(output, 'utf8'), '1\n00:00:30,300 --> 00:00:30,900\nはい\n');
}));
test('no detected speech stops generation without transcribing the full audio', () =>
test('silent audio with no detected speech stops generation without transcription', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
const vadModelPath = path.join(directory, 'vad.bin');
@@ -255,13 +266,41 @@ test('no detected speech stops generation without transcribing the full audio',
generateJapaneseSubtitles({ ...input, config: { ...input.config, vadModelPath, vadPath } }),
/No spoken dialogue detected/,
);
assert.equal((await readFile(input.callsPath, 'utf8')).trim().split('\n').length, 2);
assert.equal((await readFile(input.callsPath, 'utf8')).trim().split('\n').length, 3);
assert.deepEqual(
(await readdir(directory)).filter((file) => file.endsWith('.srt')),
[],
);
}));
test('dialogue generation retains audible audio rejected by VAD', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
const vadModelPath = path.join(directory, 'vad.bin');
await writeFile(vadModelPath, 'speech detector model');
const vadPath = await executable(
directory,
'vad',
"process.stdout.write('Detected 0 speech segments:\\n');",
);
const ffmpegPath = await executable(
directory,
'audible-ffmpeg',
`
const args = process.argv.slice(2);
if (args.at(-1) === '-') {
process.stdout.write('out_time_us=19000000\\nprogress=end\\n');
} else {
require('node:fs').writeFileSync(args.at(-1), 'wav');
}`,
);
const output = await generateJapaneseSubtitles({
...input,
config: { ...input.config, vadModelPath, vadPath, ffmpegPath },
});
assert.match(await readFile(output, 'utf8'), /00:00:03,500 --> 00:00:04,500\nこんにちは/);
}));
test('empty executable paths find tools on PATH and explicit overrides take precedence', () =>
fixture(async (directory) => {
const input = await generationFixture(directory);
+2 -2
View File
@@ -708,8 +708,8 @@
</label>
<div id="subtitleGenerationVadModel" class="subtitle-generation-hint"></div>
<p id="subtitleGenerationVadHint" class="subtitle-generation-hint">
Uses a small Silero model to transcribe separate speech passages. May skip songs,
quiet speech, and dialogue under music. Requires whisper.cpp's speech detector.
Keeps uncertain audio to avoid losing dialogue under music. Songs may also be
transcribed. Requires whisper.cpp's speech detector.
</p>
<p class="subtitle-generation-hint">
Applies for this session. Set VAD Model Path in Settings to enable it by default.