mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-09-19 05:16:27 -07:00
feat(subtitles): use loaded subtitles to guide generation timing (#249)
This commit is contained in:
@@ -29,11 +29,21 @@ With speech detection configured, SubMiner keeps detected speech and other audib
|
||||
|
||||
Passages that fit within Whisper's 30-second audio window stay intact. Longer passages prefer nearby detected speech starts when choosing cuts, falling back to quiet pauses, with a small overlap to provide context. This reduces early subtitles caused by starting a clip well before its dialogue, while keeping all retained audio covered. Matching overlapping cues are combined even when punctuation differs; repeated dialogue at separate times remains separate. SubMiner runs each passage in a fresh Whisper process so decoder state from earlier audio cannot affect later passages. This reloads the model for each passage and can increase generation time. Subtitle cues stay within the supplied audio and retain each passage's position on the original timeline. Progress reports the current passage.
|
||||
|
||||
This mode favors retaining dialogue over excluding music, so songs and background sounds may also produce subtitles. It can take longer than transcribing only VAD-approved speech. Whisper can still miss or misrecognize dialogue, and its timestamps remain estimates. Uncheck **Focus on spoken dialogue** to return to full-audio transcription for the session, or clear `vadModelPath` to change the default. A selected detector or model that fails stops generation with an error. Existing subtitles are preserved.
|
||||
This mode favors retaining dialogue over excluding music, so songs and background sounds may also produce subtitles. It can take longer than transcribing only VAD-approved speech. Whisper can still miss or misrecognize dialogue, and its timestamps remain estimates. Uncheck **Focus on spoken dialogue** to disable VAD for the session, or clear `vadModelPath` to change the default. Without a usable subtitle reference, disabling VAD returns to full-audio transcription. A selected detector or model that fails stops generation with an error. Existing subtitles are preserved.
|
||||
|
||||
## Using loaded subtitles as timing references
|
||||
|
||||
When generating for the video currently open in mpv, SubMiner automatically looks for a dialogue subtitle track among its embedded subtitles and loaded external SRT, ASS/SSA, or WebVTT files. It prefers English, then tracks labeled full or dialogue. Forced tracks, image subtitles, generated subtitles, and tracks whose titles or filenames identify signs, songs, lyrics, karaoke, or opening/ending subtitles are skipped. These checks rely on metadata; an unlabeled signs-only file cannot always be identified.
|
||||
|
||||
The selected reference appears in generation progress. SubMiner reads its timestamps, including the active primary or secondary track's subtitle delay, and uses nearby cue starts to guide cuts in long audio passages. Short passages stay intact. The reference works with or without **Focus on spoken dialogue**. With that option enabled, reference starts take priority over VAD starts when choosing a nearby cut; VAD still helps identify speech. Audio outside reference cues remains eligible for transcription, and Whisper still supplies the Japanese text and final timestamps. The reference is assumed to be timed for the playing video; this does not automatically sync a mistimed reference.
|
||||
|
||||
Unreadable or empty references are skipped in favor of another eligible loaded track. If none can be read, generation uses its normal audio timing. The launcher uses loaded references only when its input matches the video currently open in mpv; standalone generation keeps its existing behavior. It captures reference tracks and delays together with the initial audio selection, before checking or downloading a model. When relying on mpv's selected audio, it stops and asks you to retry if the media changes or cannot be verified during capture.
|
||||
|
||||
## Choosing a model
|
||||
|
||||
Start with **small** for a balance of Japanese recognition quality and CPU time. This is a general starting recommendation, not a benchmark for your hardware. Tiny and base need less memory and usually finish sooner, with more recognition errors. Medium and large models favor accuracy but need more resources. Large-v3-turbo is optimized for speed compared with large-v3, with some accuracy tradeoff; actual performance depends on your CPU, GPU, whisper.cpp build, and audio.
|
||||
The modal recommends **large-v3-turbo** when it detects an NVIDIA GPU through `nvidia-smi` and the selected `whisper-cli` discovers an available CUDA device. Otherwise it recommends **small** for a balance of Japanese recognition quality and CPU time. The check works before downloading a model and falls back to small if a tool is missing, fails, times out, or reports an unrecognized result. Vulkan, AMD, and Apple GPU support do not qualify for the turbo recommendation. Checks are cached for up to 30 seconds; changing the Whisper executable path triggers a new check.
|
||||
|
||||
The recommendation labels the model picker and explains the detected support. It does not change your configured model, current selection, external model path, or launcher's model choice. It also does not force a GPU backend during transcription. These are starting recommendations, not hardware benchmarks or guarantees that every model fits in available GPU memory. Tiny and base need less memory and usually finish sooner, with more recognition errors. Medium and large models favor accuracy but need more resources. Large-v3-turbo is optimized for speed compared with large-v3, with some accuracy tradeoff; actual performance depends on your CPU, GPU, whisper.cpp build, and audio.
|
||||
|
||||
The picker includes whisper.cpp's official multilingual tiny, base, small, medium, large-v1, large-v2, large-v3, and large-v3-turbo downloads, including their available quantized variants. Quantized models use less disk space and memory, with possible accuracy loss. English-only `.en` models are excluded. See the [upstream model list](https://github.com/ggml-org/whisper.cpp/blob/master/models/download-ggml-model.sh) and [Whisper's model guidance](https://github.com/openai/whisper#available-models-and-languages).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user