mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-09-10 17:16:20 -07:00
feat(subtitles): add local Japanese subtitle generation
- Generate Japanese SRTs through the overlay modal or launcher - Support managed whisper.cpp models and optional dialogue detection
This commit is contained in:
@@ -369,6 +369,7 @@ const sidebar: DefaultTheme.SidebarItem[] = [
|
||||
{ text: 'Jellyfin', link: '/jellyfin-integration' },
|
||||
{ text: 'YouTube', link: '/youtube-integration' },
|
||||
{ text: 'Jimaku', link: '/jimaku-integration' },
|
||||
{ text: 'Subtitle Generation', link: '/subtitle-generation' },
|
||||
{ text: 'TsukiHime', link: '/tsukihime-integration' },
|
||||
{ text: 'AniList', link: '/anilist-integration' },
|
||||
{ text: 'AniSkip', link: '/aniskip-integration' },
|
||||
|
||||
@@ -1054,7 +1054,7 @@ This example is intentionally compact. The option table below documents availabl
|
||||
| `metadata.pattern` | string | Format pattern for metadata: `%f`=filename, `%F`=filename+ext, `%t`=time, `%T`=time with milliseconds, `<br>`=newline |
|
||||
| `isLapis` | object | Lapis/shared sentence-card config: `{ enabled, sentenceCardModel }`. Sentence/audio field names are fixed to `Sentence` and `SentenceAudio`. |
|
||||
| `isKiku` | object | Kiku-only config: `{ enabled, fieldGrouping, deleteDuplicateInAuto }` (shared sentence/audio/model settings are inherited from `isLapis`) |
|
||||
| `isSenren` | object | Senren-only config: `{ enabled, fieldGrouping, deleteDuplicateInAuto }`. Merges duplicates using Senren's scene-switching markup. Mutually exclusive with `isKiku.enabled`. |
|
||||
| `isSenren` | object | Senren-only config: `{ enabled, fieldGrouping, deleteDuplicateInAuto }`. Merges duplicates using Senren's scene-switching markup. Mutually exclusive with `isKiku.enabled`. |
|
||||
|
||||
`ankiConnect.ai` only controls feature-local enablement plus optional `model` / `systemPrompt` overrides.
|
||||
API key resolution, base URL, and timeout live under the shared top-level [`ai`](#shared-ai-provider) config.
|
||||
@@ -1199,6 +1199,14 @@ The keyboard shortcut lives under `shortcuts.openTsukihime` (default `Ctrl+Shift
|
||||
|
||||
See [TsukiHime Integration](/tsukihime-integration) for the modal workflow, language tabs, and troubleshooting.
|
||||
|
||||
### Japanese Subtitle Generation
|
||||
|
||||
Open the standalone modal with `Ctrl+Shift+G`, configurable through `shortcuts.openSubtitleGeneration`, or use the subtitle sidebar button. See [shortcuts](/shortcuts) for the shared mpv and overlay keybindings.
|
||||
|
||||
`subtitleGeneration` configures local Japanese transcription for both the launcher and overlay. In **Settings → Integrations → Japanese Subtitle Generation**, set `modelPath` to an existing multilingual whisper.cpp GGML model, or leave it empty and choose a `managedModel` as the default. The generation modal lets you select another model for the current session, with download sizes and accuracy versus speed guidance. Downloads are explicit. Leave `whisperPath`, `ffmpegPath`, and `ffprobePath` empty to find the executables on `PATH`, or set them to override the executable paths. `threads` controls the CPU thread count. Settings apply to the next operation. See [subtitle generation](/subtitle-generation) for setup and behavior, and the [generated configuration example](/config.example.jsonc) for defaults.
|
||||
|
||||
The generation modal offers an optional **Prioritize dialogue** checkbox and a separate Silero model download. Set `subtitleGeneration.vadModelPath` to a Silero GGML VAD model to make dialogue mode the default. `vadPath` overrides the speech detector executable. See [dialogue generation setup](/subtitle-generation#prioritizing-spoken-dialogue) for session behavior, the additional tool, and limitations.
|
||||
|
||||
### Subtitle Sync
|
||||
|
||||
Sync a subtitle track from the overlay picker using `alass` or `ffsubsync`. The picker lets you choose which track gets retimed (the active primary track by default) and, for alass, which reference it is aligned against (the secondary subtitle track by default). Both are **optional external tools** that must be installed separately and available on your `PATH` (or configured via the path options below).
|
||||
|
||||
@@ -14,6 +14,10 @@ Only **mpv** is strictly required to run SubMiner. Everything else enhances the
|
||||
|
||||
Several entries below exist only for the `subminer` command-line launcher, which is Linux and macOS only. On Windows you launch playback with the **SubMiner mpv** shortcut instead, so you can ignore those rows.
|
||||
|
||||
[Local Japanese subtitle generation](/subtitle-generation) additionally requires whisper.cpp's `whisper-cli`, FFmpeg, and `ffprobe`. Configure their executable paths in Settings if needed. SubMiner can download a speech model explicitly, or use your existing multilingual GGML model.
|
||||
|
||||
Optional [dialogue-focused generation](/subtitle-generation#prioritizing-spoken-dialogue) also uses whisper.cpp's speech segment detector and a separate Silero GGML VAD model.
|
||||
|
||||
| Dependency | Status | Platforms | What it does |
|
||||
| -------------------- | ----------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| mpv | Required | All | The video player SubMiner overlays on. Must support `--input-ipc-server`. |
|
||||
|
||||
@@ -159,6 +159,7 @@ subminer stats -b # start background stats daemon
|
||||
| `subminer stats rebuild` / `backfill` | Rebuild or backfill rollup data |
|
||||
| `subminer doctor` | Dependency + config + socket diagnostics (`--refresh-known-words` refreshes the known-word cache) |
|
||||
| `subminer settings` | Open the SubMiner settings window |
|
||||
| `subminer generate-subs [video]` | Generate [Japanese subtitles](/usage#generate-japanese-subtitles-locally) locally |
|
||||
| `subminer logs -e` | Export a sanitized local-date log ZIP and print its path |
|
||||
| `subminer config path` | Print active config file path |
|
||||
| `subminer config show` | Print active config contents |
|
||||
|
||||
@@ -6,6 +6,23 @@
|
||||
*/
|
||||
{
|
||||
|
||||
// ==========================================
|
||||
// Japanese Subtitle Generation
|
||||
// Generate timed Japanese subtitles from local audio using whisper.cpp.
|
||||
// Configure an existing GGML model path or explicitly download a SubMiner-managed model.
|
||||
// Hot-reload: settings apply to the next generation or model download.
|
||||
// ==========================================
|
||||
"subtitleGeneration": {
|
||||
"whisperPath": "", // Optional path override for whisper.cpp. Leave empty to find whisper-cli on PATH.
|
||||
"modelPath": "", // Path to an existing multilingual whisper.cpp GGML model. Leave empty to use a SubMiner-managed model. A configured path always takes precedence.
|
||||
"managedModel": "small", // Multilingual whisper.cpp model to use when modelPath is empty. Download it explicitly from the generation modal or launcher. Values: tiny | tiny-q5_1 | tiny-q8_0 | base | base-q5_1 | base-q8_0 | small | small-q5_1 | small-q8_0 | medium | medium-q5_0 | medium-q8_0 | large-v1 | large-v2 | large-v2-q5_0 | large-v2-q8_0 | large-v3 | large-v3-q5_0 | large-v3-turbo | large-v3-turbo-q5_0 | large-v3-turbo-q8_0
|
||||
"threads": 4, // Positive integer CPU thread count for whisper.cpp Japanese transcription.
|
||||
"ffmpegPath": "", // Optional FFmpeg path override for audio extraction. Leave empty to find ffmpeg on PATH.
|
||||
"ffprobePath": "", // Optional FFprobe path override for audio tracks and timing. Leave empty to find ffprobe on PATH.
|
||||
"vadModelPath": "", // Path to a whisper.cpp Silero VAD model. Enables dialogue-focused generation from separate speech passages. Leave empty to transcribe the full audio, including songs.
|
||||
"vadPath": "" // Optional speech detector executable override. With vadModelPath configured, leave empty to find whisper-vad-speech-segments on PATH.
|
||||
}, // Generate timed Japanese subtitles from local audio using whisper.cpp.
|
||||
|
||||
// ==========================================
|
||||
// Visible Overlay Auto-Start
|
||||
// Show the visible subtitle overlay automatically after managed mpv playback starts SubMiner.
|
||||
@@ -206,6 +223,7 @@
|
||||
"openRuntimeOptions": "CommandOrControl+Shift+O", // Accelerator that opens the runtime options modal.
|
||||
"openJimaku": "Ctrl+Shift+J", // Accelerator that opens the Jimaku subtitle search modal.
|
||||
"openTsukihime": "Ctrl+Shift+T", // Accelerator that opens the TsukiHime subtitle search modal (configured secondary/Japanese primary tabs).
|
||||
"openSubtitleGeneration": "Ctrl+Shift+G", // Accelerator that opens the standalone Japanese subtitle generation modal.
|
||||
"openSessionHelp": "CommandOrControl+Slash", // Accelerator that opens the session help / keybinding cheatsheet.
|
||||
"openControllerSelect": "Alt+C", // Accelerator that opens the controller selection and learn-mode modal.
|
||||
"openControllerDebug": "Alt+Shift+C", // Accelerator that opens the controller debug modal with live axis/button readouts.
|
||||
|
||||
@@ -82,6 +82,7 @@ Mouse-hover playback behavior is configured separately from shortcuts: `subtitle
|
||||
| `Ctrl/Cmd+Shift+O` | Open runtime options palette | `shortcuts.openRuntimeOptions` |
|
||||
| `Ctrl/Cmd+/` | Open session help modal | `shortcuts.openSessionHelp` |
|
||||
| `Ctrl+Shift+J` | Open Jimaku subtitle search modal | `shortcuts.openJimaku` |
|
||||
| `Ctrl+Shift+G` | Open Japanese subtitle generation modal | `shortcuts.openSubtitleGeneration` |
|
||||
| `Ctrl+Shift+T` | Open TsukiHime subtitle search modal (EN/JA tabs) | `shortcuts.openTsukihime` |
|
||||
| `Ctrl/Cmd+N` | Toggle overlay notification history panel | `shortcuts.toggleNotificationHistory` |
|
||||
| `Ctrl+Alt+C` | Open the manual YouTube subtitle picker | `keybindings` |
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
# Japanese subtitle generation
|
||||
|
||||
Generate Japanese SRT subtitles from a local video's audio using [whisper.cpp](https://github.com/ggml-org/whisper.cpp). The launcher and overlay use the same local generation service. Audio stays on your computer. Model downloads require an internet connection; generation with an installed model does not.
|
||||
|
||||
## Setup
|
||||
|
||||
Install whisper.cpp's `whisper-cli` executable and FFmpeg, including `ffprobe`. SubMiner downloads models, not these executables. Leave `whisperPath`, `ffmpegPath`, and `ffprobePath` empty to find the executables on `PATH`. To use a specific installation, set a path override under **Settings → Integrations → Japanese Subtitle Generation**.
|
||||
|
||||
Choose one model source:
|
||||
|
||||
- Set `subtitleGeneration.modelPath` to an existing **multilingual whisper.cpp GGML `.bin` model**. Python Whisper checkpoints and English-only models are not suitable for Japanese transcription.
|
||||
- Leave that path empty and choose a model directly in the generation modal. Each option shows its download size; the selected model has speed and accuracy guidance. The modal offers **Download model** when that model is missing. Your choice lasts for the current SubMiner session, including closing and reopening the modal. Set `subtitleGeneration.managedModel` in Settings to change the default for future sessions.
|
||||
|
||||
Managed models are stored in `models/whisper/` beside your SubMiner configuration file. Downloads show progress, verify the expected file size and SHA256, and publish the model only after verification. Cancelling or failing a download removes its temporary files. A configured external path always takes precedence; an unreadable path displays an error instead of silently downloading another model.
|
||||
|
||||
See the [generated configuration example](/config.example.jsonc) for current defaults. Changes apply to the next operation.
|
||||
|
||||
## Prioritizing spoken dialogue
|
||||
|
||||
To focus on dialogue, check the optional **Prioritize dialogue** box in the generation modal. If the speech detection model is missing, click **Download Silero** to install it. This separate download uses the same progress, cancellation, and integrity checks as Whisper downloads. Checking the box never downloads automatically, and leaving it unchecked lets you generate without the Silero model.
|
||||
|
||||
You also need whisper.cpp's [speech segment detector](https://github.com/ggml-org/whisper.cpp/tree/master/examples/vad-speech-segments). SubMiner downloads the model, not this executable. The detector is found as `whisper-vad-speech-segments` on `PATH`. Builds from the upstream source may name it `vad-speech-segments`; set `vadPath` in **Settings → Integrations → Japanese Subtitle Generation** when needed.
|
||||
|
||||
The checkbox choice lasts for the current SubMiner session, including closing and reopening the modal. To make dialogue mode your default, set `vadModelPath` in Settings to a [Silero GGML VAD model](https://huggingface.co/ggml-org/whisper-vad/tree/main). The modal downloads `ggml-silero-v6.2.0.bin` into the same `models/whisper/` directory as managed Whisper models. An existing configured VAD path takes precedence and checks the box initially. Unchecking it temporarily disables dialogue mode without changing that path. Downloading the model alone does not enable dialogue mode.
|
||||
|
||||
With speech detection configured, SubMiner transcribes short speech passages separately and restores each passage's position on the original audio timeline. It resets transcription context between passages and limits subtitle cues to the passage that produced them. A line cannot stretch across an omitted music break, and repeated dialogue in separate passages remains separate. Progress reports completed batches of dialogue passages.
|
||||
|
||||
This mode prioritizes spoken dialogue over songs and background sounds. It can miss quiet speech or speech mixed with loud music, and recognition errors are still possible. Uncheck **Prioritize dialogue** to return to full-audio transcription for the session, or clear `vadModelPath` to change the default. A selected detector or model that fails stops generation with an error. Existing subtitles are preserved.
|
||||
|
||||
## Choosing a model
|
||||
|
||||
Start with **small** for a balance of Japanese recognition quality and CPU time. This is a general starting recommendation, not a benchmark for your hardware. Tiny and base need less memory and usually finish sooner, with more recognition errors. Medium and large models favor accuracy but need more resources. Large-v3-turbo is optimized for speed compared with large-v3, with some accuracy tradeoff; actual performance depends on your CPU, GPU, whisper.cpp build, and audio.
|
||||
|
||||
The picker includes whisper.cpp's official multilingual tiny, base, small, medium, large-v1, large-v2, large-v3, and large-v3-turbo downloads, including their available quantized variants. Quantized models use less disk space and memory, with possible accuracy loss. English-only `.en` models are excluded. See the [upstream model list](https://github.com/ggml-org/whisper.cpp/blob/master/models/download-ggml-model.sh) and [Whisper's model guidance](https://github.com/openai/whisper#available-models-and-languages).
|
||||
|
||||
A configured external Model Path takes precedence and hides the managed model picker. Clear it in Settings to choose a managed model. Changing the picker never downloads automatically, and it cannot change the model during an active download or generation.
|
||||
|
||||
## From the overlay
|
||||
|
||||
1. Open a local video in mpv and select its Japanese audio track.
|
||||
2. Press **Ctrl+Shift+G** to open the standalone generation modal. You can also click **Generate Japanese subtitles** in the subtitle sidebar. Neither an open sidebar nor an existing subtitle track is required.
|
||||
3. Choose a model and download it if prompted, or configure your existing model path in Settings and click **Check again**.
|
||||
4. Optionally check **Prioritize dialogue** and download Silero if prompted.
|
||||
5. Click **Generate subtitles**.
|
||||
|
||||
The modal shows audio preparation, transcription, and saving progress. Percentages appear when the underlying tool reports them. **Cancel** stops the current operation. Closing the modal lets the job continue; reopening it shows the current progress or result.
|
||||
|
||||
**Escape** or **Close** closes the modal using the same focus and overlay restoration as other SubMiner modals. Change or disable its shortcut with `shortcuts.openSubtitleGeneration` in Settings. Ctrl+G remains assigned to field grouping.
|
||||
|
||||
SubMiner saves `<video>.ja.generated.srt` beside the media, adding a numeric suffix if that name already exists. It selects the generated Japanese subtitle track and resets the subtitle delay when mpv is still playing the same file. If playback changes, the subtitles remain saved and are not attached to the new video. The result includes the saved path even if mpv cannot load it.
|
||||
|
||||
## From the launcher
|
||||
|
||||
```bash
|
||||
subminer generate-subs episode.mkv --download-model
|
||||
subminer generate-subs episode.mkv --model-path /path/to/ggml-small.bin
|
||||
subminer generate-subs
|
||||
```
|
||||
|
||||
With no file argument, the command uses the current local mpv media and its selected audio track. With an explicit file, it prefers an audio stream tagged Japanese, otherwise the first audio stream. Use `--audio-stream` to choose an absolute FFmpeg stream index. `--output` specifies a new destination SRT; existing output files are never overwritten. See [launcher usage](/usage) for all flags. Ctrl+C cancels the operation.
|
||||
|
||||
## Timing and limitations
|
||||
|
||||
The SRT includes whisper.cpp's timestamps, adjusted for the audio stream's position on the media timeline and, when speech detection is configured, each passage's original start time. No alass step is required to load it. This version uses native Whisper timing; it does not run WhisperX or another forced aligner. Recognition can repeat or invent lines, and timing can be imperfect, especially with music or overlapping speech. Review generated text and audio boundaries when mining.
|
||||
|
||||
Generation supports local files and internal audio tracks. Remote URLs, subtitle translation, and transcription of a separately attached mpv audio track are not supported by the modal. Pass a separate local audio file to the launcher if needed. The destination directory needs writable space for subtitles; temporary storage needs enough space for the extracted mono audio.
|
||||
@@ -6,9 +6,12 @@ The sidebar is enabled by default. Set `subtitleSidebar.enabled` to `false` if y
|
||||
|
||||
## How It Works
|
||||
|
||||
The **Generate Japanese subtitles** button opens [local subtitle generation](/subtitle-generation). You can use it even when the current video has no subtitle track.
|
||||
|
||||
When SubMiner parses the active subtitle source into a cue list, the sidebar becomes available. Toggle it with the `\` key (configurable via `subtitleSidebar.toggleKey`). While open:
|
||||
|
||||
- The active cue is highlighted and kept in view as playback advances (when `autoScroll` is `true`).
|
||||
- Between subtitle lines, the sidebar follows playback to the next cue without jumping back to a cue at the start of the file.
|
||||
- Clicking any cue seeks mpv into that line. For overlapping ASS karaoke, SubMiner moves past the previous line's exit animation when the selected cue has enough time remaining.
|
||||
- The sidebar stays synchronized with the overlay - media transitions and subtitle source changes update both simultaneously.
|
||||
|
||||
|
||||
@@ -70,6 +70,7 @@ subminer https://youtu.be/... # Play a YouTube URL
|
||||
subminer stats # Open the immersion stats dashboard
|
||||
subminer doctor # Check dependencies, config, and the mpv socket
|
||||
subminer settings # Open the SubMiner settings window
|
||||
subminer generate-subs video.mkv # Generate Japanese subtitles from local audio
|
||||
subminer app --setup # Re-open first-run setup
|
||||
subminer -u # Check for updates
|
||||
```
|
||||
@@ -81,6 +82,40 @@ Two flags are worth knowing early:
|
||||
- `-a/--args` passes extra arguments straight to mpv, for example `subminer --args "--ao=alsa --volume=80" video.mkv`.
|
||||
- `--log-level debug` turns on verbose logging when something is not working.
|
||||
|
||||
### Generate Japanese subtitles locally
|
||||
|
||||
`generate-subs` transcribes local audio with whisper.cpp, saves a timed Japanese SRT file,
|
||||
and loads it into mpv if that same media file is still playing, clearing the previous subtitle
|
||||
delay. It also works with no running
|
||||
SubMiner app or mpv instance when you provide a file path. Omit the path to use the current
|
||||
mpv file and selected audio track.
|
||||
|
||||
```bash
|
||||
subminer generate-subs video.mkv --download-model
|
||||
subminer generate-subs video.mkv --model-path ~/models/ggml-medium.bin
|
||||
subminer generate-subs --model medium --download-model
|
||||
subminer generate-subs video.mkv --audio-stream 2 --output ~/Subs/video.ja.srt
|
||||
```
|
||||
|
||||
Install `whisper-cli` from whisper.cpp, `ffmpeg`, and `ffprobe`, or configure their paths in
|
||||
`subtitleGeneration.whisperPath`, `subtitleGeneration.ffmpegPath`, and `subtitleGeneration.ffprobePath`.
|
||||
Set `subtitleGeneration.modelPath` in settings to reuse an existing whisper.cpp model.
|
||||
With no external path, SubMiner uses `subtitleGeneration.managedModel` and stores downloaded
|
||||
models under `models/whisper` beside its config file. `--model` selects an official multilingual model, including available quantized variants, for
|
||||
this invocation and overrides a configured external model path. Run `subminer generate-subs --help`
|
||||
for accepted names. See [model selection](/subtitle-generation#choosing-a-model) for accuracy and speed guidance.
|
||||
|
||||
Downloads only happen when you pass `--download-model` or choose the download action in the
|
||||
generation modal. The launcher reports each stage and percentages when available. Press Ctrl+C
|
||||
to cancel. `--audio-stream` takes an absolute ffprobe stream index. When you provide a file
|
||||
path without that flag, generation uses a Japanese audio track when tagged, falling back
|
||||
to the first audio track. With no file path, mpv must have an identifiable selected audio
|
||||
track, or you must provide `--audio-stream`.
|
||||
|
||||
Generated files include Whisper's native timing. Speech recognition can make mistakes,
|
||||
especially over music or overlapping dialogue, so check the wording before mining. Existing
|
||||
output files are preserved. See [configuration](/configuration) for the generation settings.
|
||||
|
||||
<details>
|
||||
<summary><b>Less common launcher commands</b></summary>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user