Compare commits

...

43 Commits

Author SHA1 Message Date
sudacode 8797719a09 docs(docs-site): correct and clarify docs against current code
Comprehensive accuracy pass over docs-site verifying every page against
current source. Fixes wrong/stale claims (AniSkip default, YouTube track
selection, Anki sentence-card requirements, immersion schema v18 + SQL
column names, plugin entrypoint, launcher flags, Cloudflare deploy path,
etc.) and fills gaps (watch history, mediaCache.maxHeight, youtubeSubgen,
character-dictionary refresh/eviction, expanded hot-reload lists).
2026-07-11 00:53:34 -07:00
sudacode 8acc78cc1c test(tokenizer): add golden-file regression corpus with 11 fixtures (#157) 2026-07-11 00:50:03 -07:00
sudacode 321461c50f feat(launcher): add subminer sync for cross-machine stats/history merge over SSH (#152) 2026-07-11 00:40:15 -07:00
sudacode beeab564b4 chore(release): prepare v0.18.0 2026-07-10 02:03:10 -07:00
sudacode cab7975a2b chore(docs): remove implemented plan docs
- Drop early-managed-overlay-startup and macos-notification-hover design docs (both implemented, no longer needed)
2026-07-10 01:09:52 -07:00
sudacode 846292809c docs(changes): consolidate and tighten changelog entries
- Merge related fix entries into single files (name-match-span-merge into character-name-split-and-scan-recovery, kanji-non-independent-noun-annotations + lexicalized-expression-frequency into content-adverb-annotation-stop-list, greedy-name-parsing-priority into character-name-split-and-scan-recovery, kiku-field-grouping-cancel-double-toast into kiku-field-grouping-modal-recovery, history-rofi-cover-art into launcher-history-command)
- Rewrite remaining entries for brevity and reclassify stats-trend-title-limits as "changed"
2026-07-10 01:04:01 -07:00
sudacode 8712780d08 fix(tokenizer): keep frequency rank for lexicalized kana expressions (#156) 2026-07-10 00:54:17 -07:00
sudacode 7b1a005a65 feat(anki): mirror mpv software volume into generated sentence audio (#155) 2026-07-10 00:44:23 -07:00
sudacode 84c75f50aa fix(release): skip gh attribution without CI token 2026-07-09 02:26:00 -07:00
sudacode 8b838f2c7d chore(release): prepare 0.18.0-beta.3 2026-07-09 01:20:02 -07:00
sudacode db4139ba0b fix(tokenizer): block kanji reading collisions and name/generic length t
- Add allowReadingOnlyMatch:false to kanji token known-word lookups so 渓谷/けいこく no longer matches a mined 警告/けいこく card via reading-only index
- Greedy name pre-pass yields when a strictly longer generic word starts at the same position (空 no longer splits 空気; ties still go to the name)
2026-07-09 01:03:58 -07:00
sudacode 6c251502b3 fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153) 2026-07-09 00:28:38 -07:00
sudacode ae40934d3a fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names (#151) 2026-07-08 23:53:45 -07:00
sudacode cdb1475a54 chore(assets): update icons and favicons 2026-07-08 22:35:47 -07:00
sudacode a2e49b369b fix(tokenizer): merge scanner metadata per token instead of all-or-nothi
- Replace hasSameTokenSpans + full-discard with mergeScannerTokensIntoParseTokens
- Grafts isNameMatch/frequencyRank/etc onto matching parse spans; filler chunks degrade only themselves
- Fixes name annotations dropping for entire subtitle lines containing unmatched interjections
2026-07-08 22:26:34 -07:00
sudacode 7f13aed50a fix(overlay): keep frequency/JLPT highlight for kanji non-independent nouns (#150) 2026-07-08 22:25:32 -07:00
sudacode 8b21a2bca8 fix(overlay): remove content adverbs from annotation stop-word list
- 確かに and やはり no longer excluded from frequency/JLPT highlighting and vocab stats
- Stop-word list now covers only interjections, pronouns, and grammar fragments
2026-07-08 20:15:45 -07:00
sudacode 925413adfe chore(release): prepare 0.18.0-beta.2 2026-07-08 02:31:14 -07:00
sudacode d0644ab2eb fix(stats): parse v3 reading-aware known-word cache in stats server (#149) 2026-07-08 02:15:58 -07:00
sudacode d253710c2e fix(stats): fetch cover art eagerly at session start instead of on series page visit (#148) 2026-07-08 00:59:35 -07:00
sudacode c3df510e4f docs(release): reclassify audio normalization as added, not fixed
- Move card audio normalization entry from Fixed → Added in prerelease notes
- Update changes/audio-normalization.md type: fixed → added
2026-07-08 00:38:38 -07:00
sudacode 187f68e5b6 fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings (#147) 2026-07-07 23:57:47 -07:00
sudacode 0e254cbbef fix(launcher): move fzf previews below menus 2026-07-07 22:36:56 -07:00
sudacode 7b94adafbd chore(release): prepare 0.18.0-beta.1 2026-07-07 02:40:05 -07:00
sudacode 61f39d1e09 fix(overlay): resolve unspaced Japanese name splits and scan recovery (#146) 2026-07-07 02:33:46 -07:00
sudacode e7739de51c fix(tokenizer): keep unparsed Yomitan tokens hoverable (#145) 2026-07-07 02:01:04 -07:00
sudacode ad1d240f20 refactor(tests): centralize lane definitions and add per-file isolation (#143) 2026-07-07 01:51:05 -07:00
sudacode 8b0ef662bc chore: remove shipped changelog entries and demo asset
- Drop changes/kiku-field-grouping-modal-wiring.md (landed in #138)
- Drop changes/known-word-partial-reading-highlight.md (landed in #142)
- Remove assets/minecard.mkv demo video
2026-07-07 00:18:48 -07:00
sudacode 0a58c20ad7 fix(stats): start stats server on background app launch (#144) 2026-07-07 00:16:57 -07:00
sudacode 38ddb29aa0 feat(anki): reading-aware known-word matching (cache v3) (#142) 2026-07-07 00:13:10 -07:00
sudacode 8b9a70c5a6 feat(stats): Trends dashboard overhaul — title visibility, ranking modes, calendar-accurate windows, tooltips (#140) 2026-07-06 23:52:43 -07:00
sudacode 48a084914a feat(branding): replace app icon with contributed pixel-art set (#141) 2026-07-06 23:31:19 -07:00
sudacode a042b04357 fix(overlay): prevent field grouping modal from freezing overlay on Hyprland (#138) 2026-07-06 22:13:14 -07:00
sudacode 35ca2afc6f fix(overlay): collapse karaoke syllable spam in secondary subtitles (#139) 2026-07-06 01:09:06 -07:00
sudacode b14f977e33 fix(tokenizer): stop partial furigana readings from marking words known (#137) 2026-07-05 17:02:44 -07:00
sudacode eef4500599 feat(launcher): add -H/--history command to browse local watch history (#136) 2026-07-05 16:46:05 -07:00
sudacode 73af1451b7 fix(anki): lower audio normalization target to -23 LUFS (EBU R128) 2026-07-05 01:13:54 -07:00
sudacode 36a3704815 fix(youtube): parse mpv EDL stream URLs with byte-length guards (#134) 2026-07-02 01:37:37 -07:00
sudacode 359cb0a301 Normalize generated Anki audio by default (#135) 2026-07-02 00:34:16 -07:00
sudacode 4b10e85053 chore(release): prepare v0.17.2 2026-06-28 23:03:18 -07:00
sudacode c942a2cf2d fFix(youtube): recover source URL for background media cache on direct mpv open (#132) 2026-06-28 22:46:11 -07:00
sudacode f65afa6046 feat(youtube): notify on manual picker open and show success after track load (#133) 2026-06-28 22:43:16 -07:00
sudacode 389d8e06e0 chore: remove stale npm artifacts 2026-06-28 20:18:08 -07:00
238 changed files with 130433 additions and 5500 deletions
+3 -3
View File
@@ -55,9 +55,6 @@ jobs:
- name: Verify generated config examples
run: bun run verify:config-example
- name: Internal docs knowledge-base checks
run: bun run test:docs:kb
- name: Test suite (source)
run: bun run test:fast
@@ -71,6 +68,9 @@ jobs:
path: coverage/test-src/lcov.info
if-no-files-found: error
- name: Stats UI tests
run: bun run test:stats
- name: Launcher smoke suite (source)
run: bun run test:launcher:smoke:src
+3
View File
@@ -70,6 +70,9 @@ jobs:
path: coverage/test-src/lcov.info
if-no-files-found: error
- name: Stats UI tests
run: bun run test:stats
- name: Launcher smoke suite (source)
run: bun run test:launcher:smoke:src
+3
View File
@@ -61,6 +61,9 @@ jobs:
path: coverage/test-src/lcov.info
if-no-files-found: error
- name: Stats UI tests
run: bun run test:stats
- name: Launcher smoke suite (source)
run: bun run test:launcher:smoke:src
+3
View File
@@ -1,5 +1,6 @@
# Dependencies
node_modules/
/package-lock.json
# Superpowers brainstorming
.superpowers/
@@ -16,6 +17,8 @@ coverage/
# Launcher build artifact (produced by make build-launcher)
/subminer
/main-entry.js
/main-entry.js.map
# Logs
*.log
+1
View File
@@ -4,3 +4,4 @@ release
coverage
vendor
*.log
src/core/services/tokenizer/__fixtures__/golden/*.json
+3
View File
@@ -42,7 +42,10 @@ Start here, then leave this file.
- Config/schema/defaults: `bun run test:config`; if template/defaults changed, `bun run generate:config-example`
- Launcher/plugin: `bun run test:launcher` or `bun run test:env`
- Runtime-compat / dist-sensitive: `bun run test:runtime:compat`
- Stats dashboard UI (`stats/`): `bun run test:stats`
- Build/release scripts (`scripts/**`): `bun run test:scripts`
- Docs-only: `bun run docs:test`, then `bun run docs:build`
- Test lanes are directory-discovered via `scripts/test-lanes.ts`; never hand-list test files in `package.json`
## Docs Upkeep
+37
View File
@@ -1,5 +1,42 @@
# Changelog
## v0.18.0 (2026-07-10)
### Added
- Sentence Audio Normalization: Generated sentence audio is now normalized to -23 LUFS by default, and clips mined from playback mirror mpv's software volume curve with a limiter to prevent clipping. Both behaviors are configurable independently.
- Watch History Command: Added `subminer -H` / `--history` to browse watch history, replay or continue episodes, or pick one via fzf or rofi, with cover art shown in the rofi picker.
### Changed
- Fzf Preview Layout: Moved fzf previews below launcher menus, giving long titles and metadata more room.
- Known-Word Highlighting: Now compares subtitle and Anki-card readings, preventing false matches between homographs and unrelated words that share a reading, while still supporting matching across kana and kanji spellings.
- Annotation Filtering: Standalone suffix tokens (e.g. さん, れる) are now excluded from JLPT/frequency/N+1 highlighting by default, matching how particles and interjections are treated; configurable via the pos2 exclusion setting.
- App Icon: Replaced the app icon with new pixel-art submarine artwork contributed by the community, used across the app icon, tray, notifications, README, docs site, and Stats page.
- Stats Trend Charts: Overhauled with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and value-sorted tooltips.
### Fixed
- Background Stats Server: `subminer app` background launches now auto-start the stats server when enabled, and skip startup if one is already running.
- Character Name Highlighting: Character dictionaries now split unspaced native names more reliably, and portraits, highlights, and hover lookup survive punctuation, unmatched text, and competing dictionary matches without incorrectly splitting longer words.
- Highlighting Coverage: Frequency/JLPT highlighting and vocabulary stats now include content adverbs (e.g. 確かに, やはり) and kanji nouns MeCab tags as non-independent (e.g. 日, 点, 以外), while still suppressing interjections, pronouns, and grammar fragments; lexicalized kana expressions like かといって keep their annotations.
- Cover Art Fetching: Stats now fetches AniList cover art as soon as a new series starts playing, and backfills missing art for existing series on the next Stats visit.
- Kiku Field Grouping: The manual field-grouping dialog now stays above fullscreen mpv, remains usable across repeated attempts, closes abandoned windows after timeouts, and reports a clear error when the original card can no longer be loaded.
- Karaoke Subtitle Collapse: Secondary subtitles no longer stack dozens of one-syllable lines during openings and endings; repeated events collapse into one line, capped to a strip at the top.
- Unparsed Token Hover: Subtitle text Yomitan can't parse (truncated inflections, elongation runs) remains hoverable, while staying excluded from highlighting, N+1 candidate math, and vocabulary stats.
- YouTube Streaming: Fixed direct YouTube stream extraction that could corrupt signed URLs and cause ffmpeg 403 errors.
<details>
<summary>Internal changes</summary>
### Internal
- Test lanes moved to `scripts/test-lanes.ts` with per-directory discovery and isolated per-file timeouts; CI now covers previously orphaned stats, scripts, plugin process-retry, and runtime-compat suites, plus a new stats lane in the change-verification workflow.
</details>
## v0.17.2 (2026-06-28)
### Fixed
- YouTube Background Cache: Fixed Windows YouTube background media cache startup for YouTube URLs opened directly in mpv, including resolved stream URLs when mpv still exposes the original YouTube playlist entry, so queued Anki media updates can append audio and images after the cache finishes.
- YouTube Subtitle Picker: Manual subtitle picker requests now show an immediate configured notification while SubMiner probes tracks and opens the modal. Subtitle download progress is replaced with a transient success notification after tracks load.
## v0.17.1 (2026-06-27)
### Added
+1
View File
@@ -213,6 +213,7 @@ On **Windows**, just run `SubMiner.exe` and the setup will open automatically on
subminer video.mkv # launch mpv with SubMiner
subminer /path/to/dir # pick a file with fzf
subminer -R /path/to/dir # pick a file with rofi (Linux only)
subminer -H # browse local watch history (replay / next episode / browse)
```
On **Windows**, use the **SubMiner mpv** shortcut created during setup. Double-click it or drag a video file onto it.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.5 MiB

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 364 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 MiB

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 433 B

After

Width:  |  Height:  |  Size: 253 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 580 B

After

Width:  |  Height:  |  Size: 366 B

Binary file not shown.
+7
View File
@@ -0,0 +1,7 @@
type: internal
area: tokenizer
- Added a golden-file regression corpus for the tokenizer/annotation pipeline: recorded Yomitan backend responses and MeCab tokens replay through the real tokenizeSubtitle pipeline in bun tests without Electron or dictionaries.
- Added `record-tokenizer-fixture:electron` script to capture new fixtures from a live Yomitan/MeCab session, with flags for known words, JLPT levels, and annotation toggles.
- Seeded eleven fixtures covering the #147#156 regression classes (grammar-helper suppression, lexical くれる, kanji non-independent nouns, N+1 targeting, reading collisions, unparsed runs, ordinal/honorific prefixes).
- Added `compare-yomitan-api:electron` script that diffs SubMiner tokenization against a stock Yomitan instance via the yomitan-api bridge (segmentation, readings, headword forms).
+4
View File
@@ -0,0 +1,4 @@
type: added
area: launcher
- Added `subminer sync <host>` to merge immersion stats and watch history between two machines over SSH, with `--push` and `--pull` for one-way insert-only transfers. Each side snapshots its database (`VACUUM INTO`), snapshots are exchanged with `scp`, and each machine merges the other's data as an insert-only union keyed on session UUIDs / video keys / series title keys, so re-syncing is idempotent and nothing is double-counted. Lifetime totals and daily/monthly rollups are updated incrementally (history older than the session retention window is preserved); remote-only historical rollups are copied only when they do not conflict with retained local session history. `subminer sync --snapshot <file>` and `subminer sync --merge <file>` expose the underlying steps for manual transfers. The command refuses to run while the stats daemon or a live mpv session is active (`--force` overrides), ignores stale mpv socket files, keeps the guard in place through local/remote merges, supplies standard SubMiner and Bun paths to non-interactive SSH commands, verifies the remote launcher starts, reports remote stderr on failures, and aborts on stats schema version mismatches.
+4 -2
View File
@@ -515,7 +515,7 @@
// ==========================================
// AnkiConnect Integration
// Automatic Anki updates and media generation options.
// Hot-reload: ankiConnect.ai.enabled, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Shared AI provider transport settings are read from top-level ai and typically require restart.
// Most other AnkiConnect settings still require restart.
// ==========================================
@@ -559,6 +559,8 @@
"animatedMaxHeight": 0, // Maximum height for animated AVIF captures, in pixels. Set to 0 to preserve aspect ratio.
"animatedCrf": 35, // Animated AVIF CRF quality target. Lower values produce larger, higher-quality files.
"syncAnimatedImageToWordAudio": true, // For animated AVIF images, prepend a frozen first frame matching the existing word-audio duration so motion starts with sentence audio. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Changes apply live. Values: true | false
"mirrorMpvVolume": true, // Apply mpv's current software volume curve to generated sentence audio. Changes apply live. Values: true | false
"audioPadding": 0, // Seconds of padding appended to both ends of generated sentence audio and animated AVIF clips.
"fallbackDuration": 3, // Fallback clip duration in seconds when subtitle timing data is unavailable.
"maxMediaDuration": 30 // Maximum allowed media clip duration in seconds.
@@ -568,7 +570,7 @@
"refreshMinutes": 1440, // Minutes between known-word cache refreshes.
"addMinedWordsImmediately": true, // Immediately append newly mined card words into the known-word cache. Values: true | false
"matchMode": "headword", // Known-word matching strategy for subtitle annotations. Cache matches always receive known-word highlighting even when POS filters suppress other annotation types. Values: headword | surface
"decks": {} // Decks and expression/word fields for known-word cache. Object mapping deck names to arrays of field names to extract, e.g. { "Kaishi 1.5k": ["Word"] }.
"decks": {} // Decks and expression/word fields for known-word cache. Object mapping deck names to arrays of field names to extract, e.g. { "Kaishi 1.5k": ["Word"] }. Reading fields (Reading, Word Reading, ExpressionReading) are always probed so cached words match only in the reading their note teaches; words from notes without readings match in any reading.
}, // Known words setting.
"behavior": {
"overwriteAudio": true, // When updating an existing card, overwrite the audio field instead of skipping it. Values: true | false
+4 -3
View File
@@ -41,7 +41,7 @@ The update flow:
1. **Title detection** -- SubMiner extracts the anime title, season, and episode number from the media filename and path. Season folders such as `Season 2` are treated as a strong season signal. SubMiner tries [`guessit`](https://github.com/guessit-io/guessit) first for accurate parsing, then falls back to an internal filename parser if guessit is unavailable.
2. **AniList search** -- The detected title is searched against the AniList GraphQL API. For season 2 and later files, SubMiner searches the season-specific title first, then falls back to the base title. SubMiner picks the best match by comparing titles (romaji, English, native) and filtering by episode count.
3. **Progress check** -- SubMiner fetches your current list entry for the matched media. The media must already be in Planning or Watching; otherwise SubMiner shows an MPV message explaining that the update is not possible. If your recorded progress already meets or exceeds the detected episode, the update is skipped.
4. **Mutation** -- A `SaveMediaListEntry` mutation sets the new progress and marks the entry as `CURRENT`.
4. **Mutation** -- A `SaveMediaListEntry` mutation sets the new progress and marks the entry as `CURRENT`, or `COMPLETED` when the watched episode is the final episode of the season (the "already at this progress" skip is bypassed for the final episode so completion still lands).
## Update Queue and Retry
@@ -81,7 +81,6 @@ All AniList API calls go through a shared rate limiter that enforces a sliding w
"enabled": true,
"accessToken": "",
"characterDictionary": {
"enabled": false,
"maxLoaded": 3,
"profileScope": "all",
"collapsibleSections": {
@@ -99,10 +98,12 @@ All AniList API calls go through a shared rate limiter that enforces a sliding w
| `enabled` | `true`, `false` | Enable AniList post-watch progress updates (default: `false`) |
| `accessToken` | string | Explicit AniList access token override; when blank, SubMiner uses the stored encrypted token (default: `""`) |
| `characterDictionary.maxLoaded` | number | Number of recent media snapshots kept in the merged dictionary (default: `3`) |
| `characterDictionary.refreshTtlHours` | number | Hours before a cached media snapshot is refreshed (default: `168`, clamped to 18760) |
| `characterDictionary.evictionPolicy` | `"delete"`, `"disable"` | What happens to snapshots evicted beyond `maxLoaded` (default: `"delete"`) |
| `characterDictionary.profileScope` | `"all"`, `"active"` | Apply dictionary to all Yomitan profiles or only the active one |
| `characterDictionary.collapsibleSections.*` | `true`, `false` | Control which dictionary entry sections start expanded |
See the [Character Dictionary](/character-dictionary) page for full details on the character dictionary feature, including name generation, matching, auto-sync lifecycle, and dictionary entry format. Character dictionary sync follows `subtitleStyle.nameMatchEnabled`.
There is no `characterDictionary.enabled` key: character dictionary sync is enabled by `subtitleStyle.nameMatchEnabled`. See the [Character Dictionary](/character-dictionary) page for full details on the character dictionary feature, including name generation, matching, auto-sync lifecycle, and dictionary entry format.
## CLI Commands
+5 -3
View File
@@ -6,12 +6,12 @@ Intro detection runs in the SubMiner app over the mpv IPC socket. It is availabl
## Setup
AniSkip is opt-in. Enable it in your config:
AniSkip is enabled by default. Disable it or change the skip key in your config:
```jsonc
{
"mpv": {
"aniskipEnabled": true,
"aniskipEnabled": true, // default: true
"aniskipButtonKey": "TAB",
},
}
@@ -37,7 +37,9 @@ On each local file load:
4. If an interval is found, SubMiner adds `AniSkip Intro Start` and `AniSkip Intro End` chapter markers to the current file and binds the skip key (`mpv.aniskipButtonKey`, default `TAB`).
5. At the start of the intro, an OSD prompt appears for 3 seconds: `You can skip by pressing TAB` (reflects your configured key). Pressing the key at any point during the intro seeks to the intro end.
Results are cached per file for the app session. Reload detection is also handled: if mpv reloads the same file, SubMiner re-applies the chapter markers without a new API lookup.
When a custom key (other than `TAB` or `y-k`) is configured, the legacy `y-k` chord is also bound as a fallback skip trigger.
Results are cached per file for the app session; only definitive "no intro found" results are cached, so transient lookup failures are retried on the next file load. Reload detection is also handled: if mpv reloads the same file, SubMiner re-applies the chapter markers without a new API lookup.
## Triggering from mpv
+48 -30
View File
@@ -37,7 +37,7 @@ In both modes, the enrichment workflow is the same:
4. Fills the translation field from the secondary subtitle or AI.
5. Writes metadata to the miscInfo field.
Polling mode uses the query `"deck:<ankiConnect.deck>" added:1` to find recently added cards. If no deck is configured, it uses Yomitan's current mining deck when available; otherwise it searches all decks. In Settings, the AnkiConnect deck dropdown auto-fills and persists Yomitan's current mining deck when available, then falls back to the decks reported by AnkiConnect.
Polling mode uses the query `"deck:<ankiConnect.deck>" added:1` to find recently added cards. If no deck is configured, it searches all decks (`added:1`). In Settings, the AnkiConnect deck dropdown auto-fills and persists Yomitan's current mining deck when available, then falls back to the decks reported by AnkiConnect; stats-dashboard mining also falls back to Yomitan's mining deck when `ankiConnect.deck` is empty.
Known-word sync scope is controlled by `ankiConnect.knownWords.decks`.
### Proxy Mode Setup (Yomitan / Texthooker)
@@ -56,12 +56,12 @@ Known-word sync scope is controlled by `ankiConnect.knownWords.decks`.
Then point Yomitan/clients to `http://127.0.0.1:8766` instead of `8765`.
When SubMiner loads the bundled Yomitan extension, it also attempts to update the **default Yomitan profile** (`profiles[0].options.anki.server`) to the active SubMiner endpoint:
When SubMiner loads the bundled Yomitan extension, it also attempts to update the **currently active Yomitan profile**'s Anki server to the active SubMiner endpoint (falling back to `profiles[0]` if the active-profile index is invalid):
- proxy URL when `ankiConnect.proxy.enabled` is `true`
- direct `ankiConnect.url` when proxy mode is disabled
To avoid clobbering custom setups, this auto-update only changes the default profile when its current server is blank or the stock Yomitan default (`http://127.0.0.1:8765`).
To avoid clobbering custom setups, this auto-update only changes the profile when its current server is blank or the stock Yomitan default (`http://127.0.0.1:8765`).
For browser-based Yomitan or other external clients (for example Texthooker in a normal browser profile), set their Anki server to the same proxy URL separately: `http://127.0.0.1:8766` (or your configured `proxy.host` + `proxy.port`).
@@ -81,7 +81,7 @@ In Yomitan, go to Settings → Profile and:
3. Set server to `http://127.0.0.1:8766` (or your configured proxy URL).
4. Save and make that profile active when using SubMiner.
This is only for non-bundled, external/browser Yomitan or other clients. The bundled profile auto-update logic only targets `profiles[0]` when it is blank or still default.
This is only for non-bundled, external/browser Yomitan or other clients. The bundled profile auto-update logic only targets the active profile when its server is blank or still default.
### Proxy Troubleshooting (quick checks)
@@ -101,10 +101,11 @@ curl -sS http://127.0.0.1:8766 \
-d '{"action":"version","version":2}'
```
3. Check both log sinks:
3. Check the log sinks in `~/.config/SubMiner/logs/`:
- Launcher/mpv-integrated log: `~/.cache/SubMiner/mp.log`
- App runtime log: `~/.config/SubMiner/logs/SubMiner-YYYY-MM-DD.log`
- App runtime log: `app-YYYY-MM-DD.log`
- Launcher log: `launcher-YYYY-MM-DD.log`
- mpv log: `mpv-YYYY-MM-DD.log`
4. Ensure config JSONC is valid and logging shape is correct:
@@ -133,7 +134,9 @@ SubMiner maps its data to your Anki note fields. Configure these under `ankiConn
}
```
Field names must match your Anki note type exactly (case-sensitive). If a configured field does not exist on the note type, SubMiner skips it without error.
Field names are matched against your Anki note type case-insensitively (an exact match wins, then a lowercase comparison). If a configured field does not exist on the note type, SubMiner skips it without error.
Two related options live alongside `fields`: `ankiConnect.deck` (target deck; empty falls back as described above) and `ankiConnect.tags` (tags added to mined cards, default `["SubMiner"]`; set `[]` to disable tagging). The `miscInfo` content is controlled by `ankiConnect.metadata.pattern` (default `[SubMiner] %f (%t)`; tokens: `%f` filename, `%F` filename with extension, `%t` timestamp, `%T` timestamp with milliseconds, `<br>` newline).
### Minimal Config
@@ -161,13 +164,17 @@ Audio is extracted from the video file using the subtitle's start and end timest
"ankiConnect": {
"media": {
"generateAudio": true,
"normalizeAudio": true, // normalize generated clip loudness
"mirrorMpvVolume": true, // apply the current mpv volume level
"audioPadding": 0, // optional seconds before and after subtitle timing
"maxMediaDuration": 30 // cap total duration in seconds
}
}
```
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream.
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream. Generated sentence audio is loudness-normalized to -23 LUFS by default during extraction; set `normalizeAudio` to `false` to keep raw source loudness. When subtitle timing is missing, clips fall back to `media.fallbackDuration` seconds (default `3`). Changing these settings applies to the next extraction without restarting SubMiner.
`mirrorMpvVolume` is also enabled by default. Immediately before extracting each playback-overlay card's audio, SubMiner reads mpv's numeric `volume` and applies mpv's cubic software-volume curve after loudness normalization. For example, mpv volume `50` produces `0.5³ = 0.125` gain. Amplified output above mpv volume `100` is limited to a `-1 dBFS` ceiling before MP3 encoding to prevent clipping. It ignores mpv's separate `mute` state. If the volume property is missing, invalid, or unavailable, extraction continues with unity scaling; disabling this option skips the query and volume filter. Changing this setting applies to the next extraction without restarting SubMiner. YouTube cards queued for a background media-cache download retain the volume captured when the card was mined. Stats-dashboard mining does not currently have access to the active mpv property client, so it does not apply mpv volume scaling.
The audio is uploaded to Anki's media folder and inserted as `[sound:audio_<timestamp>.mp3]`.
@@ -182,8 +189,8 @@ A single frame is captured at the current playback position.
"imageType": "static",
"imageFormat": "jpg", // "jpg", "png", or "webp"
"imageQuality": 92, // 1100
"imageMaxWidth": null, // optional, preserves aspect ratio
"imageMaxHeight": null
"imageMaxWidth": 0, // 0 = preserve source resolution
"imageMaxHeight": 0
}
}
```
@@ -199,13 +206,13 @@ Instead of a static screenshot, SubMiner can generate an animated AVIF covering
"imageType": "avif",
"animatedFps": 10,
"animatedMaxWidth": 640,
"animatedMaxHeight": null,
"animatedMaxHeight": 0, // 0 = preserve aspect ratio
"animatedCrf": 35 // 063, lower = better quality
}
}
```
Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`) in your FFmpeg build. Generation timeout is 60 seconds.
Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`) in your FFmpeg build. Generation timeout is 60 seconds. `media.syncAnimatedImageToWordAudio` (default `true`) prepends a frozen first frame matching the existing word-audio duration, so the motion starts together with the sentence audio.
### Behavior Options
@@ -216,6 +223,7 @@ Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`)
"overwriteImage": true, // replace existing image, or append
"mediaInsertMode": "append", // "append" or "prepend" to field content
"autoUpdateNewCards": true, // auto-update when new card detected
"highlightWord": true, // bold the mined word inside the sentence field
"notificationType": "overlay" // "overlay", "system", "both", or "none"
}
}
@@ -265,14 +273,14 @@ The built-in translation request asks for English output by default. Customize t
SubMiner can create standalone sentence cards (without a word/expression) using a separate note type. This is designed for use with [Lapis](https://github.com/donkuri/Lapis) and similar sentence-focused note types.
::: warning Required config
Sentence card creation and audio card marking both require `ankiConnect.isLapis.enabled: true` and a valid `sentenceCardModel` pointing to your Lapis/Kiku note type. Without this, the `Ctrl/Cmd+S` and `Ctrl/Cmd+Shift+A` shortcuts will not create cards.
Sentence card creation and audio card marking require a non-empty `ankiConnect.isLapis.sentenceCardModel` naming a note type that exists in Anki (default: `"Lapis"`). If the model is empty or missing, the `Ctrl/Cmd+S` and `Ctrl/Cmd+Shift+A` shortcuts will not create cards.
:::
```jsonc
"ankiConnect": {
"isLapis": {
"enabled": true,
"sentenceCardModel": "Japanese sentences"
"sentenceCardModel": "Lapis" // default; point at your Lapis/Kiku note type
}
}
```
@@ -299,25 +307,28 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
**Disabled** (`"disabled"`): No duplicate detection. Each card is independent.
**Auto** (`"auto"`): When a duplicate expression is found, SubMiner merges the new card into the existing one automatically. Both sentences, audio clips, and images are preserved, and exact duplicate values are collapsed to one entry. If `deleteDuplicateInAuto` is true, the new card is deleted after merging.
**Auto** (`"auto"`): When a duplicate expression is found, SubMiner merges the new card into the existing one automatically. Both cards' sentences, audio clips, and images are preserved as grouped entries. If `deleteDuplicateInAuto` is true, the new card is deleted after merging.
**Manual** (`"manual"`): A modal appears in the overlay showing both cards. You choose which card to keep, preview the merge result, then confirm. The modal has a 90-second timeout, after which it cancels automatically.
### What Gets Merged
| Field | Merge behavior |
| -------- | --------------------------------------------------------------- |
| Sentence | Both sentences preserved (exact duplicate text is deduplicated) |
| Audio | Both `[sound:...]` entries kept (exact duplicates deduplicated) |
| Image | Both images kept (exact duplicates deduplicated) |
| Field | Merge behavior |
| -------- | ---------------------------------------- |
| Sentence | Both cards' sentences kept as grouped entries |
| Audio | Both cards' `[sound:...]` entries kept |
| Image | Both cards' images kept |
Identical values from both cards are kept as separate grouped entries; the merge does not deduplicate.
### Keyboard Shortcuts in the Modal
| Key | Action |
| --------- | ---------------------------------- |
| `1` / `2` | Select card 1 or card 2 to keep |
| `Enter` | Confirm selection |
| `Esc` | Cancel (keep both cards unchanged) |
| Key | Action |
| ----------- | ---------------------------------- |
| `1` / `2` | Select card 1 or card 2 to keep |
| `Enter` | Confirm selection |
| `Backspace` | Go back from the merge preview |
| `Esc` | Cancel (keep both cards unchanged) |
## Full Config Example
@@ -327,8 +338,10 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
"enabled": true,
"url": "http://127.0.0.1:8765",
"pollingRate": 3000,
"deck": "",
"tags": ["SubMiner"],
"proxy": {
"enabled": false,
"enabled": true, // default
"host": "127.0.0.1",
"port": 8766,
"upstreamUrl": "http://127.0.0.1:8765",
@@ -347,6 +360,8 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
"imageType": "static",
"imageFormat": "jpg",
"imageQuality": 92,
"normalizeAudio": true,
"mirrorMpvVolume": true,
"audioPadding": 0,
"maxMediaDuration": 30,
},
@@ -357,10 +372,13 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
"autoUpdateNewCards": true,
"notificationType": "overlay",
},
"metadata": {
"pattern": "[SubMiner] %f (%t)",
},
"ai": {
"enabled": false,
"model": "openai/gpt-4o-mini",
"systemPrompt": "Translate mined sentence text only.",
"model": "", // e.g. "openai/gpt-4o-mini"
"systemPrompt": "",
},
"isKiku": {
"enabled": false,
@@ -369,7 +387,7 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
},
"isLapis": {
"enabled": false,
"sentenceCardModel": "Japanese sentences",
"sentenceCardModel": "Lapis",
},
},
"ai": {
+23 -14
View File
@@ -6,7 +6,7 @@ SubMiner is split into three cooperating runtimes:
- Electron desktop app (`src/`) for overlay/UI/runtime orchestration.
- Launcher CLI (`launcher/`) for mpv/app command workflows.
- mpv Lua plugin (`plugin/subminer/init.lua` + module files) for player-side controls and IPC handoff.
- mpv Lua plugin (`plugin/subminer/main.lua` + module files) for player-side controls and IPC handoff.
Within the desktop app, `src/main.ts` is a composition root that wires small runtime/domain modules plus core services.
@@ -24,13 +24,14 @@ Within the desktop app, `src/main.ts` is a composition root that wires small run
```text
launcher/ # Standalone CLI launcher wrapper and mpv helpers
commands/ # Command modules (doctor/config/mpv/jellyfin/playback/app passthrough)
commands/ # Command modules (doctor/config/mpv/jellyfin/playback/app passthrough/
# dictionary/history/logs/stats/update)
config/ # Launcher config parsers + CLI parser builder
main.ts # Launcher entrypoint and command dispatch
plugin/
subminer/ # Modular mpv plugin (init · main · bootstrap · lifecycle · process
subminer/ # Modular mpv plugin (main · init · bootstrap · lifecycle · process
# state · messages · hover · ui · options · environment · log
# binary)
# binary · session_bindings · version)
src/
ai/ # AI translation provider utilities (client, config)
main-entry.ts # Background-mode bootstrap wrapper before loading main.js
@@ -38,6 +39,7 @@ src/
preload.ts # Electron preload bridge
types.ts # Shared type definitions
main/ # Main-process composition/runtime adapters
boot/ # Pre-ready boot helpers
app-lifecycle.ts # App lifecycle + app-ready runtime runner factories
character-dictionary-runtime.ts # Character-dictionary orchestration/public runtime API
cli-runtime.ts # CLI command runtime service adapters
@@ -71,17 +73,17 @@ src/
resolve/ # Domain-specific config resolution pipeline stages
shared/ipc/ # Cross-process IPC channel constants + payload validators
renderer/ # Overlay renderer (modularized UI/runtime)
handlers/ # Keyboard/mouse interaction modules
modals/ # Jimaku/Kiku/subsync/runtime-options/session-help modals
handlers/ # Keyboard/mouse/gamepad interaction modules
modals/ # Modal flows (Jimaku, Kiku, subsync, runtime options, session help,
# character dictionary, playlist browser, subtitle sidebar,
# YouTube track picker, controller config/debug/select)
positioning/ # Subtitle position controller (drag-to-reposition)
settings/ # Settings window UI (model, controls, markup)
types/ # Domain type modules (anki, config, integrations, ...)
window-trackers/ # Backend-specific tracker implementations (Hyprland, Sway, X11, macOS, Windows)
jimaku/ # Jimaku API integration helpers
subsync/ # Subtitle sync (alass/ffsubsync) helpers
subtitle/ # Subtitle processing utilities
tokenizers/ # Tokenizer implementations
anki-integration/ # AnkiConnect proxy server + note-update enrichment workflow
token-mergers/ # Token merge strategies
translators/ # AI translation providers
```
### Service Layer (`src/core/services/`)
@@ -92,7 +94,7 @@ src/
- **Mining + Anki/Jimaku runtime:** `mining.ts`, `field-grouping.ts`, `field-grouping-overlay.ts`, `anki-jimaku.ts`, `anki-jimaku-ipc.ts`
- **Subtitle/token pipeline:** `subtitle-processing-controller.ts`, `subtitle-position.ts`, `subtitle-ws.ts`, `tokenizer.ts` + `tokenizer/*` stage modules (including `parser-enrichment-worker-runtime.ts` for async MeCab enrichment and `yomitan-parser-runtime.ts`)
- **Integrations:** `jimaku.ts`, `subsync.ts`, `subsync-runner.ts`, `texthooker.ts`, `jellyfin.ts`, `jellyfin-remote.ts`, `discord-presence.ts`, `yomitan-extension-loader.ts`, `yomitan-settings.ts`
- **Anki integration:** `anki-integration.ts`, `anki-integration/anki-connect-proxy.ts` (local proxy for push-based auto-enrichment), `anki-integration/note-update-workflow.ts`
- **Anki integration (repo `src/` root, not under `core/services/`):** `src/anki-integration.ts`, `src/anki-integration/anki-connect-proxy.ts` (local proxy for push-based auto-enrichment), `src/anki-integration/note-update-workflow.ts`
- **Config/runtime controls:** `config-hot-reload.ts`, `runtime-options-ipc.ts`, `cli-command.ts`, `startup.ts`
- **Domain submodules:** `anilist/*` (token/update queue/updater), `immersion-tracker/*` (storage/session/metadata/query/reducer)
@@ -116,12 +118,19 @@ src/renderer/
handlers/
keyboard.ts # Keybindings, chord handling, modal key routing
mouse.ts # Hover/drag behavior, selection + observer wiring
gamepad-controller.ts # Gamepad/controller input handling
controller-binding-capture.ts # Controller binding capture flow
modals/
jimaku.ts # Jimaku modal flow
kiku.ts # Kiku field-grouping modal flow
runtime-options.ts # Runtime options modal flow
session-help.ts # Keyboard shortcuts/help modal flow
subsync.ts # Manual subsync modal flow
character-dictionary.ts # Character dictionary modal flow
playlist-browser.ts # Playlist browser modal flow
subtitle-sidebar.ts # Subtitle sidebar modal flow
youtube-track-picker.ts # YouTube subtitle track picker
controller-*.ts # Controller config/debug/select modals
utils/
dom.ts # Required DOM lookups + typed handles
platform.ts # Layer/platform capability detection
@@ -130,7 +139,7 @@ src/renderer/
### Launcher + Plugin Runtimes
- `launcher/main.ts` dispatches commands through `launcher/commands/*` and shared config readers in `launcher/config/*`. It handles mpv startup, app passthrough, Jellyfin helper commands, and playback handoff.
- `plugin/subminer/init.lua` runs inside mpv and loads modular Lua files: `main.lua` (orchestration), `bootstrap.lua` (startup), `lifecycle.lua` (connect/disconnect), `process.lua` (process management), `state.lua` (shared state), `messages.lua` (IPC), `hover.lua` (hover-token highlight rendering), `ui.lua` (OSD rendering), `options.lua` (config), `environment.lua` (detection), `log.lua` (logging), `binary.lua` (path resolution). AniSkip intro detection lives in the SubMiner app (`src/main/runtime/aniskip-runtime.ts`), which drives mpv chapters and the skip key over the IPC socket.
- `plugin/subminer/main.lua` is the mpv entrypoint: it sets up the module path and loads `init.lua`, a thin shim that boots the modular Lua files: `bootstrap.lua` (startup), `lifecycle.lua` (connect/disconnect), `process.lua` (process management), `state.lua` (shared state), `messages.lua` (IPC), `hover.lua` (hover-token highlight rendering), `ui.lua` (OSD rendering), `options.lua` (config), `environment.lua` (detection), `log.lua` (logging), `binary.lua` (path resolution), `session_bindings.lua` (configurable session keybindings), `version.lua` (version metadata). AniSkip intro detection lives in the SubMiner app (`src/main/runtime/aniskip-runtime.ts`), which drives mpv chapters and the skip key over the IPC socket.
## Flow Diagram
@@ -313,7 +322,7 @@ The runtime sockets in this flow are detailed in [IPC + Runtime Contracts](./ipc
- **Critical-path init:** Once `app.whenReady()` fires, `composeAppReadyRuntime()` runs strict config reload, resolves keybindings, creates the `MpvIpcClient` (which immediately connects and subscribes to mpv subtitle/playback properties via `observe_property`), and initializes the `RuntimeOptionsManager`, `SubtitleTimingTracker`, and `ImmersionTrackerService`.
- **Overlay runtime:** `initializeOverlayRuntime()` creates the primary overlay window (interactive Yomitan lookups and subtitle rendering), registers global shortcuts, and sets up bounds tracking via the active window tracker. mpv subtitle suppression is handled by a dedicated `overlay-mpv-sub-visibility` service.
- **Background warmups:** Non-critical services are launched asynchronously: MeCab tokenizer check (with async worker thread), Yomitan extension load, JLPT + frequency dictionary prewarm, optional Jellyfin remote session, Discord presence service, AniList token refresh, and optional AnkiConnect proxy server. Warmup coverage is configurable through `startupWarmups` (including low-power mode that defers all but Yomitan).
- **Runtime:** Event-driven. mpv property changes, IPC messages, CLI commands, overlay shortcuts, and hot-reload notifications route through runtime handlers/composers. Subtitle text flows through `SubtitlePipeline` (normalize → tokenize → merge), and results are sent to the main overlay renderer and modal surfaces.
- **Runtime:** Event-driven. mpv property changes, IPC messages, CLI commands, overlay shortcuts, and hot-reload notifications route through runtime handlers/composers. Subtitle text flows through the `SubtitleProcessingController` (normalize → tokenize → merge), and results are sent to the main overlay renderer and modal surfaces.
- **Shutdown:** `onWillQuitCleanup` destroys tray + config watcher, unregisters shortcuts, stops WebSocket + texthooker servers, closes the mpv socket + flushes OSD log, stops the window tracker, closes the Yomitan parser window, flushes the immersion tracker (SQLite), stops Jellyfin/Discord services, stops the AnkiConnect proxy server, and cleans Anki/AniList state.
```mermaid
@@ -380,7 +389,7 @@ flowchart TB
## Subtitle Prefetch Pipeline
SubMiner can pre-tokenize upcoming subtitle lines before they appear on screen. When an external subtitle file (SRT, VTT, or ASS) is detected on the active track, the `SubtitlePrefetchService` parses all cues via the `SubtitleCueParser`, identifies a priority window of upcoming lines based on the current playback position, and tokenizes them in the background through the same pipeline used for live subtitles. Results are stored directly into the `SubtitleProcessingController` cache, so when a subtitle actually appears during playback, it hits a warm cache and renders in ~30-50ms instead of ~200-320ms.
SubMiner can pre-tokenize upcoming subtitle lines before they appear on screen. When an external subtitle file (SRT, VTT, or ASS) is detected on the active track, the `SubtitlePrefetchService` parses all cues via the subtitle cue parser (`subtitle-cue-parser.ts`), identifies a priority window of upcoming lines based on the current playback position, and tokenizes them in the background through the same pipeline used for live subtitles. Results are stored directly into the `SubtitleProcessingController` cache, so when a subtitle actually appears during playback, it hits a warm cache and renders in ~30-50ms instead of ~200-320ms.
The prefetcher yields to live subtitle processing (which always takes priority over background work) and re-computes its priority window on seek. Cache invalidation events (e.g. marking a word as known) trigger re-prefetching of the current window to keep results fresh.
+45 -3
View File
@@ -1,6 +1,48 @@
# Changelog
## v0.17.1 (2026-06-27)
## v0.18.0 (2026-07-10)
**Added**
- Sentence Audio Normalization: Generated sentence audio is now normalized to -23 LUFS by default, and clips mined from playback mirror mpv's software volume curve with a limiter to prevent clipping. Both behaviors are configurable independently.
- Watch History Command: Added `subminer -H` / `--history` to browse watch history, replay or continue episodes, or pick one via fzf or rofi, with cover art shown in the rofi picker.
**Changed**
- Fzf Preview Layout: Moved fzf previews below launcher menus, giving long titles and metadata more room.
- Known-Word Highlighting: Now compares subtitle and Anki-card readings, preventing false matches between homographs and unrelated words that share a reading, while still supporting matching across kana and kanji spellings.
- Annotation Filtering: Standalone suffix tokens (e.g. さん, れる) are now excluded from JLPT/frequency/N+1 highlighting by default, matching how particles and interjections are treated; configurable via the pos2 exclusion setting.
- App Icon: Replaced the app icon with new pixel-art submarine artwork contributed by the community, used across the app icon, tray, notifications, README, docs site, and Stats page.
- Stats Trend Charts: Overhauled with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and value-sorted tooltips.
**Fixed**
- Background Stats Server: `subminer app` background launches now auto-start the stats server when enabled, and skip startup if one is already running.
- Character Name Highlighting: Character dictionaries now split unspaced native names more reliably, and portraits, highlights, and hover lookup survive punctuation, unmatched text, and competing dictionary matches without incorrectly splitting longer words.
- Highlighting Coverage: Frequency/JLPT highlighting and vocabulary stats now include content adverbs (e.g. 確かに, やはり) and kanji nouns MeCab tags as non-independent (e.g. 日, 点, 以外), while still suppressing interjections, pronouns, and grammar fragments; lexicalized kana expressions like かといって keep their annotations.
- Cover Art Fetching: Stats now fetches AniList cover art as soon as a new series starts playing, and backfills missing art for existing series on the next Stats visit.
- Kiku Field Grouping: The manual field-grouping dialog now stays above fullscreen mpv, remains usable across repeated attempts, closes abandoned windows after timeouts, and reports a clear error when the original card can no longer be loaded.
- Karaoke Subtitle Collapse: Secondary subtitles no longer stack dozens of one-syllable lines during openings and endings; repeated events collapse into one line, capped to a strip at the top.
- Unparsed Token Hover: Subtitle text Yomitan can't parse (truncated inflections, elongation runs) remains hoverable, while staying excluded from highlighting, N+1 candidate math, and vocabulary stats.
- YouTube Streaming: Fixed direct YouTube stream extraction that could corrupt signed URLs and cause ffmpeg 403 errors.
<details>
<summary>Internal changes</summary>
**Internal**
- Test lanes moved to `scripts/test-lanes.ts` with per-directory discovery and isolated per-file timeouts; CI now covers previously orphaned stats, scripts, plugin process-retry, and runtime-compat suites, plus a new stats lane in the change-verification workflow.
</details>
## Previous Versions
<details>
<summary>v0.17.x</summary>
<h2>v0.17.2 (2026-06-28)</h2>
**Fixed**
- YouTube Background Cache: Fixed Windows YouTube background media cache startup for YouTube URLs opened directly in mpv, including resolved stream URLs when mpv still exposes the original YouTube playlist entry, so queued Anki media updates can append audio and images after the cache finishes.
- YouTube Subtitle Picker: Manual subtitle picker requests now show an immediate configured notification while SubMiner probes tracks and opens the modal. Subtitle download progress is replaced with a transient success notification after tracks load.
<h2>v0.17.1 (2026-06-27)</h2>
**Added**
- YouTube Media Cache Mode: Adds `youtube.mediaCache.mode` with `direct` and `background` options. Background mode uses a yt-dlp cache download when direct stream extraction is unreliable — creates a text-only card immediately, queues media updates for mined notes, and fills audio/image fields once the download finishes. Progress is announced via overlay/OSD notifications. Downloads are capped at 720p by default (`youtube.mediaCache.maxHeight`). Switching back to direct mode cancels any in-flight background download.
@@ -9,7 +51,7 @@
- Log Export: Fixed log filenames to use the local date so exports around UTC midnight include the current day's logs rather than stale prior-day files. Expanded export redaction to mask IPs, emails, auth and cookie headers, yt-dlp cookie arguments, URL credentials, token/key/password fields, and signed YouTube media URL parameters.
- YouTube Card Media: Improved media generation reliability by sending safer ffmpeg options for resolved streams and skipping stale stream maps (including cached YouTube files). Hardened background cache downloads with IPv4 and extractor retry flags; failed downloads now notify the user and clear queued media updates instead of leaving them silently pending. Stale background cache files are cleaned on startup and before each new download.
## v0.17.0 (2026-06-15)
<h2>v0.17.0 (2026-06-15)</h2>
**Changed**
@@ -44,7 +86,7 @@
</details>
## Previous Versions
</details>
<details>
<summary>v0.16.x</summary>
+10 -6
View File
@@ -56,6 +56,8 @@ A single character produces many searchable terms so that names are recognized r
- Family name alone: 須々木
- Given name alone: 心一
Unspaced native names (AniList often stores 渡辺真奈美 without a separator) are split into family/given parts with MeCab when it is available: person-name POS tags (姓/名) decide the boundary, validated against AniList's romanized first/last name readings. Without MeCab, a length heuristic based on the romanized readings guesses the boundary — and because that guess can be ambiguous (東紫乃 could be 東+紫乃 or 東紫+乃), terms are generated for the top two candidate boundaries so the real surname still matches. Snapshots built without MeCab are regenerated automatically once MeCab becomes available, upgrading them to the exact splits.
**Middle-dot removal** (common in katakana foreign names):
- ア・リ・ス → アリス (combined), plus individual segments
@@ -132,7 +134,7 @@ Each character entry in the Yomitan dictionary includes structured content:
- **Name** - the matched Japanese name form
- **Known names** - generated non-honorific Japanese aliases for that character, excluding raw romanized/English aliases from lookup results
- **Role badge** - color-coded by role: main (score 100), supporting (90), side (80), background (70)
- **Role badge** - color-coded by role: main / "Protagonist" (score 100), primary / "Main Character" (75), side / "Side Character" (50), appears / "Minor Role" (25). AniList's MAIN maps to main, SUPPORTING to primary, and BACKGROUND to side.
- **Portrait** - character image from AniList, embedded in the ZIP
- **Description** - biography text from AniList (collapsible)
- **Character information** - age, birthday, gender, blood type (collapsible)
@@ -164,7 +166,7 @@ These phases are emitted through the configured notification surface. Some phase
1. **checking** - Is there already a cached snapshot for this media ID?
2. **generating** - No cache hit: fetch characters from AniList GraphQL, download portraits (250ms throttle between image requests), save snapshot JSON.
3. **syncing** - Add the media ID to the most-recently-used list. Evict old entries beyond `maxLoaded`.
3. MRU update (no notification) - add the media ID to the most-recently-used list and evict old entries beyond `maxLoaded`.
4. **building** - Merge active snapshots into a single Yomitan ZIP. A SHA-1 revision hash is computed from the media set - if it matches the previously imported revision, the import is skipped.
5. **importing** - Push the ZIP into Yomitan. Waits for Yomitan mutation readiness (7-second timeout per operation).
6. **ready** - Dictionary is live. Character names will match on the next subtitle line.
@@ -173,12 +175,14 @@ These phases are emitted through the configured notification surface. Some phase
```jsonc
{
"activeMediaIds": [170942, 163134, 154587],
"activeMediaIds": ["170942 - Frieren", "163134 - ...", "154587 - ..."],
"mergedRevision": "a1b2c3d4e5f6",
"mergedDictionaryTitle": "SubMiner Character Dictionary",
}
```
(Entries are `"<mediaId> - <title>"` label strings; bare numeric IDs from older versions are still read.)
The `maxLoaded` setting (default: 3) controls how many media snapshots stay in the active set. When you start a 4th title, the oldest is evicted and the merged dictionary is rebuilt without it.
## Manual Generation
@@ -226,7 +230,7 @@ Manual selections are stored in `character-dictionaries/anilist-overrides.json`
Open the manager with `Ctrl/Cmd+D` (`shortcuts.openCharacterDictionaryManager`). The manager shows the merged dictionary's active MRU entries, marks the current anime, and lets you adjust eviction priority for the other loaded entries.
- **Remove** drops a non-current entry from the active merged dictionary and rebuilds/imports once.
- **Up/Down** changes MRU order for future eviction without rebuilding.
- **Up/Down** changes MRU order for future eviction; the merged dictionary is rebuilt and re-imported after a reorder.
- **Override** opens the AniList selector for that entry's title so you can replace a saved loaded entry.
The current anime cannot be removed while you are watching it; it stays loaded until playback changes.
@@ -248,13 +252,13 @@ character-dictionaries/
m170942-va67890.jpg # Voice actor portrait
```
**Snapshot format** (v17): each snapshot contains the media ID, title, entry count, timestamp, an array of Yomitan term entries, and base64-encoded images.
**Snapshot format** (v19, `CHARACTER_DICTIONARY_FORMAT_VERSION`): each snapshot contains the media ID, title, entry count, timestamp, an array of Yomitan term entries, and base64-encoded images. Snapshots with a different format version are regenerated.
**ZIP structure** follows the Yomitan dictionary format:
```text
merged.zip
index.json # { title, revision, format: 3, author: "SubMiner" }
index.json # { title, revision, format: 3, author: "SubMiner", description }
tag_bank_1.json # Tag definitions
term_bank_1.json # Up to 10,000 terms per bank
term_bank_2.json
+37 -30
View File
@@ -46,12 +46,13 @@ The Settings window groups options by workflow instead of mirroring the raw conf
- Appearance
- Behavior
- Mining & Anki
- Playback & Sources
- Input
- Integrations
- Tracking & App
- Advanced
Playback-related fields live as sections inside these groups (for example "Playback Behavior" under **Behavior** and "mpv Playback" / "YouTube Playback Settings" under **Integrations**).
Each field still writes to its current `config.jsonc` path. For example, subtitle hover pause appears under **Behavior** / playback behavior, but saves to `subtitleStyle.autoPauseVideoOnHover`. Anki-aware fields can query AnkiConnect for deck names, note types, and field names. The AnkiConnect deck field also reads Yomitan's current mining deck and persists it into an empty setting when one is found. Stats mining also uses Yomitan's current mining deck when `ankiConnect.deck` is empty. Keybinding fields use click-to-learn controls instead of raw text boxes.
The Settings window preserves existing JSONC comments, trailing commas, and unrelated keys. Resetting a field removes the explicit config path so the built-in default applies.
@@ -95,9 +96,11 @@ On macOS, these validation warnings also open a native dialog with full details
SubMiner watches the active config file (`config.jsonc` or `config.json`) while running and applies supported updates automatically.
Hot-reloadable settings include subtitle appearance, sidebar controls, keybindings,
logging level, selected source-language preferences, Jimaku/Subsync settings, and
the Anki deck, known-word, N+1, field, sentence-card, and Kiku options listed
in the reference tables below.
shortcuts, notifications, logging level, selected source-language preferences,
Jimaku/Subsync settings, AniSkip settings (`mpv.aniskipEnabled`, `mpv.aniskipButtonKey`),
stats keys (`stats.toggleKey`, `stats.markWatchedKey`), the secondary-subtitle default
mode, and the Anki deck, known-word, N+1, field, sentence-card, AI, and Kiku options
listed in the reference tables below.
When these values change, SubMiner applies them live. Invalid config edits are rejected and the previous valid runtime config remains active.
@@ -482,6 +485,8 @@ Configure the parsed-subtitle sidebar modal.
| `autoScroll` | boolean | Keep the active cue in view while playback advances |
| `subtitleSidebar.css` | object | CSS declaration object applied to the sidebar. Use CSS properties plus sidebar custom properties below. |
Direct style keys are also available under `subtitleSidebar` and map to the same visuals as the CSS custom properties: `maxWidth` (default `420`), `opacity` (`0.95`), `backgroundColor`, `textColor`, `fontFamily`, `fontSize` (`16`), `timestampColor`, `activeLineColor`, `activeLineBackgroundColor`, and `hoverLineBackgroundColor`.
Sidebar CSS custom properties:
| CSS Property | Default | Description |
@@ -498,7 +503,7 @@ The sidebar is only available when the active subtitle source has been parsed in
For full details on layout modes, behavior, and the keyboard shortcut, see the [Subtitle Sidebar](/subtitle-sidebar) page.
`jlptColors` keys are:
`subtitleStyle.jlptColors` keys are:
| Key | Default | Description |
| ---- | --------- | ----------------------- |
@@ -508,12 +513,6 @@ For full details on layout modes, behavior, and the keyboard shortcut, see the [
| `N4` | `#8bd5ca` | JLPT N4 underline color |
| `N5` | `#8aadf4` | JLPT N5 underline color |
**Image Quality Notes:**
- `imageQuality` affects JPG and WebP only; PNG is lossless and ignores this setting
- JPG quality is mapped to FFmpeg's scale (2-31, lower = better)
- WebP quality uses FFmpeg's native 0-100 scale
### Subtitle Position
Set the initial vertical subtitle position (measured from the bottom of the screen):
@@ -797,7 +796,7 @@ If you bind a discrete action to an axis manually, include `direction`:
Treat the button-index map as reference-only unless you are copying values from the debug modal. Updating it alone does not rewrite the hardcoded raw numeric values already present in controller bindings or controller profiles. If you need a real remap, prefer the `Alt+C` learn flow so both the source and the descriptor shape stay correct.
If you choose to bind `L2` or `R2` manually, set `triggerInputMode` to `analog` and tune `triggerDeadzone` when your controller reports triggers as analog values instead of digital pressed/not-pressed buttons. `auto` accepts either style and remains the default.
If you choose to bind `L2` or `R2` manually, set `triggerInputMode` to `analog` and tune `triggerDeadzone` when your controller reports triggers as analog values instead of digital pressed/not-pressed buttons. `digital` forces pressed/not-pressed handling; `auto` accepts either style and remains the default.
If one controller reports non-standard raw button numbers, override that controller profile's button-index map using values from the `Alt+Shift+C` debug modal. Use the global button-index map only when the mapping should apply to every controller without a profile.
@@ -834,7 +833,7 @@ These shortcuts are only active when the overlay window is visible and automatic
### Session Help Modal
The session help modal opens from the overlay with `Ctrl/Cmd+/` by default. The mpv plugin also exposes it through the `Y-H` chord (falling back to `Y-K` if needed). It shows the current session keybindings and color legend.
The session help modal opens from the overlay with `Ctrl/Cmd+/` by default. The mpv plugin also exposes it through the `y-h` chord. It shows the current session keybindings and color legend.
You can filter the modal quickly with `/`:
@@ -861,8 +860,8 @@ When config hot-reload updates shortcut/keybinding/style values, close and reope
Use the runtime options palette to toggle settings live while SubMiner is running. These changes are session-only and reset on restart.
Current runtime options cover automatic card updates, known-word highlighting,
JLPT underlines, frequency highlighting, known-word match mode, and Kiku field
grouping mode.
N+1 annotation, JLPT underlines, frequency highlighting, known-word match mode,
and Kiku field grouping mode.
Annotation toggles only apply to new subtitle lines after the toggle. The currently displayed line is not re-tokenized in place.
@@ -879,7 +878,7 @@ Palette controls:
### Shared AI Provider
This is the single, shared connection to an OpenAI-compatible LLM endpoint. Configure it **once** here at the top level, and SubMiner reuses it wherever AI is needed (today: Anki translation/enrichment). Per-feature toggles and prompt/model tweaks live in their own sections (for example `ankiConnect.ai`) and inherit this transport.
This is the single, shared connection to an OpenAI-compatible LLM endpoint. Configure it **once** here at the top level, and SubMiner reuses it wherever AI is needed (Anki translation/enrichment and YouTube subtitle fixing). Per-feature toggles and prompt/model tweaks live in their own sections (for example `ankiConnect.ai` and `youtubeSubgen.ai`) and inherit this transport.
```json
{
@@ -907,6 +906,7 @@ This is the single, shared connection to an OpenAI-compatible LLM endpoint. Conf
SubMiner uses the shared provider for:
- Anki translation/enrichment when Anki AI is enabled
- YouTube generated-subtitle fixing when `youtubeSubgen.fixWithAi` is enabled (with optional `youtubeSubgen.ai.model` / `systemPrompt` overrides)
### AnkiConnect
@@ -951,6 +951,8 @@ Enable automatic Anki card creation and updates with media generation:
"animatedMaxWidth": 640,
"animatedMaxHeight": 0,
"animatedCrf": 35,
"normalizeAudio": true,
"mirrorMpvVolume": true,
"audioPadding": 0,
"fallbackDuration": 3,
"maxMediaDuration": 30
@@ -1001,10 +1003,12 @@ This example is intentionally compact. The option table below documents availabl
| `ankiConnect.ai.model` | string | Optional model override for Anki AI translation/enrichment flows. |
| `ankiConnect.ai.systemPrompt` | string | Optional system prompt override for Anki AI translation/enrichment flows. |
| `media.generateAudio` | `true`, `false` | Generate audio clips from video (default: `true`) |
| `media.normalizeAudio` | `true`, `false` | Normalize generated sentence-audio loudness during media extraction (default: `true`). Set to `false` to keep raw source loudness. Changes apply live. |
| `media.mirrorMpvVolume` | `true`, `false` | Apply mpv's cubic software-volume curve to each generated sentence-audio clip (default: `true`). This ignores mpv's separate mute state, falls back to unity scaling if volume cannot be read, and applies changes live. |
| `media.generateImage` | `true`, `false` | Generate image/animation screenshots (default: `true`) |
| `media.imageType` | `"static"`, `"avif"` | Image type: static screenshot or animated AVIF (default: `"static"`) |
| `media.imageFormat` | `"jpg"`, `"png"`, `"webp"` | Image format (default: `"jpg"`) |
| `media.imageQuality` | number (1-100) | Image quality for JPG/WebP; PNG ignores this (default: `92`) |
| `media.imageQuality` | number (1-100) | Image quality for JPG/WebP; PNG ignores this (default: `92`). JPG values are mapped onto FFmpeg's 2-31 quality scale; WebP uses the value directly. |
| `media.imageMaxWidth` | number (px) | Optional max width for static screenshots. Unset keeps source width. |
| `media.imageMaxHeight` | number (px) | Optional max height for static screenshots. Unset keeps source height. |
| `media.animatedFps` | number (1-60) | FPS for animated AVIF (default: `10`) |
@@ -1028,7 +1032,7 @@ This example is intentionally compact. The option table below documents availabl
| `ankiConnect.nPlusOne.minSentenceWords` | number | Minimum number of words required in a sentence before single unknown-word N+1 highlighting can trigger (default: `3`). |
| `behavior.notificationType` | `"overlay"`, `"system"`, `"both"`, `"none"` | Notification type on card update (default: `"overlay"`). `"both"` means overlay + system. `osd` and `osd-system` are legacy config-file-only values; use `"osd-system"` to keep the old OSD + system behavior. |
| `behavior.autoUpdateNewCards` | `true`, `false` | Automatically update cards on creation (default: `true`) |
| `metadata.pattern` | string | Format pattern for metadata: `%f`=filename, `%F`=filename+ext, `%t`=time |
| `metadata.pattern` | string | Format pattern for metadata: `%f`=filename, `%F`=filename+ext, `%t`=time, `%T`=time with milliseconds, `<br>`=newline |
| `isLapis` | object | Lapis/shared sentence-card config: `{ enabled, sentenceCardModel }`. Sentence/audio field names are fixed to `Sentence` and `SentenceAudio`. |
| `isKiku` | object | Kiku-only config: `{ enabled, fieldGrouping, deleteDuplicateInAuto }` (shared sentence/audio/model settings are inherited from `isLapis`) |
@@ -1132,8 +1136,6 @@ Configure Jimaku API access and defaults:
Jimaku is rate limited; if you hit a limit, SubMiner will surface the retry delay from the API response.
Set `openBrowser` to `false` to only print the URL without opening a browser.
### Subtitle Sync
Sync the active subtitle track from the overlay picker using `alass` or `ffsubsync`. Both are **optional external tools** that must be installed separately and available on your `PATH` (or configured via the path options below).
@@ -1154,8 +1156,8 @@ Sync the active subtitle track from the overlay picker using `alass` or `ffsubsy
| Option | Values | Description |
| ---------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `alass_path` | string path | Path to `alass` executable. Empty or `null` resolves from `PATH`. `alass` must be installed separately. |
| `ffsubsync_path` | string path | Path to `ffsubsync` executable. Empty or `null` resolves from `PATH`. `ffsubsync` must be installed separately. |
| `alass_path` | string path | Path to `alass` executable. Empty falls back to `/usr/bin/alass`. `alass` must be installed separately. |
| `ffsubsync_path` | string path | Path to `ffsubsync` executable. Empty falls back to `/usr/bin/ffsubsync`. `ffsubsync` must be installed separately. |
| `ffmpeg_path` | string path | Path to `ffmpeg` (used for internal subtitle extraction). Empty or `null` falls back to `/usr/bin/ffmpeg`. |
| `replace` | `true`, `false` | When `true` (default), overwrite the active subtitle file on successful sync. When `false`, write `<name>_retimed.<ext>`. |
@@ -1191,6 +1193,8 @@ AniList integration is opt-in and disabled by default. Enable it to allow SubMin
| `anilist.enabled` | `true`, `false` | Enable AniList post-watch progress updates (default: `false`) |
| `accessToken` | string | Optional explicit AniList access token override (default: empty string) |
| `characterDictionary.maxLoaded` | number | Maximum number of most-recently-used AniList media snapshots included in the merged dictionary (default: `3`) |
| `characterDictionary.refreshTtlHours` | number | Hours before a cached media snapshot is refreshed (default: `168`, clamped to 18760) |
| `characterDictionary.evictionPolicy` | `"delete"`, `"disable"` | What happens to snapshots evicted beyond `maxLoaded` (default: `"delete"`) |
| `characterDictionary.collapsibleSections.description` | `true`, `false` | Open the Description section by default in generated dictionary entries |
| `characterDictionary.collapsibleSections.characterInformation` | `true`, `false` | Open the Character Information section by default in generated dictionary entries |
| `characterDictionary.collapsibleSections.voicedBy` | `true`, `false` | Open the Voiced by section by default in generated dictionary entries |
@@ -1500,7 +1504,7 @@ Configure the mpv executable, profile, and window state for SubMiner-managed mpv
| `executablePath` | string | Absolute path to `mpv.exe` for Windows launch flows. Leave empty to auto-discover from `SUBMINER_MPV_PATH` or `PATH` (default `""`) |
| `profile` | string | mpv profile name passed as `--profile=<name>`. Leave empty to pass no profile (default `""`) |
| `launchMode` | `"normal"` \| `"maximized"` \| `"fullscreen"` | Window state when SubMiner spawns mpv (default `"normal"`) |
| `socketPath` | string | mpv IPC socket path used by SubMiner-managed playback and the bundled mpv plugin (default: `\\\\.\\pipe\\subminer-socket`) |
| `socketPath` | string | mpv IPC socket path used by SubMiner-managed playback and the bundled mpv plugin (platform-dependent default: `/tmp/subminer-socket`, or `\\\\.\\pipe\\subminer-socket` on Windows) |
| `backend` | `"auto"` \| `"hyprland"` \| `"sway"` \| `"x11"` \| `"macos"` \| `"windows"` | Window tracking backend passed to the bundled mpv plugin. Auto detects the current platform (default: `"auto"`) |
| `autoStartSubMiner` | `true`, `false` | Start SubMiner in the background when SubMiner-managed mpv loads a file (default: `true`) |
| `pauseUntilOverlayReady` | `true`, `false` | Pause mpv on visible-overlay auto-start until SubMiner signals subtitle tokenization readiness, with a 30-second fallback (default: `true`) |
@@ -1538,7 +1542,7 @@ Set defaults used by managed subtitle auto-selection and the `subminer` launcher
| `mediaCache.mode` | `direct` \| `background` | YouTube card audio/image extraction mode (default `direct`) |
| `mediaCache.maxHeight` | number | Maximum background cache download height. Set `0` for unlimited (default `720`) |
`mediaCache.mode: "direct"` extracts card media from the active YouTube stream URL. `mediaCache.mode: "background"` starts a separate yt-dlp media download after YouTube playback has loaded. Playback and subtitle loading do not wait for that download. Use background mode if direct card media generation hits YouTube `403` errors from expiring stream URLs.
`mediaCache.mode: "direct"` extracts card media from the active YouTube stream URL. `mediaCache.mode: "background"` starts a separate yt-dlp media download after YouTube playback has loaded, including YouTube URLs opened directly in mpv and resolved stream URLs when mpv still exposes the original YouTube playlist entry. Playback and subtitle loading do not wait for that download. Use background mode if direct card media generation hits YouTube `403` errors from expiring stream URLs.
Background cache downloads are capped by `mediaCache.maxHeight`, which defaults to 720p; set it to `0` to let yt-dlp choose the best available height. Downloads use IPv4 and yt-dlp retry flags to reduce YouTube throttling failures. SubMiner announces when the background cache download starts and when the cache is ready, using the configured notification surface; overlay and OSD messages queue until the overlay or mpv is ready. If you mine cards before the cache is ready, SubMiner creates the text fields immediately, queues the audio/image work for those note IDs, shows a status notification, and fills the media fields once the cached file is ready. If the cache download fails, SubMiner shows a failure notification, shows queued-card failure notifications, and clears the pending updates.
@@ -1548,15 +1552,18 @@ Current launcher behavior:
- If YouTube/mpv already exposes an authoritative matching subtitle track, SubMiner reuses it; otherwise it downloads and injects only the missing side.
- SubMiner loads the primary subtitle plus a best-effort secondary subtitle.
- Playback waits only for primary subtitle readiness; secondary failures do not block playback.
- English secondary subtitles are selected from the secondary-subtitle language list when primary language matches are unavailable.
- Native mpv secondary subtitle rendering stays hidden during this flow so the SubMiner overlay remains the visible secondary subtitle surface.
- If primary subtitle loading fails, use `Ctrl+Alt+C` to open the subtitle modal and pick a track.
Language targets are derived from subtitle config:
Track selection:
- primary track: `youtube.primarySubLanguages` (falls back to `["ja","jpn"]`)
- secondary track: secondary-subtitle language list (falls back to English when empty)
- Local playback uses the same priorities after mpv reports subtitle track metadata, so sidecar/internal mixed sets can override an incorrect initial `sid=auto` pick.
- YouTube auto-selection always targets a Japanese primary track and an English secondary track, preferring manual uploads over auto-generated captions.
- `youtube.primarySubLanguages` (default `["ja","jpn"]`) defines which loaded track counts as a satisfactory primary for the "primary subtitle missing" notification and for managed local/playlist subtitle selection.
- Local playback applies these priorities after mpv reports subtitle track metadata, so sidecar/internal mixed sets can override an incorrect initial `sid=auto` pick.
- Tracks are resolved and loaded before mpv starts; the older launcher mode switch has been removed.
Precedence for launcher defaults is: CLI flag > environment variable > `config.jsonc` > built-in default.
These settings come from `config.jsonc` (or built-in defaults); there are no CLI flags or environment variables for subtitle language selection.
#### YouTube Subtitle Generation (`youtubeSubgen`)
An advanced, template-hidden section for Whisper-based YouTube subtitle generation: `whisperBin`, `whisperModel`, `whisperVadModel`, `whisperThreads` (default `4`), and `fixWithAi` (default `false`), which post-processes generated subtitles through the [Shared AI Provider](#shared-ai-provider) with optional `youtubeSubgen.ai.model` / `systemPrompt` overrides. These keys are accepted in `config.jsonc` but intentionally omitted from the generated template.
+16 -17
View File
@@ -5,6 +5,8 @@ For internal architecture/workflow guidance, use `docs/README.md` at the repo ro
## Prerequisites
- [Bun](https://bun.sh)
- A system `lua` interpreter for `bun run test:launcher` / `bun run test:plugin:src`
- macOS builds compile a Swift helper via `scripts/build-macos-helper.sh` (skip with `SUBMINER_SKIP_MACOS_HELPER_BUILD=1`)
## Setup
@@ -80,18 +82,24 @@ Default lanes:
```bash
bun run test # alias for test:fast
bun run test:fast # default fast lane
bun run test:full # maintained source + launcher-unit + runtime compat surface
bun run test:fast # full source lanes: src + launcher-unit + scripts + runtime compat
bun run test:runtime:compat # compiled/runtime compatibility slice only
bun run test:env # launcher/plugin + env-sensitive verification
bun run test:stats # stats dashboard UI suite
bun run test:immersion:sqlite # SQLite persistence lane
bun run test:subtitle # maintained alass/ffsubsync subtitle surface
```
- `bun run test` and `bun run test:fast` cover config/core suites plus representative entry/runtime, Anki integration, release-workflow coverage, typecheck, and runtime-registry checks.
- `bun run test:full` is the maintained full surface: Bun-compatible `src/**` discovery, Bun-compatible launcher unit discovery, and the compiled/runtime compatibility lane for suites routed through `dist/**`.
Test lane membership is defined once in `scripts/test-lanes.ts` and discovered by
directory, so new test files join their lane automatically. `scripts/run-test-lane.mjs`
runs each test file in its own `bun test` process (per-file isolation) so a hanging
test or leaked global in one file cannot cascade into the rest of the lane; pass
`--jobs N` to parallelize or `--single-process` for one shared process.
- `bun run test` and `bun run test:fast` cover the full discovered `src/**` suite, launcher unit tests, `scripts/**` tests, and the compiled/runtime compatibility lane.
- `bun run test:runtime:compat` covers the compiled/runtime slice directly: `ipc`, `anki-jimaku-ipc`, `overlay-manager`, `config-validation`, `startup-config`, and `registry`.
- `bun run test:env` covers environment-sensitive checks: launcher smoke/plugin verification plus the Bun source SQLite lane.
- `bun run test:stats` runs the stats dashboard suite under `stats/src/**`.
- `bun run test:immersion:sqlite` is the reproducible persistence lane when you need real DB-backed SQLite coverage under Bun.
The Bun-managed discovery lanes intentionally exclude a small compiled/runtime-focused set: `src/core/services/ipc.test.ts`, `src/core/services/anki-jimaku-ipc.test.ts`, `src/core/services/overlay-manager.test.ts`, `src/main/config-validation.test.ts`, `src/main/runtime/startup-config.test.ts`, and `src/main/runtime/registry.test.ts`. `bun run test:runtime:compat` keeps them in the standard workflow via `dist/**`.
@@ -126,11 +134,11 @@ Focused commands:
```bash
bun run test:config # Source-level config schema/validation tests
bun run test:launcher # Launcher regression tests (config discovery + command routing)
bun run test:core # Source-level core regression tests (default lane)
bun run test:launcher:smoke:src # Launcher e2e smoke: launcher -> mpv IPC -> overlay start/stop wiring
bun run test:launcher:env:src # Launcher smoke + Lua plugin gate
bun run test:src # Bun-managed maintained src/** discovery lane
bun run test:launcher:unit:src # Bun-managed maintained launcher unit lane
bun run test:scripts # Bun-managed scripts/** test lane
bun run test:immersion:sqlite:src # Bun source lane
```
@@ -144,8 +152,6 @@ Smoke and optional deep dist commands:
bun run build # compile dist artifacts
bun run test:immersion:sqlite # compile + run SQLite-backed immersion tests under Bun
bun run test:smoke:dist # explicit smoke scope for compiled runtime
bun run test:config:dist # optional full dist config suite
bun run test:core:dist # optional full dist core suite
```
Use `bun run test:immersion:sqlite` when you need real DB-backed coverage for the immersion tracker.
@@ -159,7 +165,7 @@ make pretty
bun run format:check:src
```
- `make pretty` runs the maintained Prettier allowlist only (`format:src`).
- `make pretty` runs the maintained Prettier allowlists (`format:src` and `format:stats`).
- `bun run format:check:src` checks the same scoped set without writing changes.
- `bun run format` remains the broad repo-wide Prettier command; use it intentionally.
@@ -192,15 +198,7 @@ bun run docs:preview # Preview built site at http://localhost:4173
bun run docs:test # Docs regression tests
```
Cloudflare Pages deploy settings:
- Git repo: `ksyasuda/SubMiner`
- Root directory: `docs-site`
- Build command: `bun run docs:build`
- Build output directory: `.vitepress/dist`
- Build watch paths: `docs-site/*`
Use Cloudflare's single `*` wildcard syntax for watch paths. `docs-site/*` covers nested docs-site changes in the repo; `docs-site/**` is not the correct Pages pattern and may skip docs-only pushes.
Deployment: production docs are built with `bun run docs:build:versioned` and uploaded directly to Cloudflare Pages by the `docs-pages` GitHub Actions workflow using Wrangler (from `.tmp/docs-versioned-site`). Cloudflare's automatic Git-integration deployments are intentionally disabled - see `docs-site/README.md` for the deployment contract. Do not re-enable Pages build settings in the Cloudflare dashboard.
## Makefile Reference
@@ -237,6 +235,7 @@ Run `make help` for a full list of targets. Key ones:
| `SUBMINER_APPIMAGE_PATH` | Override SubMiner app binary path for launcher playback commands |
| `SUBMINER_BINARY_PATH` | Alias for `SUBMINER_APPIMAGE_PATH` |
| `SUBMINER_ROFI_THEME` | Override rofi theme path for launcher picker |
| `SUBMINER_MPV_PLUGIN_PATH` | Override the mpv plugin directory injected by the launcher |
| `SUBMINER_LOG_LEVEL` | Override app logger level (`debug`, `info`, `warn`, `error`) |
| `SUBMINER_MPV_LOG` | Override mpv/app shared log file path |
| `SUBMINER_JIMAKU_API_KEY` | Override Jimaku API key for launcher subtitle downloads |
+20 -13
View File
@@ -25,15 +25,16 @@ Episode completion for local `watched` state uses the shared `DEFAULT_MIN_WATCH_
- Leave `dbPath` empty to use the default location (`immersion.sqlite` in SubMiner's app-data directory).
- Set an explicit path to move the database (useful for backups, cloud syncing, or external tools).
- To share stats and watch history between two machines, use [`subminer sync <host>`](/launcher-script#sync-between-machines) instead of file-level cloud sync — it merges both databases without one side overwriting the other.
## Stats Dashboard
The same immersion data powers the stats dashboard.
- In-app overlay: focus the visible overlay, then press the key from `stats.toggleKey` (default: `` ` `` / `Backquote`).
- Launcher command: run `subminer stats` to start the local stats server on demand and open the dashboard in your browser.
- Launcher command: run `subminer stats` to start the local stats server on demand (it also opens the dashboard in your browser when `stats.autoOpenBrowser` is enabled; the default is `false`).
- Background server: run `subminer stats -b` to start or reuse a dedicated background stats daemon without keeping the launcher attached, and `subminer stats -s` to stop that daemon.
- Maintenance command: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand.
- Maintenance commands: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand; `subminer stats cleanup -l` repairs lifetime summary tables. `subminer stats rebuild` and `subminer stats backfill` rebuild or backfill rollup data.
- Browser page: open `http://127.0.0.1:6969` directly if the local stats server is already running.
### Dashboard Tabs
@@ -86,6 +87,7 @@ Stats server config lives under `stats`:
{
"stats": {
"toggleKey": "Backquote",
"markWatchedKey": "KeyW",
"serverPort": 6969,
"autoStartServer": true,
"autoOpenBrowser": false,
@@ -94,8 +96,9 @@ Stats server config lives under `stats`:
```
- `toggleKey` is overlay-local, not a system-wide shortcut.
- `markWatchedKey` toggles the watched state of the highlighted entry inside the stats dashboard.
- `serverPort` controls the localhost dashboard URL.
- `autoStartServer` starts the local stats HTTP server on launch once immersion tracking is active, or reuses the dedicated background stats server when one is already running.
- `autoStartServer` starts the local stats HTTP server on launch once immersion tracking is active, or reuses the dedicated background stats server when one is already running. Background app launches (`subminer app`) start the stats server immediately, registering it so later launches reuse it instead of starting another one.
- `autoOpenBrowser` controls whether `subminer stats` launches the dashboard URL in your browser after ensuring the server is running.
- `subminer stats` forces the dashboard server to start even when `autoStartServer` is `false`.
- `subminer stats -b` starts or reuses the dedicated background stats daemon and exits after startup acknowledgement.
@@ -185,7 +188,6 @@ SELECT
total_watched_ms,
active_watched_ms,
lines_seen,
words_seen,
tokens_seen,
cards_mined
FROM imm_session_telemetry
@@ -203,13 +205,13 @@ SELECT
s.started_at_ms,
s.ended_at_ms,
COALESCE(s.active_watched_ms, 0) AS active_watched_ms,
COALESCE(s.words_seen, 0) AS words_seen,
COALESCE(s.tokens_seen, 0) AS tokens_seen,
COALESCE(s.cards_mined, 0) AS cards_mined,
CASE
WHEN COALESCE(s.active_watched_ms, 0) > 0
THEN COALESCE(s.words_seen, 0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
THEN COALESCE(s.tokens_seen, 0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
ELSE NULL
END AS words_per_min,
END AS tokens_per_min,
CASE
WHEN COALESCE(s.active_watched_ms, 0) > 0
THEN (COALESCE(s.cards_mined, 0) * 60.0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
@@ -229,7 +231,7 @@ SELECT
la.total_sessions,
la.total_active_ms,
la.total_cards,
la.total_words_seen,
la.total_tokens_seen,
la.total_lines_seen,
la.first_watched_ms,
la.last_watched_ms
@@ -248,11 +250,10 @@ SELECT
total_sessions,
total_active_min,
total_lines_seen,
total_words_seen,
total_tokens_seen,
total_cards,
cards_per_hour,
words_per_min,
tokens_per_min,
lookup_hit_rate
FROM imm_daily_rollups
ORDER BY rollup_day DESC, video_id DESC
@@ -268,7 +269,6 @@ SELECT
total_sessions,
total_active_min,
total_lines_seen,
total_words_seen,
total_tokens_seen,
total_cards
FROM imm_monthly_rollups
@@ -287,15 +287,19 @@ LIMIT ?;
- Large-table reads are index-backed for `sample_ms`, session time windows, frequency-ranked words/kanji, and cover-art identity lookups.
- Workload-dependent tuning knobs remain at defaults unless you change them: `cache_size`, `mmap_size`, `temp_store`, `auto_vacuum`.
### Schema (v4)
### Schema (v18)
The exact schema version lives in `SCHEMA_VERSION` (`src/core/services/immersion-tracker/types.ts`) and is recorded in the `imm_schema_version` table.
Core tables:
- `imm_videos` - video key/title/source metadata
- `imm_anime` - anime/series metadata referenced by videos and lifetime tables
- `imm_sessions` - session UUID, video reference, timing/status, final denormalized totals
- `imm_session_telemetry` - high-frequency session aggregates over time
- `imm_session_events` - event stream with compact numeric event types
- `imm_subtitle_lines` - persisted subtitle text and timing per session/video
- `imm_youtube_videos` - YouTube video/channel metadata for tracked videos
Lifetime summary tables:
@@ -308,11 +312,14 @@ Rollup tables:
- `imm_daily_rollups`
- `imm_monthly_rollups`
- `imm_rollup_state` - incremental rollup progress bookkeeping
Vocabulary tables:
- `imm_words(id, headword, word, reading, first_seen, last_seen, frequency)`
- `imm_words(id, headword, word, reading, part_of_speech, pos1, pos2, pos3, first_seen, last_seen, frequency, frequency_rank)` with `UNIQUE(headword, word, reading)`
- `imm_kanji(id, kanji, first_seen, last_seen, frequency)`
- `imm_word_line_occurrences` / `imm_kanji_line_occurrences` - word/kanji ↔ subtitle-line occurrence links
- `imm_stats_excluded_words` - vocabulary exclusion list managed from the dashboard
Media-art tables:
+2 -1
View File
@@ -79,6 +79,7 @@ SubMiner extracts media info from the current video path to pre-fill the search
- **Season + episode patterns:** `S01E03`, `1x03`
- **Episode-only patterns:** `E03`, `EP03`, or dash-separated numbers like `Title - 03 -`
- **Season folders:** a parent directory named `Season 2` or `S2` fills in the season when the filename lacks one
- **Bracket tags:** `[SubGroup]`, `[1080p]`, `[HEVC]` - stripped before title extraction
- **Year tags:** `(2024)` - stripped
- **Dots and underscores:** treated as spaces
@@ -94,7 +95,7 @@ Configure `jimaku.apiKey` or `jimaku.apiKeyCommand` in your config. If using `ap
**"Jimaku request failed" or HTTP 429**
The Jimaku API has rate limits. If you see 429 errors, wait for the retry duration shown in the OSD message and try again. An API key provides higher rate limits.
The Jimaku API has rate limits. If you see 429 errors, wait for the retry duration shown in the OSD message and try again.
**No entries found**
+66 -16
View File
@@ -54,6 +54,7 @@ The theme is auto-detected from these paths (first match wins):
- `/usr/local/share/SubMiner/themes/subminer.rasi`
- `/usr/share/SubMiner/themes/subminer.rasi`
- macOS: `~/Library/Application Support/SubMiner/themes/subminer.rasi`
- `assets/themes/subminer.rasi` next to the launcher script (final fallback)
Override with the `SUBMINER_ROFI_THEME` environment variable:
@@ -61,10 +62,54 @@ Override with the `SUBMINER_ROFI_THEME` environment variable:
SUBMINER_ROFI_THEME=/path/to/custom-theme.rasi subminer -R
```
## Watch History
`subminer -H` (or `--history`) browses your local watch history, sourced from the immersion tracker database. It works with both pickers: fzf by default, rofi with `-R -H`.
```bash
subminer -H # fzf history browser
subminer -R -H # rofi history browser
```
The first menu lists every locally watched series, most recently watched first, using the parsed media title (e.g. the anime title) when available and the directory name otherwise. Selecting a series opens an action menu:
- **Replay last watched** — replays the most recently watched episode
- **Next episode** — plays the episode after the last watched one (continues into the next season directory when the season ends)
- **Browse episodes** — lists the video files in the series directory in episode order, using the same fzf/rofi episode picker as directory browsing; if the series has multiple season directories, a season menu is shown first
Series whose directories are not currently accessible (e.g. an unmounted network share) are hidden from the list. Watch history requires the immersion tracker database (`immersionTracking.dbPath`, default `<config dir>/immersion.sqlite`), which SubMiner populates during playback.
## Sync Between Machines
`subminer sync <host>` merges immersion stats and watch history between two machines over SSH, so both end up with the union of sessions, lifetime totals, vocabulary counts, daily/monthly charts, and `--history` entries. `<host>` is anything `ssh` accepts (`user@hostname` or an ssh config alias); SubMiner must be installed on both machines at the same version.
```bash
subminer sync macbook # two-way sync with the host "macbook"
subminer sync macbook --push # merge local data into macbook only
subminer sync macbook --pull # merge macbook data into local only
subminer sync user@192.168.1.20 # explicit user@host
subminer sync macbook --remote-cmd ~/bin/subminer # custom remote launcher path
```
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over `scp`, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time — syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
For a one-way transfer, `--push` snapshots the local database and merges it into the host without changing the local database. `--pull` snapshots the host and merges it into the local database without changing the host. These modes add missing data; they do not delete destination-only data or make the destination an exact mirror.
Close SubMiner (and stop the background stats daemon, `subminer stats -s`) on both machines before syncing; the command refuses to run while a SubMiner process may be writing the database (`--force` overrides). The mpv safety check requires a live socket connection, so a stale socket file left after mpv exits does not block sync. Both machines must be on the same SubMiner version — the sync aborts on a stats schema mismatch. Remote sync checks standard SubMiner and Bun locations (`~/.local/bin`, `~/.bun/bin`, Homebrew, `/usr/local/bin`, `/usr/bin`, and `/bin`) even when the non-interactive SSH shell omits them from `PATH`.
Two lower-level modes are used internally over SSH and also work standalone for manual transfers (e.g. via a USB drive):
```bash
subminer sync --snapshot /tmp/stats.sqlite # write a consistent snapshot of the local database
subminer sync --merge /tmp/stats.sqlite # merge a snapshot file into the local database
```
Unfinished sessions (a crash mid-playback) are skipped until the app finalizes them; they sync on the next run. Word/kanji "known" state from Anki is not part of the database and does not sync — each machine derives it from its own Anki collection.
## Common Commands
```bash
subminer video.mkv # play a specific file (default plugin config auto-starts visible overlay)
subminer video.mkv # play a specific file (managed launches auto-start the visible overlay by default)
subminer https://youtu.be/... # YouTube playback (requires yt-dlp)
subminer --backend x11 video.mkv # Force x11 backend for a specific file
subminer -u # check for SubMiner updates
@@ -77,11 +122,12 @@ subminer stats -b # start background stats daemon
| Subcommand | Purpose |
| ------------------------------------------ | ------------------------------------------------------------------ |
| `subminer jellyfin` / `jf` | Jellyfin workflows (`-d` discovery, `-p` play, `-l` login) |
| `subminer stats` | Start stats server and open immersion dashboard in browser |
| `subminer stats -b` | Start or reuse background stats daemon (non-blocking) |
| `subminer stats cleanup` | Backfill vocabulary metadata and prune stale rows |
| `subminer doctor` | Dependency + config + socket diagnostics |
| `subminer jellyfin` / `jf` | Jellyfin workflows (`-d` discovery, `-p` play, `-l` login, `--logout`, `--setup`) |
| `subminer stats` | Start the stats server (opens the dashboard when `stats.autoOpenBrowser` is on) |
| `subminer stats -b` / `-s` | Start/reuse or stop the background stats daemon |
| `subminer stats cleanup` | Backfill vocabulary metadata and prune stale rows (`-v` vocab, `-l` lifetime summaries) |
| `subminer stats rebuild` / `backfill` | Rebuild or backfill rollup data |
| `subminer doctor` | Dependency + config + socket diagnostics (`--refresh-known-words` refreshes the known-word cache) |
| `subminer settings` | Open the SubMiner settings window |
| `subminer logs -e` | Export a sanitized local-date log ZIP and print its path |
| `subminer config path` | Print active config file path |
@@ -89,12 +135,15 @@ subminer stats -b # start background stats daemon
| `subminer mpv status` | Check mpv socket readiness |
| `subminer mpv socket` | Print active socket path |
| `subminer mpv idle` | Launch detached idle mpv instance |
| `subminer dictionary <path>` | Generate character dictionary ZIP from file/dir target |
| `subminer sync <host>` | Two-way stats/history sync with another machine over SSH |
| `subminer sync <host> --push` | Merge local stats/history into another machine only |
| `subminer sync <host> --pull` | Merge another machine's stats/history into the local database only |
| `subminer dictionary <path>` / `dict` | Generate character dictionary ZIP from file/dir target |
| `subminer dictionary --candidates <path>` | List AniList candidate matches for character dictionary correction |
| `subminer dictionary --select <id> <path>` | Pin an AniList media ID for that target series |
| `subminer texthooker` | Launch texthooker-only mode |
| `subminer texthooker -o` | Launch texthooker and open it in the default browser |
| `subminer app` | Pass arguments directly to SubMiner binary |
| `subminer app` / `bin` | Pass arguments directly to SubMiner binary (e.g. `subminer app --setup`) |
Use `subminer <subcommand> -h` for command-specific help.
@@ -105,24 +154,25 @@ Use `subminer <subcommand> -h` for command-specific help.
| `-d, --directory` | Video search directory (default: cwd) |
| `-r, --recursive` | Search directories recursively |
| `-R, --rofi` | Use rofi instead of fzf |
| `--setup` | Open first-run setup popup manually |
| `-v, --version` | Print installed SubMiner version |
| `-H, --history` | Browse local watch history (see [Watch History](#watch-history)) |
| `-v, --version` | Print the launcher's own version (can differ from the installed app binary) |
| `-u, --update` | Check for SubMiner updates and update the app/launcher when possible |
| `--start` | Explicitly start overlay after mpv launches |
| `-S, --start-overlay` | Explicitly start overlay after mpv launches |
| `-S, --start-overlay` | Force the visible overlay on start |
| `-T, --no-texthooker` | Disable texthooker server |
| `-p, --profile` | mpv profile name (no default; omitted unless set) |
| `-a, --args` | Pass additional mpv arguments as a quoted string |
| `-b, --backend` | Force window backend (`hyprland`, `sway`, `x11`, `macos`, `windows`) |
| `--settings` | Open the SubMiner settings window |
| `--log-level` | Logger verbosity (`debug`, `info`, `warn`, `error`) |
| `--dev`, `--debug` | Enable app dev-mode (not tied to log level) |
App-binary flags such as `--setup`, `--dev`, and `--debug` are not launcher flags - pass them through with `subminer app`, for example `subminer app --setup`.
On Linux, `subminer -u` updates from the launcher process itself. It can check and replace the AppImage, launcher, runtime plugin copy, and rofi theme even when SubMiner is already running in the tray.
With default plugin settings (`auto_start=yes`, `auto_start_visible_overlay=yes`, `auto_start_pause_until_ready=yes`), explicit start flags are usually unnecessary.
Managed launches inject `auto_start=yes`, `auto_start_visible_overlay=yes`, and `auto_start_pause_until_ready=yes` as plugin script-opts from SubMiner's config defaults (`mpv.autoStartSubMiner`, `auto_start_overlay`), so explicit start flags are usually unnecessary. The plugin's own built-in defaults are off - mpv launched outside SubMiner does not auto-start the overlay.
## Logging
- Default log level is `info`
- `--background` mode defaults to `warn` unless `--log-level` is explicitly set
- `--dev` / `--debug` control app behavior, not logging verbosity - use `--log-level` for that
- Default log level is `warn` (launcher and app; configurable via `logging.level`)
- `--dev` / `--debug` are app-binary flags that control app dev-mode, not logging verbosity - use `--log-level` for that
+5 -5
View File
@@ -56,20 +56,20 @@ This is useful when auto-update is disabled or when you want explicit control ov
Create a standalone sentence card without going through Yomitan:
- **Mine current sentence**: `Ctrl/Cmd+S` (configurable via `shortcuts.mineSentence`)
- **Mine multiple lines**: `Ctrl/Cmd+Shift+S` followed by a digit 19 to select how many recent subtitle lines to combine.
- **Mine multiple lines**: `Ctrl/Cmd+Shift+S` followed by a digit 19 to select how many recent subtitle lines to combine (the digit selector times out after 3 seconds, configurable via `shortcuts.multiCopyTimeoutMs`).
The sentence card uses the note type configured in `isLapis.sentenceCardModel` and always maps sentence/audio to `Sentence` and `SentenceAudio`.
::: warning Requires Lapis/Kiku note type
Sentence card creation requires a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type and `ankiConnect.isLapis.enabled: true` in your config. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
Sentence card creation requires `ankiConnect.isLapis.sentenceCardModel` to name a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type that exists in Anki (default: `"Lapis"`). See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
:::
### 4. Mark as Audio Card
After adding a word via Yomitan, press the audio card shortcut to overwrite the audio with a longer clip spanning the full subtitle timing.
After adding a word via Yomitan, press the audio card shortcut (`Ctrl/Cmd+Shift+A` by default, `shortcuts.markAudioCard`) to mark the card as an audio card. This sets the audio-card flag and fills sentence, image, and metadata fields alongside the full-subtitle audio clip.
::: warning Requires Lapis/Kiku note type
Audio card marking requires a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type and `ankiConnect.isLapis.enabled: true` in your config. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
Audio card marking uses the same `ankiConnect.isLapis.sentenceCardModel` note type as sentence cards. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
:::
### Field Grouping (Kiku)
@@ -172,7 +172,7 @@ Install the sync tools separately - see [Troubleshooting](/troubleshooting#subti
## Texthooker
SubMiner runs a local HTTP server at `http://127.0.0.1:5174` (configurable port) that serves a texthooker UI. This allows external tools - such as a browser-based Yomitan instance - to receive subtitle text in real time.
SubMiner runs a local HTTP server at `http://127.0.0.1:5174` (fixed default port; overridable only via the mpv plugin's `texthooker_port` script-opt) that serves a texthooker UI. This allows external tools - such as a browser-based Yomitan instance - to receive subtitle text in real time.
The texthooker page displays the current subtitle and updates as new lines arrive. This is useful if you prefer to do lookups in a browser rather than through the overlay's built-in Yomitan.
+42 -11
View File
@@ -4,7 +4,7 @@
**Who needs this page:** Most users never touch the plugin directly - SubMiner-managed launches (the app, the `subminer` launcher, or the Windows shortcut) inject the bundled plugin automatically for that session, so there is nothing to install into mpv's global `scripts` directory. Read on if you launch mpv from another tool and want SubMiner's in-player controls, or you want to script mpv against SubMiner.
The plugin ships as a modular Lua package under `plugin/subminer/` (entry point `init.lua`, which loads `main.lua` and sibling modules). Earlier releases shipped a single global `main.lua`; runtime loading replaces it.
The plugin ships as a modular Lua package under `plugin/subminer/` (entry point `main.lua`, which loads `init.lua` and sibling modules). Earlier releases shipped a single global `main.lua`; runtime loading replaces it.
## Runtime Loading
@@ -27,6 +27,25 @@ On Windows, use a named pipe instead:
input-ipc-server=\\.\pipe\subminer-socket
```
## Configuration (script-opts)
The plugin reads options from `script-opts` with the `subminer-` prefix (for example `--script-opts=subminer-backend=hyprland`). Managed launches inject these automatically from your SubMiner config; the shipped `subminer.conf` is intentionally empty so command-line opts always win.
| Option | Default | Description |
| -------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------- |
| `binary_path` | `""` | Path to the SubMiner binary; empty enables [auto-detection](#binary-auto-detection) |
| `socket_path` | platform default | mpv IPC socket path (`/tmp/subminer-socket`, or `\\.\pipe\subminer-socket` on Windows) |
| `texthooker_enabled` | `no` | Start the texthooker server with the overlay |
| `texthooker_port` | `5174` | Texthooker server port |
| `backend` | `auto` | Window backend (`auto`, `hyprland`, `sway`, `x11`, `macos`) |
| `auto_start` | `no` | Start the overlay app on `file-loaded` (managed launches set this from `mpv.autoStartSubMiner`) |
| `auto_start_visible_overlay` | `no` | Show the visible overlay on auto-start (from `auto_start_overlay` in config) |
| `overlay_loading_osd` | `no` | Show an OSD loading spinner while the overlay starts |
| `auto_start_pause_until_ready` | `yes` | Keep mpv paused until the overlay reports tokenization-ready |
| `auto_start_pause_until_ready_timeout_seconds` | `30` | Timeout before resuming playback anyway |
| `osd_messages` | `yes` | Show plugin OSD status messages |
| `log_level` | `info` | Plugin log verbosity |
## Keybindings
Most plugin actions use a `y` chord prefix - press `y`, then the second key (a "chord"):
@@ -44,7 +63,7 @@ Most plugin actions use a `y` chord prefix - press `y`, then the second key (a "
| `v` | Toggle primary subtitle bar visibility |
| `TAB` (default) | Skip intro (AniSkip) |
The AniSkip key is **not** a `y` chord and is not bound by the plugin: the SubMiner app binds it over the mpv IPC socket while it is connected. It defaults to `TAB` and is configurable via `mpv.aniskipButtonKey`. The legacy `y-k` chord still works as a fallback unless you remap the AniSkip key onto it. See [AniSkip Integration](/aniskip-integration) for setup and details.
The AniSkip key is **not** a `y` chord and is not bound by the plugin: the SubMiner app binds it over the mpv IPC socket while it is connected. It defaults to `TAB` and is configurable via `mpv.aniskipButtonKey`. When a custom key (other than `TAB` or `y-k`) is configured, the legacy `y-k` chord is also bound as a fallback. See [AniSkip Integration](/aniskip-integration) for setup and details.
The bare `v` binding is a forced mpv binding. It overrides mpv's default primary subtitle visibility toggle and routes the action to SubMiner's primary subtitle bar instead.
@@ -70,7 +89,7 @@ Notes:
## Menu
Press `y-y` to open an interactive menu in mpv's OSD:
Press `y-y` to open an interactive menu (rendered with mpv's console selector):
```text
SubMiner:
@@ -80,6 +99,7 @@ SubMiner:
4. Open options
5. Restart overlay
6. Check status
7. Stats
```
Select an item by pressing its number.
@@ -92,8 +112,8 @@ When `binary_path` is empty, the plugin searches platform-specific locations:
1. `~/.local/bin/SubMiner.AppImage`
2. `/opt/SubMiner/SubMiner.AppImage`
3. `/usr/local/bin/SubMiner`
4. `/usr/bin/SubMiner`
3. `/usr/local/bin/SubMiner` / `/usr/local/bin/subminer`
4. `/usr/bin/SubMiner` / `/usr/bin/subminer`
**macOS:**
@@ -102,11 +122,14 @@ When `binary_path` is empty, the plugin searches platform-specific locations:
**Windows:**
1. `C:\Program Files\SubMiner\SubMiner.exe`
2. `C:\Program Files (x86)\SubMiner\SubMiner.exe`
3. `C:\SubMiner\SubMiner.exe`
A PowerShell system lookup runs first (running SubMiner process, registry App Paths, `Get-Command`), then static paths:
Packaged Windows plugin installs also rewrite `socket_path` to `\\.\pipe\subminer-socket` automatically.
1. `%LOCALAPPDATA%\Programs\SubMiner\SubMiner.exe` (the default per-user install location)
2. `C:\Program Files\SubMiner\SubMiner.exe`
3. `C:\Program Files (x86)\SubMiner\SubMiner.exe`
4. `C:\SubMiner\SubMiner.exe`
On Windows the plugin also normalizes a Unix-style `socket_path` (`/tmp/subminer-socket`) to the named pipe `\\.\pipe\subminer-socket` at runtime.
## Backend Detection
@@ -135,8 +158,16 @@ script-message subminer-options
script-message subminer-restart
script-message subminer-status
script-message subminer-autoplay-ready
script-message subminer-stats-toggle
script-message subminer-visible-overlay-shown
script-message subminer-visible-overlay-hidden
script-message subminer-managed-subtitles-loading
script-message subminer-overlay-loading-ready
script-message subminer-reload-session-bindings
```
The last five are primarily used by the SubMiner app to notify the plugin of overlay/loading state and to trigger session-binding reloads.
The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) still exist, but they are handled by the SubMiner app over the IPC socket rather than by the plugin - see [AniSkip Integration](/aniskip-integration#triggering-from-mpv).
The `subminer-start` message accepts overrides:
@@ -155,8 +186,8 @@ For how the plugin's auto-start fits into the full launch sequence - including w
- **File loaded**: If `auto_start=yes`, the plugin starts the overlay.
- **Auto-start pause gate**: If `auto_start_visible_overlay=yes` and `auto_start_pause_until_ready=yes`, launcher starts mpv paused. On cold managed background startup, SubMiner opens the tray and visible overlay shell before tokenization warmups finish, then the plugin resumes playback after SubMiner reports tokenization-ready (with a 30-second timeout fallback).
- **Duplicate auto-start events**: Repeated `file-loaded` hooks while overlay is already running are ignored for auto-start triggers (prevents duplicate start attempts).
- **MPV shutdown**: The plugin sends a stop command to gracefully shut down both the overlay and the texthooker server.
- **Texthooker**: Starts as a separate subprocess before the overlay to ensure the app lock is acquired first.
- **MPV shutdown**: The plugin clears its hover/OSD/gate state on shutdown; the overlay app notices the closed IPC socket and shuts itself down.
- **Texthooker**: When `texthooker_enabled=yes`, the plugin appends `--texthooker` to the overlay start command so the app starts the texthooker server alongside the overlay.
## Using with the `subminer` Wrapper
Binary file not shown.

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 8.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 MiB

After

Width:  |  Height:  |  Size: 15 KiB

+4 -2
View File
@@ -515,7 +515,7 @@
// ==========================================
// AnkiConnect Integration
// Automatic Anki updates and media generation options.
// Hot-reload: ankiConnect.ai.enabled, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Shared AI provider transport settings are read from top-level ai and typically require restart.
// Most other AnkiConnect settings still require restart.
// ==========================================
@@ -559,6 +559,8 @@
"animatedMaxHeight": 0, // Maximum height for animated AVIF captures, in pixels. Set to 0 to preserve aspect ratio.
"animatedCrf": 35, // Animated AVIF CRF quality target. Lower values produce larger, higher-quality files.
"syncAnimatedImageToWordAudio": true, // For animated AVIF images, prepend a frozen first frame matching the existing word-audio duration so motion starts with sentence audio. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Changes apply live. Values: true | false
"mirrorMpvVolume": true, // Apply mpv's current software volume curve to generated sentence audio. Changes apply live. Values: true | false
"audioPadding": 0, // Seconds of padding appended to both ends of generated sentence audio and animated AVIF clips.
"fallbackDuration": 3, // Fallback clip duration in seconds when subtitle timing data is unavailable.
"maxMediaDuration": 30 // Maximum allowed media clip duration in seconds.
@@ -568,7 +570,7 @@
"refreshMinutes": 1440, // Minutes between known-word cache refreshes.
"addMinedWordsImmediately": true, // Immediately append newly mined card words into the known-word cache. Values: true | false
"matchMode": "headword", // Known-word matching strategy for subtitle annotations. Cache matches always receive known-word highlighting even when POS filters suppress other annotation types. Values: headword | surface
"decks": {} // Decks and expression/word fields for known-word cache. Object mapping deck names to arrays of field names to extract, e.g. { "Kaishi 1.5k": ["Word"] }.
"decks": {} // Decks and expression/word fields for known-word cache. Object mapping deck names to arrays of field names to extract, e.g. { "Kaishi 1.5k": ["Word"] }. Reading fields (Reading, Word Reading, ExpressionReading) are always probed so cached words match only in the reading their note teaches; words from notes without readings match in any reading.
}, // Known words setting.
"behavior": {
"overwriteAudio": true, // When updating an existing card, overwrite the audio field instead of skipping it. Values: true | false
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 633 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 KiB

After

Width:  |  Height:  |  Size: 1.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

+8 -10
View File
@@ -10,17 +10,15 @@ A few terms used throughout:
All shortcuts are configurable in `config.jsonc` under `shortcuts` and `keybindings`. Set any shortcut to `null` to disable it.
## Global Shortcuts
## App-Wide Shortcuts
These work system-wide regardless of which window has focus.
| Shortcut | Action | Configurable |
| ------------- | ---------------------- | -------------------------------------- |
| `Alt+Shift+O` | Toggle visible overlay | `shortcuts.toggleVisibleOverlayGlobal` |
| `Alt+Shift+Y` | Open Yomitan settings | Fixed (not configurable) |
| Shortcut | Action | Scope | Configurable |
| ------------- | ---------------------- | -------------------------------------------- | -------------------------------------- |
| `Alt+Shift+O` | Toggle visible overlay | Works while the overlay or mpv has focus | `shortcuts.toggleVisibleOverlayGlobal` |
| `Alt+Shift+Y` | Open Yomitan settings | OS-global (registered with the OS) | Fixed (not configurable) |
::: tip
Global shortcuts are registered with the OS. If they conflict with another application, update them in `shortcuts` config and restart SubMiner.
`Alt+Shift+O` is dispatched by the overlay window and the mpv plugin, so it works from either surface without OS registration. Only `Alt+Shift+Y` is registered with the OS; if it conflicts with another application, that binding cannot be changed. All `shortcuts.*` keys hot-reload - no restart needed.
:::
## Mining Shortcuts
@@ -88,7 +86,7 @@ Mouse-hover playback behavior is configured separately from shortcuts: `subtitle
| `Ctrl/Cmd+N` | Toggle overlay notification history panel | `shortcuts.toggleNotificationHistory` |
| `Ctrl+Alt+C` | Open the manual YouTube subtitle picker | `keybindings` |
| `Ctrl+Alt+S` | Open subtitle sync (subsync) modal | `shortcuts.triggerSubsync` |
| `\` | Toggle subtitle sidebar | `subtitleSidebar.toggleKey` |
| `\` | Toggle subtitle sidebar | `subtitleSidebar.toggleKey` (overlay) / `shortcuts.toggleSubtitleSidebar` (mpv session binding) |
| `` ` `` | Toggle stats overlay | `stats.toggleKey` |
| `W` | Mark current video watched and advance to next in queue | `stats.markWatchedKey` |
@@ -109,7 +107,7 @@ Controller input only drives the overlay while keyboard-only mode is enabled. Th
## MPV Plugin Chords
When the mpv plugin is installed, all commands use a `y` chord prefix - press `y`, then the second key within 1 second.
When the mpv plugin is installed, all commands use a `y` chord prefix - press `y`, then the second key (the overlay-side chord times out after 1 second; the mpv plugin uses native mpv key sequences).
| Chord | Action |
| ----- | ---------------------------------------------------------- |
+11 -9
View File
@@ -64,7 +64,7 @@ For full details on dictionary generation, name variant expansion, auto-sync lif
## Frequency Highlighting
Frequency highlighting colors tokens based on how common they are, using dictionary frequency rank data. This helps you spot high-value vocabulary at a glance. Frequency ranks are sourced from the **highest-ranked frequency dictionary** installed in Yomitan - other frequency dictionaries are not consulted.
Frequency highlighting colors tokens based on how common they are, using dictionary frequency rank data. This helps you spot high-value vocabulary at a glance. For each token, ranks from the installed Yomitan frequency dictionaries are consulted in priority order: the highest-priority dictionary that has the term wins, lower-priority dictionaries fill in terms it lacks, and occurrence-based dictionaries are skipped.
**Modes:**
@@ -90,7 +90,7 @@ SubMiner looks up each token's `frequencyRank` from `term_meta_bank_*.json` file
When `sourcePath` is omitted, SubMiner searches default install/runtime locations for `frequency-dictionary` directories automatically.
::: info
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist.
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist. For merged kana tokens, SubMiner keeps a rank when the dictionary headword reading covers the full token (for example, `かと言って` / `かといって`), while grammar wrapped around a shorter lemma remains unannotated.
:::
::: info
@@ -126,22 +126,24 @@ All colors are customizable via the `subtitleStyle.jlptColors` object.
## Runtime Toggles
All annotation layers can be toggled at runtime via the mpv command menu without restarting:
These annotation layers can be toggled at runtime via the runtime options palette (`Ctrl/Cmd+Shift+O`) without restarting:
- `ankiConnect.knownWords.highlightEnabled` (`On` / `Off`)
- `subtitleStyle.nameMatchEnabled` (`On` / `Off`)
- `subtitleStyle.nameMatchImagesEnabled` (`On` / `Off`)
- `ankiConnect.knownWords.matchMode`
- `ankiConnect.nPlusOne.enabled` (`On` / `Off`)
- `subtitleStyle.enableJlpt` (`On` / `Off`)
- `subtitleStyle.frequencyDictionary.enabled` (`On` / `Off`)
(Character-name matching, `subtitleStyle.nameMatchEnabled`, is toggled through config or the Settings window, not the runtime palette.)
Toggles only apply to new subtitle lines after the change - the currently displayed line is not re-tokenized in place.
## Rendering Priority
When multiple annotations apply to the same token, the visual priority is:
1. **N+1 target** (highest) - the single unknown word in an N+1 sentence
2. **Character-name match** - dictionary-driven character-name token styling
1. **Character-name match** (highest) - dictionary-driven character-name token styling; it clears the token's N+1, frequency, and JLPT annotations
2. **N+1 target** - the single unknown word in an N+1 sentence
3. **Known-word color** - already-learned token tint
4. **Frequency highlight** - common-word coloring (not applied when N+1/character-name/known-word already matched)
5. **JLPT underline** - level-based underline (stacks with the above since it uses underline rather than text color)
4. **Frequency highlight** - common-word coloring (not applied when a higher layer already matched)
5. **JLPT underline** - level-based underline (stacks with N+1/known/frequency since it uses underline rather than text color, but not with a character-name match)
+10 -10
View File
@@ -164,7 +164,7 @@ Shown when SubMiner tries to update a card that no longer exists, or when AnkiCo
**Overlay does not appear**
- Confirm SubMiner is running: `SubMiner.AppImage --start` or check for the process.
- On Linux, the overlay requires a supported window backend. Hyprland and Sway have native Wayland support; all other compositors require both mpv and SubMiner to run under X11 or Xwayland (`xdotool` and `xwininfo` must be installed).
- On Linux, the overlay requires a supported window backend. Hyprland and Sway have native Wayland support; all other compositors require both mpv and SubMiner to run under X11 or Xwayland (`xdotool`, `xprop`, and `xwininfo` must be installed).
- On macOS, grant Accessibility permission to SubMiner in System Settings > Privacy & Security > Accessibility.
**Overlay appears but clicks pass through / cannot interact**
@@ -177,7 +177,7 @@ Shown when SubMiner tries to update a card that no longer exists, or when AnkiCo
- Renderer errors now trigger an automatic recovery path. You should see a short toast ("Renderer error recovered. Overlay is still running.").
- Recovery closes any open modal and restores click-through/shortcuts automatically without interrupting mpv playback.
- If errors keep recurring, toggle the overlay's DevTools using overlay chord `y` then `d` (or global `F12`) and inspect the `renderer overlay recovery` error payload for stack trace + modal/subtitle context.
- If errors keep recurring, toggle the overlay's DevTools using overlay chord `y` then `d` (`F12` also works in dev builds) and inspect the `renderer overlay recovery` error payload for stack trace + modal/subtitle context.
**Overlay is on the wrong monitor or position**
@@ -258,22 +258,22 @@ Without FFmpeg, card creation still works but audio and image fields will be emp
Media generation has a 30-second timeout (60 seconds for animated AVIF). If your video file is on a slow network mount or the codec requires software decoding, generation may time out. Try:
- Using a local copy of the video file.
- Reducing `media.imageQuality` or switching from `avif` to `static` image type.
- Checking that `media.maxMediaDuration` is not set too high.
- Reducing `ankiConnect.media.imageQuality` or switching from `avif` to `static` image type.
- Checking that `ankiConnect.media.maxMediaDuration` is not set too high.
## Shortcuts
**"Failed to register global shortcut"**
Global shortcuts (`Alt+Shift+O`, `Alt+Shift+Y`) may conflict with other applications or desktop environment keybindings.
This warning refers to the OS-registered shortcut `Alt+Shift+Y` (Yomitan settings), which is fixed and may conflict with other applications or desktop environment keybindings.
- Check your DE/WM keybinding settings for conflicts.
- Change the shortcut in your config under `shortcuts.toggleVisibleOverlayGlobal`.
- Check your DE/WM keybinding settings for conflicts and free up `Alt+Shift+Y` there.
- `Alt+Shift+O` (`shortcuts.toggleVisibleOverlayGlobal`) is not OS-registered - it is handled by the overlay window and the mpv plugin, so it does not trigger this warning and only needs those windows focused.
- On Wayland, global shortcut registration has limitations depending on the compositor. Only Hyprland and Sway are supported natively - see the [Hyprland](#hyprland) section below for shortcut passthrough rules. Other Wayland compositors require X11/Xwayland.
**Overlay keybindings not working**
Overlay-local shortcuts (Space, arrow keys, etc.) only work when the overlay window has focus. Click on the overlay or use the global shortcut to toggle it to give it focus.
Overlay-local shortcuts (Space, arrow keys, etc.) only work when the overlay window has focus. Click on the overlay or use `Alt+Shift+O` (with the overlay or mpv focused) to toggle it and give it focus.
## Subtitle Timing
@@ -307,9 +307,9 @@ Install ffsubsync or configure the path:
- **pip**: `pip install ffsubsync`
- Must be on `PATH` or configured via `subsync.ffsubsync_path` in your config.
**"Subtitle synchronization failed"**
**"alass synchronization failed" / "ffsubsync synchronization failed"**
If subtitle sync fails:
If subtitle sync fails (the error message is prefixed with the engine name):
- Ensure the reference subtitle track exists in the video (alass requires a source track).
- Check that `ffmpeg` is available (used to extract the internal subtitle track).
+37 -15
View File
@@ -60,13 +60,20 @@ The mpv plugin is always available - it's bundled with SubMiner and injected at
While SubMiner is running, it watches your active config file and applies safe updates automatically.
Live-updated settings:
Live-updated settings include:
- `subtitleStyle`
- `keybindings`
- `shortcuts`
- `secondarySub.defaultMode`
- `ankiConnect.ai`
- `subtitleSidebar`
- `notifications`
- `logging`
- `jimaku`, `subsync`
- `mpv.aniskipEnabled`, `mpv.aniskipButtonKey`
- `stats.toggleKey`, `stats.markWatchedKey`
- `youtube.primarySubLanguages`
- most `ankiConnect.*` settings (including `ankiConnect.ai`)
Invalid config edits are rejected; SubMiner keeps the previous valid runtime config and shows an error notification.
For restart-required sections, SubMiner shows a restart-needed notification.
@@ -82,12 +89,13 @@ subminer -R # Use rofi instead of fzf
subminer -d ~/Videos # Specific directory
subminer -r -d ~/Anime # Recursive search
subminer video.mkv # Play specific file (overlay auto-starts)
subminer --start video.mkv # Explicit overlay start (use when auto_start=no in config)
subminer -S video.mkv # Same as above via --start-overlay
subminer --start video.mkv # Explicit overlay start (use when mpv.autoStartSubMiner is false in config)
subminer -S video.mkv # Also force the visible overlay on start (--start-overlay)
subminer https://youtu.be/... # Play a YouTube URL
subminer ytsearch:"jp news" # Play first YouTube search result
subminer --setup # Open first-run setup popup
subminer --version # Print installed SubMiner version
subminer -H # Browse watch history (replay/continue episodes, fzf or rofi picker)
subminer app --setup # Open first-run setup popup
subminer --version # Print the launcher's version
subminer -v # Same as above
subminer --log-level debug video.mkv # Enable verbose logs for launch/debugging
subminer --log-level warn video.mkv # Set logging level explicitly
@@ -111,11 +119,19 @@ subminer config show # Print active config contents
subminer mpv socket # Print active mpv socket path
subminer mpv status # Exit 0 if socket is ready, else exit 1
subminer mpv idle # Launch detached idle mpv with SubMiner defaults
subminer sync media-box # Sync stats/watch history with an SSH host
subminer sync media-box --push # Merge this machine's stats into the host only
subminer sync media-box --pull # Merge the host's stats into this machine only
subminer sync --snapshot ~/subminer-snapshot.sqlite # Write a local DB snapshot
subminer sync --merge ~/subminer-snapshot.sqlite # Merge a snapshot into the local DB
subminer dictionary /path/to/file-or-directory # Generate character dictionary ZIP from target (manual Yomitan import)
subminer dictionary --candidates /path/to/file.mkv
subminer dictionary --select 21355 /path/to/file.mkv
subminer texthooker # Launch texthooker-only mode
subminer texthooker -o # Launch texthooker and open it in your browser
subminer stats # Start the local stats server (see Immersion Tracking)
subminer stats -b # Start/reuse the background stats daemon
subminer stats -s # Stop the background stats daemon
subminer app --anilist-setup # Pass args directly to SubMiner binary (example: AniList login flow)
# Direct packaged app control
@@ -129,6 +145,7 @@ SubMiner.AppImage --start --toggle # Start MPV IPC + toggle visibility
SubMiner.AppImage --show-visible-overlay # Force show visible overlay
SubMiner.AppImage --hide-visible-overlay # Force hide visible overlay
SubMiner.AppImage --toggle-primary-subtitle-bar # Toggle primary subtitle bar visibility
SubMiner.AppImage --toggle-subtitle-sidebar # Toggle the subtitle sidebar
SubMiner.AppImage --start --dev # Enable app/dev mode only
SubMiner.AppImage --start --debug # Alias for --dev
SubMiner.AppImage --start --log-level debug # Force verbose logging without app/dev mode
@@ -194,6 +211,7 @@ This flow requires `mpv.exe` to be discoverable. Leave `mpv.executablePath` blan
- `subminer logs -e`: export a sanitized ZIP of today's local-date logs, or the most recent logs when no current-day log exists. The exported copy masks common PII and secrets; on-disk logs are unchanged.
- `subminer config`: config file helpers (`path`, `show`).
- `subminer mpv`: mpv helpers (`status`, `socket`, `idle`).
- `subminer sync <host>`: sync immersion stats and watch history with another machine over SSH. The host is the SSH destination (`user@host` or an SSH config alias). Use `--push` to merge only this machine's data into the host, or `--pull` to merge only the host's data into this machine; both remain insert-only and do not make either database an exact mirror. Remote launcher checks include standard SubMiner and Bun paths even when SSH omits them from `PATH`. Use `--snapshot <file>` to write a consistent local stats DB snapshot, `--merge <file>` to merge a snapshot into the local stats DB, and `--force` to skip the running stats/mpv safety check. Advanced options: `--db <file>` overrides the local stats DB path, and `--remote-cmd <cmd>` overrides the `subminer` command used on the remote host.
- `subminer dictionary <path>`: generates a Yomitan-importable character dictionary ZIP from a file/directory target.
- Use `subminer dictionary --candidates <path>` and `subminer dictionary --select <id> <path>` to correct AniList character-dictionary matches for a whole series.
- `subminer texthooker`: texthooker-only shortcut (same behavior as `--texthooker`). A _texthooker_ is a web page that displays the current subtitle line as selectable text, so browser-based dictionary extensions and other tools can read along with playback.
@@ -207,7 +225,7 @@ Setup popup appears on first launch, or when setup has not been completed.
You can also open it manually:
```bash
subminer --setup
subminer app --setup
SubMiner.AppImage --setup
```
@@ -243,6 +261,7 @@ Top-level launcher flags like `--jellyfin-*` are intentionally rejected.
- `--sub-file-paths=.;subs;subtitles`
- `--sid=auto`
- `--secondary-sid=auto`
- `--sub-visibility=no` (the overlay renders subtitles instead of mpv)
- `--secondary-sub-visibility=no`
You can append additional MPV arguments with launcher `-a/--args`, for example `--args "--ao=alsa --volume=80"`.
@@ -288,8 +307,8 @@ Notes:
- Press `Ctrl+Alt+C` during active YouTube playback to open the manual YouTube subtitle picker and retry track selection.
- For YouTube URLs, `subminer` probes available YouTube subtitle tracks, reuses existing authoritative tracks when available, and downloads only missing sides.
- Native mpv secondary subtitle rendering stays hidden so the overlay remains the visible secondary subtitle surface.
- Primary subtitle target languages come from `youtube.primarySubLanguages` (defaults to `["ja","jpn"]`).
- Secondary target languages come from `secondarySub.secondarySubLanguages` (empty by default; when empty, no language-based secondary track is auto-selected, though mpv's `--slang` list above still prefers English variants). When multiple matching secondary tracks exist, SubMiner prefers a non-Signs/Songs track.
- YouTube auto-selection always targets a Japanese primary track and an English secondary track (manual uploads preferred over auto-generated captions). `youtube.primarySubLanguages` (defaults to `["ja","jpn"]`) defines which loaded track counts as a satisfactory primary for the missing-subtitle notification and for managed local/playlist selection.
- When multiple matching secondary tracks exist, SubMiner prefers a non-Signs/Songs track.
- Configure defaults in `$XDG_CONFIG_HOME/SubMiner/config.jsonc` (or `~/.config/SubMiner/config.jsonc`) under `youtube` and `secondarySub`.
For local video files, SubMiner uses the same config-driven language priorities to auto-select the primary and secondary subtitle tracks from internal and external subtitle sources.
@@ -324,6 +343,8 @@ By default SubMiner uses the first connected controller after controller support
| `Select` / `Minus` | Quit mpv |
| `L2` / `R2` | Unbound (available for custom bindings) |
Note: the default quit binding uses gamepad button index 6. Pads that follow the W3C standard gamepad layout report L2 as index 6 (Select is index 8), so on those controllers the quit action may fire on L2 instead - use `Alt+C` learn mode to remap it for your pad.
### Analog Controls
| Input | Action |
@@ -341,18 +362,18 @@ All button and axis mappings are configurable under the `controller` config bloc
See [Keyboard Shortcuts](/shortcuts) for the full reference, including mining shortcuts, overlay controls, and customization.
**Global shortcuts** (work system-wide):
**App-wide shortcuts:**
| Keybind | Action |
| ------------- | ---------------------- |
| `Alt+Shift+O` | Toggle visible overlay |
| `Alt+Shift+Y` | Open Yomitan settings |
| Keybind | Action | Scope |
| ------------- | ---------------------- | -------------------------------------------------------------------------------------------------- |
| `Alt+Shift+O` | Toggle visible overlay | Works while the overlay or mpv has focus (configurable via `shortcuts.toggleVisibleOverlayGlobal`) |
| `Alt+Shift+Y` | Open Yomitan settings | OS-global - registered with the system, works from any window |
`Alt+Shift+Y` is fixed and not configurable. All other shortcuts can be changed under `shortcuts` in your config.
Useful overlay-local default keybinding: `Ctrl+Alt+P` opens the playlist browser for the current video's parent directory and the live mpv queue so you can append, reorder, remove, or jump between episodes without leaving playback.
Press `V` to hide or restore the primary SubMiner subtitle bar. The bundled mpv plugin also binds bare `v` to the same action (injected at runtime).
Press `V` to cycle the primary SubMiner subtitle bar through hidden → visible → hover modes. The bundled mpv plugin also binds bare `v` to the same action (injected at runtime).
`Ctrl/Cmd+/` opens the session help modal with the current overlay and mpv keybindings. The same help view is also available through the `y-h` chord in mpv.
@@ -362,5 +383,6 @@ Hovering over subtitle text pauses mpv by default; leaving resumes it. Yomitan p
- Drop video files onto the overlay to replace current playback.
- Hold `Shift` while dropping to append to the playlist instead.
- Drop subtitle files onto the overlay to load them as a new subtitle track.
Next: [Mining Workflow](/mining-workflow) - word lookup, card creation, and the full mining loop.
+7 -1
View File
@@ -265,9 +265,15 @@ script-message subminer-options
script-message subminer-restart
script-message subminer-status
script-message subminer-autoplay-ready
script-message subminer-stats-toggle
script-message subminer-visible-overlay-shown
script-message subminer-visible-overlay-hidden
script-message subminer-managed-subtitles-loading
script-message subminer-overlay-loading-ready
script-message subminer-reload-session-bindings
```
The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) are handled by the SubMiner app over the mpv IPC socket while it is connected.
The overlay/loading/session-binding messages are primarily sent by the SubMiner app to keep the plugin's state in sync. The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) are handled by the SubMiner app over the mpv IPC socket while it is connected.
The start command also accepts inline overrides:
+26 -20
View File
@@ -74,25 +74,29 @@ Press **Ctrl+Alt+C** during YouTube playback to open the subtitle picker overlay
- Select different primary and secondary tracks
- Retry track loading if the auto-load failed or picked the wrong track
SubMiner shows an "Opening YouTube subtitle picker..." status through your configured notification
surface while it probes tracks and prepares the modal, then updates the subtitle download progress
card to a success notification after the selected tracks load.
The picker displays each track with its language, kind (manual/auto), and title when available.
## Subtitle Format Handling
SubMiner handles several YouTube subtitle formats transparently:
| Format | Handling |
| ------ | -------- |
| `srt`, `vtt` | Used directly (preferred for manual tracks) |
| Format | Handling |
| ---------------------- | -------------------------------------------------------- |
| `srt`, `vtt` | Used directly (preferred for manual tracks) |
| `srv1`, `srv2`, `srv3` | YouTube TimedText XML --- converted to VTT automatically |
| Auto-generated VTT | Normalized to remove rolling-caption text duplication |
| Auto-generated VTT | Normalized to remove rolling-caption text duplication |
For auto-generated tracks, SubMiner prefers `srv3` > `srv2` > `srv1` > `vtt` (TimedText XML produces cleaner output). For manual tracks, `srt` > `vtt` is preferred.
## Card Media Cache
By default, YouTube card audio and screenshots are extracted directly from mpv's active stream URLs. If generated card media fails with YouTube `403` errors, set `youtube.mediaCache.mode` to `"background"`. Background mode starts a separate `yt-dlp` media download after playback loads, creates text fields immediately, queues audio/image work for mined notes, and fills those fields once the local cache file is ready.
By default, YouTube card audio and screenshots are extracted directly from mpv's active stream URLs. If generated card media fails with YouTube `403` errors, set `youtube.mediaCache.mode` to `"background"`. Background mode starts a separate `yt-dlp` media download after playback loads, including YouTube URLs opened directly in mpv and resolved stream URLs when mpv still exposes the original YouTube playlist entry. It creates text fields immediately, queues audio/image work for mined notes, and fills those fields once the local cache file is ready.
Background cache downloads use IPv4 and retry flags to reduce YouTube throttling failures. If the background download still fails, SubMiner shows a cache failure notification, shows queued-card failure notifications, and clears those pending updates so cards are not left waiting silently.
Background cache downloads are capped at 720p by default (`youtube.mediaCache.maxHeight`; set `0` for unlimited) and use IPv4 and retry flags to reduce YouTube throttling failures. If the background download still fails, SubMiner shows a cache failure notification, shows queued-card failure notifications, and clears those pending updates so cards are not left waiting silently.
## Configuration Reference
@@ -101,36 +105,38 @@ Background cache downloads use IPv4 and retry flags to reduce YouTube throttling
```jsonc
{
"youtube": {
"primarySubLanguages": ["ja", "jpn"]
}
"primarySubLanguages": ["ja", "jpn"],
},
}
```
| Option | Type | Description |
| ------ | ---- | ----------- |
| `primarySubLanguages` | `string[]` | Language priority for YouTube primary subtitle auto-loading (default `["ja", "jpn"]`) |
| Option | Type | Description |
| --------------------- | ---------- | ------------------------------------------------------------------------------------- |
| `primarySubLanguages` | `string[]` | Languages that count as a satisfactory primary subtitle (default `["ja", "jpn"]`). Used by the "primary subtitle missing" notification and by managed local/playlist subtitle selection. |
YouTube auto-selection itself always picks a Japanese track first (manual over auto), then falls back to any manual track — `primarySubLanguages` does not change which YouTube track is auto-picked.
### Secondary Subtitle Languages
Secondary track selection uses the shared `secondarySub` config:
YouTube secondary selection is fixed: SubMiner always tries an English track (manual over auto) and loads it when found. The shared `secondarySub` config does not change YouTube track selection — `secondarySubLanguages` and `autoLoadSecondarySub` apply only to local/Jellyfin sidecar selection — but `defaultMode` still controls how the loaded secondary bar is displayed:
```jsonc
{
"secondarySub": {
"secondarySubLanguages": [],
"autoLoadSecondarySub": false,
"defaultMode": "hover"
}
"defaultMode": "hover",
},
}
```
| Option | Type | Description |
| ------ | ---- | ----------- |
| `secondarySubLanguages` | `string[]` | Extra language codes (e.g. `["eng", "en"]`) used when auto-selecting a secondary track. Default is empty (`[]`). For YouTube, SubMiner always tries an English track first regardless of this list. |
| `autoLoadSecondarySub` | `boolean` | Auto-detect and load a matching secondary track (default: `false`) |
| `defaultMode` | `"hidden"` / `"visible"` / `"hover"` | Initial display mode for secondary subtitles (default: `"hover"`) |
| Option | Type | Description |
| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `secondarySubLanguages` | `string[]` | Extra language codes (e.g. `["eng", "en"]`) used when auto-selecting a secondary track for local/Jellyfin sidecar files. Default is empty (`[]`). Not used for YouTube. |
| `autoLoadSecondarySub` | `boolean` | Auto-detect and load a matching secondary sidecar track for local files (default: `false`). Not used for YouTube. |
| `defaultMode` | `"hidden"` / `"visible"` / `"hover"` | Initial display mode for secondary subtitles (default: `"hover"`) |
Precedence: CLI flag > environment variable > `config.jsonc` > built-in default.
These settings come from `config.jsonc` (or built-in defaults); there are no CLI flags or environment variables for subtitle language selection.
## Limitations and Troubleshooting
@@ -1,29 +0,0 @@
<!-- read_when: changing managed mpv startup, pause-until-ready, or visible overlay boot ordering -->
# Early Managed Overlay Startup Design
Status: approved
Date: 2026-06-06
## Problem
Managed mpv startup can pause playback immediately, then leave SubMiner's tray and visible overlay
unavailable until Yomitan/tokenization warmups finish. Startup notifications therefore miss the
overlay surface and fall back to non-overlay status paths.
## Chosen Approach
For cold `--start --background --managed-playback` launches, handle initial args before waiting for
the deferred overlay warmup. That lets the tray and visible overlay shell initialize immediately
while the existing tokenization warmups continue in the background.
The mpv plugin pause gate stays armed. Playback release still waits for SubMiner's autoplay-ready
signal, which is emitted only after tokenization warmup and visible-overlay readiness. Existing
second-instance attach behavior remains unchanged: when the launcher finds an already-running
background app, it sends the same control command to that process and reuses its warmups/tokenizer.
## Checks
- Add a startup ordering regression test for managed background playback.
- Keep the existing deferred startup ordering for non-managed launches.
- Run the startup/runtime test slice plus SubMiner verification lane.
@@ -1,27 +0,0 @@
<!-- read_when: changing overlay notification hover, macOS mouse passthrough, or notification actions -->
# macOS Notification Hover Stability Design
Status: approved
Date: 2026-06-09
## Problem
On macOS, hovering a character dictionary build notification can make the card flicker and slide as
if it is hiding, then snap back. The likely trigger is the notification stack changing the overlay
window's mouse-passthrough state for a progress card that has no user action.
## Chosen Approach
Keep non-action overlay notifications visually stable and click-through on hover. Only notifications
with explicit actions should request interactive overlay input. The notification history panel keeps
its existing interactive behavior.
This avoids a macOS mouseenter/mouseleave passthrough loop for passive progress cards while
preserving clickable notification actions.
## Checks
- Add a renderer regression test for passive notification hover.
- Keep action-bearing notification cards interactive.
- Run the targeted overlay notification and mouse-ignore tests.
+15 -1
View File
@@ -3,10 +3,22 @@
# Verification
Status: active
Last verified: 2026-05-23
Last verified: 2026-07-06
Owner: Kyle Yasuda
Read when: selecting the right verification lane for a change
## Lane Infrastructure
- Lane membership is defined once in `scripts/test-lanes.ts` and discovered by
directory — new test files join their lane automatically; never hand-list test
files in `package.json`.
- `scripts/run-test-lane.mjs` runs each test file in its own `bun test` process
(per-file isolation with a wall timeout) so a hanging test or leaked global in
one file cannot cascade into the rest of the lane. `--jobs N` parallelizes;
`--single-process` restores the shared-process mode for debugging.
- `bun run test:fast` is the full source gate: discovered `src/**`, launcher
unit, `scripts/**`, and the compiled runtime-compat slice.
## Default Handoff Gate
```bash
@@ -31,6 +43,8 @@ bun run docs:build
- Config/schema/defaults: `bun run test:config`, then `bun run generate:config-example` if template/defaults changed
- Launcher/plugin: `bun run test:launcher` or `bun run test:env`
- Runtime-compat / compiled behavior: `bun run test:runtime:compat`
- Stats dashboard UI: `bun run test:stats`
- Build/release scripts (`scripts/**`): `bun run test:scripts`
- Coverage for the maintained source lane: `bun run test:coverage:src`
- Deep/local full gate: default handoff gate above
+30
View File
@@ -0,0 +1,30 @@
// Minimal ambient typing for bun:sqlite. The launcher always runs under bun
// (see the build banner in package.json), but the repo typechecks with plain
// tsc which has no bun type definitions.
declare module 'bun:sqlite' {
export interface RunResult {
changes: number;
lastInsertRowid: number | bigint;
}
export interface Statement<ReturnType = unknown, ParamsType extends unknown[] = unknown[]> {
all(...params: ParamsType): ReturnType[];
get(...params: ParamsType): ReturnType | undefined;
run(...params: ParamsType): RunResult;
}
export class Database {
constructor(
filename: string,
options?: { readonly?: boolean; readwrite?: boolean; create?: boolean },
);
query<ReturnType = unknown, ParamsType extends unknown[] = unknown[]>(
sql: string,
): Statement<ReturnType, ParamsType>;
prepare<ReturnType = unknown, ParamsType extends unknown[] = unknown[]>(
sql: string,
): Statement<ReturnType, ParamsType>;
run(sql: string, ...params: unknown[]): RunResult;
close(throwOnError?: boolean): void;
}
}
+228
View File
@@ -0,0 +1,228 @@
import fs from 'node:fs';
import path from 'node:path';
import { spawnSync } from 'node:child_process';
import { fail, log } from '../log.js';
import { commandExists } from '../util.js';
import {
collectVideos,
findRofiTheme,
formatPickerLaunchError,
showFzfMenu,
showRofiMenu,
} from '../picker.js';
import {
findNextEpisode,
groupHistoryBySeries,
listSeasonDirs,
materializeCoverArt,
queryLocalWatchHistory,
resolveImmersionDbPath,
sortVideosByEpisode,
type HistorySeriesEntry,
} from '../history.js';
import type { Args } from '../types.js';
import type { LauncherCommandContext } from './context.js';
function checkPickerDependencies(args: Args): void {
if (args.useRofi) {
if (!commandExists('rofi')) fail('Missing dependency: rofi');
return;
}
if (!commandExists('fzf')) fail('Missing dependency: fzf');
}
function showRofiIndexMenu(
labels: string[],
prompt: string,
themePath: string | null,
icons: Array<string | null> = [],
): number {
const rofiArgs = ['-dmenu', '-i', '-matching', 'fuzzy', '-format', 'i', '-p', prompt];
const hasIcons = icons.some(Boolean);
if (hasIcons) rofiArgs.push('-show-icons');
if (themePath) {
rofiArgs.push('-theme', themePath);
} else {
rofiArgs.push('-theme-str', 'configuration { font: "Noto Sans CJK JP Regular 8";}');
}
if (hasIcons) {
rofiArgs.push('-theme-str', 'configuration { show-icons: true; }');
rofiArgs.push('-theme-str', 'element-icon { enabled: true; size: 3em; }');
}
const lines = labels.map((label, index) =>
icons[index] ? `${label}\u0000icon\u001f${icons[index]}` : label,
);
const result = spawnSync('rofi', rofiArgs, {
input: `${lines.join('\n')}\n`,
encoding: 'utf8',
stdio: ['pipe', 'pipe', 'ignore'],
});
if (result.error) {
fail(formatPickerLaunchError('rofi', result.error as NodeJS.ErrnoException));
}
const out = (result.stdout || '').trim();
if (!out) return -1;
const idx = Number.parseInt(out, 10);
return Number.isInteger(idx) && idx >= 0 && idx < labels.length ? idx : -1;
}
function showFzfIndexMenu(labels: string[], prompt: string): number {
const lines = labels.map((label, index) => `${index}\t${label}`);
const result = spawnSync(
'fzf',
[
'--ansi',
'--reverse',
'--ignore-case',
`--prompt=${prompt}: `,
'--delimiter=\t',
'--with-nth=2..',
],
{
input: `${lines.join('\n')}\n`,
encoding: 'utf8',
stdio: ['pipe', 'pipe', 'inherit'],
},
);
if (result.error) {
fail(formatPickerLaunchError('fzf', result.error as NodeJS.ErrnoException));
}
const picked = (result.stdout || '').trim();
const tab = picked.indexOf('\t');
if (tab === -1) return -1;
const idx = Number.parseInt(picked.slice(0, tab), 10);
return Number.isInteger(idx) && idx >= 0 && idx < labels.length ? idx : -1;
}
function pickIndex(
labels: string[],
prompt: string,
useRofi: boolean,
themePath: string | null,
icons: Array<string | null> = [],
): number {
if (labels.length === 0) return -1;
return useRofi
? showRofiIndexMenu(labels, prompt, themePath, icons)
: showFzfIndexMenu(labels, prompt);
}
function formatEpisodeLabel(entry: HistorySeriesEntry): string {
const { parsedSeason, parsedEpisode } = entry.lastWatched;
if (parsedEpisode === null) return '';
return parsedSeason !== null ? `S${parsedSeason}E${parsedEpisode}` : `E${parsedEpisode}`;
}
function formatSeriesLabel(entry: HistorySeriesEntry): string {
const episodeLabel = formatEpisodeLabel(entry);
return episodeLabel ? `${entry.displayName} [last: ${episodeLabel}]` : entry.displayName;
}
function pickEpisodeFromDir(dir: string, context: LauncherCommandContext): string | null {
const { args, scriptPath } = context;
const videos = sortVideosByEpisode(collectVideos(dir, false));
if (videos.length === 0) {
fail(`No video files found in: ${dir}`);
}
const selected = args.useRofi
? showRofiMenu(videos, dir, false, scriptPath, args.logLevel)
: showFzfMenu(videos);
return selected || null;
}
function browseEpisodes(
entry: HistorySeriesEntry,
context: LauncherCommandContext,
themePath: string | null,
): string | null {
const { args } = context;
const seasons = listSeasonDirs(entry.seriesRoot);
let dir = entry.seriesRoot;
if (seasons.length > 1) {
const idx = pickIndex(
seasons.map((season) => season.name),
`${entry.displayName} — Season`,
args.useRofi,
themePath,
);
if (idx < 0) return null;
dir = seasons[idx]!.path;
} else if (seasons.length === 1 && collectVideos(dir, false).length === 0) {
dir = seasons[0]!.path;
}
return pickEpisodeFromDir(dir, context);
}
export async function runHistoryCommand(context: LauncherCommandContext): Promise<string | null> {
const { args, scriptPath } = context;
checkPickerDependencies(args);
const themePath = args.useRofi ? findRofiTheme(scriptPath) : null;
const dbPath = resolveImmersionDbPath();
if (!fs.existsSync(dbPath)) {
fail(`Watch history database not found: ${dbPath}`);
}
const rows = queryLocalWatchHistory(dbPath);
const series = groupHistoryBySeries(rows);
if (series.length === 0) {
fail('No local watch history found (or watched directories are not accessible).');
}
log('info', args.logLevel, `Watch history: ${series.length} series found in ${dbPath}`);
const coverPaths = args.useRofi
? materializeCoverArt(
dbPath,
series.map((seriesEntry) => seriesEntry.coverBlobHash),
)
: new Map<string, string>();
const seriesIcons = series.map((seriesEntry) =>
seriesEntry.coverBlobHash ? (coverPaths.get(seriesEntry.coverBlobHash) ?? null) : null,
);
const seriesIdx = pickIndex(
series.map(formatSeriesLabel),
'Watch History',
args.useRofi,
themePath,
seriesIcons,
);
if (seriesIdx < 0) return null;
const entry = series[seriesIdx]!;
const lastPath = path.resolve(entry.lastWatched.sourcePath);
const lastExists = fs.existsSync(lastPath);
const nextEpisode = findNextEpisode(lastPath);
const actions: Array<{ kind: 'replay' | 'next' | 'browse'; label: string }> = [];
if (lastExists) {
actions.push({ kind: 'replay', label: `Replay last watched — ${path.basename(lastPath)}` });
}
if (nextEpisode) {
actions.push({ kind: 'next', label: `Next episode — ${path.basename(nextEpisode)}` });
}
actions.push({ kind: 'browse', label: 'Browse episodes' });
const entryIcon = seriesIcons[seriesIdx] ?? null;
const actionIdx = pickIndex(
actions.map((action) => action.label),
entry.displayName,
args.useRofi,
themePath,
actions.map(() => entryIcon),
);
if (actionIdx < 0) return null;
switch (actions[actionIdx]!.kind) {
case 'replay':
return lastPath;
case 'next':
return nextEpisode;
case 'browse':
return browseEpisodes(entry, context, themePath);
}
}
@@ -35,6 +35,15 @@ function createContext(): LauncherCommandContext {
texthookerOnly: false,
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncDirection: 'both',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'info',
logRotation: 7,
passwordStore: '',
+257
View File
@@ -0,0 +1,257 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import type { Args } from '../types.js';
import { createEmptyMergeSummary } from '../sync/sync-shared.js';
import type { LauncherCommandContext } from './context.js';
import { ensureTrackerQuiescent, runSyncCommand, type SyncCommandDeps } from './sync-command.js';
function makeContext(overrides: Partial<Args>): LauncherCommandContext {
return {
args: {
sync: true,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncDirection: 'both',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'warn',
...overrides,
} as Args,
scriptPath: '/tmp/subminer',
scriptName: 'subminer',
mpvSocketPath: '',
pluginRuntimeConfig: {},
appPath: null,
launcherJellyfinConfig: {},
processAdapter: process,
} as unknown as LauncherCommandContext;
}
function ok(stdout = ''): { status: number; stdout: string; stderr: string } {
return { status: 0, stdout, stderr: '' };
}
test('ensureTrackerQuiescent ignores stale sockets but rejects live sockets', async () => {
const context = makeContext({ syncDbPath: '/tmp/local.sqlite' });
context.mpvSocketPath = '/tmp/subminer-socket';
let socketConnectable = false;
const deps: Partial<SyncCommandDeps> = {
realpathSync: (() => '/tracker.sqlite') as unknown as typeof fs.realpathSync,
findLiveStatsDaemonPid: () => null,
canConnectUnixSocket: async () => socketConnectable,
fail: (message: string): never => {
throw new Error(message);
},
};
await ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps);
socketConnectable = true;
await assert.rejects(
async () => ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps),
/mpv\/SubMiner session appears to be running/,
);
});
test('runSyncCommand dispatches snapshot, merge, host, and missing-target modes', async () => {
const calls: string[] = [];
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (dbPath: string, outPath: string) => {
calls.push(`snapshot:${dbPath}->${outPath}`);
},
mergeSnapshotIntoDb: (dbPath: string, snapshotPath: string) => {
calls.push(`merge:${dbPath}<-${snapshotPath}`);
return createEmptyMergeSummary();
},
formatMergeSummary: () => 'summary',
ensureTrackerQuiescent: async () => {
calls.push('quiescent');
},
assertSafeSshHost: (host: string) => {
calls.push(`host:${host}`);
},
resolveRemoteSubminerCommand: () => 'subminer',
runSsh: (_host: string, command: string) => {
calls.push(`ssh:${command}`);
return command.startsWith('mktemp ') ? ok('/tmp/subminer-sync.remote\n') : ok();
},
runScp: (from: string, to: string) => {
calls.push(`scp:${from}->${to}`);
},
fail: (message: string): never => {
throw new Error(message);
},
};
assert.equal(
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncSnapshotPath: '/tmp/out.sqlite' }),
deps,
),
true,
);
assert.ok(calls.includes('snapshot:/tmp/local.sqlite->/tmp/out.sqlite'));
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncMergePath: '/tmp/in.sqlite' }),
deps,
);
assert.ok(calls.includes('quiescent'));
assert.ok(calls.includes('merge:/tmp/local.sqlite<-/tmp/in.sqlite'));
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }),
deps,
);
assert.ok(calls.includes('host:media-box'));
await assert.rejects(
() => runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite' }), deps),
/sync requires a host, --snapshot <file>, or --merge <file>/,
);
});
test('runHostSync keeps tracker quiescent through local and remote merge and cleans up after failure', async () => {
const calls: string[] = [];
let localTmpDir = '';
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (_dbPath: string, outPath: string) => {
calls.push(`snapshot:${outPath}`);
fs.writeFileSync(outPath, 'snapshot');
},
mergeSnapshotIntoDb: () => {
calls.push('local-merge');
return createEmptyMergeSummary();
},
formatMergeSummary: () => 'summary',
ensureTrackerQuiescent: async () => {
calls.push('quiescent');
},
assertSafeSshHost: () => {},
resolveRemoteSubminerCommand: () => 'subminer',
mkdtempSync: ((prefix: string) => {
localTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), path.basename(prefix)));
return localTmpDir;
}) as typeof fs.mkdtempSync,
runSsh: (_host: string, command: string) => {
calls.push(`ssh:${command}`);
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
if (command.includes(' sync --snapshot ')) return ok();
if (command.includes(' sync --merge ')) {
return { status: 9, stdout: 'remote output', stderr: 'remote merge exploded' };
}
return ok();
},
runScp: (from: string, to: string) => {
calls.push(`scp:${from}->${to}`);
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
},
};
await assert.rejects(
() =>
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
/Remote merge failed on media-box[\s\S]*remote merge exploded/,
);
assert.equal(calls.filter((call) => call === 'quiescent').length, 3);
assert.ok(calls.indexOf('quiescent') < calls.findIndex((call) => call.startsWith('snapshot:')));
assert.ok(calls.includes('local-merge'));
assert.ok(calls.some((call) => call.startsWith('ssh:rm -rf ')));
assert.equal(fs.existsSync(localTmpDir), false);
});
test('runHostSync includes remote snapshot stderr in failures', async () => {
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (_dbPath: string, outPath: string) => {
fs.writeFileSync(outPath, 'snapshot');
},
ensureTrackerQuiescent: async () => {},
assertSafeSshHost: () => {},
resolveRemoteSubminerCommand: () => 'subminer',
runSsh: (_host: string, command: string) => {
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
if (command.includes(' sync --snapshot ')) {
return { status: 5, stdout: '', stderr: 'snapshot permission denied' };
}
return ok();
},
runScp: () => {},
};
await assert.rejects(
() =>
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
/Remote snapshot failed on media-box[\s\S]*snapshot permission denied/,
);
});
function makeDirectionDeps(calls: string[]): Partial<SyncCommandDeps> {
return {
createDbSnapshot: (_dbPath: string, outPath: string) => {
calls.push(`snapshot:${outPath}`);
fs.writeFileSync(outPath, 'snapshot');
},
mergeSnapshotIntoDb: () => {
calls.push('local-merge');
return createEmptyMergeSummary();
},
formatMergeSummary: () => 'summary',
ensureTrackerQuiescent: async () => {
calls.push('quiescent');
},
assertSafeSshHost: () => {},
resolveRemoteSubminerCommand: () => 'subminer',
runSsh: (_host: string, command: string) => {
calls.push(`ssh:${command}`);
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
return ok();
},
runScp: (from: string, to: string) => {
calls.push(`scp:${from}->${to}`);
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
},
};
}
test('runHostSync push only snapshots locally and merges remotely', async () => {
const calls: string[] = [];
await runSyncCommand(
makeContext({
syncDbPath: '/tmp/local.sqlite',
syncHost: 'media-box',
syncDirection: 'push',
}),
makeDirectionDeps(calls),
);
assert.ok(calls.some((call) => call.startsWith('snapshot:')));
assert.ok(calls.some((call) => call.includes(' sync --merge ')));
assert.ok(calls.some((call) => call.startsWith('scp:') && call.includes('->media-box:')));
assert.ok(!calls.some((call) => call.includes(' sync --snapshot ')));
assert.ok(!calls.includes('local-merge'));
});
test('runHostSync pull only snapshots remotely and merges locally', async () => {
const calls: string[] = [];
await runSyncCommand(
makeContext({
syncDbPath: '/tmp/local.sqlite',
syncHost: 'media-box',
syncDirection: 'pull',
}),
makeDirectionDeps(calls),
);
assert.ok(calls.some((call) => call.includes(' sync --snapshot ')));
assert.ok(calls.some((call) => call.startsWith('scp:media-box:')));
assert.ok(calls.includes('local-merge'));
assert.ok(!calls.some((call) => call.startsWith('snapshot:')));
assert.ok(!calls.some((call) => call.includes(' sync --merge ')));
});
+253
View File
@@ -0,0 +1,253 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fail, log } from '../log.js';
import { resolveImmersionDbPath } from '../history-db.js';
import {
createDbSnapshot,
findLiveStatsDaemonPid,
formatMergeSummary,
mergeSnapshotIntoDb,
} from '../sync/sync-db.js';
import {
assertSafeSshHost,
resolveRemoteSubminerCommand,
runScp,
runSsh,
shellQuote,
} from '../sync/ssh.js';
import { resolvePathMaybe } from '../util.js';
import { canConnectUnixSocket } from '../mpv.js';
import type { LauncherCommandContext } from './context.js';
import type { RemoteRunResult } from '../sync/ssh.js';
export interface SyncCommandDeps {
createDbSnapshot: typeof createDbSnapshot;
mergeSnapshotIntoDb: typeof mergeSnapshotIntoDb;
formatMergeSummary: typeof formatMergeSummary;
findLiveStatsDaemonPid: typeof findLiveStatsDaemonPid;
assertSafeSshHost: typeof assertSafeSshHost;
resolveRemoteSubminerCommand: typeof resolveRemoteSubminerCommand;
runScp: typeof runScp;
runSsh: typeof runSsh;
fail: typeof fail;
log: typeof log;
canConnectUnixSocket: typeof canConnectUnixSocket;
realpathSync: typeof fs.realpathSync;
mkdtempSync: typeof fs.mkdtempSync;
rmSync: typeof fs.rmSync;
consoleLog: typeof console.log;
writeStdout: typeof process.stdout.write;
ensureTrackerQuiescent: (context: LauncherCommandContext, dbPath: string) => Promise<void>;
}
function resolveDbPath(context: LauncherCommandContext): string {
const override = context.args.syncDbPath.trim();
return override ? resolvePathMaybe(override) : resolveImmersionDbPath();
}
function isTrackerDb(dbPath: string, deps: SyncCommandDeps): boolean {
const trackerDbPath = resolveImmersionDbPath();
try {
return deps.realpathSync(dbPath) === deps.realpathSync(trackerDbPath);
} catch {
return dbPath === trackerDbPath;
}
}
export async function ensureTrackerQuiescent(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
if (context.args.syncForce) return;
// A running SubMiner only holds the tracker's own database; --db pointed
// elsewhere needs no guard.
if (!isTrackerDb(dbPath, deps)) return;
const daemonPid = deps.findLiveStatsDaemonPid(dbPath);
if (daemonPid !== null) {
deps.fail(
`The SubMiner stats server is running (pid ${daemonPid}). Stop it with "subminer stats -s" (or close SubMiner) before syncing, or pass --force.`,
);
}
if (context.mpvSocketPath && (await deps.canConnectUnixSocket(context.mpvSocketPath))) {
deps.fail(
`An mpv/SubMiner session appears to be running (socket ${context.mpvSocketPath}). Close it before syncing, or pass --force.`,
);
}
}
const defaultSyncCommandDeps: SyncCommandDeps = {
createDbSnapshot,
mergeSnapshotIntoDb,
formatMergeSummary,
findLiveStatsDaemonPid,
assertSafeSshHost,
resolveRemoteSubminerCommand,
runScp,
runSsh,
fail,
log,
canConnectUnixSocket,
realpathSync: fs.realpathSync,
mkdtempSync: fs.mkdtempSync,
rmSync: fs.rmSync,
consoleLog: console.log,
writeStdout: process.stdout.write.bind(process.stdout),
ensureTrackerQuiescent: async (context, dbPath) => ensureTrackerQuiescent(context, dbPath),
};
function resolveSyncCommandDeps(inputDeps: Partial<SyncCommandDeps> = {}): SyncCommandDeps {
return { ...defaultSyncCommandDeps, ...inputDeps };
}
export function runSnapshotMode(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): void {
const deps = resolveSyncCommandDeps(inputDeps);
const outPath = resolvePathMaybe(context.args.syncSnapshotPath);
deps.createDbSnapshot(dbPath, outPath);
deps.consoleLog(outPath);
}
export async function runMergeMode(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
await deps.ensureTrackerQuiescent(context, dbPath);
const snapshotPath = resolvePathMaybe(context.args.syncMergePath);
const summary = deps.mergeSnapshotIntoDb(dbPath, snapshotPath);
deps.consoleLog(deps.formatMergeSummary(summary));
}
function cleanupRemote(host: string, remoteTmpDir: string, deps: SyncCommandDeps): void {
if (!remoteTmpDir.startsWith('/tmp/')) return;
deps.runSsh(host, `rm -rf ${shellQuote(remoteTmpDir)}`);
}
function formatRemoteRunError(message: string, run: RemoteRunResult): string {
const stderr = run.stderr.trim();
return stderr ? `${message}\n${stderr}` : message;
}
export async function runHostSync(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
const { args } = context;
const host = args.syncHost;
const direction = args.syncDirection ?? 'both';
const shouldPull = direction !== 'push';
const shouldPush = direction !== 'pull';
deps.assertSafeSshHost(host);
await deps.ensureTrackerQuiescent(context, dbPath);
const remoteCmd = deps.resolveRemoteSubminerCommand(host, args.syncRemoteCmd || null);
deps.log('debug', args.logLevel, `Remote subminer command: ${remoteCmd}`);
const localTmpDir = deps.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-'));
let remoteTmpDir = '';
try {
// Signal failures by throwing (not fail(), which exits synchronously and
// would skip the finally cleanup, leaking temp dirs holding snapshot data).
// main().catch() reports the message the same way fail() would.
const mktemp = deps.runSsh(host, 'mktemp -d /tmp/subminer-sync.XXXXXX');
remoteTmpDir = mktemp.stdout.trim();
if (mktemp.status !== 0 || !remoteTmpDir.startsWith('/tmp/')) {
throw new Error(`Could not create a temporary directory on ${host}.`);
}
const forceFlag = args.syncForce ? ' --force' : '';
const localSnapshot = path.join(localTmpDir, 'local.sqlite');
if (shouldPush) {
deps.consoleLog(`Snapshotting local database (${dbPath})...`);
deps.createDbSnapshot(dbPath, localSnapshot);
}
const remoteSnapshot = `${remoteTmpDir}/snapshot.sqlite`;
if (shouldPull) {
deps.consoleLog(`Snapshotting ${host}...`);
const snapshotRun = deps.runSsh(
host,
`${remoteCmd} sync --snapshot ${shellQuote(remoteSnapshot)}${forceFlag}`,
);
if (snapshotRun.status !== 0) {
throw new Error(formatRemoteRunError(`Remote snapshot failed on ${host}.`, snapshotRun));
}
}
const pulledSnapshot = path.join(localTmpDir, 'remote.sqlite');
if (shouldPull) deps.runScp(`${host}:${remoteSnapshot}`, pulledSnapshot);
const incomingSnapshot = `${remoteTmpDir}/incoming.sqlite`;
if (shouldPush) deps.runScp(localSnapshot, `${host}:${incomingSnapshot}`);
if (shouldPull) {
deps.consoleLog(`\nMerging ${host} -> local:`);
await deps.ensureTrackerQuiescent(context, dbPath);
const summary = deps.mergeSnapshotIntoDb(dbPath, pulledSnapshot);
deps.consoleLog(deps.formatMergeSummary(summary));
}
if (shouldPush) {
deps.consoleLog(`\nMerging local -> ${host}:`);
await deps.ensureTrackerQuiescent(context, dbPath);
const mergeRun = deps.runSsh(
host,
`${remoteCmd} sync --merge ${shellQuote(incomingSnapshot)}${forceFlag}`,
);
deps.writeStdout(mergeRun.stdout);
if (mergeRun.status !== 0) {
const retryCommand =
direction === 'push' ? `subminer sync ${host} --push` : `subminer sync ${host}`;
const localUpdate = shouldPull ? ' The local database was updated;' : '';
throw new Error(
formatRemoteRunError(
`Remote merge failed on ${host}.${localUpdate} re-run "${retryCommand}" once the remote issue is fixed.`,
mergeRun,
),
);
}
}
deps.consoleLog('\nSync complete.');
} finally {
deps.rmSync(localTmpDir, { recursive: true, force: true });
if (remoteTmpDir) {
try {
cleanupRemote(host, remoteTmpDir, deps);
} catch {
// best effort
}
}
}
}
export async function runSyncCommand(
context: LauncherCommandContext,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<boolean> {
const deps = resolveSyncCommandDeps(inputDeps);
const { args } = context;
if (!args.sync) return false;
const dbPath = resolveDbPath(context);
if (args.syncSnapshotPath) {
runSnapshotMode(context, dbPath, deps);
} else if (args.syncMergePath) {
await runMergeMode(context, dbPath, deps);
} else if (args.syncHost) {
await runHostSync(context, dbPath, deps);
} else {
deps.fail('sync requires a host, --snapshot <file>, or --merge <file>.');
}
return true;
}
+36
View File
@@ -135,6 +135,15 @@ test('applyInvocationsToArgs maps config and jellyfin invocation state', () => {
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncDirection: 'both',
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -181,6 +190,15 @@ test('applyInvocationsToArgs maps settings invocation to settings window', () =>
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncDirection: 'both',
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -220,6 +238,15 @@ test('applyInvocationsToArgs fails when config invocation has no action', () =>
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncDirection: 'both',
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -257,6 +284,15 @@ test('applyInvocationsToArgs maps texthooker browser-open request', () => {
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncDirection: 'both',
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
+21
View File
@@ -198,6 +198,15 @@ export function createDefaultArgs(
texthookerOnly: false,
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncDirection: 'both',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: loggingConfig.level ?? 'warn',
logRotation: loggingConfig.rotation ?? 7,
passwordStore: '',
@@ -231,6 +240,7 @@ export function applyRootOptionsToArgs(
if (typeof options.logLevel === 'string') parsed.logLevel = parseLogLevel(options.logLevel);
if (typeof options.passwordStore === 'string') parsed.passwordStore = options.passwordStore;
if (options.rofi === true) parsed.useRofi = true;
if (options.history === true) parsed.history = true;
if (options.update === true) parsed.update = true;
if (options.version === true) parsed.version = true;
if (options.settings === true) parsed.settings = true;
@@ -262,6 +272,17 @@ export function applyInvocationsToArgs(parsed: Args, invocations: CliInvocations
) {
fail('Dictionary target path is required.');
}
if (invocations.syncTriggered) {
parsed.sync = true;
parsed.syncHost = invocations.syncHost ?? '';
parsed.syncSnapshotPath = invocations.syncSnapshotPath ?? '';
parsed.syncMergePath = invocations.syncMergePath ?? '';
parsed.syncDirection = invocations.syncDirection;
parsed.syncRemoteCmd = invocations.syncRemoteCmd ?? '';
parsed.syncDbPath = invocations.syncDbPath ?? '';
parsed.syncForce = invocations.syncForce;
if (invocations.syncLogLevel) parsed.logLevel = parseLogLevel(invocations.syncLogLevel);
}
if (invocations.doctorTriggered) parsed.doctor = true;
if (invocations.doctorRefreshKnownWords) parsed.doctorRefreshKnownWords = true;
if (invocations.logsTriggered && !invocations.logsExport) {
@@ -42,3 +42,22 @@ test('parseCliPrograms captures texthooker browser-open flag', () => {
assert.equal(result.invocations.texthookerTriggered, true);
assert.equal(result.invocations.texthookerOpenBrowser, true);
});
test('parseCliPrograms captures one-way sync directions', () => {
const push = parseCliPrograms(['sync', 'media-box', '--push'], 'subminer');
const pull = parseCliPrograms(['sync', 'media-box', '--pull'], 'subminer');
assert.equal(push.invocations.syncDirection, 'push');
assert.equal(pull.invocations.syncDirection, 'pull');
});
test('parseCliPrograms rejects conflicting or hostless one-way sync directions', () => {
assert.throws(
() => parseCliPrograms(['sync', 'media-box', '--push', '--pull'], 'subminer'),
/--push and --pull cannot be combined/,
);
assert.throws(
() => parseCliPrograms(['sync', '--snapshot', '/tmp/stats.sqlite', '--push'], 'subminer'),
/--push and --pull require a host/,
);
});
+72
View File
@@ -38,6 +38,15 @@ export interface CliInvocations {
statsCleanupVocab: boolean;
statsCleanupLifetime: boolean;
statsLogLevel: string | null;
syncTriggered: boolean;
syncHost: string | null;
syncSnapshotPath: string | null;
syncMergePath: string | null;
syncDirection: 'both' | 'push' | 'pull';
syncRemoteCmd: string | null;
syncDbPath: string | null;
syncForce: boolean;
syncLogLevel: string | null;
doctorTriggered: boolean;
doctorLogLevel: string | null;
doctorRefreshKnownWords: boolean;
@@ -64,6 +73,7 @@ function applyRootOptions(program: Command): void {
.option('--settings', 'Open settings window')
.option('-u, --update', 'Check for updates')
.option('-R, --rofi', 'Use rofi picker')
.option('-H, --history', 'Browse local watch history')
.option('-S, --start-overlay', 'Auto-start overlay')
.option('-T, --no-texthooker', 'Disable texthooker-ui server');
}
@@ -97,6 +107,7 @@ function getTopLevelCommand(argv: string[]): { name: string; index: number } | n
'dictionary',
'dict',
'stats',
'sync',
'texthooker',
'app',
'bin',
@@ -159,6 +170,15 @@ export function parseCliPrograms(
let statsCleanupVocab = false;
let statsCleanupLifetime = false;
let statsLogLevel: string | null = null;
let syncTriggered = false;
let syncHost: string | null = null;
let syncSnapshotPath: string | null = null;
let syncMergePath: string | null = null;
let syncDirection: 'both' | 'push' | 'pull' = 'both';
let syncRemoteCmd: string | null = null;
let syncDbPath: string | null = null;
let syncForce = false;
let syncLogLevel: string | null = null;
let doctorLogLevel: string | null = null;
let doctorRefreshKnownWords = false;
let logsTriggered = false;
@@ -288,6 +308,49 @@ export function parseCliPrograms(
statsLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
});
commandProgram
.command('sync')
.description('Sync stats and watch history with another machine over SSH')
.argument('[host]', 'SSH destination (user@host or an ssh config alias)')
.option('--snapshot <file>', 'Write a consistent snapshot of the local stats database')
.option('--merge <file>', 'Merge a snapshot database file into the local stats database')
.option('--push', 'Only merge local stats into the SSH host')
.option('--pull', 'Only merge stats from the SSH host into the local database')
.option('--db <file>', 'Override the local stats database path')
.option('--remote-cmd <cmd>', 'subminer command to run on the remote host')
.option('-f, --force', 'Skip the running-app safety check')
.option('--log-level <level>', 'Log level')
.action((rawHost: string | undefined, options: Record<string, unknown>) => {
const host = typeof rawHost === 'string' ? rawHost.trim() : '';
const snapshot = typeof options.snapshot === 'string' ? options.snapshot.trim() : '';
const merge = typeof options.merge === 'string' ? options.merge.trim() : '';
const push = options.push === true;
const pull = options.pull === true;
if (push && pull) {
throw new Error('Sync --push and --pull cannot be combined.');
}
if ((push || pull) && !host) {
throw new Error('Sync --push and --pull require a host.');
}
const modes = [Boolean(host), Boolean(snapshot), Boolean(merge)].filter(Boolean).length;
if (modes === 0) {
throw new Error('Sync requires a host, --snapshot <file>, or --merge <file>.');
}
if (modes > 1) {
throw new Error('Sync host, --snapshot, and --merge cannot be combined.');
}
syncTriggered = true;
syncHost = host || null;
syncSnapshotPath = snapshot || null;
syncMergePath = merge || null;
syncDirection = push ? 'push' : pull ? 'pull' : 'both';
syncRemoteCmd =
typeof options.remoteCmd === 'string' ? options.remoteCmd.trim() || null : null;
syncDbPath = typeof options.db === 'string' ? options.db.trim() || null : null;
syncForce = options.force === true;
syncLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
});
commandProgram
.command('doctor')
.description('Run dependency and environment checks')
@@ -399,6 +462,15 @@ export function parseCliPrograms(
statsCleanupVocab,
statsCleanupLifetime,
statsLogLevel,
syncTriggered,
syncHost,
syncSnapshotPath,
syncMergePath,
syncDirection,
syncRemoteCmd,
syncDbPath,
syncForce,
syncLogLevel,
doctorTriggered,
doctorLogLevel,
doctorRefreshKnownWords,
+126
View File
@@ -0,0 +1,126 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { withReadonlyWalRetry } from './history-db.js';
const COVER_EXTENSIONS = ['.jpg', '.png', '.webp', '.gif'] as const;
const SAFE_COVER_HASH_PATTERN = /^[a-z0-9_-]+$/i;
export function getDefaultCoverCacheDir(): string {
return path.join(os.homedir(), '.cache', 'subminer', 'covers');
}
export function detectImageExtension(blob: Buffer): string {
if (blob.length >= 8 && blob.subarray(0, 8).equals(Buffer.from('89504e470d0a1a0a', 'hex'))) {
return '.png';
}
if (blob.length >= 3 && blob[0] === 0xff && blob[1] === 0xd8 && blob[2] === 0xff) {
return '.jpg';
}
if (
blob.length >= 12 &&
blob.subarray(0, 4).toString('ascii') === 'RIFF' &&
blob.subarray(8, 12).toString('ascii') === 'WEBP'
) {
return '.webp';
}
if (blob.length >= 4 && blob.subarray(0, 3).toString('ascii') === 'GIF') {
return '.gif';
}
return '.jpg';
}
function findCachedCover(cacheDir: string, hash: string): string | null {
for (const ext of COVER_EXTENSIONS) {
const candidate = path.join(cacheDir, `${hash}${ext}`);
try {
if (fs.statSync(candidate).size > 0) return candidate;
} catch {
// not cached with this extension
}
}
return null;
}
function queryCoverBlobs(
dbPath: string,
hashes: string[],
options: { readonly?: boolean; readwrite?: boolean; create?: boolean },
): Map<string, Buffer> {
const blobs = new Map<string, Buffer>();
const db = new Database(dbPath, options);
try {
const hasBlobTable = db
.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'imm_cover_art_blobs'`)
.get();
if (!hasBlobTable) return blobs;
const stmt = db.query<{ cover_blob: Uint8Array | null }>(
'SELECT cover_blob FROM imm_cover_art_blobs WHERE blob_hash = ?',
);
for (const hash of hashes) {
const row = stmt.get(hash);
if (row?.cover_blob && row.cover_blob.length > 0) {
blobs.set(hash, Buffer.from(row.cover_blob));
}
}
return blobs;
} finally {
db.close();
}
}
function isSafeCoverHash(hash: string | null | undefined): hash is string {
return typeof hash === 'string' && SAFE_COVER_HASH_PATTERN.test(hash);
}
/**
* Ensures cover art blobs referenced by hash exist as image files in the cache
* directory, extracting missing ones from the stats database. Returns a map of
* blob hash to on-disk image path for every cover that could be materialized.
*/
export function materializeCoverArt(
dbPath: string,
hashes: Array<string | null | undefined>,
cacheDir: string = getDefaultCoverCacheDir(),
): Map<string, string> {
const wanted = Array.from(new Set(hashes.filter(isSafeCoverHash)));
const resolved = new Map<string, string>();
if (wanted.length === 0) return resolved;
const missing: string[] = [];
for (const hash of wanted) {
const cached = findCachedCover(cacheDir, hash);
if (cached) {
resolved.set(hash, cached);
} else {
missing.push(hash);
}
}
if (missing.length === 0) return resolved;
let blobs: Map<string, Buffer>;
try {
blobs = withReadonlyWalRetry(dbPath, (options) => queryCoverBlobs(dbPath, missing, options));
} catch {
return resolved;
}
if (blobs.size === 0) return resolved;
try {
fs.mkdirSync(cacheDir, { recursive: true });
} catch {
return resolved;
}
for (const [hash, blob] of blobs) {
const target = path.join(cacheDir, `${hash}${detectImageExtension(blob)}`);
try {
fs.writeFileSync(target, blob);
resolved.set(hash, target);
} catch {
// cache write failure just means no icon for this entry
}
}
return resolved;
}
+151
View File
@@ -0,0 +1,151 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { resolveConfigDir } from '../src/config/path-resolution.js';
import { readLauncherMainConfigObject } from './config/shared-config-reader.js';
import type { HistoryVideoRow } from './history-types.js';
import { resolvePathMaybe } from './util.js';
export function resolveImmersionDbPath(): string {
const root = readLauncherMainConfigObject();
const tracking =
root?.immersionTracking &&
typeof root.immersionTracking === 'object' &&
!Array.isArray(root.immersionTracking)
? (root.immersionTracking as Record<string, unknown>)
: null;
const configured = typeof tracking?.dbPath === 'string' ? tracking.dbPath.trim() : '';
if (configured) return resolvePathMaybe(configured);
const configDir = resolveConfigDir({
platform: process.platform,
appDataDir: process.env.APPDATA,
xdgConfigHome: process.env.XDG_CONFIG_HOME,
homeDir: os.homedir(),
existsSync: fs.existsSync,
});
return path.join(configDir, 'immersion.sqlite');
}
interface RawHistoryRow {
video_id: number;
source_path: string | null;
parsed_title: string | null;
parsed_season: number | null;
parsed_episode: number | null;
anime_title: string | null;
last_watched_ms: number | bigint | null;
cover_blob_hash: string | null;
}
export function queryLocalWatchHistory(dbPath: string): HistoryVideoRow[] {
return withReadonlyWalRetry(dbPath, (options) => readHistoryRows(dbPath, options));
}
export function withReadonlyWalRetry<T>(
dbPath: string,
query: (options: { readonly?: boolean; readwrite?: boolean; create?: boolean }) => T,
): T {
try {
return query({ readonly: true });
} catch (error) {
if (!isReadonlyWalRetryError(error, dbPath)) throw error;
return query({ readwrite: true, create: false });
}
}
export function isReadonlyWalRetryError(error: unknown, dbPath: string): boolean {
if (!isWalModeSqliteDatabase(dbPath)) return false;
const code =
typeof error === 'object' && error !== null && 'code' in error
? String((error as { code?: unknown }).code ?? '')
: '';
const message = error instanceof Error ? error.message : String(error);
const text = `${code} ${message}`.toLowerCase();
return (
text.includes('readonly') ||
text.includes('read-only') ||
text.includes('attempt to write a readonly database') ||
text.includes('sqlite_cantopen') ||
text.includes('unable to open database file')
);
}
function isWalModeSqliteDatabase(dbPath: string): boolean {
const header = Buffer.alloc(20);
let fd: number | null = null;
try {
fd = fs.openSync(dbPath, 'r');
if (fs.readSync(fd, header, 0, header.length, 0) < header.length) return false;
} catch {
return false;
} finally {
if (fd !== null) fs.closeSync(fd);
}
return header.subarray(0, 16).toString('ascii') === 'SQLite format 3\0' && header[18] === 2;
}
function tableExists(db: Database, tableName: string): boolean {
return Boolean(
db.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?`).get(tableName),
);
}
function readHistoryRows(
dbPath: string,
options: { readonly?: boolean; readwrite?: boolean; create?: boolean },
): HistoryVideoRow[] {
const db = new Database(dbPath, options);
try {
const hasMediaArt = tableExists(db, 'imm_media_art');
const coverSelect = hasMediaArt
? `COALESCE(
ma.cover_blob_hash,
(SELECT ma2.cover_blob_hash
FROM imm_media_art ma2
JOIN imm_videos v2 ON v2.video_id = ma2.video_id
WHERE v2.anime_id = v.anime_id AND ma2.cover_blob_hash IS NOT NULL
LIMIT 1)
) AS cover_blob_hash`
: 'NULL AS cover_blob_hash';
const coverJoin = hasMediaArt ? 'LEFT JOIN imm_media_art ma ON ma.video_id = v.video_id' : '';
const rows = db
.query<RawHistoryRow>(
`
SELECT
v.video_id,
v.source_path,
v.parsed_title,
v.parsed_season,
v.parsed_episode,
COALESCE(a.title_romaji, a.canonical_title) AS anime_title,
MAX(CAST(s.started_at_ms AS INTEGER)) AS last_watched_ms,
${coverSelect}
FROM imm_sessions s
JOIN imm_videos v ON v.video_id = s.video_id
LEFT JOIN imm_anime a ON a.anime_id = v.anime_id
${coverJoin}
WHERE v.source_type = 1 AND v.source_path IS NOT NULL AND v.source_path != ''
GROUP BY v.video_id
ORDER BY last_watched_ms DESC
`,
)
.all();
return rows
.filter((row) => typeof row.source_path === 'string' && row.source_path.length > 0)
.map((row) => ({
videoId: row.video_id,
sourcePath: row.source_path!,
parsedTitle: row.parsed_title,
parsedSeason: row.parsed_season,
parsedEpisode: row.parsed_episode,
animeTitle: row.anime_title,
lastWatchedMs: Number(row.last_watched_ms ?? 0),
coverBlobHash: row.cover_blob_hash,
}));
} finally {
db.close();
}
}
+132
View File
@@ -0,0 +1,132 @@
import fs from 'node:fs';
import path from 'node:path';
import { parseMediaInfo } from '../src/jimaku/utils.js';
import { collectVideos } from './picker.js';
import type { HistorySeriesEntry, HistoryVideoRow, SeasonDirEntry } from './history-types.js';
const SEASON_DIR_PATTERN = /^(?:season|s)[\s._-]*(\d{1,3})\b/i;
export function seasonNumberFromDirName(name: string): number | null {
const match = name.trim().match(SEASON_DIR_PATTERN);
if (!match) return null;
const parsed = Number.parseInt(match[1]!, 10);
return Number.isFinite(parsed) ? parsed : null;
}
export function resolveSeriesRoot(filePath: string): string {
const parent = path.dirname(filePath);
if (seasonNumberFromDirName(path.basename(parent)) !== null) {
return path.dirname(parent);
}
return parent;
}
export function groupHistoryBySeries(
rows: HistoryVideoRow[],
existsFn: (candidate: string) => boolean = fs.existsSync,
): HistorySeriesEntry[] {
const byRoot = new Map<string, HistorySeriesEntry>();
const sorted = [...rows].sort((a, b) => b.lastWatchedMs - a.lastWatchedMs);
for (const row of sorted) {
const seriesRoot = resolveSeriesRoot(row.sourcePath);
const existing = byRoot.get(seriesRoot);
if (existing) {
if (existing.coverBlobHash === null && row.coverBlobHash !== null) {
existing.coverBlobHash = row.coverBlobHash;
}
continue;
}
if (!existsFn(seriesRoot)) continue;
const displayName =
row.parsedTitle?.trim() || row.animeTitle?.trim() || path.basename(seriesRoot);
byRoot.set(seriesRoot, {
seriesRoot,
displayName,
lastWatched: row,
coverBlobHash: row.coverBlobHash,
});
}
return Array.from(byRoot.values());
}
function compareNatural(a: string, b: string): number {
return a.localeCompare(b, undefined, { numeric: true, sensitivity: 'base' });
}
export function sortVideosByEpisode(videos: string[]): string[] {
const parsed = videos.map((video) => ({ video, info: parseMediaInfo(video) }));
parsed.sort((a, b) => {
if (a.info.episode !== null && b.info.episode !== null) {
const seasonA = a.info.season ?? 0;
const seasonB = b.info.season ?? 0;
if (seasonA !== seasonB) return seasonA - seasonB;
if (a.info.episode !== b.info.episode) return a.info.episode - b.info.episode;
}
return compareNatural(a.video, b.video);
});
return parsed.map((entry) => entry.video);
}
function dirContainsVideo(dir: string): boolean {
return collectVideos(dir, true).length > 0;
}
export function listSeasonDirs(seriesRoot: string): SeasonDirEntry[] {
let entries: fs.Dirent[];
try {
entries = fs.readdirSync(seriesRoot, { withFileTypes: true });
} catch {
return [];
}
const dirs = entries
.filter((entry) => entry.isDirectory())
.map((entry) => ({
name: entry.name,
path: path.join(seriesRoot, entry.name),
season: seasonNumberFromDirName(entry.name),
}))
.filter((entry) => dirContainsVideo(entry.path));
dirs.sort((a, b) => {
if (a.season !== null && b.season !== null && a.season !== b.season) {
return a.season - b.season;
}
return compareNatural(a.name, b.name);
});
return dirs;
}
function findFirstEpisodeInNextSeason(resolvedLast: string, dir: string): string | null {
const seriesRoot = resolveSeriesRoot(resolvedLast);
if (seriesRoot === dir) return null;
const seasons = listSeasonDirs(seriesRoot);
const currentIdx = seasons.findIndex((season) => path.resolve(season.path) === dir);
if (currentIdx < 0 || currentIdx + 1 >= seasons.length) return null;
const nextSeason = sortVideosByEpisode(collectVideos(seasons[currentIdx + 1]!.path, false));
return nextSeason[0] ?? null;
}
export function findNextEpisode(lastPath: string): string | null {
const resolvedLast = path.resolve(lastPath);
const dir = path.dirname(resolvedLast);
const episodes = sortVideosByEpisode(collectVideos(dir, false));
const idx = episodes.indexOf(resolvedLast);
if (idx >= 0) {
if (idx + 1 < episodes.length) return episodes[idx + 1]!;
} else {
const lastInfo = parseMediaInfo(resolvedLast);
if (lastInfo.episode !== null) {
const candidate = episodes.find((episode) => {
const info = parseMediaInfo(episode);
return info.episode !== null && info.episode > lastInfo.episode!;
});
if (candidate) return candidate;
}
}
return findFirstEpisodeInNextSeason(resolvedLast, dir);
}
+23
View File
@@ -0,0 +1,23 @@
export interface HistoryVideoRow {
videoId: number;
sourcePath: string;
parsedTitle: string | null;
parsedSeason: number | null;
parsedEpisode: number | null;
animeTitle: string | null;
lastWatchedMs: number;
coverBlobHash: string | null;
}
export interface HistorySeriesEntry {
seriesRoot: string;
displayName: string;
lastWatched: HistoryVideoRow;
coverBlobHash: string | null;
}
export interface SeasonDirEntry {
name: string;
path: string;
season: number | null;
}
+441
View File
@@ -0,0 +1,441 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import {
detectImageExtension,
findNextEpisode,
groupHistoryBySeries,
isReadonlyWalRetryError,
listSeasonDirs,
materializeCoverArt,
queryLocalWatchHistory,
resolveSeriesRoot,
seasonNumberFromDirName,
sortVideosByEpisode,
type HistoryVideoRow,
} from './history.js';
function makeRow(overrides: Partial<HistoryVideoRow> = {}): HistoryVideoRow {
return {
videoId: 1,
sourcePath: '/media/anime/Show/Season-1/Show - S01E01.mkv',
parsedTitle: 'Show',
parsedSeason: 1,
parsedEpisode: 1,
animeTitle: null,
lastWatchedMs: 1000,
coverBlobHash: null,
...overrides,
};
}
test('seasonNumberFromDirName detects common season directory names', () => {
assert.equal(seasonNumberFromDirName('Season-1'), 1);
assert.equal(seasonNumberFromDirName('Season 2'), 2);
assert.equal(seasonNumberFromDirName('S03'), 3);
assert.equal(seasonNumberFromDirName('season_04'), 4);
assert.equal(seasonNumberFromDirName('Specials'), null);
assert.equal(seasonNumberFromDirName('Show Name'), null);
});
test('resolveSeriesRoot skips season directories', () => {
assert.equal(
resolveSeriesRoot('/media/anime/Show/Season-1/Show - S01E01.mkv'),
'/media/anime/Show',
);
assert.equal(resolveSeriesRoot('/media/anime/Show/Show - 01.mkv'), '/media/anime/Show');
});
test('groupHistoryBySeries keeps most recent entry per series root', () => {
const rows = [
makeRow({ videoId: 1, parsedEpisode: 1, lastWatchedMs: 1000 }),
makeRow({
videoId: 2,
sourcePath: '/media/anime/Show/Season-1/Show - S01E02.mkv',
parsedEpisode: 2,
lastWatchedMs: 3000,
}),
makeRow({
videoId: 3,
sourcePath: '/media/anime/Other/Other - 05.mkv',
parsedTitle: 'Other',
parsedSeason: null,
parsedEpisode: 5,
lastWatchedMs: 2000,
}),
];
const series = groupHistoryBySeries(rows, () => true);
assert.equal(series.length, 2);
assert.equal(series[0]?.displayName, 'Show');
assert.equal(series[0]?.seriesRoot, '/media/anime/Show');
assert.equal(series[0]?.lastWatched.parsedEpisode, 2);
assert.equal(series[1]?.displayName, 'Other');
});
test('groupHistoryBySeries filters series roots that no longer exist', () => {
const rows = [
makeRow({ videoId: 1 }),
makeRow({
videoId: 2,
sourcePath: '/gone/anime/Missing/Season-1/Missing - S01E01.mkv',
parsedTitle: 'Missing',
lastWatchedMs: 5000,
}),
];
const series = groupHistoryBySeries(rows, (candidate) => !candidate.startsWith('/gone/'));
assert.equal(series.length, 1);
assert.equal(series[0]?.displayName, 'Show');
});
test('groupHistoryBySeries falls back to directory name for display', () => {
const rows = [
makeRow({
sourcePath: '/media/anime/Some Show Dir/video.mkv',
parsedTitle: null,
animeTitle: null,
}),
];
const series = groupHistoryBySeries(rows, () => true);
assert.equal(series[0]?.displayName, 'Some Show Dir');
});
test('sortVideosByEpisode orders by parsed episode with natural fallback', () => {
const videos = [
'/media/Show/Show - S01E10 - Ten.mkv',
'/media/Show/Show - S01E02 - Two.mkv',
'/media/Show/Show - S01E01 - One.mkv',
];
assert.deepEqual(sortVideosByEpisode(videos), [
'/media/Show/Show - S01E01 - One.mkv',
'/media/Show/Show - S01E02 - Two.mkv',
'/media/Show/Show - S01E10 - Ten.mkv',
]);
});
function createSeriesTree(): string {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-'));
const seriesRoot = path.join(root, 'Show');
const season1 = path.join(seriesRoot, 'Season-1');
const season2 = path.join(seriesRoot, 'Season-2');
fs.mkdirSync(season1, { recursive: true });
fs.mkdirSync(season2, { recursive: true });
fs.mkdirSync(path.join(seriesRoot, 'extras-empty'), { recursive: true });
for (const name of ['Show - S01E01.mkv', 'Show - S01E02.mkv', 'Show - S01E03.mkv']) {
fs.writeFileSync(path.join(season1, name), '');
}
fs.writeFileSync(path.join(season2, 'Show - S02E01.mkv'), '');
fs.writeFileSync(path.join(season1, 'notes.txt'), '');
return seriesRoot;
}
test('listSeasonDirs returns only video-bearing directories in season order', () => {
const seriesRoot = createSeriesTree();
try {
const seasons = listSeasonDirs(seriesRoot);
assert.deepEqual(
seasons.map((entry) => entry.name),
['Season-1', 'Season-2'],
);
assert.deepEqual(
seasons.map((entry) => entry.season),
[1, 2],
);
} finally {
fs.rmSync(path.dirname(seriesRoot), { recursive: true, force: true });
}
});
test('findNextEpisode advances within a season and across seasons', () => {
const seriesRoot = createSeriesTree();
try {
const season1 = path.join(seriesRoot, 'Season-1');
const season2 = path.join(seriesRoot, 'Season-2');
assert.equal(
findNextEpisode(path.join(season1, 'Show - S01E02.mkv')),
path.join(season1, 'Show - S01E03.mkv'),
);
assert.equal(
findNextEpisode(path.join(season1, 'Show - S01E03.mkv')),
path.join(season2, 'Show - S02E01.mkv'),
);
assert.equal(findNextEpisode(path.join(season2, 'Show - S02E01.mkv')), null);
} finally {
fs.rmSync(path.dirname(seriesRoot), { recursive: true, force: true });
}
});
test('findNextEpisode falls back to episode numbers when file was removed', () => {
const seriesRoot = createSeriesTree();
try {
const season1 = path.join(seriesRoot, 'Season-1');
const missing = path.join(season1, 'Show - S01E02 - Deleted Cut.mkv');
assert.equal(findNextEpisode(missing), path.join(season1, 'Show - S01E03.mkv'));
} finally {
fs.rmSync(path.dirname(seriesRoot), { recursive: true, force: true });
}
});
test('findNextEpisode advances seasons when a deleted file was the last episode', () => {
const seriesRoot = createSeriesTree();
try {
const season1 = path.join(seriesRoot, 'Season-1');
const season2 = path.join(seriesRoot, 'Season-2');
const missing = path.join(season1, 'Show - S01E03 - Deleted Cut.mkv');
assert.equal(findNextEpisode(missing), path.join(season2, 'Show - S02E01.mkv'));
} finally {
fs.rmSync(path.dirname(seriesRoot), { recursive: true, force: true });
}
});
const PNG_MAGIC = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
function createHistoryDb(
dbPath: string,
options: { wal?: boolean; coverArt?: boolean } = {},
): void {
const db = new Database(dbPath);
try {
if (options.wal) db.run('PRAGMA journal_mode = WAL;');
db.run(`
CREATE TABLE imm_anime(
anime_id INTEGER PRIMARY KEY,
canonical_title TEXT,
title_romaji TEXT
);
`);
db.run(`
CREATE TABLE imm_videos(
video_id INTEGER PRIMARY KEY,
anime_id INTEGER,
source_type INTEGER,
source_path TEXT,
parsed_title TEXT,
parsed_season INTEGER,
parsed_episode INTEGER
);
`);
db.run(`
CREATE TABLE imm_sessions(
session_id INTEGER PRIMARY KEY,
video_id INTEGER,
started_at_ms TEXT
);
`);
db.run(`INSERT INTO imm_anime VALUES (1, 'Show Season 1', 'Show Romaji');`);
db.run(`
INSERT INTO imm_videos VALUES
(1, 1, 1, '/media/Show/Season-1/Show - S01E01.mkv', 'Show', 1, 1),
(2, 1, 1, '/media/Show/Season-1/Show - S01E02.mkv', 'Show', 1, 2),
(3, NULL, 2, NULL, 'Remote Show', NULL, NULL),
(4, NULL, 1, '', 'Empty Path', NULL, NULL);
`);
db.run(`
INSERT INTO imm_sessions VALUES
(1, 1, '1000'),
(2, 1, '5000'),
(3, 2, '3000'),
(4, 3, '9000');
`);
if (options.coverArt) {
db.run(`
CREATE TABLE imm_media_art(
video_id INTEGER PRIMARY KEY,
cover_blob_hash TEXT
);
`);
db.run(`
CREATE TABLE imm_cover_art_blobs(
blob_hash TEXT PRIMARY KEY,
cover_blob BLOB NOT NULL
);
`);
// Art only on video 1; video 2 resolves it through the shared anime_id.
db.run(`INSERT INTO imm_media_art VALUES (1, 'hash-1');`);
db.query('INSERT INTO imm_cover_art_blobs VALUES (?, ?)').run('hash-1', PNG_MAGIC);
}
if (options.wal) db.run('PRAGMA wal_checkpoint(TRUNCATE);');
} finally {
db.close();
}
}
function assertHistoryRows(dbPath: string): void {
const rows = queryLocalWatchHistory(dbPath);
assert.equal(rows.length, 2);
assert.equal(rows[0]?.videoId, 1);
assert.equal(rows[0]?.lastWatchedMs, 5000);
assert.equal(rows[0]?.animeTitle, 'Show Romaji');
assert.equal(rows[1]?.videoId, 2);
assert.equal(rows[1]?.lastWatchedMs, 3000);
}
test('queryLocalWatchHistory returns local files ordered by most recent session', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-db-'));
const dbPath = path.join(dir, 'immersion.sqlite');
try {
createHistoryDb(dbPath);
assertHistoryRows(dbPath);
const rows = queryLocalWatchHistory(dbPath);
assert.equal(rows[0]?.coverBlobHash, null);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('queryLocalWatchHistory resolves cover hashes directly and via shared anime', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-art-'));
const dbPath = path.join(dir, 'immersion.sqlite');
try {
createHistoryDb(dbPath, { coverArt: true });
const rows = queryLocalWatchHistory(dbPath);
assert.equal(rows[0]?.videoId, 1);
assert.equal(rows[0]?.coverBlobHash, 'hash-1');
assert.equal(rows[1]?.videoId, 2);
assert.equal(rows[1]?.coverBlobHash, 'hash-1');
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('materializeCoverArt extracts blobs to the cache dir and reuses cached files', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-covers-'));
const dbPath = path.join(dir, 'immersion.sqlite');
const cacheDir = path.join(dir, 'covers');
try {
createHistoryDb(dbPath, { coverArt: true });
const covers = materializeCoverArt(
dbPath,
['hash-1', 'hash-1', null, 'hash-missing'],
cacheDir,
);
const coverPath = covers.get('hash-1');
assert.ok(coverPath);
assert.equal(path.extname(coverPath!), '.png');
assert.ok(fs.statSync(coverPath!).size > 0);
assert.equal(covers.has('hash-missing'), false);
// Cached file is reused even when the database has disappeared.
fs.rmSync(dbPath);
const cachedCovers = materializeCoverArt(dbPath, ['hash-1'], cacheDir);
assert.equal(cachedCovers.get('hash-1'), coverPath);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('materializeCoverArt rejects cover hashes that escape the cache dir', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-cover-safety-'));
const dbPath = path.join(dir, 'immersion.sqlite');
const cacheDir = path.join(dir, 'covers');
const unsafeHash = '../escape';
try {
createHistoryDb(dbPath, { coverArt: true });
const db = new Database(dbPath);
try {
db.query('INSERT INTO imm_cover_art_blobs VALUES (?, ?)').run(unsafeHash, PNG_MAGIC);
} finally {
db.close();
}
const covers = materializeCoverArt(dbPath, [unsafeHash], cacheDir);
assert.equal(covers.has(unsafeHash), false);
assert.equal(fs.existsSync(path.join(dir, 'escape.png')), false);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('detectImageExtension identifies common cover formats', () => {
assert.equal(detectImageExtension(PNG_MAGIC), '.png');
assert.equal(detectImageExtension(Buffer.from([0xff, 0xd8, 0xff, 0xe0])), '.jpg');
assert.equal(detectImageExtension(Buffer.from('RIFF0000WEBPVP8 ', 'ascii')), '.webp');
assert.equal(detectImageExtension(Buffer.from('GIF89a', 'ascii')), '.gif');
assert.equal(detectImageExtension(Buffer.from('unknown', 'ascii')), '.jpg');
});
test('groupHistoryBySeries backfills cover hash from older rows of the same series', () => {
const rows = [
makeRow({ videoId: 2, parsedEpisode: 2, lastWatchedMs: 3000, coverBlobHash: null }),
makeRow({ videoId: 1, parsedEpisode: 1, lastWatchedMs: 1000, coverBlobHash: 'hash-1' }),
];
const series = groupHistoryBySeries(rows, () => true);
assert.equal(series.length, 1);
assert.equal(series[0]?.lastWatched.videoId, 2);
assert.equal(series[0]?.coverBlobHash, 'hash-1');
});
test('queryLocalWatchHistory reads a cleanly-closed WAL database', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-wal-'));
const dbPath = path.join(dir, 'immersion.sqlite');
try {
createHistoryDb(dbPath, { wal: true });
// Reproduce the state after the app shuts down cleanly: WAL journal mode
// with no -wal/-shm sidecar files on disk. A read-only connection then
// fails at query time because it cannot recreate them.
fs.rmSync(`${dbPath}-wal`, { force: true });
fs.rmSync(`${dbPath}-shm`, { force: true });
assertHistoryRows(dbPath);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('isReadonlyWalRetryError only accepts readonly errors from WAL-mode databases', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-history-retry-'));
const walDbPath = path.join(dir, 'wal.sqlite');
const rollbackDbPath = path.join(dir, 'rollback.sqlite');
try {
createHistoryDb(walDbPath, { wal: true });
createHistoryDb(rollbackDbPath);
assert.equal(
isReadonlyWalRetryError(
Object.assign(new Error('attempt to write a readonly database'), {
code: 'SQLITE_READONLY',
}),
walDbPath,
),
true,
);
assert.equal(
isReadonlyWalRetryError(
Object.assign(new Error('unable to open database file'), {
code: 'SQLITE_CANTOPEN',
}),
walDbPath,
),
true,
);
assert.equal(
isReadonlyWalRetryError(new Error('no such table: imm_sessions'), walDbPath),
false,
);
assert.equal(
isReadonlyWalRetryError(
Object.assign(new Error('attempt to write a readonly database'), {
code: 'SQLITE_READONLY',
}),
rollbackDbPath,
),
false,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
+4
View File
@@ -0,0 +1,4 @@
export * from './history-art.js';
export * from './history-db.js';
export * from './history-navigation.js';
export type { HistorySeriesEntry, HistoryVideoRow, SeasonDirEntry } from './history-types.js';
+9
View File
@@ -29,6 +29,15 @@ function createArgs(): Args {
texthookerOnly: false,
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncDirection: 'both',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'info',
logRotation: 7,
passwordStore: '',
+46 -16
View File
@@ -65,11 +65,36 @@ function makeTestEnv(homeDir: string, xdgConfigHome: string): NodeJS.ProcessEnv
APPDATA: xdgConfigHome,
LOCALAPPDATA: path.join(homeDir, 'AppData', 'Local'),
XDG_CONFIG_HOME: xdgConfigHome,
// Pin the data dir under the temp home so the Linux runtime-plugin preflight
// resolves managed asset paths deterministically (not the CI runner's).
XDG_DATA_HOME: path.join(homeDir, '.local', 'share'),
PATH: pathValue,
Path: pathValue,
};
}
// On Linux the playback path runs `ensureLinuxRuntimePluginAvailable`, which —
// when the runtime plugin/theme are missing — spawns the app with
// `--ensure-linux-runtime-plugin-assets` and polls up to 30s
// (RESPONSE_TIMEOUT_MS) for an install response. A fake app that just exits
// never writes that response, so the launcher hangs and the test times out on
// Linux CI (the preflight is a no-op on macOS/Windows). This shell prelude makes
// the fake app install the managed plugin/theme and write the response, matching
// launcher/smoke.e2e.test.ts. Prepend it to each fake app that reaches playback.
const RUNTIME_PLUGIN_PREFLIGHT_SH = `if [ "$1" = "--ensure-linux-runtime-plugin-assets" ]; then
data="\${XDG_DATA_HOME:-$HOME/.local/share}/SubMiner"
mkdir -p "$data/plugin/subminer" "$data/themes"
printf -- '-- test plugin\\n' > "$data/plugin/subminer/main.lua"
printf 'test=true\\n' > "$data/plugin/subminer.conf"
printf '/* test theme */\\n' > "$data/themes/subminer.rasi"
if [ "$2" = "--ensure-linux-runtime-plugin-assets-response-path" ] && [ -n "$3" ]; then
mkdir -p "$(dirname "$3")"
printf '{"ok":true,"status":"installed","path":"%s"}' "$data/plugin/subminer/main.lua" > "$3"
fi
exit 0
fi
`;
test('config path uses XDG_CONFIG_HOME override', () => {
withTempDir((root) => {
const xdgConfigHome = path.join(root, 'xdg');
@@ -237,7 +262,7 @@ test('doctor refresh-known-words forwards app refresh command without requiring
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -264,7 +289,7 @@ test('launcher settings option forwards app settings window command', () => {
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -288,7 +313,7 @@ test('launcher settings command forwards app settings window command', () => {
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -369,7 +394,7 @@ test('launcher forwards --args to mpv as parsed tokens', { timeout: 15000 }, ()
},
}),
);
fs.writeFileSync(appPath, '#!/bin/sh\nexit 0\n');
fs.writeFileSync(appPath, `#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}exit 0\n`);
fs.chmodSync(appPath, 0o755);
fs.writeFileSync(
@@ -460,7 +485,7 @@ test('launcher forwards non-info log level into mpv logging args', { timeout: 15
},
}),
);
fs.writeFileSync(appPath, '#!/bin/sh\nexit 0\n');
fs.writeFileSync(appPath, `#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}exit 0\n`);
fs.chmodSync(appPath, 0o755);
fs.writeFileSync(
@@ -539,7 +564,7 @@ test('launcher routes youtube urls through regular playback startup', { timeout:
);
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -566,17 +591,22 @@ ${bunBinary} -e "const net=require('node:net'); const fs=require('node:fs'); con
fs.chmodSync(path.join(binDir, 'yt-dlp'), 0o755);
fs.chmodSync(path.join(binDir, 'ffmpeg'), 0o755);
// Note: no SUBMINER_TEST_CAPTURE here. When set, the launcher intercepts
// *every* app command — including the Linux runtime-plugin preflight's
// `--ensure-linux-runtime-plugin-assets` install — and returns without
// running the fake app, so the preflight would poll 30s for a response that
// never arrives and time out. This test asserts on the mpv args, not on
// captured app args, so capture isn't needed.
const env = {
...makeTestEnv(homeDir, xdgConfigHome),
PATH: `${binDir}${path.delimiter}${process.env.Path || process.env.PATH || ''}`,
Path: `${binDir}${path.delimiter}${process.env.Path || process.env.PATH || ''}`,
DISPLAY: ':99',
XDG_SESSION_TYPE: 'x11',
SUBMINER_APPIMAGE_PATH: appPath,
SUBMINER_TEST_MPV_ARGS: mpvArgsPath,
SUBMINER_TEST_CAPTURE: path.join(root, 'captured-args.txt'),
};
const result = runLauncher(['https://www.youtube.com/watch?v=abc123'], env);
// Pass an explicit backend so overlay startup doesn't probe for a display
// (headless CI has none), matching launcher/smoke.e2e.test.ts.
const result = runLauncher(['--backend', 'x11', 'https://www.youtube.com/watch?v=abc123'], env);
assert.equal(result.status, 0, `stdout:\n${result.stdout}\nstderr:\n${result.stderr}`);
const forwardedArgs = fs
@@ -597,7 +627,7 @@ test('dictionary command forwards --dictionary and --dictionary-target to app co
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -626,7 +656,7 @@ test('dictionary command forwards manual AniList selection modes to app command
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -763,7 +793,7 @@ test('jellyfin discovery routes to app --background and remote announce with log
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -790,7 +820,7 @@ test('jellyfin discovery via jf alias forwards remote announce for cast visibili
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -817,7 +847,7 @@ test('jellyfin login routes credentials to app command', () => {
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
@@ -856,7 +886,7 @@ test('jellyfin setup forwards password-store to app command', () => {
const capturePath = path.join(root, 'captured-args.txt');
fs.writeFileSync(
appPath,
'#!/bin/sh\nif [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n',
`#!/bin/sh\n${RUNTIME_PLUGIN_PREFLIGHT_SH}if [ -n "$SUBMINER_TEST_CAPTURE" ]; then printf "%s\\n" "$@" > "$SUBMINER_TEST_CAPTURE"; fi\nexit 0\n`,
);
fs.chmodSync(appPath, 0o755);
+16
View File
@@ -21,6 +21,8 @@ import { runDictionaryCommand } from './commands/dictionary-command.js';
import { runLogsCommand } from './commands/logs-command.js';
import { runStatsCommand } from './commands/stats-command.js';
import { runJellyfinCommand } from './commands/jellyfin-command.js';
import { runHistoryCommand } from './commands/history-command.js';
import { runSyncCommand } from './commands/sync-command.js';
import { runPlaybackCommand } from './commands/playback-command.js';
import { runUpdateCommand } from './commands/update-command.js';
@@ -106,6 +108,10 @@ async function main(): Promise<void> {
return;
}
if (await runSyncCommand(context)) {
return;
}
const resolvedAppPath = ensureAppPath(context);
state.appPath = resolvedAppPath;
log('debug', args.logLevel, `Using SubMiner app binary: ${resolvedAppPath}`);
@@ -142,6 +148,16 @@ async function main(): Promise<void> {
return;
}
if (appContext.args.history) {
const selected = await runHistoryCommand(appContext);
if (!selected) {
log('info', args.logLevel, 'No watch history selection made, exiting');
return;
}
appContext.args.target = selected;
appContext.args.targetKind = 'file';
}
await runPlaybackCommand(appContext);
}
+9
View File
@@ -570,6 +570,15 @@ function makeArgs(overrides: Partial<Args> = {}): Args {
texthookerOnly: false,
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncDirection: 'both',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'error',
logRotation: 7,
passwordStore: '',
+1 -1
View File
@@ -1683,7 +1683,7 @@ async function sleepMs(ms: number): Promise<void> {
await new Promise<void>((resolve) => setTimeout(resolve, ms));
}
async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
export async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
return await new Promise<boolean>((resolve) => {
const socket = net.createConnection(socketPath);
let settled = false;
+13
View File
@@ -42,6 +42,19 @@ test('parseArgs maps root settings window option', () => {
assert.equal(parsed.settings, true);
});
test('parseArgs maps root watch history flags', () => {
const shortParsed = parseArgs(['-H'], 'subminer', {});
const longParsed = parseArgs(['--history'], 'subminer', {});
const rofiParsed = parseArgs(['-R', '-H'], 'subminer', {});
const defaultParsed = parseArgs([], 'subminer', {});
assert.equal(shortParsed.history, true);
assert.equal(longParsed.history, true);
assert.equal(rofiParsed.history, true);
assert.equal(rofiParsed.useRofi, true);
assert.equal(defaultParsed.history, false);
});
test('parseArgs maps root update flags without conflicting with jellyfin username', () => {
const shortParsed = parseArgs(['-u'], 'subminer', {});
const longParsed = parseArgs(['--update'], 'subminer', {});
+2 -2
View File
@@ -56,7 +56,7 @@ export function showFzfFlatMenu(
`--prompt=${prompt}`,
'--delimiter=\t',
'--with-nth=2',
'--preview-window=right:50%:wrap',
'--preview-window=down:50%:wrap',
'--preview',
previewCommand,
];
@@ -468,7 +468,7 @@ thumb=$(get_thumb)
'--prompt=Select Video: ',
'--delimiter=\t',
'--with-nth=1',
'--preview-window=right:50%:wrap',
'--preview-window=down:50%:wrap',
'--preview',
previewCmd,
],
+423
View File
@@ -0,0 +1,423 @@
import { Database } from 'bun:sqlite';
import { insertRow, tableExists, type SyncMergeSummary } from './sync-shared.js';
const ANIME_COPY_COLUMNS = [
'normalized_title_key',
'canonical_title',
'anilist_id',
'title_romaji',
'title_english',
'title_native',
'episodes_total',
'description',
'metadata_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const VIDEO_COPY_COLUMNS = [
'video_key',
'canonical_title',
'source_type',
'source_path',
'source_url',
'parsed_basename',
'parsed_title',
'parsed_season',
'parsed_episode',
'parser_source',
'parser_confidence',
'parse_metadata_json',
'watched',
'duration_ms',
'file_size_bytes',
'codec_id',
'container_id',
'width_px',
'height_px',
'fps_x100',
'bitrate_kbps',
'audio_codec_id',
'hash_sha256',
'screenshot_path',
'metadata_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const MEDIA_ART_COPY_COLUMNS = [
'anilist_id',
'cover_url',
'cover_blob',
'cover_blob_hash',
'title_romaji',
'title_english',
'episodes_total',
'fetched_at_ms',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const YOUTUBE_COPY_COLUMNS = [
'youtube_video_id',
'video_url',
'video_title',
'video_thumbnail_url',
'channel_id',
'channel_name',
'channel_url',
'channel_thumbnail_url',
'uploader_id',
'uploader_url',
'description',
'metadata_json',
'fetched_at_ms',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const WORD_COPY_COLUMNS = [
'headword',
'word',
'reading',
'part_of_speech',
'pos1',
'pos2',
'pos3',
'first_seen',
'last_seen',
'frequency',
'frequency_rank',
] as const;
type SqlRow = Record<string, unknown>;
function selectAll(db: Database, sql: string, params: unknown[] = []): SqlRow[] {
return db.query<SqlRow>(sql).all(...params);
}
export function mergeAnime(
local: Database,
remote: Database,
summary: SyncMergeSummary,
): Map<number, number> {
const map = new Map<number, number>();
const byAnilist = local.prepare<SqlRow>('SELECT anime_id FROM imm_anime WHERE anilist_id = ?');
const byTitleKey = local.prepare<SqlRow>(
'SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?',
);
const fillMissing = local.prepare(
`UPDATE imm_anime
SET
title_romaji = COALESCE(title_romaji, ?),
title_english = COALESCE(title_english, ?),
title_native = COALESCE(title_native, ?),
episodes_total = COALESCE(episodes_total, ?),
description = COALESCE(description, ?)
WHERE anime_id = ?`,
);
for (const row of selectAll(
remote,
`SELECT anime_id, ${ANIME_COPY_COLUMNS.join(', ')} FROM imm_anime`,
)) {
const remoteId = Number(row.anime_id);
const existing =
(row.anilist_id !== null ? byAnilist.get(row.anilist_id) : undefined) ??
byTitleKey.get(row.normalized_title_key);
if (existing) {
const localId = Number(existing.anime_id);
map.set(remoteId, localId);
fillMissing.run(
row.title_romaji,
row.title_english,
row.title_native,
row.episodes_total,
row.description,
localId,
);
continue;
}
// No local row matched by anilist_id (checked first in `existing` above)
// or title key, so the remote anilist_id — if any — is free to insert as-is.
const values = ANIME_COPY_COLUMNS.map((column) => row[column]);
map.set(remoteId, insertRow(local, 'imm_anime', ANIME_COPY_COLUMNS, values));
summary.animeAdded += 1;
}
return map;
}
export interface VideoMergeResult {
videoIdMap: Map<number, number>;
addedVideoIds: Set<number>;
}
export function mergeVideos(
local: Database,
remote: Database,
animeIdMap: Map<number, number>,
summary: SyncMergeSummary,
): VideoMergeResult {
const videoIdMap = new Map<number, number>();
const addedVideoIds = new Set<number>();
const byKey = local.prepare<SqlRow>(
'SELECT video_id, watched FROM imm_videos WHERE video_key = ?',
);
const setWatched = local.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?');
for (const row of selectAll(
remote,
`SELECT video_id, anime_id, ${VIDEO_COPY_COLUMNS.join(', ')} FROM imm_videos`,
)) {
const remoteId = Number(row.video_id);
const mappedAnimeId =
row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
const existing = byKey.get(row.video_key);
if (existing) {
const localId = Number(existing.video_id);
videoIdMap.set(remoteId, localId);
if (Number(row.watched) > 0 && Number(existing.watched) <= 0) {
setWatched.run(localId);
}
continue;
}
const columns = ['anime_id', ...VIDEO_COPY_COLUMNS];
const values = [mappedAnimeId, ...VIDEO_COPY_COLUMNS.map((column) => row[column])];
const localId = insertRow(local, 'imm_videos', columns, values);
videoIdMap.set(remoteId, localId);
addedVideoIds.add(remoteId);
summary.videosAdded += 1;
}
return { videoIdMap, addedVideoIds };
}
export function mergeMediaMetadata(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
addedVideoIds: Set<number>,
): void {
if (videoIdMap.size === 0) return;
const metadataVideoIds = new Set<number>([...addedVideoIds, ...videoIdMap.keys()]);
const hasBlobStore =
tableExists(local, 'imm_cover_art_blobs') && tableExists(remote, 'imm_cover_art_blobs');
const copyBlob = hasBlobStore
? local.prepare(
`INSERT INTO imm_cover_art_blobs (blob_hash, cover_blob, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)
ON CONFLICT(blob_hash) DO NOTHING`,
)
: null;
const readBlob = hasBlobStore
? remote.prepare<SqlRow>('SELECT * FROM imm_cover_art_blobs WHERE blob_hash = ?')
: null;
if (tableExists(remote, 'imm_media_art') && tableExists(local, 'imm_media_art')) {
const localArtExists = local.prepare<SqlRow>(
'SELECT 1 FROM imm_media_art WHERE video_id = ? LIMIT 1',
);
for (const remoteVideoId of metadataVideoIds) {
const localVideoId = videoIdMap.get(remoteVideoId)!;
if (localArtExists.get(localVideoId)) continue;
const row = remote
.query<SqlRow>(
`SELECT ${MEDIA_ART_COPY_COLUMNS.join(', ')} FROM imm_media_art WHERE video_id = ?`,
)
.get(remoteVideoId);
if (!row) continue;
if (row.cover_blob_hash && copyBlob && readBlob) {
const blob = readBlob.get(row.cover_blob_hash);
if (blob) {
copyBlob.run(blob.blob_hash, blob.cover_blob, blob.CREATED_DATE, blob.LAST_UPDATE_DATE);
}
}
insertRow(
local,
'imm_media_art',
['video_id', ...MEDIA_ART_COPY_COLUMNS],
[localVideoId, ...MEDIA_ART_COPY_COLUMNS.map((column) => row[column])],
);
}
}
if (tableExists(remote, 'imm_youtube_videos') && tableExists(local, 'imm_youtube_videos')) {
const localYoutubeExists = local.prepare<SqlRow>(
'SELECT 1 FROM imm_youtube_videos WHERE video_id = ? LIMIT 1',
);
for (const remoteVideoId of metadataVideoIds) {
const localVideoId = videoIdMap.get(remoteVideoId)!;
if (localYoutubeExists.get(localVideoId)) continue;
const row = remote
.query<SqlRow>(
`SELECT ${YOUTUBE_COPY_COLUMNS.join(', ')} FROM imm_youtube_videos WHERE video_id = ?`,
)
.get(remoteVideoId);
if (!row) continue;
insertRow(
local,
'imm_youtube_videos',
['video_id', ...YOUTUBE_COPY_COLUMNS],
[localVideoId, ...YOUTUBE_COPY_COLUMNS.map((column) => row[column])],
);
}
}
}
export function mergeExcludedWords(
local: Database,
remote: Database,
summary: SyncMergeSummary,
): void {
if (
!tableExists(remote, 'imm_stats_excluded_words') ||
!tableExists(local, 'imm_stats_excluded_words')
) {
return;
}
const insert = local.prepare(
`INSERT INTO imm_stats_excluded_words (headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(headword, word, reading) DO NOTHING`,
);
for (const row of selectAll(
remote,
'SELECT headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE FROM imm_stats_excluded_words',
)) {
const result = insert.run(
row.headword,
row.word,
row.reading,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.excludedWordsAdded += result.changes;
}
}
/**
* Lazily maps remote imm_words / imm_kanji ids onto local rows by natural key
* ((headword, word, reading) / kanji). New rows are copied with the remote's
* accumulated frequency; rows that already exist locally get their frequency
* incremented later with only the occurrence counts this merge adds (the
* remote total would double-count lines merged in earlier syncs).
*/
export class LexiconResolver {
private readonly wordMap = new Map<number, { localId: number; isNew: boolean }>();
private readonly kanjiMap = new Map<number, { localId: number; isNew: boolean }>();
readonly wordFrequencyDeltas = new Map<number, number>();
readonly kanjiFrequencyDeltas = new Map<number, number>();
constructor(
private readonly local: Database,
private readonly remote: Database,
private readonly summary: SyncMergeSummary,
) {}
resolveWord(remoteWordId: number): number {
const cached = this.wordMap.get(remoteWordId);
if (cached) return cached.localId;
const row = this.remote
.query<SqlRow>(`SELECT ${WORD_COPY_COLUMNS.join(', ')} FROM imm_words WHERE id = ?`)
.get(remoteWordId);
if (!row) throw new Error(`Snapshot references missing imm_words row ${remoteWordId}`);
const existing = this.local
.query<SqlRow>('SELECT id FROM imm_words WHERE headword IS ? AND word IS ? AND reading IS ?')
.get(row.headword, row.word, row.reading);
let entry: { localId: number; isNew: boolean };
if (existing) {
entry = { localId: Number(existing.id), isNew: false };
this.local
.prepare(
`UPDATE imm_words
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
WHERE id = ?`,
)
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
} else {
const localId = insertRow(
this.local,
'imm_words',
WORD_COPY_COLUMNS,
WORD_COPY_COLUMNS.map((column) => row[column]),
);
entry = { localId, isNew: true };
this.summary.wordsAdded += 1;
}
this.wordMap.set(remoteWordId, entry);
return entry.localId;
}
resolveKanji(remoteKanjiId: number): number {
const cached = this.kanjiMap.get(remoteKanjiId);
if (cached) return cached.localId;
const row = this.remote
.query<SqlRow>('SELECT kanji, first_seen, last_seen, frequency FROM imm_kanji WHERE id = ?')
.get(remoteKanjiId);
if (!row) throw new Error(`Snapshot references missing imm_kanji row ${remoteKanjiId}`);
const existing = this.local
.query<SqlRow>('SELECT id FROM imm_kanji WHERE kanji IS ?')
.get(row.kanji);
let entry: { localId: number; isNew: boolean };
if (existing) {
entry = { localId: Number(existing.id), isNew: false };
this.local
.prepare(
`UPDATE imm_kanji
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
WHERE id = ?`,
)
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
} else {
const localId = insertRow(
this.local,
'imm_kanji',
['kanji', 'first_seen', 'last_seen', 'frequency'],
[row.kanji, row.first_seen, row.last_seen, row.frequency],
);
entry = { localId, isNew: true };
this.summary.kanjiAdded += 1;
}
this.kanjiMap.set(remoteKanjiId, entry);
return entry.localId;
}
addWordOccurrences(remoteWordId: number, count: number): void {
const entry = this.wordMap.get(remoteWordId);
if (!entry || entry.isNew) return;
this.wordFrequencyDeltas.set(
entry.localId,
(this.wordFrequencyDeltas.get(entry.localId) ?? 0) + count,
);
}
addKanjiOccurrences(remoteKanjiId: number, count: number): void {
const entry = this.kanjiMap.get(remoteKanjiId);
if (!entry || entry.isNew) return;
this.kanjiFrequencyDeltas.set(
entry.localId,
(this.kanjiFrequencyDeltas.get(entry.localId) ?? 0) + count,
);
}
applyFrequencyDeltas(): void {
const updateWord = this.local.prepare(
'UPDATE imm_words SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
);
for (const [localId, delta] of this.wordFrequencyDeltas) {
updateWord.run(delta, localId);
}
const updateKanji = this.local.prepare(
'UPDATE imm_kanji SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
);
for (const [localId, delta] of this.kanjiFrequencyDeltas) {
updateKanji.run(delta, localId);
}
}
}
+265
View File
@@ -0,0 +1,265 @@
import { Database } from 'bun:sqlite';
import { nowDbTimestamp, tableExists, type SyncMergeSummary } from './sync-shared.js';
type SqlRow = Record<string, unknown>;
const LOCAL_DAY_EXPR = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
const LOCAL_MONTH_EXPR = `CAST(strftime('%Y%m', CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER)`;
// Ported from upsertDailyRollupsForGroups / upsertMonthlyRollupsForGroups in
// src/core/services/immersion-tracker/maintenance.ts — must stay in sync.
const DAILY_ROLLUP_UPSERT = `
WITH matching_sessions AS (
SELECT * FROM imm_sessions
WHERE ${LOCAL_DAY_EXPR} = ? AND video_id = ?
),
session_metrics AS (
SELECT
t.session_id,
MAX(t.active_watched_ms) AS max_active_ms,
MAX(t.lines_seen) AS max_lines,
MAX(t.tokens_seen) AS max_tokens,
MAX(t.cards_mined) AS max_cards,
MAX(t.lookup_count) AS max_lookups,
MAX(t.lookup_hits) AS max_hits
FROM imm_session_telemetry t
JOIN matching_sessions s ON s.session_id = t.session_id
GROUP BY t.session_id
)
INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
CREATED_DATE, LAST_UPDATE_DATE
)
SELECT
${LOCAL_DAY_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_day,
s.video_id AS video_id,
COUNT(DISTINCT s.session_id) AS total_sessions,
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
THEN (COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) * 60.0)
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
ELSE NULL
END AS cards_per_hour,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
THEN COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0)
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
ELSE NULL
END AS tokens_per_min,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) > 0
THEN CAST(COALESCE(SUM(COALESCE(sm.max_hits, s.lookup_hits)), 0) AS REAL)
/ CAST(COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) AS REAL)
ELSE NULL
END AS lookup_hit_rate,
? AS CREATED_DATE,
? AS LAST_UPDATE_DATE
FROM matching_sessions s
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
GROUP BY rollup_day, s.video_id
ON CONFLICT (rollup_day, video_id) DO UPDATE SET
total_sessions = excluded.total_sessions,
total_active_min = excluded.total_active_min,
total_lines_seen = excluded.total_lines_seen,
total_tokens_seen = excluded.total_tokens_seen,
total_cards = excluded.total_cards,
cards_per_hour = excluded.cards_per_hour,
tokens_per_min = excluded.tokens_per_min,
lookup_hit_rate = excluded.lookup_hit_rate,
CREATED_DATE = COALESCE(imm_daily_rollups.CREATED_DATE, excluded.CREATED_DATE),
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
`;
const MONTHLY_ROLLUP_UPSERT = `
WITH matching_sessions AS (
SELECT * FROM imm_sessions
WHERE ${LOCAL_MONTH_EXPR} = ? AND video_id = ?
),
session_metrics AS (
SELECT
t.session_id,
MAX(t.active_watched_ms) AS max_active_ms,
MAX(t.lines_seen) AS max_lines,
MAX(t.tokens_seen) AS max_tokens,
MAX(t.cards_mined) AS max_cards
FROM imm_session_telemetry t
JOIN matching_sessions s ON s.session_id = t.session_id
GROUP BY t.session_id
)
INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
)
SELECT
${LOCAL_MONTH_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_month,
s.video_id AS video_id,
COUNT(DISTINCT s.session_id) AS total_sessions,
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
? AS CREATED_DATE,
? AS LAST_UPDATE_DATE
FROM matching_sessions s
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
GROUP BY rollup_month, s.video_id
ON CONFLICT (rollup_month, video_id) DO UPDATE SET
total_sessions = excluded.total_sessions,
total_active_min = excluded.total_active_min,
total_lines_seen = excluded.total_lines_seen,
total_tokens_seen = excluded.total_tokens_seen,
total_cards = excluded.total_cards,
CREATED_DATE = COALESCE(imm_monthly_rollups.CREATED_DATE, excluded.CREATED_DATE),
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
`;
/**
* Recompute daily/monthly rollup groups touched by the newly merged sessions
* from the (now merged) local session + telemetry data. The maintenance
* watermark is left alone: telemetry newer than it gets recomputed again by
* the app later, which is idempotent.
*/
export function refreshRollupsForNewSessions(
local: Database,
newSessionIds: number[],
summary: SyncMergeSummary,
): void {
if (newSessionIds.length === 0) return;
const groups = new Map<string, { day: number; month: number; videoId: number }>();
for (let offset = 0; offset < newSessionIds.length; offset += 500) {
const chunk = newSessionIds.slice(offset, offset + 500);
const rows = local
.query<SqlRow>(
`SELECT DISTINCT ${LOCAL_DAY_EXPR} AS rollup_day, ${LOCAL_MONTH_EXPR} AS rollup_month, video_id
FROM imm_sessions WHERE session_id IN (${chunk.map(() => '?').join(',')})`,
)
.all(...chunk);
for (const row of rows) {
const day = Number(row.rollup_day);
const month = Number(row.rollup_month);
const videoId = Number(row.video_id);
groups.set(`${day}-${videoId}`, { day, month, videoId });
}
}
const stampMs = nowDbTimestamp();
const deleteDaily = local.prepare(
'DELETE FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ?',
);
const deleteMonthly = local.prepare(
'DELETE FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ?',
);
const upsertDaily = local.prepare(DAILY_ROLLUP_UPSERT);
const upsertMonthly = local.prepare(MONTHLY_ROLLUP_UPSERT);
const monthlyGroups = new Set<string>();
for (const { day, month, videoId } of groups.values()) {
deleteDaily.run(day, videoId);
upsertDaily.run(day, videoId, stampMs, stampMs);
summary.rollupGroupsRecomputed += 1;
const monthKey = `${month}-${videoId}`;
if (!monthlyGroups.has(monthKey)) {
monthlyGroups.add(monthKey);
deleteMonthly.run(month, videoId);
upsertMonthly.run(month, videoId, stampMs, stampMs);
}
}
}
/**
* Sessions are pruned after a retention window, but rollups are kept much
* longer the remote's older rollup history can't be reconstructed from
* merged sessions. Copy remote rollup rows for groups where the local DB has
* neither a rollup row nor any sessions (i.e. history only the remote knows).
* Groups both machines have data for are never summed, to avoid
* double-counting sessions that earlier syncs already shared.
*/
export function copyRemoteOnlyRollups(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
summary: SyncMergeSummary,
): void {
if (!tableExists(remote, 'imm_daily_rollups') || !tableExists(local, 'imm_daily_rollups')) return;
const localDailyExists = local.prepare(
'SELECT 1 FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ? LIMIT 1',
);
const localDaySessions = local.prepare(
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_DAY_EXPR} = ? LIMIT 1`,
);
const localMonthSessions = local.prepare(
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_MONTH_EXPR} = ? LIMIT 1`,
);
const localMonthSessionsForDay = local.prepare(
`SELECT 1 FROM imm_sessions
WHERE video_id = ?
AND ${LOCAL_MONTH_EXPR} = CAST(strftime('%Y%m', CAST(? AS INTEGER) * 86400, 'unixepoch', 'localtime') AS INTEGER)
LIMIT 1`,
);
const insertDaily = local.prepare(
`INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
);
for (const row of remote.query<SqlRow>('SELECT * FROM imm_daily_rollups').all()) {
if (row.video_id === null) continue;
const localVideoId = videoIdMap.get(Number(row.video_id));
if (localVideoId === undefined) continue;
if (localDailyExists.get(row.rollup_day, localVideoId)) continue;
if (localDaySessions.get(localVideoId, row.rollup_day)) continue;
if (localMonthSessionsForDay.get(localVideoId, row.rollup_day)) continue;
insertDaily.run(
row.rollup_day,
localVideoId,
row.total_sessions,
row.total_active_min,
row.total_lines_seen,
row.total_tokens_seen,
row.total_cards,
row.cards_per_hour,
row.tokens_per_min,
row.lookup_hit_rate,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.dailyRollupsCopied += 1;
}
const localMonthlyExists = local.prepare(
'SELECT 1 FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ? LIMIT 1',
);
const insertMonthly = local.prepare(
`INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
);
for (const row of remote.query<SqlRow>('SELECT * FROM imm_monthly_rollups').all()) {
if (row.video_id === null) continue;
const localVideoId = videoIdMap.get(Number(row.video_id));
if (localVideoId === undefined) continue;
if (localMonthlyExists.get(row.rollup_month, localVideoId)) continue;
if (localMonthSessions.get(localVideoId, row.rollup_month)) continue;
insertMonthly.run(
row.rollup_month,
localVideoId,
row.total_sessions,
row.total_active_min,
row.total_lines_seen,
row.total_tokens_seen,
row.total_cards,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.monthlyRollupsCopied += 1;
}
}
+465
View File
@@ -0,0 +1,465 @@
import { Database } from 'bun:sqlite';
import type { LexiconResolver } from './merge-catalog.js';
import { insertRow, nowDbTimestamp, type SyncMergeSummary } from './sync-shared.js';
const SESSION_COPY_COLUMNS = [
'session_uuid',
'started_at_ms',
'ended_at_ms',
'status',
'locale_id',
'target_lang_id',
'difficulty_tier',
'subtitle_mode',
'ended_media_ms',
'total_watched_ms',
'active_watched_ms',
'lines_seen',
'tokens_seen',
'cards_mined',
'lookup_count',
'lookup_hits',
'yomitan_lookup_count',
'pause_count',
'pause_ms',
'seek_forward_count',
'seek_backward_count',
'media_buffer_events',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const TELEMETRY_COPY_COLUMNS = [
'sample_ms',
'total_watched_ms',
'active_watched_ms',
'lines_seen',
'tokens_seen',
'cards_mined',
'lookup_count',
'lookup_hits',
'yomitan_lookup_count',
'pause_count',
'pause_ms',
'seek_forward_count',
'seek_backward_count',
'media_buffer_events',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const EVENT_COPY_COLUMNS = [
'ts_ms',
'event_type',
'line_index',
'segment_start_ms',
'segment_end_ms',
'tokens_delta',
'cards_delta',
'payload_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const LINE_COPY_COLUMNS = [
'line_index',
'segment_start_ms',
'segment_end_ms',
'text',
'secondary_text',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
type SqlRow = Record<string, unknown>;
export interface SessionMergeResult {
newSessionIds: number[];
}
export function mergeSessions(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
animeIdMap: Map<number, number>,
lexicon: LexiconResolver,
summary: SyncMergeSummary,
): SessionMergeResult {
const newSessionIds: number[] = [];
const uuidExists = local.prepare<SqlRow>(
'SELECT session_id FROM imm_sessions WHERE session_uuid = ?',
);
const remoteSessions = remote
.query<SqlRow>(
`SELECT session_id, video_id, ${SESSION_COPY_COLUMNS.join(', ')}
FROM imm_sessions
ORDER BY CAST(started_at_ms AS REAL) ASC, session_id ASC`,
)
.all();
for (const session of remoteSessions) {
if (session.ended_at_ms === null) {
// Stale ACTIVE sessions are finalized by the app on its next startup;
// they will sync once they carry final numbers.
summary.activeSessionsSkipped += 1;
continue;
}
if (uuidExists.get(session.session_uuid)) {
summary.sessionsAlreadyPresent += 1;
continue;
}
const localVideoId = videoIdMap.get(Number(session.video_id));
if (localVideoId === undefined) {
throw new Error(`Snapshot session ${String(session.session_uuid)} references missing video row`);
}
const localSessionId = insertRow(
local,
'imm_sessions',
['video_id', ...SESSION_COPY_COLUMNS],
[localVideoId, ...SESSION_COPY_COLUMNS.map((column) => session[column])],
);
newSessionIds.push(localSessionId);
summary.sessionsMerged += 1;
const remoteSessionId = Number(session.session_id);
copyTelemetry(local, remote, remoteSessionId, localSessionId, summary);
const eventIdMap = copyEvents(local, remote, remoteSessionId, localSessionId, summary);
copySubtitleLines(
local,
remote,
remoteSessionId,
localSessionId,
localVideoId,
animeIdMap,
eventIdMap,
lexicon,
summary,
);
applyMergedSessionLifetime(local, localSessionId, localVideoId, session);
}
return { newSessionIds };
}
function copyTelemetry(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
summary: SyncMergeSummary,
): void {
const rows = remote
.query<SqlRow>(
`SELECT ${TELEMETRY_COPY_COLUMNS.join(', ')} FROM imm_session_telemetry
WHERE session_id = ? ORDER BY telemetry_id ASC`,
)
.all(remoteSessionId);
for (const row of rows) {
insertRow(
local,
'imm_session_telemetry',
['session_id', ...TELEMETRY_COPY_COLUMNS],
[localSessionId, ...TELEMETRY_COPY_COLUMNS.map((column) => row[column])],
);
summary.telemetryRowsAdded += 1;
}
}
function copyEvents(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
summary: SyncMergeSummary,
): Map<number, number> {
const eventIdMap = new Map<number, number>();
const rows = remote
.query<SqlRow>(
`SELECT event_id, ${EVENT_COPY_COLUMNS.join(', ')} FROM imm_session_events
WHERE session_id = ? ORDER BY event_id ASC`,
)
.all(remoteSessionId);
for (const row of rows) {
const localEventId = insertRow(
local,
'imm_session_events',
['session_id', ...EVENT_COPY_COLUMNS],
[localSessionId, ...EVENT_COPY_COLUMNS.map((column) => row[column])],
);
eventIdMap.set(Number(row.event_id), localEventId);
summary.eventsAdded += 1;
}
return eventIdMap;
}
function copySubtitleLines(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
localVideoId: number,
animeIdMap: Map<number, number>,
eventIdMap: Map<number, number>,
lexicon: LexiconResolver,
summary: SyncMergeSummary,
): void {
const rows = remote
.query<SqlRow>(
`SELECT line_id, event_id, anime_id, ${LINE_COPY_COLUMNS.join(', ')} FROM imm_subtitle_lines
WHERE session_id = ? ORDER BY line_id ASC`,
)
.all(remoteSessionId);
const wordOccurrences = remote.prepare<SqlRow>(
'SELECT word_id, occurrence_count FROM imm_word_line_occurrences WHERE line_id = ?',
);
const kanjiOccurrences = remote.prepare<SqlRow>(
'SELECT kanji_id, occurrence_count FROM imm_kanji_line_occurrences WHERE line_id = ?',
);
const insertWordOccurrence = local.prepare(
`INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)
ON CONFLICT(line_id, word_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
);
const insertKanjiOccurrence = local.prepare(
`INSERT INTO imm_kanji_line_occurrences (line_id, kanji_id, occurrence_count) VALUES (?, ?, ?)
ON CONFLICT(line_id, kanji_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
);
for (const row of rows) {
const localEventId = row.event_id === null ? null : (eventIdMap.get(Number(row.event_id)) ?? null);
const localAnimeId = row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
const localLineId = insertRow(
local,
'imm_subtitle_lines',
['session_id', 'event_id', 'video_id', 'anime_id', ...LINE_COPY_COLUMNS],
[
localSessionId,
localEventId,
localVideoId,
localAnimeId,
...LINE_COPY_COLUMNS.map((column) => row[column]),
],
);
summary.subtitleLinesAdded += 1;
for (const occurrence of wordOccurrences.all(row.line_id)) {
const localWordId = lexicon.resolveWord(Number(occurrence.word_id));
const count = Number(occurrence.occurrence_count);
insertWordOccurrence.run(localLineId, localWordId, count);
lexicon.addWordOccurrences(Number(occurrence.word_id), count);
}
for (const occurrence of kanjiOccurrences.all(row.line_id)) {
const localKanjiId = lexicon.resolveKanji(Number(occurrence.kanji_id));
const count = Number(occurrence.occurrence_count);
insertKanjiOccurrence.run(localLineId, localKanjiId, count);
lexicon.addKanjiOccurrences(Number(occurrence.kanji_id), count);
}
}
}
/**
* Port of applySessionLifetimeSummary (src/core/services/immersion-tracker/
* lifetime.ts) for sessions arriving out of chronological order. The
* "first session of the day / for this video" checks are order-independent
* here (any other session counts, not just earlier ones): the local machine
* already credited active_days/episodes_started when its own session was
* applied, even if the merged session started earlier that day.
*/
function applyMergedSessionLifetime(
local: Database,
sessionId: number,
videoId: number,
session: SqlRow,
): void {
const updatedAtMs = nowDbTimestamp();
const applied = local
.prepare(
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)
ON CONFLICT(session_id) DO NOTHING`,
)
.run(sessionId, session.ended_at_ms, updatedAtMs, updatedAtMs);
if (applied.changes <= 0) return;
const telemetry = local
.query<SqlRow>(
`SELECT active_watched_ms, cards_mined, lines_seen, tokens_seen
FROM imm_session_telemetry
WHERE session_id = ?
ORDER BY sample_ms DESC, telemetry_id DESC
LIMIT 1`,
)
.get(sessionId);
const metric = (telemetryValue: unknown, sessionValue: unknown): number => {
const fromTelemetry = telemetry ? Number(telemetryValue) : Number.NaN;
const value = Number.isFinite(fromTelemetry) ? fromTelemetry : Number(sessionValue);
return Math.max(0, Math.floor(Number.isFinite(value) ? value : 0));
};
const activeMs = metric(telemetry?.active_watched_ms, session.active_watched_ms);
const cardsMined = metric(telemetry?.cards_mined, session.cards_mined);
const linesSeen = metric(telemetry?.lines_seen, session.lines_seen);
const tokensSeen = metric(telemetry?.tokens_seen, session.tokens_seen);
const video = local
.query<SqlRow>('SELECT anime_id, watched FROM imm_videos WHERE video_id = ?')
.get(videoId);
const watched = Number(video?.watched ?? 0);
const animeId = video?.anime_id === null || video?.anime_id === undefined ? null : Number(video.anime_id);
const mediaLifetime = local
.query<SqlRow>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
.get(videoId);
const hasOtherSessionForVideo = Boolean(
local
.query('SELECT 1 FROM imm_sessions WHERE video_id = ? AND session_id != ? LIMIT 1')
.get(videoId, sessionId),
);
const isFirstSessionForVideoRun = !mediaLifetime && !hasOtherSessionForVideo;
const isFirstCompletedSessionForVideoRun = watched > 0 && Number(mediaLifetime?.completed ?? 0) <= 0;
const hasOtherSessionOnDay = Boolean(
local
.query(
`SELECT 1 FROM imm_sessions
WHERE session_id != ?
AND CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
= CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
LIMIT 1`,
)
.get(sessionId, session.started_at_ms),
);
let animeCompletedDelta = 0;
if (animeId !== null && watched > 0 && isFirstCompletedSessionForVideoRun) {
const animeLifetime = local
.query<SqlRow>('SELECT episodes_completed FROM imm_lifetime_anime WHERE anime_id = ?')
.get(animeId);
const anime = local
.query<SqlRow>('SELECT episodes_total FROM imm_anime WHERE anime_id = ?')
.get(animeId);
const episodesCompletedBefore = Number(animeLifetime?.episodes_completed ?? 0);
const episodesTotal = anime?.episodes_total === null || anime?.episodes_total === undefined
? null
: Number(anime.episodes_total);
if (
episodesTotal !== null &&
episodesTotal > 0 &&
episodesCompletedBefore < episodesTotal &&
episodesCompletedBefore + 1 >= episodesTotal
) {
animeCompletedDelta = 1;
}
}
local
.prepare(
`UPDATE imm_lifetime_global
SET total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + ?,
total_cards = total_cards + ?,
active_days = active_days + ?,
episodes_started = episodes_started + ?,
episodes_completed = episodes_completed + ?,
anime_completed = anime_completed + ?,
LAST_UPDATE_DATE = ?
WHERE global_id = 1`,
)
.run(
activeMs,
cardsMined,
hasOtherSessionOnDay ? 0 : 1,
isFirstSessionForVideoRun ? 1 : 0,
isFirstCompletedSessionForVideoRun ? 1 : 0,
animeCompletedDelta,
updatedAtMs,
);
local
.prepare(
`INSERT INTO imm_lifetime_media(
video_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(video_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
completed = MAX(completed, excluded.completed),
first_watched_ms = CASE
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
ELSE first_watched_ms
END,
last_watched_ms = CASE
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
ELSE last_watched_ms
END,
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
)
.run(
videoId,
activeMs,
cardsMined,
linesSeen,
tokensSeen,
watched > 0 ? 1 : 0,
session.started_at_ms,
session.ended_at_ms,
updatedAtMs,
updatedAtMs,
);
if (animeId !== null) {
local
.prepare(
`INSERT INTO imm_lifetime_anime(
anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
total_tokens_seen, episodes_started, episodes_completed, first_watched_ms,
last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(anime_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
episodes_started = episodes_started + excluded.episodes_started,
episodes_completed = episodes_completed + excluded.episodes_completed,
first_watched_ms = CASE
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
ELSE first_watched_ms
END,
last_watched_ms = CASE
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
ELSE last_watched_ms
END,
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
)
.run(
animeId,
activeMs,
cardsMined,
linesSeen,
tokensSeen,
isFirstSessionForVideoRun ? 1 : 0,
isFirstCompletedSessionForVideoRun ? 1 : 0,
session.started_at_ms,
session.ended_at_ms,
updatedAtMs,
updatedAtMs,
);
}
}
+51
View File
@@ -0,0 +1,51 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { assertSafeSshHost, resolveRemoteSubminerCommand, runScp, shellQuote } from './ssh.js';
test('assertSafeSshHost rejects option-like hosts', () => {
assert.throws(() => assertSafeSshHost('-oProxyCommand=touch pwned'), /looks like an option/);
assert.throws(() => assertSafeSshHost('-lroot'), /looks like an option/);
});
test('assertSafeSshHost accepts normal destinations', () => {
assert.doesNotThrow(() => assertSafeSshHost('macbook'));
assert.doesNotThrow(() => assertSafeSshHost('user@192.168.1.20'));
assert.doesNotThrow(() => assertSafeSshHost('ssh-alias'));
});
test('shellQuote escapes single quotes and wraps in quotes', () => {
assert.equal(shellQuote('subminer'), `'subminer'`);
assert.equal(shellQuote(`a'; rm -rf ~; '`), `'a'\\''; rm -rf ~; '\\'''`);
});
test('runScp rejects option-like local endpoints before spawning scp', () => {
assert.throws(() => runScp('-oProxyCommand=sh', '/tmp/out.sqlite'), /looks like an option/);
assert.throws(() => runScp('/tmp/in.sqlite', '-bad-destination'), /looks like an option/);
});
test('runScp rejects option-like remote host components', () => {
assert.throws(
() => runScp('-oProxyCommand=sh:/tmp/in.sqlite', '/tmp/out.sqlite'),
/SSH host that looks like an option/,
);
});
test('resolveRemoteSubminerCommand verifies the launcher under the remote runtime PATH', () => {
const calls: Array<{ host: string; remoteCommand: string }> = [];
const command = resolveRemoteSubminerCommand('macbook', null, (host, remoteCommand) => {
calls.push({ host, remoteCommand });
return { status: 0, stdout: '', stderr: '' };
});
assert.equal(
command,
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer',
);
assert.deepEqual(calls, [
{
host: 'macbook',
remoteCommand:
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer --help >/dev/null 2>&1',
},
]);
});
+106
View File
@@ -0,0 +1,106 @@
import { spawnSync } from 'node:child_process';
export interface RemoteRunResult {
status: number;
stdout: string;
stderr: string;
}
/**
* ssh/scp have no `--` terminator for the destination, so a host that starts
* with `-` (e.g. `-oProxyCommand=...`) is parsed as an option. Reject those
* before spawning.
*/
export function assertSafeSshHost(host: string): void {
if (host.startsWith('-')) {
throw new Error(`Refusing to use SSH host that looks like an option: ${host}`);
}
}
/**
* Run a command on the SSH host. stdin stays attached so interactive prompts
* can still read from the terminal; stdout/stderr are captured for callers
* that need actionable remote failure messages.
*/
export function runSsh(host: string, remoteCommand: string): RemoteRunResult {
assertSafeSshHost(host);
const result = spawnSync('ssh', [host, remoteCommand], {
encoding: 'utf8',
stdio: ['inherit', 'pipe', 'pipe'],
});
if (result.error) {
throw new Error(`Failed to run ssh: ${(result.error as Error).message}`);
}
return { status: result.status ?? 1, stdout: result.stdout ?? '', stderr: result.stderr ?? '' };
}
function assertSafeScpEndpoint(endpoint: string): void {
const colon = endpoint.indexOf(':');
const slash = endpoint.indexOf('/');
if (colon <= 0 || (slash !== -1 && slash < colon)) {
if (endpoint.startsWith('-')) {
throw new Error(`Refusing to use scp endpoint that looks like an option: ${endpoint}`);
}
return;
}
const host = endpoint.slice(0, colon);
const remotePath = endpoint.slice(colon + 1);
assertSafeSshHost(host);
if (remotePath.startsWith('-')) {
throw new Error(`Refusing to use scp remote path that looks like an option: ${remotePath}`);
}
}
export function runScp(from: string, to: string): void {
assertSafeScpEndpoint(from);
assertSafeScpEndpoint(to);
const result = spawnSync('scp', ['-q', from, to], {
encoding: 'utf8',
stdio: ['inherit', 'inherit', 'inherit'],
});
if (result.error) {
throw new Error(`Failed to run scp: ${(result.error as Error).message}`);
}
if ((result.status ?? 1) !== 0) {
throw new Error(`scp failed copying ${from} -> ${to}`);
}
}
export function shellQuote(value: string): string {
return `'${value.replaceAll("'", `'\\''`)}'`;
}
const REMOTE_RUNTIME_PATH =
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"';
/**
* Non-interactive SSH shells often miss user-installed launchers and Bun.
* Probe the launcher under the same deterministic PATH used by sync itself.
*/
export function resolveRemoteSubminerCommand(
host: string,
preferred: string | null,
runRemote: typeof runSsh = runSsh,
): string {
// Trusted defaults stay unquoted so the remote shell expands `~`; a
// user-supplied override is shell-quoted to prevent command injection.
const candidates: Array<{ value: string; invocation: string }> = preferred
? [{ value: preferred, invocation: shellQuote(preferred) }]
: [
{ value: 'subminer', invocation: 'subminer' },
{ value: '~/.local/bin/subminer', invocation: '~/.local/bin/subminer' },
];
for (const candidate of candidates) {
const command = `${REMOTE_RUNTIME_PATH} ${candidate.invocation}`;
const probe = runRemote(host, `${command} --help >/dev/null 2>&1`);
if (probe.status === 0) {
return command;
}
}
throw new Error(
preferred
? `Remote command not found on ${host}: ${preferred}`
: `subminer not found on ${host} (tried PATH and ~/.local/bin/subminer). Pass --remote-cmd <path>.`,
);
}
+598
View File
@@ -0,0 +1,598 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { createDbSnapshot, mergeSnapshotIntoDb } from './sync-db.js';
import {
createImmersionDbFixture,
insertFixtureSession,
} from '../test-support/immersion-db-fixture.js';
const DAY_MS = 86_400_000;
const BASE_MS = Date.UTC(2026, 5, 1, 12, 0, 0);
function makeTmpDir(): string {
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-test-'));
}
function makeDbPair(): { dir: string; localPath: string; remotePath: string } {
const dir = makeTmpDir();
const localPath = path.join(dir, 'local.sqlite');
const remotePath = path.join(dir, 'remote.sqlite');
createImmersionDbFixture(localPath);
createImmersionDbFixture(remotePath);
return { dir, localPath, remotePath };
}
function queryOne<T extends Record<string, unknown>>(
dbPath: string,
sql: string,
params: unknown[] = [],
): T | undefined {
const db = new Database(dbPath, { readonly: true });
try {
return db.query<T>(sql).get(...params) as T | undefined;
} finally {
db.close();
}
}
function count(dbPath: string, sql: string, params: unknown[] = []): number {
return Number(queryOne<{ n: number }>(dbPath, sql, params)?.n ?? 0);
}
function withWritableDb<T>(dbPath: string, fn: (db: Database) => T): T {
const db = new Database(dbPath, { readwrite: true });
try {
return fn(db);
} finally {
db.close();
}
}
test('merges remote-only sessions with catalog, lifetime, and rollups', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 2 }],
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
animeTitleKey: 'showb',
startedAtMs: BASE_MS + DAY_MS,
activeWatchedMs: 900_000,
cardsMined: 3,
applyLifetime: true,
words: [
{ headword: '見る', word: '見た', reading: 'みた', count: 5 },
{ headword: '食べる', word: '食べた', reading: 'たべた', count: 1 },
],
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 1);
assert.equal(summary.animeAdded, 1);
assert.equal(summary.videosAdded, 1);
assert.equal(summary.wordsAdded, 1);
assert.equal(summary.subtitleLinesAdded, 2);
assert.equal(summary.telemetryRowsAdded, 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 2);
const global = queryOne<{
total_sessions: number;
total_active_ms: number;
total_cards: number;
active_days: number;
episodes_started: number;
}>(
localPath,
'SELECT total_sessions, total_active_ms, total_cards, active_days, episodes_started FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.total_active_ms, 1_200_000 + 900_000);
assert.equal(global?.total_cards, 2 + 3);
assert.equal(global?.active_days, 2);
assert.equal(global?.episodes_started, 2);
// Existing word: local 2 + merged 5; new word carries remote frequency.
const sharedWord = queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '見た'`,
);
assert.equal(sharedWord?.frequency, 7);
const newWord = queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '食べた'`,
);
assert.equal(newWord?.frequency, 1);
// The merged session's rollup group was recomputed.
assert.equal(summary.rollupGroupsRecomputed, 1);
const mergedVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'showb-e1'`,
)?.video_id,
);
assert.equal(
count(localPath, 'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE video_id = ?', [
mergedVideoId,
]),
1,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('is idempotent: re-merging the same snapshot changes nothing', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 4 }],
});
mergeSnapshotIntoDb(localPath, remotePath);
const globalAfterFirst = queryOne<Record<string, unknown>>(
localPath,
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
);
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 0);
assert.equal(summary.sessionsAlreadyPresent, 1);
assert.equal(summary.wordsAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_subtitle_lines'), 1);
const globalAfterSecond = queryOne<Record<string, unknown>>(
localPath,
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
);
assert.deepEqual(
{ ...globalAfterSecond, LAST_UPDATE_DATE: null },
{ ...globalAfterFirst, LAST_UPDATE_DATE: null },
);
assert.equal(
Number(
queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '見た'`,
)?.frequency,
),
4,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('preserves anilist_id when inserting a new anime from the snapshot', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
animeTitleKey: 'showb',
startedAtMs: BASE_MS,
applyLifetime: true,
});
withWritableDb(remotePath, (remoteDb) => {
remoteDb
.prepare(`UPDATE imm_anime SET anilist_id = 12345 WHERE normalized_title_key = 'showb'`)
.run();
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.animeAdded, 1);
const anime = queryOne<{ anilist_id: number }>(
localPath,
`SELECT anilist_id FROM imm_anime WHERE normalized_title_key = 'showb'`,
);
assert.equal(anime?.anilist_id, 12345);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('matches an existing local anime by anilist_id even when the title key differs', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'show-romaji',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e2',
animeTitleKey: 'show-native',
startedAtMs: BASE_MS + DAY_MS,
applyLifetime: true,
});
withWritableDb(localPath, (localDb) => {
localDb
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-romaji'`)
.run();
});
withWritableDb(remotePath, (remoteDb) => {
remoteDb
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-native'`)
.run();
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
// Same anilist_id → one anime, not two.
assert.equal(summary.animeAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_anime'), 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('matches shared videos by video_key and merges the watched flag', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS + DAY_MS,
watched: true,
applyLifetime: true,
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.videosAdded, 0);
assert.equal(summary.animeAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_videos'), 1);
const video = queryOne<{ watched: number }>(
localPath,
`SELECT watched FROM imm_videos WHERE video_key = 'showa-e1'`,
);
assert.equal(video?.watched, 1);
// Both sessions now credit the same video; episode was started once and
// completed once (by the remote session that watched it to the end).
const global = queryOne<{
episodes_started: number;
episodes_completed: number;
total_sessions: number;
}>(
localPath,
'SELECT episodes_started, episodes_completed, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.episodes_started, 1);
assert.equal(global?.episodes_completed, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('backfills media metadata for matched videos when local metadata is absent', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS + DAY_MS,
applyLifetime: true,
});
const remoteDb = new Database(remotePath, { readwrite: true });
try {
const remoteVideoId = Number(
remoteDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`)
.get()?.video_id,
);
remoteDb
.prepare(
`INSERT INTO imm_media_art (video_id, anilist_id, cover_url, title_romaji, fetched_at_ms)
VALUES (?, 123, 'https://example.test/cover.jpg', 'Show A', ?)`,
)
.run(remoteVideoId, String(BASE_MS));
remoteDb
.prepare(
`INSERT INTO imm_youtube_videos (video_id, youtube_video_id, video_url, video_title, fetched_at_ms)
VALUES (?, 'yt-1', 'https://youtube.test/watch?v=yt-1', 'Remote Video', ?)`,
)
.run(remoteVideoId, String(BASE_MS));
} finally {
remoteDb.close();
}
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.videosAdded, 0);
const localVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`,
)?.video_id,
);
const art = queryOne<{ cover_url: string; title_romaji: string }>(
localPath,
'SELECT cover_url, title_romaji FROM imm_media_art WHERE video_id = ?',
[localVideoId],
);
assert.equal(art?.cover_url, 'https://example.test/cover.jpg');
assert.equal(art?.title_romaji, 'Show A');
const youtube = queryOne<{ youtube_video_id: string; video_title: string }>(
localPath,
'SELECT youtube_video_id, video_title FROM imm_youtube_videos WHERE video_id = ?',
[localVideoId],
);
assert.equal(youtube?.youtube_video_id, 'yt-1');
assert.equal(youtube?.video_title, 'Remote Video');
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('does not double-count active_days when a merged session starts earlier on an already-credited day', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
startedAtMs: BASE_MS + 3_600_000,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
mergeSnapshotIntoDb(localPath, remotePath);
const global = queryOne<{ active_days: number; total_sessions: number }>(
localPath,
'SELECT active_days, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.active_days, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('skips unfinished sessions', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-active',
videoKey: 'showa-e1',
startedAtMs: BASE_MS,
endedAtMs: null,
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 0);
assert.equal(summary.activeSessionsSkipped, 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 0);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('copies remote-only historical rollups but never sums shared groups', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
// Remote has a video plus an old rollup row whose sessions were pruned.
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'old-show-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
withWritableDb(remotePath, (remoteDb) => {
const remoteVideoId = Number(
remoteDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
.get()?.video_id,
);
remoteDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (10000, ?, 4, 120.5, 400, 3200, 9)`,
)
.run(remoteVideoId);
});
// Local already has its own rollup row for a shared group.
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'old-show-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const localVideoId = withWritableDb(localPath, (localDb) => {
const videoId = Number(
localDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
.get()?.video_id,
);
localDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (10000, ?, 2, 60.0, 200, 1600, 4)`,
)
.run(videoId);
return videoId;
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
// Shared group (day 10000) untouched; the remote-only session's own group
// was recomputed, not copied.
assert.equal(summary.dailyRollupsCopied, 0);
const shared = queryOne<{ total_sessions: number }>(
localPath,
'SELECT total_sessions FROM imm_daily_rollups WHERE rollup_day = 10000 AND video_id = ?',
[localVideoId],
);
assert.equal(shared?.total_sessions, 2);
// Now a rollup for a video with no local sessions at all gets copied.
withWritableDb(remotePath, (remoteDb2) => {
const orphanVideoId = Number(
remoteDb2
.prepare(
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
VALUES ('pruned-show-e1', 'pruned-show-e1', 1, 1, 1440000)`,
)
.run().lastInsertRowid,
);
remoteDb2
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (9000, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(orphanVideoId);
});
const summary2 = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary2.dailyRollupsCopied, 1);
const copiedVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'pruned-show-e1'`,
)?.video_id,
);
const copied = queryOne<{ total_sessions: number; total_active_min: number }>(
localPath,
'SELECT total_sessions, total_active_min FROM imm_daily_rollups WHERE rollup_day = 9000 AND video_id = ?',
[copiedVideoId],
);
assert.equal(copied?.total_sessions, 3);
assert.equal(copied?.total_active_min, 90.0);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('does not copy remote-only daily rollups into months with local sessions', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'mixed-month-e1',
animeTitleKey: 'mixed-month',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const remoteDb = new Database(remotePath, { readwrite: true });
try {
const remoteVideoId = Number(
remoteDb
.prepare(
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
VALUES ('mixed-month-e1', 'mixed-month-e1', 1, 1, 1440000)`,
)
.run().lastInsertRowid,
);
remoteDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (20615, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(remoteVideoId);
remoteDb
.prepare(
`INSERT INTO imm_monthly_rollups (rollup_month, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (202606, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(remoteVideoId);
} finally {
remoteDb.close();
}
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.dailyRollupsCopied, 0);
assert.equal(summary.monthlyRollupsCopied, 0);
const localVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'mixed-month-e1'`,
)?.video_id,
);
assert.equal(
count(
localPath,
'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE rollup_day = 20615 AND video_id = ?',
[localVideoId],
),
0,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('rejects snapshots at a different schema version', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
withWritableDb(remotePath, (remoteDb) => {
remoteDb.prepare('UPDATE imm_schema_version SET schema_version = 17').run();
});
assert.throws(() => mergeSnapshotIntoDb(localPath, remotePath), /schema version 17/);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('createDbSnapshot produces a mergeable copy', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const snapshotPath = path.join(dir, 'snapshot.sqlite');
createDbSnapshot(remotePath, snapshotPath);
assert.ok(fs.existsSync(snapshotPath));
const summary = mergeSnapshotIntoDb(localPath, snapshotPath);
assert.equal(summary.sessionsMerged, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
+105
View File
@@ -0,0 +1,105 @@
import fs from 'node:fs';
import { Database } from 'bun:sqlite';
import {
LexiconResolver,
mergeAnime,
mergeExcludedWords,
mergeMediaMetadata,
mergeVideos,
} from './merge-catalog.js';
import { mergeSessions } from './merge-sessions.js';
import { copyRemoteOnlyRollups, refreshRollupsForNewSessions } from './merge-rollups.js';
import {
assertMergeableSchema,
createEmptyMergeSummary,
type SyncMergeSummary,
} from './sync-shared.js';
export type { SyncMergeSummary } from './sync-shared.js';
export { createDbSnapshot, findLiveStatsDaemonPid } from './sync-shared.js';
/**
* Merge a snapshot of another machine's immersion database into the local
* one. Insert-only union keyed on natural keys (session_uuid, video_key,
* normalized_title_key, word/kanji identity); lifetime and rollup aggregates
* are updated incrementally so history older than the session retention
* window is preserved on both sides. Idempotent: re-merging the same
* snapshot is a no-op.
*/
export function mergeSnapshotIntoDb(localDbPath: string, snapshotPath: string): SyncMergeSummary {
if (!fs.existsSync(localDbPath)) {
throw new Error(`Local stats database not found: ${localDbPath}`);
}
if (!fs.existsSync(snapshotPath)) {
throw new Error(`Snapshot database not found: ${snapshotPath}`);
}
const remote = new Database(snapshotPath, { readonly: true });
let local: Database;
try {
local = new Database(localDbPath, { readwrite: true, create: false });
} catch (error) {
remote.close();
throw error;
}
try {
assertMergeableSchema(remote, 'Snapshot');
assertMergeableSchema(local, 'Local');
const summary = createEmptyMergeSummary();
local.run('PRAGMA foreign_keys = ON');
local.run('PRAGMA busy_timeout = 5000');
local.run('BEGIN IMMEDIATE');
try {
const animeIdMap = mergeAnime(local, remote, summary);
const { videoIdMap, addedVideoIds } = mergeVideos(local, remote, animeIdMap, summary);
mergeMediaMetadata(local, remote, videoIdMap, addedVideoIds);
mergeExcludedWords(local, remote, summary);
const lexicon = new LexiconResolver(local, remote, summary);
const { newSessionIds } = mergeSessions(
local,
remote,
videoIdMap,
animeIdMap,
lexicon,
summary,
);
lexicon.applyFrequencyDeltas();
refreshRollupsForNewSessions(local, newSessionIds, summary);
copyRemoteOnlyRollups(local, remote, videoIdMap, summary);
local.run('COMMIT');
return summary;
} catch (error) {
local.run('ROLLBACK');
throw error;
}
} finally {
local.close();
remote.close();
}
}
export function formatMergeSummary(summary: SyncMergeSummary): string {
const lines = [
`Sessions merged: ${summary.sessionsMerged} (${summary.sessionsAlreadyPresent} already present, ${summary.activeSessionsSkipped} unfinished skipped)`,
];
const detail: string[] = [];
if (summary.animeAdded) detail.push(`${summary.animeAdded} series`);
if (summary.videosAdded) detail.push(`${summary.videosAdded} videos`);
if (summary.wordsAdded) detail.push(`${summary.wordsAdded} words`);
if (summary.kanjiAdded) detail.push(`${summary.kanjiAdded} kanji`);
if (summary.subtitleLinesAdded) detail.push(`${summary.subtitleLinesAdded} subtitle lines`);
if (summary.excludedWordsAdded) detail.push(`${summary.excludedWordsAdded} excluded words`);
if (detail.length > 0) lines.push(`Added: ${detail.join(', ')}`);
if (summary.dailyRollupsCopied || summary.monthlyRollupsCopied) {
lines.push(
`Historical rollups copied: ${summary.dailyRollupsCopied} daily, ${summary.monthlyRollupsCopied} monthly`,
);
}
if (summary.rollupGroupsRecomputed) {
lines.push(`Rollup groups recomputed: ${summary.rollupGroupsRecomputed}`);
}
return lines.join('\n');
}
+173
View File
@@ -0,0 +1,173 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
import { withReadonlyWalRetry } from '../history-db.js';
import { resolveConfigDir } from '../../src/config/path-resolution.js';
export { SCHEMA_VERSION };
export interface SyncMergeSummary {
sessionsMerged: number;
sessionsAlreadyPresent: number;
activeSessionsSkipped: number;
animeAdded: number;
videosAdded: number;
wordsAdded: number;
kanjiAdded: number;
subtitleLinesAdded: number;
telemetryRowsAdded: number;
eventsAdded: number;
excludedWordsAdded: number;
dailyRollupsCopied: number;
monthlyRollupsCopied: number;
rollupGroupsRecomputed: number;
}
export function createEmptyMergeSummary(): SyncMergeSummary {
return {
sessionsMerged: 0,
sessionsAlreadyPresent: 0,
activeSessionsSkipped: 0,
animeAdded: 0,
videosAdded: 0,
wordsAdded: 0,
kanjiAdded: 0,
subtitleLinesAdded: 0,
telemetryRowsAdded: 0,
eventsAdded: 0,
excludedWordsAdded: 0,
dailyRollupsCopied: 0,
monthlyRollupsCopied: 0,
rollupGroupsRecomputed: 0,
};
}
export function nowDbTimestamp(): string {
return String(Date.now());
}
export function tableExists(db: Database, tableName: string): boolean {
return Boolean(
db.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?`).get(tableName),
);
}
export function readSchemaVersion(db: Database): number | null {
if (!tableExists(db, 'imm_schema_version')) return null;
const row = db
.query<{ schema_version: number }>(
'SELECT MAX(schema_version) AS schema_version FROM imm_schema_version',
)
.get();
return typeof row?.schema_version === 'number' ? row.schema_version : null;
}
export function assertMergeableSchema(db: Database, label: string): void {
const version = readSchemaVersion(db);
if (version === null) {
throw new Error(
`${label} database has no schema version. Run SubMiner once on that machine so the stats database is initialized.`,
);
}
if (version !== SCHEMA_VERSION) {
throw new Error(
`${label} database is at schema version ${version} but this launcher expects ${SCHEMA_VERSION}. Update SubMiner on both machines to the same version and run each app once before syncing.`,
);
}
for (const table of ['imm_sessions', 'imm_videos', 'imm_lifetime_global']) {
if (!tableExists(db, table)) {
throw new Error(`${label} database is missing table ${table}; cannot sync.`);
}
}
}
export function insertRow(
db: Database,
table: string,
columns: readonly string[],
values: unknown[],
): number {
const sql = `INSERT INTO ${table} (${columns.join(', ')}) VALUES (${columns.map(() => '?').join(', ')})`;
// db.query() caches the prepared statement per SQL string; this runs once
// per copied row, so re-preparing via db.prepare() would dominate merge time.
const result = db.query(sql).run(...values);
return Number(result.lastInsertRowid);
}
export function createDbSnapshot(dbPath: string, outPath: string): void {
if (!fs.existsSync(dbPath)) {
throw new Error(`Stats database not found: ${dbPath}`);
}
fs.rmSync(outPath, { force: true });
fs.mkdirSync(path.dirname(outPath), { recursive: true });
withReadonlyWalRetry(dbPath, (options) => {
const db = new Database(dbPath, options);
try {
assertMergeableSchema(db, 'Local');
db.prepare('VACUUM INTO ?').run(outPath);
} finally {
db.close();
}
});
}
interface DaemonStateFile {
pid?: unknown;
}
function isProcessAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch (error) {
// EPERM means the process exists but we can't signal it → still alive.
// Only ESRCH (no such process) means it's actually gone.
return (error as NodeJS.ErrnoException)?.code === 'EPERM';
}
}
function statsDaemonStateCandidates(dbPath: string): string[] {
const homeDir = os.homedir();
const candidates = new Set<string>([path.join(path.dirname(dbPath), 'stats-daemon.json')]);
const configDir = resolveConfigDir({
platform: process.platform,
appDataDir: process.env.APPDATA,
xdgConfigHome: process.env.XDG_CONFIG_HOME,
homeDir,
existsSync: fs.existsSync,
});
candidates.add(path.join(configDir, 'stats-daemon.json'));
if (process.platform === 'darwin') {
candidates.add(path.join(homeDir, 'Library', 'Application Support', 'SubMiner', 'stats-daemon.json'));
}
return [...candidates];
}
/**
* Best-effort guard against merging while a SubMiner process holds the
* tracker's write queue in memory. Detects the background stats daemon via
* its pid state file; the interactive app is caught by the mpv-socket check
* in the sync command.
*/
export function findLiveStatsDaemonPid(dbPath: string): number | null {
for (const statePath of statsDaemonStateCandidates(dbPath)) {
let raw: string;
try {
raw = fs.readFileSync(statePath, 'utf8');
} catch {
continue;
}
try {
const parsed = JSON.parse(raw) as DaemonStateFile;
const pid = typeof parsed.pid === 'number' && Number.isInteger(parsed.pid) ? parsed.pid : 0;
if (pid > 0 && isProcessAlive(pid)) {
return pid;
}
} catch {
continue;
}
}
return null;
}
@@ -0,0 +1,323 @@
import { Database } from 'bun:sqlite';
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
import { IMMERSION_DB_FIXTURE_DDL } from './immersion-db-schema.js';
export function createImmersionDbFixture(dbPath: string): void {
const db = new Database(dbPath, { create: true });
try {
db.run('PRAGMA foreign_keys = ON');
db.run('PRAGMA journal_mode = WAL');
for (const statement of IMMERSION_DB_FIXTURE_DDL.split(';')) {
const sql = statement.trim();
if (sql) db.run(sql);
}
db.prepare('INSERT INTO imm_schema_version (schema_version, applied_at_ms) VALUES (?, ?)').run(
SCHEMA_VERSION,
String(Date.now()),
);
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0)`,
).run();
db.prepare(
`INSERT INTO imm_lifetime_global(global_id, CREATED_DATE, LAST_UPDATE_DATE) VALUES (1, ?, ?)`,
).run(String(Date.now()), String(Date.now()));
} finally {
db.close();
}
}
let uniqueCounter = 0;
export interface FixtureSessionInput {
uuid: string;
videoKey: string;
animeTitleKey?: string | null;
animeEpisodesTotal?: number | null;
startedAtMs: number;
endedAtMs?: number | null;
activeWatchedMs?: number;
cardsMined?: number;
linesSeen?: number;
tokensSeen?: number;
watched?: boolean;
words?: Array<{ headword: string; word: string; reading: string; count: number }>;
applyLifetime?: boolean;
}
/**
* Insert a session (with video/anime/subtitle-line/occurrence rows) the way
* the app would have recorded it, optionally crediting lifetime aggregates
* the way applySessionLifetimeSummary does for locally-recorded sessions.
*/
export function insertFixtureSession(dbPath: string, input: FixtureSessionInput): void {
const db = new Database(dbPath, { readwrite: true });
const stamp = String(input.startedAtMs);
try {
db.run('PRAGMA foreign_keys = ON');
let animeId: number | null = null;
if (input.animeTitleKey) {
const existing = db
.query<{
anime_id: number;
}>('SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?')
.get(input.animeTitleKey);
animeId = existing
? existing.anime_id
: Number(
db
.prepare(
`INSERT INTO imm_anime (normalized_title_key, canonical_title, episodes_total, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?)`,
)
.run(
input.animeTitleKey,
input.animeTitleKey,
input.animeEpisodesTotal ?? null,
stamp,
stamp,
).lastInsertRowid,
);
}
const existingVideo = db
.query<{ video_id: number }>('SELECT video_id FROM imm_videos WHERE video_key = ?')
.get(input.videoKey);
const videoId = existingVideo
? existingVideo.video_id
: Number(
db
.prepare(
`INSERT INTO imm_videos (video_key, anime_id, canonical_title, source_type, source_path, watched, duration_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, 1, ?, ?, 1440000, ?, ?)`,
)
.run(
input.videoKey,
animeId,
input.videoKey,
`/videos/${input.videoKey}.mkv`,
input.watched ? 1 : 0,
stamp,
stamp,
).lastInsertRowid,
);
if (input.watched) {
db.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?').run(videoId);
}
const endedAtMs =
input.endedAtMs === undefined ? input.startedAtMs + 1_500_000 : input.endedAtMs;
const activeWatchedMs = input.activeWatchedMs ?? 1_200_000;
const cardsMined = input.cardsMined ?? 2;
const linesSeen = input.linesSeen ?? 100;
const tokensSeen = input.tokensSeen ?? 800;
const sessionId = Number(
db
.prepare(
`INSERT INTO imm_sessions (
session_uuid, video_id, started_at_ms, ended_at_ms, status,
total_watched_ms, active_watched_ms, lines_seen, tokens_seen, cards_mined,
CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run(
input.uuid,
videoId,
String(input.startedAtMs),
endedAtMs === null ? null : String(endedAtMs),
endedAtMs === null ? 1 : 2,
activeWatchedMs,
activeWatchedMs,
linesSeen,
tokensSeen,
cardsMined,
stamp,
stamp,
).lastInsertRowid,
);
db.prepare(
`INSERT INTO imm_session_telemetry (
session_id, sample_ms, total_watched_ms, active_watched_ms, lines_seen, tokens_seen,
cards_mined, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
sessionId,
String(endedAtMs ?? input.startedAtMs),
activeWatchedMs,
activeWatchedMs,
linesSeen,
tokensSeen,
cardsMined,
stamp,
stamp,
);
for (const word of input.words ?? []) {
const existing = db
.query<{
id: number;
}>('SELECT id FROM imm_words WHERE headword = ? AND word = ? AND reading = ?')
.get(word.headword, word.word, word.reading);
const wordId = existing
? existing.id
: Number(
db
.prepare(
`INSERT INTO imm_words (headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 0)`,
)
.run(
word.headword,
word.word,
word.reading,
Math.floor(input.startedAtMs / 1000),
Math.floor(input.startedAtMs / 1000),
).lastInsertRowid,
);
uniqueCounter += 1;
const lineId = Number(
db
.prepare(
`INSERT INTO imm_subtitle_lines (session_id, video_id, anime_id, line_index, text, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
)
.run(
sessionId,
videoId,
animeId,
uniqueCounter,
`line ${word.word}`,
input.startedAtMs,
input.startedAtMs,
).lastInsertRowid,
);
db.prepare(
'INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)',
).run(lineId, wordId, word.count);
db.prepare('UPDATE imm_words SET frequency = frequency + ? WHERE id = ?').run(
word.count,
wordId,
);
}
if (input.applyLifetime && endedAtMs !== null) {
applyFixtureLifetime(db, sessionId, videoId, animeId, {
endedAtMs,
startedAtMs: input.startedAtMs,
activeWatchedMs,
cardsMined,
linesSeen,
tokensSeen,
watched: Boolean(input.watched),
episodesTotal: input.animeEpisodesTotal ?? null,
});
}
} finally {
db.close();
}
}
function applyFixtureLifetime(
db: Database,
sessionId: number,
videoId: number,
animeId: number | null,
data: {
endedAtMs: number;
startedAtMs: number;
activeWatchedMs: number;
cardsMined: number;
linesSeen: number;
tokensSeen: number;
watched: boolean;
episodesTotal: number | null;
},
): void {
const stamp = String(data.endedAtMs);
db.prepare(
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)`,
).run(sessionId, String(data.endedAtMs), stamp, stamp);
const mediaLifetime = db
.query<{ completed: number }>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
.get(videoId);
const isFirstSessionForVideo = !mediaLifetime;
const isFirstCompleted = data.watched && Number(mediaLifetime?.completed ?? 0) <= 0;
const dayExpr = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
const otherOnDay = db
.query(
`SELECT 1 FROM imm_sessions WHERE session_id != ? AND ${dayExpr} =
CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER) LIMIT 1`,
)
.get(sessionId, String(data.startedAtMs));
db.prepare(
`UPDATE imm_lifetime_global SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + ?,
total_cards = total_cards + ?,
active_days = active_days + ?,
episodes_started = episodes_started + ?,
episodes_completed = episodes_completed + ?
WHERE global_id = 1`,
).run(
data.activeWatchedMs,
data.cardsMined,
otherOnDay ? 0 : 1,
isFirstSessionForVideo ? 1 : 0,
isFirstCompleted ? 1 : 0,
);
db.prepare(
`INSERT INTO imm_lifetime_media (video_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(video_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
completed = MAX(completed, excluded.completed),
last_watched_ms = excluded.last_watched_ms`,
).run(
videoId,
data.activeWatchedMs,
data.cardsMined,
data.linesSeen,
data.tokensSeen,
data.watched ? 1 : 0,
String(data.startedAtMs),
String(data.endedAtMs),
stamp,
stamp,
);
if (animeId !== null) {
db.prepare(
`INSERT INTO imm_lifetime_anime (anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, episodes_started, episodes_completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(anime_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
episodes_started = episodes_started + excluded.episodes_started,
episodes_completed = episodes_completed + excluded.episodes_completed,
last_watched_ms = excluded.last_watched_ms`,
).run(
animeId,
data.activeWatchedMs,
data.cardsMined,
data.linesSeen,
data.tokensSeen,
isFirstSessionForVideo ? 1 : 0,
isFirstCompleted ? 1 : 0,
String(data.startedAtMs),
String(data.endedAtMs),
stamp,
stamp,
);
}
}
@@ -0,0 +1,141 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database as BunDatabase } from 'bun:sqlite';
import { ensureSchema } from '../../src/core/services/immersion-tracker/storage.js';
import { createImmersionDbFixture, insertFixtureSession } from './immersion-db-fixture.js';
type SchemaRow = { type: string; name: string; tbl_name: string; sql: string | null };
const SYNC_SCHEMA_OBJECTS = [
'imm_anime',
'imm_videos',
'imm_sessions',
'imm_session_telemetry',
'imm_session_events',
'imm_daily_rollups',
'imm_monthly_rollups',
'imm_words',
'imm_kanji',
'imm_subtitle_lines',
'imm_word_line_occurrences',
'imm_kanji_line_occurrences',
'imm_media_art',
'imm_youtube_videos',
'imm_cover_art_blobs',
'imm_lifetime_global',
'imm_lifetime_anime',
'imm_lifetime_media',
'imm_lifetime_applied_sessions',
'imm_stats_excluded_words',
'idx_anime_normalized_title',
'idx_anime_anilist_id',
'idx_videos_anime_id',
'idx_sessions_video_started',
'idx_sessions_status_started',
'idx_sessions_started_at',
'idx_sessions_ended_at',
'idx_telemetry_session_sample',
'idx_telemetry_sample_ms',
'idx_events_session_ts',
'idx_events_type_ts',
'idx_rollups_day_video',
'idx_rollups_month_video',
'idx_words_headword_word_reading',
'idx_words_frequency',
'idx_kanji_kanji',
'idx_kanji_frequency',
'idx_subtitle_lines_session_line',
'idx_subtitle_lines_video_line',
'idx_subtitle_lines_anime_line',
'idx_word_line_occurrences_word',
'idx_kanji_line_occurrences_kanji',
'idx_media_art_cover_blob_hash',
'idx_media_art_anilist_id',
'idx_media_art_cover_url',
'idx_youtube_videos_channel_id',
'idx_youtube_videos_youtube_video_id',
] as const;
function normalizeSql(sql: string | null): string {
return (sql ?? '')
.replace(/\bIF NOT EXISTS\b/gi, '')
.replace(/\s+/g, ' ')
.replace(/\s+([(),])/g, '$1')
.replace(/,\s+/g, ', ')
.trim();
}
function readSchema(dbPath: string): Map<string, string> {
const db = new BunDatabase(dbPath, { readonly: true });
try {
const rows = db
.query<SchemaRow>(
`SELECT type, name, tbl_name, sql
FROM sqlite_schema
WHERE name IN (${SYNC_SCHEMA_OBJECTS.map(() => '?').join(',')})
ORDER BY type, name`,
)
.all(...SYNC_SCHEMA_OBJECTS);
return new Map(rows.map((row) => [row.name, normalizeSql(row.sql)]));
} finally {
db.close();
}
}
test('fixture schema stays aligned with production sync-touched tables and indexes', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-schema-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
const productionPath = path.join(dir, 'production.sqlite');
const productionDb = new BunDatabase(productionPath, { create: true });
try {
createImmersionDbFixture(fixturePath);
ensureSchema(productionDb as never);
productionDb.close();
const fixtureSchema = readSchema(fixturePath);
const productionSchema = readSchema(productionPath);
assert.deepEqual(fixtureSchema, productionSchema);
} finally {
try {
productionDb.close();
} catch {
// already closed
}
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('fixture session inserts enforce foreign keys', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-foreign-keys-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
try {
createImmersionDbFixture(fixturePath);
const db = new BunDatabase(fixturePath, { readwrite: true });
try {
db.run(`
CREATE TRIGGER remove_fixture_video
BEFORE INSERT ON imm_sessions
BEGIN
DELETE FROM imm_videos WHERE video_id = NEW.video_id;
END
`);
} finally {
db.close();
}
assert.throws(
() =>
insertFixtureSession(fixturePath, {
uuid: 'foreign-key-check',
videoKey: 'foreign-key-check',
startedAtMs: Date.UTC(2026, 6, 9),
}),
/FOREIGN KEY constraint failed/,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
@@ -0,0 +1,323 @@
// Schema-version-18 shape of the tables the sync merge touches (plus the
// app's indexes), mirroring ensureSchema / ensureLifetimeSummaryTables /
// ensureStatsExcludedWordsTable in src/core/services/immersion-tracker/storage.ts.
export const IMMERSION_DB_FIXTURE_DDL = `
CREATE TABLE imm_schema_version (
schema_version INTEGER PRIMARY KEY,
applied_at_ms TEXT NOT NULL
);
CREATE TABLE imm_rollup_state(
state_key TEXT PRIMARY KEY,
state_value TEXT NOT NULL
);
CREATE TABLE imm_anime(
anime_id INTEGER PRIMARY KEY AUTOINCREMENT,
normalized_title_key TEXT NOT NULL UNIQUE,
canonical_title TEXT NOT NULL,
anilist_id INTEGER UNIQUE,
title_romaji TEXT,
title_english TEXT,
title_native TEXT,
episodes_total INTEGER,
description TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_videos(
video_id INTEGER PRIMARY KEY AUTOINCREMENT,
video_key TEXT NOT NULL UNIQUE,
anime_id INTEGER,
canonical_title TEXT NOT NULL,
source_type INTEGER NOT NULL,
source_path TEXT,
source_url TEXT,
parsed_basename TEXT,
parsed_title TEXT,
parsed_season INTEGER,
parsed_episode INTEGER,
parser_source TEXT,
parser_confidence REAL,
parse_metadata_json TEXT,
watched INTEGER NOT NULL DEFAULT 0,
duration_ms INTEGER NOT NULL CHECK(duration_ms>=0),
file_size_bytes INTEGER CHECK(file_size_bytes>=0),
codec_id INTEGER, container_id INTEGER,
width_px INTEGER, height_px INTEGER, fps_x100 INTEGER,
bitrate_kbps INTEGER, audio_codec_id INTEGER,
hash_sha256 TEXT, screenshot_path TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_sessions(
session_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_uuid TEXT NOT NULL UNIQUE,
video_id INTEGER NOT NULL,
started_at_ms TEXT NOT NULL, ended_at_ms TEXT,
status INTEGER NOT NULL,
locale_id INTEGER, target_lang_id INTEGER,
difficulty_tier INTEGER, subtitle_mode INTEGER,
ended_media_ms INTEGER,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id)
);
CREATE TABLE imm_session_telemetry(
telemetry_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
sample_ms TEXT NOT NULL,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_session_events(
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
ts_ms TEXT NOT NULL,
event_type INTEGER NOT NULL,
line_index INTEGER,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
tokens_delta INTEGER NOT NULL DEFAULT 0,
cards_delta INTEGER NOT NULL DEFAULT 0,
payload_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_daily_rollups(
rollup_day INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
cards_per_hour REAL,
tokens_per_min REAL,
lookup_hit_rate REAL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_day, video_id)
);
CREATE TABLE imm_monthly_rollups(
rollup_month INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_month, video_id)
);
CREATE TABLE imm_words(
id INTEGER PRIMARY KEY AUTOINCREMENT,
headword TEXT,
word TEXT,
reading TEXT,
part_of_speech TEXT,
pos1 TEXT,
pos2 TEXT,
pos3 TEXT,
first_seen REAL,
last_seen REAL,
frequency INTEGER,
frequency_rank INTEGER,
UNIQUE(headword, word, reading)
);
CREATE TABLE imm_kanji(
id INTEGER PRIMARY KEY AUTOINCREMENT,
kanji TEXT,
first_seen REAL,
last_seen REAL,
frequency INTEGER,
UNIQUE(kanji)
);
CREATE TABLE imm_subtitle_lines(
line_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
event_id INTEGER,
video_id INTEGER NOT NULL,
anime_id INTEGER,
line_index INTEGER NOT NULL,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
text TEXT NOT NULL,
secondary_text TEXT,
CREATED_DATE INTEGER,
LAST_UPDATE_DATE INTEGER,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE,
FOREIGN KEY(event_id) REFERENCES imm_session_events(event_id) ON DELETE SET NULL,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_word_line_occurrences(
line_id INTEGER NOT NULL,
word_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, word_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(word_id) REFERENCES imm_words(id) ON DELETE CASCADE
);
CREATE TABLE imm_kanji_line_occurrences(
line_id INTEGER NOT NULL,
kanji_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, kanji_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(kanji_id) REFERENCES imm_kanji(id) ON DELETE CASCADE
);
CREATE TABLE imm_media_art(
video_id INTEGER PRIMARY KEY,
anilist_id INTEGER,
cover_url TEXT,
cover_blob BLOB,
cover_blob_hash TEXT,
title_romaji TEXT,
title_english TEXT,
episodes_total INTEGER,
fetched_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_youtube_videos(
video_id INTEGER PRIMARY KEY,
youtube_video_id TEXT NOT NULL,
video_url TEXT NOT NULL,
video_title TEXT,
video_thumbnail_url TEXT,
channel_id TEXT,
channel_name TEXT,
channel_url TEXT,
channel_thumbnail_url TEXT,
uploader_id TEXT,
uploader_url TEXT,
description TEXT,
metadata_json TEXT,
fetched_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_cover_art_blobs(
blob_hash TEXT PRIMARY KEY,
cover_blob BLOB NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_lifetime_global(
global_id INTEGER PRIMARY KEY CHECK(global_id = 1),
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
active_days INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
anime_completed INTEGER NOT NULL DEFAULT 0,
last_rebuilt_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_lifetime_anime(
anime_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_media(
video_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_applied_sessions(
session_id INTEGER PRIMARY KEY,
applied_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_stats_excluded_words(
headword TEXT NOT NULL,
word TEXT NOT NULL,
reading TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY(headword, word, reading)
);
CREATE INDEX idx_anime_normalized_title ON imm_anime(normalized_title_key);
CREATE INDEX idx_anime_anilist_id ON imm_anime(anilist_id);
CREATE INDEX idx_videos_anime_id ON imm_videos(anime_id);
CREATE INDEX idx_sessions_video_started ON imm_sessions(video_id, started_at_ms DESC);
CREATE INDEX idx_sessions_status_started ON imm_sessions(status, started_at_ms DESC);
CREATE INDEX idx_sessions_started_at ON imm_sessions(started_at_ms DESC);
CREATE INDEX idx_sessions_ended_at ON imm_sessions(ended_at_ms DESC);
CREATE INDEX idx_telemetry_session_sample ON imm_session_telemetry(session_id, sample_ms DESC);
CREATE INDEX idx_telemetry_sample_ms ON imm_session_telemetry(sample_ms DESC);
CREATE INDEX idx_events_session_ts ON imm_session_events(session_id, ts_ms DESC);
CREATE INDEX idx_events_type_ts ON imm_session_events(event_type, ts_ms DESC);
CREATE INDEX idx_rollups_day_video ON imm_daily_rollups(rollup_day, video_id);
CREATE INDEX idx_rollups_month_video ON imm_monthly_rollups(rollup_month, video_id);
CREATE INDEX idx_words_headword_word_reading ON imm_words(headword, word, reading);
CREATE INDEX idx_words_frequency ON imm_words(frequency DESC);
CREATE INDEX idx_kanji_kanji ON imm_kanji(kanji);
CREATE INDEX idx_kanji_frequency ON imm_kanji(frequency DESC);
CREATE INDEX idx_subtitle_lines_session_line ON imm_subtitle_lines(session_id, line_index);
CREATE INDEX idx_subtitle_lines_video_line ON imm_subtitle_lines(video_id, line_index);
CREATE INDEX idx_subtitle_lines_anime_line ON imm_subtitle_lines(anime_id, line_index);
CREATE INDEX idx_word_line_occurrences_word ON imm_word_line_occurrences(word_id, line_id);
CREATE INDEX idx_kanji_line_occurrences_kanji ON imm_kanji_line_occurrences(kanji_id, line_id);
CREATE INDEX idx_media_art_cover_blob_hash ON imm_media_art(cover_blob_hash);
CREATE INDEX idx_media_art_anilist_id ON imm_media_art(anilist_id);
CREATE INDEX idx_media_art_cover_url ON imm_media_art(cover_url);
CREATE INDEX idx_youtube_videos_channel_id ON imm_youtube_videos(channel_id);
CREATE INDEX idx_youtube_videos_youtube_video_id ON imm_youtube_videos(youtube_video_id);
`;
+9
View File
@@ -112,6 +112,15 @@ export interface Args {
texthookerOnly: boolean;
texthookerOpenBrowser: boolean;
useRofi: boolean;
history: boolean;
sync: boolean;
syncHost: string;
syncSnapshotPath: string;
syncMergePath: string;
syncDirection: 'both' | 'push' | 'pull';
syncRemoteCmd: string;
syncDbPath: string;
syncForce: boolean;
logLevel: LogLevel;
logRotation: LogRotation;
passwordStore: string;
-233
View File
@@ -1,233 +0,0 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const node_os_1 = __importDefault(require("node:os"));
const node_child_process_1 = require("node:child_process");
const electron_1 = require("electron");
const help_1 = require("./cli/help");
const main_entry_runtime_1 = require("./main-entry-runtime");
const early_single_instance_1 = require("./main/early-single-instance");
const main_entry_launch_config_1 = require("./main-entry-launch-config");
const app_control_client_1 = require("./shared/app-control-client");
const first_run_setup_plugin_1 = require("./main/runtime/first-run-setup-plugin");
const windows_mpv_launch_1 = require("./main/runtime/windows-mpv-launch");
const stats_daemon_entry_1 = require("./stats-daemon-entry");
const fatal_error_1 = require("./main/fatal-error");
const mpv_logging_args_1 = require("./shared/mpv-logging-args");
const log_files_1 = require("./shared/log-files");
const DEFAULT_TEXTHOOKER_PORT = 5174;
function appendWindowsMpvLaunchLog(message, logRotation) {
if (!(0, log_files_1.isLogFileEnabled)('app')) {
return;
}
const timestamp = new Date().toISOString().replace('T', ' ').slice(0, 19);
(0, log_files_1.appendLogLine)(process.env.SUBMINER_APP_LOG?.trim() || (0, log_files_1.resolveDefaultLogFilePath)('app'), `[subminer] - ${timestamp} - INFO - [main:windows-mpv-launch] ${message}`, { rotation: logRotation });
}
function applySanitizedEnv(sanitizedEnv) {
if (sanitizedEnv.NODE_NO_WARNINGS) {
process.env.NODE_NO_WARNINGS = sanitizedEnv.NODE_NO_WARNINGS;
}
if (sanitizedEnv.VK_INSTANCE_LAYERS) {
process.env.VK_INSTANCE_LAYERS = sanitizedEnv.VK_INSTANCE_LAYERS;
}
else {
delete process.env.VK_INSTANCE_LAYERS;
}
}
function resolveBundledWindowsMpvPluginEntrypoint() {
return ((0, first_run_setup_plugin_1.resolvePackagedRuntimePluginPath)({
dirname: __dirname,
appPath: electron_1.app.getAppPath(),
resourcesPath: process.resourcesPath,
}) ?? undefined);
}
function buildInstalledWindowsMpvPluginMessage(pathValue, version) {
return [
'SubMiner detected an installed mpv plugin at:',
pathValue,
'',
"This mpv session will use the installed plugin. Remove it to use SubMiner's bundled runtime plugin automatically.",
`Detected plugin version: ${version ?? 'unknown or legacy'}`,
].join('\n');
}
async function promptForWindowsLegacyMpvPluginRemoval(mpvPath, detection) {
const response = await electron_1.dialog.showMessageBox({
type: 'warning',
title: 'SubMiner mpv plugin detected',
message: buildInstalledWindowsMpvPluginMessage(detection.path ?? 'unknown path', detection.version),
detail: 'Remove the legacy SubMiner mpv plugin files from mpv before launching this video? This moves the files to the OS trash. SubMiner-managed playback will then use the bundled runtime plugin.',
buttons: ['Remove legacy plugin', 'Continue with installed plugin', 'Cancel'],
defaultId: 0,
cancelId: 2,
});
if (response.response === 2) {
return 'cancel';
}
if (response.response === 1) {
return 'continue';
}
const candidates = (0, first_run_setup_plugin_1.detectInstalledFirstRunPluginCandidates)({
platform: 'win32',
homeDir: node_os_1.default.homedir(),
appDataDir: electron_1.app.getPath('appData'),
mpvExecutablePath: mpvPath,
});
const result = await (0, first_run_setup_plugin_1.removeLegacyMpvPluginCandidates)({
candidates,
trashItem: (candidatePath) => electron_1.shell.trashItem(candidatePath),
});
if (result.ok) {
await electron_1.dialog.showMessageBox({
type: 'info',
title: 'Legacy mpv plugin removed',
message: 'Legacy mpv plugin removed. SubMiner-managed playback will use the bundled runtime plugin.',
});
return 'removed';
}
await electron_1.dialog.showMessageBox({
type: 'error',
title: 'Could not remove legacy mpv plugin',
message: 'Some legacy SubMiner mpv plugin files could not be moved to the trash.',
detail: result.failedPaths.map((failure) => `${failure.path}: ${failure.message}`).join('\n'),
});
return 'cancel';
}
function createWindowsRuntimePluginPolicy() {
return {
detectInstalledMpvPlugin: (mpvPath) => (0, first_run_setup_plugin_1.detectInstalledMpvPlugin)({
platform: 'win32',
homeDir: node_os_1.default.homedir(),
appDataDir: electron_1.app.getPath('appData'),
mpvExecutablePath: mpvPath,
}),
notifyInstalledPluginDetected: (detection) => {
if (!detection.installed || !detection.path)
return;
electron_1.dialog.showMessageBoxSync({
type: 'warning',
title: 'SubMiner mpv plugin detected',
message: buildInstalledWindowsMpvPluginMessage(detection.path, detection.version),
});
},
resolveInstalledPluginBeforeLaunch: (detection, mpvPath) => promptForWindowsLegacyMpvPluginRemoval(mpvPath, detection),
};
}
process.argv = (0, main_entry_runtime_1.normalizeStartupArgv)(process.argv, process.env);
(0, main_entry_runtime_1.applyEarlyLinuxCommandLineSwitches)(electron_1.app.commandLine, process.argv);
applySanitizedEnv((0, main_entry_runtime_1.sanitizeStartupEnv)(process.env));
const userDataPath = (0, main_entry_runtime_1.configureEarlyAppPaths)(electron_1.app);
const reportFatalError = (0, fatal_error_1.createFatalErrorReporter)({
showErrorBox: (title, details) => electron_1.dialog.showErrorBox(title, details),
consoleError: (message, error) => console.error(message, error),
});
(0, fatal_error_1.registerFatalErrorHandlers)({
reportFatalError,
exit: (code) => electron_1.app.exit(code),
});
function startMainProcess() {
const gotSingleInstanceLock = (0, early_single_instance_1.requestSingleInstanceLockEarly)(electron_1.app);
if (!gotSingleInstanceLock) {
electron_1.app.exit(0);
return;
}
try {
require('./main.js');
}
catch (error) {
reportFatalError(error, {
title: 'SubMiner startup failed',
context: 'SubMiner failed while loading the main process.',
});
electron_1.app.exit(1);
}
}
async function forwardStartupArgvViaAppControlIfAvailable() {
if (!(0, main_entry_runtime_1.shouldForwardStartupArgvViaAppControl)(process.argv, process.env)) {
return false;
}
const result = await (0, app_control_client_1.sendAppControlCommand)(process.argv, {
configDir: userDataPath,
timeoutMs: 500,
});
if (result.ok) {
electron_1.app.exit(0);
return true;
}
if (!result.unavailable) {
console.error(`SubMiner app-control handoff failed: ${result.error ?? 'unknown error'}`);
electron_1.app.exit(1);
return true;
}
return false;
}
async function runEntryProcess() {
if ((0, main_entry_runtime_1.shouldHandleHelpOnlyAtEntry)(process.argv, process.env)) {
const sanitizedEnv = (0, main_entry_runtime_1.sanitizeHelpEnv)(process.env);
process.env.NODE_NO_WARNINGS = sanitizedEnv.NODE_NO_WARNINGS;
if (!sanitizedEnv.VK_INSTANCE_LAYERS) {
delete process.env.VK_INSTANCE_LAYERS;
}
(0, help_1.printHelp)(DEFAULT_TEXTHOOKER_PORT);
process.exit(0);
return;
}
if ((0, main_entry_runtime_1.shouldHandleLaunchMpvAtEntry)(process.argv, process.env)) {
const sanitizedEnv = (0, main_entry_runtime_1.sanitizeLaunchMpvEnv)(process.env);
applySanitizedEnv(sanitizedEnv);
await electron_1.app.whenReady();
const configuredMpvLaunch = (0, main_entry_launch_config_1.readConfiguredWindowsMpvLaunch)(userDataPath);
const extraArgs = (0, main_entry_runtime_1.normalizeLaunchMpvExtraArgs)(process.argv);
(0, log_files_1.applyLogFileTogglesToEnv)(configuredMpvLaunch.logFiles);
const mpvLogPath = (0, log_files_1.isLogFileEnabled)('mpv')
? process.env.SUBMINER_MPV_LOG?.trim() || (0, log_files_1.resolveDefaultLogFilePath)('mpv')
: '';
if (mpvLogPath) {
(0, log_files_1.pruneLogDirectoryForPath)(mpvLogPath, configuredMpvLaunch.logRotation);
}
const result = await (0, windows_mpv_launch_1.launchWindowsMpv)((0, main_entry_runtime_1.normalizeLaunchMpvTargets)(process.argv), (0, windows_mpv_launch_1.createWindowsMpvLaunchDeps)({
getEnv: (name) => process.env[name],
isAppControlServerAvailable: () => (0, app_control_client_1.isAppControlServerAvailable)({
configDir: userDataPath,
timeoutMs: 350,
}),
sendAppControlCommand: (argv) => (0, app_control_client_1.sendAppControlCommand)(argv, {
configDir: userDataPath,
timeoutMs: 1000,
}),
showError: (title, content) => {
electron_1.dialog.showErrorBox(title, content);
},
logInfo: (message) => appendWindowsMpvLaunchLog(message, configuredMpvLaunch.logRotation),
}), [...extraArgs, ...(0, mpv_logging_args_1.buildMpvLoggingArgs)(configuredMpvLaunch.logLevel, mpvLogPath, extraArgs)], process.execPath, resolveBundledWindowsMpvPluginEntrypoint(), configuredMpvLaunch.executablePath, configuredMpvLaunch.launchMode, createWindowsRuntimePluginPolicy(), configuredMpvLaunch.pluginRuntimeConfig);
electron_1.app.exit(result.ok ? 0 : 1);
return;
}
if ((0, main_entry_runtime_1.shouldHandleStatsDaemonCommandAtEntry)(process.argv, process.env)) {
await electron_1.app.whenReady();
const exitCode = await (0, stats_daemon_entry_1.runStatsDaemonControlFromProcess)(electron_1.app.getPath('userData'));
electron_1.app.exit(exitCode);
return;
}
if (await forwardStartupArgvViaAppControlIfAvailable()) {
return;
}
if ((0, main_entry_runtime_1.shouldDetachBackgroundLaunch)(process.argv, process.env)) {
const childArgs = (0, main_entry_runtime_1.hasTransportedStartupArgs)(process.env) ? [] : process.argv.slice(1);
const child = (0, node_child_process_1.spawn)(process.execPath, childArgs, {
detached: true,
stdio: 'ignore',
env: (0, main_entry_runtime_1.sanitizeBackgroundEnv)(process.env),
});
child.unref();
process.exit(0);
return;
}
startMainProcess();
}
void runEntryProcess().catch((error) => {
console.error('SubMiner app-control handoff failed:', error);
startMainProcess();
});
//# sourceMappingURL=main-entry.js.map
-4095
View File
File diff suppressed because it is too large Load Diff
+10 -11
View File
File diff suppressed because one or more lines are too long
@@ -105,6 +105,8 @@ bash plugins/subminer-workflow/skills/subminer-change-verification/scripts/verif
- For `docs-site/`, `docs/`, and doc-only edits.
- `config`
- For `src/config/` and config-template-sensitive edits.
- `stats`
- For `stats/` dashboard UI edits.
- `core`
- For general source changes where `typecheck` + `test:fast` is the best cheap signal.
- `launcher-plugin`
@@ -108,6 +108,14 @@ for path in "${PATHS[@]}"; do
;;
esac
case "$path" in
stats/*)
add_lane "stats"
add_reason "$path -> stats"
specialized=1
;;
esac
case "$path" in
launcher/*|plugin/subminer/*|plugin/subminer.conf|scripts/test-plugin-*|scripts/get-mpv-window-*|scripts/configure-plugin-binary-path.mjs)
add_lane "launcher-plugin"
@@ -255,8 +255,18 @@ write_summary_files() {
local lane_lines
lane_lines=$(printf '%s\n' "${SELECTED_LANES[@]}")
printf '%s\n' "$lane_lines" >"$ARTIFACT_DIR/lanes.txt"
printf '%s\n' "${BLOCKERS[@]}" >"$ARTIFACT_DIR/blockers.txt"
printf '%s\n' "${PATH_ARGS[@]}" >"$ARTIFACT_DIR/requested-paths.txt"
# bash 3.2 raises "unbound variable" under set -u when expanding an empty
# array, so guard on length (matching the idiom used elsewhere here).
if [[ ${#BLOCKERS[@]} -gt 0 ]]; then
printf '%s\n' "${BLOCKERS[@]}" >"$ARTIFACT_DIR/blockers.txt"
else
: >"$ARTIFACT_DIR/blockers.txt"
fi
if [[ ${#PATH_ARGS[@]} -gt 0 ]]; then
printf '%s\n' "${PATH_ARGS[@]}" >"$ARTIFACT_DIR/requested-paths.txt"
else
: >"$ARTIFACT_DIR/requested-paths.txt"
fi
ARTIFACT_DIR_ENV="$ARTIFACT_DIR" \
SESSION_ID_ENV="$SESSION_ID" \
@@ -482,6 +492,9 @@ for lane in "${SELECTED_LANES[@]}"; do
config)
run_step "$lane" "config" "bun run test:config" || break
;;
stats)
run_step "$lane" "stats" "bun run test:stats" || break
;;
core)
run_step "$lane" "typecheck" "bun run typecheck" || break
run_step "$lane" "fast-tests" "bun run test:fast" || break
+59 -10
View File
@@ -1,24 +1,73 @@
> This is a prerelease build for testing. Stable changelog and docs-site updates remain pending until the final stable release.
<!-- prerelease-base-version: 0.17.1 -->
<!-- prerelease-base-version: 0.18.0 -->
## Highlights
### Added
- **Watch History Browser**
- New `subminer -H` / `--history` command lets you browse your local watch history, replay the last episode, jump to the next one, or pick an episode via fzf or rofi.
- The rofi picker now shows AniList cover art for each show, making it easier to spot the right title at a glance.
- **Card Audio Normalization**
- Audio extracted for Anki cards is now volume-normalized by default, giving more consistent playback loudness across cards.
- Prefer the original source volume? Disable it via the new `ankiConnect.media.normalizeAudio` setting.
- **YouTube Media Cache Mode**: A new `youtube.mediaCache.mode` setting (`direct` or `background`) lets you choose how SubMiner extracts audio and image from YouTube cards.
- In background mode, SubMiner creates a text-only card immediately, downloads a yt-dlp media cache (capped at 720p by default), and fills audio and image fields once the file is ready — with overlay and OSD notifications when the download starts and when media is available.
- If a background download fails, SubMiner now notifies you and clears any pending media updates rather than leaving cards silently incomplete.
### Changed
- **New App Icon**
- SubMiner now ships pixel-art submarine artwork contributed by an anonymous community member.
- Applied across the app icon, tray icon, notifications, README, docs site, and stats page.
- **Launcher Preview Layout**
- fzf previews in the launcher now sit below the menu instead of beside it, giving long titles and metadata more horizontal room.
### Fixed
- **Log Export**: Log filenames now use your local date, so exporting logs near midnight no longer pulls stale files from the previous UTC day. Export redaction has also been expanded to mask a broader range of sensitive data, including IP addresses, email addresses, authentication and cookie headers, yt-dlp cookie arguments, URL credentials, and signed YouTube media URLs.
- **YouTube Card Media Reliability**: Direct stream extraction now uses safer ffmpeg options and skips stale or cached stream map entries to reduce failed media generation. Background cache downloads are hardened with IPv4 and extractor retry flags, stale cache files are cleaned up on startup and before new downloads, and in-flight background downloads are stopped automatically when switching back to direct mode.
- **Character Name Highlighting in Subtitles**
- Fixed unspaced Japanese names (e.g. 東紫乃, 渡辺真奈美) being split at the wrong point, which left surnames like 東 and 渡辺 without their character portrait or hover lookup.
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, misclassified by grammar tagging, or swallowed entirely by a longer generic dictionary match (e.g. ヨータ disappearing inside a false とヨー match).
- Fixed a single unrecognized word in a subtitle line (like a stray interjection) causing character-name highlighting to drop for the whole line instead of just that word.
- No action needed — existing data upgrades automatically the next time a matching name is seen.
- **Known-Word Highlighting**
- Words are no longer marked "known" (green) just because they share spelling with a known Anki card that actually teaches a different reading (e.g. 床 read as とこ no longer falsely matches a known 床/ゆか card).
- Kanji words are also no longer marked known just because a different mined word happens to share their reading (e.g. 渓谷/けいこく no longer falsely matches a known 警告/けいこく card).
- Single-kana grammar tokens (particles like よ, え) no longer borrow an unrelated card's reading and get falsely painted as known.
- Stats sessions now correctly reflect known-word counts again after the reading-aware matching upgrade, instead of showing 0 everywhere.
- **Annotation Highlighting Refinements**
- Restored frequency/JLPT highlighting and vocabulary-stat counting for words like 確かに and やはり, which were wrongly treated as grammar noise.
- Kanji nouns that MeCab tags as "non-independent" (e.g. 日, 点, 以外) also keep their highlighting and stats counting again.
- Suffix-only tokens (e.g. さん, れる) are now excluded from JLPT/frequency highlighting by default to match how particles and interjections are treated; known-word highlighting for them still works, and this is configurable.
- **Unparsed Subtitle Text**
- Subtitle text the dictionary can't recognize (like a truncated verb form) is now still hoverable for lookup and correctly counted toward a sentence's difficulty, instead of showing as dead, non-interactive text.
- **Kiku Manual Field Grouping**
- Fixed the field-grouping dialog getting stuck invisible behind fullscreen video on Hyprland/Wayland, and failing silently on repeated attempts after the first use.
- Fixed a timed-out or failed grouping request leaving an invisible, stuck dialog covering the video; it now closes automatically so the overlay recovers.
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut, and added a proper error message for the previously-silent case where the original card can no longer be loaded.
- **Secondary Subtitles**
- Karaoke-style secondary subtitles (common in opening/ending songs) no longer spam dozens of lines down the screen; repeated lines are now collapsed and the subtitle area is capped to a strip at the top.
- **YouTube Extraction**
- Fixed direct YouTube stream extraction occasionally corrupting the stream URL and causing failed audio/video capture.
- **Background Stats Server**
- Launching SubMiner in the background now correctly auto-starts the stats server when enabled, and won't start a duplicate if one's already running.
- **Stats Trend Charts**
- All trend chart titles now show by default, with the ability to hide specific titles (remembered across sessions) and cap how many top titles a chart displays.
- **Stats Cover Art**
- Cover art now loads as soon as a series starts playing instead of waiting for your first visit to its detail page, so the stats timeline shows artwork right away.
- Existing series missing art are backfilled automatically the next time you open the stats page.
## What's Changed
- feat(youtube): add mediaCache mode and safer stream media extraction by @ksyasuda in #130
- fix(logs): use local date for log filenames and expand export redaction by @ksyasuda in #131
- fix(youtube): parse mpv EDL stream URLs with byte-length guards by @ksyasuda in #134
- Normalize generated Anki audio by default by @ksyasuda in #135
- feat(launcher): add -H/--history command to browse local watch history by @ksyasuda in #136
- fix(overlay): prevent field grouping modal from freezing overlay on Hyprland by @ksyasuda in #138
- fix(overlay): collapse karaoke syllable spam in secondary subtitles by @ksyasuda in #139
- feat(stats): Trends dashboard overhaul — title visibility, ranking modes, calendar-accurate windows, tooltips by @ksyasuda in #140
- feat(branding): replace app icon with contributed pixel-art set by @ksyasuda in #141
- feat(anki): reading-aware known-word matching (cache v3) by @ksyasuda in #142
- fix(stats): start stats server on background app launch by @ksyasuda in #144
- fix(tokenizer): keep unparsed Yomitan tokens hoverable by @ksyasuda in #145
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
- fix(overlay): keep frequency/JLPT highlight for kanji non-independent nouns by @ksyasuda in #150
- fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names by @ksyasuda in #151
## Installation
+66 -12
View File
@@ -1,25 +1,79 @@
## Highlights
### Added
- **YouTube Media Cache Mode:** New `youtube.mediaCache.mode` setting lets you choose between `direct` (the existing behavior) and `background` modes for extracting audio and images when creating cards from YouTube videos.
- Background mode uses a separate yt-dlp download to cache the video file locally, which can rescue media extraction when direct stream URLs are unreliable or expire quickly.
- While the cache downloads, SubMiner creates a text-only card immediately, then fills in audio and image once the file is ready — no need to re-mine the sentence.
- Cache downloads are capped at 720p by default (`youtube.mediaCache.maxHeight`) to keep file sizes reasonable.
- **Sentence Audio Normalization**
- Generated sentence audio is now normalized to -23 LUFS by default, giving mined clips consistent volume across shows.
- Clips captured from playback can also mirror mpv's software volume curve, with a limiter to prevent clipping when boosted.
- Both behaviors are controlled independently and can be turned off in the Anki Connect media settings.
- **Watch History Browser**
- Added `subminer -H` / `--history` to browse local watch history, replay the last episode, continue to the next one, or jump to any past episode.
- Works with fzf or rofi; the rofi picker shows AniList cover art already stored in the stats database.
### Changed
- **Known-Word Highlighting Accuracy**
- Highlighting now compares subtitle and Anki-card readings, so it no longer confuses homographs like 床/とこ vs 床/ゆか or unrelated kanji that happen to share a reading.
- Standalone suffix words such as さん or れる are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are already treated (configurable if you'd rather keep them annotated).
- Cards without readings still fall back to word-only matching; the highlighting cache upgrades automatically with no action needed.
- **Stats Trend Charts**
- Overhauled trend charts with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and sortable multi-column tooltips.
- **New App Icon**
- Replaced the SubMiner icon with new pixel-art submarine artwork contributed by an anonymous community member, now used across the app icon, tray, notifications, README, docs site, and stats page.
- **Launcher Preview Layout**
- fzf previews now sit below the launcher menu instead of beside it, giving long titles and metadata more horizontal room.
### Fixed
- **Log Export:** Log filenames now use your local date, so exporting logs around UTC midnight correctly captures today's file instead of yesterday's.
- Log exports also now redact a broader set of sensitive data — IP addresses, email addresses, auth and cookie headers, yt-dlp cookie arguments, URL credentials, API keys, and signed YouTube media URLs.
- **Character Name Recognition**
- Character dictionaries now correctly split unspaced AniList native names and validate readings, so overlay portraits, highlights, and hover lookups work reliably even without MeCab installed.
- Name matches survive punctuation and unmatched text and no longer get overridden by generic dictionary matches or wrongly split from longer words like 空気.
- Existing installs regenerate automatically and upgrade to exact splits once MeCab is available — no action needed.
- **YouTube Card Media Generation:** Extraction is more reliable and resilient across a wider range of stream conditions.
- Stale or cached stream maps are skipped automatically, and ffmpeg requests use safer options for resolved streams to reduce failures.
- Background cache downloads retry on network issues (IPv4 and extractor fallbacks), notify you on failure instead of silently hanging, and clean up stale cache files on startup and before each new download.
- Switching back to direct mode now cancels any in-progress background downloads cleanly.
- **Frequency & JLPT Highlighting Coverage**
- Content adverbs (確かに, やはり) and kanji nouns MeCab tags as non-independent (日, 点, 以外) are now correctly included in frequency/JLPT highlighting and vocabulary stats.
- Lexicalized kana expressions like かといって keep their frequency annotations, while interjections, pronouns, and pure grammar fragments are still filtered out as noise.
- **Unparsed Text Hover Lookup**
- Subtitle text Yomitan can't fully parse — truncated inflections like とこ戻ろ… or elongation runs like ぅ~ — is hoverable again for dictionary lookup.
- These runs stay excluded from frequency/JLPT highlighting and vocabulary stats, same as bracketed captions and punctuation-only text.
- **Karaoke-Style Secondary Subtitles**
- Secondary subtitles no longer flood the screen with dozens of one-syllable lines during karaoke-style openings and endings.
- Repeated events now collapse into a single line, and the secondary subtitle area stays capped to a strip at the top.
- **Kiku Field Grouping Reliability**
- The manual field-grouping dialog now stays above fullscreen mpv on Hyprland/Wayland and keeps working across repeated attempts.
- Abandoned grouping windows close automatically after a timeout or failure, and each attempt now reports a clear success or error, including when the original card can no longer be loaded.
- **Background Stats Server Startup**
- Background `subminer app` launches now start the stats server automatically when enabled, and skip startup if one is already running.
- **AniList Cover Art Timing**
- Stats now fetches the best-match AniList cover as soon as a new series starts playing, so artwork appears in the timeline immediately instead of only after visiting the series page.
- Existing series missing art get backfilled automatically on the next Stats page visit.
- **YouTube Direct Stream Playback**
- Fixed direct YouTube stream extraction so mpv's EDL stream URLs are parsed correctly, preventing corrupted signed video URLs and the resulting ffmpeg 403 errors.
## What's Changed
- feat(youtube): add mediaCache mode and safer stream media extraction by @ksyasuda in #130
- fix(logs): use local date for log filenames and expand export redaction by @ksyasuda in #131
- fix(youtube): parse mpv EDL stream URLs with byte-length guards by @ksyasuda in #134
- Normalize generated Anki audio by default by @ksyasuda in #135
- feat(launcher): add -H/--history command to browse local watch history by @ksyasuda in #136
- fix(overlay): prevent field grouping modal from freezing overlay on Hyprland by @ksyasuda in #138
- fix(overlay): collapse karaoke syllable spam in secondary subtitles by @ksyasuda in #139
- feat(stats): Trends dashboard overhaul — title visibility, ranking modes, calendar-accurate windows, tooltips by @ksyasuda in #140
- feat(branding): replace app icon with contributed pixel-art set by @ksyasuda in #141
- feat(anki): reading-aware known-word matching (cache v3) by @ksyasuda in #142
- fix(stats): start stats server on background app launch by @ksyasuda in #144
- fix(tokenizer): keep unparsed Yomitan tokens hoverable by @ksyasuda in #145
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
## Installation
+96
View File
@@ -1213,6 +1213,102 @@ test('writeChangelogArtifacts appends contributor attribution and a new-contribu
}
});
test('writeChangelogArtifacts skips contributor attribution in GitHub Actions without a token', async () => {
const { writeChangelogArtifacts } = await loadModule();
const workspace = createWorkspace('release-notes-actions-no-token');
const projectRoot = path.join(workspace, 'SubMiner');
const originalActions = process.env.GITHUB_ACTIONS;
const originalGhToken = process.env.GH_TOKEN;
const originalGithubToken = process.env.GITHUB_TOKEN;
const originalPath = process.env.PATH;
const originalWarn = console.warn;
const warnings: string[] = [];
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(path.join(projectRoot, 'CHANGELOG.md'), '# Changelog\n', 'utf8');
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: added', 'area: release', '', '- Added a feature.'].join('\n'),
'utf8',
);
try {
process.env.GITHUB_ACTIONS = 'true';
delete process.env.GH_TOKEN;
delete process.env.GITHUB_TOKEN;
process.env.PATH = workspace;
console.warn = (message?: unknown) => {
warnings.push(String(message));
};
writeChangelogArtifacts({
cwd: projectRoot,
version: '0.6.0',
date: '2026-05-06',
deps: { runClaude: defaultStubClaude().runClaude },
});
assert.deepEqual(warnings, []);
const releaseNotes = fs.readFileSync(
path.join(projectRoot, 'release', 'release-notes.md'),
'utf8',
);
assert.doesNotMatch(releaseNotes, /## What's Changed/);
} finally {
console.warn = originalWarn;
if (originalActions === undefined) {
delete process.env.GITHUB_ACTIONS;
} else {
process.env.GITHUB_ACTIONS = originalActions;
}
if (originalGhToken === undefined) {
delete process.env.GH_TOKEN;
} else {
process.env.GH_TOKEN = originalGhToken;
}
if (originalGithubToken === undefined) {
delete process.env.GITHUB_TOKEN;
} else {
process.env.GITHUB_TOKEN = originalGithubToken;
}
if (originalPath === undefined) {
delete process.env.PATH;
} else {
process.env.PATH = originalPath;
}
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('shouldSkipDefaultContributionLookup skips GitHub Actions without a gh token', async () => {
const { shouldSkipDefaultContributionLookup } = await loadModule();
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: 'true',
GH_TOKEN: undefined,
GITHUB_TOKEN: undefined,
}),
true,
);
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: 'true',
GH_TOKEN: 'ghs_test',
GITHUB_TOKEN: undefined,
}),
false,
);
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: undefined,
GH_TOKEN: undefined,
GITHUB_TOKEN: undefined,
}),
false,
);
});
test('writeReleaseNotesForVersion preserves committed contributor attribution before installation', async () => {
const { writeReleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('release-notes-preserve-attribution');
+11 -6
View File
@@ -108,15 +108,11 @@ function renderPrereleaseBaseVersionMarker(version: string): string {
}
function extractPrereleaseBaseVersionMarker(notes: string): string | null {
return (
/<!--\s*prerelease-base-version:\s*(\d+\.\d+\.\d+)\s*-->/u.exec(notes)?.[1] ?? null
);
return /<!--\s*prerelease-base-version:\s*(\d+\.\d+\.\d+)\s*-->/u.exec(notes)?.[1] ?? null;
}
function stripPrereleaseMetadata(notes: string): string {
return notes
.replace(/<!--\s*prerelease-base-version:\s*\d+\.\d+\.\d+\s*-->\s*/u, '')
.trim();
return notes.replace(/<!--\s*prerelease-base-version:\s*\d+\.\d+\.\d+\s*-->\s*/u, '').trim();
}
function resolveReusablePrereleaseNotes(notes: string, version: string): string | undefined {
@@ -349,6 +345,12 @@ function resolveFragmentRelativePath(fragmentPath: string, cwd: string): string
return path.relative(cwd, fragmentPath).split(path.sep).join('/');
}
export function shouldSkipDefaultContributionLookup(
env: Partial<Record<'GITHUB_ACTIONS' | 'GH_TOKEN' | 'GITHUB_TOKEN', string>> = process.env,
): boolean {
return env.GITHUB_ACTIONS === 'true' && !env.GH_TOKEN && !env.GITHUB_TOKEN;
}
// Walks git history + the GitHub API to attribute each released fragment to the
// PR (and author) that introduced it. One git call and one gh call per fragment,
// plus one gh call per unique author for the first-contribution check. Best
@@ -358,6 +360,9 @@ function defaultResolveContributions(fragmentPaths: string[], cwd: string): Cont
if (fragmentPaths.length === 0) {
return [];
}
if (shouldSkipDefaultContributionLookup()) {
return [];
}
try {
const slug = execFileSync(
+26
View File
@@ -0,0 +1,26 @@
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import test from 'node:test';
const source = readFileSync('scripts/compare-yomitan-api.ts', 'utf8');
test('creates the configured Yomitan user-data directory before Electron uses it', () => {
const setPathIndex = source.indexOf(
"electronModule.app.setPath('userData', options.yomitanUserDataPath)",
);
assert.notEqual(setPathIndex, -1);
const mkdirIndex = source.lastIndexOf(
'fs.mkdirSync(options.yomitanUserDataPath, { recursive: true })',
setPathIndex,
);
assert.ok(mkdirIndex > -1, 'must create the user-data directory');
assert.ok(mkdirIndex < setPathIndex, 'must create the directory before app.setPath');
});
test('lets output drain by using exitCode in the top-level completion handlers', () => {
const completionHandlers = source.slice(source.lastIndexOf('\nmain()'));
assert.match(completionHandlers, /process\.exitCode = process\.exitCode \?\? 0/);
assert.match(completionHandlers, /process\.exitCode = 1/);
assert.doesNotMatch(completionHandlers, /process\.exit\(/);
});
+589
View File
@@ -0,0 +1,589 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import process from 'node:process';
import { createTokenizerDepsRuntime, tokenizeSubtitle } from '../src/core/services/tokenizer.js';
import { selectYomitanParseTokens } from '../src/core/services/tokenizer/parser-selection-stage.js';
import { enrichTokensWithMecabPos1 } from '../src/core/services/tokenizer/parser-enrichment-stage.js';
import { resolveYomitanExtensionPath as resolveBuiltYomitanExtensionPath } from '../src/core/services/yomitan-extension-paths.js';
import { MecabTokenizer } from '../src/mecab-tokenizer.js';
import type { MergedToken } from '../src/types.js';
import { fetchYomitanApi } from './yomitan-api-request.js';
// Compares SubMiner's full tokenization pipeline against a stock Yomitan
// instance reached through the yomitan-api native-messaging bridge
// (https://github.com/yomidevs/yomitan-api, default http://127.0.0.1:19633).
// The API response is raw parseText output; it is mapped through SubMiner's
// own selectYomitanParseTokens so both sides share identical mapping
// semantics and the diff isolates real segmentation/headword divergence.
const DEFAULT_YOMITAN_USER_DATA_PATH = path.join(os.homedir(), '.config', 'SubMiner');
const DEFAULT_API_URL = 'http://127.0.0.1:19633';
const DEFAULT_SCAN_LENGTH = 40;
const FIXTURE_DIR = path.join('src', 'core', 'services', 'tokenizer', '__fixtures__', 'golden');
interface CliOptions {
sentences: string[];
fromFixtures: boolean;
apiUrl: string;
scanLength: number;
emitJson: boolean;
nameMatch: boolean;
yomitanExtensionPath?: string;
yomitanUserDataPath: string;
mecabCommand?: string;
mecabDictionaryPath?: string;
}
function printUsage(): void {
process.stdout.write(`Usage:
bun run compare-yomitan-api:electron -- [options] [sentence ...]
Compares SubMiner tokenization against a stock Yomitan instance via the
yomitan-api bridge. Without sentences, compares every golden-corpus fixture
text plus any --file lines.
--from-fixtures Include golden-corpus fixture texts (default when
no sentences are given).
--file <path> Read additional sentences, one per line.
--api-url <url> yomitan-api base URL (default: ${DEFAULT_API_URL}).
--scan-length <n> Scan length for the API request (default: ${DEFAULT_SCAN_LENGTH}).
--name-match Keep SubMiner's character-name pre-pass enabled.
Off by default: stock Yomitan has no name regions,
so it would only add noise to the diff.
--json Emit machine-readable JSON instead of a report.
--yomitan-extension <path> Path to built Yomitan extension directory.
--yomitan-user-data <path> Electron userData directory (default: ~/.config/SubMiner).
--mecab-command <path> MeCab binary (default: mecab).
--mecab-dictionary <path> MeCab dictionary directory.
-h, --help Show usage.
Exits 1 when any sentence diverges, 0 when everything matches.
`);
}
function requireValue(args: string[], flag: string): string {
const next = args.shift();
if (!next) {
throw new Error(`Missing value for ${flag}`);
}
return next;
}
function parseCliArgs(argv: string[]): CliOptions {
const args = [...argv];
const sentences: string[] = [];
let fromFixtures = false;
let apiUrl = DEFAULT_API_URL;
let scanLength = DEFAULT_SCAN_LENGTH;
let emitJson = false;
let nameMatch = false;
let yomitanExtensionPath: string | undefined;
let yomitanUserDataPath = DEFAULT_YOMITAN_USER_DATA_PATH;
let mecabCommand: string | undefined;
let mecabDictionaryPath: string | undefined;
while (args.length > 0) {
const arg = args.shift();
if (!arg) break;
if (arg === '--help' || arg === '-h') {
printUsage();
process.exit(0);
}
if (arg === '--from-fixtures') {
fromFixtures = true;
continue;
}
if (arg === '--file') {
const filePath = requireValue(args, arg);
const lines = fs
.readFileSync(filePath, 'utf8')
.split('\n')
.map((line) => line.trim())
.filter((line) => line.length > 0);
sentences.push(...lines);
continue;
}
if (arg === '--api-url') {
apiUrl = requireValue(args, arg).replace(/\/$/, '');
continue;
}
if (arg === '--scan-length') {
const value = Number(requireValue(args, arg));
if (!Number.isInteger(value) || value <= 0) {
throw new Error('Invalid --scan-length value');
}
scanLength = value;
continue;
}
if (arg === '--json') {
emitJson = true;
continue;
}
if (arg === '--name-match') {
nameMatch = true;
continue;
}
if (arg === '--yomitan-extension') {
yomitanExtensionPath = requireValue(args, arg);
continue;
}
if (arg === '--yomitan-user-data') {
yomitanUserDataPath = requireValue(args, arg);
continue;
}
if (arg === '--mecab-command') {
mecabCommand = requireValue(args, arg);
continue;
}
if (arg === '--mecab-dictionary') {
mecabDictionaryPath = requireValue(args, arg);
continue;
}
if (arg.startsWith('-') && arg !== '-') {
throw new Error(`Unknown flag: ${arg}`);
}
sentences.push(arg);
}
if (sentences.length === 0) {
fromFixtures = true;
}
return {
sentences,
fromFixtures,
apiUrl,
scanLength,
emitJson,
nameMatch,
yomitanExtensionPath,
yomitanUserDataPath,
mecabCommand,
mecabDictionaryPath,
};
}
function loadFixtureSentences(): string[] {
if (!fs.existsSync(FIXTURE_DIR)) {
return [];
}
return fs
.readdirSync(FIXTURE_DIR)
.filter((entry) => entry.endsWith('.json'))
.sort()
.map((entry) => {
const fixture = JSON.parse(fs.readFileSync(path.join(FIXTURE_DIR, entry), 'utf8')) as {
input?: { text?: string };
};
return fixture.input?.text ?? '';
})
.filter((text) => text.length > 0);
}
function normalizeTokenizerText(text: string): string {
return text
.replace(/\r\n/g, '\n')
.replace(/\\N/g, '\n')
.replace(/\\n/g, '\n')
.trim()
.replace(/\n/g, ' ')
.replace(/\s+/g, ' ')
.trim();
}
interface FuriganaChunk {
startPos: number;
endPos: number;
reading: string;
headword: string;
}
// Walk the raw parseText content the same way the parse-result mapper does
// (whole segments, char offsets accumulated across lines) and capture, per
// chunk, the effective reading (kana chunks read as themselves) and the first
// dictionary headword term. The mapper's token spans are authoritative for
// grouping; these chunks supply the complete reading/lemma the legacy mapper
// does not synthesize.
function collectFuriganaChunks(parseResults: unknown): FuriganaChunk[] {
const chunks: FuriganaChunk[] = [];
if (!Array.isArray(parseResults)) {
return chunks;
}
const content = (parseResults[0] as { content?: unknown } | undefined)?.content;
if (!Array.isArray(content)) {
return chunks;
}
let charOffset = 0;
for (const line of content) {
if (!Array.isArray(line)) {
continue;
}
for (const segment of line as Array<Record<string, unknown>>) {
const text = typeof segment?.text === 'string' ? segment.text : '';
if (!text) {
continue;
}
const reading =
typeof segment.reading === 'string' && segment.reading.length > 0 ? segment.reading : text;
let headword = '';
const headwords = segment.headwords;
if (Array.isArray(headwords)) {
for (const row of headwords) {
const first = Array.isArray(row) ? (row[0] as { term?: unknown } | undefined) : undefined;
if (first && typeof first.term === 'string' && first.term.trim().length > 0) {
headword = first.term.trim();
break;
}
}
}
chunks.push({ startPos: charOffset, endPos: charOffset + text.length, reading, headword });
charOffset += text.length;
}
}
return chunks;
}
function overlayChunkMetadata(tokens: MergedToken[], chunks: FuriganaChunk[]): MergedToken[] {
return tokens.map((token) => {
const covered = chunks.filter(
(chunk) => chunk.startPos >= token.startPos && chunk.endPos <= token.endPos,
);
if (covered.length === 0) {
return token;
}
const reading = covered.map((chunk) => chunk.reading).join('');
const headword = covered.find((chunk) => chunk.headword)?.headword ?? token.surface;
return { ...token, reading, headword };
});
}
async function fetchApiTokens(
apiUrl: string,
text: string,
scanLength: number,
): Promise<MergedToken[] | null> {
const response = await fetchYomitanApi(`${apiUrl}/tokenize`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, scanLength, parser: 'scanning-parser' }),
});
if (!response.ok) {
throw new Error(`yomitan-api /tokenize responded ${response.status}`);
}
const parseResults = (await response.json()) as unknown;
const tokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
if (!tokens) {
return tokens;
}
return overlayChunkMetadata(tokens, collectFuriganaChunks(parseResults));
}
interface ComparedToken {
surface: string;
reading: string;
headword: string;
startPos: number;
endPos: number;
frequencyRank?: number;
}
function toComparedTokens(tokens: MergedToken[] | null): ComparedToken[] {
return (tokens ?? []).map((token) => ({
surface: token.surface,
reading: token.reading,
headword: token.headword,
startPos: token.startPos,
endPos: token.endPos,
frequencyRank: token.frequencyRank,
}));
}
interface MismatchRegion {
subminer: ComparedToken[];
yomitan: ComparedToken[];
}
interface PairDiff {
surface: string;
field: 'headword' | 'reading';
subminer: string;
yomitan: string;
}
interface SentenceComparison {
text: string;
subminerTokens: ComparedToken[];
yomitanTokens: ComparedToken[];
segmentationMatches: boolean;
mismatchRegions: MismatchRegion[];
pairDiffs: PairDiff[];
error?: string;
}
// Walk both token lists in span order. Tokens whose [startPos, endPos) agree
// are paired; disagreeing runs are grouped until the boundaries realign.
function compareTokenLists(
subminer: ComparedToken[],
yomitan: ComparedToken[],
): Pick<SentenceComparison, 'segmentationMatches' | 'mismatchRegions' | 'pairDiffs'> {
const mismatchRegions: MismatchRegion[] = [];
const pairDiffs: PairDiff[] = [];
let i = 0;
let j = 0;
while (i < subminer.length || j < yomitan.length) {
const a = subminer[i];
const b = yomitan[j];
if (a && b && a.startPos === b.startPos && a.endPos === b.endPos) {
if (a.headword !== b.headword) {
pairDiffs.push({
surface: a.surface,
field: 'headword',
subminer: a.headword,
yomitan: b.headword,
});
}
// An empty reading means "reads as its surface" (kana tokens, unparsed
// runs) — normalize before comparing.
if ((a.reading || a.surface) !== (b.reading || b.surface)) {
pairDiffs.push({
surface: a.surface,
field: 'reading',
subminer: a.reading,
yomitan: b.reading,
});
}
i += 1;
j += 1;
continue;
}
const region: MismatchRegion = { subminer: [], yomitan: [] };
let aEnd = a ? a.startPos : Number.MAX_SAFE_INTEGER;
let bEnd = b ? b.startPos : Number.MAX_SAFE_INTEGER;
do {
if (aEnd <= bEnd && i < subminer.length) {
const token = subminer[i];
if (token) {
region.subminer.push(token);
aEnd = token.endPos;
}
i += 1;
} else if (j < yomitan.length) {
const token = yomitan[j];
if (token) {
region.yomitan.push(token);
bEnd = token.endPos;
}
j += 1;
} else {
break;
}
} while (aEnd !== bEnd && (i < subminer.length || j < yomitan.length));
mismatchRegions.push(region);
}
return {
segmentationMatches: mismatchRegions.length === 0,
mismatchRegions,
pairDiffs,
};
}
function renderTokenRun(tokens: ComparedToken[]): string {
if (tokens.length === 0) {
return '(no tokens)';
}
return tokens.map((token) => token.surface).join('|');
}
function renderReport(comparisons: SentenceComparison[]): void {
let divergent = 0;
for (const comparison of comparisons) {
const clean =
!comparison.error && comparison.segmentationMatches && comparison.pairDiffs.length === 0;
if (!clean) {
divergent += 1;
}
process.stdout.write(`\n${clean ? 'MATCH' : 'DIFF '} ${comparison.text}\n`);
if (comparison.error) {
process.stdout.write(` error: ${comparison.error}\n`);
continue;
}
if (clean) {
process.stdout.write(
` ${comparison.subminerTokens.length} tokens: ${renderTokenRun(comparison.subminerTokens)}\n`,
);
continue;
}
if (!comparison.segmentationMatches) {
process.stdout.write(' segmentation:\n');
for (const region of comparison.mismatchRegions) {
process.stdout.write(` subminer: ${renderTokenRun(region.subminer)}\n`);
process.stdout.write(` yomitan: ${renderTokenRun(region.yomitan)}\n`);
}
}
for (const diff of comparison.pairDiffs) {
process.stdout.write(
` ${diff.surface}: ${diff.field} "${diff.subminer}" (subminer) vs "${diff.yomitan}" (yomitan)\n`,
);
}
}
process.stdout.write(
`\n${comparisons.length - divergent}/${comparisons.length} sentences match stock Yomitan\n`,
);
}
async function main(): Promise<void> {
const options = parseCliArgs(process.argv.slice(2));
const sentences = [...(options.fromFixtures ? loadFixtureSentences() : []), ...options.sentences];
if (sentences.length === 0) {
throw new Error('No sentences to compare (no fixtures found and none provided).');
}
// Fail fast with a clear message when the bridge is not running.
try {
const probe = await fetchYomitanApi(`${options.apiUrl}/serverVersion`, {
method: 'POST',
body: '{}',
});
if (!probe.ok) {
throw new Error(`status ${probe.status}`);
}
} catch (error) {
throw new Error(
`yomitan-api is not reachable at ${options.apiUrl} (${(error as Error).message}). ` +
'Ensure the browser is running and "Enable Yomitan API" is on in Yomitan settings.',
);
}
const electronModule = await import('electron').catch(() => null);
if (!electronModule?.app || !electronModule?.session) {
throw new Error('Electron runtime required; run via `bun run compare-yomitan-api:electron`.');
}
const mecabTokenizer = new MecabTokenizer({
mecabCommand: options.mecabCommand,
dictionaryPath: options.mecabDictionaryPath,
});
if (!(await mecabTokenizer.checkAvailability())) {
throw new Error('MeCab is not available on this system.');
}
if (typeof electronModule.app.setPath === 'function') {
fs.mkdirSync(options.yomitanUserDataPath, { recursive: true });
electronModule.app.setPath('userData', options.yomitanUserDataPath);
}
await electronModule.app.whenReady();
const extensionPath = resolveBuiltYomitanExtensionPath({
explicitPath: options.yomitanExtensionPath,
cwd: process.cwd(),
});
if (!extensionPath) {
throw new Error('No built Yomitan extension found; run `bun run build:yomitan` first.');
}
const extension = await electronModule.session.defaultSession.loadExtension(extensionPath, {
allowFileAccess: true,
});
let parserWindow: Electron.BrowserWindow | null = null;
let parserReadyPromise: Promise<void> | null = null;
let parserInitPromise: Promise<boolean> | null = null;
const runtimeDeps = createTokenizerDepsRuntime({
getYomitanExt: () => extension,
getYomitanParserWindow: () => parserWindow,
setYomitanParserWindow: (window) => {
parserWindow = window;
},
getYomitanParserReadyPromise: () => parserReadyPromise,
setYomitanParserReadyPromise: (promise) => {
parserReadyPromise = promise;
},
getYomitanParserInitPromise: () => parserInitPromise,
setYomitanParserInitPromise: (promise) => {
parserInitPromise = promise;
},
isKnownWord: () => false,
getKnownWordMatchMode: () => 'headword',
getJlptLevel: () => null,
getNameMatchEnabled: () => options.nameMatch,
getMecabTokenizer: () => ({
tokenize: (text: string) => mecabTokenizer.tokenize(text),
}),
});
const deps = {
...runtimeDeps,
enrichTokensWithMecab: async (
tokens: Parameters<typeof enrichTokensWithMecabPos1>[0],
mecabTokens: Parameters<typeof enrichTokensWithMecabPos1>[1],
) => enrichTokensWithMecabPos1(tokens, mecabTokens),
};
const comparisons: SentenceComparison[] = [];
try {
for (const sentence of sentences) {
const text = normalizeTokenizerText(sentence);
try {
const [subtitleData, apiTokens] = await Promise.all([
tokenizeSubtitle(text, deps),
fetchApiTokens(options.apiUrl, text, options.scanLength),
]);
const subminerTokens = toComparedTokens(subtitleData.tokens);
const yomitanTokens = toComparedTokens(apiTokens);
comparisons.push({
text,
subminerTokens,
yomitanTokens,
...compareTokenLists(subminerTokens, yomitanTokens),
});
} catch (error) {
comparisons.push({
text,
subminerTokens: [],
yomitanTokens: [],
segmentationMatches: false,
mismatchRegions: [],
pairDiffs: [],
error: (error as Error).message,
});
}
}
} finally {
if (parserWindow) {
const window = parserWindow as Electron.BrowserWindow;
if (!window.isDestroyed()) {
window.destroy();
}
}
electronModule.app.quit();
}
if (options.emitJson) {
process.stdout.write(`${JSON.stringify(comparisons, null, 2)}\n`);
} else {
renderReport(comparisons);
}
const anyDivergence = comparisons.some(
(comparison) =>
comparison.error !== undefined ||
!comparison.segmentationMatches ||
comparison.pairDiffs.length > 0,
);
process.exitCode = anyDivergence ? 1 : 0;
}
main()
.then(() => {
process.exitCode = process.exitCode ?? 0;
})
.catch((error) => {
console.error(`Error: ${(error as Error).message}`);
process.exitCode = 1;
});
+496
View File
@@ -0,0 +1,496 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import process from 'node:process';
import { createTokenizerDepsRuntime, tokenizeSubtitle } from '../src/core/services/tokenizer.js';
import {
buildFixtureLookups,
classifyInjectedScript,
fixtureConfigGetters,
hashInjectedScript,
projectGoldenTokens,
pruneGoldenMessages,
} from '../src/core/services/tokenizer/golden-corpus-harness.js';
import type {
GoldenFixture,
GoldenFixtureConfig,
GoldenRecordedMessage,
GoldenRecordedScript,
} from '../src/core/services/tokenizer/golden-corpus-harness.js';
import { enrichTokensWithMecabPos1 } from '../src/core/services/tokenizer/parser-enrichment-stage.js';
import { resolveYomitanExtensionPath as resolveBuiltYomitanExtensionPath } from '../src/core/services/yomitan-extension-paths.js';
import { MecabTokenizer } from '../src/mecab-tokenizer.js';
import type { JlptLevel, Token } from '../src/types.js';
const DEFAULT_YOMITAN_USER_DATA_PATH = path.join(os.homedir(), '.config', 'SubMiner');
const DEFAULT_FIXTURE_DIR = path.join(
'src',
'core',
'services',
'tokenizer',
'__fixtures__',
'golden',
);
const JLPT_LEVELS: ReadonlySet<string> = new Set(['N1', 'N2', 'N3', 'N4', 'N5']);
interface CliOptions {
name: string;
text: string;
description?: string;
issueRefs: string[];
config: GoldenFixtureConfig;
outDir: string;
force: boolean;
yomitanExtensionPath?: string;
yomitanUserDataPath: string;
mecabCommand?: string;
mecabDictionaryPath?: string;
}
function printUsage(): void {
process.stdout.write(`Usage:
bun run record-tokenizer-fixture:electron -- --name <slug> [options] <subtitle text>
Records a golden tokenizer fixture: raw Yomitan backend responses, raw MeCab
tokens, and the annotated tokens the full pipeline produced for the text.
Review the "expected" block before committing it becomes the assertion.
--name <slug> Fixture name (file becomes <slug>.json). Required.
--description <text> What behavior this fixture pins down.
--issue <ref> Related issue/PR ref, repeatable (e.g. --issue "#156").
--known-word <w[:reading]> Mark a word known, repeatable.
--jlpt <term=level> JLPT level for a term, repeatable (level N1..N5).
--local-frequency <t=rank> Local frequency-dictionary rank, repeatable.
--match-mode <mode> Known-word match mode: headword|surface.
--min-sentence-words <n> Minimum sentence words for N+1.
--no-known-words Disable known-word annotation.
--no-nplusone Disable N+1 targeting.
--no-jlpt Disable JLPT annotation.
--no-name-match Disable character-name matching.
--no-frequency Disable frequency annotation.
--out-dir <path> Fixture directory (default: ${DEFAULT_FIXTURE_DIR}).
--force Overwrite an existing fixture file.
--yomitan-extension <path> Path to built Yomitan extension directory.
--yomitan-user-data <path> Electron userData directory (default: ~/.config/SubMiner).
--mecab-command <path> MeCab binary (default: mecab).
--mecab-dictionary <path> MeCab dictionary directory.
-h, --help Show usage.
`);
}
function requireValue(args: string[], flag: string): string {
const next = args.shift();
if (!next) {
throw new Error(`Missing value for ${flag}`);
}
return next;
}
function parseCliArgs(argv: string[]): CliOptions {
const args = [...argv];
const inputParts: string[] = [];
const issueRefs: string[] = [];
const knownWords: GoldenFixtureConfig['knownWords'] = [];
const jlptLevels: Record<string, JlptLevel> = {};
const localFrequencyRanks: Record<string, number> = {};
const config: GoldenFixtureConfig = {};
let name: string | undefined;
let description: string | undefined;
let outDir = DEFAULT_FIXTURE_DIR;
let force = false;
let yomitanExtensionPath: string | undefined;
let yomitanUserDataPath = DEFAULT_YOMITAN_USER_DATA_PATH;
let mecabCommand: string | undefined;
let mecabDictionaryPath: string | undefined;
while (args.length > 0) {
const arg = args.shift();
if (!arg) break;
if (arg === '--help' || arg === '-h') {
printUsage();
process.exit(0);
}
if (arg === '--name') {
name = requireValue(args, arg);
continue;
}
if (arg === '--description') {
description = requireValue(args, arg);
continue;
}
if (arg === '--issue') {
issueRefs.push(requireValue(args, arg));
continue;
}
if (arg === '--known-word') {
const value = requireValue(args, arg);
const separator = value.indexOf(':');
knownWords.push(
separator === -1
? value
: { text: value.slice(0, separator), reading: value.slice(separator + 1) },
);
continue;
}
if (arg === '--jlpt') {
const value = requireValue(args, arg);
const separator = value.indexOf('=');
const level = separator === -1 ? '' : value.slice(separator + 1);
if (separator === -1 || !JLPT_LEVELS.has(level)) {
throw new Error(`Invalid --jlpt value "${value}"; expected <term>=N1..N5`);
}
jlptLevels[value.slice(0, separator)] = level as JlptLevel;
continue;
}
if (arg === '--local-frequency') {
const value = requireValue(args, arg);
const separator = value.indexOf('=');
const rank = separator === -1 ? Number.NaN : Number(value.slice(separator + 1));
if (!Number.isInteger(rank) || rank <= 0) {
throw new Error(`Invalid --local-frequency value "${value}"; expected <term>=<rank>`);
}
localFrequencyRanks[value.slice(0, separator)] = rank;
continue;
}
if (arg === '--match-mode') {
const value = requireValue(args, arg);
if (value !== 'headword' && value !== 'surface') {
throw new Error(`Invalid --match-mode "${value}"; expected headword|surface`);
}
config.knownWordMatchMode = value;
config.frequencyMatchMode = value;
continue;
}
if (arg === '--min-sentence-words') {
const value = Number(requireValue(args, arg));
if (!Number.isInteger(value) || value < 0) {
throw new Error('Invalid --min-sentence-words value');
}
config.minSentenceWordsForNPlusOne = value;
continue;
}
if (arg === '--no-known-words') {
config.knownWordsEnabled = false;
continue;
}
if (arg === '--no-nplusone') {
config.nPlusOneEnabled = false;
continue;
}
if (arg === '--no-jlpt') {
config.jlptEnabled = false;
continue;
}
if (arg === '--no-name-match') {
config.nameMatchEnabled = false;
continue;
}
if (arg === '--no-frequency') {
config.frequencyEnabled = false;
continue;
}
if (arg === '--out-dir') {
outDir = requireValue(args, arg);
continue;
}
if (arg === '--force') {
force = true;
continue;
}
if (arg === '--yomitan-extension') {
yomitanExtensionPath = requireValue(args, arg);
continue;
}
if (arg === '--yomitan-user-data') {
yomitanUserDataPath = requireValue(args, arg);
continue;
}
if (arg === '--mecab-command') {
mecabCommand = requireValue(args, arg);
continue;
}
if (arg === '--mecab-dictionary') {
mecabDictionaryPath = requireValue(args, arg);
continue;
}
if (arg.startsWith('-') && arg !== '-') {
throw new Error(`Unknown flag: ${arg}`);
}
inputParts.push(arg);
}
if (!name || !/^[a-z0-9][a-z0-9-]*$/.test(name)) {
throw new Error('A kebab-case --name is required (e.g. --name kureru-lexical-keep)');
}
const text = inputParts.join(' ').trim();
if (!text) {
throw new Error('Provide the subtitle text to record as positional arguments.');
}
if (knownWords.length > 0) config.knownWords = knownWords;
if (Object.keys(jlptLevels).length > 0) config.jlptLevels = jlptLevels;
if (Object.keys(localFrequencyRanks).length > 0) {
config.localFrequencyRanks = localFrequencyRanks;
}
return {
name,
text,
description,
issueRefs,
config,
outDir,
force,
yomitanExtensionPath,
yomitanUserDataPath,
mecabCommand,
mecabDictionaryPath,
};
}
const RECORDER_SHIM_SCRIPT = `
(() => {
const g = globalThis;
if (g.__subminerGoldenRecorderInstalled) { return true; }
g.__subminerGoldenRecorderInstalled = true;
g.__subminerGoldenMessages = [];
const runtime = chrome.runtime;
const original = runtime.sendMessage.bind(runtime);
runtime.sendMessage = (payload, callback) => {
return original(payload, (response) => {
try {
g.__subminerGoldenMessages.push(JSON.parse(JSON.stringify({
action: payload && payload.action ? payload.action : '',
params: payload && typeof payload.params !== 'undefined' ? payload.params : null,
response: typeof response !== 'undefined' ? response : null,
})));
} catch {}
if (callback) { callback(response); }
});
};
return true;
})();
`;
const RECORDER_DRAIN_SCRIPT = `
(() => {
const g = globalThis;
const log = g.__subminerGoldenMessages || [];
g.__subminerGoldenMessages = [];
return JSON.stringify(log);
})();
`;
interface RecordingSink {
messages: GoldenRecordedMessage[];
scripts: GoldenRecordedScript[];
mecab: Record<string, Token[] | null>;
}
function patchParserWindowForRecording(
window: Electron.BrowserWindow | null,
patched: WeakSet<object>,
sink: RecordingSink,
): void {
if (!window || window.isDestroyed()) {
return;
}
const webContents = window.webContents as unknown as {
executeJavaScript: (script: string, userGesture?: boolean) => Promise<unknown>;
};
if (patched.has(webContents)) {
return;
}
patched.add(webContents);
const original = webContents.executeJavaScript.bind(webContents);
let shimReady: Promise<unknown> | null = null;
const drain = async (): Promise<void> => {
try {
const raw = (await original(RECORDER_DRAIN_SCRIPT, true)) as string;
const drained = JSON.parse(raw) as GoldenRecordedMessage[];
sink.messages.push(...drained);
} catch (error) {
process.stderr.write(
`Warning: failed to drain recorded messages: ${(error as Error).message}\n`,
);
}
};
webContents.executeJavaScript = async (script: string, userGesture?: boolean) => {
if (!shimReady) {
shimReady = original(RECORDER_SHIM_SCRIPT, true);
}
try {
await shimReady;
} catch {
shimReady = null;
}
let result: unknown;
try {
result = await original(script, userGesture);
} finally {
await drain();
}
sink.scripts.push({
sha256: hashInjectedScript(script),
marker: classifyInjectedScript(script),
result: result === undefined ? null : result,
});
return result;
};
}
async function main(): Promise<void> {
const options = parseCliArgs(process.argv.slice(2));
const outPath = path.join(options.outDir, `${options.name}.json`);
if (fs.existsSync(outPath) && !options.force) {
throw new Error(`${outPath} already exists; pass --force to overwrite.`);
}
const electronModule = await import('electron').catch(() => null);
if (!electronModule?.app || !electronModule?.session) {
throw new Error(
'Electron runtime is required; run via `bun run record-tokenizer-fixture:electron`.',
);
}
const mecabTokenizer = new MecabTokenizer({
mecabCommand: options.mecabCommand,
dictionaryPath: options.mecabDictionaryPath,
});
if (!(await mecabTokenizer.checkAvailability())) {
throw new Error('MeCab is not available on this system.');
}
if (typeof electronModule.app.setPath === 'function') {
electronModule.app.setPath('userData', options.yomitanUserDataPath);
}
await electronModule.app.whenReady();
const extensionPath = resolveBuiltYomitanExtensionPath({
explicitPath: options.yomitanExtensionPath,
cwd: process.cwd(),
});
if (!extensionPath) {
throw new Error('No built Yomitan extension found; run `bun run build:yomitan` first.');
}
const extension = await electronModule.session.defaultSession.loadExtension(extensionPath, {
allowFileAccess: true,
});
const sink: RecordingSink = { messages: [], scripts: [], mecab: {} };
const patched = new WeakSet<object>();
let parserWindow: Electron.BrowserWindow | null = null;
let parserReadyPromise: Promise<void> | null = null;
let parserInitPromise: Promise<boolean> | null = null;
const lookups = buildFixtureLookups(options.config);
const runtimeDeps = createTokenizerDepsRuntime({
getYomitanExt: () => extension,
getYomitanParserWindow: () => {
patchParserWindowForRecording(parserWindow, patched, sink);
return parserWindow;
},
setYomitanParserWindow: (window) => {
parserWindow = window;
patchParserWindowForRecording(parserWindow, patched, sink);
},
getYomitanParserReadyPromise: () => parserReadyPromise,
setYomitanParserReadyPromise: (promise) => {
parserReadyPromise = promise;
},
getYomitanParserInitPromise: () => parserInitPromise,
setYomitanParserInitPromise: (promise) => {
parserInitPromise = promise;
},
isKnownWord: lookups.isKnownWord,
getJlptLevel: lookups.getJlptLevel,
getFrequencyRank: lookups.getFrequencyRank,
...fixtureConfigGetters(options.config),
getMecabTokenizer: () => ({
tokenize: async (text: string) => {
const rawTokens = await mecabTokenizer.tokenize(text);
sink.mecab[text] = rawTokens;
return rawTokens;
},
}),
});
const deps = {
...runtimeDeps,
// Same pinned enrichment implementation the replay harness uses.
enrichTokensWithMecab: async (
tokens: Parameters<typeof enrichTokensWithMecabPos1>[0],
mecabTokens: Parameters<typeof enrichTokensWithMecabPos1>[1],
) => enrichTokensWithMecabPos1(tokens, mecabTokens),
};
try {
const subtitleData = await tokenizeSubtitle(options.text, deps);
const expectedTokens = projectGoldenTokens(subtitleData.tokens);
const prunedMessages = pruneGoldenMessages(sink.messages);
const fixture: GoldenFixture = {
name: options.name,
...(options.description ? { description: options.description } : {}),
...(options.issueRefs.length > 0 ? { issueRefs: options.issueRefs } : {}),
recordedAt: new Date().toISOString(),
input: { text: options.text },
config: options.config,
recording: {
messages: prunedMessages,
scripts: sink.scripts,
mecab: sink.mecab,
},
expected: { tokens: expectedTokens },
};
fs.mkdirSync(options.outDir, { recursive: true });
fs.writeFileSync(outPath, `${JSON.stringify(fixture, null, 2)}\n`);
process.stdout.write(`Recorded ${outPath}\n`);
process.stdout.write(
` messages=${prunedMessages.length} scripts=${sink.scripts.length} mecabTexts=${Object.keys(sink.mecab).length}\n`,
);
process.stdout.write('\nExpected tokens (review before committing):\n');
if (!expectedTokens || expectedTokens.length === 0) {
process.stdout.write(' (none)\n');
} else {
for (const token of expectedTokens) {
const flags = [
token.isKnown ? 'known' : null,
token.isNPlusOneTarget ? 'n+1' : null,
token.isNameMatch ? 'name' : null,
token.isUnparsedRun ? 'unparsed' : null,
token.jlptLevel ?? null,
token.frequencyRank !== undefined ? `freq=${token.frequencyRank}` : null,
]
.filter(Boolean)
.join(', ');
process.stdout.write(
` ${token.surface} -> ${token.headword} (${token.reading})${flags ? ` [${flags}]` : ''}\n`,
);
}
}
process.stdout.write(
'\nVerify replay with: bun test src/core/services/tokenizer/golden-corpus.test.ts\n',
);
} finally {
if (parserWindow) {
const window = parserWindow as Electron.BrowserWindow;
if (!window.isDestroyed()) {
window.destroy();
}
}
electronModule.app.quit();
}
}
main()
.then(() => process.exit(0))
.catch((error) => {
console.error(`Error: ${(error as Error).message}`);
process.exit(1);
});
+9 -66
View File
@@ -1,12 +1,7 @@
import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { isAbsolute, join, relative, resolve } from 'node:path';
type LaneConfig = {
roots: string[];
include: string[];
exclude: Set<string>;
};
import { collectLaneFiles } from './test-lanes';
type LcovRecord = {
sourceFile: string;
@@ -18,64 +13,6 @@ type LcovRecord = {
const repoRoot = resolve(__dirname, '..');
const lanes: Record<string, LaneConfig> = {
'bun-src-full': {
roots: ['src'],
include: ['.test.ts', '.type-test.ts'],
exclude: new Set([
'src/core/services/anki-jimaku-ipc.test.ts',
'src/core/services/ipc.test.ts',
'src/core/services/overlay-manager.test.ts',
'src/main/config-validation.test.ts',
'src/main/runtime/registry.test.ts',
'src/main/runtime/startup-config.test.ts',
]),
},
'bun-launcher-unit': {
roots: ['launcher'],
include: ['.test.ts'],
exclude: new Set(['launcher/smoke.e2e.test.ts']),
},
};
function collectFiles(
rootDir: string,
includeSuffixes: string[],
excludeSet: Set<string>,
): string[] {
const out: string[] = [];
const visit = (currentDir: string) => {
for (const entry of readdirSync(currentDir, { withFileTypes: true })) {
const fullPath = resolve(currentDir, entry.name);
if (entry.isDirectory()) {
visit(fullPath);
continue;
}
const relPath = relative(repoRoot, fullPath).replaceAll('\\', '/');
if (excludeSet.has(relPath)) continue;
if (includeSuffixes.some((suffix) => relPath.endsWith(suffix))) {
out.push(relPath);
}
}
};
visit(resolve(repoRoot, rootDir));
out.sort();
return out;
}
function getLaneFiles(laneName: string): string[] {
const lane = lanes[laneName];
if (!lane) {
throw new Error(`Unknown coverage lane: ${laneName}`);
}
const files = lane.roots.flatMap((rootDir) => collectFiles(rootDir, lane.include, lane.exclude));
if (files.length === 0) {
throw new Error(`No test files found for coverage lane: ${laneName}`);
}
return files;
}
function parseCoverageDirArg(argv: string[]): string {
for (let index = 0; index < argv.length; index += 1) {
if (argv[index] === '--coverage-dir') {
@@ -277,7 +214,13 @@ function runCoverageLane(): number {
rmSync(shardRoot, { recursive: true, force: true });
mkdirSync(shardRoot, { recursive: true });
const files = getLaneFiles(laneName);
let files: string[];
try {
files = collectLaneFiles(repoRoot, laneName);
} catch (error) {
process.stderr.write(`${error instanceof Error ? error.message : error}\n`);
return 1;
}
const reports: string[] = [];
try {
+133 -53
View File
@@ -1,73 +1,153 @@
import { readdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { relative, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
import { resolve } from 'node:path';
import { spawn, spawnSync } from 'node:child_process';
import { collectLaneFiles } from './test-lanes.ts';
// Runs a test lane with per-file process isolation: one `bun test` process per
// test file so a hanging test or leaked global in one file cannot poison the
// rest of the lane. Use --single-process for the old all-in-one-process mode.
//
// Usage: bun scripts/run-test-lane.mjs <lane> [--jobs N] [--timeout-secs N] [--single-process]
const repoRoot = resolve(fileURLToPath(new URL('..', import.meta.url)));
const lanes = {
'bun-src-full': {
roots: ['src'],
include: ['.test.ts', '.type-test.ts'],
exclude: new Set([
'src/core/services/anki-jimaku-ipc.test.ts',
'src/core/services/ipc.test.ts',
'src/core/services/overlay-manager.test.ts',
'src/main/config-validation.test.ts',
'src/main/runtime/registry.test.ts',
'src/main/runtime/startup-config.test.ts',
]),
},
'bun-launcher-unit': {
roots: ['launcher'],
include: ['.test.ts'],
exclude: new Set(['launcher/smoke.e2e.test.ts']),
},
};
// Cap per-file buffered output so a long or noisy test cannot grow the string
// without bound and exhaust memory.
const MAX_OUTPUT_BYTES = 1024 * 1024;
function collectFiles(rootDir, includeSuffixes, excludeSet) {
const out = [];
const visit = (currentDir) => {
for (const entry of readdirSync(currentDir, { withFileTypes: true })) {
const fullPath = resolve(currentDir, entry.name);
if (entry.isDirectory()) {
visit(fullPath);
continue;
// Track spawned `bun test` children so we can kill them if the runner is
// interrupted, avoiding orphaned in-flight test processes.
const activeChildren = new Set();
function terminateChildren() {
for (const child of activeChildren) {
child.kill('SIGKILL');
}
activeChildren.clear();
}
for (const signal of ['SIGINT', 'SIGTERM']) {
process.on(signal, () => {
terminateChildren();
process.exit(130);
});
}
function parseArgs(argv) {
const options = { lane: undefined, jobs: 1, timeoutSecs: 300, singleProcess: false };
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === '--jobs') {
options.jobs = Math.max(1, Number(argv[(index += 1)]) || 1);
} else if (arg === '--timeout-secs') {
options.timeoutSecs = Math.max(1, Number(argv[(index += 1)]) || 300);
} else if (arg === '--single-process') {
options.singleProcess = true;
} else if (!arg.startsWith('--') && options.lane === undefined) {
options.lane = arg;
} else {
process.stderr.write(`Unknown argument: ${arg}\n`);
process.exit(1);
}
}
return options;
}
function runFile(file, timeoutSecs) {
return new Promise((resolvePromise) => {
const child = spawn('bun', ['test', `./${file}`], { cwd: repoRoot });
activeChildren.add(child);
let output = '';
let truncated = false;
let timedOut = false;
const append = (chunk) => {
if (truncated) return;
output += chunk;
if (output.length > MAX_OUTPUT_BYTES) {
output = `${output.slice(0, MAX_OUTPUT_BYTES)}\n[output truncated at ${MAX_OUTPUT_BYTES} bytes]\n`;
truncated = true;
}
const relPath = relative(repoRoot, fullPath).replaceAll('\\', '/');
if (excludeSet.has(relPath)) continue;
if (includeSuffixes.some((suffix) => relPath.endsWith(suffix))) {
out.push(relPath);
};
child.stdout.on('data', append);
child.stderr.on('data', append);
const timer = setTimeout(() => {
timedOut = true;
child.kill('SIGKILL');
}, timeoutSecs * 1000);
child.on('close', (code) => {
clearTimeout(timer);
activeChildren.delete(child);
resolvePromise({ file, code: timedOut ? 124 : (code ?? 1), output, timedOut });
});
child.on('error', (error) => {
clearTimeout(timer);
activeChildren.delete(child);
resolvePromise({ file, code: 1, output: String(error), timedOut: false });
});
});
}
async function runIsolated(files, options) {
const failures = [];
let nextIndex = 0;
let completed = 0;
async function worker() {
while (nextIndex < files.length) {
const file = files[nextIndex];
nextIndex += 1;
const result = await runFile(file, options.timeoutSecs);
completed += 1;
if (result.code !== 0) {
failures.push(result);
const reason = result.timedOut ? `timed out after ${options.timeoutSecs}s` : 'failed';
process.stderr.write(`\n[${completed}/${files.length}] ${file} ${reason}\n`);
process.stderr.write(result.output);
}
}
};
}
visit(resolve(repoRoot, rootDir));
out.sort();
return out;
await Promise.all(Array.from({ length: Math.min(options.jobs, files.length) }, worker));
if (failures.length > 0) {
process.stderr.write(`\n${failures.length} of ${files.length} test files failed:\n`);
for (const failure of failures) {
process.stderr.write(` ${failure.file}${failure.timedOut ? ' (timeout)' : ''}\n`);
}
return 1;
}
process.stdout.write(`All ${files.length} test files passed.\n`);
return 0;
}
const lane = lanes[process.argv[2]];
function runSingleProcess(files) {
const result = spawnSync('bun', ['test', ...files.map((file) => `./${file}`)], {
cwd: repoRoot,
stdio: 'inherit',
});
if (result.error) {
throw result.error;
}
return result.status ?? 1;
}
if (!lane) {
process.stderr.write(`Unknown test lane: ${process.argv[2] ?? '(missing)'}\n`);
const options = parseArgs(process.argv.slice(2));
if (!options.lane) {
process.stderr.write('Missing test lane name\n');
process.exit(1);
}
const files = lane.roots.flatMap((rootDir) => collectFiles(rootDir, lane.include, lane.exclude));
if (files.length === 0) {
process.stderr.write(`No test files found for lane: ${process.argv[2]}\n`);
let files;
try {
files = collectLaneFiles(repoRoot, options.lane);
} catch (error) {
process.stderr.write(`${error instanceof Error ? error.message : error}\n`);
process.exit(1);
}
const result = spawnSync('bun', ['test', ...files.map((file) => `./${file}`)], {
cwd: repoRoot,
stdio: 'inherit',
});
if (result.error) {
throw result.error;
if (options.singleProcess) {
process.exit(runSingleProcess(files));
}
process.exit(result.status ?? 1);
process.exit(await runIsolated(files, options));
+97
View File
@@ -0,0 +1,97 @@
import { readdirSync } from 'node:fs';
import { relative, resolve } from 'node:path';
export type TestLane = {
roots: string[];
include: string[];
exclude?: string[];
extraFiles?: string[];
};
// Single source of truth for test-lane membership. Consumed by
// scripts/run-test-lane.mjs (plain runs) and scripts/run-coverage-lane.ts
// (per-file coverage shards). Lanes discover files by directory so new test
// files join their lane automatically.
export const testLanes: Record<string, TestLane> = {
'bun-src-full': {
roots: ['src'],
include: ['.test.ts', '.type-test.ts'],
// Node-compat suites; their dist builds run via test:runtime:compat.
exclude: [
'src/core/services/anki-jimaku-ipc.test.ts',
'src/core/services/ipc.test.ts',
'src/core/services/overlay-manager.test.ts',
'src/main/config-validation.test.ts',
'src/main/runtime/registry.test.ts',
'src/main/runtime/startup-config.test.ts',
],
},
config: {
roots: ['src/config'],
include: ['.test.ts'],
extraFiles: ['src/generate-config-example.test.ts', 'src/verify-config-example.test.ts'],
},
launcher: {
roots: ['launcher'],
include: ['.test.ts'],
},
'bun-launcher-unit': {
roots: ['launcher'],
include: ['.test.ts'],
exclude: ['launcher/smoke.e2e.test.ts'],
},
scripts: {
roots: ['scripts'],
include: ['.test.ts'],
},
stats: {
roots: ['stats/src'],
include: ['.test.ts', '.test.tsx'],
},
};
function collectFiles(
repoRoot: string,
rootDir: string,
includeSuffixes: string[],
excludeSet: Set<string>,
): string[] {
const out: string[] = [];
const visit = (currentDir: string): void => {
for (const entry of readdirSync(currentDir, { withFileTypes: true })) {
const fullPath = resolve(currentDir, entry.name);
if (entry.isDirectory()) {
visit(fullPath);
continue;
}
const relPath = relative(repoRoot, fullPath).replaceAll('\\', '/');
if (excludeSet.has(relPath)) continue;
if (includeSuffixes.some((suffix) => relPath.endsWith(suffix))) {
out.push(relPath);
}
}
};
visit(resolve(repoRoot, rootDir));
out.sort();
return out;
}
export function collectLaneFiles(repoRoot: string, laneName: string): string[] {
const lane = testLanes[laneName];
if (!lane) {
throw new Error(`Unknown test lane: ${laneName}`);
}
const excludeSet = new Set(lane.exclude ?? []);
const files = lane.roots.flatMap((rootDir) =>
collectFiles(repoRoot, rootDir, lane.include, excludeSet),
);
for (const extra of lane.extraFiles ?? []) {
if (!files.includes(extra)) files.push(extra);
}
files.sort();
if (files.length === 0) {
throw new Error(`No test files found for lane: ${laneName}`);
}
return files;
}
+69
View File
@@ -0,0 +1,69 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { fetchYomitanApi } from './yomitan-api-request.js';
function createStalledServer(): ReturnType<typeof Bun.serve> {
return Bun.serve({
hostname: '127.0.0.1',
port: 0,
fetch: () => new Promise<Response>(() => {}),
});
}
test('fetchYomitanApi reports a clear timeout when the bridge stalls', async () => {
const server = createStalledServer();
try {
await assert.rejects(
() => fetchYomitanApi(`${server.url}tokenize`, { method: 'POST' }, 20),
/Yomitan API request timed out after 20ms/,
);
} finally {
server.stop(true);
}
});
test('fetchYomitanApi preserves an init cancellation signal', async () => {
const server = createStalledServer();
const controller = new AbortController();
const reason = new Error('cancelled by caller');
try {
const request = fetchYomitanApi(`${server.url}tokenize`, { signal: controller.signal }, 50);
controller.abort(reason);
await assert.rejects(request, (error) => error === reason);
} finally {
server.stop(true);
}
});
test('fetchYomitanApi preserves a Request input cancellation signal', async () => {
const server = createStalledServer();
const controller = new AbortController();
const reason = new Error('cancelled by request');
const input = new Request(`${server.url}tokenize`, { signal: controller.signal });
try {
const request = fetchYomitanApi(input, undefined, 50);
controller.abort(reason);
await assert.rejects(request, (error) => error === reason);
} finally {
server.stop(true);
}
});
test('fetchYomitanApi accepts a null init cancellation signal', async () => {
const server = Bun.serve({
hostname: '127.0.0.1',
port: 0,
fetch: () => new Response('ok'),
});
try {
const response = await fetchYomitanApi(`${server.url}tokenize`, { signal: null }, 50);
assert.equal(response.status, 200);
} finally {
server.stop(true);
}
});
+26
View File
@@ -0,0 +1,26 @@
const DEFAULT_YOMITAN_API_TIMEOUT_MS = 10_000;
export async function fetchYomitanApi(
input: RequestInfo | URL,
init?: RequestInit,
timeoutMs = DEFAULT_YOMITAN_API_TIMEOUT_MS,
): Promise<Response> {
const controller = new AbortController();
const requestSignal = input instanceof Request ? input.signal : undefined;
const signals = [controller.signal, init?.signal, requestSignal].filter(
(signal): signal is AbortSignal => signal != null,
);
const signal = signals.length === 1 ? controller.signal : AbortSignal.any(signals);
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
return await fetch(input, { ...init, signal });
} catch (error) {
if (controller.signal.aborted) {
throw new Error(`Yomitan API request timed out after ${timeoutMs}ms`, { cause: error });
}
throw error;
} finally {
clearTimeout(timeout);
}
}
+164 -8
View File
@@ -573,6 +573,8 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
endTime: number,
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
generateScreenshot: (path: MediaInput) => Promise<Buffer>;
};
@@ -587,6 +589,7 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName?: string;
generateAudio: boolean;
generateImage: boolean;
volumeScale?: number;
}) => void;
};
internals.client = {
@@ -606,9 +609,17 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
},
};
internals.mediaGenerator = {
generateAudio: async (mediaPath, _startTime, _endTime, _audioPadding, audioStreamIndex) => {
generateAudio: async (
mediaPath,
_startTime,
_endTime,
_audioPadding,
audioStreamIndex,
_normalizeAudio,
volumeScale,
) => {
mediaInputs.push(
`audio:${describeMediaInputForTest(mediaPath)}:${audioStreamIndex ?? 'auto'}`,
`audio:${describeMediaInputForTest(mediaPath)}:${audioStreamIndex ?? 'auto'}:${volumeScale}`,
);
return Buffer.from('audio');
},
@@ -628,6 +639,7 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName: 'Picture',
generateAudio: true,
generateImage: true,
volumeScale: 0.25,
});
internals.queuePendingYoutubeMediaUpdate({
sourceUrl: 'https://youtu.be/abc123',
@@ -640,14 +652,15 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName: 'Picture',
generateAudio: true,
generateImage: true,
volumeScale: 0.8,
});
await integration.handleYoutubeMediaCacheReady('https://youtu.be/abc123', '/tmp/media.mkv');
assert.deepEqual(mediaInputs, [
'audio:/tmp/media.mkv:youtube-cache:auto',
'audio:/tmp/media.mkv:youtube-cache:auto:0.25',
'image:/tmp/media.mkv:youtube-cache',
'audio:/tmp/media.mkv:youtube-cache:auto',
'audio:/tmp/media.mkv:youtube-cache:auto:0.8',
'image:/tmp/media.mkv:youtube-cache',
]);
assert.deepEqual(
@@ -768,13 +781,139 @@ test('AnkiIntegration reports partial queued YouTube media updates separately fr
);
});
test('AnkiIntegration does not use mpv stream indexes for ready cached YouTube audio', async () => {
const audioCalls: Array<{ path: string; audioStreamIndex?: number }> = [];
test('AnkiIntegration queues YouTube media updates against recovered source URLs', async () => {
const updatedNotes: Array<{ noteId: number; fields: Record<string, string> }> = [];
const storedMedia: string[] = [];
const audioVolumeScales: Array<number | undefined> = [];
let mpvVolume = 30;
const integration = new AnkiIntegration(
{
fields: {
image: 'Picture',
},
media: {
imageFormat: 'jpg',
},
},
{} as never,
{
currentVideoPath: 'https://rr1---sn.example.googlevideo.com/videoplayback?expire=1777777777',
currentSubStart: 10,
currentSubEnd: 12,
currentTimePos: 11,
requestProperty: async (name: string) => {
assert.equal(name, 'volume');
return mpvVolume;
},
} as never,
() => undefined,
undefined,
undefined,
undefined,
{},
undefined,
undefined,
async () => null,
() => true,
() => 'https://www.youtube.com/watch?v=abc123',
);
const internals = integration as unknown as {
client: {
notesInfo: (noteIds: number[]) => Promise<unknown[]>;
updateNoteFields: (noteId: number, fields: Record<string, string>) => Promise<void>;
storeMediaFile: (filename: string) => Promise<void>;
};
mediaGenerator: {
generateAudio: (
path: MediaInput,
startTime: number,
endTime: number,
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
generateScreenshot: () => Promise<Buffer>;
};
queuePendingYoutubeMediaUpdateForNote: (job: {
noteId: number;
noteInfo: { noteId: number; fields: Record<string, { value: string }> };
label: string | number;
}) => Promise<boolean>;
showNotification: () => Promise<void>;
};
internals.client = {
notesInfo: async (noteIds) =>
noteIds.map((noteId) => ({
noteId,
fields: {
SentenceAudio: { value: '' },
Picture: { value: '' },
},
})),
updateNoteFields: async (noteId, fields) => {
updatedNotes.push({ noteId, fields });
},
storeMediaFile: async (filename) => {
storedMedia.push(filename);
},
};
internals.mediaGenerator = {
generateAudio: async (
_path,
_startTime,
_endTime,
_audioPadding,
_audioStreamIndex,
_normalizeAudio,
volumeScale,
) => {
audioVolumeScales.push(volumeScale);
return Buffer.from('audio');
},
generateScreenshot: async () => Buffer.from('image'),
};
internals.showNotification = async () => undefined;
const queued = await internals.queuePendingYoutubeMediaUpdateForNote({
noteId: 404,
noteInfo: {
noteId: 404,
fields: {
SentenceAudio: { value: '' },
Picture: { value: '' },
},
},
label: 'resolved source',
});
mpvVolume = 90;
await integration.handleYoutubeMediaCacheReady('https://youtu.be/abc123', '/tmp/media.mkv');
assert.equal(queued, true);
assert.equal(updatedNotes.length, 1);
assert.equal(updatedNotes[0]?.noteId, 404);
assert.match(updatedNotes[0]?.fields.SentenceAudio ?? '', /^\[sound:audio_/);
assert.match(updatedNotes[0]?.fields.Picture ?? '', /^<img src="image_/);
assert.equal(storedMedia.length, 2);
assert.deepEqual(audioVolumeScales, [0.3 ** 3]);
});
test('AnkiIntegration passes audio normalization config for ready cached YouTube audio', async () => {
const audioCalls: Array<{
path: string;
audioStreamIndex?: number;
normalizeAudio?: boolean;
volumeScale?: number;
}> = [];
const requestedProperties: string[] = [];
const integration = new AnkiIntegration(
{
media: {
audioPadding: 0,
normalizeAudio: false,
},
},
{} as never,
@@ -784,6 +923,10 @@ test('AnkiIntegration does not use mpv stream indexes for ready cached YouTube a
currentSubStart: 10,
currentSubEnd: 12,
currentTimePos: 11,
requestProperty: async (name: string) => {
requestedProperties.push(name);
return 55;
},
} as never,
() => undefined,
undefined,
@@ -804,23 +947,36 @@ test('AnkiIntegration does not use mpv stream indexes for ready cached YouTube a
endTime: number,
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
};
generateAudio: () => Promise<Buffer | null>;
};
internals.mediaGenerator = {
generateAudio: async (path, _startTime, _endTime, _audioPadding, audioStreamIndex) => {
audioCalls.push({ path: path.path, audioStreamIndex });
generateAudio: async (
path,
_startTime,
_endTime,
_audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) => {
audioCalls.push({ path: path.path, audioStreamIndex, normalizeAudio, volumeScale });
return Buffer.from('audio');
},
};
await internals.generateAudio();
assert.deepEqual(requestedProperties, ['volume']);
assert.deepEqual(audioCalls, [
{
path: '/tmp/subminer-youtube-media-cache/media.mkv',
audioStreamIndex: undefined,
normalizeAudio: false,
volumeScale: 0.55 ** 3,
},
]);
});
+51 -5
View File
@@ -73,6 +73,7 @@ import type {
PendingYoutubeMediaQueueFailedOptions,
PendingYoutubeMediaQueueReadyOptions,
} from './anki-integration/pending-youtube-media-queue';
import { resolveMpvVolumeScale } from './anki-integration/mpv-volume';
const log = createLogger('anki').child('integration');
@@ -239,6 +240,9 @@ export class AnkiIntegration {
private getCachedMediaPath: MediaGenerationInputResolverOptions['getCachedMediaPath'] | null =
null;
private shouldRequireRemoteMediaCache: (() => boolean) | null = null;
private getYoutubeMediaSourceUrl:
| (() => Promise<string | null | undefined> | string | null | undefined)
| null = null;
private pendingYoutubeMediaQueue: PendingYoutubeMediaQueue;
constructor(
@@ -257,6 +261,7 @@ export class AnkiIntegration {
overlayNotificationCallback?: (payload: OverlayNotificationPayload) => void,
getCachedMediaPath?: MediaGenerationInputResolverOptions['getCachedMediaPath'],
shouldRequireRemoteMediaCache?: () => boolean,
getYoutubeMediaSourceUrl?: () => Promise<string | null | undefined> | string | null | undefined,
) {
this.config = normalizeAnkiIntegrationConfig(config);
this.aiConfig = { ...aiConfig };
@@ -271,6 +276,7 @@ export class AnkiIntegration {
this.recordCardsMinedCallback = recordCardsMined ?? null;
this.getCachedMediaPath = getCachedMediaPath ?? null;
this.shouldRequireRemoteMediaCache = shouldRequireRemoteMediaCache ?? null;
this.getYoutubeMediaSourceUrl = getYoutubeMediaSourceUrl ?? null;
this.pendingYoutubeMediaQueue = this.createPendingYoutubeMediaQueue();
this.knownWordCache = this.createKnownWordCache(knownWordCacheStatePath);
this.pollingRunner = this.createPollingRunner();
@@ -336,13 +342,23 @@ export class AnkiIntegration {
storeMediaFile: (filename, data) => this.client.storeMediaFile(filename, data),
},
mediaGenerator: {
generateAudio: (videoPath, startTime, endTime, audioPadding, audioStreamIndex) =>
generateAudio: (
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) =>
this.mediaGenerator.generateAudio(
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
),
generateScreenshot: (videoPath, timestamp, options) =>
this.mediaGenerator.generateScreenshot(videoPath, timestamp, options),
@@ -356,7 +372,7 @@ export class AnkiIntegration {
),
},
getConfig: () => this.config,
getCurrentVideoPath: () => this.mpvClient.currentVideoPath,
getCurrentVideoPath: () => this.getCurrentYoutubeMediaSourceUrl(),
getCachedMediaPath: this.getCachedMediaPath,
shouldRequireRemoteMediaCache: () => this.shouldRequireRemoteMediaCache?.() === true,
getSubtitleMediaRange: (context) => this.getSubtitleMediaRange(context),
@@ -367,6 +383,7 @@ export class AnkiIntegration {
mergeFieldValue: (existing, newValue, overwrite) =>
this.mergeFieldValue(existing, newValue, overwrite),
getAnimatedImageLeadInSeconds: (noteInfo) => this.getAnimatedImageLeadInSeconds(noteInfo),
getMpvVolumeScale: () => this.getMpvVolumeScale(),
generateAudioFilename: () => this.generateAudioFilename(),
generateImageFilename: () => this.generateImageFilename(),
formatMiscInfoPatternForMediaPath: (
@@ -474,6 +491,7 @@ export class AnkiIntegration {
getMpvClient: () => this.mpvClient,
...(this.getCachedMediaPath ? { getCachedMediaPath: this.getCachedMediaPath } : {}),
shouldRequireRemoteMediaCache: () => this.shouldRequireRemoteMediaCache?.() === true,
getYoutubeMediaSourceUrl: () => this.getCurrentYoutubeMediaSourceUrl(),
queuePendingYoutubeMediaUpdate: (job) => this.queuePendingYoutubeMediaUpdate(job),
getDeck: () => this.config.deck,
client: {
@@ -489,13 +507,23 @@ export class AnkiIntegration {
retrieveMediaFile: (filename) => this.client.retrieveMediaFile(filename),
},
mediaGenerator: {
generateAudio: (videoPath, startTime, endTime, audioPadding, audioStreamIndex) =>
generateAudio: (
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) =>
this.mediaGenerator.generateAudio(
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
),
generateScreenshot: (videoPath, timestamp, options) =>
this.mediaGenerator.generateScreenshot(videoPath, timestamp, options),
@@ -695,8 +723,12 @@ export class AnkiIntegration {
});
}
isKnownWord(text: string): boolean {
return this.knownWordCache.isKnownWord(text);
isKnownWord(
text: string,
reading?: string,
options?: { allowReadingOnlyMatch?: boolean },
): boolean {
return this.knownWordCache.isKnownWord(text, reading, options);
}
getKnownWordMatchMode(): NPlusOneMatchMode {
@@ -945,6 +977,18 @@ export class AnkiIntegration {
return this.pendingYoutubeMediaQueue.queueFromNote(job);
}
private async getCurrentYoutubeMediaSourceUrl(): Promise<string> {
return (
trimToNonEmptyString(await this.getYoutubeMediaSourceUrl?.()) ??
trimToNonEmptyString(this.mpvClient.currentVideoPath) ??
''
);
}
private async getMpvVolumeScale(): Promise<number | undefined> {
return resolveMpvVolumeScale(this.mpvClient, this.config.media?.mirrorMpvVolume !== false);
}
async handleYoutubeMediaCacheReady(
sourceUrl: string,
cachedPath: string,
@@ -982,6 +1026,8 @@ export class AnkiIntegration {
endTime,
this.config.media?.audioPadding,
resolveAudioStreamIndexForMediaGeneration(videoPath, this.mpvClient.currentAudioStreamIndex),
this.config.media?.normalizeAudio !== false,
await this.getMpvVolumeScale(),
);
}
@@ -2,6 +2,7 @@ import assert from 'node:assert/strict';
import test from 'node:test';
import { CardCreationService } from './card-creation';
import { toMpvEdlValue } from './mpv-edl-test-utils';
import type { MediaInput } from '../media-generator';
import type { AnkiConnectConfig } from '../types/anki';
@@ -269,9 +270,11 @@ test('manual clipboard subtitle update uses resolved mpv stream URLs for remote
const imagePaths: string[] = [];
const recordMediaPath = (mediaInput: MediaInput): string =>
typeof mediaInput === 'string' ? mediaInput : mediaInput.path;
const audioUrl = 'https://audio.example/videoplayback?mime=audio%2Fwebm';
const videoUrl = 'https://video.example/videoplayback?mime=video%2Fmp4';
const edlSource = [
'edl://!new_stream;!no_clip;!no_chapters;%70%https://audio.example/videoplayback?mime=audio%2Fwebm',
'!new_stream;!no_clip;!no_chapters;%69%https://video.example/videoplayback?mime=video%2Fmp4',
`edl://!new_stream;!no_clip;!no_chapters;${toMpvEdlValue(audioUrl)}`,
`!new_stream;!no_clip;!no_chapters;${toMpvEdlValue(videoUrl)}`,
'!global_tags,title=test',
].join(';');
@@ -354,8 +357,8 @@ test('manual clipboard subtitle update uses resolved mpv stream URLs for remote
await service.updateLastAddedFromClipboard('一行目\n\n二行目');
assert.deepEqual(audioPaths, ['https://audio.example/videoplayback?mime=audio%2Fwebm']);
assert.deepEqual(imagePaths, ['https://video.example/videoplayback?mime=video%2Fmp4']);
assert.deepEqual(audioPaths, [audioUrl]);
assert.deepEqual(imagePaths, [videoUrl]);
assert.equal(storedMedia.length, 2);
assert.equal(updatedFields.length, 1);
assert.equal(updatedFields[0]?.Sentence, '一行目 二行目');

Some files were not shown because too many files have changed in this diff Show More