Compare commits

..

10 Commits

Author SHA1 Message Date
sudacode beeab564b4 chore(release): prepare v0.18.0 2026-07-10 02:03:10 -07:00
sudacode cab7975a2b chore(docs): remove implemented plan docs
- Drop early-managed-overlay-startup and macos-notification-hover design docs (both implemented, no longer needed)
2026-07-10 01:09:52 -07:00
sudacode 846292809c docs(changes): consolidate and tighten changelog entries
- Merge related fix entries into single files (name-match-span-merge into character-name-split-and-scan-recovery, kanji-non-independent-noun-annotations + lexicalized-expression-frequency into content-adverb-annotation-stop-list, greedy-name-parsing-priority into character-name-split-and-scan-recovery, kiku-field-grouping-cancel-double-toast into kiku-field-grouping-modal-recovery, history-rofi-cover-art into launcher-history-command)
- Rewrite remaining entries for brevity and reclassify stats-trend-title-limits as "changed"
2026-07-10 01:04:01 -07:00
sudacode 8712780d08 fix(tokenizer): keep frequency rank for lexicalized kana expressions (#156) 2026-07-10 00:54:17 -07:00
sudacode 7b1a005a65 feat(anki): mirror mpv software volume into generated sentence audio (#155) 2026-07-10 00:44:23 -07:00
sudacode 84c75f50aa fix(release): skip gh attribution without CI token 2026-07-09 02:26:00 -07:00
sudacode 8b838f2c7d chore(release): prepare 0.18.0-beta.3 2026-07-09 01:20:02 -07:00
sudacode db4139ba0b fix(tokenizer): block kanji reading collisions and name/generic length t
- Add allowReadingOnlyMatch:false to kanji token known-word lookups so 渓谷/けいこく no longer matches a mined 警告/けいこく card via reading-only index
- Greedy name pre-pass yields when a strictly longer generic word starts at the same position (空 no longer splits 空気; ties still go to the name)
2026-07-09 01:03:58 -07:00
sudacode 6c251502b3 fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153) 2026-07-09 00:28:38 -07:00
sudacode ae40934d3a fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names (#151) 2026-07-08 23:53:45 -07:00
104 changed files with 1513 additions and 3912 deletions
+31
View File
@@ -1,5 +1,36 @@
# Changelog
## v0.18.0 (2026-07-10)
### Added
- Sentence Audio Normalization: Generated sentence audio is now normalized to -23 LUFS by default, and clips mined from playback mirror mpv's software volume curve with a limiter to prevent clipping. Both behaviors are configurable independently.
- Watch History Command: Added `subminer -H` / `--history` to browse watch history, replay or continue episodes, or pick one via fzf or rofi, with cover art shown in the rofi picker.
### Changed
- Fzf Preview Layout: Moved fzf previews below launcher menus, giving long titles and metadata more room.
- Known-Word Highlighting: Now compares subtitle and Anki-card readings, preventing false matches between homographs and unrelated words that share a reading, while still supporting matching across kana and kanji spellings.
- Annotation Filtering: Standalone suffix tokens (e.g. さん, れる) are now excluded from JLPT/frequency/N+1 highlighting by default, matching how particles and interjections are treated; configurable via the pos2 exclusion setting.
- App Icon: Replaced the app icon with new pixel-art submarine artwork contributed by the community, used across the app icon, tray, notifications, README, docs site, and Stats page.
- Stats Trend Charts: Overhauled with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and value-sorted tooltips.
### Fixed
- Background Stats Server: `subminer app` background launches now auto-start the stats server when enabled, and skip startup if one is already running.
- Character Name Highlighting: Character dictionaries now split unspaced native names more reliably, and portraits, highlights, and hover lookup survive punctuation, unmatched text, and competing dictionary matches without incorrectly splitting longer words.
- Highlighting Coverage: Frequency/JLPT highlighting and vocabulary stats now include content adverbs (e.g. 確かに, やはり) and kanji nouns MeCab tags as non-independent (e.g. 日, 点, 以外), while still suppressing interjections, pronouns, and grammar fragments; lexicalized kana expressions like かといって keep their annotations.
- Cover Art Fetching: Stats now fetches AniList cover art as soon as a new series starts playing, and backfills missing art for existing series on the next Stats visit.
- Kiku Field Grouping: The manual field-grouping dialog now stays above fullscreen mpv, remains usable across repeated attempts, closes abandoned windows after timeouts, and reports a clear error when the original card can no longer be loaded.
- Karaoke Subtitle Collapse: Secondary subtitles no longer stack dozens of one-syllable lines during openings and endings; repeated events collapse into one line, capped to a strip at the top.
- Unparsed Token Hover: Subtitle text Yomitan can't parse (truncated inflections, elongation runs) remains hoverable, while staying excluded from highlighting, N+1 candidate math, and vocabulary stats.
- YouTube Streaming: Fixed direct YouTube stream extraction that could corrupt signed URLs and cause ffmpeg 403 errors.
<details>
<summary>Internal changes</summary>
### Internal
- Test lanes moved to `scripts/test-lanes.ts` with per-directory discovery and isolated per-file timeouts; CI now covers previously orphaned stats, scripts, plugin process-retry, and runtime-compat suites, plus a new stats lane in the change-verification workflow.
</details>
## v0.17.2 (2026-06-28)
### Fixed
-4
View File
@@ -1,4 +0,0 @@
type: added
area: mining
- Normalized generated card audio by default during media extraction, with `ankiConnect.media.normalizeAudio` available to keep raw source loudness when needed.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: stats
- `subminer app` background launches now start the stats server automatically when `stats.autoStartServer` is enabled, and skip startup when a background stats server is already running.
@@ -1,6 +0,0 @@
type: fixed
area: overlay
- Fixed character-dictionary surname/given-name splitting for unspaced AniList native names where romanized reading length misled the boundary guess (東紫乃 split as 東紫+乃, 渡辺真奈美 as 渡辺真+奈美), so surnames like 東 and 渡辺 were never highlighted with character portraits and unmatched leading kanji (渡) lost hover lookup entirely. Name parts are now resolved with MeCab person-name POS tags and reading validation when MeCab is available; without MeCab, terms are generated for both plausible boundaries so the real surname still matches. Existing snapshots regenerate automatically (format version bump), and snapshots built without MeCab are upgraded to exact MeCab splits once MeCab is detected.
- Fixed subtitle tokenization dropping a character entirely when Yomitan's greedy scan consumed trailing punctuation/whitespace past the term (e.g. 平 in 「(東)平 (平)え?」 matched across the paren with no exact-source headword). The scanner now retries shorter windows so the prefix term still tokenizes with its name match and portrait.
- Fixed character-name highlights being stripped when MeCab's context-dependent POS tag put the token in the annotation noise filter (e.g. 平 tagged as prefix 接頭詞 in 「あっ 平 これ…」 lost its highlight and portrait even though hover lookup showed the character). Confirmed name matches now survive the POS exclusion filter, and name matching alone keeps the annotation stage active when all other annotations are disabled.
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Removed 確かに and やはり from the annotation stop-word list so they get frequency/JLPT highlighting again and count toward vocabulary stats; the list now only covers interjections, pronouns, and grammar fragments.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: stats
- Cover art is now fetched eagerly when a new series starts playing, instead of waiting for the first visit to its series detail page, so the stats timeline shows the best-guess AniList image right away. The stats covers endpoint also backfills missing series art in the background, so existing series without an image pick one up on the next stats page load.
-4
View File
@@ -1,4 +0,0 @@
type: changed
area: launcher
- Moved fzf previews below launcher menus so long titles and metadata have more horizontal room.
-4
View File
@@ -1,4 +0,0 @@
type: added
area: launcher
- Show cover art icons in the rofi watch-history picker, reusing AniList covers already stored in the stats database (extracted to `~/.cache/subminer/covers`).
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Kanji-bearing nouns that MeCab tags as non-independent (非自立) — e.g. 日 in いい日だったな, 点, 以外 — now keep frequency/JLPT highlighting and count toward vocabulary stats. Yomitan segments them as standalone vocabulary tokens, so the MeCab POS filter only suppresses kana grammar nouns (こと, もの, とき) it was meant for.
@@ -1,4 +0,0 @@
type: fixed
area: anki
- Fixed cancelling the Kiku field grouping dialog showing two "Field grouping cancelled" notifications when grouping was started via the trigger shortcut: the manual workflow already notifies about its outcome (cancelled, UI unavailable, failed), and the trigger path re-notified on top of it. The workflow now owns all outcome notifications, and a previously silent failure (the original card no longer loadable) gets its own message.
@@ -1,6 +0,0 @@
type: fixed
area: overlay
- Fixed Kiku manual field grouping freezing the overlay after adding a duplicate card: the field grouping modal now reliably appears above fullscreen mpv on Hyprland/Wayland by re-asserting window placement until the compositor maps the modal window, instead of a single post-show attempt that raced the async map and left the dialog invisible.
- Fixed manual field grouping staying broken after the first attempt: the request resolver is now always cleared once a choice is made or the request is abandoned, so later grouping attempts no longer short-circuit to an instant "Field grouping cancelled".
- Fixed a timed-out or failed field grouping request leaving an orphaned, invisible modal window covering mpv: abandoned requests now tear down the modal window and close the dialog so the overlay recovers immediately.
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Fixed words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" matching a known 床/ゆか "floor" card). The known-word cache now stores each card's word together with its reading and only matches when the token's reading agrees; cards without a reading field keep matching in any reading as before.
-4
View File
@@ -1,4 +0,0 @@
type: added
area: launcher
- Added `subminer -H` / `--history` to browse local watch history, replay the last watched episode, continue to the next episode, or browse episodes with fzf/rofi.
-4
View File
@@ -1,4 +0,0 @@
type: added
area: launcher
- Added `subminer sync <host>` to merge immersion stats and watch history between two machines over SSH. Each side snapshots its database (`VACUUM INTO`), snapshots are exchanged with `scp`, and each machine merges the other's data as an insert-only union keyed on session UUIDs / video keys / series title keys, so re-syncing is idempotent and nothing is double-counted. Lifetime totals and daily/monthly rollups are updated incrementally (history older than the session retention window is preserved); remote-only historical rollups are copied only when they do not conflict with retained local session history. `subminer sync --snapshot <file>` and `subminer sync --merge <file>` expose the underlying steps for manual transfers. The command refuses to run while the stats daemon or a live mpv session is active (`--force` overrides), ignores stale mpv socket files, keeps the guard in place through local/remote merges, supplies standard SubMiner and Bun paths to non-interactive SSH commands, verifies the remote launcher starts, reports remote stderr on failures, and aborts on stats schema version mismatches.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Fixed character-name annotations dropping for an entire subtitle line when it contained any chunk the dictionary scanner could not match (e.g. an interjection like やほっ before a name): scanner metadata is now merged per token into the parseText segmentation instead of being discarded on any span mismatch.
-5
View File
@@ -1,5 +0,0 @@
type: fixed
area: overlay
- Fixed single-kana grammar tokens counting as known words by borrowing the reading of an unrelated Anki note (よ in 全然いいよ matched a card read よ such as 夜, standalone え matched 絵), which painted them with the known-word highlight. Reading-only known-word matching now requires at least two kana; single-kana cards still match by their word field, and tokens genuinely present in the known-words cache (e.g. です) keep their highlight.
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.
-4
View File
@@ -1,4 +0,0 @@
type: changed
area: branding
- Replaced the SubMiner app icon with new pixel-art submarine artwork contributed by an anonymous community member, used across the app icon, tray, notifications, README, docs site, and stats page.
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Fixed secondary subtitles stacking dozens of one-syllable lines down the screen during karaoke-typeset openings/endings, which made the hover-pause band cover the whole video: karaoke-like event spam is now collapsed into a single deduped line, and the secondary subtitle area is height-capped so it always stays a strip at the top.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: stats
- Session known-word counts no longer show 0 everywhere. The stats server's known-word cache parser only understood the v1/v2 cache formats, so after the reading-aware v3 cache upgrade it silently treated the cache as missing; it now flattens v3 note entries into the headword set.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: stats
- Show all trend chart titles by default, persist hidden-title choices, and add a per-chart top-title limit selector.
-7
View File
@@ -1,7 +0,0 @@
type: internal
area: testing
- Test lanes are now defined once in `scripts/test-lanes.ts` and discovered by directory instead of hand-maintained file lists in `package.json`; the unused `test:core:*`/`test:config:dist`/`test:full` scripts were removed.
- `scripts/run-test-lane.mjs` runs each test file in an isolated `bun test` process with a wall timeout, so a hanging test or leaked global can no longer cascade failures across the lane.
- Previously orphaned suites now run in CI: the stats dashboard tests (`bun run test:stats`), the `scripts/**` tests (`bun run test:scripts`, including the change-verification skill tests), the `test-plugin-process-start-retries.lua` plugin test, and the runtime-compat dist slice (now part of `bun run test:fast`).
- The change-verification skill gained a `stats` lane for `stats/` edits.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: overlay
- Fixed subtitle text that Yomitan's parser cannot match (e.g. the truncated volitional in とこ戻ろ…) being rendered as plain, non-interactive text: it was invisible to hover/lookup and excluded from the n+1 word count, which could wrongly mark a sentence as n+1. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation; bracketed SFX/speaker captions and punctuation-only runs are still skipped.
-4
View File
@@ -1,4 +0,0 @@
type: fixed
area: youtube
- Fixed direct YouTube stream media extraction by parsing mpv EDL stream URLs with their byte-length guards, preventing trailing EDL segment options from corrupting signed googlevideo URLs and causing ffmpeg 403 errors.
+3 -2
View File
@@ -515,7 +515,7 @@
// ==========================================
// AnkiConnect Integration
// Automatic Anki updates and media generation options.
// Hot-reload: ankiConnect.ai.enabled, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Shared AI provider transport settings are read from top-level ai and typically require restart.
// Most other AnkiConnect settings still require restart.
// ==========================================
@@ -559,7 +559,8 @@
"animatedMaxHeight": 0, // Maximum height for animated AVIF captures, in pixels. Set to 0 to preserve aspect ratio.
"animatedCrf": 35, // Animated AVIF CRF quality target. Lower values produce larger, higher-quality files.
"syncAnimatedImageToWordAudio": true, // For animated AVIF images, prepend a frozen first frame matching the existing word-audio duration so motion starts with sentence audio. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Changes apply live. Values: true | false
"mirrorMpvVolume": true, // Apply mpv's current software volume curve to generated sentence audio. Changes apply live. Values: true | false
"audioPadding": 0, // Seconds of padding appended to both ends of generated sentence audio and animated AVIF clips.
"fallbackDuration": 3, // Fallback clip duration in seconds when subtitle timing data is unavailable.
"maxMediaDuration": 30 // Maximum allowed media clip duration in seconds.
+5 -1
View File
@@ -162,13 +162,16 @@ Audio is extracted from the video file using the subtitle's start and end timest
"media": {
"generateAudio": true,
"normalizeAudio": true, // normalize generated clip loudness
"mirrorMpvVolume": true, // apply the current mpv volume level
"audioPadding": 0, // optional seconds before and after subtitle timing
"maxMediaDuration": 30 // cap total duration in seconds
}
}
```
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream. Generated sentence audio is loudness-normalized by default during extraction; set `normalizeAudio` to `false` to keep raw source loudness.
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream. Generated sentence audio is loudness-normalized by default during extraction; set `normalizeAudio` to `false` to keep raw source loudness. Changing this setting applies to the next extraction without restarting SubMiner.
`mirrorMpvVolume` is also enabled by default. Immediately before extracting each playback-overlay card's audio, SubMiner reads mpv's numeric `volume` and applies mpv's cubic software-volume curve after loudness normalization. For example, mpv volume `50` produces `0.5³ = 0.125` gain. Amplified output above mpv volume `100` is limited to a `-1 dBFS` ceiling before MP3 encoding to prevent clipping. It ignores mpv's separate `mute` state. If the volume property is missing, invalid, or unavailable, extraction continues with unity scaling; disabling this option skips the query and volume filter. Changing this setting applies to the next extraction without restarting SubMiner. YouTube cards queued for a background media-cache download retain the volume captured when the card was mined. Stats-dashboard mining does not currently have access to the active mpv property client, so it does not apply mpv volume scaling.
The audio is uploaded to Anki's media folder and inserted as `[sound:audio_<timestamp>.mp3]`.
@@ -349,6 +352,7 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
"imageFormat": "jpg",
"imageQuality": 92,
"normalizeAudio": true,
"mirrorMpvVolume": true,
"audioPadding": 0,
"maxMediaDuration": 30,
},
+40 -4
View File
@@ -1,12 +1,48 @@
# Changelog
## v0.17.2 (2026-06-28)
## v0.18.0 (2026-07-10)
**Added**
- Sentence Audio Normalization: Generated sentence audio is now normalized to -23 LUFS by default, and clips mined from playback mirror mpv's software volume curve with a limiter to prevent clipping. Both behaviors are configurable independently.
- Watch History Command: Added `subminer -H` / `--history` to browse watch history, replay or continue episodes, or pick one via fzf or rofi, with cover art shown in the rofi picker.
**Changed**
- Fzf Preview Layout: Moved fzf previews below launcher menus, giving long titles and metadata more room.
- Known-Word Highlighting: Now compares subtitle and Anki-card readings, preventing false matches between homographs and unrelated words that share a reading, while still supporting matching across kana and kanji spellings.
- Annotation Filtering: Standalone suffix tokens (e.g. さん, れる) are now excluded from JLPT/frequency/N+1 highlighting by default, matching how particles and interjections are treated; configurable via the pos2 exclusion setting.
- App Icon: Replaced the app icon with new pixel-art submarine artwork contributed by the community, used across the app icon, tray, notifications, README, docs site, and Stats page.
- Stats Trend Charts: Overhauled with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and value-sorted tooltips.
**Fixed**
- Background Stats Server: `subminer app` background launches now auto-start the stats server when enabled, and skip startup if one is already running.
- Character Name Highlighting: Character dictionaries now split unspaced native names more reliably, and portraits, highlights, and hover lookup survive punctuation, unmatched text, and competing dictionary matches without incorrectly splitting longer words.
- Highlighting Coverage: Frequency/JLPT highlighting and vocabulary stats now include content adverbs (e.g. 確かに, やはり) and kanji nouns MeCab tags as non-independent (e.g. 日, 点, 以外), while still suppressing interjections, pronouns, and grammar fragments; lexicalized kana expressions like かといって keep their annotations.
- Cover Art Fetching: Stats now fetches AniList cover art as soon as a new series starts playing, and backfills missing art for existing series on the next Stats visit.
- Kiku Field Grouping: The manual field-grouping dialog now stays above fullscreen mpv, remains usable across repeated attempts, closes abandoned windows after timeouts, and reports a clear error when the original card can no longer be loaded.
- Karaoke Subtitle Collapse: Secondary subtitles no longer stack dozens of one-syllable lines during openings and endings; repeated events collapse into one line, capped to a strip at the top.
- Unparsed Token Hover: Subtitle text Yomitan can't parse (truncated inflections, elongation runs) remains hoverable, while staying excluded from highlighting, N+1 candidate math, and vocabulary stats.
- YouTube Streaming: Fixed direct YouTube stream extraction that could corrupt signed URLs and cause ffmpeg 403 errors.
<details>
<summary>Internal changes</summary>
**Internal**
- Test lanes moved to `scripts/test-lanes.ts` with per-directory discovery and isolated per-file timeouts; CI now covers previously orphaned stats, scripts, plugin process-retry, and runtime-compat suites, plus a new stats lane in the change-verification workflow.
</details>
## Previous Versions
<details>
<summary>v0.17.x</summary>
<h2>v0.17.2 (2026-06-28)</h2>
**Fixed**
- YouTube Background Cache: Fixed Windows YouTube background media cache startup for YouTube URLs opened directly in mpv, including resolved stream URLs when mpv still exposes the original YouTube playlist entry, so queued Anki media updates can append audio and images after the cache finishes.
- YouTube Subtitle Picker: Manual subtitle picker requests now show an immediate configured notification while SubMiner probes tracks and opens the modal. Subtitle download progress is replaced with a transient success notification after tracks load.
## v0.17.1 (2026-06-27)
<h2>v0.17.1 (2026-06-27)</h2>
**Added**
- YouTube Media Cache Mode: Adds `youtube.mediaCache.mode` with `direct` and `background` options. Background mode uses a yt-dlp cache download when direct stream extraction is unreliable — creates a text-only card immediately, queues media updates for mined notes, and fills audio/image fields once the download finishes. Progress is announced via overlay/OSD notifications. Downloads are capped at 720p by default (`youtube.mediaCache.maxHeight`). Switching back to direct mode cancels any in-flight background download.
@@ -15,7 +51,7 @@
- Log Export: Fixed log filenames to use the local date so exports around UTC midnight include the current day's logs rather than stale prior-day files. Expanded export redaction to mask IPs, emails, auth and cookie headers, yt-dlp cookie arguments, URL credentials, token/key/password fields, and signed YouTube media URL parameters.
- YouTube Card Media: Improved media generation reliability by sending safer ffmpeg options for resolved streams and skipping stale stream maps (including cached YouTube files). Hardened background cache downloads with IPv4 and extractor retry flags; failed downloads now notify the user and clear queued media updates instead of leaving them silently pending. Stale background cache files are cleaned on startup and before each new download.
## v0.17.0 (2026-06-15)
<h2>v0.17.0 (2026-06-15)</h2>
**Changed**
@@ -50,7 +86,7 @@
</details>
## Previous Versions
</details>
<details>
<summary>v0.16.x</summary>
+3 -1
View File
@@ -952,6 +952,7 @@ Enable automatic Anki card creation and updates with media generation:
"animatedMaxHeight": 0,
"animatedCrf": 35,
"normalizeAudio": true,
"mirrorMpvVolume": true,
"audioPadding": 0,
"fallbackDuration": 3,
"maxMediaDuration": 30
@@ -1002,7 +1003,8 @@ This example is intentionally compact. The option table below documents availabl
| `ankiConnect.ai.model` | string | Optional model override for Anki AI translation/enrichment flows. |
| `ankiConnect.ai.systemPrompt` | string | Optional system prompt override for Anki AI translation/enrichment flows. |
| `media.generateAudio` | `true`, `false` | Generate audio clips from video (default: `true`) |
| `media.normalizeAudio` | `true`, `false` | Normalize generated sentence-audio loudness during media extraction (default: `true`). Set to `false` to keep raw source loudness. |
| `media.normalizeAudio` | `true`, `false` | Normalize generated sentence-audio loudness during media extraction (default: `true`). Set to `false` to keep raw source loudness. Changes apply live. |
| `media.mirrorMpvVolume` | `true`, `false` | Apply mpv's cubic software-volume curve to each generated sentence-audio clip (default: `true`). This ignores mpv's separate mute state, falls back to unity scaling if volume cannot be read, and applies changes live. |
| `media.generateImage` | `true`, `false` | Generate image/animation screenshots (default: `true`) |
| `media.imageType` | `"static"`, `"avif"` | Image type: static screenshot or animated AVIF (default: `"static"`) |
| `media.imageFormat` | `"jpg"`, `"png"`, `"webp"` | Image format (default: `"jpg"`) |
-1
View File
@@ -25,7 +25,6 @@ Episode completion for local `watched` state uses the shared `DEFAULT_MIN_WATCH_
- Leave `dbPath` empty to use the default location (`immersion.sqlite` in SubMiner's app-data directory).
- Set an explicit path to move the database (useful for backups, cloud syncing, or external tools).
- To share stats and watch history between two machines, use [`subminer sync <host>`](/launcher-script#sync-between-machines) instead of file-level cloud sync — it merges both databases without one side overwriting the other.
## Stats Dashboard
-24
View File
@@ -78,29 +78,6 @@ The first menu lists every locally watched series, most recently watched first,
Series whose directories are not currently accessible (e.g. an unmounted network share) are hidden from the list. Watch history requires the immersion tracker database (`immersionTracking.dbPath`, default `<config dir>/immersion.sqlite`), which SubMiner populates during playback.
## Sync Between Machines
`subminer sync <host>` merges immersion stats and watch history between two machines over SSH, so both end up with the union of sessions, lifetime totals, vocabulary counts, daily/monthly charts, and `--history` entries. `<host>` is anything `ssh` accepts (`user@hostname` or an ssh config alias); SubMiner must be installed on both machines at the same version.
```bash
subminer sync macbook # two-way sync with the host "macbook"
subminer sync user@192.168.1.20 # explicit user@host
subminer sync macbook --remote-cmd ~/bin/subminer # custom remote launcher path
```
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over `scp`, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time — syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
Close SubMiner (and stop the background stats daemon, `subminer stats -s`) on both machines before syncing; the command refuses to run while a SubMiner process may be writing the database (`--force` overrides). The mpv safety check requires a live socket connection, so a stale socket file left after mpv exits does not block sync. Both machines must be on the same SubMiner version — the sync aborts on a stats schema mismatch. Remote sync checks standard SubMiner and Bun locations (`~/.local/bin`, `~/.bun/bin`, Homebrew, `/usr/local/bin`, `/usr/bin`, and `/bin`) even when the non-interactive SSH shell omits them from `PATH`.
Two lower-level modes are used internally over SSH and also work standalone for manual transfers (e.g. via a USB drive):
```bash
subminer sync --snapshot /tmp/stats.sqlite # write a consistent snapshot of the local database
subminer sync --merge /tmp/stats.sqlite # merge a snapshot file into the local database
```
Unfinished sessions (a crash mid-playback) are skipped until the app finalizes them; they sync on the next run. Word/kanji "known" state from Anki is not part of the database and does not sync — each machine derives it from its own Anki collection.
## Common Commands
```bash
@@ -129,7 +106,6 @@ subminer stats -b # start background stats daemon
| `subminer mpv status` | Check mpv socket readiness |
| `subminer mpv socket` | Print active socket path |
| `subminer mpv idle` | Launch detached idle mpv instance |
| `subminer sync <host>` | Two-way stats/history sync with another machine over SSH |
| `subminer dictionary <path>` | Generate character dictionary ZIP from file/dir target |
| `subminer dictionary --candidates <path>` | List AniList candidate matches for character dictionary correction |
| `subminer dictionary --select <id> <path>` | Pin an AniList media ID for that target series |
+3 -2
View File
@@ -515,7 +515,7 @@
// ==========================================
// AnkiConnect Integration
// Automatic Anki updates and media generation options.
// Hot-reload: ankiConnect.ai.enabled, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
// Shared AI provider transport settings are read from top-level ai and typically require restart.
// Most other AnkiConnect settings still require restart.
// ==========================================
@@ -559,7 +559,8 @@
"animatedMaxHeight": 0, // Maximum height for animated AVIF captures, in pixels. Set to 0 to preserve aspect ratio.
"animatedCrf": 35, // Animated AVIF CRF quality target. Lower values produce larger, higher-quality files.
"syncAnimatedImageToWordAudio": true, // For animated AVIF images, prepend a frozen first frame matching the existing word-audio duration so motion starts with sentence audio. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Values: true | false
"normalizeAudio": true, // Normalize generated sentence audio loudness during media extraction. Changes apply live. Values: true | false
"mirrorMpvVolume": true, // Apply mpv's current software volume curve to generated sentence audio. Changes apply live. Values: true | false
"audioPadding": 0, // Seconds of padding appended to both ends of generated sentence audio and animated AVIF clips.
"fallbackDuration": 3, // Fallback clip duration in seconds when subtitle timing data is unavailable.
"maxMediaDuration": 30 // Maximum allowed media clip duration in seconds.
+1 -1
View File
@@ -90,7 +90,7 @@ SubMiner looks up each token's `frequencyRank` from `term_meta_bank_*.json` file
When `sourcePath` is omitted, SubMiner searches default install/runtime locations for `frequency-dictionary` directories automatically.
::: info
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist.
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist. For merged kana tokens, SubMiner keeps a rank when the dictionary headword reading covers the full token (for example, `かと言って` / `かといって`), while grammar wrapped around a shorter lemma remains unannotated.
:::
::: info
-4
View File
@@ -111,9 +111,6 @@ subminer config show # Print active config contents
subminer mpv socket # Print active mpv socket path
subminer mpv status # Exit 0 if socket is ready, else exit 1
subminer mpv idle # Launch detached idle mpv with SubMiner defaults
subminer sync media-box # Sync stats/watch history with an SSH host
subminer sync --snapshot ~/subminer-snapshot.sqlite # Write a local DB snapshot
subminer sync --merge ~/subminer-snapshot.sqlite # Merge a snapshot into the local DB
subminer dictionary /path/to/file-or-directory # Generate character dictionary ZIP from target (manual Yomitan import)
subminer dictionary --candidates /path/to/file.mkv
subminer dictionary --select 21355 /path/to/file.mkv
@@ -197,7 +194,6 @@ This flow requires `mpv.exe` to be discoverable. Leave `mpv.executablePath` blan
- `subminer logs -e`: export a sanitized ZIP of today's local-date logs, or the most recent logs when no current-day log exists. The exported copy masks common PII and secrets; on-disk logs are unchanged.
- `subminer config`: config file helpers (`path`, `show`).
- `subminer mpv`: mpv helpers (`status`, `socket`, `idle`).
- `subminer sync <host>`: sync immersion stats and watch history with another machine over SSH. The host is the SSH destination (`user@host` or an SSH config alias). Remote launcher checks include standard SubMiner and Bun paths even when SSH omits them from `PATH`. Use `--snapshot <file>` to write a consistent local stats DB snapshot, `--merge <file>` to merge a snapshot into the local stats DB, and `--force` to skip the running stats/mpv safety check. Advanced options: `--db <file>` overrides the local stats DB path, and `--remote-cmd <cmd>` overrides the `subminer` command used on the remote host.
- `subminer dictionary <path>`: generates a Yomitan-importable character dictionary ZIP from a file/directory target.
- Use `subminer dictionary --candidates <path>` and `subminer dictionary --select <id> <path>` to correct AniList character-dictionary matches for a whole series.
- `subminer texthooker`: texthooker-only shortcut (same behavior as `--texthooker`). A _texthooker_ is a web page that displays the current subtitle line as selectable text, so browser-based dictionary extensions and other tools can read along with playback.
@@ -1,29 +0,0 @@
<!-- read_when: changing managed mpv startup, pause-until-ready, or visible overlay boot ordering -->
# Early Managed Overlay Startup Design
Status: approved
Date: 2026-06-06
## Problem
Managed mpv startup can pause playback immediately, then leave SubMiner's tray and visible overlay
unavailable until Yomitan/tokenization warmups finish. Startup notifications therefore miss the
overlay surface and fall back to non-overlay status paths.
## Chosen Approach
For cold `--start --background --managed-playback` launches, handle initial args before waiting for
the deferred overlay warmup. That lets the tray and visible overlay shell initialize immediately
while the existing tokenization warmups continue in the background.
The mpv plugin pause gate stays armed. Playback release still waits for SubMiner's autoplay-ready
signal, which is emitted only after tokenization warmup and visible-overlay readiness. Existing
second-instance attach behavior remains unchanged: when the launcher finds an already-running
background app, it sends the same control command to that process and reuses its warmups/tokenizer.
## Checks
- Add a startup ordering regression test for managed background playback.
- Keep the existing deferred startup ordering for non-managed launches.
- Run the startup/runtime test slice plus SubMiner verification lane.
@@ -1,27 +0,0 @@
<!-- read_when: changing overlay notification hover, macOS mouse passthrough, or notification actions -->
# macOS Notification Hover Stability Design
Status: approved
Date: 2026-06-09
## Problem
On macOS, hovering a character dictionary build notification can make the card flicker and slide as
if it is hiding, then snap back. The likely trigger is the notification stack changing the overlay
window's mouse-passthrough state for a progress card that has no user action.
## Chosen Approach
Keep non-action overlay notifications visually stable and click-through on hover. Only notifications
with explicit actions should request interactive overlay input. The notification history panel keeps
its existing interactive behavior.
This avoids a macOS mouseenter/mouseleave passthrough loop for passive progress cards while
preserving clickable notification actions.
## Checks
- Add a renderer regression test for passive notification hover.
- Keep action-bearing notification cards interactive.
- Run the targeted overlay notification and mouse-ignore tests.
@@ -1,33 +0,0 @@
# Remote Sync Runtime PATH Design
## Problem
`subminer sync <host>` can locate a remote launcher while still failing to run
it. The generated launcher uses `#!/usr/bin/env bun`, and non-interactive SSH
shells commonly omit Bun's install directory from `PATH`.
## Design
Remote launcher probes and invocations will use a deterministic `PATH` prefix
covering SubMiner and Bun's supported/common locations:
- `$HOME/.local/bin`
- `$HOME/.bun/bin`
- `/opt/homebrew/bin`
- `/usr/local/bin`
- `/usr/bin`
- `/bin`
Resolution will run `<candidate> --help` under that PATH instead of only using
`command -v`. This confirms both the launcher and its shebang runtime work
before sync creates or transfers snapshots. The same prefixed command returned
by resolution will be used for remote snapshot and merge operations.
`--remote-cmd` remains a launcher path override and receives the same runtime
PATH. Remote commands continue to be shell-quoted where user-controlled.
## Validation
Add focused resolver coverage that simulates an SSH shell where only the
prefixed PATH can start SubMiner. Run launcher unit tests, type checking, docs
tests/build, and the SubMiner launcher/docs verification lanes.
@@ -1,31 +0,0 @@
# SSH Resolver Testability Design
## Problem
The remote launcher resolver test installs a temporary `ssh` executable and
mutates `process.env.PATH` after Bun starts. Bun's synchronous process launcher
does not reliably use that late PATH change, so the focused test fails locally
and in CI even though the command-building behavior is correct.
## Design
Allow `resolveRemoteSubminerCommand` to accept an optional remote runner that
defaults to the production `runSsh` function. Existing callers retain the same
behavior and two-argument API. The focused test will provide a small runner,
exercise the real candidate and PATH construction, and assert the host and
remote command passed across the SSH boundary.
Alternatives rejected:
- Explicitly forwarding `process.env` to `spawnSync` still depends on Bun's
executable-resolution behavior and changes production code to support a test
harness.
- Module-level mocking couples the test to loader behavior and obscures the
resolver's actual dependency.
## Validation
First update the focused test and confirm it remains red while the resolver
ignores the injected runner. Then add the minimal runner parameter and confirm
the focused test passes. Run the launcher verification lane, type checking,
and the repository's default handoff gate.
@@ -1,26 +0,0 @@
# Sync Stale Socket Guard Design
## Problem
The sync quiescence guard treats any configured mpv socket path as a running
session when the filesystem entry exists. Unix socket files can remain after
mpv exits, so a stale `/tmp/subminer-socket` blocks the remote merge even when
nothing is listening.
## Design
Reuse the launcher's existing Unix socket connection check. The sync guard will
block only when it can connect to the configured socket. Missing, stale, or
otherwise unreachable socket paths will not block sync, and sync will not
delete them.
Because socket connection checks are asynchronous, the sync dispatch, merge
mode, host mode, and quiescence checks will become async. The already-async
launcher entrypoint will await sync dispatch. Snapshot, transfer, merge,
cleanup, and error behavior otherwise remain unchanged.
## Validation
Add focused tests proving a stale socket is ignored and a live socket remains a
hard failure. Re-run sync command and socket tests, type checking, launcher
tests, docs checks, and the SubMiner launcher/docs verification lanes.
@@ -36,13 +36,6 @@ function createContext(): LauncherCommandContext {
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'info',
logRotation: 7,
passwordStore: '',
-190
View File
@@ -1,190 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import type { Args } from '../types.js';
import { createEmptyMergeSummary } from '../sync/sync-shared.js';
import type { LauncherCommandContext } from './context.js';
import { ensureTrackerQuiescent, runSyncCommand, type SyncCommandDeps } from './sync-command.js';
function makeContext(overrides: Partial<Args>): LauncherCommandContext {
return {
args: {
sync: true,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'warn',
...overrides,
} as Args,
scriptPath: '/tmp/subminer',
scriptName: 'subminer',
mpvSocketPath: '',
pluginRuntimeConfig: {},
appPath: null,
launcherJellyfinConfig: {},
processAdapter: process,
} as unknown as LauncherCommandContext;
}
function ok(stdout = ''): { status: number; stdout: string; stderr: string } {
return { status: 0, stdout, stderr: '' };
}
test('ensureTrackerQuiescent ignores stale sockets but rejects live sockets', async () => {
const context = makeContext({ syncDbPath: '/tmp/local.sqlite' });
context.mpvSocketPath = '/tmp/subminer-socket';
let socketConnectable = false;
const deps: Partial<SyncCommandDeps> = {
realpathSync: (() => '/tracker.sqlite') as unknown as typeof fs.realpathSync,
findLiveStatsDaemonPid: () => null,
canConnectUnixSocket: async () => socketConnectable,
fail: (message: string): never => {
throw new Error(message);
},
};
await ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps);
socketConnectable = true;
await assert.rejects(
async () => ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps),
/mpv\/SubMiner session appears to be running/,
);
});
test('runSyncCommand dispatches snapshot, merge, host, and missing-target modes', async () => {
const calls: string[] = [];
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (dbPath: string, outPath: string) => {
calls.push(`snapshot:${dbPath}->${outPath}`);
},
mergeSnapshotIntoDb: (dbPath: string, snapshotPath: string) => {
calls.push(`merge:${dbPath}<-${snapshotPath}`);
return createEmptyMergeSummary();
},
formatMergeSummary: () => 'summary',
ensureTrackerQuiescent: async () => {
calls.push('quiescent');
},
assertSafeSshHost: (host: string) => {
calls.push(`host:${host}`);
},
resolveRemoteSubminerCommand: () => 'subminer',
runSsh: (_host: string, command: string) => {
calls.push(`ssh:${command}`);
return command.startsWith('mktemp ') ? ok('/tmp/subminer-sync.remote\n') : ok();
},
runScp: (from: string, to: string) => {
calls.push(`scp:${from}->${to}`);
},
fail: (message: string): never => {
throw new Error(message);
},
};
assert.equal(
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncSnapshotPath: '/tmp/out.sqlite' }),
deps,
),
true,
);
assert.ok(calls.includes('snapshot:/tmp/local.sqlite->/tmp/out.sqlite'));
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncMergePath: '/tmp/in.sqlite' }),
deps,
);
assert.ok(calls.includes('quiescent'));
assert.ok(calls.includes('merge:/tmp/local.sqlite<-/tmp/in.sqlite'));
await runSyncCommand(
makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }),
deps,
);
assert.ok(calls.includes('host:media-box'));
await assert.rejects(
() => runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite' }), deps),
/sync requires a host, --snapshot <file>, or --merge <file>/,
);
});
test('runHostSync keeps tracker quiescent through local and remote merge and cleans up after failure', async () => {
const calls: string[] = [];
let localTmpDir = '';
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (_dbPath: string, outPath: string) => {
calls.push(`snapshot:${outPath}`);
fs.writeFileSync(outPath, 'snapshot');
},
mergeSnapshotIntoDb: () => {
calls.push('local-merge');
return createEmptyMergeSummary();
},
formatMergeSummary: () => 'summary',
ensureTrackerQuiescent: async () => {
calls.push('quiescent');
},
assertSafeSshHost: () => {},
resolveRemoteSubminerCommand: () => 'subminer',
mkdtempSync: ((prefix: string) => {
localTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), path.basename(prefix)));
return localTmpDir;
}) as typeof fs.mkdtempSync,
runSsh: (_host: string, command: string) => {
calls.push(`ssh:${command}`);
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
if (command.includes(' sync --snapshot ')) return ok();
if (command.includes(' sync --merge ')) {
return { status: 9, stdout: 'remote output', stderr: 'remote merge exploded' };
}
return ok();
},
runScp: (from: string, to: string) => {
calls.push(`scp:${from}->${to}`);
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
},
};
await assert.rejects(
() =>
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
/Remote merge failed on media-box[\s\S]*remote merge exploded/,
);
assert.equal(calls.filter((call) => call === 'quiescent').length, 3);
assert.ok(calls.indexOf('quiescent') < calls.findIndex((call) => call.startsWith('snapshot:')));
assert.ok(calls.includes('local-merge'));
assert.ok(calls.some((call) => call.startsWith('ssh:rm -rf ')));
assert.equal(fs.existsSync(localTmpDir), false);
});
test('runHostSync includes remote snapshot stderr in failures', async () => {
const deps: Partial<SyncCommandDeps> = {
createDbSnapshot: (_dbPath: string, outPath: string) => {
fs.writeFileSync(outPath, 'snapshot');
},
ensureTrackerQuiescent: async () => {},
assertSafeSshHost: () => {},
resolveRemoteSubminerCommand: () => 'subminer',
runSsh: (_host: string, command: string) => {
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
if (command.includes(' sync --snapshot ')) {
return { status: 5, stdout: '', stderr: 'snapshot permission denied' };
}
return ok();
},
runScp: () => {},
};
await assert.rejects(
() =>
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
/Remote snapshot failed on media-box[\s\S]*snapshot permission denied/,
);
});
-239
View File
@@ -1,239 +0,0 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fail, log } from '../log.js';
import { resolveImmersionDbPath } from '../history-db.js';
import {
createDbSnapshot,
findLiveStatsDaemonPid,
formatMergeSummary,
mergeSnapshotIntoDb,
} from '../sync/sync-db.js';
import {
assertSafeSshHost,
resolveRemoteSubminerCommand,
runScp,
runSsh,
shellQuote,
} from '../sync/ssh.js';
import { resolvePathMaybe } from '../util.js';
import { canConnectUnixSocket } from '../mpv.js';
import type { LauncherCommandContext } from './context.js';
import type { RemoteRunResult } from '../sync/ssh.js';
export interface SyncCommandDeps {
createDbSnapshot: typeof createDbSnapshot;
mergeSnapshotIntoDb: typeof mergeSnapshotIntoDb;
formatMergeSummary: typeof formatMergeSummary;
findLiveStatsDaemonPid: typeof findLiveStatsDaemonPid;
assertSafeSshHost: typeof assertSafeSshHost;
resolveRemoteSubminerCommand: typeof resolveRemoteSubminerCommand;
runScp: typeof runScp;
runSsh: typeof runSsh;
fail: typeof fail;
log: typeof log;
canConnectUnixSocket: typeof canConnectUnixSocket;
realpathSync: typeof fs.realpathSync;
mkdtempSync: typeof fs.mkdtempSync;
rmSync: typeof fs.rmSync;
consoleLog: typeof console.log;
writeStdout: typeof process.stdout.write;
ensureTrackerQuiescent: (context: LauncherCommandContext, dbPath: string) => Promise<void>;
}
function resolveDbPath(context: LauncherCommandContext): string {
const override = context.args.syncDbPath.trim();
return override ? resolvePathMaybe(override) : resolveImmersionDbPath();
}
function isTrackerDb(dbPath: string, deps: SyncCommandDeps): boolean {
const trackerDbPath = resolveImmersionDbPath();
try {
return deps.realpathSync(dbPath) === deps.realpathSync(trackerDbPath);
} catch {
return dbPath === trackerDbPath;
}
}
export async function ensureTrackerQuiescent(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
if (context.args.syncForce) return;
// A running SubMiner only holds the tracker's own database; --db pointed
// elsewhere needs no guard.
if (!isTrackerDb(dbPath, deps)) return;
const daemonPid = deps.findLiveStatsDaemonPid(dbPath);
if (daemonPid !== null) {
deps.fail(
`The SubMiner stats server is running (pid ${daemonPid}). Stop it with "subminer stats -s" (or close SubMiner) before syncing, or pass --force.`,
);
}
if (context.mpvSocketPath && (await deps.canConnectUnixSocket(context.mpvSocketPath))) {
deps.fail(
`An mpv/SubMiner session appears to be running (socket ${context.mpvSocketPath}). Close it before syncing, or pass --force.`,
);
}
}
const defaultSyncCommandDeps: SyncCommandDeps = {
createDbSnapshot,
mergeSnapshotIntoDb,
formatMergeSummary,
findLiveStatsDaemonPid,
assertSafeSshHost,
resolveRemoteSubminerCommand,
runScp,
runSsh,
fail,
log,
canConnectUnixSocket,
realpathSync: fs.realpathSync,
mkdtempSync: fs.mkdtempSync,
rmSync: fs.rmSync,
consoleLog: console.log,
writeStdout: process.stdout.write.bind(process.stdout),
ensureTrackerQuiescent: async (context, dbPath) => ensureTrackerQuiescent(context, dbPath),
};
function resolveSyncCommandDeps(inputDeps: Partial<SyncCommandDeps> = {}): SyncCommandDeps {
return { ...defaultSyncCommandDeps, ...inputDeps };
}
export function runSnapshotMode(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): void {
const deps = resolveSyncCommandDeps(inputDeps);
const outPath = resolvePathMaybe(context.args.syncSnapshotPath);
deps.createDbSnapshot(dbPath, outPath);
deps.consoleLog(outPath);
}
export async function runMergeMode(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
await deps.ensureTrackerQuiescent(context, dbPath);
const snapshotPath = resolvePathMaybe(context.args.syncMergePath);
const summary = deps.mergeSnapshotIntoDb(dbPath, snapshotPath);
deps.consoleLog(deps.formatMergeSummary(summary));
}
function cleanupRemote(host: string, remoteTmpDir: string, deps: SyncCommandDeps): void {
if (!remoteTmpDir.startsWith('/tmp/')) return;
deps.runSsh(host, `rm -rf ${shellQuote(remoteTmpDir)}`);
}
function formatRemoteRunError(message: string, run: RemoteRunResult): string {
const stderr = run.stderr.trim();
return stderr ? `${message}\n${stderr}` : message;
}
export async function runHostSync(
context: LauncherCommandContext,
dbPath: string,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<void> {
const deps = resolveSyncCommandDeps(inputDeps);
const { args } = context;
const host = args.syncHost;
deps.assertSafeSshHost(host);
await deps.ensureTrackerQuiescent(context, dbPath);
const remoteCmd = deps.resolveRemoteSubminerCommand(host, args.syncRemoteCmd || null);
deps.log('debug', args.logLevel, `Remote subminer command: ${remoteCmd}`);
const localTmpDir = deps.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-'));
let remoteTmpDir = '';
try {
// Signal failures by throwing (not fail(), which exits synchronously and
// would skip the finally cleanup, leaking temp dirs holding snapshot data).
// main().catch() reports the message the same way fail() would.
const mktemp = deps.runSsh(host, 'mktemp -d /tmp/subminer-sync.XXXXXX');
remoteTmpDir = mktemp.stdout.trim();
if (mktemp.status !== 0 || !remoteTmpDir.startsWith('/tmp/')) {
throw new Error(`Could not create a temporary directory on ${host}.`);
}
const forceFlag = args.syncForce ? ' --force' : '';
deps.consoleLog(`Snapshotting local database (${dbPath})...`);
const localSnapshot = path.join(localTmpDir, 'local.sqlite');
deps.createDbSnapshot(dbPath, localSnapshot);
deps.consoleLog(`Snapshotting ${host}...`);
const remoteSnapshot = `${remoteTmpDir}/snapshot.sqlite`;
const snapshotRun = deps.runSsh(
host,
`${remoteCmd} sync --snapshot ${shellQuote(remoteSnapshot)}${forceFlag}`,
);
if (snapshotRun.status !== 0) {
throw new Error(formatRemoteRunError(`Remote snapshot failed on ${host}.`, snapshotRun));
}
const pulledSnapshot = path.join(localTmpDir, 'remote.sqlite');
deps.runScp(`${host}:${remoteSnapshot}`, pulledSnapshot);
const incomingSnapshot = `${remoteTmpDir}/incoming.sqlite`;
deps.runScp(localSnapshot, `${host}:${incomingSnapshot}`);
deps.consoleLog(`\nMerging ${host} -> local:`);
await deps.ensureTrackerQuiescent(context, dbPath);
const summary = deps.mergeSnapshotIntoDb(dbPath, pulledSnapshot);
deps.consoleLog(deps.formatMergeSummary(summary));
deps.consoleLog(`\nMerging local -> ${host}:`);
await deps.ensureTrackerQuiescent(context, dbPath);
const mergeRun = deps.runSsh(
host,
`${remoteCmd} sync --merge ${shellQuote(incomingSnapshot)}${forceFlag}`,
);
deps.writeStdout(mergeRun.stdout);
if (mergeRun.status !== 0) {
throw new Error(
formatRemoteRunError(
`Remote merge failed on ${host}. The local database was updated; re-run "subminer sync ${host}" once the remote issue is fixed.`,
mergeRun,
),
);
}
deps.consoleLog('\nSync complete.');
} finally {
deps.rmSync(localTmpDir, { recursive: true, force: true });
if (remoteTmpDir) {
try {
cleanupRemote(host, remoteTmpDir, deps);
} catch {
// best effort
}
}
}
}
export async function runSyncCommand(
context: LauncherCommandContext,
inputDeps: Partial<SyncCommandDeps> = {},
): Promise<boolean> {
const deps = resolveSyncCommandDeps(inputDeps);
const { args } = context;
if (!args.sync) return false;
const dbPath = resolveDbPath(context);
if (args.syncSnapshotPath) {
runSnapshotMode(context, dbPath, deps);
} else if (args.syncMergePath) {
await runMergeMode(context, dbPath, deps);
} else if (args.syncHost) {
await runHostSync(context, dbPath, deps);
} else {
deps.fail('sync requires a host, --snapshot <file>, or --merge <file>.');
}
return true;
}
-32
View File
@@ -135,14 +135,6 @@ test('applyInvocationsToArgs maps config and jellyfin invocation state', () => {
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -189,14 +181,6 @@ test('applyInvocationsToArgs maps settings invocation to settings window', () =>
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -236,14 +220,6 @@ test('applyInvocationsToArgs fails when config invocation has no action', () =>
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
@@ -281,14 +257,6 @@ test('applyInvocationsToArgs maps texthooker browser-open request', () => {
statsCleanupVocab: false,
statsCleanupLifetime: false,
statsLogLevel: null,
syncTriggered: false,
syncHost: null,
syncSnapshotPath: null,
syncMergePath: null,
syncRemoteCmd: null,
syncDbPath: null,
syncForce: false,
syncLogLevel: null,
doctorTriggered: false,
doctorLogLevel: null,
doctorRefreshKnownWords: false,
-17
View File
@@ -199,13 +199,6 @@ export function createDefaultArgs(
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: loggingConfig.level ?? 'warn',
logRotation: loggingConfig.rotation ?? 7,
passwordStore: '',
@@ -271,16 +264,6 @@ export function applyInvocationsToArgs(parsed: Args, invocations: CliInvocations
) {
fail('Dictionary target path is required.');
}
if (invocations.syncTriggered) {
parsed.sync = true;
parsed.syncHost = invocations.syncHost ?? '';
parsed.syncSnapshotPath = invocations.syncSnapshotPath ?? '';
parsed.syncMergePath = invocations.syncMergePath ?? '';
parsed.syncRemoteCmd = invocations.syncRemoteCmd ?? '';
parsed.syncDbPath = invocations.syncDbPath ?? '';
parsed.syncForce = invocations.syncForce;
if (invocations.syncLogLevel) parsed.logLevel = parseLogLevel(invocations.syncLogLevel);
}
if (invocations.doctorTriggered) parsed.doctor = true;
if (invocations.doctorRefreshKnownWords) parsed.doctorRefreshKnownWords = true;
if (invocations.logsTriggered && !invocations.logsExport) {
-56
View File
@@ -38,14 +38,6 @@ export interface CliInvocations {
statsCleanupVocab: boolean;
statsCleanupLifetime: boolean;
statsLogLevel: string | null;
syncTriggered: boolean;
syncHost: string | null;
syncSnapshotPath: string | null;
syncMergePath: string | null;
syncRemoteCmd: string | null;
syncDbPath: string | null;
syncForce: boolean;
syncLogLevel: string | null;
doctorTriggered: boolean;
doctorLogLevel: string | null;
doctorRefreshKnownWords: boolean;
@@ -106,7 +98,6 @@ function getTopLevelCommand(argv: string[]): { name: string; index: number } | n
'dictionary',
'dict',
'stats',
'sync',
'texthooker',
'app',
'bin',
@@ -169,14 +160,6 @@ export function parseCliPrograms(
let statsCleanupVocab = false;
let statsCleanupLifetime = false;
let statsLogLevel: string | null = null;
let syncTriggered = false;
let syncHost: string | null = null;
let syncSnapshotPath: string | null = null;
let syncMergePath: string | null = null;
let syncRemoteCmd: string | null = null;
let syncDbPath: string | null = null;
let syncForce = false;
let syncLogLevel: string | null = null;
let doctorLogLevel: string | null = null;
let doctorRefreshKnownWords = false;
let logsTriggered = false;
@@ -306,37 +289,6 @@ export function parseCliPrograms(
statsLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
});
commandProgram
.command('sync')
.description('Sync stats and watch history with another machine over SSH')
.argument('[host]', 'SSH destination (user@host or an ssh config alias)')
.option('--snapshot <file>', 'Write a consistent snapshot of the local stats database')
.option('--merge <file>', 'Merge a snapshot database file into the local stats database')
.option('--db <file>', 'Override the local stats database path')
.option('--remote-cmd <cmd>', 'subminer command to run on the remote host')
.option('-f, --force', 'Skip the running-app safety check')
.option('--log-level <level>', 'Log level')
.action((rawHost: string | undefined, options: Record<string, unknown>) => {
const host = typeof rawHost === 'string' ? rawHost.trim() : '';
const snapshot = typeof options.snapshot === 'string' ? options.snapshot.trim() : '';
const merge = typeof options.merge === 'string' ? options.merge.trim() : '';
const modes = [Boolean(host), Boolean(snapshot), Boolean(merge)].filter(Boolean).length;
if (modes === 0) {
throw new Error('Sync requires a host, --snapshot <file>, or --merge <file>.');
}
if (modes > 1) {
throw new Error('Sync host, --snapshot, and --merge cannot be combined.');
}
syncTriggered = true;
syncHost = host || null;
syncSnapshotPath = snapshot || null;
syncMergePath = merge || null;
syncRemoteCmd = typeof options.remoteCmd === 'string' ? options.remoteCmd.trim() || null : null;
syncDbPath = typeof options.db === 'string' ? options.db.trim() || null : null;
syncForce = options.force === true;
syncLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
});
commandProgram
.command('doctor')
.description('Run dependency and environment checks')
@@ -448,14 +400,6 @@ export function parseCliPrograms(
statsCleanupVocab,
statsCleanupLifetime,
statsLogLevel,
syncTriggered,
syncHost,
syncSnapshotPath,
syncMergePath,
syncRemoteCmd,
syncDbPath,
syncForce,
syncLogLevel,
doctorTriggered,
doctorLogLevel,
doctorRefreshKnownWords,
-7
View File
@@ -30,13 +30,6 @@ function createArgs(): Args {
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'info',
logRotation: 7,
passwordStore: '',
-5
View File
@@ -22,7 +22,6 @@ import { runLogsCommand } from './commands/logs-command.js';
import { runStatsCommand } from './commands/stats-command.js';
import { runJellyfinCommand } from './commands/jellyfin-command.js';
import { runHistoryCommand } from './commands/history-command.js';
import { runSyncCommand } from './commands/sync-command.js';
import { runPlaybackCommand } from './commands/playback-command.js';
import { runUpdateCommand } from './commands/update-command.js';
@@ -108,10 +107,6 @@ async function main(): Promise<void> {
return;
}
if (await runSyncCommand(context)) {
return;
}
const resolvedAppPath = ensureAppPath(context);
state.appPath = resolvedAppPath;
log('debug', args.logLevel, `Using SubMiner app binary: ${resolvedAppPath}`);
-7
View File
@@ -571,13 +571,6 @@ function makeArgs(overrides: Partial<Args> = {}): Args {
texthookerOpenBrowser: false,
useRofi: false,
history: false,
sync: false,
syncHost: '',
syncSnapshotPath: '',
syncMergePath: '',
syncRemoteCmd: '',
syncDbPath: '',
syncForce: false,
logLevel: 'error',
logRotation: 7,
passwordStore: '',
+1 -1
View File
@@ -1683,7 +1683,7 @@ async function sleepMs(ms: number): Promise<void> {
await new Promise<void>((resolve) => setTimeout(resolve, ms));
}
export async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
return await new Promise<boolean>((resolve) => {
const socket = net.createConnection(socketPath);
let settled = false;
-423
View File
@@ -1,423 +0,0 @@
import { Database } from 'bun:sqlite';
import { insertRow, tableExists, type SyncMergeSummary } from './sync-shared.js';
const ANIME_COPY_COLUMNS = [
'normalized_title_key',
'canonical_title',
'anilist_id',
'title_romaji',
'title_english',
'title_native',
'episodes_total',
'description',
'metadata_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const VIDEO_COPY_COLUMNS = [
'video_key',
'canonical_title',
'source_type',
'source_path',
'source_url',
'parsed_basename',
'parsed_title',
'parsed_season',
'parsed_episode',
'parser_source',
'parser_confidence',
'parse_metadata_json',
'watched',
'duration_ms',
'file_size_bytes',
'codec_id',
'container_id',
'width_px',
'height_px',
'fps_x100',
'bitrate_kbps',
'audio_codec_id',
'hash_sha256',
'screenshot_path',
'metadata_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const MEDIA_ART_COPY_COLUMNS = [
'anilist_id',
'cover_url',
'cover_blob',
'cover_blob_hash',
'title_romaji',
'title_english',
'episodes_total',
'fetched_at_ms',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const YOUTUBE_COPY_COLUMNS = [
'youtube_video_id',
'video_url',
'video_title',
'video_thumbnail_url',
'channel_id',
'channel_name',
'channel_url',
'channel_thumbnail_url',
'uploader_id',
'uploader_url',
'description',
'metadata_json',
'fetched_at_ms',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const WORD_COPY_COLUMNS = [
'headword',
'word',
'reading',
'part_of_speech',
'pos1',
'pos2',
'pos3',
'first_seen',
'last_seen',
'frequency',
'frequency_rank',
] as const;
type SqlRow = Record<string, unknown>;
function selectAll(db: Database, sql: string, params: unknown[] = []): SqlRow[] {
return db.query<SqlRow>(sql).all(...params);
}
export function mergeAnime(
local: Database,
remote: Database,
summary: SyncMergeSummary,
): Map<number, number> {
const map = new Map<number, number>();
const byAnilist = local.prepare<SqlRow>('SELECT anime_id FROM imm_anime WHERE anilist_id = ?');
const byTitleKey = local.prepare<SqlRow>(
'SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?',
);
const fillMissing = local.prepare(
`UPDATE imm_anime
SET
title_romaji = COALESCE(title_romaji, ?),
title_english = COALESCE(title_english, ?),
title_native = COALESCE(title_native, ?),
episodes_total = COALESCE(episodes_total, ?),
description = COALESCE(description, ?)
WHERE anime_id = ?`,
);
for (const row of selectAll(
remote,
`SELECT anime_id, ${ANIME_COPY_COLUMNS.join(', ')} FROM imm_anime`,
)) {
const remoteId = Number(row.anime_id);
const existing =
(row.anilist_id !== null ? byAnilist.get(row.anilist_id) : undefined) ??
byTitleKey.get(row.normalized_title_key);
if (existing) {
const localId = Number(existing.anime_id);
map.set(remoteId, localId);
fillMissing.run(
row.title_romaji,
row.title_english,
row.title_native,
row.episodes_total,
row.description,
localId,
);
continue;
}
// No local row matched by anilist_id (checked first in `existing` above)
// or title key, so the remote anilist_id — if any — is free to insert as-is.
const values = ANIME_COPY_COLUMNS.map((column) => row[column]);
map.set(remoteId, insertRow(local, 'imm_anime', ANIME_COPY_COLUMNS, values));
summary.animeAdded += 1;
}
return map;
}
export interface VideoMergeResult {
videoIdMap: Map<number, number>;
addedVideoIds: Set<number>;
}
export function mergeVideos(
local: Database,
remote: Database,
animeIdMap: Map<number, number>,
summary: SyncMergeSummary,
): VideoMergeResult {
const videoIdMap = new Map<number, number>();
const addedVideoIds = new Set<number>();
const byKey = local.prepare<SqlRow>(
'SELECT video_id, watched FROM imm_videos WHERE video_key = ?',
);
const setWatched = local.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?');
for (const row of selectAll(
remote,
`SELECT video_id, anime_id, ${VIDEO_COPY_COLUMNS.join(', ')} FROM imm_videos`,
)) {
const remoteId = Number(row.video_id);
const mappedAnimeId =
row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
const existing = byKey.get(row.video_key);
if (existing) {
const localId = Number(existing.video_id);
videoIdMap.set(remoteId, localId);
if (Number(row.watched) > 0 && Number(existing.watched) <= 0) {
setWatched.run(localId);
}
continue;
}
const columns = ['anime_id', ...VIDEO_COPY_COLUMNS];
const values = [mappedAnimeId, ...VIDEO_COPY_COLUMNS.map((column) => row[column])];
const localId = insertRow(local, 'imm_videos', columns, values);
videoIdMap.set(remoteId, localId);
addedVideoIds.add(remoteId);
summary.videosAdded += 1;
}
return { videoIdMap, addedVideoIds };
}
export function mergeMediaMetadata(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
addedVideoIds: Set<number>,
): void {
if (videoIdMap.size === 0) return;
const metadataVideoIds = new Set<number>([...addedVideoIds, ...videoIdMap.keys()]);
const hasBlobStore =
tableExists(local, 'imm_cover_art_blobs') && tableExists(remote, 'imm_cover_art_blobs');
const copyBlob = hasBlobStore
? local.prepare(
`INSERT INTO imm_cover_art_blobs (blob_hash, cover_blob, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)
ON CONFLICT(blob_hash) DO NOTHING`,
)
: null;
const readBlob = hasBlobStore
? remote.prepare<SqlRow>('SELECT * FROM imm_cover_art_blobs WHERE blob_hash = ?')
: null;
if (tableExists(remote, 'imm_media_art') && tableExists(local, 'imm_media_art')) {
const localArtExists = local.prepare<SqlRow>(
'SELECT 1 FROM imm_media_art WHERE video_id = ? LIMIT 1',
);
for (const remoteVideoId of metadataVideoIds) {
const localVideoId = videoIdMap.get(remoteVideoId)!;
if (localArtExists.get(localVideoId)) continue;
const row = remote
.query<SqlRow>(
`SELECT ${MEDIA_ART_COPY_COLUMNS.join(', ')} FROM imm_media_art WHERE video_id = ?`,
)
.get(remoteVideoId);
if (!row) continue;
if (row.cover_blob_hash && copyBlob && readBlob) {
const blob = readBlob.get(row.cover_blob_hash);
if (blob) {
copyBlob.run(blob.blob_hash, blob.cover_blob, blob.CREATED_DATE, blob.LAST_UPDATE_DATE);
}
}
insertRow(
local,
'imm_media_art',
['video_id', ...MEDIA_ART_COPY_COLUMNS],
[localVideoId, ...MEDIA_ART_COPY_COLUMNS.map((column) => row[column])],
);
}
}
if (tableExists(remote, 'imm_youtube_videos') && tableExists(local, 'imm_youtube_videos')) {
const localYoutubeExists = local.prepare<SqlRow>(
'SELECT 1 FROM imm_youtube_videos WHERE video_id = ? LIMIT 1',
);
for (const remoteVideoId of metadataVideoIds) {
const localVideoId = videoIdMap.get(remoteVideoId)!;
if (localYoutubeExists.get(localVideoId)) continue;
const row = remote
.query<SqlRow>(
`SELECT ${YOUTUBE_COPY_COLUMNS.join(', ')} FROM imm_youtube_videos WHERE video_id = ?`,
)
.get(remoteVideoId);
if (!row) continue;
insertRow(
local,
'imm_youtube_videos',
['video_id', ...YOUTUBE_COPY_COLUMNS],
[localVideoId, ...YOUTUBE_COPY_COLUMNS.map((column) => row[column])],
);
}
}
}
export function mergeExcludedWords(
local: Database,
remote: Database,
summary: SyncMergeSummary,
): void {
if (
!tableExists(remote, 'imm_stats_excluded_words') ||
!tableExists(local, 'imm_stats_excluded_words')
) {
return;
}
const insert = local.prepare(
`INSERT INTO imm_stats_excluded_words (headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(headword, word, reading) DO NOTHING`,
);
for (const row of selectAll(
remote,
'SELECT headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE FROM imm_stats_excluded_words',
)) {
const result = insert.run(
row.headword,
row.word,
row.reading,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.excludedWordsAdded += result.changes;
}
}
/**
* Lazily maps remote imm_words / imm_kanji ids onto local rows by natural key
* ((headword, word, reading) / kanji). New rows are copied with the remote's
* accumulated frequency; rows that already exist locally get their frequency
* incremented later with only the occurrence counts this merge adds (the
* remote total would double-count lines merged in earlier syncs).
*/
export class LexiconResolver {
private readonly wordMap = new Map<number, { localId: number; isNew: boolean }>();
private readonly kanjiMap = new Map<number, { localId: number; isNew: boolean }>();
readonly wordFrequencyDeltas = new Map<number, number>();
readonly kanjiFrequencyDeltas = new Map<number, number>();
constructor(
private readonly local: Database,
private readonly remote: Database,
private readonly summary: SyncMergeSummary,
) {}
resolveWord(remoteWordId: number): number {
const cached = this.wordMap.get(remoteWordId);
if (cached) return cached.localId;
const row = this.remote
.query<SqlRow>(`SELECT ${WORD_COPY_COLUMNS.join(', ')} FROM imm_words WHERE id = ?`)
.get(remoteWordId);
if (!row) throw new Error(`Snapshot references missing imm_words row ${remoteWordId}`);
const existing = this.local
.query<SqlRow>('SELECT id FROM imm_words WHERE headword IS ? AND word IS ? AND reading IS ?')
.get(row.headword, row.word, row.reading);
let entry: { localId: number; isNew: boolean };
if (existing) {
entry = { localId: Number(existing.id), isNew: false };
this.local
.prepare(
`UPDATE imm_words
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
WHERE id = ?`,
)
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
} else {
const localId = insertRow(
this.local,
'imm_words',
WORD_COPY_COLUMNS,
WORD_COPY_COLUMNS.map((column) => row[column]),
);
entry = { localId, isNew: true };
this.summary.wordsAdded += 1;
}
this.wordMap.set(remoteWordId, entry);
return entry.localId;
}
resolveKanji(remoteKanjiId: number): number {
const cached = this.kanjiMap.get(remoteKanjiId);
if (cached) return cached.localId;
const row = this.remote
.query<SqlRow>('SELECT kanji, first_seen, last_seen, frequency FROM imm_kanji WHERE id = ?')
.get(remoteKanjiId);
if (!row) throw new Error(`Snapshot references missing imm_kanji row ${remoteKanjiId}`);
const existing = this.local
.query<SqlRow>('SELECT id FROM imm_kanji WHERE kanji IS ?')
.get(row.kanji);
let entry: { localId: number; isNew: boolean };
if (existing) {
entry = { localId: Number(existing.id), isNew: false };
this.local
.prepare(
`UPDATE imm_kanji
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
WHERE id = ?`,
)
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
} else {
const localId = insertRow(
this.local,
'imm_kanji',
['kanji', 'first_seen', 'last_seen', 'frequency'],
[row.kanji, row.first_seen, row.last_seen, row.frequency],
);
entry = { localId, isNew: true };
this.summary.kanjiAdded += 1;
}
this.kanjiMap.set(remoteKanjiId, entry);
return entry.localId;
}
addWordOccurrences(remoteWordId: number, count: number): void {
const entry = this.wordMap.get(remoteWordId);
if (!entry || entry.isNew) return;
this.wordFrequencyDeltas.set(
entry.localId,
(this.wordFrequencyDeltas.get(entry.localId) ?? 0) + count,
);
}
addKanjiOccurrences(remoteKanjiId: number, count: number): void {
const entry = this.kanjiMap.get(remoteKanjiId);
if (!entry || entry.isNew) return;
this.kanjiFrequencyDeltas.set(
entry.localId,
(this.kanjiFrequencyDeltas.get(entry.localId) ?? 0) + count,
);
}
applyFrequencyDeltas(): void {
const updateWord = this.local.prepare(
'UPDATE imm_words SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
);
for (const [localId, delta] of this.wordFrequencyDeltas) {
updateWord.run(delta, localId);
}
const updateKanji = this.local.prepare(
'UPDATE imm_kanji SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
);
for (const [localId, delta] of this.kanjiFrequencyDeltas) {
updateKanji.run(delta, localId);
}
}
}
-265
View File
@@ -1,265 +0,0 @@
import { Database } from 'bun:sqlite';
import { nowDbTimestamp, tableExists, type SyncMergeSummary } from './sync-shared.js';
type SqlRow = Record<string, unknown>;
const LOCAL_DAY_EXPR = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
const LOCAL_MONTH_EXPR = `CAST(strftime('%Y%m', CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER)`;
// Ported from upsertDailyRollupsForGroups / upsertMonthlyRollupsForGroups in
// src/core/services/immersion-tracker/maintenance.ts — must stay in sync.
const DAILY_ROLLUP_UPSERT = `
WITH matching_sessions AS (
SELECT * FROM imm_sessions
WHERE ${LOCAL_DAY_EXPR} = ? AND video_id = ?
),
session_metrics AS (
SELECT
t.session_id,
MAX(t.active_watched_ms) AS max_active_ms,
MAX(t.lines_seen) AS max_lines,
MAX(t.tokens_seen) AS max_tokens,
MAX(t.cards_mined) AS max_cards,
MAX(t.lookup_count) AS max_lookups,
MAX(t.lookup_hits) AS max_hits
FROM imm_session_telemetry t
JOIN matching_sessions s ON s.session_id = t.session_id
GROUP BY t.session_id
)
INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
CREATED_DATE, LAST_UPDATE_DATE
)
SELECT
${LOCAL_DAY_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_day,
s.video_id AS video_id,
COUNT(DISTINCT s.session_id) AS total_sessions,
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
THEN (COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) * 60.0)
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
ELSE NULL
END AS cards_per_hour,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
THEN COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0)
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
ELSE NULL
END AS tokens_per_min,
CASE
WHEN COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) > 0
THEN CAST(COALESCE(SUM(COALESCE(sm.max_hits, s.lookup_hits)), 0) AS REAL)
/ CAST(COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) AS REAL)
ELSE NULL
END AS lookup_hit_rate,
? AS CREATED_DATE,
? AS LAST_UPDATE_DATE
FROM matching_sessions s
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
GROUP BY rollup_day, s.video_id
ON CONFLICT (rollup_day, video_id) DO UPDATE SET
total_sessions = excluded.total_sessions,
total_active_min = excluded.total_active_min,
total_lines_seen = excluded.total_lines_seen,
total_tokens_seen = excluded.total_tokens_seen,
total_cards = excluded.total_cards,
cards_per_hour = excluded.cards_per_hour,
tokens_per_min = excluded.tokens_per_min,
lookup_hit_rate = excluded.lookup_hit_rate,
CREATED_DATE = COALESCE(imm_daily_rollups.CREATED_DATE, excluded.CREATED_DATE),
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
`;
const MONTHLY_ROLLUP_UPSERT = `
WITH matching_sessions AS (
SELECT * FROM imm_sessions
WHERE ${LOCAL_MONTH_EXPR} = ? AND video_id = ?
),
session_metrics AS (
SELECT
t.session_id,
MAX(t.active_watched_ms) AS max_active_ms,
MAX(t.lines_seen) AS max_lines,
MAX(t.tokens_seen) AS max_tokens,
MAX(t.cards_mined) AS max_cards
FROM imm_session_telemetry t
JOIN matching_sessions s ON s.session_id = t.session_id
GROUP BY t.session_id
)
INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
)
SELECT
${LOCAL_MONTH_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_month,
s.video_id AS video_id,
COUNT(DISTINCT s.session_id) AS total_sessions,
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
? AS CREATED_DATE,
? AS LAST_UPDATE_DATE
FROM matching_sessions s
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
GROUP BY rollup_month, s.video_id
ON CONFLICT (rollup_month, video_id) DO UPDATE SET
total_sessions = excluded.total_sessions,
total_active_min = excluded.total_active_min,
total_lines_seen = excluded.total_lines_seen,
total_tokens_seen = excluded.total_tokens_seen,
total_cards = excluded.total_cards,
CREATED_DATE = COALESCE(imm_monthly_rollups.CREATED_DATE, excluded.CREATED_DATE),
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
`;
/**
* Recompute daily/monthly rollup groups touched by the newly merged sessions
* from the (now merged) local session + telemetry data. The maintenance
* watermark is left alone: telemetry newer than it gets recomputed again by
* the app later, which is idempotent.
*/
export function refreshRollupsForNewSessions(
local: Database,
newSessionIds: number[],
summary: SyncMergeSummary,
): void {
if (newSessionIds.length === 0) return;
const groups = new Map<string, { day: number; month: number; videoId: number }>();
for (let offset = 0; offset < newSessionIds.length; offset += 500) {
const chunk = newSessionIds.slice(offset, offset + 500);
const rows = local
.query<SqlRow>(
`SELECT DISTINCT ${LOCAL_DAY_EXPR} AS rollup_day, ${LOCAL_MONTH_EXPR} AS rollup_month, video_id
FROM imm_sessions WHERE session_id IN (${chunk.map(() => '?').join(',')})`,
)
.all(...chunk);
for (const row of rows) {
const day = Number(row.rollup_day);
const month = Number(row.rollup_month);
const videoId = Number(row.video_id);
groups.set(`${day}-${videoId}`, { day, month, videoId });
}
}
const stampMs = nowDbTimestamp();
const deleteDaily = local.prepare(
'DELETE FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ?',
);
const deleteMonthly = local.prepare(
'DELETE FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ?',
);
const upsertDaily = local.prepare(DAILY_ROLLUP_UPSERT);
const upsertMonthly = local.prepare(MONTHLY_ROLLUP_UPSERT);
const monthlyGroups = new Set<string>();
for (const { day, month, videoId } of groups.values()) {
deleteDaily.run(day, videoId);
upsertDaily.run(day, videoId, stampMs, stampMs);
summary.rollupGroupsRecomputed += 1;
const monthKey = `${month}-${videoId}`;
if (!monthlyGroups.has(monthKey)) {
monthlyGroups.add(monthKey);
deleteMonthly.run(month, videoId);
upsertMonthly.run(month, videoId, stampMs, stampMs);
}
}
}
/**
* Sessions are pruned after a retention window, but rollups are kept much
* longer the remote's older rollup history can't be reconstructed from
* merged sessions. Copy remote rollup rows for groups where the local DB has
* neither a rollup row nor any sessions (i.e. history only the remote knows).
* Groups both machines have data for are never summed, to avoid
* double-counting sessions that earlier syncs already shared.
*/
export function copyRemoteOnlyRollups(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
summary: SyncMergeSummary,
): void {
if (!tableExists(remote, 'imm_daily_rollups') || !tableExists(local, 'imm_daily_rollups')) return;
const localDailyExists = local.prepare(
'SELECT 1 FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ? LIMIT 1',
);
const localDaySessions = local.prepare(
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_DAY_EXPR} = ? LIMIT 1`,
);
const localMonthSessions = local.prepare(
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_MONTH_EXPR} = ? LIMIT 1`,
);
const localMonthSessionsForDay = local.prepare(
`SELECT 1 FROM imm_sessions
WHERE video_id = ?
AND ${LOCAL_MONTH_EXPR} = CAST(strftime('%Y%m', CAST(? AS INTEGER) * 86400, 'unixepoch', 'localtime') AS INTEGER)
LIMIT 1`,
);
const insertDaily = local.prepare(
`INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
);
for (const row of remote.query<SqlRow>('SELECT * FROM imm_daily_rollups').all()) {
if (row.video_id === null) continue;
const localVideoId = videoIdMap.get(Number(row.video_id));
if (localVideoId === undefined) continue;
if (localDailyExists.get(row.rollup_day, localVideoId)) continue;
if (localDaySessions.get(localVideoId, row.rollup_day)) continue;
if (localMonthSessionsForDay.get(localVideoId, row.rollup_day)) continue;
insertDaily.run(
row.rollup_day,
localVideoId,
row.total_sessions,
row.total_active_min,
row.total_lines_seen,
row.total_tokens_seen,
row.total_cards,
row.cards_per_hour,
row.tokens_per_min,
row.lookup_hit_rate,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.dailyRollupsCopied += 1;
}
const localMonthlyExists = local.prepare(
'SELECT 1 FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ? LIMIT 1',
);
const insertMonthly = local.prepare(
`INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
);
for (const row of remote.query<SqlRow>('SELECT * FROM imm_monthly_rollups').all()) {
if (row.video_id === null) continue;
const localVideoId = videoIdMap.get(Number(row.video_id));
if (localVideoId === undefined) continue;
if (localMonthlyExists.get(row.rollup_month, localVideoId)) continue;
if (localMonthSessions.get(localVideoId, row.rollup_month)) continue;
insertMonthly.run(
row.rollup_month,
localVideoId,
row.total_sessions,
row.total_active_min,
row.total_lines_seen,
row.total_tokens_seen,
row.total_cards,
row.CREATED_DATE,
row.LAST_UPDATE_DATE,
);
summary.monthlyRollupsCopied += 1;
}
}
-465
View File
@@ -1,465 +0,0 @@
import { Database } from 'bun:sqlite';
import type { LexiconResolver } from './merge-catalog.js';
import { insertRow, nowDbTimestamp, type SyncMergeSummary } from './sync-shared.js';
const SESSION_COPY_COLUMNS = [
'session_uuid',
'started_at_ms',
'ended_at_ms',
'status',
'locale_id',
'target_lang_id',
'difficulty_tier',
'subtitle_mode',
'ended_media_ms',
'total_watched_ms',
'active_watched_ms',
'lines_seen',
'tokens_seen',
'cards_mined',
'lookup_count',
'lookup_hits',
'yomitan_lookup_count',
'pause_count',
'pause_ms',
'seek_forward_count',
'seek_backward_count',
'media_buffer_events',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const TELEMETRY_COPY_COLUMNS = [
'sample_ms',
'total_watched_ms',
'active_watched_ms',
'lines_seen',
'tokens_seen',
'cards_mined',
'lookup_count',
'lookup_hits',
'yomitan_lookup_count',
'pause_count',
'pause_ms',
'seek_forward_count',
'seek_backward_count',
'media_buffer_events',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const EVENT_COPY_COLUMNS = [
'ts_ms',
'event_type',
'line_index',
'segment_start_ms',
'segment_end_ms',
'tokens_delta',
'cards_delta',
'payload_json',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
const LINE_COPY_COLUMNS = [
'line_index',
'segment_start_ms',
'segment_end_ms',
'text',
'secondary_text',
'CREATED_DATE',
'LAST_UPDATE_DATE',
] as const;
type SqlRow = Record<string, unknown>;
export interface SessionMergeResult {
newSessionIds: number[];
}
export function mergeSessions(
local: Database,
remote: Database,
videoIdMap: Map<number, number>,
animeIdMap: Map<number, number>,
lexicon: LexiconResolver,
summary: SyncMergeSummary,
): SessionMergeResult {
const newSessionIds: number[] = [];
const uuidExists = local.prepare<SqlRow>(
'SELECT session_id FROM imm_sessions WHERE session_uuid = ?',
);
const remoteSessions = remote
.query<SqlRow>(
`SELECT session_id, video_id, ${SESSION_COPY_COLUMNS.join(', ')}
FROM imm_sessions
ORDER BY CAST(started_at_ms AS REAL) ASC, session_id ASC`,
)
.all();
for (const session of remoteSessions) {
if (session.ended_at_ms === null) {
// Stale ACTIVE sessions are finalized by the app on its next startup;
// they will sync once they carry final numbers.
summary.activeSessionsSkipped += 1;
continue;
}
if (uuidExists.get(session.session_uuid)) {
summary.sessionsAlreadyPresent += 1;
continue;
}
const localVideoId = videoIdMap.get(Number(session.video_id));
if (localVideoId === undefined) {
throw new Error(`Snapshot session ${String(session.session_uuid)} references missing video row`);
}
const localSessionId = insertRow(
local,
'imm_sessions',
['video_id', ...SESSION_COPY_COLUMNS],
[localVideoId, ...SESSION_COPY_COLUMNS.map((column) => session[column])],
);
newSessionIds.push(localSessionId);
summary.sessionsMerged += 1;
const remoteSessionId = Number(session.session_id);
copyTelemetry(local, remote, remoteSessionId, localSessionId, summary);
const eventIdMap = copyEvents(local, remote, remoteSessionId, localSessionId, summary);
copySubtitleLines(
local,
remote,
remoteSessionId,
localSessionId,
localVideoId,
animeIdMap,
eventIdMap,
lexicon,
summary,
);
applyMergedSessionLifetime(local, localSessionId, localVideoId, session);
}
return { newSessionIds };
}
function copyTelemetry(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
summary: SyncMergeSummary,
): void {
const rows = remote
.query<SqlRow>(
`SELECT ${TELEMETRY_COPY_COLUMNS.join(', ')} FROM imm_session_telemetry
WHERE session_id = ? ORDER BY telemetry_id ASC`,
)
.all(remoteSessionId);
for (const row of rows) {
insertRow(
local,
'imm_session_telemetry',
['session_id', ...TELEMETRY_COPY_COLUMNS],
[localSessionId, ...TELEMETRY_COPY_COLUMNS.map((column) => row[column])],
);
summary.telemetryRowsAdded += 1;
}
}
function copyEvents(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
summary: SyncMergeSummary,
): Map<number, number> {
const eventIdMap = new Map<number, number>();
const rows = remote
.query<SqlRow>(
`SELECT event_id, ${EVENT_COPY_COLUMNS.join(', ')} FROM imm_session_events
WHERE session_id = ? ORDER BY event_id ASC`,
)
.all(remoteSessionId);
for (const row of rows) {
const localEventId = insertRow(
local,
'imm_session_events',
['session_id', ...EVENT_COPY_COLUMNS],
[localSessionId, ...EVENT_COPY_COLUMNS.map((column) => row[column])],
);
eventIdMap.set(Number(row.event_id), localEventId);
summary.eventsAdded += 1;
}
return eventIdMap;
}
function copySubtitleLines(
local: Database,
remote: Database,
remoteSessionId: number,
localSessionId: number,
localVideoId: number,
animeIdMap: Map<number, number>,
eventIdMap: Map<number, number>,
lexicon: LexiconResolver,
summary: SyncMergeSummary,
): void {
const rows = remote
.query<SqlRow>(
`SELECT line_id, event_id, anime_id, ${LINE_COPY_COLUMNS.join(', ')} FROM imm_subtitle_lines
WHERE session_id = ? ORDER BY line_id ASC`,
)
.all(remoteSessionId);
const wordOccurrences = remote.prepare<SqlRow>(
'SELECT word_id, occurrence_count FROM imm_word_line_occurrences WHERE line_id = ?',
);
const kanjiOccurrences = remote.prepare<SqlRow>(
'SELECT kanji_id, occurrence_count FROM imm_kanji_line_occurrences WHERE line_id = ?',
);
const insertWordOccurrence = local.prepare(
`INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)
ON CONFLICT(line_id, word_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
);
const insertKanjiOccurrence = local.prepare(
`INSERT INTO imm_kanji_line_occurrences (line_id, kanji_id, occurrence_count) VALUES (?, ?, ?)
ON CONFLICT(line_id, kanji_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
);
for (const row of rows) {
const localEventId = row.event_id === null ? null : (eventIdMap.get(Number(row.event_id)) ?? null);
const localAnimeId = row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
const localLineId = insertRow(
local,
'imm_subtitle_lines',
['session_id', 'event_id', 'video_id', 'anime_id', ...LINE_COPY_COLUMNS],
[
localSessionId,
localEventId,
localVideoId,
localAnimeId,
...LINE_COPY_COLUMNS.map((column) => row[column]),
],
);
summary.subtitleLinesAdded += 1;
for (const occurrence of wordOccurrences.all(row.line_id)) {
const localWordId = lexicon.resolveWord(Number(occurrence.word_id));
const count = Number(occurrence.occurrence_count);
insertWordOccurrence.run(localLineId, localWordId, count);
lexicon.addWordOccurrences(Number(occurrence.word_id), count);
}
for (const occurrence of kanjiOccurrences.all(row.line_id)) {
const localKanjiId = lexicon.resolveKanji(Number(occurrence.kanji_id));
const count = Number(occurrence.occurrence_count);
insertKanjiOccurrence.run(localLineId, localKanjiId, count);
lexicon.addKanjiOccurrences(Number(occurrence.kanji_id), count);
}
}
}
/**
* Port of applySessionLifetimeSummary (src/core/services/immersion-tracker/
* lifetime.ts) for sessions arriving out of chronological order. The
* "first session of the day / for this video" checks are order-independent
* here (any other session counts, not just earlier ones): the local machine
* already credited active_days/episodes_started when its own session was
* applied, even if the merged session started earlier that day.
*/
function applyMergedSessionLifetime(
local: Database,
sessionId: number,
videoId: number,
session: SqlRow,
): void {
const updatedAtMs = nowDbTimestamp();
const applied = local
.prepare(
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)
ON CONFLICT(session_id) DO NOTHING`,
)
.run(sessionId, session.ended_at_ms, updatedAtMs, updatedAtMs);
if (applied.changes <= 0) return;
const telemetry = local
.query<SqlRow>(
`SELECT active_watched_ms, cards_mined, lines_seen, tokens_seen
FROM imm_session_telemetry
WHERE session_id = ?
ORDER BY sample_ms DESC, telemetry_id DESC
LIMIT 1`,
)
.get(sessionId);
const metric = (telemetryValue: unknown, sessionValue: unknown): number => {
const fromTelemetry = telemetry ? Number(telemetryValue) : Number.NaN;
const value = Number.isFinite(fromTelemetry) ? fromTelemetry : Number(sessionValue);
return Math.max(0, Math.floor(Number.isFinite(value) ? value : 0));
};
const activeMs = metric(telemetry?.active_watched_ms, session.active_watched_ms);
const cardsMined = metric(telemetry?.cards_mined, session.cards_mined);
const linesSeen = metric(telemetry?.lines_seen, session.lines_seen);
const tokensSeen = metric(telemetry?.tokens_seen, session.tokens_seen);
const video = local
.query<SqlRow>('SELECT anime_id, watched FROM imm_videos WHERE video_id = ?')
.get(videoId);
const watched = Number(video?.watched ?? 0);
const animeId = video?.anime_id === null || video?.anime_id === undefined ? null : Number(video.anime_id);
const mediaLifetime = local
.query<SqlRow>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
.get(videoId);
const hasOtherSessionForVideo = Boolean(
local
.query('SELECT 1 FROM imm_sessions WHERE video_id = ? AND session_id != ? LIMIT 1')
.get(videoId, sessionId),
);
const isFirstSessionForVideoRun = !mediaLifetime && !hasOtherSessionForVideo;
const isFirstCompletedSessionForVideoRun = watched > 0 && Number(mediaLifetime?.completed ?? 0) <= 0;
const hasOtherSessionOnDay = Boolean(
local
.query(
`SELECT 1 FROM imm_sessions
WHERE session_id != ?
AND CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
= CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
LIMIT 1`,
)
.get(sessionId, session.started_at_ms),
);
let animeCompletedDelta = 0;
if (animeId !== null && watched > 0 && isFirstCompletedSessionForVideoRun) {
const animeLifetime = local
.query<SqlRow>('SELECT episodes_completed FROM imm_lifetime_anime WHERE anime_id = ?')
.get(animeId);
const anime = local
.query<SqlRow>('SELECT episodes_total FROM imm_anime WHERE anime_id = ?')
.get(animeId);
const episodesCompletedBefore = Number(animeLifetime?.episodes_completed ?? 0);
const episodesTotal = anime?.episodes_total === null || anime?.episodes_total === undefined
? null
: Number(anime.episodes_total);
if (
episodesTotal !== null &&
episodesTotal > 0 &&
episodesCompletedBefore < episodesTotal &&
episodesCompletedBefore + 1 >= episodesTotal
) {
animeCompletedDelta = 1;
}
}
local
.prepare(
`UPDATE imm_lifetime_global
SET total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + ?,
total_cards = total_cards + ?,
active_days = active_days + ?,
episodes_started = episodes_started + ?,
episodes_completed = episodes_completed + ?,
anime_completed = anime_completed + ?,
LAST_UPDATE_DATE = ?
WHERE global_id = 1`,
)
.run(
activeMs,
cardsMined,
hasOtherSessionOnDay ? 0 : 1,
isFirstSessionForVideoRun ? 1 : 0,
isFirstCompletedSessionForVideoRun ? 1 : 0,
animeCompletedDelta,
updatedAtMs,
);
local
.prepare(
`INSERT INTO imm_lifetime_media(
video_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(video_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
completed = MAX(completed, excluded.completed),
first_watched_ms = CASE
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
ELSE first_watched_ms
END,
last_watched_ms = CASE
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
ELSE last_watched_ms
END,
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
)
.run(
videoId,
activeMs,
cardsMined,
linesSeen,
tokensSeen,
watched > 0 ? 1 : 0,
session.started_at_ms,
session.ended_at_ms,
updatedAtMs,
updatedAtMs,
);
if (animeId !== null) {
local
.prepare(
`INSERT INTO imm_lifetime_anime(
anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
total_tokens_seen, episodes_started, episodes_completed, first_watched_ms,
last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(anime_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
episodes_started = episodes_started + excluded.episodes_started,
episodes_completed = episodes_completed + excluded.episodes_completed,
first_watched_ms = CASE
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
ELSE first_watched_ms
END,
last_watched_ms = CASE
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
ELSE last_watched_ms
END,
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
)
.run(
animeId,
activeMs,
cardsMined,
linesSeen,
tokensSeen,
isFirstSessionForVideoRun ? 1 : 0,
isFirstCompletedSessionForVideoRun ? 1 : 0,
session.started_at_ms,
session.ended_at_ms,
updatedAtMs,
updatedAtMs,
);
}
}
-51
View File
@@ -1,51 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { assertSafeSshHost, resolveRemoteSubminerCommand, runScp, shellQuote } from './ssh.js';
test('assertSafeSshHost rejects option-like hosts', () => {
assert.throws(() => assertSafeSshHost('-oProxyCommand=touch pwned'), /looks like an option/);
assert.throws(() => assertSafeSshHost('-lroot'), /looks like an option/);
});
test('assertSafeSshHost accepts normal destinations', () => {
assert.doesNotThrow(() => assertSafeSshHost('macbook'));
assert.doesNotThrow(() => assertSafeSshHost('user@192.168.1.20'));
assert.doesNotThrow(() => assertSafeSshHost('ssh-alias'));
});
test('shellQuote escapes single quotes and wraps in quotes', () => {
assert.equal(shellQuote('subminer'), `'subminer'`);
assert.equal(shellQuote(`a'; rm -rf ~; '`), `'a'\\''; rm -rf ~; '\\'''`);
});
test('runScp rejects option-like local endpoints before spawning scp', () => {
assert.throws(() => runScp('-oProxyCommand=sh', '/tmp/out.sqlite'), /looks like an option/);
assert.throws(() => runScp('/tmp/in.sqlite', '-bad-destination'), /looks like an option/);
});
test('runScp rejects option-like remote host components', () => {
assert.throws(
() => runScp('-oProxyCommand=sh:/tmp/in.sqlite', '/tmp/out.sqlite'),
/SSH host that looks like an option/,
);
});
test('resolveRemoteSubminerCommand verifies the launcher under the remote runtime PATH', () => {
const calls: Array<{ host: string; remoteCommand: string }> = [];
const command = resolveRemoteSubminerCommand('macbook', null, (host, remoteCommand) => {
calls.push({ host, remoteCommand });
return { status: 0, stdout: '', stderr: '' };
});
assert.equal(
command,
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer',
);
assert.deepEqual(calls, [
{
host: 'macbook',
remoteCommand:
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer --help >/dev/null 2>&1',
},
]);
});
-106
View File
@@ -1,106 +0,0 @@
import { spawnSync } from 'node:child_process';
export interface RemoteRunResult {
status: number;
stdout: string;
stderr: string;
}
/**
* ssh/scp have no `--` terminator for the destination, so a host that starts
* with `-` (e.g. `-oProxyCommand=...`) is parsed as an option. Reject those
* before spawning.
*/
export function assertSafeSshHost(host: string): void {
if (host.startsWith('-')) {
throw new Error(`Refusing to use SSH host that looks like an option: ${host}`);
}
}
/**
* Run a command on the SSH host. stdin stays attached so interactive prompts
* can still read from the terminal; stdout/stderr are captured for callers
* that need actionable remote failure messages.
*/
export function runSsh(host: string, remoteCommand: string): RemoteRunResult {
assertSafeSshHost(host);
const result = spawnSync('ssh', [host, remoteCommand], {
encoding: 'utf8',
stdio: ['inherit', 'pipe', 'pipe'],
});
if (result.error) {
throw new Error(`Failed to run ssh: ${(result.error as Error).message}`);
}
return { status: result.status ?? 1, stdout: result.stdout ?? '', stderr: result.stderr ?? '' };
}
function assertSafeScpEndpoint(endpoint: string): void {
const colon = endpoint.indexOf(':');
const slash = endpoint.indexOf('/');
if (colon <= 0 || (slash !== -1 && slash < colon)) {
if (endpoint.startsWith('-')) {
throw new Error(`Refusing to use scp endpoint that looks like an option: ${endpoint}`);
}
return;
}
const host = endpoint.slice(0, colon);
const remotePath = endpoint.slice(colon + 1);
assertSafeSshHost(host);
if (remotePath.startsWith('-')) {
throw new Error(`Refusing to use scp remote path that looks like an option: ${remotePath}`);
}
}
export function runScp(from: string, to: string): void {
assertSafeScpEndpoint(from);
assertSafeScpEndpoint(to);
const result = spawnSync('scp', ['-q', from, to], {
encoding: 'utf8',
stdio: ['inherit', 'inherit', 'inherit'],
});
if (result.error) {
throw new Error(`Failed to run scp: ${(result.error as Error).message}`);
}
if ((result.status ?? 1) !== 0) {
throw new Error(`scp failed copying ${from} -> ${to}`);
}
}
export function shellQuote(value: string): string {
return `'${value.replaceAll("'", `'\\''`)}'`;
}
const REMOTE_RUNTIME_PATH =
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"';
/**
* Non-interactive SSH shells often miss user-installed launchers and Bun.
* Probe the launcher under the same deterministic PATH used by sync itself.
*/
export function resolveRemoteSubminerCommand(
host: string,
preferred: string | null,
runRemote: typeof runSsh = runSsh,
): string {
// Trusted defaults stay unquoted so the remote shell expands `~`; a
// user-supplied override is shell-quoted to prevent command injection.
const candidates: Array<{ value: string; invocation: string }> = preferred
? [{ value: preferred, invocation: shellQuote(preferred) }]
: [
{ value: 'subminer', invocation: 'subminer' },
{ value: '~/.local/bin/subminer', invocation: '~/.local/bin/subminer' },
];
for (const candidate of candidates) {
const command = `${REMOTE_RUNTIME_PATH} ${candidate.invocation}`;
const probe = runRemote(host, `${command} --help >/dev/null 2>&1`);
if (probe.status === 0) {
return command;
}
}
throw new Error(
preferred
? `Remote command not found on ${host}: ${preferred}`
: `subminer not found on ${host} (tried PATH and ~/.local/bin/subminer). Pass --remote-cmd <path>.`,
);
}
-598
View File
@@ -1,598 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { createDbSnapshot, mergeSnapshotIntoDb } from './sync-db.js';
import {
createImmersionDbFixture,
insertFixtureSession,
} from '../test-support/immersion-db-fixture.js';
const DAY_MS = 86_400_000;
const BASE_MS = Date.UTC(2026, 5, 1, 12, 0, 0);
function makeTmpDir(): string {
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-test-'));
}
function makeDbPair(): { dir: string; localPath: string; remotePath: string } {
const dir = makeTmpDir();
const localPath = path.join(dir, 'local.sqlite');
const remotePath = path.join(dir, 'remote.sqlite');
createImmersionDbFixture(localPath);
createImmersionDbFixture(remotePath);
return { dir, localPath, remotePath };
}
function queryOne<T extends Record<string, unknown>>(
dbPath: string,
sql: string,
params: unknown[] = [],
): T | undefined {
const db = new Database(dbPath, { readonly: true });
try {
return db.query<T>(sql).get(...params) as T | undefined;
} finally {
db.close();
}
}
function count(dbPath: string, sql: string, params: unknown[] = []): number {
return Number(queryOne<{ n: number }>(dbPath, sql, params)?.n ?? 0);
}
function withWritableDb<T>(dbPath: string, fn: (db: Database) => T): T {
const db = new Database(dbPath, { readwrite: true });
try {
return fn(db);
} finally {
db.close();
}
}
test('merges remote-only sessions with catalog, lifetime, and rollups', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 2 }],
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
animeTitleKey: 'showb',
startedAtMs: BASE_MS + DAY_MS,
activeWatchedMs: 900_000,
cardsMined: 3,
applyLifetime: true,
words: [
{ headword: '見る', word: '見た', reading: 'みた', count: 5 },
{ headword: '食べる', word: '食べた', reading: 'たべた', count: 1 },
],
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 1);
assert.equal(summary.animeAdded, 1);
assert.equal(summary.videosAdded, 1);
assert.equal(summary.wordsAdded, 1);
assert.equal(summary.subtitleLinesAdded, 2);
assert.equal(summary.telemetryRowsAdded, 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 2);
const global = queryOne<{
total_sessions: number;
total_active_ms: number;
total_cards: number;
active_days: number;
episodes_started: number;
}>(
localPath,
'SELECT total_sessions, total_active_ms, total_cards, active_days, episodes_started FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.total_active_ms, 1_200_000 + 900_000);
assert.equal(global?.total_cards, 2 + 3);
assert.equal(global?.active_days, 2);
assert.equal(global?.episodes_started, 2);
// Existing word: local 2 + merged 5; new word carries remote frequency.
const sharedWord = queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '見た'`,
);
assert.equal(sharedWord?.frequency, 7);
const newWord = queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '食べた'`,
);
assert.equal(newWord?.frequency, 1);
// The merged session's rollup group was recomputed.
assert.equal(summary.rollupGroupsRecomputed, 1);
const mergedVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'showb-e1'`,
)?.video_id,
);
assert.equal(
count(localPath, 'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE video_id = ?', [
mergedVideoId,
]),
1,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('is idempotent: re-merging the same snapshot changes nothing', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 4 }],
});
mergeSnapshotIntoDb(localPath, remotePath);
const globalAfterFirst = queryOne<Record<string, unknown>>(
localPath,
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
);
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 0);
assert.equal(summary.sessionsAlreadyPresent, 1);
assert.equal(summary.wordsAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_subtitle_lines'), 1);
const globalAfterSecond = queryOne<Record<string, unknown>>(
localPath,
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
);
assert.deepEqual(
{ ...globalAfterSecond, LAST_UPDATE_DATE: null },
{ ...globalAfterFirst, LAST_UPDATE_DATE: null },
);
assert.equal(
Number(
queryOne<{ frequency: number }>(
localPath,
`SELECT frequency FROM imm_words WHERE word = '見た'`,
)?.frequency,
),
4,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('preserves anilist_id when inserting a new anime from the snapshot', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
animeTitleKey: 'showb',
startedAtMs: BASE_MS,
applyLifetime: true,
});
withWritableDb(remotePath, (remoteDb) => {
remoteDb
.prepare(`UPDATE imm_anime SET anilist_id = 12345 WHERE normalized_title_key = 'showb'`)
.run();
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.animeAdded, 1);
const anime = queryOne<{ anilist_id: number }>(
localPath,
`SELECT anilist_id FROM imm_anime WHERE normalized_title_key = 'showb'`,
);
assert.equal(anime?.anilist_id, 12345);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('matches an existing local anime by anilist_id even when the title key differs', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'show-romaji',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e2',
animeTitleKey: 'show-native',
startedAtMs: BASE_MS + DAY_MS,
applyLifetime: true,
});
withWritableDb(localPath, (localDb) => {
localDb
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-romaji'`)
.run();
});
withWritableDb(remotePath, (remoteDb) => {
remoteDb
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-native'`)
.run();
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
// Same anilist_id → one anime, not two.
assert.equal(summary.animeAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_anime'), 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('matches shared videos by video_key and merges the watched flag', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS + DAY_MS,
watched: true,
applyLifetime: true,
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.videosAdded, 0);
assert.equal(summary.animeAdded, 0);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_videos'), 1);
const video = queryOne<{ watched: number }>(
localPath,
`SELECT watched FROM imm_videos WHERE video_key = 'showa-e1'`,
);
assert.equal(video?.watched, 1);
// Both sessions now credit the same video; episode was started once and
// completed once (by the remote session that watched it to the end).
const global = queryOne<{
episodes_started: number;
episodes_completed: number;
total_sessions: number;
}>(
localPath,
'SELECT episodes_started, episodes_completed, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.episodes_started, 1);
assert.equal(global?.episodes_completed, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('backfills media metadata for matched videos when local metadata is absent', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
animeTitleKey: 'showa',
startedAtMs: BASE_MS + DAY_MS,
applyLifetime: true,
});
const remoteDb = new Database(remotePath, { readwrite: true });
try {
const remoteVideoId = Number(
remoteDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`)
.get()?.video_id,
);
remoteDb
.prepare(
`INSERT INTO imm_media_art (video_id, anilist_id, cover_url, title_romaji, fetched_at_ms)
VALUES (?, 123, 'https://example.test/cover.jpg', 'Show A', ?)`,
)
.run(remoteVideoId, String(BASE_MS));
remoteDb
.prepare(
`INSERT INTO imm_youtube_videos (video_id, youtube_video_id, video_url, video_title, fetched_at_ms)
VALUES (?, 'yt-1', 'https://youtube.test/watch?v=yt-1', 'Remote Video', ?)`,
)
.run(remoteVideoId, String(BASE_MS));
} finally {
remoteDb.close();
}
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.videosAdded, 0);
const localVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`,
)?.video_id,
);
const art = queryOne<{ cover_url: string; title_romaji: string }>(
localPath,
'SELECT cover_url, title_romaji FROM imm_media_art WHERE video_id = ?',
[localVideoId],
);
assert.equal(art?.cover_url, 'https://example.test/cover.jpg');
assert.equal(art?.title_romaji, 'Show A');
const youtube = queryOne<{ youtube_video_id: string; video_title: string }>(
localPath,
'SELECT youtube_video_id, video_title FROM imm_youtube_videos WHERE video_id = ?',
[localVideoId],
);
assert.equal(youtube?.youtube_video_id, 'yt-1');
assert.equal(youtube?.video_title, 'Remote Video');
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('does not double-count active_days when a merged session starts earlier on an already-credited day', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'showa-e1',
startedAtMs: BASE_MS + 3_600_000,
applyLifetime: true,
});
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showb-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
mergeSnapshotIntoDb(localPath, remotePath);
const global = queryOne<{ active_days: number; total_sessions: number }>(
localPath,
'SELECT active_days, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
);
assert.equal(global?.total_sessions, 2);
assert.equal(global?.active_days, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('skips unfinished sessions', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-active',
videoKey: 'showa-e1',
startedAtMs: BASE_MS,
endedAtMs: null,
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.sessionsMerged, 0);
assert.equal(summary.activeSessionsSkipped, 1);
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 0);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('copies remote-only historical rollups but never sums shared groups', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
// Remote has a video plus an old rollup row whose sessions were pruned.
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'old-show-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
withWritableDb(remotePath, (remoteDb) => {
const remoteVideoId = Number(
remoteDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
.get()?.video_id,
);
remoteDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (10000, ?, 4, 120.5, 400, 3200, 9)`,
)
.run(remoteVideoId);
});
// Local already has its own rollup row for a shared group.
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'old-show-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const localVideoId = withWritableDb(localPath, (localDb) => {
const videoId = Number(
localDb
.query<{
video_id: number;
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
.get()?.video_id,
);
localDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (10000, ?, 2, 60.0, 200, 1600, 4)`,
)
.run(videoId);
return videoId;
});
const summary = mergeSnapshotIntoDb(localPath, remotePath);
// Shared group (day 10000) untouched; the remote-only session's own group
// was recomputed, not copied.
assert.equal(summary.dailyRollupsCopied, 0);
const shared = queryOne<{ total_sessions: number }>(
localPath,
'SELECT total_sessions FROM imm_daily_rollups WHERE rollup_day = 10000 AND video_id = ?',
[localVideoId],
);
assert.equal(shared?.total_sessions, 2);
// Now a rollup for a video with no local sessions at all gets copied.
withWritableDb(remotePath, (remoteDb2) => {
const orphanVideoId = Number(
remoteDb2
.prepare(
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
VALUES ('pruned-show-e1', 'pruned-show-e1', 1, 1, 1440000)`,
)
.run().lastInsertRowid,
);
remoteDb2
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (9000, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(orphanVideoId);
});
const summary2 = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary2.dailyRollupsCopied, 1);
const copiedVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'pruned-show-e1'`,
)?.video_id,
);
const copied = queryOne<{ total_sessions: number; total_active_min: number }>(
localPath,
'SELECT total_sessions, total_active_min FROM imm_daily_rollups WHERE rollup_day = 9000 AND video_id = ?',
[copiedVideoId],
);
assert.equal(copied?.total_sessions, 3);
assert.equal(copied?.total_active_min, 90.0);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('does not copy remote-only daily rollups into months with local sessions', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(localPath, {
uuid: 'local-1',
videoKey: 'mixed-month-e1',
animeTitleKey: 'mixed-month',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const remoteDb = new Database(remotePath, { readwrite: true });
try {
const remoteVideoId = Number(
remoteDb
.prepare(
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
VALUES ('mixed-month-e1', 'mixed-month-e1', 1, 1, 1440000)`,
)
.run().lastInsertRowid,
);
remoteDb
.prepare(
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (20615, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(remoteVideoId);
remoteDb
.prepare(
`INSERT INTO imm_monthly_rollups (rollup_month, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
VALUES (202606, ?, 3, 90.0, 300, 2400, 6)`,
)
.run(remoteVideoId);
} finally {
remoteDb.close();
}
const summary = mergeSnapshotIntoDb(localPath, remotePath);
assert.equal(summary.dailyRollupsCopied, 0);
assert.equal(summary.monthlyRollupsCopied, 0);
const localVideoId = Number(
queryOne<{ video_id: number }>(
localPath,
`SELECT video_id FROM imm_videos WHERE video_key = 'mixed-month-e1'`,
)?.video_id,
);
assert.equal(
count(
localPath,
'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE rollup_day = 20615 AND video_id = ?',
[localVideoId],
),
0,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('rejects snapshots at a different schema version', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
withWritableDb(remotePath, (remoteDb) => {
remoteDb.prepare('UPDATE imm_schema_version SET schema_version = 17').run();
});
assert.throws(() => mergeSnapshotIntoDb(localPath, remotePath), /schema version 17/);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('createDbSnapshot produces a mergeable copy', () => {
const { dir, localPath, remotePath } = makeDbPair();
try {
insertFixtureSession(remotePath, {
uuid: 'remote-1',
videoKey: 'showa-e1',
startedAtMs: BASE_MS,
applyLifetime: true,
});
const snapshotPath = path.join(dir, 'snapshot.sqlite');
createDbSnapshot(remotePath, snapshotPath);
assert.ok(fs.existsSync(snapshotPath));
const summary = mergeSnapshotIntoDb(localPath, snapshotPath);
assert.equal(summary.sessionsMerged, 1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
-105
View File
@@ -1,105 +0,0 @@
import fs from 'node:fs';
import { Database } from 'bun:sqlite';
import {
LexiconResolver,
mergeAnime,
mergeExcludedWords,
mergeMediaMetadata,
mergeVideos,
} from './merge-catalog.js';
import { mergeSessions } from './merge-sessions.js';
import { copyRemoteOnlyRollups, refreshRollupsForNewSessions } from './merge-rollups.js';
import {
assertMergeableSchema,
createEmptyMergeSummary,
type SyncMergeSummary,
} from './sync-shared.js';
export type { SyncMergeSummary } from './sync-shared.js';
export { createDbSnapshot, findLiveStatsDaemonPid } from './sync-shared.js';
/**
* Merge a snapshot of another machine's immersion database into the local
* one. Insert-only union keyed on natural keys (session_uuid, video_key,
* normalized_title_key, word/kanji identity); lifetime and rollup aggregates
* are updated incrementally so history older than the session retention
* window is preserved on both sides. Idempotent: re-merging the same
* snapshot is a no-op.
*/
export function mergeSnapshotIntoDb(localDbPath: string, snapshotPath: string): SyncMergeSummary {
if (!fs.existsSync(localDbPath)) {
throw new Error(`Local stats database not found: ${localDbPath}`);
}
if (!fs.existsSync(snapshotPath)) {
throw new Error(`Snapshot database not found: ${snapshotPath}`);
}
const remote = new Database(snapshotPath, { readonly: true });
let local: Database;
try {
local = new Database(localDbPath, { readwrite: true, create: false });
} catch (error) {
remote.close();
throw error;
}
try {
assertMergeableSchema(remote, 'Snapshot');
assertMergeableSchema(local, 'Local');
const summary = createEmptyMergeSummary();
local.run('PRAGMA foreign_keys = ON');
local.run('PRAGMA busy_timeout = 5000');
local.run('BEGIN IMMEDIATE');
try {
const animeIdMap = mergeAnime(local, remote, summary);
const { videoIdMap, addedVideoIds } = mergeVideos(local, remote, animeIdMap, summary);
mergeMediaMetadata(local, remote, videoIdMap, addedVideoIds);
mergeExcludedWords(local, remote, summary);
const lexicon = new LexiconResolver(local, remote, summary);
const { newSessionIds } = mergeSessions(
local,
remote,
videoIdMap,
animeIdMap,
lexicon,
summary,
);
lexicon.applyFrequencyDeltas();
refreshRollupsForNewSessions(local, newSessionIds, summary);
copyRemoteOnlyRollups(local, remote, videoIdMap, summary);
local.run('COMMIT');
return summary;
} catch (error) {
local.run('ROLLBACK');
throw error;
}
} finally {
local.close();
remote.close();
}
}
export function formatMergeSummary(summary: SyncMergeSummary): string {
const lines = [
`Sessions merged: ${summary.sessionsMerged} (${summary.sessionsAlreadyPresent} already present, ${summary.activeSessionsSkipped} unfinished skipped)`,
];
const detail: string[] = [];
if (summary.animeAdded) detail.push(`${summary.animeAdded} series`);
if (summary.videosAdded) detail.push(`${summary.videosAdded} videos`);
if (summary.wordsAdded) detail.push(`${summary.wordsAdded} words`);
if (summary.kanjiAdded) detail.push(`${summary.kanjiAdded} kanji`);
if (summary.subtitleLinesAdded) detail.push(`${summary.subtitleLinesAdded} subtitle lines`);
if (summary.excludedWordsAdded) detail.push(`${summary.excludedWordsAdded} excluded words`);
if (detail.length > 0) lines.push(`Added: ${detail.join(', ')}`);
if (summary.dailyRollupsCopied || summary.monthlyRollupsCopied) {
lines.push(
`Historical rollups copied: ${summary.dailyRollupsCopied} daily, ${summary.monthlyRollupsCopied} monthly`,
);
}
if (summary.rollupGroupsRecomputed) {
lines.push(`Rollup groups recomputed: ${summary.rollupGroupsRecomputed}`);
}
return lines.join('\n');
}
-173
View File
@@ -1,173 +0,0 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database } from 'bun:sqlite';
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
import { withReadonlyWalRetry } from '../history-db.js';
import { resolveConfigDir } from '../../src/config/path-resolution.js';
export { SCHEMA_VERSION };
export interface SyncMergeSummary {
sessionsMerged: number;
sessionsAlreadyPresent: number;
activeSessionsSkipped: number;
animeAdded: number;
videosAdded: number;
wordsAdded: number;
kanjiAdded: number;
subtitleLinesAdded: number;
telemetryRowsAdded: number;
eventsAdded: number;
excludedWordsAdded: number;
dailyRollupsCopied: number;
monthlyRollupsCopied: number;
rollupGroupsRecomputed: number;
}
export function createEmptyMergeSummary(): SyncMergeSummary {
return {
sessionsMerged: 0,
sessionsAlreadyPresent: 0,
activeSessionsSkipped: 0,
animeAdded: 0,
videosAdded: 0,
wordsAdded: 0,
kanjiAdded: 0,
subtitleLinesAdded: 0,
telemetryRowsAdded: 0,
eventsAdded: 0,
excludedWordsAdded: 0,
dailyRollupsCopied: 0,
monthlyRollupsCopied: 0,
rollupGroupsRecomputed: 0,
};
}
export function nowDbTimestamp(): string {
return String(Date.now());
}
export function tableExists(db: Database, tableName: string): boolean {
return Boolean(
db.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?`).get(tableName),
);
}
export function readSchemaVersion(db: Database): number | null {
if (!tableExists(db, 'imm_schema_version')) return null;
const row = db
.query<{ schema_version: number }>(
'SELECT MAX(schema_version) AS schema_version FROM imm_schema_version',
)
.get();
return typeof row?.schema_version === 'number' ? row.schema_version : null;
}
export function assertMergeableSchema(db: Database, label: string): void {
const version = readSchemaVersion(db);
if (version === null) {
throw new Error(
`${label} database has no schema version. Run SubMiner once on that machine so the stats database is initialized.`,
);
}
if (version !== SCHEMA_VERSION) {
throw new Error(
`${label} database is at schema version ${version} but this launcher expects ${SCHEMA_VERSION}. Update SubMiner on both machines to the same version and run each app once before syncing.`,
);
}
for (const table of ['imm_sessions', 'imm_videos', 'imm_lifetime_global']) {
if (!tableExists(db, table)) {
throw new Error(`${label} database is missing table ${table}; cannot sync.`);
}
}
}
export function insertRow(
db: Database,
table: string,
columns: readonly string[],
values: unknown[],
): number {
const sql = `INSERT INTO ${table} (${columns.join(', ')}) VALUES (${columns.map(() => '?').join(', ')})`;
// db.query() caches the prepared statement per SQL string; this runs once
// per copied row, so re-preparing via db.prepare() would dominate merge time.
const result = db.query(sql).run(...values);
return Number(result.lastInsertRowid);
}
export function createDbSnapshot(dbPath: string, outPath: string): void {
if (!fs.existsSync(dbPath)) {
throw new Error(`Stats database not found: ${dbPath}`);
}
fs.rmSync(outPath, { force: true });
fs.mkdirSync(path.dirname(outPath), { recursive: true });
withReadonlyWalRetry(dbPath, (options) => {
const db = new Database(dbPath, options);
try {
assertMergeableSchema(db, 'Local');
db.prepare('VACUUM INTO ?').run(outPath);
} finally {
db.close();
}
});
}
interface DaemonStateFile {
pid?: unknown;
}
function isProcessAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch (error) {
// EPERM means the process exists but we can't signal it → still alive.
// Only ESRCH (no such process) means it's actually gone.
return (error as NodeJS.ErrnoException)?.code === 'EPERM';
}
}
function statsDaemonStateCandidates(dbPath: string): string[] {
const homeDir = os.homedir();
const candidates = new Set<string>([path.join(path.dirname(dbPath), 'stats-daemon.json')]);
const configDir = resolveConfigDir({
platform: process.platform,
appDataDir: process.env.APPDATA,
xdgConfigHome: process.env.XDG_CONFIG_HOME,
homeDir,
existsSync: fs.existsSync,
});
candidates.add(path.join(configDir, 'stats-daemon.json'));
if (process.platform === 'darwin') {
candidates.add(path.join(homeDir, 'Library', 'Application Support', 'SubMiner', 'stats-daemon.json'));
}
return [...candidates];
}
/**
* Best-effort guard against merging while a SubMiner process holds the
* tracker's write queue in memory. Detects the background stats daemon via
* its pid state file; the interactive app is caught by the mpv-socket check
* in the sync command.
*/
export function findLiveStatsDaemonPid(dbPath: string): number | null {
for (const statePath of statsDaemonStateCandidates(dbPath)) {
let raw: string;
try {
raw = fs.readFileSync(statePath, 'utf8');
} catch {
continue;
}
try {
const parsed = JSON.parse(raw) as DaemonStateFile;
const pid = typeof parsed.pid === 'number' && Number.isInteger(parsed.pid) ? parsed.pid : 0;
if (pid > 0 && isProcessAlive(pid)) {
return pid;
}
} catch {
continue;
}
}
return null;
}
@@ -1,323 +0,0 @@
import { Database } from 'bun:sqlite';
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
import { IMMERSION_DB_FIXTURE_DDL } from './immersion-db-schema.js';
export function createImmersionDbFixture(dbPath: string): void {
const db = new Database(dbPath, { create: true });
try {
db.run('PRAGMA foreign_keys = ON');
db.run('PRAGMA journal_mode = WAL');
for (const statement of IMMERSION_DB_FIXTURE_DDL.split(';')) {
const sql = statement.trim();
if (sql) db.run(sql);
}
db.prepare('INSERT INTO imm_schema_version (schema_version, applied_at_ms) VALUES (?, ?)').run(
SCHEMA_VERSION,
String(Date.now()),
);
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0)`,
).run();
db.prepare(
`INSERT INTO imm_lifetime_global(global_id, CREATED_DATE, LAST_UPDATE_DATE) VALUES (1, ?, ?)`,
).run(String(Date.now()), String(Date.now()));
} finally {
db.close();
}
}
let uniqueCounter = 0;
export interface FixtureSessionInput {
uuid: string;
videoKey: string;
animeTitleKey?: string | null;
animeEpisodesTotal?: number | null;
startedAtMs: number;
endedAtMs?: number | null;
activeWatchedMs?: number;
cardsMined?: number;
linesSeen?: number;
tokensSeen?: number;
watched?: boolean;
words?: Array<{ headword: string; word: string; reading: string; count: number }>;
applyLifetime?: boolean;
}
/**
* Insert a session (with video/anime/subtitle-line/occurrence rows) the way
* the app would have recorded it, optionally crediting lifetime aggregates
* the way applySessionLifetimeSummary does for locally-recorded sessions.
*/
export function insertFixtureSession(dbPath: string, input: FixtureSessionInput): void {
const db = new Database(dbPath, { readwrite: true });
const stamp = String(input.startedAtMs);
try {
db.run('PRAGMA foreign_keys = ON');
let animeId: number | null = null;
if (input.animeTitleKey) {
const existing = db
.query<{
anime_id: number;
}>('SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?')
.get(input.animeTitleKey);
animeId = existing
? existing.anime_id
: Number(
db
.prepare(
`INSERT INTO imm_anime (normalized_title_key, canonical_title, episodes_total, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?)`,
)
.run(
input.animeTitleKey,
input.animeTitleKey,
input.animeEpisodesTotal ?? null,
stamp,
stamp,
).lastInsertRowid,
);
}
const existingVideo = db
.query<{ video_id: number }>('SELECT video_id FROM imm_videos WHERE video_key = ?')
.get(input.videoKey);
const videoId = existingVideo
? existingVideo.video_id
: Number(
db
.prepare(
`INSERT INTO imm_videos (video_key, anime_id, canonical_title, source_type, source_path, watched, duration_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, 1, ?, ?, 1440000, ?, ?)`,
)
.run(
input.videoKey,
animeId,
input.videoKey,
`/videos/${input.videoKey}.mkv`,
input.watched ? 1 : 0,
stamp,
stamp,
).lastInsertRowid,
);
if (input.watched) {
db.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?').run(videoId);
}
const endedAtMs =
input.endedAtMs === undefined ? input.startedAtMs + 1_500_000 : input.endedAtMs;
const activeWatchedMs = input.activeWatchedMs ?? 1_200_000;
const cardsMined = input.cardsMined ?? 2;
const linesSeen = input.linesSeen ?? 100;
const tokensSeen = input.tokensSeen ?? 800;
const sessionId = Number(
db
.prepare(
`INSERT INTO imm_sessions (
session_uuid, video_id, started_at_ms, ended_at_ms, status,
total_watched_ms, active_watched_ms, lines_seen, tokens_seen, cards_mined,
CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run(
input.uuid,
videoId,
String(input.startedAtMs),
endedAtMs === null ? null : String(endedAtMs),
endedAtMs === null ? 1 : 2,
activeWatchedMs,
activeWatchedMs,
linesSeen,
tokensSeen,
cardsMined,
stamp,
stamp,
).lastInsertRowid,
);
db.prepare(
`INSERT INTO imm_session_telemetry (
session_id, sample_ms, total_watched_ms, active_watched_ms, lines_seen, tokens_seen,
cards_mined, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
sessionId,
String(endedAtMs ?? input.startedAtMs),
activeWatchedMs,
activeWatchedMs,
linesSeen,
tokensSeen,
cardsMined,
stamp,
stamp,
);
for (const word of input.words ?? []) {
const existing = db
.query<{
id: number;
}>('SELECT id FROM imm_words WHERE headword = ? AND word = ? AND reading = ?')
.get(word.headword, word.word, word.reading);
const wordId = existing
? existing.id
: Number(
db
.prepare(
`INSERT INTO imm_words (headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 0)`,
)
.run(
word.headword,
word.word,
word.reading,
Math.floor(input.startedAtMs / 1000),
Math.floor(input.startedAtMs / 1000),
).lastInsertRowid,
);
uniqueCounter += 1;
const lineId = Number(
db
.prepare(
`INSERT INTO imm_subtitle_lines (session_id, video_id, anime_id, line_index, text, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
)
.run(
sessionId,
videoId,
animeId,
uniqueCounter,
`line ${word.word}`,
input.startedAtMs,
input.startedAtMs,
).lastInsertRowid,
);
db.prepare(
'INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)',
).run(lineId, wordId, word.count);
db.prepare('UPDATE imm_words SET frequency = frequency + ? WHERE id = ?').run(
word.count,
wordId,
);
}
if (input.applyLifetime && endedAtMs !== null) {
applyFixtureLifetime(db, sessionId, videoId, animeId, {
endedAtMs,
startedAtMs: input.startedAtMs,
activeWatchedMs,
cardsMined,
linesSeen,
tokensSeen,
watched: Boolean(input.watched),
episodesTotal: input.animeEpisodesTotal ?? null,
});
}
} finally {
db.close();
}
}
function applyFixtureLifetime(
db: Database,
sessionId: number,
videoId: number,
animeId: number | null,
data: {
endedAtMs: number;
startedAtMs: number;
activeWatchedMs: number;
cardsMined: number;
linesSeen: number;
tokensSeen: number;
watched: boolean;
episodesTotal: number | null;
},
): void {
const stamp = String(data.endedAtMs);
db.prepare(
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, ?, ?, ?)`,
).run(sessionId, String(data.endedAtMs), stamp, stamp);
const mediaLifetime = db
.query<{ completed: number }>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
.get(videoId);
const isFirstSessionForVideo = !mediaLifetime;
const isFirstCompleted = data.watched && Number(mediaLifetime?.completed ?? 0) <= 0;
const dayExpr = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
const otherOnDay = db
.query(
`SELECT 1 FROM imm_sessions WHERE session_id != ? AND ${dayExpr} =
CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER) LIMIT 1`,
)
.get(sessionId, String(data.startedAtMs));
db.prepare(
`UPDATE imm_lifetime_global SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + ?,
total_cards = total_cards + ?,
active_days = active_days + ?,
episodes_started = episodes_started + ?,
episodes_completed = episodes_completed + ?
WHERE global_id = 1`,
).run(
data.activeWatchedMs,
data.cardsMined,
otherOnDay ? 0 : 1,
isFirstSessionForVideo ? 1 : 0,
isFirstCompleted ? 1 : 0,
);
db.prepare(
`INSERT INTO imm_lifetime_media (video_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(video_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
completed = MAX(completed, excluded.completed),
last_watched_ms = excluded.last_watched_ms`,
).run(
videoId,
data.activeWatchedMs,
data.cardsMined,
data.linesSeen,
data.tokensSeen,
data.watched ? 1 : 0,
String(data.startedAtMs),
String(data.endedAtMs),
stamp,
stamp,
);
if (animeId !== null) {
db.prepare(
`INSERT INTO imm_lifetime_anime (anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, episodes_started, episodes_completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(anime_id) DO UPDATE SET
total_sessions = total_sessions + 1,
total_active_ms = total_active_ms + excluded.total_active_ms,
total_cards = total_cards + excluded.total_cards,
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
episodes_started = episodes_started + excluded.episodes_started,
episodes_completed = episodes_completed + excluded.episodes_completed,
last_watched_ms = excluded.last_watched_ms`,
).run(
animeId,
data.activeWatchedMs,
data.cardsMined,
data.linesSeen,
data.tokensSeen,
isFirstSessionForVideo ? 1 : 0,
isFirstCompleted ? 1 : 0,
String(data.startedAtMs),
String(data.endedAtMs),
stamp,
stamp,
);
}
}
@@ -1,141 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Database as BunDatabase } from 'bun:sqlite';
import { ensureSchema } from '../../src/core/services/immersion-tracker/storage.js';
import { createImmersionDbFixture, insertFixtureSession } from './immersion-db-fixture.js';
type SchemaRow = { type: string; name: string; tbl_name: string; sql: string | null };
const SYNC_SCHEMA_OBJECTS = [
'imm_anime',
'imm_videos',
'imm_sessions',
'imm_session_telemetry',
'imm_session_events',
'imm_daily_rollups',
'imm_monthly_rollups',
'imm_words',
'imm_kanji',
'imm_subtitle_lines',
'imm_word_line_occurrences',
'imm_kanji_line_occurrences',
'imm_media_art',
'imm_youtube_videos',
'imm_cover_art_blobs',
'imm_lifetime_global',
'imm_lifetime_anime',
'imm_lifetime_media',
'imm_lifetime_applied_sessions',
'imm_stats_excluded_words',
'idx_anime_normalized_title',
'idx_anime_anilist_id',
'idx_videos_anime_id',
'idx_sessions_video_started',
'idx_sessions_status_started',
'idx_sessions_started_at',
'idx_sessions_ended_at',
'idx_telemetry_session_sample',
'idx_telemetry_sample_ms',
'idx_events_session_ts',
'idx_events_type_ts',
'idx_rollups_day_video',
'idx_rollups_month_video',
'idx_words_headword_word_reading',
'idx_words_frequency',
'idx_kanji_kanji',
'idx_kanji_frequency',
'idx_subtitle_lines_session_line',
'idx_subtitle_lines_video_line',
'idx_subtitle_lines_anime_line',
'idx_word_line_occurrences_word',
'idx_kanji_line_occurrences_kanji',
'idx_media_art_cover_blob_hash',
'idx_media_art_anilist_id',
'idx_media_art_cover_url',
'idx_youtube_videos_channel_id',
'idx_youtube_videos_youtube_video_id',
] as const;
function normalizeSql(sql: string | null): string {
return (sql ?? '')
.replace(/\bIF NOT EXISTS\b/gi, '')
.replace(/\s+/g, ' ')
.replace(/\s+([(),])/g, '$1')
.replace(/,\s+/g, ', ')
.trim();
}
function readSchema(dbPath: string): Map<string, string> {
const db = new BunDatabase(dbPath, { readonly: true });
try {
const rows = db
.query<SchemaRow>(
`SELECT type, name, tbl_name, sql
FROM sqlite_schema
WHERE name IN (${SYNC_SCHEMA_OBJECTS.map(() => '?').join(',')})
ORDER BY type, name`,
)
.all(...SYNC_SCHEMA_OBJECTS);
return new Map(rows.map((row) => [row.name, normalizeSql(row.sql)]));
} finally {
db.close();
}
}
test('fixture schema stays aligned with production sync-touched tables and indexes', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-schema-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
const productionPath = path.join(dir, 'production.sqlite');
const productionDb = new BunDatabase(productionPath, { create: true });
try {
createImmersionDbFixture(fixturePath);
ensureSchema(productionDb as never);
productionDb.close();
const fixtureSchema = readSchema(fixturePath);
const productionSchema = readSchema(productionPath);
assert.deepEqual(fixtureSchema, productionSchema);
} finally {
try {
productionDb.close();
} catch {
// already closed
}
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('fixture session inserts enforce foreign keys', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-foreign-keys-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
try {
createImmersionDbFixture(fixturePath);
const db = new BunDatabase(fixturePath, { readwrite: true });
try {
db.run(`
CREATE TRIGGER remove_fixture_video
BEFORE INSERT ON imm_sessions
BEGIN
DELETE FROM imm_videos WHERE video_id = NEW.video_id;
END
`);
} finally {
db.close();
}
assert.throws(
() =>
insertFixtureSession(fixturePath, {
uuid: 'foreign-key-check',
videoKey: 'foreign-key-check',
startedAtMs: Date.UTC(2026, 6, 9),
}),
/FOREIGN KEY constraint failed/,
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
@@ -1,323 +0,0 @@
// Schema-version-18 shape of the tables the sync merge touches (plus the
// app's indexes), mirroring ensureSchema / ensureLifetimeSummaryTables /
// ensureStatsExcludedWordsTable in src/core/services/immersion-tracker/storage.ts.
export const IMMERSION_DB_FIXTURE_DDL = `
CREATE TABLE imm_schema_version (
schema_version INTEGER PRIMARY KEY,
applied_at_ms TEXT NOT NULL
);
CREATE TABLE imm_rollup_state(
state_key TEXT PRIMARY KEY,
state_value TEXT NOT NULL
);
CREATE TABLE imm_anime(
anime_id INTEGER PRIMARY KEY AUTOINCREMENT,
normalized_title_key TEXT NOT NULL UNIQUE,
canonical_title TEXT NOT NULL,
anilist_id INTEGER UNIQUE,
title_romaji TEXT,
title_english TEXT,
title_native TEXT,
episodes_total INTEGER,
description TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_videos(
video_id INTEGER PRIMARY KEY AUTOINCREMENT,
video_key TEXT NOT NULL UNIQUE,
anime_id INTEGER,
canonical_title TEXT NOT NULL,
source_type INTEGER NOT NULL,
source_path TEXT,
source_url TEXT,
parsed_basename TEXT,
parsed_title TEXT,
parsed_season INTEGER,
parsed_episode INTEGER,
parser_source TEXT,
parser_confidence REAL,
parse_metadata_json TEXT,
watched INTEGER NOT NULL DEFAULT 0,
duration_ms INTEGER NOT NULL CHECK(duration_ms>=0),
file_size_bytes INTEGER CHECK(file_size_bytes>=0),
codec_id INTEGER, container_id INTEGER,
width_px INTEGER, height_px INTEGER, fps_x100 INTEGER,
bitrate_kbps INTEGER, audio_codec_id INTEGER,
hash_sha256 TEXT, screenshot_path TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_sessions(
session_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_uuid TEXT NOT NULL UNIQUE,
video_id INTEGER NOT NULL,
started_at_ms TEXT NOT NULL, ended_at_ms TEXT,
status INTEGER NOT NULL,
locale_id INTEGER, target_lang_id INTEGER,
difficulty_tier INTEGER, subtitle_mode INTEGER,
ended_media_ms INTEGER,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id)
);
CREATE TABLE imm_session_telemetry(
telemetry_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
sample_ms TEXT NOT NULL,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_session_events(
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
ts_ms TEXT NOT NULL,
event_type INTEGER NOT NULL,
line_index INTEGER,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
tokens_delta INTEGER NOT NULL DEFAULT 0,
cards_delta INTEGER NOT NULL DEFAULT 0,
payload_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_daily_rollups(
rollup_day INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
cards_per_hour REAL,
tokens_per_min REAL,
lookup_hit_rate REAL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_day, video_id)
);
CREATE TABLE imm_monthly_rollups(
rollup_month INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_month, video_id)
);
CREATE TABLE imm_words(
id INTEGER PRIMARY KEY AUTOINCREMENT,
headword TEXT,
word TEXT,
reading TEXT,
part_of_speech TEXT,
pos1 TEXT,
pos2 TEXT,
pos3 TEXT,
first_seen REAL,
last_seen REAL,
frequency INTEGER,
frequency_rank INTEGER,
UNIQUE(headword, word, reading)
);
CREATE TABLE imm_kanji(
id INTEGER PRIMARY KEY AUTOINCREMENT,
kanji TEXT,
first_seen REAL,
last_seen REAL,
frequency INTEGER,
UNIQUE(kanji)
);
CREATE TABLE imm_subtitle_lines(
line_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
event_id INTEGER,
video_id INTEGER NOT NULL,
anime_id INTEGER,
line_index INTEGER NOT NULL,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
text TEXT NOT NULL,
secondary_text TEXT,
CREATED_DATE INTEGER,
LAST_UPDATE_DATE INTEGER,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE,
FOREIGN KEY(event_id) REFERENCES imm_session_events(event_id) ON DELETE SET NULL,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_word_line_occurrences(
line_id INTEGER NOT NULL,
word_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, word_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(word_id) REFERENCES imm_words(id) ON DELETE CASCADE
);
CREATE TABLE imm_kanji_line_occurrences(
line_id INTEGER NOT NULL,
kanji_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, kanji_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(kanji_id) REFERENCES imm_kanji(id) ON DELETE CASCADE
);
CREATE TABLE imm_media_art(
video_id INTEGER PRIMARY KEY,
anilist_id INTEGER,
cover_url TEXT,
cover_blob BLOB,
cover_blob_hash TEXT,
title_romaji TEXT,
title_english TEXT,
episodes_total INTEGER,
fetched_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_youtube_videos(
video_id INTEGER PRIMARY KEY,
youtube_video_id TEXT NOT NULL,
video_url TEXT NOT NULL,
video_title TEXT,
video_thumbnail_url TEXT,
channel_id TEXT,
channel_name TEXT,
channel_url TEXT,
channel_thumbnail_url TEXT,
uploader_id TEXT,
uploader_url TEXT,
description TEXT,
metadata_json TEXT,
fetched_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_cover_art_blobs(
blob_hash TEXT PRIMARY KEY,
cover_blob BLOB NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_lifetime_global(
global_id INTEGER PRIMARY KEY CHECK(global_id = 1),
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
active_days INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
anime_completed INTEGER NOT NULL DEFAULT 0,
last_rebuilt_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_lifetime_anime(
anime_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_media(
video_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_applied_sessions(
session_id INTEGER PRIMARY KEY,
applied_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_stats_excluded_words(
headword TEXT NOT NULL,
word TEXT NOT NULL,
reading TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY(headword, word, reading)
);
CREATE INDEX idx_anime_normalized_title ON imm_anime(normalized_title_key);
CREATE INDEX idx_anime_anilist_id ON imm_anime(anilist_id);
CREATE INDEX idx_videos_anime_id ON imm_videos(anime_id);
CREATE INDEX idx_sessions_video_started ON imm_sessions(video_id, started_at_ms DESC);
CREATE INDEX idx_sessions_status_started ON imm_sessions(status, started_at_ms DESC);
CREATE INDEX idx_sessions_started_at ON imm_sessions(started_at_ms DESC);
CREATE INDEX idx_sessions_ended_at ON imm_sessions(ended_at_ms DESC);
CREATE INDEX idx_telemetry_session_sample ON imm_session_telemetry(session_id, sample_ms DESC);
CREATE INDEX idx_telemetry_sample_ms ON imm_session_telemetry(sample_ms DESC);
CREATE INDEX idx_events_session_ts ON imm_session_events(session_id, ts_ms DESC);
CREATE INDEX idx_events_type_ts ON imm_session_events(event_type, ts_ms DESC);
CREATE INDEX idx_rollups_day_video ON imm_daily_rollups(rollup_day, video_id);
CREATE INDEX idx_rollups_month_video ON imm_monthly_rollups(rollup_month, video_id);
CREATE INDEX idx_words_headword_word_reading ON imm_words(headword, word, reading);
CREATE INDEX idx_words_frequency ON imm_words(frequency DESC);
CREATE INDEX idx_kanji_kanji ON imm_kanji(kanji);
CREATE INDEX idx_kanji_frequency ON imm_kanji(frequency DESC);
CREATE INDEX idx_subtitle_lines_session_line ON imm_subtitle_lines(session_id, line_index);
CREATE INDEX idx_subtitle_lines_video_line ON imm_subtitle_lines(video_id, line_index);
CREATE INDEX idx_subtitle_lines_anime_line ON imm_subtitle_lines(anime_id, line_index);
CREATE INDEX idx_word_line_occurrences_word ON imm_word_line_occurrences(word_id, line_id);
CREATE INDEX idx_kanji_line_occurrences_kanji ON imm_kanji_line_occurrences(kanji_id, line_id);
CREATE INDEX idx_media_art_cover_blob_hash ON imm_media_art(cover_blob_hash);
CREATE INDEX idx_media_art_anilist_id ON imm_media_art(anilist_id);
CREATE INDEX idx_media_art_cover_url ON imm_media_art(cover_url);
CREATE INDEX idx_youtube_videos_channel_id ON imm_youtube_videos(channel_id);
CREATE INDEX idx_youtube_videos_youtube_video_id ON imm_youtube_videos(youtube_video_id);
`;
-7
View File
@@ -113,13 +113,6 @@ export interface Args {
texthookerOpenBrowser: boolean;
useRofi: boolean;
history: boolean;
sync: boolean;
syncHost: string;
syncSnapshotPath: string;
syncMergePath: string;
syncRemoteCmd: string;
syncDbPath: string;
syncForce: boolean;
logLevel: LogLevel;
logRotation: LogRotation;
passwordStore: string;
+1 -1
View File
@@ -2,7 +2,7 @@
"name": "subminer",
"productName": "SubMiner",
"desktopName": "SubMiner.desktop",
"version": "0.18.0-beta.2",
"version": "0.18.0",
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
"packageManager": "bun@1.3.5",
"main": "dist/main-entry.js",
+11 -3
View File
@@ -21,17 +21,24 @@
### Fixed
- **Character Name Highlighting in Subtitles**
- Fixed unspaced Japanese names (e.g. 東紫乃, 渡辺真奈美) being split at the wrong point, which left surnames like 東 and 渡辺 without their character portrait or hover lookup.
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, or stripped entirely when grammar tagging misclassified the name token.
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, misclassified by grammar tagging, or swallowed entirely by a longer generic dictionary match (e.g. ヨータ disappearing inside a false とヨー match).
- Fixed a single unrecognized word in a subtitle line (like a stray interjection) causing character-name highlighting to drop for the whole line instead of just that word.
- No action needed — existing data upgrades automatically the next time a matching name is seen.
- **Known-Word Highlighting**
- Words are no longer marked "known" (green) just because they share spelling with a known Anki card that actually teaches a different reading (e.g. 床 read as とこ no longer falsely matches a known 床/ゆか card).
- Kanji words are also no longer marked known just because a different mined word happens to share their reading (e.g. 渓谷/けいこく no longer falsely matches a known 警告/けいこく card).
- Single-kana grammar tokens (particles like よ, え) no longer borrow an unrelated card's reading and get falsely painted as known.
- Stats sessions now correctly reflect known-word counts again after the reading-aware matching upgrade, instead of showing 0 everywhere.
- **Annotation Highlighting Refinements**
- Restored frequency/JLPT highlighting and vocabulary-stat counting for words like 確かに and やはり, which were wrongly treated as grammar noise.
- Kanji nouns that MeCab tags as "non-independent" (e.g. 日, 点, 以外) also keep their highlighting and stats counting again.
- Suffix-only tokens (e.g. さん, れる) are now excluded from JLPT/frequency highlighting by default to match how particles and interjections are treated; known-word highlighting for them still works, and this is configurable.
- **Unparsed Subtitle Text**
- Subtitle text the dictionary can't recognize (like a truncated verb form) is now still hoverable for lookup and correctly counted toward a sentence's difficulty, instead of showing as dead, non-interactive text.
- **Kiku Manual Field Grouping**
- Fixed the field-grouping dialog getting stuck invisible behind fullscreen video on Hyprland/Wayland, and failing silently on repeated attempts after the first use.
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut.
- Fixed a timed-out or failed grouping request leaving an invisible, stuck dialog covering the video; it now closes automatically so the overlay recovers.
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut, and added a proper error message for the previously-silent case where the original card can no longer be loaded.
- **Secondary Subtitles**
- Karaoke-style secondary subtitles (common in opening/ending songs) no longer spam dozens of lines down the screen; repeated lines are now collapsed and the subtitle area is capped to a strip at the top.
- **YouTube Extraction**
@@ -59,7 +66,8 @@
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
- fix(stats): parse v3 reading-aware known-word cache in stats server by @ksyasuda in #149
- fix(overlay): keep frequency/JLPT highlight for kanji non-independent nouns by @ksyasuda in #150
- fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names by @ksyasuda in #151
## Installation
+89
View File
@@ -0,0 +1,89 @@
## Highlights
### Added
- **Sentence Audio Normalization**
- Generated sentence audio is now normalized to -23 LUFS by default, giving mined clips consistent volume across shows.
- Clips captured from playback can also mirror mpv's software volume curve, with a limiter to prevent clipping when boosted.
- Both behaviors are controlled independently and can be turned off in the Anki Connect media settings.
- **Watch History Browser**
- Added `subminer -H` / `--history` to browse local watch history, replay the last episode, continue to the next one, or jump to any past episode.
- Works with fzf or rofi; the rofi picker shows AniList cover art already stored in the stats database.
### Changed
- **Known-Word Highlighting Accuracy**
- Highlighting now compares subtitle and Anki-card readings, so it no longer confuses homographs like 床/とこ vs 床/ゆか or unrelated kanji that happen to share a reading.
- Standalone suffix words such as さん or れる are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are already treated (configurable if you'd rather keep them annotated).
- Cards without readings still fall back to word-only matching; the highlighting cache upgrades automatically with no action needed.
- **Stats Trend Charts**
- Overhauled trend charts with persisted title visibility, per-chart title limits, "top" and "most recent" ranking modes, an option to show or hide empty days, calendar-aligned periods, and sortable multi-column tooltips.
- **New App Icon**
- Replaced the SubMiner icon with new pixel-art submarine artwork contributed by an anonymous community member, now used across the app icon, tray, notifications, README, docs site, and stats page.
- **Launcher Preview Layout**
- fzf previews now sit below the launcher menu instead of beside it, giving long titles and metadata more horizontal room.
### Fixed
- **Character Name Recognition**
- Character dictionaries now correctly split unspaced AniList native names and validate readings, so overlay portraits, highlights, and hover lookups work reliably even without MeCab installed.
- Name matches survive punctuation and unmatched text and no longer get overridden by generic dictionary matches or wrongly split from longer words like 空気.
- Existing installs regenerate automatically and upgrade to exact splits once MeCab is available — no action needed.
- **Frequency & JLPT Highlighting Coverage**
- Content adverbs (確かに, やはり) and kanji nouns MeCab tags as non-independent (日, 点, 以外) are now correctly included in frequency/JLPT highlighting and vocabulary stats.
- Lexicalized kana expressions like かといって keep their frequency annotations, while interjections, pronouns, and pure grammar fragments are still filtered out as noise.
- **Unparsed Text Hover Lookup**
- Subtitle text Yomitan can't fully parse — truncated inflections like とこ戻ろ… or elongation runs like ぅ~ — is hoverable again for dictionary lookup.
- These runs stay excluded from frequency/JLPT highlighting and vocabulary stats, same as bracketed captions and punctuation-only text.
- **Karaoke-Style Secondary Subtitles**
- Secondary subtitles no longer flood the screen with dozens of one-syllable lines during karaoke-style openings and endings.
- Repeated events now collapse into a single line, and the secondary subtitle area stays capped to a strip at the top.
- **Kiku Field Grouping Reliability**
- The manual field-grouping dialog now stays above fullscreen mpv on Hyprland/Wayland and keeps working across repeated attempts.
- Abandoned grouping windows close automatically after a timeout or failure, and each attempt now reports a clear success or error, including when the original card can no longer be loaded.
- **Background Stats Server Startup**
- Background `subminer app` launches now start the stats server automatically when enabled, and skip startup if one is already running.
- **AniList Cover Art Timing**
- Stats now fetches the best-match AniList cover as soon as a new series starts playing, so artwork appears in the timeline immediately instead of only after visiting the series page.
- Existing series missing art get backfilled automatically on the next Stats page visit.
- **YouTube Direct Stream Playback**
- Fixed direct YouTube stream extraction so mpv's EDL stream URLs are parsed correctly, preventing corrupted signed video URLs and the resulting ffmpeg 403 errors.
## What's Changed
- fix(youtube): parse mpv EDL stream URLs with byte-length guards by @ksyasuda in #134
- Normalize generated Anki audio by default by @ksyasuda in #135
- feat(launcher): add -H/--history command to browse local watch history by @ksyasuda in #136
- fix(overlay): prevent field grouping modal from freezing overlay on Hyprland by @ksyasuda in #138
- fix(overlay): collapse karaoke syllable spam in secondary subtitles by @ksyasuda in #139
- feat(stats): Trends dashboard overhaul — title visibility, ranking modes, calendar-accurate windows, tooltips by @ksyasuda in #140
- feat(branding): replace app icon with contributed pixel-art set by @ksyasuda in #141
- feat(anki): reading-aware known-word matching (cache v3) by @ksyasuda in #142
- fix(stats): start stats server on background app launch by @ksyasuda in #144
- fix(tokenizer): keep unparsed Yomitan tokens hoverable by @ksyasuda in #145
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
## Installation
See the README and docs/installation guide for full setup steps.
## Assets
- Linux: `SubMiner.AppImage`
- macOS: `SubMiner-*.dmg` and `SubMiner-*.zip`
- Windows: `SubMiner-*.exe` and `SubMiner-*-win.zip`
- Optional extras: `subminer-assets.tar.gz` and the `subminer` launcher
Note: the `subminer` wrapper script uses Bun (`#!/usr/bin/env bun`), so `bun` must be installed and on `PATH`.
+96
View File
@@ -1213,6 +1213,102 @@ test('writeChangelogArtifacts appends contributor attribution and a new-contribu
}
});
test('writeChangelogArtifacts skips contributor attribution in GitHub Actions without a token', async () => {
const { writeChangelogArtifacts } = await loadModule();
const workspace = createWorkspace('release-notes-actions-no-token');
const projectRoot = path.join(workspace, 'SubMiner');
const originalActions = process.env.GITHUB_ACTIONS;
const originalGhToken = process.env.GH_TOKEN;
const originalGithubToken = process.env.GITHUB_TOKEN;
const originalPath = process.env.PATH;
const originalWarn = console.warn;
const warnings: string[] = [];
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(path.join(projectRoot, 'CHANGELOG.md'), '# Changelog\n', 'utf8');
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: added', 'area: release', '', '- Added a feature.'].join('\n'),
'utf8',
);
try {
process.env.GITHUB_ACTIONS = 'true';
delete process.env.GH_TOKEN;
delete process.env.GITHUB_TOKEN;
process.env.PATH = workspace;
console.warn = (message?: unknown) => {
warnings.push(String(message));
};
writeChangelogArtifacts({
cwd: projectRoot,
version: '0.6.0',
date: '2026-05-06',
deps: { runClaude: defaultStubClaude().runClaude },
});
assert.deepEqual(warnings, []);
const releaseNotes = fs.readFileSync(
path.join(projectRoot, 'release', 'release-notes.md'),
'utf8',
);
assert.doesNotMatch(releaseNotes, /## What's Changed/);
} finally {
console.warn = originalWarn;
if (originalActions === undefined) {
delete process.env.GITHUB_ACTIONS;
} else {
process.env.GITHUB_ACTIONS = originalActions;
}
if (originalGhToken === undefined) {
delete process.env.GH_TOKEN;
} else {
process.env.GH_TOKEN = originalGhToken;
}
if (originalGithubToken === undefined) {
delete process.env.GITHUB_TOKEN;
} else {
process.env.GITHUB_TOKEN = originalGithubToken;
}
if (originalPath === undefined) {
delete process.env.PATH;
} else {
process.env.PATH = originalPath;
}
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('shouldSkipDefaultContributionLookup skips GitHub Actions without a gh token', async () => {
const { shouldSkipDefaultContributionLookup } = await loadModule();
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: 'true',
GH_TOKEN: undefined,
GITHUB_TOKEN: undefined,
}),
true,
);
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: 'true',
GH_TOKEN: 'ghs_test',
GITHUB_TOKEN: undefined,
}),
false,
);
assert.equal(
shouldSkipDefaultContributionLookup({
GITHUB_ACTIONS: undefined,
GH_TOKEN: undefined,
GITHUB_TOKEN: undefined,
}),
false,
);
});
test('writeReleaseNotesForVersion preserves committed contributor attribution before installation', async () => {
const { writeReleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('release-notes-preserve-attribution');
+9
View File
@@ -345,6 +345,12 @@ function resolveFragmentRelativePath(fragmentPath: string, cwd: string): string
return path.relative(cwd, fragmentPath).split(path.sep).join('/');
}
export function shouldSkipDefaultContributionLookup(
env: Partial<Record<'GITHUB_ACTIONS' | 'GH_TOKEN' | 'GITHUB_TOKEN', string>> = process.env,
): boolean {
return env.GITHUB_ACTIONS === 'true' && !env.GH_TOKEN && !env.GITHUB_TOKEN;
}
// Walks git history + the GitHub API to attribute each released fragment to the
// PR (and author) that introduced it. One git call and one gh call per fragment,
// plus one gh call per unique author for the first-contribution check. Best
@@ -354,6 +360,9 @@ function defaultResolveContributions(fragmentPaths: string[], cwd: string): Cont
if (fragmentPaths.length === 0) {
return [];
}
if (shouldSkipDefaultContributionLookup()) {
return [];
}
try {
const slug = execFileSync(
+57 -7
View File
@@ -573,6 +573,8 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
endTime: number,
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
generateScreenshot: (path: MediaInput) => Promise<Buffer>;
};
@@ -587,6 +589,7 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName?: string;
generateAudio: boolean;
generateImage: boolean;
volumeScale?: number;
}) => void;
};
internals.client = {
@@ -606,9 +609,17 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
},
};
internals.mediaGenerator = {
generateAudio: async (mediaPath, _startTime, _endTime, _audioPadding, audioStreamIndex) => {
generateAudio: async (
mediaPath,
_startTime,
_endTime,
_audioPadding,
audioStreamIndex,
_normalizeAudio,
volumeScale,
) => {
mediaInputs.push(
`audio:${describeMediaInputForTest(mediaPath)}:${audioStreamIndex ?? 'auto'}`,
`audio:${describeMediaInputForTest(mediaPath)}:${audioStreamIndex ?? 'auto'}:${volumeScale}`,
);
return Buffer.from('audio');
},
@@ -628,6 +639,7 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName: 'Picture',
generateAudio: true,
generateImage: true,
volumeScale: 0.25,
});
internals.queuePendingYoutubeMediaUpdate({
sourceUrl: 'https://youtu.be/abc123',
@@ -640,14 +652,15 @@ test('AnkiIntegration applies ready YouTube cache media to every queued note id'
imageFieldName: 'Picture',
generateAudio: true,
generateImage: true,
volumeScale: 0.8,
});
await integration.handleYoutubeMediaCacheReady('https://youtu.be/abc123', '/tmp/media.mkv');
assert.deepEqual(mediaInputs, [
'audio:/tmp/media.mkv:youtube-cache:auto',
'audio:/tmp/media.mkv:youtube-cache:auto:0.25',
'image:/tmp/media.mkv:youtube-cache',
'audio:/tmp/media.mkv:youtube-cache:auto',
'audio:/tmp/media.mkv:youtube-cache:auto:0.8',
'image:/tmp/media.mkv:youtube-cache',
]);
assert.deepEqual(
@@ -771,6 +784,8 @@ test('AnkiIntegration reports partial queued YouTube media updates separately fr
test('AnkiIntegration queues YouTube media updates against recovered source URLs', async () => {
const updatedNotes: Array<{ noteId: number; fields: Record<string, string> }> = [];
const storedMedia: string[] = [];
const audioVolumeScales: Array<number | undefined> = [];
let mpvVolume = 30;
const integration = new AnkiIntegration(
{
@@ -787,6 +802,10 @@ test('AnkiIntegration queues YouTube media updates against recovered source URLs
currentSubStart: 10,
currentSubEnd: 12,
currentTimePos: 11,
requestProperty: async (name: string) => {
assert.equal(name, 'volume');
return mpvVolume;
},
} as never,
() => undefined,
undefined,
@@ -807,7 +826,15 @@ test('AnkiIntegration queues YouTube media updates against recovered source URLs
storeMediaFile: (filename: string) => Promise<void>;
};
mediaGenerator: {
generateAudio: () => Promise<Buffer>;
generateAudio: (
path: MediaInput,
startTime: number,
endTime: number,
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
generateScreenshot: () => Promise<Buffer>;
};
queuePendingYoutubeMediaUpdateForNote: (job: {
@@ -834,7 +861,18 @@ test('AnkiIntegration queues YouTube media updates against recovered source URLs
},
};
internals.mediaGenerator = {
generateAudio: async () => Buffer.from('audio'),
generateAudio: async (
_path,
_startTime,
_endTime,
_audioPadding,
_audioStreamIndex,
_normalizeAudio,
volumeScale,
) => {
audioVolumeScales.push(volumeScale);
return Buffer.from('audio');
},
generateScreenshot: async () => Buffer.from('image'),
};
internals.showNotification = async () => undefined;
@@ -850,6 +888,7 @@ test('AnkiIntegration queues YouTube media updates against recovered source URLs
},
label: 'resolved source',
});
mpvVolume = 90;
await integration.handleYoutubeMediaCacheReady('https://youtu.be/abc123', '/tmp/media.mkv');
assert.equal(queued, true);
@@ -858,6 +897,7 @@ test('AnkiIntegration queues YouTube media updates against recovered source URLs
assert.match(updatedNotes[0]?.fields.SentenceAudio ?? '', /^\[sound:audio_/);
assert.match(updatedNotes[0]?.fields.Picture ?? '', /^<img src="image_/);
assert.equal(storedMedia.length, 2);
assert.deepEqual(audioVolumeScales, [0.3 ** 3]);
});
test('AnkiIntegration passes audio normalization config for ready cached YouTube audio', async () => {
@@ -865,7 +905,9 @@ test('AnkiIntegration passes audio normalization config for ready cached YouTube
path: string;
audioStreamIndex?: number;
normalizeAudio?: boolean;
volumeScale?: number;
}> = [];
const requestedProperties: string[] = [];
const integration = new AnkiIntegration(
{
@@ -881,6 +923,10 @@ test('AnkiIntegration passes audio normalization config for ready cached YouTube
currentSubStart: 10,
currentSubEnd: 12,
currentTimePos: 11,
requestProperty: async (name: string) => {
requestedProperties.push(name);
return 55;
},
} as never,
() => undefined,
undefined,
@@ -902,6 +948,7 @@ test('AnkiIntegration passes audio normalization config for ready cached YouTube
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
) => Promise<Buffer>;
};
generateAudio: () => Promise<Buffer | null>;
@@ -914,19 +961,22 @@ test('AnkiIntegration passes audio normalization config for ready cached YouTube
_audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) => {
audioCalls.push({ path: path.path, audioStreamIndex, normalizeAudio });
audioCalls.push({ path: path.path, audioStreamIndex, normalizeAudio, volumeScale });
return Buffer.from('audio');
},
};
await internals.generateAudio();
assert.deepEqual(requestedProperties, ['volume']);
assert.deepEqual(audioCalls, [
{
path: '/tmp/subminer-youtube-media-cache/media.mkv',
audioStreamIndex: undefined,
normalizeAudio: false,
volumeScale: 0.55 ** 3,
},
]);
});
+35 -6
View File
@@ -73,6 +73,7 @@ import type {
PendingYoutubeMediaQueueFailedOptions,
PendingYoutubeMediaQueueReadyOptions,
} from './anki-integration/pending-youtube-media-queue';
import { resolveMpvVolumeScale } from './anki-integration/mpv-volume';
const log = createLogger('anki').child('integration');
@@ -341,14 +342,23 @@ export class AnkiIntegration {
storeMediaFile: (filename, data) => this.client.storeMediaFile(filename, data),
},
mediaGenerator: {
generateAudio: (videoPath, startTime, endTime, audioPadding, audioStreamIndex) =>
generateAudio: (
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) =>
this.mediaGenerator.generateAudio(
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
this.config.media?.normalizeAudio !== false,
normalizeAudio,
volumeScale,
),
generateScreenshot: (videoPath, timestamp, options) =>
this.mediaGenerator.generateScreenshot(videoPath, timestamp, options),
@@ -373,6 +383,7 @@ export class AnkiIntegration {
mergeFieldValue: (existing, newValue, overwrite) =>
this.mergeFieldValue(existing, newValue, overwrite),
getAnimatedImageLeadInSeconds: (noteInfo) => this.getAnimatedImageLeadInSeconds(noteInfo),
getMpvVolumeScale: () => this.getMpvVolumeScale(),
generateAudioFilename: () => this.generateAudioFilename(),
generateImageFilename: () => this.generateImageFilename(),
formatMiscInfoPatternForMediaPath: (
@@ -496,14 +507,23 @@ export class AnkiIntegration {
retrieveMediaFile: (filename) => this.client.retrieveMediaFile(filename),
},
mediaGenerator: {
generateAudio: (videoPath, startTime, endTime, audioPadding, audioStreamIndex) =>
generateAudio: (
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
normalizeAudio,
volumeScale,
) =>
this.mediaGenerator.generateAudio(
videoPath,
startTime,
endTime,
audioPadding,
audioStreamIndex,
this.config.media?.normalizeAudio !== false,
normalizeAudio,
volumeScale,
),
generateScreenshot: (videoPath, timestamp, options) =>
this.mediaGenerator.generateScreenshot(videoPath, timestamp, options),
@@ -703,8 +723,12 @@ export class AnkiIntegration {
});
}
isKnownWord(text: string, reading?: string): boolean {
return this.knownWordCache.isKnownWord(text, reading);
isKnownWord(
text: string,
reading?: string,
options?: { allowReadingOnlyMatch?: boolean },
): boolean {
return this.knownWordCache.isKnownWord(text, reading, options);
}
getKnownWordMatchMode(): NPlusOneMatchMode {
@@ -961,6 +985,10 @@ export class AnkiIntegration {
);
}
private async getMpvVolumeScale(): Promise<number | undefined> {
return resolveMpvVolumeScale(this.mpvClient, this.config.media?.mirrorMpvVolume !== false);
}
async handleYoutubeMediaCacheReady(
sourceUrl: string,
cachedPath: string,
@@ -999,6 +1027,7 @@ export class AnkiIntegration {
this.config.media?.audioPadding,
resolveAudioStreamIndexForMediaGeneration(videoPath, this.mpvClient.currentAudioStreamIndex),
this.config.media?.normalizeAudio !== false,
await this.getMpvVolumeScale(),
);
}
@@ -10,6 +10,8 @@ test('sentence card writes generated audio only to sentence audio field', async
const addedFields: Record<string, string>[] = [];
const updatedFields: Record<string, string>[] = [];
const storedMedia: string[] = [];
const requestedProperties: string[] = [];
const audioVolumeScales: Array<number | undefined> = [];
const deps: CardCreationDeps = {
getConfig: () =>
@@ -24,6 +26,7 @@ test('sentence card writes generated audio only to sentence audio field', async
media: {
generateAudio: true,
generateImage: false,
mirrorMpvVolume: true,
maxMediaDuration: 30,
},
behavior: {},
@@ -39,6 +42,10 @@ test('sentence card writes generated audio only to sentence audio field', async
currentSubEnd: 14,
currentTimePos: 13,
currentAudioStreamIndex: 0,
requestProperty: async (name: string) => {
requestedProperties.push(name);
return 40;
},
}) as never,
client: {
addNote: async (_deck, _modelName, fields) => {
@@ -68,7 +75,18 @@ test('sentence card writes generated audio only to sentence audio field', async
retrieveMediaFile: async () => '',
},
mediaGenerator: {
generateAudio: async () => Buffer.from('audio'),
generateAudio: async (
_path,
_startTime,
_endTime,
_audioPadding,
_audioStreamIndex,
_normalizeAudio,
volumeScale,
) => {
audioVolumeScales.push(volumeScale);
return Buffer.from('audio');
},
generateScreenshot: async () => null,
generateAnimatedImage: async () => null,
},
@@ -124,6 +142,8 @@ test('sentence card writes generated audio only to sentence audio field', async
Expression: '字幕',
});
assert.equal(storedMedia.length, 1);
assert.deepEqual(requestedProperties, ['volume']);
assert.deepEqual(audioVolumeScales, [0.4 ** 3]);
const mediaUpdate = updatedFields.find((fields) => 'SentenceAudio' in fields);
assert.equal(mediaUpdate?.SentenceAudio, `[sound:${storedMedia[0]}]`);
assert.equal('ExpressionAudio' in mediaUpdate!, false);
+6 -1
View File
@@ -525,6 +525,7 @@ test('CardCreationService queues YouTube media when required cache is not ready'
miscInfoFieldName?: string;
generateAudio: boolean;
generateImage: boolean;
volumeScale?: number;
}> = [];
let streamRequests = 0;
@@ -556,7 +557,10 @@ test('CardCreationService queues YouTube media when required cache is not ready'
currentSubEnd: 12,
currentTimePos: 11,
currentAudioStreamIndex: 2,
requestProperty: async () => {
requestProperty: async (name: string) => {
if (name === 'volume') {
return 35;
}
streamRequests += 1;
return 'https://rr1---sn.example.googlevideo.com/videoplayback?id=123';
},
@@ -652,6 +656,7 @@ test('CardCreationService queues YouTube media when required cache is not ready'
miscInfoFieldName: 'MiscInfo',
generateAudio: true,
generateImage: true,
volumeScale: 0.35 ** 3,
},
]);
assert.deepEqual(updates, []);
+13
View File
@@ -17,6 +17,7 @@ import {
} from './media-source';
import { shouldMarkWordAndSentenceCard } from './note-field-utils';
import type { PendingYoutubeMediaUpdate } from './pending-youtube-media';
import { resolveMpvVolumeScale } from './mpv-volume';
const log = createLogger('anki').child('integration.card-creation');
@@ -66,6 +67,7 @@ interface CardCreationMediaGenerator {
audioPadding?: number,
audioStreamIndex?: number,
normalizeAudio?: boolean,
volumeScale?: number,
): Promise<Buffer | null>;
generateScreenshot(
path: MediaInput,
@@ -714,6 +716,12 @@ export class CardCreationService {
const label = sentence.length > 30 ? sentence.substring(0, 30) + '...' : sentence;
if (shouldQueuePendingYoutubeMedia) {
const volumeScale = generateAudio
? await resolveMpvVolumeScale(
mpvClient,
this.deps.getConfig().media?.mirrorMpvVolume !== false,
)
: undefined;
this.deps.queuePendingYoutubeMediaUpdate?.({
sourceUrl:
trimToNonEmptyString(await this.deps.getYoutubeMediaSourceUrl?.()) ??
@@ -727,6 +735,7 @@ export class CardCreationService {
miscInfoFieldName: resolvedMiscInfoField ?? undefined,
generateAudio,
generateImage,
volumeScale,
});
await this.deps.showNotification(noteId, label, 'media queued');
return true;
@@ -844,6 +853,10 @@ export class CardCreationService {
mpvClient.currentAudioStreamIndex ?? undefined,
),
this.deps.getConfig().media?.normalizeAudio !== false,
await resolveMpvVolumeScale(
mpvClient,
this.deps.getConfig().media?.mirrorMpvVolume !== false,
),
);
}
@@ -734,6 +734,42 @@ test('KnownWordCacheManager disambiguates known words by note reading', async ()
}
});
test('KnownWordCacheManager suppresses reading-only matches when disallowed', async () => {
const config: AnkiConnectConfig = {
fields: {
word: 'Word',
},
knownWords: {
highlightEnabled: true,
},
};
const { manager, clientState, cleanup } = createKnownWordCacheHarness(config);
try {
clientState.findNotesResult = [1];
clientState.notesInfoResult = [
{
noteId: 1,
fields: {
Word: { value: '警告' },
'Word Reading': { value: 'けいこく' },
},
},
];
await manager.refresh(true);
// Reading-only match stays available for kana subtitle text…
assert.equal(manager.isKnownWord('けいこく'), true);
// …but a kanji token's reading (渓谷/けいこく) must not borrow 警告's.
assert.equal(manager.isKnownWord('けいこく', undefined, { allowReadingOnlyMatch: false }), false);
// Mined word texts still match regardless of the flag.
assert.equal(manager.isKnownWord('警告', undefined, { allowReadingOnlyMatch: false }), true);
} finally {
cleanup();
}
});
test('KnownWordCacheManager does not match single-kana text by reading alone', async () => {
const config: AnkiConnectConfig = {
fields: {
+13 -1
View File
@@ -142,7 +142,11 @@ export class KnownWordCacheManager {
);
}
isKnownWord(text: string, reading?: string): boolean {
isKnownWord(
text: string,
reading?: string,
options?: { allowReadingOnlyMatch?: boolean },
): boolean {
if (!this.isKnownWordCacheEnabled()) {
return false;
}
@@ -163,6 +167,14 @@ export class KnownWordCacheManager {
);
}
// Callers that look up a kanji token's reading (not subtitle text) must
// opt out of the reading-only fallback: readingCounts holds readings of
// every note including kanji words, so 渓谷's けいこく would match a
// mined 警告/けいこく.
if (options?.allowReadingOnlyMatch === false) {
return false;
}
// Reading-only fallback, except for single-kana text: particles and
// interjections (よ, ね, え…) would otherwise borrow the reading of an
// unrelated note (夜「よ」, 絵「え」) and count as known.
+56
View File
@@ -0,0 +1,56 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { resolveMpvVolumeScale } from './mpv-volume';
test('resolveMpvVolumeScale converts numeric mpv volume with mpv software volume curve', async () => {
const requested: string[] = [];
const scale = await resolveMpvVolumeScale(
{
requestProperty: async (name) => {
requested.push(name);
return 75;
},
},
true,
);
assert.equal(scale, 0.421875);
assert.deepEqual(requested, ['volume']);
});
test('resolveMpvVolumeScale skips mpv when mirroring is disabled', async () => {
let requested = false;
const scale = await resolveMpvVolumeScale(
{
requestProperty: async () => {
requested = true;
return 50;
},
},
false,
);
assert.equal(scale, undefined);
assert.equal(requested, false);
});
test('resolveMpvVolumeScale falls back to unity for missing, failed, or invalid values', async () => {
assert.equal(await resolveMpvVolumeScale({}, true), 1);
assert.equal(
await resolveMpvVolumeScale(
{
requestProperty: async () => {
throw new Error('disconnected');
},
},
true,
),
1,
);
assert.equal(await resolveMpvVolumeScale({ requestProperty: async () => '50' }, true), 1);
assert.equal(await resolveMpvVolumeScale({ requestProperty: async () => Number.NaN }, true), 1);
assert.equal(await resolveMpvVolumeScale({ requestProperty: async () => -1 }, true), 1);
});
+26
View File
@@ -0,0 +1,26 @@
export interface MpvVolumePropertySource {
requestProperty?: (name: string) => Promise<unknown>;
}
export async function resolveMpvVolumeScale(
mpvClient: MpvVolumePropertySource,
enabled: boolean,
): Promise<number | undefined> {
if (!enabled) {
return undefined;
}
if (!mpvClient.requestProperty) {
return 1;
}
try {
const volume = await mpvClient.requestProperty('volume');
if (typeof volume !== 'number' || !Number.isFinite(volume) || volume < 0) {
return 1;
}
return (volume / 100) ** 3;
} catch {
return 1;
}
}
@@ -35,6 +35,7 @@ function createDeps(
resolveConfiguredFieldName: () => 'Picture',
mergeFieldValue: (_existing, newValue) => newValue,
getAnimatedImageLeadInSeconds: async () => 0,
getMpvVolumeScale: async () => 1,
generateAudioFilename: () => 'audio.mp3',
generateImageFilename: () => 'image.webp',
formatMiscInfoPatternForMediaPath: () => '',
@@ -46,6 +46,7 @@ export interface PendingYoutubeMediaQueueDeps {
) => string | null;
mergeFieldValue: (existing: string, newValue: string, overwrite: boolean) => string;
getAnimatedImageLeadInSeconds: (noteInfo: PendingYoutubeMediaNoteInfo) => Promise<number>;
getMpvVolumeScale: () => Promise<number | undefined>;
generateAudioFilename: () => string;
generateImageFilename: () => string;
formatMiscInfoPatternForMediaPath: (
@@ -126,6 +127,9 @@ export class PendingYoutubeMediaQueue {
const config = this.deps.getConfig();
const mediaRange = this.deps.getSubtitleMediaRange(job.context);
const volumeScale = shouldGenerateAudio(config)
? await this.deps.getMpvVolumeScale()
: undefined;
this.enqueue({
sourceUrl,
noteId: job.noteId,
@@ -143,6 +147,7 @@ export class PendingYoutubeMediaQueue {
this.deps.resolveConfiguredFieldName(job.noteInfo, config.fields?.miscInfo) ?? undefined,
generateAudio: shouldGenerateAudio(config),
generateImage: shouldGenerateImage(config),
volumeScale,
});
return true;
}
@@ -273,6 +278,7 @@ export class PendingYoutubeMediaQueue {
config.media?.audioPadding,
undefined,
config.media?.normalizeAudio !== false,
job.volumeScale,
);
if (audioBuffer) {
await this.deps.client.storeMediaFile(audioFilename, audioBuffer);
@@ -9,6 +9,7 @@ export interface PendingYoutubeMediaUpdate {
miscInfoFieldName?: string;
generateAudio: boolean;
generateImage: boolean;
volumeScale?: number;
}
function trimToNonEmptyString(value: unknown): string | null {
+26
View File
@@ -93,6 +93,7 @@ test('loads defaults when config is missing', () => {
systemPrompt: '',
});
assert.equal(config.ankiConnect.media.normalizeAudio, true);
assert.equal(config.ankiConnect.media.mirrorMpvVolume, true);
assert.equal(config.startupWarmups.lowPowerMode, false);
assert.equal(config.startupWarmups.mecab, true);
assert.equal(config.startupWarmups.yomitanExtension, true);
@@ -168,6 +169,31 @@ test('loads defaults when config is missing', () => {
assert.equal(config.mpv.aniskipButtonKey, 'TAB');
});
test('rejects invalid mpv volume mirroring values', () => {
const dir = makeTempDir();
fs.writeFileSync(
path.join(dir, 'config.jsonc'),
`{
"ankiConnect": {
"media": {
"mirrorMpvVolume": "false"
}
}
}`,
'utf-8',
);
const service = new ConfigService(dir);
assert.equal(
service.getConfig().ankiConnect.media.mirrorMpvVolume,
DEFAULT_CONFIG.ankiConnect.media.mirrorMpvVolume,
);
assert.ok(
service.getWarnings().some((warning) => warning.path === 'ankiConnect.media.mirrorMpvVolume'),
);
});
test('parses updates config and warns on invalid values', () => {
const validDir = makeTempDir();
fs.writeFileSync(
@@ -52,6 +52,7 @@ export const INTEGRATIONS_DEFAULT_CONFIG: Pick<
animatedCrf: 35,
syncAnimatedImageToWordAudio: true,
normalizeAudio: true,
mirrorMpvVolume: true,
audioPadding: 0,
fallbackDuration: 3.0,
maxMediaDuration: 30,
@@ -111,6 +111,7 @@ test('config option registry includes critical paths and has unique entries', ()
'ankiConnect.enabled',
'subtitleStyle.nameMatchEnabled',
'ankiConnect.media.normalizeAudio',
'ankiConnect.media.mirrorMpvVolume',
'anilist.characterDictionary.collapsibleSections.description',
'mpv.executablePath',
'mpv.launchMode',
@@ -185,7 +185,15 @@ export function buildIntegrationConfigOptionRegistry(
path: 'ankiConnect.media.normalizeAudio',
kind: 'boolean',
defaultValue: defaultConfig.ankiConnect.media.normalizeAudio,
description: 'Normalize generated sentence audio loudness during media extraction.',
description:
'Normalize generated sentence audio loudness during media extraction. Changes apply live.',
},
{
path: 'ankiConnect.media.mirrorMpvVolume',
kind: 'boolean',
defaultValue: defaultConfig.ankiConnect.media.mirrorMpvVolume,
description:
"Apply mpv's current software volume curve to generated sentence audio. Changes apply live.",
},
{
path: 'ankiConnect.media.generateImage',
+1 -1
View File
@@ -135,7 +135,7 @@ const INTEGRATION_TEMPLATE_SECTIONS: ConfigTemplateSection[] = [
title: 'AnkiConnect Integration',
description: ['Automatic Anki updates and media generation options.'],
notes: [
'Hot-reload: ankiConnect.ai.enabled, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.',
'Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.',
'Shared AI provider transport settings are read from top-level ai and typically require restart.',
'Most other AnkiConnect settings still require restart.',
],
+16
View File
@@ -106,6 +106,22 @@ export function applyAnkiConnectResolution(context: ResolveContext): void {
},
};
if (hasOwn(media, 'mirrorMpvVolume')) {
const parsed = asBoolean(media.mirrorMpvVolume);
if (parsed === undefined) {
context.resolved.ankiConnect.media.mirrorMpvVolume =
DEFAULT_CONFIG.ankiConnect.media.mirrorMpvVolume;
context.warn(
'ankiConnect.media.mirrorMpvVolume',
media.mirrorMpvVolume,
context.resolved.ankiConnect.media.mirrorMpvVolume,
'Expected boolean.',
);
} else {
context.resolved.ankiConnect.media.mirrorMpvVolume = parsed;
}
}
if (hasOwn(behavior, 'notificationType')) {
const parsed = asNotificationType(behavior.notificationType);
if (parsed === undefined) {
+11
View File
@@ -165,6 +165,15 @@ test('settings registry exposes specialized controls for config-assisted inputs'
assert.equal(field('discordPresence.presenceStyle').control, 'select');
});
test('settings registry exposes mpv volume mirroring as a mining toggle', () => {
const volumeField = field('ankiConnect.media.mirrorMpvVolume');
assert.equal(volumeField.category, 'mining-anki');
assert.equal(volumeField.section, 'Media Capture');
assert.equal(volumeField.control, 'boolean');
assert.equal(volumeField.defaultValue, true);
});
test('settings registry exposes YouTube media cache mode as a labeled select', () => {
const mediaCacheMode = field('youtube.mediaCache.mode');
const mediaCacheMaxHeight = field('youtube.mediaCache.maxHeight');
@@ -313,6 +322,8 @@ test('settings registry marks safe live config paths as hot-reloadable', () => {
'subsync.replace',
'ankiConnect.behavior.autoUpdateNewCards',
'ankiConnect.deck',
'ankiConnect.media.normalizeAudio',
'ankiConnect.media.mirrorMpvVolume',
'ankiConnect.knownWords.highlightEnabled',
'ankiConnect.knownWords.refreshMinutes',
'ankiConnect.knownWords.addMinedWordsImmediately',
+3
View File
@@ -236,6 +236,7 @@ const LABEL_OVERRIDES: Record<string, string> = {
'mpv.pauseUntilOverlayReady': 'Pause Until Overlay Ready',
'mpv.aniskipEnabled': 'Enable AniSkip',
'mpv.aniskipButtonKey': 'AniSkip Button Key',
'ankiConnect.media.mirrorMpvVolume': 'Mirror mpv Volume',
'discordPresence.updateIntervalMs': 'Update Interval (ms)',
};
@@ -671,6 +672,8 @@ function restartBehaviorForPath(path: string): ConfigSettingsRestartBehavior {
path === 'secondarySub.defaultMode' ||
path === 'ankiConnect.deck' ||
path === 'ankiConnect.ai.enabled' ||
path === 'ankiConnect.media.normalizeAudio' ||
path === 'ankiConnect.media.mirrorMpvVolume' ||
path === 'ankiConnect.behavior.autoUpdateNewCards' ||
path === 'ankiConnect.knownWords.highlightEnabled' ||
path === 'ankiConnect.knownWords.refreshMinutes' ||
@@ -31,6 +31,8 @@ test('classifyConfigHotReloadDiff treats safe nested config paths as hot-reloada
next.jimaku.maxEntryResults = prev.jimaku.maxEntryResults + 1;
next.subsync.replace = !prev.subsync.replace;
next.ankiConnect.deck = 'Mining';
next.ankiConnect.media.normalizeAudio = !prev.ankiConnect.media.normalizeAudio;
next.ankiConnect.media.mirrorMpvVolume = !prev.ankiConnect.media.mirrorMpvVolume;
next.ankiConnect.behavior.autoUpdateNewCards = !prev.ankiConnect.behavior.autoUpdateNewCards;
next.ankiConnect.knownWords.highlightEnabled = !prev.ankiConnect.knownWords.highlightEnabled;
next.ankiConnect.knownWords.refreshMinutes = prev.ankiConnect.knownWords.refreshMinutes + 5;
@@ -65,6 +67,8 @@ test('classifyConfigHotReloadDiff treats safe nested config paths as hot-reloada
'jimaku.maxEntryResults',
'subsync.replace',
'ankiConnect.deck',
'ankiConnect.media.normalizeAudio',
'ankiConnect.media.mirrorMpvVolume',
'ankiConnect.behavior.autoUpdateNewCards',
'ankiConnect.knownWords.highlightEnabled',
'ankiConnect.knownWords.refreshMinutes',
+2
View File
@@ -68,6 +68,8 @@ const HOT_RELOAD_EXACT_OR_PREFIX_PATHS = [
'jimaku',
'subsync',
'ankiConnect.deck',
'ankiConnect.media.normalizeAudio',
'ankiConnect.media.mirrorMpvVolume',
'ankiConnect.behavior.autoUpdateNewCards',
'ankiConnect.knownWords.highlightEnabled',
'ankiConnect.knownWords.refreshMinutes',
+34
View File
@@ -28,6 +28,7 @@ interface YomitanTokenInput {
frequencyRank?: number;
isNameMatch?: boolean;
wordClasses?: string[];
isUnparsedRun?: boolean;
}
function makeDepsFromYomitanTokens(
@@ -60,6 +61,7 @@ function makeDepsFromYomitanTokens(
isNameMatch: token.isNameMatch ?? false,
frequencyRank: token.frequencyRank,
wordClasses: token.wordClasses,
isUnparsedRun: token.isUnparsedRun,
};
});
},
@@ -4223,6 +4225,38 @@ test('tokenizeSubtitle clears all annotations for explanatory pondering endings'
);
});
test('tokenizeSubtitle ignores unparsed-run tokens for annotations and N+1', async () => {
// もう いるぅ~!: the ぅ~ elongation has no Yomitan dictionary entry; it must
// not become the sole N+1 candidate or receive frequency/JLPT annotations.
const result = await tokenizeSubtitle(
'もう いるぅ~!',
makeDepsFromYomitanTokens(
[
{ surface: 'もう', reading: 'もう', headword: 'もう' },
{ surface: 'いる', reading: 'いる', headword: 'いる' },
{ surface: 'ぅ~', reading: '', headword: 'ぅ~', isUnparsedRun: true, frequencyRank: 999 },
],
{
getFrequencyDictionaryEnabled: () => true,
getJlptLevel: (text) => (text === 'ぅ~' ? 'N5' : null),
isKnownWord: (text) => text === 'もう' || text === 'いる',
getMinSentenceWordsForNPlusOne: () => 2,
tokenizeWithMecab: async () => null,
},
),
);
const filler = result.tokens?.find((token) => token.surface === 'ぅ~');
assert.ok(filler);
assert.equal(filler?.isNPlusOneTarget, false);
assert.equal(filler?.frequencyRank, undefined);
assert.equal(filler?.jlptLevel, undefined);
assert.equal(
result.tokens?.some((token) => token.isNPlusOneTarget),
false,
);
});
test('tokenizeSubtitle keeps frequency for content-led merged token with trailing colloquial suffixes', async () => {
const result = await tokenizeSubtitle(
'張り切ってんじゃ',
+13 -3
View File
@@ -33,6 +33,15 @@ type MecabTokenEnrichmentFn = (
mecabTokens: MergedToken[] | null,
) => Promise<MergedToken[]>;
// allowReadingOnlyMatch: false suppresses the cache's reading-only index for
// lookups that pass a kanji token's reading as the text (see
// computeTokenKnownStatus in annotation-stage).
export type KnownWordLookupFn = (
text: string,
reading?: string,
options?: { allowReadingOnlyMatch?: boolean },
) => boolean;
export interface TokenizerServiceDeps {
getYomitanExt: () => Extension | null;
getYomitanSession?: () => Session | null;
@@ -42,7 +51,7 @@ export interface TokenizerServiceDeps {
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
getYomitanParserInitPromise: () => Promise<boolean> | null;
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
isKnownWord: (text: string, reading?: string) => boolean;
isKnownWord: KnownWordLookupFn;
getKnownWordMatchMode: () => NPlusOneMatchMode;
getKnownWordsEnabled?: () => boolean;
getJlptLevel: (text: string) => JlptLevel | null;
@@ -77,7 +86,7 @@ export interface TokenizerDepsRuntimeOptions {
setYomitanParserReadyPromise: (promise: Promise<void> | null) => void;
getYomitanParserInitPromise: () => Promise<boolean> | null;
setYomitanParserInitPromise: (promise: Promise<boolean> | null) => void;
isKnownWord: (text: string, reading?: string) => boolean;
isKnownWord: KnownWordLookupFn;
getKnownWordMatchMode: () => NPlusOneMatchMode;
getKnownWordsEnabled?: () => boolean;
getJlptLevel: (text: string) => JlptLevel | null;
@@ -129,7 +138,7 @@ const INVISIBLE_SEPARATOR_PATTERN = /[\u200b\u2060\ufeff]/g;
function getKnownWordLookup(
deps: TokenizerServiceDeps,
options: TokenizerAnnotationOptions,
): (text: string, reading?: string) => boolean {
): KnownWordLookupFn {
if (!options.knownWordsEnabled && !options.nPlusOneEnabled) {
return () => false;
}
@@ -734,6 +743,7 @@ async function parseWithYomitanInternalParser(
isNPlusOneTarget: false,
isNameMatch: token.isNameMatch ?? false,
frequencyRank: token.frequencyRank,
isUnparsedRun: token.isUnparsedRun === true ? true : undefined,
};
}),
);
@@ -208,6 +208,48 @@ test('annotateTokens hides known-word marks while still using known words for N+
assert.equal(result[2]?.isNPlusOneTarget, true);
});
test('shouldExcludeTokenFromSubtitleAnnotations excludes unparsed-run tokens', () => {
// 戻 from 「…とこ戻ろ…」: Yomitan had no dictionary entry, headword falls back
// to the surface. Without the flag the token passes every other filter.
const unflagged = makeToken({ surface: '戻', headword: '戻', reading: '' });
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(unflagged), false);
const flagged = makeToken({ surface: '戻', headword: '戻', reading: '', isUnparsedRun: true });
assert.equal(shouldExcludeTokenFromSubtitleAnnotations(flagged), true);
assert.equal(shouldExcludeTokenFromVocabularyPersistence(flagged), true);
});
test('annotateTokens ignores unparsed-run tokens for annotations and N+1', () => {
// もう いるぅ~!-style line: the elongation run is the only unknown token and
// used to become the sole N+1 candidate despite having no dictionary entry.
const tokens = [
makeToken({ surface: 'みんな', headword: '皆', reading: 'みんな', startPos: 0, endPos: 3 }),
makeToken({ surface: 'とこ', headword: '所', reading: 'とこ', startPos: 3, endPos: 5 }),
makeToken({
surface: '戻',
headword: '戻',
reading: '',
startPos: 5,
endPos: 6,
isUnparsedRun: true,
frequencyRank: 12,
}),
];
const result = annotateTokens(
tokens,
makeDeps({
isKnownWord: (text) => text === '皆' || text === '所',
getJlptLevel: (text) => (text === '戻' ? 'N5' : null),
}),
{ minSentenceWordsForNPlusOne: 2 },
);
assert.equal(result[2]?.isNPlusOneTarget, false);
assert.equal(result[2]?.jlptLevel, undefined);
assert.equal(result[2]?.frequencyRank, undefined);
});
test('annotateTokens falls back to reading for known-word matches when headword lookup misses', () => {
const tokens = [
makeToken({
@@ -231,6 +273,29 @@ test('annotateTokens falls back to reading for known-word matches when headword
assert.equal(result[0]?.frequencyRank, 1895);
});
test('annotateTokens reading fallback does not match kanji tokens sharing a mined reading', () => {
const tokens = [
makeToken({
surface: '渓谷',
headword: '渓谷',
reading: 'けいこく',
endPos: 2,
}),
];
const result = annotateTokens(
tokens,
makeDeps({
// Mimics the cache with a mined 警告/けいこく: けいこく matches through
// the reading-only index unless the lookup opts out of it.
isKnownWord: (text, _reading, options) =>
text === '警告' || (options?.allowReadingOnlyMatch !== false && text === 'けいこく'),
}),
);
assert.equal(result[0]?.isKnown, false);
});
test('annotateTokens ignores partial furigana readings for known-word fallback', () => {
const tokens = [
makeToken({
@@ -1648,7 +1713,9 @@ test('annotateTokens excludes kana-only composite function/content tokens from f
const tokens = [
makeToken({
surface: 'になれば',
reading: 'になれば',
headword: 'なる',
headwordReading: 'なる',
pos1: '助詞|動詞',
pos2: '格助詞|自立|接続助詞',
startPos: 0,
@@ -1665,6 +1732,28 @@ test('annotateTokens excludes kana-only composite function/content tokens from f
assert.equal(result[0]?.isNPlusOneTarget, false);
});
test('annotateTokens keeps frequency for mixed kana tokens whose headword reading covers the token', () => {
const tokens = [
makeToken({
surface: 'かといって',
reading: 'カトイッテ',
headword: 'かと言って',
headwordReading: 'かといって',
pos1: '助詞|助詞|動詞|助詞',
pos2: '副助詞|格助詞|自立|接続助詞',
startPos: 0,
endPos: 5,
frequencyRank: 4898,
}),
];
const result = annotateTokens(tokens, makeDeps(), {
minSentenceWordsForNPlusOne: 1,
});
assert.equal(result[0]?.frequencyRank, 4898);
});
test('annotateTokens excludes composite tokens when all component pos tags are excluded', () => {
const tokens = [
makeToken({
@@ -26,7 +26,11 @@ const jlptLevelLookupCaches = new WeakMap<
>();
export interface AnnotationStageDeps {
isKnownWord: (text: string, reading?: string) => boolean;
isKnownWord: (
text: string,
reading?: string,
options?: { allowReadingOnlyMatch?: boolean },
) => boolean;
knownWordMatchMode: NPlusOneMatchMode;
getJlptLevel: (text: string) => JlptLevel | null;
}
@@ -546,10 +550,20 @@ function isKanaOnlyMixedFunctionContentToken(
}
const pos1Parts = splitNormalizedTagParts(normalizePos1Tag(token.pos1));
return (
const hasMixedFunctionContentParts =
pos1Parts.length >= 2 &&
pos1Parts.some((part) => pos1Exclusions.has(part)) &&
pos1Parts.some((part) => !pos1Exclusions.has(part))
pos1Parts.some((part) => !pos1Exclusions.has(part));
if (!hasMixedFunctionContentParts) {
return false;
}
const normalizedReading = normalizeJlptTextForExclusion(token.reading);
const normalizedHeadwordReading = normalizeJlptTextForExclusion(token.headwordReading ?? '');
return (
!normalizedReading ||
!normalizedHeadwordReading ||
normalizedReading !== normalizedHeadwordReading
);
}
@@ -654,7 +668,7 @@ function resolveKnownWordReadingForMatch(
function computeTokenKnownStatus(
token: MergedToken,
isKnownWord: (text: string, reading?: string) => boolean,
isKnownWord: AnnotationStageDeps['isKnownWord'],
knownWordMatchMode: NPlusOneMatchMode,
): boolean {
const matchText = resolveKnownWordText(token.surface, token.headword, knownWordMatchMode);
@@ -668,7 +682,14 @@ function computeTokenKnownStatus(
return false;
}
return fallbackReading !== matchText.trim() && isKnownWord(fallbackReading);
// This fallback covers words mined in kana (token 大体, mined word だいたい),
// so the reading must only match mined word texts — the cache's reading-only
// index would let any kanji token match an unrelated note that shares its
// reading (渓谷/けいこく vs a mined 警告/けいこく).
return (
fallbackReading !== matchText.trim() &&
isKnownWord(fallbackReading, undefined, { allowReadingOnlyMatch: false })
);
}
function filterTokenFrequencyRank(
@@ -146,13 +146,17 @@ test('emits unparsed non-caption text as a token with surface headword', () => {
const tokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
assert.deepEqual(
tokens?.map((token) => ({ surface: token.surface, headword: token.headword })),
tokens?.map((token) => ({
surface: token.surface,
headword: token.headword,
isUnparsedRun: token.isUnparsedRun ?? false,
})),
[
{ surface: 'みんな', headword: '皆' },
{ surface: 'の', headword: 'の' },
{ surface: 'とこ', headword: '所' },
{ surface: '戻', headword: '戻' },
{ surface: 'ろ', headword: '櫓' },
{ surface: 'みんな', headword: '皆', isUnparsedRun: false },
{ surface: 'の', headword: 'の', isUnparsedRun: false },
{ surface: 'とこ', headword: '所', isUnparsedRun: false },
{ surface: '戻', headword: '戻', isUnparsedRun: true },
{ surface: 'ろ', headword: '櫓', isUnparsedRun: false },
],
);
});
@@ -239,6 +239,7 @@ export function mapYomitanParseResultItemToMergedTokens(
headword: string,
start: number,
end: number,
isUnparsedRun = false,
): void => {
tokens.push({
surface,
@@ -254,6 +255,7 @@ export function mapYomitanParseResultItemToMergedTokens(
const matchText = resolveKnownWordText(surface, headword, knownWordMatchMode);
return matchText ? isKnownWord(matchText) : false;
})(),
...(isUnparsedRun ? { isUnparsedRun: true } : {}),
});
};
@@ -275,10 +277,11 @@ export function mapYomitanParseResultItemToMergedTokens(
previousToken.reading += combinedReading;
previousToken.endPos = end;
} else if (shouldEmitUnparsedRunAsToken(combinedSurface)) {
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional). Keep it
// as a token with its surface as headword so it stays hoverable and counts in
// the n+1 math — matching what the embedded Yomitan actually returns.
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end);
// Yomitan couldn't parse this run (e.g. 戻ろ… truncated volitional, ぅ~
// elongations). Keep it as a token with its surface as headword so it stays
// hoverable, but flag it so annotation/N+1/vocab logic ignores it — there is
// no dictionary entry behind it.
pushToken(combinedSurface, combinedReading, combinedSurface, combinedStart, end, true);
}
} else {
hasDictionaryMatch = true;
@@ -474,6 +474,13 @@ export function shouldExcludeTokenFromSubtitleAnnotations(
token: MergedToken,
options: SubtitleAnnotationFilterOptions = {},
): boolean {
// No Yomitan dictionary entry backs this token (ぅ~ elongations, truncated
// inflections) — it exists only to stay hoverable and must never receive
// annotations or count in the N+1 math.
if (token.isUnparsedRun === true) {
return true;
}
const pos1Exclusions = resolvePos1Exclusions(options);
const pos2Exclusions = resolvePos2Exclusions(options);
const normalizedPos1 = normalizePosTag(token.pos1);
@@ -934,6 +934,7 @@ test('requestYomitanScanTokens keeps scanner metadata for matching spans when pa
headword: 'っ ',
startPos: 2,
endPos: 4,
isUnparsedRun: true,
},
{
surface: 'ミナト',
@@ -1889,6 +1890,364 @@ test('requestYomitanScanTokens accepts SubMiner character entries with structure
assert.equal((result as Array<{ isNameMatch?: boolean }>)[0]?.isNameMatch, true);
});
test('requestYomitanScanTokens greedily tokenizes character names before longer generic matches', async () => {
let scannerScript = '';
const deps = createDeps(async (script) => {
if (script.includes('termsFind')) {
scannerScript = script;
return [];
}
if (script.includes('optionsGetFull')) {
return {
profileCurrent: 0,
profiles: [
{
options: {
scanning: { length: 40 },
dictionaries: [
{ name: 'JMdict', enabled: true },
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
],
},
},
],
};
}
return null;
});
await requestYomitanScanTokens(
'美姫とヨータ',
deps,
{ error: () => undefined },
{ includeNameMatchMetadata: true },
);
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
const nameEntry = (term: string, reading: string) => ({
headwords: [
{
term,
reading,
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
},
],
definitions: [
{
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
},
],
});
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
headwords: [
{
term,
reading,
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
},
],
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
});
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
if (action !== 'termsFind') {
throw new Error(`unexpected action: ${action}`);
}
const text = (params as { text?: string } | undefined)?.text ?? '';
if (text.startsWith('美姫')) {
return { originalTextLength: 2, dictionaryEntries: [nameEntry('美姫', 'みき')] };
}
if (text.startsWith('とヨータ')) {
// Greedy generic match: とヨー normalizes to とよう (渡洋). Without the
// name pre-pass this consumes the ヨ of ヨータ.
return {
originalTextLength: 3,
dictionaryEntries: [jmdictEntry('渡洋', 'とよう', 'とヨー'), jmdictEntry('と', 'と', 'と')],
};
}
if (text.startsWith('ヨータ')) {
return { originalTextLength: 3, dictionaryEntries: [nameEntry('ヨータ', 'よーた')] };
}
if (text === 'と') {
return { originalTextLength: 1, dictionaryEntries: [jmdictEntry('と', 'と', 'と')] };
}
return { originalTextLength: 0, dictionaryEntries: [] };
});
assert.equal(Array.isArray(result), true);
assert.deepEqual(
(result as Array<Record<string, unknown>>).map(
({ surface, headword, startPos, endPos, isNameMatch }) => ({
surface,
headword,
startPos,
endPos,
isNameMatch,
}),
),
[
{ surface: '美姫', headword: '美姫', startPos: 0, endPos: 2, isNameMatch: true },
{ surface: 'と', headword: 'と', startPos: 2, endPos: 3, isNameMatch: false },
{ surface: 'ヨータ', headword: 'ヨータ', startPos: 3, endPos: 6, isNameMatch: true },
],
);
});
test('requestYomitanScanTokens lets a longer generic word beat a shorter name at the same position', async () => {
let scannerScript = '';
const deps = createDeps(async (script) => {
if (script.includes('termsFind')) {
scannerScript = script;
return [];
}
if (script.includes('optionsGetFull')) {
return {
profileCurrent: 0,
profiles: [
{
options: {
scanning: { length: 40 },
dictionaries: [
{ name: 'JMdict', enabled: true },
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
],
},
},
],
};
}
return null;
});
await requestYomitanScanTokens(
'空気変わって',
deps,
{ error: () => undefined },
{ includeNameMatchMetadata: true },
);
assert.match(scannerScript, /const greedyNameScanEnabled = true;/);
const nameEntry = (term: string, reading: string) => ({
headwords: [
{
term,
reading,
sources: [{ originalText: term, isPrimary: true, matchType: 'exact' }],
},
],
definitions: [
{
dictionary: 'SubMiner Character Dictionary (AniList 130298)',
dictionaryAlias: 'SubMiner Character Dictionary (AniList 130298)',
},
],
});
const jmdictEntry = (term: string, reading: string, originalText: string) => ({
headwords: [
{
term,
reading,
sources: [{ originalText, isPrimary: true, matchType: 'exact' }],
},
],
definitions: [{ dictionary: 'JMdict', dictionaryAlias: 'JMdict' }],
});
const result = await runInjectedYomitanScript(scannerScript, (action, params) => {
if (action !== 'termsFind') {
throw new Error(`unexpected action: ${action}`);
}
const text = (params as { text?: string } | undefined)?.text ?? '';
if (text.startsWith('空気')) {
// A character named 空 matches here, but the generic 空気 is longer and
// must win the position.
return {
originalTextLength: 2,
dictionaryEntries: [nameEntry('空', 'くう'), jmdictEntry('空気', 'くうき', '空気')],
};
}
if (text.startsWith('変わって')) {
return {
originalTextLength: 4,
dictionaryEntries: [jmdictEntry('変わる', 'かわる', '変わって')],
};
}
return { originalTextLength: 0, dictionaryEntries: [] };
});
assert.equal(Array.isArray(result), true);
assert.deepEqual(
(result as Array<Record<string, unknown>>).map(
({ surface, headword, startPos, endPos, isNameMatch }) => ({
surface,
headword,
startPos,
endPos,
isNameMatch,
}),
),
[
{ surface: '空気', headword: '空気', startPos: 0, endPos: 2, isNameMatch: false },
{ surface: '変わって', headword: '変わる', startPos: 2, endPos: 6, isNameMatch: false },
],
);
});
test('requestYomitanScanTokens skips greedy name scan without an enabled character dictionary', async () => {
let scannerScript = '';
const deps = createDeps(async (script) => {
if (script.includes('termsFind')) {
scannerScript = script;
return [];
}
if (script.includes('optionsGetFull')) {
return {
profileCurrent: 0,
profiles: [
{
options: {
scanning: { length: 40 },
dictionaries: [{ name: 'JMdict', enabled: true }],
},
},
],
};
}
return null;
});
await requestYomitanScanTokens(
'アクア',
deps,
{ error: () => undefined },
{ includeNameMatchMetadata: true },
);
assert.match(scannerScript, /const greedyNameScanEnabled = false;/);
});
test('requestYomitanScanTokens replaces parseText segmentation where greedy name tokens re-segment', async () => {
const deps = createDeps(async (script) => {
if (script.includes('optionsGetFull')) {
return {
profileCurrent: 0,
profiles: [
{
options: {
scanning: { length: 40 },
dictionaries: [
{ name: 'JMdict', enabled: true },
{ name: 'SubMiner Character Dictionary (AniList 130298)', enabled: true },
],
},
},
],
};
}
if (script.includes('parseText')) {
// parseText walks greedily too, so it merges と with ヨー into 渡洋 and
// strands the タ.
return [
{
source: 'scanning-parser',
index: 0,
content: [
[
{
text: '美姫',
reading: 'みき',
headwords: [[{ term: '美姫' }]],
},
],
[
{
text: 'とヨー',
reading: 'とよう',
headwords: [[{ term: '渡洋' }]],
},
],
[
{
text: 'タ',
reading: '',
},
],
],
},
];
}
return [
{
surface: '美姫',
reading: 'みき',
headword: '美姫',
headwordReading: 'みき',
startPos: 0,
endPos: 2,
isNameMatch: true,
},
{
surface: 'と',
reading: 'と',
headword: 'と',
headwordReading: 'と',
startPos: 2,
endPos: 3,
isNameMatch: false,
},
{
surface: 'ヨータ',
reading: 'ヨータ',
headword: 'ヨータ',
headwordReading: 'よーた',
startPos: 3,
endPos: 6,
isNameMatch: true,
},
];
});
const result = await requestYomitanScanTokens(
'美姫とヨータ',
deps,
{ error: () => undefined },
{ includeNameMatchMetadata: true },
);
assert.deepEqual(result, [
{
surface: '美姫',
reading: 'みき',
headword: '美姫',
headwordReading: 'みき',
startPos: 0,
endPos: 2,
isNameMatch: true,
},
{
surface: 'と',
reading: 'と',
headword: 'と',
headwordReading: 'と',
startPos: 2,
endPos: 3,
isNameMatch: false,
},
{
surface: 'ヨータ',
reading: 'ヨータ',
headword: 'ヨータ',
headwordReading: 'よーた',
startPos: 3,
endPos: 6,
isNameMatch: true,
},
]);
});
test('requestYomitanScanTokens preserves matched headword word classes', async () => {
let scannerScript = '';
const deps = createDeps(async (script) => {
@@ -57,6 +57,7 @@ export interface YomitanScanToken {
isNameMatch?: boolean;
frequencyRank?: number;
wordClasses?: string[];
isUnparsedRun?: boolean;
}
interface YomitanProfileMetadata {
@@ -73,6 +74,7 @@ export interface YomitanAddNoteResult {
}
const DEFAULT_YOMITAN_SCAN_LENGTH = 40;
const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
const yomitanProfileMetadataByWindow = new WeakMap<BrowserWindow, YomitanProfileMetadata>();
const yomitanProfileDiagnosticsLoggedByWindow = new WeakSet<BrowserWindow>();
const yomitanFrequencyCacheByWindow = new WeakMap<
@@ -109,12 +111,39 @@ function scanTokenSpanKey(token: YomitanScanToken): string {
return `${token.startPos}:${token.endPos}:${token.surface}`;
}
// Maps a parse-selected token to the scanner-token shape carried out of the
// parser runtime. Shared by both selectYomitanParseTokens fallback paths so the
// projected fields stay in sync as the shape changes.
function toYomitanScanToken(token: {
surface: string;
reading: string;
headword: string;
startPos: number;
endPos: number;
isUnparsedRun?: boolean;
}): YomitanScanToken {
return {
surface: token.surface,
reading: token.reading,
headword: token.headword,
startPos: token.startPos,
endPos: token.endPos,
...(token.isUnparsedRun === true ? { isUnparsedRun: true } : {}),
};
}
// parseText segmentation is authoritative (it emits filler chunks for text the
// termsFind scanner skips), but only the termsFind scanner carries annotation
// metadata (isNameMatch, frequencyRank, headwordReading, wordClasses). Graft
// scanner tokens onto the parseText segmentation per matching span so one
// unmatched chunk degrades only itself instead of dropping the whole line's
// metadata.
//
// Exception: character-name tokens. The greedy name scan can re-segment text
// around a name (e.g. とヨータ → と + ヨータ instead of とヨー + タ), so
// parseText segmentation cannot be authoritative there. Each name span is
// expanded until it aligns with token boundaries in both segmentations, then
// the parse tokens inside are replaced with the scanner tokens.
function mergeScannerTokensIntoParseTokens(
parseScanTokens: YomitanScanToken[],
scannerTokens: YomitanScanToken[],
@@ -123,8 +152,44 @@ function mergeScannerTokensIntoParseTokens(
for (const token of scannerTokens) {
scannerTokensBySpan.set(scanTokenSpanKey(token), token);
}
const graftedTokens = parseScanTokens.map(
(token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token,
);
return parseScanTokens.map((token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token);
const nameTokens = scannerTokens.filter((token) => token.isNameMatch === true);
if (nameTokens.length === 0) {
return graftedTokens;
}
const regions = nameTokens.map((token) => ({ start: token.startPos, end: token.endPos }));
const allTokens = [...parseScanTokens, ...scannerTokens];
let expanded = true;
while (expanded) {
expanded = false;
for (const region of regions) {
for (const token of allTokens) {
const overlaps = token.startPos < region.end && token.endPos > region.start;
const extendsBeyond = token.startPos < region.start || token.endPos > region.end;
if (overlaps && extendsBeyond) {
region.start = Math.min(region.start, token.startPos);
region.end = Math.max(region.end, token.endPos);
expanded = true;
}
}
}
}
const isInsideNameRegion = (token: YomitanScanToken): boolean =>
regions.some((region) => token.startPos >= region.start && token.endPos <= region.end);
const merged = graftedTokens.filter((token) => !isInsideNameRegion(token));
for (const token of scannerTokens) {
if (isInsideNameRegion(token)) {
merged.push(token);
}
}
merged.sort((a, b) => a.startPos - b.startPos || a.endPos - b.endPos);
return merged;
}
function makeTermReadingCacheKey(term: string, reading: string | null): string {
@@ -1110,8 +1175,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
}
return best;
}
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
function normalizeWordClasses(headword) {
function normalizeWordClasses(headword) {
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
return classes.length > 0 ? classes : undefined;
@@ -1152,7 +1216,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
return false;
}
return getDictionaryEntryNames(entry).some((name) => name.startsWith("SubMiner Character Dictionary"));
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
}
function parseSubMinerMediaIdFromString(value) {
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
@@ -1160,7 +1224,7 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
const parsed = Number.parseInt(imageMatch[1], 10);
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
}
const titleMatch = value.match(/SubMiner Character Dictionary[^\d]*(?:AniList\s*)?(\d+)/i);
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
if (titleMatch) {
const parsed = Number.parseInt(titleMatch[1], 10);
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
@@ -1220,7 +1284,43 @@ const YOMITAN_SCANNING_HELPERS = String.raw`
}
const mediaIds = getSubMinerMediaIds(entry);
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
}
function findLongestNameMatch(dictionaryEntries, textWindow) {
let best = null;
for (const dictionaryEntry of dictionaryEntries || []) {
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
const headword = headwords[headwordIndex];
for (const src of headword?.sources || []) {
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
if (best === null || originalText.length > best.sourceLength) {
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
}
}
}
}
return best;
}
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
let best = 0;
for (const dictionaryEntry of dictionaryEntries || []) {
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
for (const headword of headwords) {
for (const src of headword?.sources || []) {
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
if (originalText.length > best) { best = originalText.length; }
}
}
}
return best;
}
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
const currentMediaDictionaryEntries =
currentCharacterDictionaryMediaId === null
? (dictionaryEntries || [])
@@ -1267,6 +1367,7 @@ function buildYomitanScanningScript(
profileIndex: number,
scanLength: number,
includeNameMatchMetadata: boolean,
greedyNameScanEnabled: boolean,
currentCharacterDictionaryMediaId: number | null,
dictionaryPriorityByName: Record<string, number>,
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>,
@@ -1293,6 +1394,7 @@ function buildYomitanScanningScript(
});
${YOMITAN_SCANNING_HELPERS}
const includeNameMatchMetadata = ${includeNameMatchMetadata ? 'true' : 'false'};
const greedyNameScanEnabled = ${greedyNameScanEnabled ? 'true' : 'false'};
const currentCharacterDictionaryMediaId = ${
currentCharacterDictionaryMediaId !== null
? String(currentCharacterDictionaryMediaId)
@@ -1303,26 +1405,17 @@ ${YOMITAN_SCANNING_HELPERS}
const text = ${JSON.stringify(text)};
const details = {matchType: "exact", deinflect: true};
const tokens = [];
async function findTokenAt(position, windowLength) {
const codePoint = text.codePointAt(position);
const character = String.fromCodePoint(codePoint);
const termsFindCache = new Map();
async function termsFindAt(position, windowLength) {
const cacheKey = position + ":" + windowLength;
const cached = termsFindCache.get(cacheKey);
if (cached) { return cached; }
const substring = text.substring(position, position + windowLength);
const result = await invoke("termsFind", { text: substring, details, optionsContext: { index: ${profileIndex} } });
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
return { token: null, matchedLength: 0 };
}
const source = substring.substring(0, originalTextLength);
const preferredHeadword = getPreferredHeadword(
dictionaryEntries,
source,
dictionaryPriorityByName,
dictionaryFrequencyModeByName
);
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
return { token: null, matchedLength: originalTextLength };
}
termsFindCache.set(cacheKey, result);
return result;
}
function buildScanToken(position, source, preferredHeadword) {
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
const tokenPayload = {
@@ -1331,7 +1424,7 @@ ${YOMITAN_SCANNING_HELPERS}
headword: preferredHeadword.term,
headwordReading: reading || undefined,
startPos: position,
endPos: position + originalTextLength,
endPos: position + source.length,
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
frequencyRank:
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
@@ -1341,17 +1434,93 @@ ${YOMITAN_SCANNING_HELPERS}
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
tokenPayload.wordClasses = preferredHeadword.wordClasses;
}
return { token: tokenPayload, matchedLength: originalTextLength };
return tokenPayload;
}
async function findTokenAt(position, windowLength) {
const codePoint = text.codePointAt(position);
const character = String.fromCodePoint(codePoint);
const result = await termsFindAt(position, windowLength);
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
return { token: null, matchedLength: 0 };
}
const source = text.substring(position, position + originalTextLength);
const preferredHeadword = getPreferredHeadword(
dictionaryEntries,
source,
dictionaryPriorityByName,
dictionaryFrequencyModeByName
);
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
return { token: null, matchedLength: originalTextLength };
}
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
}
// Greedy name pre-pass: character-name matches claim their spans before
// the left-to-right walk, so a longer generic match starting earlier
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
const nameTokens = [];
if (greedyNameScanEnabled) {
let namePos = 0;
while (namePos < text.length) {
const codePoint = text.codePointAt(namePos);
if (!isCodePointJapanese(codePoint)) {
namePos += String.fromCodePoint(codePoint).length;
continue;
}
const result = await termsFindAt(namePos, ${scanLength});
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
const textWindow = text.substring(namePos, namePos + ${scanLength});
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
// A name only claims its span when no strictly longer generic word
// starts at the same position (a character named 空 must not split
// 空気). Ties go to the name. Generic matches that start earlier and
// overlap the name are still blocked by the reservation.
if (
!nameMatch ||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
) {
namePos += String.fromCodePoint(codePoint).length;
continue;
}
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
nameTokens.push(buildScanToken(namePos, source, {
term: nameMatch.headword.term,
reading: nameMatch.headword.reading,
wordClasses: normalizeWordClasses(nameMatch.headword),
isNameMatch: true,
frequencyRank: getBestFrequencyRank(
nameMatch.dictionaryEntry,
nameMatch.headwordIndex,
dictionaryPriorityByName,
dictionaryFrequencyModeByName
)
}));
namePos += nameMatch.sourceLength;
}
}
let i = 0;
let nameIndex = 0;
while (i < text.length) {
let attempt = await findTokenAt(i, ${scanLength});
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
if (nextNameToken && nextNameToken.startPos === i) {
tokens.push(nextNameToken);
i = nextNameToken.endPos;
nameIndex += 1;
continue;
}
// Cap the window at the next reserved name span so a generic match
// cannot consume into it.
const windowLength = nextNameToken ? Math.min(${scanLength}, nextNameToken.startPos - i) : ${scanLength};
let attempt = await findTokenAt(i, windowLength);
// Yomitan text normalization can consume characters (whitespace,
// punctuation) beyond the matched term, leaving no headword whose
// source equals the consumed text. Retry with shorter windows so a
// valid prefix term (e.g. a character name before a paren) still
// tokenizes instead of the position being skipped.
let retryLength = Math.min(attempt.matchedLength, ${scanLength}) - 1;
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
while (!attempt.token && retryLength >= 1) {
const retry = await findTokenAt(i, retryLength);
if (retry.token) {
@@ -1488,18 +1657,17 @@ export async function requestYomitanScanTokens(
const parseResults = await requestYomitanParseResults(text, deps, logger);
const selectedParseTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
const parseScanTokens =
selectedParseTokens?.map((token) => ({
surface: token.surface,
reading: token.reading,
headword: token.headword,
startPos: token.startPos,
endPos: token.endPos,
})) ?? null;
const parseScanTokens = selectedParseTokens?.map(toYomitanScanToken) ?? null;
const metadata = await requestYomitanProfileMetadata(parserWindow, logger);
const profileIndex = metadata?.profileIndex ?? 0;
const scanLength = metadata?.scanLength ?? DEFAULT_YOMITAN_SCAN_LENGTH;
const includeNameMatchMetadata = options?.includeNameMatchMetadata === true;
const greedyNameScanEnabled =
includeNameMatchMetadata &&
(metadata?.dictionaries ?? []).some((name) =>
name.startsWith(CHARACTER_DICTIONARY_TITLE_PREFIX),
);
try {
const rawResult = await parserWindow.webContents.executeJavaScript(
@@ -1507,7 +1675,8 @@ export async function requestYomitanScanTokens(
text,
profileIndex,
scanLength,
options?.includeNameMatchMetadata === true,
includeNameMatchMetadata,
greedyNameScanEnabled,
typeof options?.currentCharacterDictionaryMediaId === 'number' &&
Number.isFinite(options.currentCharacterDictionaryMediaId) &&
options.currentCharacterDictionaryMediaId > 0
@@ -1526,15 +1695,7 @@ export async function requestYomitanScanTokens(
}
if (Array.isArray(rawResult)) {
const selectedTokens = selectYomitanParseTokens(rawResult, () => false, 'headword');
return (
selectedTokens?.map((token) => ({
surface: token.surface,
reading: token.reading,
headword: token.headword,
startPos: token.startPos,
endPos: token.endPos,
})) ?? null
);
return selectedTokens?.map(toYomitanScanToken) ?? null;
}
if (parseScanTokens && parseScanTokens.length > 0) {
return parseScanTokens;
+2 -1
View File
@@ -4652,7 +4652,8 @@ const {
setYomitanParserInitPromise: (promise) => {
appState.yomitanParserInitPromise = promise;
},
isKnownWord: (text, reading) => Boolean(appState.ankiIntegration?.isKnownWord(text, reading)),
isKnownWord: (text, reading, options) =>
Boolean(appState.ankiIntegration?.isKnownWord(text, reading, options)),
recordLookup: (hit) => {
ensureImmersionTrackerStarted();
appState.immersionTracker?.recordLookup(hit);
@@ -149,6 +149,42 @@ test('createConfigHotReloadAppliedHandler applies safe Anki, annotation, and log
assert.ok(calls.includes('broadcast:config:hot-reload'));
});
test('createConfigHotReloadAppliedHandler applies only changed Anki media options', () => {
const config = deepCloneConfig(DEFAULT_CONFIG);
config.ankiConnect.media.normalizeAudio = false;
config.ankiConnect.media.mirrorMpvVolume = false;
const ankiPatches: unknown[] = [];
const applyHotReload = createConfigHotReloadAppliedHandler({
setKeybindings: () => undefined,
setSessionBindings: () => undefined,
refreshGlobalAndOverlayShortcuts: () => undefined,
setSecondarySubMode: () => undefined,
broadcastToOverlayWindows: () => undefined,
applyAnkiRuntimeConfigPatch: (patch) => ankiPatches.push(patch),
});
applyHotReload(
{
hotReloadFields: ['ankiConnect.media.normalizeAudio'],
restartRequiredFields: [],
},
config,
);
applyHotReload(
{
hotReloadFields: ['ankiConnect.media.mirrorMpvVolume'],
restartRequiredFields: [],
},
config,
);
assert.deepEqual(ankiPatches, [
{ media: { normalizeAudio: false } },
{ media: { mirrorMpvVolume: false } },
]);
});
test('buildConfigHotReloadPayload includes independent primary subtitle mode', () => {
const config = deepCloneConfig(DEFAULT_CONFIG);
config.subtitleStyle.primaryDefaultMode = 'hover';
@@ -93,6 +93,16 @@ function buildAnkiRuntimeConfigPatch(
if (diff.hotReloadFields.includes('ankiConnect.deck')) {
patch.deck = config.ankiConnect.deck;
}
const mediaPatch: NonNullable<AnkiConnectConfig['media']> = {};
if (diff.hotReloadFields.includes('ankiConnect.media.normalizeAudio')) {
mediaPatch.normalizeAudio = config.ankiConnect.media.normalizeAudio;
}
if (diff.hotReloadFields.includes('ankiConnect.media.mirrorMpvVolume')) {
mediaPatch.mirrorMpvVolume = config.ankiConnect.media.mirrorMpvVolume;
}
if (Object.keys(mediaPatch).length > 0) {
patch.media = mediaPatch;
}
if (hasAnyHotReloadField(diff, ['ankiConnect.knownWords'])) {
patch.knownWords = config.ankiConnect.knownWords;
}
@@ -37,8 +37,8 @@ export function createBuildTokenizerDepsMainHandler(deps: TokenizerMainDeps) {
getYomitanParserInitPromise: () => deps.getYomitanParserInitPromise(),
setYomitanParserInitPromise: (promise: Promise<boolean> | null) =>
deps.setYomitanParserInitPromise(promise),
isKnownWord: (text: string, reading?: string) => {
const hit = deps.isKnownWord(text, reading);
isKnownWord: (text, reading, options) => {
const hit = deps.isKnownWord(text, reading, options);
deps.recordLookup(hit);
return hit;
},
+48
View File
@@ -181,6 +181,54 @@ test('generateAudio can preserve raw sentence audio loudness', async () => {
});
});
test('generateAudio applies mpv volume after loudness normalization', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 10, 12, 0, null, true, 0.42);
const args = readFfmpegArgs(argsPath);
assert.equal(args[args.indexOf('-af') + 1], 'loudnorm=I=-23:TP=-2:LRA=11,volume=0.42');
});
});
test('generateAudio limits amplified mpv volume after applying gain', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 10, 12, 0, null, true, 2);
const args = readFfmpegArgs(argsPath);
assert.equal(
args[args.indexOf('-af') + 1],
'loudnorm=I=-23:TP=-2:LRA=11,volume=2,alimiter=limit=0.891251:level=false',
);
});
});
test('generateAudio applies mpv volume without loudness normalization', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 10, 12, 0, null, false, 0.75);
const args = readFfmpegArgs(argsPath);
assert.equal(args[args.indexOf('-af') + 1], 'volume=0.75');
});
});
test('generateAudio omits no-op mpv volume filters', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 10, 12, 0, null, false, 1);
const args = readFfmpegArgs(argsPath);
assert.equal(args.includes('-af'), false);
});
});
test('generateAudio preserves a zero numeric mpv volume', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 10, 12, 0, null, false, 0);
const args = readFfmpegArgs(argsPath);
assert.equal(args[args.indexOf('-af') + 1], 'volume=0');
});
});
test('generateAudio clips leading padding without adding it to trailing duration', async () => {
await withStubbedFfmpeg(async (generator, argsPath) => {
await generator.generateAudio('/video.mp4', 0.2, 1.2, 0.5);

Some files were not shown because too many files have changed in this diff Show More