mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-07-27 16:49:51 -07:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sudacode
|
8797719a09
|
||
|
sudacode
|
8acc78cc1c | ||
|
sudacode
|
321461c50f |
@@ -4,3 +4,4 @@ release
|
||||
coverage
|
||||
vendor
|
||||
*.log
|
||||
src/core/services/tokenizer/__fixtures__/golden/*.json
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
type: internal
|
||||
area: tokenizer
|
||||
|
||||
- Added a golden-file regression corpus for the tokenizer/annotation pipeline: recorded Yomitan backend responses and MeCab tokens replay through the real tokenizeSubtitle pipeline in bun tests without Electron or dictionaries.
|
||||
- Added `record-tokenizer-fixture:electron` script to capture new fixtures from a live Yomitan/MeCab session, with flags for known words, JLPT levels, and annotation toggles.
|
||||
- Seeded eleven fixtures covering the #147–#156 regression classes (grammar-helper suppression, lexical くれる, kanji non-independent nouns, N+1 targeting, reading collisions, unparsed runs, ordinal/honorific prefixes).
|
||||
- Added `compare-yomitan-api:electron` script that diffs SubMiner tokenization against a stock Yomitan instance via the yomitan-api bridge (segmentation, readings, headword forms).
|
||||
@@ -0,0 +1,4 @@
|
||||
type: added
|
||||
area: launcher
|
||||
|
||||
- Added `subminer sync <host>` to merge immersion stats and watch history between two machines over SSH, with `--push` and `--pull` for one-way insert-only transfers. Each side snapshots its database (`VACUUM INTO`), snapshots are exchanged with `scp`, and each machine merges the other's data as an insert-only union keyed on session UUIDs / video keys / series title keys, so re-syncing is idempotent and nothing is double-counted. Lifetime totals and daily/monthly rollups are updated incrementally (history older than the session retention window is preserved); remote-only historical rollups are copied only when they do not conflict with retained local session history. `subminer sync --snapshot <file>` and `subminer sync --merge <file>` expose the underlying steps for manual transfers. The command refuses to run while the stats daemon or a live mpv session is active (`--force` overrides), ignores stale mpv socket files, keeps the guard in place through local/remote merges, supplies standard SubMiner and Bun paths to non-interactive SSH commands, verifies the remote launcher starts, reports remote stderr on failures, and aborts on stats schema version mismatches.
|
||||
@@ -41,7 +41,7 @@ The update flow:
|
||||
1. **Title detection** -- SubMiner extracts the anime title, season, and episode number from the media filename and path. Season folders such as `Season 2` are treated as a strong season signal. SubMiner tries [`guessit`](https://github.com/guessit-io/guessit) first for accurate parsing, then falls back to an internal filename parser if guessit is unavailable.
|
||||
2. **AniList search** -- The detected title is searched against the AniList GraphQL API. For season 2 and later files, SubMiner searches the season-specific title first, then falls back to the base title. SubMiner picks the best match by comparing titles (romaji, English, native) and filtering by episode count.
|
||||
3. **Progress check** -- SubMiner fetches your current list entry for the matched media. The media must already be in Planning or Watching; otherwise SubMiner shows an MPV message explaining that the update is not possible. If your recorded progress already meets or exceeds the detected episode, the update is skipped.
|
||||
4. **Mutation** -- A `SaveMediaListEntry` mutation sets the new progress and marks the entry as `CURRENT`.
|
||||
4. **Mutation** -- A `SaveMediaListEntry` mutation sets the new progress and marks the entry as `CURRENT`, or `COMPLETED` when the watched episode is the final episode of the season (the "already at this progress" skip is bypassed for the final episode so completion still lands).
|
||||
|
||||
## Update Queue and Retry
|
||||
|
||||
@@ -81,7 +81,6 @@ All AniList API calls go through a shared rate limiter that enforces a sliding w
|
||||
"enabled": true,
|
||||
"accessToken": "",
|
||||
"characterDictionary": {
|
||||
"enabled": false,
|
||||
"maxLoaded": 3,
|
||||
"profileScope": "all",
|
||||
"collapsibleSections": {
|
||||
@@ -99,10 +98,12 @@ All AniList API calls go through a shared rate limiter that enforces a sliding w
|
||||
| `enabled` | `true`, `false` | Enable AniList post-watch progress updates (default: `false`) |
|
||||
| `accessToken` | string | Explicit AniList access token override; when blank, SubMiner uses the stored encrypted token (default: `""`) |
|
||||
| `characterDictionary.maxLoaded` | number | Number of recent media snapshots kept in the merged dictionary (default: `3`) |
|
||||
| `characterDictionary.refreshTtlHours` | number | Hours before a cached media snapshot is refreshed (default: `168`, clamped to 1–8760) |
|
||||
| `characterDictionary.evictionPolicy` | `"delete"`, `"disable"` | What happens to snapshots evicted beyond `maxLoaded` (default: `"delete"`) |
|
||||
| `characterDictionary.profileScope` | `"all"`, `"active"` | Apply dictionary to all Yomitan profiles or only the active one |
|
||||
| `characterDictionary.collapsibleSections.*` | `true`, `false` | Control which dictionary entry sections start expanded |
|
||||
|
||||
See the [Character Dictionary](/character-dictionary) page for full details on the character dictionary feature, including name generation, matching, auto-sync lifecycle, and dictionary entry format. Character dictionary sync follows `subtitleStyle.nameMatchEnabled`.
|
||||
There is no `characterDictionary.enabled` key: character dictionary sync is enabled by `subtitleStyle.nameMatchEnabled`. See the [Character Dictionary](/character-dictionary) page for full details on the character dictionary feature, including name generation, matching, auto-sync lifecycle, and dictionary entry format.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
|
||||
@@ -6,12 +6,12 @@ Intro detection runs in the SubMiner app over the mpv IPC socket. It is availabl
|
||||
|
||||
## Setup
|
||||
|
||||
AniSkip is opt-in. Enable it in your config:
|
||||
AniSkip is enabled by default. Disable it or change the skip key in your config:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"mpv": {
|
||||
"aniskipEnabled": true,
|
||||
"aniskipEnabled": true, // default: true
|
||||
"aniskipButtonKey": "TAB",
|
||||
},
|
||||
}
|
||||
@@ -37,7 +37,9 @@ On each local file load:
|
||||
4. If an interval is found, SubMiner adds `AniSkip Intro Start` and `AniSkip Intro End` chapter markers to the current file and binds the skip key (`mpv.aniskipButtonKey`, default `TAB`).
|
||||
5. At the start of the intro, an OSD prompt appears for 3 seconds: `You can skip by pressing TAB` (reflects your configured key). Pressing the key at any point during the intro seeks to the intro end.
|
||||
|
||||
Results are cached per file for the app session. Reload detection is also handled: if mpv reloads the same file, SubMiner re-applies the chapter markers without a new API lookup.
|
||||
When a custom key (other than `TAB` or `y-k`) is configured, the legacy `y-k` chord is also bound as a fallback skip trigger.
|
||||
|
||||
Results are cached per file for the app session; only definitive "no intro found" results are cached, so transient lookup failures are retried on the next file load. Reload detection is also handled: if mpv reloads the same file, SubMiner re-applies the chapter markers without a new API lookup.
|
||||
|
||||
## Triggering from mpv
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ In both modes, the enrichment workflow is the same:
|
||||
4. Fills the translation field from the secondary subtitle or AI.
|
||||
5. Writes metadata to the miscInfo field.
|
||||
|
||||
Polling mode uses the query `"deck:<ankiConnect.deck>" added:1` to find recently added cards. If no deck is configured, it uses Yomitan's current mining deck when available; otherwise it searches all decks. In Settings, the AnkiConnect deck dropdown auto-fills and persists Yomitan's current mining deck when available, then falls back to the decks reported by AnkiConnect.
|
||||
Polling mode uses the query `"deck:<ankiConnect.deck>" added:1` to find recently added cards. If no deck is configured, it searches all decks (`added:1`). In Settings, the AnkiConnect deck dropdown auto-fills and persists Yomitan's current mining deck when available, then falls back to the decks reported by AnkiConnect; stats-dashboard mining also falls back to Yomitan's mining deck when `ankiConnect.deck` is empty.
|
||||
Known-word sync scope is controlled by `ankiConnect.knownWords.decks`.
|
||||
|
||||
### Proxy Mode Setup (Yomitan / Texthooker)
|
||||
@@ -56,12 +56,12 @@ Known-word sync scope is controlled by `ankiConnect.knownWords.decks`.
|
||||
|
||||
Then point Yomitan/clients to `http://127.0.0.1:8766` instead of `8765`.
|
||||
|
||||
When SubMiner loads the bundled Yomitan extension, it also attempts to update the **default Yomitan profile** (`profiles[0].options.anki.server`) to the active SubMiner endpoint:
|
||||
When SubMiner loads the bundled Yomitan extension, it also attempts to update the **currently active Yomitan profile**'s Anki server to the active SubMiner endpoint (falling back to `profiles[0]` if the active-profile index is invalid):
|
||||
|
||||
- proxy URL when `ankiConnect.proxy.enabled` is `true`
|
||||
- direct `ankiConnect.url` when proxy mode is disabled
|
||||
|
||||
To avoid clobbering custom setups, this auto-update only changes the default profile when its current server is blank or the stock Yomitan default (`http://127.0.0.1:8765`).
|
||||
To avoid clobbering custom setups, this auto-update only changes the profile when its current server is blank or the stock Yomitan default (`http://127.0.0.1:8765`).
|
||||
|
||||
For browser-based Yomitan or other external clients (for example Texthooker in a normal browser profile), set their Anki server to the same proxy URL separately: `http://127.0.0.1:8766` (or your configured `proxy.host` + `proxy.port`).
|
||||
|
||||
@@ -81,7 +81,7 @@ In Yomitan, go to Settings → Profile and:
|
||||
3. Set server to `http://127.0.0.1:8766` (or your configured proxy URL).
|
||||
4. Save and make that profile active when using SubMiner.
|
||||
|
||||
This is only for non-bundled, external/browser Yomitan or other clients. The bundled profile auto-update logic only targets `profiles[0]` when it is blank or still default.
|
||||
This is only for non-bundled, external/browser Yomitan or other clients. The bundled profile auto-update logic only targets the active profile when its server is blank or still default.
|
||||
|
||||
### Proxy Troubleshooting (quick checks)
|
||||
|
||||
@@ -101,10 +101,11 @@ curl -sS http://127.0.0.1:8766 \
|
||||
-d '{"action":"version","version":2}'
|
||||
```
|
||||
|
||||
3. Check both log sinks:
|
||||
3. Check the log sinks in `~/.config/SubMiner/logs/`:
|
||||
|
||||
- Launcher/mpv-integrated log: `~/.cache/SubMiner/mp.log`
|
||||
- App runtime log: `~/.config/SubMiner/logs/SubMiner-YYYY-MM-DD.log`
|
||||
- App runtime log: `app-YYYY-MM-DD.log`
|
||||
- Launcher log: `launcher-YYYY-MM-DD.log`
|
||||
- mpv log: `mpv-YYYY-MM-DD.log`
|
||||
|
||||
4. Ensure config JSONC is valid and logging shape is correct:
|
||||
|
||||
@@ -133,7 +134,9 @@ SubMiner maps its data to your Anki note fields. Configure these under `ankiConn
|
||||
}
|
||||
```
|
||||
|
||||
Field names must match your Anki note type exactly (case-sensitive). If a configured field does not exist on the note type, SubMiner skips it without error.
|
||||
Field names are matched against your Anki note type case-insensitively (an exact match wins, then a lowercase comparison). If a configured field does not exist on the note type, SubMiner skips it without error.
|
||||
|
||||
Two related options live alongside `fields`: `ankiConnect.deck` (target deck; empty falls back as described above) and `ankiConnect.tags` (tags added to mined cards, default `["SubMiner"]`; set `[]` to disable tagging). The `miscInfo` content is controlled by `ankiConnect.metadata.pattern` (default `[SubMiner] %f (%t)`; tokens: `%f` filename, `%F` filename with extension, `%t` timestamp, `%T` timestamp with milliseconds, `<br>` newline).
|
||||
|
||||
### Minimal Config
|
||||
|
||||
@@ -169,7 +172,7 @@ Audio is extracted from the video file using the subtitle's start and end timest
|
||||
}
|
||||
```
|
||||
|
||||
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream. Generated sentence audio is loudness-normalized by default during extraction; set `normalizeAudio` to `false` to keep raw source loudness. Changing this setting applies to the next extraction without restarting SubMiner.
|
||||
Output format: MP3 at 44100 Hz. If the video has multiple audio streams, SubMiner uses the active stream. Generated sentence audio is loudness-normalized to -23 LUFS by default during extraction; set `normalizeAudio` to `false` to keep raw source loudness. When subtitle timing is missing, clips fall back to `media.fallbackDuration` seconds (default `3`). Changing these settings applies to the next extraction without restarting SubMiner.
|
||||
|
||||
`mirrorMpvVolume` is also enabled by default. Immediately before extracting each playback-overlay card's audio, SubMiner reads mpv's numeric `volume` and applies mpv's cubic software-volume curve after loudness normalization. For example, mpv volume `50` produces `0.5³ = 0.125` gain. Amplified output above mpv volume `100` is limited to a `-1 dBFS` ceiling before MP3 encoding to prevent clipping. It ignores mpv's separate `mute` state. If the volume property is missing, invalid, or unavailable, extraction continues with unity scaling; disabling this option skips the query and volume filter. Changing this setting applies to the next extraction without restarting SubMiner. YouTube cards queued for a background media-cache download retain the volume captured when the card was mined. Stats-dashboard mining does not currently have access to the active mpv property client, so it does not apply mpv volume scaling.
|
||||
|
||||
@@ -186,8 +189,8 @@ A single frame is captured at the current playback position.
|
||||
"imageType": "static",
|
||||
"imageFormat": "jpg", // "jpg", "png", or "webp"
|
||||
"imageQuality": 92, // 1–100
|
||||
"imageMaxWidth": null, // optional, preserves aspect ratio
|
||||
"imageMaxHeight": null
|
||||
"imageMaxWidth": 0, // 0 = preserve source resolution
|
||||
"imageMaxHeight": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -203,13 +206,13 @@ Instead of a static screenshot, SubMiner can generate an animated AVIF covering
|
||||
"imageType": "avif",
|
||||
"animatedFps": 10,
|
||||
"animatedMaxWidth": 640,
|
||||
"animatedMaxHeight": null,
|
||||
"animatedMaxHeight": 0, // 0 = preserve aspect ratio
|
||||
"animatedCrf": 35 // 0–63, lower = better quality
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`) in your FFmpeg build. Generation timeout is 60 seconds.
|
||||
Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`) in your FFmpeg build. Generation timeout is 60 seconds. `media.syncAnimatedImageToWordAudio` (default `true`) prepends a frozen first frame matching the existing word-audio duration, so the motion starts together with the sentence audio.
|
||||
|
||||
### Behavior Options
|
||||
|
||||
@@ -220,6 +223,7 @@ Animated AVIF requires an AV1 encoder (`libaom-av1`, `libsvtav1`, or `librav1e`)
|
||||
"overwriteImage": true, // replace existing image, or append
|
||||
"mediaInsertMode": "append", // "append" or "prepend" to field content
|
||||
"autoUpdateNewCards": true, // auto-update when new card detected
|
||||
"highlightWord": true, // bold the mined word inside the sentence field
|
||||
"notificationType": "overlay" // "overlay", "system", "both", or "none"
|
||||
}
|
||||
}
|
||||
@@ -269,14 +273,14 @@ The built-in translation request asks for English output by default. Customize t
|
||||
SubMiner can create standalone sentence cards (without a word/expression) using a separate note type. This is designed for use with [Lapis](https://github.com/donkuri/Lapis) and similar sentence-focused note types.
|
||||
|
||||
::: warning Required config
|
||||
Sentence card creation and audio card marking both require `ankiConnect.isLapis.enabled: true` and a valid `sentenceCardModel` pointing to your Lapis/Kiku note type. Without this, the `Ctrl/Cmd+S` and `Ctrl/Cmd+Shift+A` shortcuts will not create cards.
|
||||
Sentence card creation and audio card marking require a non-empty `ankiConnect.isLapis.sentenceCardModel` naming a note type that exists in Anki (default: `"Lapis"`). If the model is empty or missing, the `Ctrl/Cmd+S` and `Ctrl/Cmd+Shift+A` shortcuts will not create cards.
|
||||
:::
|
||||
|
||||
```jsonc
|
||||
"ankiConnect": {
|
||||
"isLapis": {
|
||||
"enabled": true,
|
||||
"sentenceCardModel": "Japanese sentences"
|
||||
"sentenceCardModel": "Lapis" // default; point at your Lapis/Kiku note type
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -303,25 +307,28 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
|
||||
|
||||
**Disabled** (`"disabled"`): No duplicate detection. Each card is independent.
|
||||
|
||||
**Auto** (`"auto"`): When a duplicate expression is found, SubMiner merges the new card into the existing one automatically. Both sentences, audio clips, and images are preserved, and exact duplicate values are collapsed to one entry. If `deleteDuplicateInAuto` is true, the new card is deleted after merging.
|
||||
**Auto** (`"auto"`): When a duplicate expression is found, SubMiner merges the new card into the existing one automatically. Both cards' sentences, audio clips, and images are preserved as grouped entries. If `deleteDuplicateInAuto` is true, the new card is deleted after merging.
|
||||
|
||||
**Manual** (`"manual"`): A modal appears in the overlay showing both cards. You choose which card to keep, preview the merge result, then confirm. The modal has a 90-second timeout, after which it cancels automatically.
|
||||
|
||||
### What Gets Merged
|
||||
|
||||
| Field | Merge behavior |
|
||||
| -------- | --------------------------------------------------------------- |
|
||||
| Sentence | Both sentences preserved (exact duplicate text is deduplicated) |
|
||||
| Audio | Both `[sound:...]` entries kept (exact duplicates deduplicated) |
|
||||
| Image | Both images kept (exact duplicates deduplicated) |
|
||||
| Field | Merge behavior |
|
||||
| -------- | ---------------------------------------- |
|
||||
| Sentence | Both cards' sentences kept as grouped entries |
|
||||
| Audio | Both cards' `[sound:...]` entries kept |
|
||||
| Image | Both cards' images kept |
|
||||
|
||||
Identical values from both cards are kept as separate grouped entries; the merge does not deduplicate.
|
||||
|
||||
### Keyboard Shortcuts in the Modal
|
||||
|
||||
| Key | Action |
|
||||
| --------- | ---------------------------------- |
|
||||
| `1` / `2` | Select card 1 or card 2 to keep |
|
||||
| `Enter` | Confirm selection |
|
||||
| `Esc` | Cancel (keep both cards unchanged) |
|
||||
| Key | Action |
|
||||
| ----------- | ---------------------------------- |
|
||||
| `1` / `2` | Select card 1 or card 2 to keep |
|
||||
| `Enter` | Confirm selection |
|
||||
| `Backspace` | Go back from the merge preview |
|
||||
| `Esc` | Cancel (keep both cards unchanged) |
|
||||
|
||||
## Full Config Example
|
||||
|
||||
@@ -331,8 +338,10 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
|
||||
"enabled": true,
|
||||
"url": "http://127.0.0.1:8765",
|
||||
"pollingRate": 3000,
|
||||
"deck": "",
|
||||
"tags": ["SubMiner"],
|
||||
"proxy": {
|
||||
"enabled": false,
|
||||
"enabled": true, // default
|
||||
"host": "127.0.0.1",
|
||||
"port": 8766,
|
||||
"upstreamUrl": "http://127.0.0.1:8765",
|
||||
@@ -363,10 +372,13 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
|
||||
"autoUpdateNewCards": true,
|
||||
"notificationType": "overlay",
|
||||
},
|
||||
"metadata": {
|
||||
"pattern": "[SubMiner] %f (%t)",
|
||||
},
|
||||
"ai": {
|
||||
"enabled": false,
|
||||
"model": "openai/gpt-4o-mini",
|
||||
"systemPrompt": "Translate mined sentence text only.",
|
||||
"model": "", // e.g. "openai/gpt-4o-mini"
|
||||
"systemPrompt": "",
|
||||
},
|
||||
"isKiku": {
|
||||
"enabled": false,
|
||||
@@ -375,7 +387,7 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
|
||||
},
|
||||
"isLapis": {
|
||||
"enabled": false,
|
||||
"sentenceCardModel": "Japanese sentences",
|
||||
"sentenceCardModel": "Lapis",
|
||||
},
|
||||
},
|
||||
"ai": {
|
||||
|
||||
+23
-14
@@ -6,7 +6,7 @@ SubMiner is split into three cooperating runtimes:
|
||||
|
||||
- Electron desktop app (`src/`) for overlay/UI/runtime orchestration.
|
||||
- Launcher CLI (`launcher/`) for mpv/app command workflows.
|
||||
- mpv Lua plugin (`plugin/subminer/init.lua` + module files) for player-side controls and IPC handoff.
|
||||
- mpv Lua plugin (`plugin/subminer/main.lua` + module files) for player-side controls and IPC handoff.
|
||||
|
||||
Within the desktop app, `src/main.ts` is a composition root that wires small runtime/domain modules plus core services.
|
||||
|
||||
@@ -24,13 +24,14 @@ Within the desktop app, `src/main.ts` is a composition root that wires small run
|
||||
|
||||
```text
|
||||
launcher/ # Standalone CLI launcher wrapper and mpv helpers
|
||||
commands/ # Command modules (doctor/config/mpv/jellyfin/playback/app passthrough)
|
||||
commands/ # Command modules (doctor/config/mpv/jellyfin/playback/app passthrough/
|
||||
# dictionary/history/logs/stats/update)
|
||||
config/ # Launcher config parsers + CLI parser builder
|
||||
main.ts # Launcher entrypoint and command dispatch
|
||||
plugin/
|
||||
subminer/ # Modular mpv plugin (init · main · bootstrap · lifecycle · process
|
||||
subminer/ # Modular mpv plugin (main · init · bootstrap · lifecycle · process
|
||||
# state · messages · hover · ui · options · environment · log
|
||||
# binary)
|
||||
# binary · session_bindings · version)
|
||||
src/
|
||||
ai/ # AI translation provider utilities (client, config)
|
||||
main-entry.ts # Background-mode bootstrap wrapper before loading main.js
|
||||
@@ -38,6 +39,7 @@ src/
|
||||
preload.ts # Electron preload bridge
|
||||
types.ts # Shared type definitions
|
||||
main/ # Main-process composition/runtime adapters
|
||||
boot/ # Pre-ready boot helpers
|
||||
app-lifecycle.ts # App lifecycle + app-ready runtime runner factories
|
||||
character-dictionary-runtime.ts # Character-dictionary orchestration/public runtime API
|
||||
cli-runtime.ts # CLI command runtime service adapters
|
||||
@@ -71,17 +73,17 @@ src/
|
||||
resolve/ # Domain-specific config resolution pipeline stages
|
||||
shared/ipc/ # Cross-process IPC channel constants + payload validators
|
||||
renderer/ # Overlay renderer (modularized UI/runtime)
|
||||
handlers/ # Keyboard/mouse interaction modules
|
||||
modals/ # Jimaku/Kiku/subsync/runtime-options/session-help modals
|
||||
handlers/ # Keyboard/mouse/gamepad interaction modules
|
||||
modals/ # Modal flows (Jimaku, Kiku, subsync, runtime options, session help,
|
||||
# character dictionary, playlist browser, subtitle sidebar,
|
||||
# YouTube track picker, controller config/debug/select)
|
||||
positioning/ # Subtitle position controller (drag-to-reposition)
|
||||
settings/ # Settings window UI (model, controls, markup)
|
||||
types/ # Domain type modules (anki, config, integrations, ...)
|
||||
window-trackers/ # Backend-specific tracker implementations (Hyprland, Sway, X11, macOS, Windows)
|
||||
jimaku/ # Jimaku API integration helpers
|
||||
subsync/ # Subtitle sync (alass/ffsubsync) helpers
|
||||
subtitle/ # Subtitle processing utilities
|
||||
tokenizers/ # Tokenizer implementations
|
||||
anki-integration/ # AnkiConnect proxy server + note-update enrichment workflow
|
||||
token-mergers/ # Token merge strategies
|
||||
translators/ # AI translation providers
|
||||
```
|
||||
|
||||
### Service Layer (`src/core/services/`)
|
||||
@@ -92,7 +94,7 @@ src/
|
||||
- **Mining + Anki/Jimaku runtime:** `mining.ts`, `field-grouping.ts`, `field-grouping-overlay.ts`, `anki-jimaku.ts`, `anki-jimaku-ipc.ts`
|
||||
- **Subtitle/token pipeline:** `subtitle-processing-controller.ts`, `subtitle-position.ts`, `subtitle-ws.ts`, `tokenizer.ts` + `tokenizer/*` stage modules (including `parser-enrichment-worker-runtime.ts` for async MeCab enrichment and `yomitan-parser-runtime.ts`)
|
||||
- **Integrations:** `jimaku.ts`, `subsync.ts`, `subsync-runner.ts`, `texthooker.ts`, `jellyfin.ts`, `jellyfin-remote.ts`, `discord-presence.ts`, `yomitan-extension-loader.ts`, `yomitan-settings.ts`
|
||||
- **Anki integration:** `anki-integration.ts`, `anki-integration/anki-connect-proxy.ts` (local proxy for push-based auto-enrichment), `anki-integration/note-update-workflow.ts`
|
||||
- **Anki integration (repo `src/` root, not under `core/services/`):** `src/anki-integration.ts`, `src/anki-integration/anki-connect-proxy.ts` (local proxy for push-based auto-enrichment), `src/anki-integration/note-update-workflow.ts`
|
||||
- **Config/runtime controls:** `config-hot-reload.ts`, `runtime-options-ipc.ts`, `cli-command.ts`, `startup.ts`
|
||||
- **Domain submodules:** `anilist/*` (token/update queue/updater), `immersion-tracker/*` (storage/session/metadata/query/reducer)
|
||||
|
||||
@@ -116,12 +118,19 @@ src/renderer/
|
||||
handlers/
|
||||
keyboard.ts # Keybindings, chord handling, modal key routing
|
||||
mouse.ts # Hover/drag behavior, selection + observer wiring
|
||||
gamepad-controller.ts # Gamepad/controller input handling
|
||||
controller-binding-capture.ts # Controller binding capture flow
|
||||
modals/
|
||||
jimaku.ts # Jimaku modal flow
|
||||
kiku.ts # Kiku field-grouping modal flow
|
||||
runtime-options.ts # Runtime options modal flow
|
||||
session-help.ts # Keyboard shortcuts/help modal flow
|
||||
subsync.ts # Manual subsync modal flow
|
||||
character-dictionary.ts # Character dictionary modal flow
|
||||
playlist-browser.ts # Playlist browser modal flow
|
||||
subtitle-sidebar.ts # Subtitle sidebar modal flow
|
||||
youtube-track-picker.ts # YouTube subtitle track picker
|
||||
controller-*.ts # Controller config/debug/select modals
|
||||
utils/
|
||||
dom.ts # Required DOM lookups + typed handles
|
||||
platform.ts # Layer/platform capability detection
|
||||
@@ -130,7 +139,7 @@ src/renderer/
|
||||
### Launcher + Plugin Runtimes
|
||||
|
||||
- `launcher/main.ts` dispatches commands through `launcher/commands/*` and shared config readers in `launcher/config/*`. It handles mpv startup, app passthrough, Jellyfin helper commands, and playback handoff.
|
||||
- `plugin/subminer/init.lua` runs inside mpv and loads modular Lua files: `main.lua` (orchestration), `bootstrap.lua` (startup), `lifecycle.lua` (connect/disconnect), `process.lua` (process management), `state.lua` (shared state), `messages.lua` (IPC), `hover.lua` (hover-token highlight rendering), `ui.lua` (OSD rendering), `options.lua` (config), `environment.lua` (detection), `log.lua` (logging), `binary.lua` (path resolution). AniSkip intro detection lives in the SubMiner app (`src/main/runtime/aniskip-runtime.ts`), which drives mpv chapters and the skip key over the IPC socket.
|
||||
- `plugin/subminer/main.lua` is the mpv entrypoint: it sets up the module path and loads `init.lua`, a thin shim that boots the modular Lua files: `bootstrap.lua` (startup), `lifecycle.lua` (connect/disconnect), `process.lua` (process management), `state.lua` (shared state), `messages.lua` (IPC), `hover.lua` (hover-token highlight rendering), `ui.lua` (OSD rendering), `options.lua` (config), `environment.lua` (detection), `log.lua` (logging), `binary.lua` (path resolution), `session_bindings.lua` (configurable session keybindings), `version.lua` (version metadata). AniSkip intro detection lives in the SubMiner app (`src/main/runtime/aniskip-runtime.ts`), which drives mpv chapters and the skip key over the IPC socket.
|
||||
|
||||
## Flow Diagram
|
||||
|
||||
@@ -313,7 +322,7 @@ The runtime sockets in this flow are detailed in [IPC + Runtime Contracts](./ipc
|
||||
- **Critical-path init:** Once `app.whenReady()` fires, `composeAppReadyRuntime()` runs strict config reload, resolves keybindings, creates the `MpvIpcClient` (which immediately connects and subscribes to mpv subtitle/playback properties via `observe_property`), and initializes the `RuntimeOptionsManager`, `SubtitleTimingTracker`, and `ImmersionTrackerService`.
|
||||
- **Overlay runtime:** `initializeOverlayRuntime()` creates the primary overlay window (interactive Yomitan lookups and subtitle rendering), registers global shortcuts, and sets up bounds tracking via the active window tracker. mpv subtitle suppression is handled by a dedicated `overlay-mpv-sub-visibility` service.
|
||||
- **Background warmups:** Non-critical services are launched asynchronously: MeCab tokenizer check (with async worker thread), Yomitan extension load, JLPT + frequency dictionary prewarm, optional Jellyfin remote session, Discord presence service, AniList token refresh, and optional AnkiConnect proxy server. Warmup coverage is configurable through `startupWarmups` (including low-power mode that defers all but Yomitan).
|
||||
- **Runtime:** Event-driven. mpv property changes, IPC messages, CLI commands, overlay shortcuts, and hot-reload notifications route through runtime handlers/composers. Subtitle text flows through `SubtitlePipeline` (normalize → tokenize → merge), and results are sent to the main overlay renderer and modal surfaces.
|
||||
- **Runtime:** Event-driven. mpv property changes, IPC messages, CLI commands, overlay shortcuts, and hot-reload notifications route through runtime handlers/composers. Subtitle text flows through the `SubtitleProcessingController` (normalize → tokenize → merge), and results are sent to the main overlay renderer and modal surfaces.
|
||||
- **Shutdown:** `onWillQuitCleanup` destroys tray + config watcher, unregisters shortcuts, stops WebSocket + texthooker servers, closes the mpv socket + flushes OSD log, stops the window tracker, closes the Yomitan parser window, flushes the immersion tracker (SQLite), stops Jellyfin/Discord services, stops the AnkiConnect proxy server, and cleans Anki/AniList state.
|
||||
|
||||
```mermaid
|
||||
@@ -380,7 +389,7 @@ flowchart TB
|
||||
|
||||
## Subtitle Prefetch Pipeline
|
||||
|
||||
SubMiner can pre-tokenize upcoming subtitle lines before they appear on screen. When an external subtitle file (SRT, VTT, or ASS) is detected on the active track, the `SubtitlePrefetchService` parses all cues via the `SubtitleCueParser`, identifies a priority window of upcoming lines based on the current playback position, and tokenizes them in the background through the same pipeline used for live subtitles. Results are stored directly into the `SubtitleProcessingController` cache, so when a subtitle actually appears during playback, it hits a warm cache and renders in ~30-50ms instead of ~200-320ms.
|
||||
SubMiner can pre-tokenize upcoming subtitle lines before they appear on screen. When an external subtitle file (SRT, VTT, or ASS) is detected on the active track, the `SubtitlePrefetchService` parses all cues via the subtitle cue parser (`subtitle-cue-parser.ts`), identifies a priority window of upcoming lines based on the current playback position, and tokenizes them in the background through the same pipeline used for live subtitles. Results are stored directly into the `SubtitleProcessingController` cache, so when a subtitle actually appears during playback, it hits a warm cache and renders in ~30-50ms instead of ~200-320ms.
|
||||
|
||||
The prefetcher yields to live subtitle processing (which always takes priority over background work) and re-computes its priority window on seek. Cache invalidation events (e.g. marking a word as known) trigger re-prefetching of the current window to keep results fresh.
|
||||
|
||||
|
||||
@@ -134,7 +134,7 @@ Each character entry in the Yomitan dictionary includes structured content:
|
||||
|
||||
- **Name** - the matched Japanese name form
|
||||
- **Known names** - generated non-honorific Japanese aliases for that character, excluding raw romanized/English aliases from lookup results
|
||||
- **Role badge** - color-coded by role: main (score 100), supporting (90), side (80), background (70)
|
||||
- **Role badge** - color-coded by role: main / "Protagonist" (score 100), primary / "Main Character" (75), side / "Side Character" (50), appears / "Minor Role" (25). AniList's MAIN maps to main, SUPPORTING to primary, and BACKGROUND to side.
|
||||
- **Portrait** - character image from AniList, embedded in the ZIP
|
||||
- **Description** - biography text from AniList (collapsible)
|
||||
- **Character information** - age, birthday, gender, blood type (collapsible)
|
||||
@@ -166,7 +166,7 @@ These phases are emitted through the configured notification surface. Some phase
|
||||
|
||||
1. **checking** - Is there already a cached snapshot for this media ID?
|
||||
2. **generating** - No cache hit: fetch characters from AniList GraphQL, download portraits (250ms throttle between image requests), save snapshot JSON.
|
||||
3. **syncing** - Add the media ID to the most-recently-used list. Evict old entries beyond `maxLoaded`.
|
||||
3. MRU update (no notification) - add the media ID to the most-recently-used list and evict old entries beyond `maxLoaded`.
|
||||
4. **building** - Merge active snapshots into a single Yomitan ZIP. A SHA-1 revision hash is computed from the media set - if it matches the previously imported revision, the import is skipped.
|
||||
5. **importing** - Push the ZIP into Yomitan. Waits for Yomitan mutation readiness (7-second timeout per operation).
|
||||
6. **ready** - Dictionary is live. Character names will match on the next subtitle line.
|
||||
@@ -175,12 +175,14 @@ These phases are emitted through the configured notification surface. Some phase
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"activeMediaIds": [170942, 163134, 154587],
|
||||
"activeMediaIds": ["170942 - Frieren", "163134 - ...", "154587 - ..."],
|
||||
"mergedRevision": "a1b2c3d4e5f6",
|
||||
"mergedDictionaryTitle": "SubMiner Character Dictionary",
|
||||
}
|
||||
```
|
||||
|
||||
(Entries are `"<mediaId> - <title>"` label strings; bare numeric IDs from older versions are still read.)
|
||||
|
||||
The `maxLoaded` setting (default: 3) controls how many media snapshots stay in the active set. When you start a 4th title, the oldest is evicted and the merged dictionary is rebuilt without it.
|
||||
|
||||
## Manual Generation
|
||||
@@ -228,7 +230,7 @@ Manual selections are stored in `character-dictionaries/anilist-overrides.json`
|
||||
Open the manager with `Ctrl/Cmd+D` (`shortcuts.openCharacterDictionaryManager`). The manager shows the merged dictionary's active MRU entries, marks the current anime, and lets you adjust eviction priority for the other loaded entries.
|
||||
|
||||
- **Remove** drops a non-current entry from the active merged dictionary and rebuilds/imports once.
|
||||
- **Up/Down** changes MRU order for future eviction without rebuilding.
|
||||
- **Up/Down** changes MRU order for future eviction; the merged dictionary is rebuilt and re-imported after a reorder.
|
||||
- **Override** opens the AniList selector for that entry's title so you can replace a saved loaded entry.
|
||||
|
||||
The current anime cannot be removed while you are watching it; it stays loaded until playback changes.
|
||||
@@ -250,13 +252,13 @@ character-dictionaries/
|
||||
m170942-va67890.jpg # Voice actor portrait
|
||||
```
|
||||
|
||||
**Snapshot format** (v17): each snapshot contains the media ID, title, entry count, timestamp, an array of Yomitan term entries, and base64-encoded images.
|
||||
**Snapshot format** (v19, `CHARACTER_DICTIONARY_FORMAT_VERSION`): each snapshot contains the media ID, title, entry count, timestamp, an array of Yomitan term entries, and base64-encoded images. Snapshots with a different format version are regenerated.
|
||||
|
||||
**ZIP structure** follows the Yomitan dictionary format:
|
||||
|
||||
```text
|
||||
merged.zip
|
||||
index.json # { title, revision, format: 3, author: "SubMiner" }
|
||||
index.json # { title, revision, format: 3, author: "SubMiner", description }
|
||||
tag_bank_1.json # Tag definitions
|
||||
term_bank_1.json # Up to 10,000 terms per bank
|
||||
term_bank_2.json
|
||||
|
||||
+32
-29
@@ -46,12 +46,13 @@ The Settings window groups options by workflow instead of mirroring the raw conf
|
||||
- Appearance
|
||||
- Behavior
|
||||
- Mining & Anki
|
||||
- Playback & Sources
|
||||
- Input
|
||||
- Integrations
|
||||
- Tracking & App
|
||||
- Advanced
|
||||
|
||||
Playback-related fields live as sections inside these groups (for example "Playback Behavior" under **Behavior** and "mpv Playback" / "YouTube Playback Settings" under **Integrations**).
|
||||
|
||||
Each field still writes to its current `config.jsonc` path. For example, subtitle hover pause appears under **Behavior** / playback behavior, but saves to `subtitleStyle.autoPauseVideoOnHover`. Anki-aware fields can query AnkiConnect for deck names, note types, and field names. The AnkiConnect deck field also reads Yomitan's current mining deck and persists it into an empty setting when one is found. Stats mining also uses Yomitan's current mining deck when `ankiConnect.deck` is empty. Keybinding fields use click-to-learn controls instead of raw text boxes.
|
||||
|
||||
The Settings window preserves existing JSONC comments, trailing commas, and unrelated keys. Resetting a field removes the explicit config path so the built-in default applies.
|
||||
@@ -95,9 +96,11 @@ On macOS, these validation warnings also open a native dialog with full details
|
||||
SubMiner watches the active config file (`config.jsonc` or `config.json`) while running and applies supported updates automatically.
|
||||
|
||||
Hot-reloadable settings include subtitle appearance, sidebar controls, keybindings,
|
||||
logging level, selected source-language preferences, Jimaku/Subsync settings, and
|
||||
the Anki deck, known-word, N+1, field, sentence-card, and Kiku options listed
|
||||
in the reference tables below.
|
||||
shortcuts, notifications, logging level, selected source-language preferences,
|
||||
Jimaku/Subsync settings, AniSkip settings (`mpv.aniskipEnabled`, `mpv.aniskipButtonKey`),
|
||||
stats keys (`stats.toggleKey`, `stats.markWatchedKey`), the secondary-subtitle default
|
||||
mode, and the Anki deck, known-word, N+1, field, sentence-card, AI, and Kiku options
|
||||
listed in the reference tables below.
|
||||
|
||||
When these values change, SubMiner applies them live. Invalid config edits are rejected and the previous valid runtime config remains active.
|
||||
|
||||
@@ -482,6 +485,8 @@ Configure the parsed-subtitle sidebar modal.
|
||||
| `autoScroll` | boolean | Keep the active cue in view while playback advances |
|
||||
| `subtitleSidebar.css` | object | CSS declaration object applied to the sidebar. Use CSS properties plus sidebar custom properties below. |
|
||||
|
||||
Direct style keys are also available under `subtitleSidebar` and map to the same visuals as the CSS custom properties: `maxWidth` (default `420`), `opacity` (`0.95`), `backgroundColor`, `textColor`, `fontFamily`, `fontSize` (`16`), `timestampColor`, `activeLineColor`, `activeLineBackgroundColor`, and `hoverLineBackgroundColor`.
|
||||
|
||||
Sidebar CSS custom properties:
|
||||
|
||||
| CSS Property | Default | Description |
|
||||
@@ -498,7 +503,7 @@ The sidebar is only available when the active subtitle source has been parsed in
|
||||
|
||||
For full details on layout modes, behavior, and the keyboard shortcut, see the [Subtitle Sidebar](/subtitle-sidebar) page.
|
||||
|
||||
`jlptColors` keys are:
|
||||
`subtitleStyle.jlptColors` keys are:
|
||||
|
||||
| Key | Default | Description |
|
||||
| ---- | --------- | ----------------------- |
|
||||
@@ -508,12 +513,6 @@ For full details on layout modes, behavior, and the keyboard shortcut, see the [
|
||||
| `N4` | `#8bd5ca` | JLPT N4 underline color |
|
||||
| `N5` | `#8aadf4` | JLPT N5 underline color |
|
||||
|
||||
**Image Quality Notes:**
|
||||
|
||||
- `imageQuality` affects JPG and WebP only; PNG is lossless and ignores this setting
|
||||
- JPG quality is mapped to FFmpeg's scale (2-31, lower = better)
|
||||
- WebP quality uses FFmpeg's native 0-100 scale
|
||||
|
||||
### Subtitle Position
|
||||
|
||||
Set the initial vertical subtitle position (measured from the bottom of the screen):
|
||||
@@ -797,7 +796,7 @@ If you bind a discrete action to an axis manually, include `direction`:
|
||||
|
||||
Treat the button-index map as reference-only unless you are copying values from the debug modal. Updating it alone does not rewrite the hardcoded raw numeric values already present in controller bindings or controller profiles. If you need a real remap, prefer the `Alt+C` learn flow so both the source and the descriptor shape stay correct.
|
||||
|
||||
If you choose to bind `L2` or `R2` manually, set `triggerInputMode` to `analog` and tune `triggerDeadzone` when your controller reports triggers as analog values instead of digital pressed/not-pressed buttons. `auto` accepts either style and remains the default.
|
||||
If you choose to bind `L2` or `R2` manually, set `triggerInputMode` to `analog` and tune `triggerDeadzone` when your controller reports triggers as analog values instead of digital pressed/not-pressed buttons. `digital` forces pressed/not-pressed handling; `auto` accepts either style and remains the default.
|
||||
|
||||
If one controller reports non-standard raw button numbers, override that controller profile's button-index map using values from the `Alt+Shift+C` debug modal. Use the global button-index map only when the mapping should apply to every controller without a profile.
|
||||
|
||||
@@ -834,7 +833,7 @@ These shortcuts are only active when the overlay window is visible and automatic
|
||||
|
||||
### Session Help Modal
|
||||
|
||||
The session help modal opens from the overlay with `Ctrl/Cmd+/` by default. The mpv plugin also exposes it through the `Y-H` chord (falling back to `Y-K` if needed). It shows the current session keybindings and color legend.
|
||||
The session help modal opens from the overlay with `Ctrl/Cmd+/` by default. The mpv plugin also exposes it through the `y-h` chord. It shows the current session keybindings and color legend.
|
||||
|
||||
You can filter the modal quickly with `/`:
|
||||
|
||||
@@ -861,8 +860,8 @@ When config hot-reload updates shortcut/keybinding/style values, close and reope
|
||||
Use the runtime options palette to toggle settings live while SubMiner is running. These changes are session-only and reset on restart.
|
||||
|
||||
Current runtime options cover automatic card updates, known-word highlighting,
|
||||
JLPT underlines, frequency highlighting, known-word match mode, and Kiku field
|
||||
grouping mode.
|
||||
N+1 annotation, JLPT underlines, frequency highlighting, known-word match mode,
|
||||
and Kiku field grouping mode.
|
||||
|
||||
Annotation toggles only apply to new subtitle lines after the toggle. The currently displayed line is not re-tokenized in place.
|
||||
|
||||
@@ -879,7 +878,7 @@ Palette controls:
|
||||
|
||||
### Shared AI Provider
|
||||
|
||||
This is the single, shared connection to an OpenAI-compatible LLM endpoint. Configure it **once** here at the top level, and SubMiner reuses it wherever AI is needed (today: Anki translation/enrichment). Per-feature toggles and prompt/model tweaks live in their own sections (for example `ankiConnect.ai`) and inherit this transport.
|
||||
This is the single, shared connection to an OpenAI-compatible LLM endpoint. Configure it **once** here at the top level, and SubMiner reuses it wherever AI is needed (Anki translation/enrichment and YouTube subtitle fixing). Per-feature toggles and prompt/model tweaks live in their own sections (for example `ankiConnect.ai` and `youtubeSubgen.ai`) and inherit this transport.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -907,6 +906,7 @@ This is the single, shared connection to an OpenAI-compatible LLM endpoint. Conf
|
||||
SubMiner uses the shared provider for:
|
||||
|
||||
- Anki translation/enrichment when Anki AI is enabled
|
||||
- YouTube generated-subtitle fixing when `youtubeSubgen.fixWithAi` is enabled (with optional `youtubeSubgen.ai.model` / `systemPrompt` overrides)
|
||||
|
||||
### AnkiConnect
|
||||
|
||||
@@ -1008,7 +1008,7 @@ This example is intentionally compact. The option table below documents availabl
|
||||
| `media.generateImage` | `true`, `false` | Generate image/animation screenshots (default: `true`) |
|
||||
| `media.imageType` | `"static"`, `"avif"` | Image type: static screenshot or animated AVIF (default: `"static"`) |
|
||||
| `media.imageFormat` | `"jpg"`, `"png"`, `"webp"` | Image format (default: `"jpg"`) |
|
||||
| `media.imageQuality` | number (1-100) | Image quality for JPG/WebP; PNG ignores this (default: `92`) |
|
||||
| `media.imageQuality` | number (1-100) | Image quality for JPG/WebP; PNG ignores this (default: `92`). JPG values are mapped onto FFmpeg's 2-31 quality scale; WebP uses the value directly. |
|
||||
| `media.imageMaxWidth` | number (px) | Optional max width for static screenshots. Unset keeps source width. |
|
||||
| `media.imageMaxHeight` | number (px) | Optional max height for static screenshots. Unset keeps source height. |
|
||||
| `media.animatedFps` | number (1-60) | FPS for animated AVIF (default: `10`) |
|
||||
@@ -1032,7 +1032,7 @@ This example is intentionally compact. The option table below documents availabl
|
||||
| `ankiConnect.nPlusOne.minSentenceWords` | number | Minimum number of words required in a sentence before single unknown-word N+1 highlighting can trigger (default: `3`). |
|
||||
| `behavior.notificationType` | `"overlay"`, `"system"`, `"both"`, `"none"` | Notification type on card update (default: `"overlay"`). `"both"` means overlay + system. `osd` and `osd-system` are legacy config-file-only values; use `"osd-system"` to keep the old OSD + system behavior. |
|
||||
| `behavior.autoUpdateNewCards` | `true`, `false` | Automatically update cards on creation (default: `true`) |
|
||||
| `metadata.pattern` | string | Format pattern for metadata: `%f`=filename, `%F`=filename+ext, `%t`=time |
|
||||
| `metadata.pattern` | string | Format pattern for metadata: `%f`=filename, `%F`=filename+ext, `%t`=time, `%T`=time with milliseconds, `<br>`=newline |
|
||||
| `isLapis` | object | Lapis/shared sentence-card config: `{ enabled, sentenceCardModel }`. Sentence/audio field names are fixed to `Sentence` and `SentenceAudio`. |
|
||||
| `isKiku` | object | Kiku-only config: `{ enabled, fieldGrouping, deleteDuplicateInAuto }` (shared sentence/audio/model settings are inherited from `isLapis`) |
|
||||
|
||||
@@ -1136,8 +1136,6 @@ Configure Jimaku API access and defaults:
|
||||
|
||||
Jimaku is rate limited; if you hit a limit, SubMiner will surface the retry delay from the API response.
|
||||
|
||||
Set `openBrowser` to `false` to only print the URL without opening a browser.
|
||||
|
||||
### Subtitle Sync
|
||||
|
||||
Sync the active subtitle track from the overlay picker using `alass` or `ffsubsync`. Both are **optional external tools** that must be installed separately and available on your `PATH` (or configured via the path options below).
|
||||
@@ -1158,8 +1156,8 @@ Sync the active subtitle track from the overlay picker using `alass` or `ffsubsy
|
||||
|
||||
| Option | Values | Description |
|
||||
| ---------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `alass_path` | string path | Path to `alass` executable. Empty or `null` resolves from `PATH`. `alass` must be installed separately. |
|
||||
| `ffsubsync_path` | string path | Path to `ffsubsync` executable. Empty or `null` resolves from `PATH`. `ffsubsync` must be installed separately. |
|
||||
| `alass_path` | string path | Path to `alass` executable. Empty falls back to `/usr/bin/alass`. `alass` must be installed separately. |
|
||||
| `ffsubsync_path` | string path | Path to `ffsubsync` executable. Empty falls back to `/usr/bin/ffsubsync`. `ffsubsync` must be installed separately. |
|
||||
| `ffmpeg_path` | string path | Path to `ffmpeg` (used for internal subtitle extraction). Empty or `null` falls back to `/usr/bin/ffmpeg`. |
|
||||
| `replace` | `true`, `false` | When `true` (default), overwrite the active subtitle file on successful sync. When `false`, write `<name>_retimed.<ext>`. |
|
||||
|
||||
@@ -1195,6 +1193,8 @@ AniList integration is opt-in and disabled by default. Enable it to allow SubMin
|
||||
| `anilist.enabled` | `true`, `false` | Enable AniList post-watch progress updates (default: `false`) |
|
||||
| `accessToken` | string | Optional explicit AniList access token override (default: empty string) |
|
||||
| `characterDictionary.maxLoaded` | number | Maximum number of most-recently-used AniList media snapshots included in the merged dictionary (default: `3`) |
|
||||
| `characterDictionary.refreshTtlHours` | number | Hours before a cached media snapshot is refreshed (default: `168`, clamped to 1–8760) |
|
||||
| `characterDictionary.evictionPolicy` | `"delete"`, `"disable"` | What happens to snapshots evicted beyond `maxLoaded` (default: `"delete"`) |
|
||||
| `characterDictionary.collapsibleSections.description` | `true`, `false` | Open the Description section by default in generated dictionary entries |
|
||||
| `characterDictionary.collapsibleSections.characterInformation` | `true`, `false` | Open the Character Information section by default in generated dictionary entries |
|
||||
| `characterDictionary.collapsibleSections.voicedBy` | `true`, `false` | Open the Voiced by section by default in generated dictionary entries |
|
||||
@@ -1504,7 +1504,7 @@ Configure the mpv executable, profile, and window state for SubMiner-managed mpv
|
||||
| `executablePath` | string | Absolute path to `mpv.exe` for Windows launch flows. Leave empty to auto-discover from `SUBMINER_MPV_PATH` or `PATH` (default `""`) |
|
||||
| `profile` | string | mpv profile name passed as `--profile=<name>`. Leave empty to pass no profile (default `""`) |
|
||||
| `launchMode` | `"normal"` \| `"maximized"` \| `"fullscreen"` | Window state when SubMiner spawns mpv (default `"normal"`) |
|
||||
| `socketPath` | string | mpv IPC socket path used by SubMiner-managed playback and the bundled mpv plugin (default: `\\\\.\\pipe\\subminer-socket`) |
|
||||
| `socketPath` | string | mpv IPC socket path used by SubMiner-managed playback and the bundled mpv plugin (platform-dependent default: `/tmp/subminer-socket`, or `\\\\.\\pipe\\subminer-socket` on Windows) |
|
||||
| `backend` | `"auto"` \| `"hyprland"` \| `"sway"` \| `"x11"` \| `"macos"` \| `"windows"` | Window tracking backend passed to the bundled mpv plugin. Auto detects the current platform (default: `"auto"`) |
|
||||
| `autoStartSubMiner` | `true`, `false` | Start SubMiner in the background when SubMiner-managed mpv loads a file (default: `true`) |
|
||||
| `pauseUntilOverlayReady` | `true`, `false` | Pause mpv on visible-overlay auto-start until SubMiner signals subtitle tokenization readiness, with a 30-second fallback (default: `true`) |
|
||||
@@ -1552,15 +1552,18 @@ Current launcher behavior:
|
||||
- If YouTube/mpv already exposes an authoritative matching subtitle track, SubMiner reuses it; otherwise it downloads and injects only the missing side.
|
||||
- SubMiner loads the primary subtitle plus a best-effort secondary subtitle.
|
||||
- Playback waits only for primary subtitle readiness; secondary failures do not block playback.
|
||||
- English secondary subtitles are selected from the secondary-subtitle language list when primary language matches are unavailable.
|
||||
- Native mpv secondary subtitle rendering stays hidden during this flow so the SubMiner overlay remains the visible secondary subtitle surface.
|
||||
- If primary subtitle loading fails, use `Ctrl+Alt+C` to open the subtitle modal and pick a track.
|
||||
|
||||
Language targets are derived from subtitle config:
|
||||
Track selection:
|
||||
|
||||
- primary track: `youtube.primarySubLanguages` (falls back to `["ja","jpn"]`)
|
||||
- secondary track: secondary-subtitle language list (falls back to English when empty)
|
||||
- Local playback uses the same priorities after mpv reports subtitle track metadata, so sidecar/internal mixed sets can override an incorrect initial `sid=auto` pick.
|
||||
- YouTube auto-selection always targets a Japanese primary track and an English secondary track, preferring manual uploads over auto-generated captions.
|
||||
- `youtube.primarySubLanguages` (default `["ja","jpn"]`) defines which loaded track counts as a satisfactory primary for the "primary subtitle missing" notification and for managed local/playlist subtitle selection.
|
||||
- Local playback applies these priorities after mpv reports subtitle track metadata, so sidecar/internal mixed sets can override an incorrect initial `sid=auto` pick.
|
||||
- Tracks are resolved and loaded before mpv starts; the older launcher mode switch has been removed.
|
||||
|
||||
Precedence for launcher defaults is: CLI flag > environment variable > `config.jsonc` > built-in default.
|
||||
These settings come from `config.jsonc` (or built-in defaults); there are no CLI flags or environment variables for subtitle language selection.
|
||||
|
||||
#### YouTube Subtitle Generation (`youtubeSubgen`)
|
||||
|
||||
An advanced, template-hidden section for Whisper-based YouTube subtitle generation: `whisperBin`, `whisperModel`, `whisperVadModel`, `whisperThreads` (default `4`), and `fixWithAi` (default `false`), which post-processes generated subtitles through the [Shared AI Provider](#shared-ai-provider) with optional `youtubeSubgen.ai.model` / `systemPrompt` overrides. These keys are accepted in `config.jsonc` but intentionally omitted from the generated template.
|
||||
|
||||
@@ -5,6 +5,8 @@ For internal architecture/workflow guidance, use `docs/README.md` at the repo ro
|
||||
## Prerequisites
|
||||
|
||||
- [Bun](https://bun.sh)
|
||||
- A system `lua` interpreter for `bun run test:launcher` / `bun run test:plugin:src`
|
||||
- macOS builds compile a Swift helper via `scripts/build-macos-helper.sh` (skip with `SUBMINER_SKIP_MACOS_HELPER_BUILD=1`)
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -163,7 +165,7 @@ make pretty
|
||||
bun run format:check:src
|
||||
```
|
||||
|
||||
- `make pretty` runs the maintained Prettier allowlist only (`format:src`).
|
||||
- `make pretty` runs the maintained Prettier allowlists (`format:src` and `format:stats`).
|
||||
- `bun run format:check:src` checks the same scoped set without writing changes.
|
||||
- `bun run format` remains the broad repo-wide Prettier command; use it intentionally.
|
||||
|
||||
@@ -196,15 +198,7 @@ bun run docs:preview # Preview built site at http://localhost:4173
|
||||
bun run docs:test # Docs regression tests
|
||||
```
|
||||
|
||||
Cloudflare Pages deploy settings:
|
||||
|
||||
- Git repo: `ksyasuda/SubMiner`
|
||||
- Root directory: `docs-site`
|
||||
- Build command: `bun run docs:build`
|
||||
- Build output directory: `.vitepress/dist`
|
||||
- Build watch paths: `docs-site/*`
|
||||
|
||||
Use Cloudflare's single `*` wildcard syntax for watch paths. `docs-site/*` covers nested docs-site changes in the repo; `docs-site/**` is not the correct Pages pattern and may skip docs-only pushes.
|
||||
Deployment: production docs are built with `bun run docs:build:versioned` and uploaded directly to Cloudflare Pages by the `docs-pages` GitHub Actions workflow using Wrangler (from `.tmp/docs-versioned-site`). Cloudflare's automatic Git-integration deployments are intentionally disabled - see `docs-site/README.md` for the deployment contract. Do not re-enable Pages build settings in the Cloudflare dashboard.
|
||||
|
||||
## Makefile Reference
|
||||
|
||||
@@ -241,6 +235,7 @@ Run `make help` for a full list of targets. Key ones:
|
||||
| `SUBMINER_APPIMAGE_PATH` | Override SubMiner app binary path for launcher playback commands |
|
||||
| `SUBMINER_BINARY_PATH` | Alias for `SUBMINER_APPIMAGE_PATH` |
|
||||
| `SUBMINER_ROFI_THEME` | Override rofi theme path for launcher picker |
|
||||
| `SUBMINER_MPV_PLUGIN_PATH` | Override the mpv plugin directory injected by the launcher |
|
||||
| `SUBMINER_LOG_LEVEL` | Override app logger level (`debug`, `info`, `warn`, `error`) |
|
||||
| `SUBMINER_MPV_LOG` | Override mpv/app shared log file path |
|
||||
| `SUBMINER_JIMAKU_API_KEY` | Override Jimaku API key for launcher subtitle downloads |
|
||||
|
||||
@@ -25,15 +25,16 @@ Episode completion for local `watched` state uses the shared `DEFAULT_MIN_WATCH_
|
||||
|
||||
- Leave `dbPath` empty to use the default location (`immersion.sqlite` in SubMiner's app-data directory).
|
||||
- Set an explicit path to move the database (useful for backups, cloud syncing, or external tools).
|
||||
- To share stats and watch history between two machines, use [`subminer sync <host>`](/launcher-script#sync-between-machines) instead of file-level cloud sync — it merges both databases without one side overwriting the other.
|
||||
|
||||
## Stats Dashboard
|
||||
|
||||
The same immersion data powers the stats dashboard.
|
||||
|
||||
- In-app overlay: focus the visible overlay, then press the key from `stats.toggleKey` (default: `` ` `` / `Backquote`).
|
||||
- Launcher command: run `subminer stats` to start the local stats server on demand and open the dashboard in your browser.
|
||||
- Launcher command: run `subminer stats` to start the local stats server on demand (it also opens the dashboard in your browser when `stats.autoOpenBrowser` is enabled; the default is `false`).
|
||||
- Background server: run `subminer stats -b` to start or reuse a dedicated background stats daemon without keeping the launcher attached, and `subminer stats -s` to stop that daemon.
|
||||
- Maintenance command: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand.
|
||||
- Maintenance commands: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand; `subminer stats cleanup -l` repairs lifetime summary tables. `subminer stats rebuild` and `subminer stats backfill` rebuild or backfill rollup data.
|
||||
- Browser page: open `http://127.0.0.1:6969` directly if the local stats server is already running.
|
||||
|
||||
### Dashboard Tabs
|
||||
@@ -86,6 +87,7 @@ Stats server config lives under `stats`:
|
||||
{
|
||||
"stats": {
|
||||
"toggleKey": "Backquote",
|
||||
"markWatchedKey": "KeyW",
|
||||
"serverPort": 6969,
|
||||
"autoStartServer": true,
|
||||
"autoOpenBrowser": false,
|
||||
@@ -94,6 +96,7 @@ Stats server config lives under `stats`:
|
||||
```
|
||||
|
||||
- `toggleKey` is overlay-local, not a system-wide shortcut.
|
||||
- `markWatchedKey` toggles the watched state of the highlighted entry inside the stats dashboard.
|
||||
- `serverPort` controls the localhost dashboard URL.
|
||||
- `autoStartServer` starts the local stats HTTP server on launch once immersion tracking is active, or reuses the dedicated background stats server when one is already running. Background app launches (`subminer app`) start the stats server immediately, registering it so later launches reuse it instead of starting another one.
|
||||
- `autoOpenBrowser` controls whether `subminer stats` launches the dashboard URL in your browser after ensuring the server is running.
|
||||
@@ -185,7 +188,6 @@ SELECT
|
||||
total_watched_ms,
|
||||
active_watched_ms,
|
||||
lines_seen,
|
||||
words_seen,
|
||||
tokens_seen,
|
||||
cards_mined
|
||||
FROM imm_session_telemetry
|
||||
@@ -203,13 +205,13 @@ SELECT
|
||||
s.started_at_ms,
|
||||
s.ended_at_ms,
|
||||
COALESCE(s.active_watched_ms, 0) AS active_watched_ms,
|
||||
COALESCE(s.words_seen, 0) AS words_seen,
|
||||
COALESCE(s.tokens_seen, 0) AS tokens_seen,
|
||||
COALESCE(s.cards_mined, 0) AS cards_mined,
|
||||
CASE
|
||||
WHEN COALESCE(s.active_watched_ms, 0) > 0
|
||||
THEN COALESCE(s.words_seen, 0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
|
||||
THEN COALESCE(s.tokens_seen, 0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
|
||||
ELSE NULL
|
||||
END AS words_per_min,
|
||||
END AS tokens_per_min,
|
||||
CASE
|
||||
WHEN COALESCE(s.active_watched_ms, 0) > 0
|
||||
THEN (COALESCE(s.cards_mined, 0) * 60.0) / (COALESCE(s.active_watched_ms, 0) / 60000.0)
|
||||
@@ -229,7 +231,7 @@ SELECT
|
||||
la.total_sessions,
|
||||
la.total_active_ms,
|
||||
la.total_cards,
|
||||
la.total_words_seen,
|
||||
la.total_tokens_seen,
|
||||
la.total_lines_seen,
|
||||
la.first_watched_ms,
|
||||
la.last_watched_ms
|
||||
@@ -248,11 +250,10 @@ SELECT
|
||||
total_sessions,
|
||||
total_active_min,
|
||||
total_lines_seen,
|
||||
total_words_seen,
|
||||
total_tokens_seen,
|
||||
total_cards,
|
||||
cards_per_hour,
|
||||
words_per_min,
|
||||
tokens_per_min,
|
||||
lookup_hit_rate
|
||||
FROM imm_daily_rollups
|
||||
ORDER BY rollup_day DESC, video_id DESC
|
||||
@@ -268,7 +269,6 @@ SELECT
|
||||
total_sessions,
|
||||
total_active_min,
|
||||
total_lines_seen,
|
||||
total_words_seen,
|
||||
total_tokens_seen,
|
||||
total_cards
|
||||
FROM imm_monthly_rollups
|
||||
@@ -287,15 +287,19 @@ LIMIT ?;
|
||||
- Large-table reads are index-backed for `sample_ms`, session time windows, frequency-ranked words/kanji, and cover-art identity lookups.
|
||||
- Workload-dependent tuning knobs remain at defaults unless you change them: `cache_size`, `mmap_size`, `temp_store`, `auto_vacuum`.
|
||||
|
||||
### Schema (v4)
|
||||
### Schema (v18)
|
||||
|
||||
The exact schema version lives in `SCHEMA_VERSION` (`src/core/services/immersion-tracker/types.ts`) and is recorded in the `imm_schema_version` table.
|
||||
|
||||
Core tables:
|
||||
|
||||
- `imm_videos` - video key/title/source metadata
|
||||
- `imm_anime` - anime/series metadata referenced by videos and lifetime tables
|
||||
- `imm_sessions` - session UUID, video reference, timing/status, final denormalized totals
|
||||
- `imm_session_telemetry` - high-frequency session aggregates over time
|
||||
- `imm_session_events` - event stream with compact numeric event types
|
||||
- `imm_subtitle_lines` - persisted subtitle text and timing per session/video
|
||||
- `imm_youtube_videos` - YouTube video/channel metadata for tracked videos
|
||||
|
||||
Lifetime summary tables:
|
||||
|
||||
@@ -308,11 +312,14 @@ Rollup tables:
|
||||
|
||||
- `imm_daily_rollups`
|
||||
- `imm_monthly_rollups`
|
||||
- `imm_rollup_state` - incremental rollup progress bookkeeping
|
||||
|
||||
Vocabulary tables:
|
||||
|
||||
- `imm_words(id, headword, word, reading, first_seen, last_seen, frequency)`
|
||||
- `imm_words(id, headword, word, reading, part_of_speech, pos1, pos2, pos3, first_seen, last_seen, frequency, frequency_rank)` with `UNIQUE(headword, word, reading)`
|
||||
- `imm_kanji(id, kanji, first_seen, last_seen, frequency)`
|
||||
- `imm_word_line_occurrences` / `imm_kanji_line_occurrences` - word/kanji ↔ subtitle-line occurrence links
|
||||
- `imm_stats_excluded_words` - vocabulary exclusion list managed from the dashboard
|
||||
|
||||
Media-art tables:
|
||||
|
||||
|
||||
@@ -79,6 +79,7 @@ SubMiner extracts media info from the current video path to pre-fill the search
|
||||
|
||||
- **Season + episode patterns:** `S01E03`, `1x03`
|
||||
- **Episode-only patterns:** `E03`, `EP03`, or dash-separated numbers like `Title - 03 -`
|
||||
- **Season folders:** a parent directory named `Season 2` or `S2` fills in the season when the filename lacks one
|
||||
- **Bracket tags:** `[SubGroup]`, `[1080p]`, `[HEVC]` - stripped before title extraction
|
||||
- **Year tags:** `(2024)` - stripped
|
||||
- **Dots and underscores:** treated as spaces
|
||||
@@ -94,7 +95,7 @@ Configure `jimaku.apiKey` or `jimaku.apiKeyCommand` in your config. If using `ap
|
||||
|
||||
**"Jimaku request failed" or HTTP 429**
|
||||
|
||||
The Jimaku API has rate limits. If you see 429 errors, wait for the retry duration shown in the OSD message and try again. An API key provides higher rate limits.
|
||||
The Jimaku API has rate limits. If you see 429 errors, wait for the retry duration shown in the OSD message and try again.
|
||||
|
||||
**No entries found**
|
||||
|
||||
|
||||
@@ -54,6 +54,7 @@ The theme is auto-detected from these paths (first match wins):
|
||||
- `/usr/local/share/SubMiner/themes/subminer.rasi`
|
||||
- `/usr/share/SubMiner/themes/subminer.rasi`
|
||||
- macOS: `~/Library/Application Support/SubMiner/themes/subminer.rasi`
|
||||
- `assets/themes/subminer.rasi` next to the launcher script (final fallback)
|
||||
|
||||
Override with the `SUBMINER_ROFI_THEME` environment variable:
|
||||
|
||||
@@ -78,10 +79,37 @@ The first menu lists every locally watched series, most recently watched first,
|
||||
|
||||
Series whose directories are not currently accessible (e.g. an unmounted network share) are hidden from the list. Watch history requires the immersion tracker database (`immersionTracking.dbPath`, default `<config dir>/immersion.sqlite`), which SubMiner populates during playback.
|
||||
|
||||
## Sync Between Machines
|
||||
|
||||
`subminer sync <host>` merges immersion stats and watch history between two machines over SSH, so both end up with the union of sessions, lifetime totals, vocabulary counts, daily/monthly charts, and `--history` entries. `<host>` is anything `ssh` accepts (`user@hostname` or an ssh config alias); SubMiner must be installed on both machines at the same version.
|
||||
|
||||
```bash
|
||||
subminer sync macbook # two-way sync with the host "macbook"
|
||||
subminer sync macbook --push # merge local data into macbook only
|
||||
subminer sync macbook --pull # merge macbook data into local only
|
||||
subminer sync user@192.168.1.20 # explicit user@host
|
||||
subminer sync macbook --remote-cmd ~/bin/subminer # custom remote launcher path
|
||||
```
|
||||
|
||||
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over `scp`, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time — syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
|
||||
|
||||
For a one-way transfer, `--push` snapshots the local database and merges it into the host without changing the local database. `--pull` snapshots the host and merges it into the local database without changing the host. These modes add missing data; they do not delete destination-only data or make the destination an exact mirror.
|
||||
|
||||
Close SubMiner (and stop the background stats daemon, `subminer stats -s`) on both machines before syncing; the command refuses to run while a SubMiner process may be writing the database (`--force` overrides). The mpv safety check requires a live socket connection, so a stale socket file left after mpv exits does not block sync. Both machines must be on the same SubMiner version — the sync aborts on a stats schema mismatch. Remote sync checks standard SubMiner and Bun locations (`~/.local/bin`, `~/.bun/bin`, Homebrew, `/usr/local/bin`, `/usr/bin`, and `/bin`) even when the non-interactive SSH shell omits them from `PATH`.
|
||||
|
||||
Two lower-level modes are used internally over SSH and also work standalone for manual transfers (e.g. via a USB drive):
|
||||
|
||||
```bash
|
||||
subminer sync --snapshot /tmp/stats.sqlite # write a consistent snapshot of the local database
|
||||
subminer sync --merge /tmp/stats.sqlite # merge a snapshot file into the local database
|
||||
```
|
||||
|
||||
Unfinished sessions (a crash mid-playback) are skipped until the app finalizes them; they sync on the next run. Word/kanji "known" state from Anki is not part of the database and does not sync — each machine derives it from its own Anki collection.
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
subminer video.mkv # play a specific file (default plugin config auto-starts visible overlay)
|
||||
subminer video.mkv # play a specific file (managed launches auto-start the visible overlay by default)
|
||||
subminer https://youtu.be/... # YouTube playback (requires yt-dlp)
|
||||
subminer --backend x11 video.mkv # Force x11 backend for a specific file
|
||||
subminer -u # check for SubMiner updates
|
||||
@@ -94,11 +122,12 @@ subminer stats -b # start background stats daemon
|
||||
|
||||
| Subcommand | Purpose |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------ |
|
||||
| `subminer jellyfin` / `jf` | Jellyfin workflows (`-d` discovery, `-p` play, `-l` login) |
|
||||
| `subminer stats` | Start stats server and open immersion dashboard in browser |
|
||||
| `subminer stats -b` | Start or reuse background stats daemon (non-blocking) |
|
||||
| `subminer stats cleanup` | Backfill vocabulary metadata and prune stale rows |
|
||||
| `subminer doctor` | Dependency + config + socket diagnostics |
|
||||
| `subminer jellyfin` / `jf` | Jellyfin workflows (`-d` discovery, `-p` play, `-l` login, `--logout`, `--setup`) |
|
||||
| `subminer stats` | Start the stats server (opens the dashboard when `stats.autoOpenBrowser` is on) |
|
||||
| `subminer stats -b` / `-s` | Start/reuse or stop the background stats daemon |
|
||||
| `subminer stats cleanup` | Backfill vocabulary metadata and prune stale rows (`-v` vocab, `-l` lifetime summaries) |
|
||||
| `subminer stats rebuild` / `backfill` | Rebuild or backfill rollup data |
|
||||
| `subminer doctor` | Dependency + config + socket diagnostics (`--refresh-known-words` refreshes the known-word cache) |
|
||||
| `subminer settings` | Open the SubMiner settings window |
|
||||
| `subminer logs -e` | Export a sanitized local-date log ZIP and print its path |
|
||||
| `subminer config path` | Print active config file path |
|
||||
@@ -106,12 +135,15 @@ subminer stats -b # start background stats daemon
|
||||
| `subminer mpv status` | Check mpv socket readiness |
|
||||
| `subminer mpv socket` | Print active socket path |
|
||||
| `subminer mpv idle` | Launch detached idle mpv instance |
|
||||
| `subminer dictionary <path>` | Generate character dictionary ZIP from file/dir target |
|
||||
| `subminer sync <host>` | Two-way stats/history sync with another machine over SSH |
|
||||
| `subminer sync <host> --push` | Merge local stats/history into another machine only |
|
||||
| `subminer sync <host> --pull` | Merge another machine's stats/history into the local database only |
|
||||
| `subminer dictionary <path>` / `dict` | Generate character dictionary ZIP from file/dir target |
|
||||
| `subminer dictionary --candidates <path>` | List AniList candidate matches for character dictionary correction |
|
||||
| `subminer dictionary --select <id> <path>` | Pin an AniList media ID for that target series |
|
||||
| `subminer texthooker` | Launch texthooker-only mode |
|
||||
| `subminer texthooker -o` | Launch texthooker and open it in the default browser |
|
||||
| `subminer app` | Pass arguments directly to SubMiner binary |
|
||||
| `subminer app` / `bin` | Pass arguments directly to SubMiner binary (e.g. `subminer app --setup`) |
|
||||
|
||||
Use `subminer <subcommand> -h` for command-specific help.
|
||||
|
||||
@@ -123,24 +155,24 @@ Use `subminer <subcommand> -h` for command-specific help.
|
||||
| `-r, --recursive` | Search directories recursively |
|
||||
| `-R, --rofi` | Use rofi instead of fzf |
|
||||
| `-H, --history` | Browse local watch history (see [Watch History](#watch-history)) |
|
||||
| `--setup` | Open first-run setup popup manually |
|
||||
| `-v, --version` | Print installed SubMiner version |
|
||||
| `-v, --version` | Print the launcher's own version (can differ from the installed app binary) |
|
||||
| `-u, --update` | Check for SubMiner updates and update the app/launcher when possible |
|
||||
| `--start` | Explicitly start overlay after mpv launches |
|
||||
| `-S, --start-overlay` | Explicitly start overlay after mpv launches |
|
||||
| `-S, --start-overlay` | Force the visible overlay on start |
|
||||
| `-T, --no-texthooker` | Disable texthooker server |
|
||||
| `-p, --profile` | mpv profile name (no default; omitted unless set) |
|
||||
| `-a, --args` | Pass additional mpv arguments as a quoted string |
|
||||
| `-b, --backend` | Force window backend (`hyprland`, `sway`, `x11`, `macos`, `windows`) |
|
||||
| `--settings` | Open the SubMiner settings window |
|
||||
| `--log-level` | Logger verbosity (`debug`, `info`, `warn`, `error`) |
|
||||
| `--dev`, `--debug` | Enable app dev-mode (not tied to log level) |
|
||||
|
||||
App-binary flags such as `--setup`, `--dev`, and `--debug` are not launcher flags - pass them through with `subminer app`, for example `subminer app --setup`.
|
||||
|
||||
On Linux, `subminer -u` updates from the launcher process itself. It can check and replace the AppImage, launcher, runtime plugin copy, and rofi theme even when SubMiner is already running in the tray.
|
||||
|
||||
With default plugin settings (`auto_start=yes`, `auto_start_visible_overlay=yes`, `auto_start_pause_until_ready=yes`), explicit start flags are usually unnecessary.
|
||||
Managed launches inject `auto_start=yes`, `auto_start_visible_overlay=yes`, and `auto_start_pause_until_ready=yes` as plugin script-opts from SubMiner's config defaults (`mpv.autoStartSubMiner`, `auto_start_overlay`), so explicit start flags are usually unnecessary. The plugin's own built-in defaults are off - mpv launched outside SubMiner does not auto-start the overlay.
|
||||
|
||||
## Logging
|
||||
|
||||
- Default log level is `info`
|
||||
- `--background` mode defaults to `warn` unless `--log-level` is explicitly set
|
||||
- `--dev` / `--debug` control app behavior, not logging verbosity - use `--log-level` for that
|
||||
- Default log level is `warn` (launcher and app; configurable via `logging.level`)
|
||||
- `--dev` / `--debug` are app-binary flags that control app dev-mode, not logging verbosity - use `--log-level` for that
|
||||
|
||||
@@ -56,20 +56,20 @@ This is useful when auto-update is disabled or when you want explicit control ov
|
||||
Create a standalone sentence card without going through Yomitan:
|
||||
|
||||
- **Mine current sentence**: `Ctrl/Cmd+S` (configurable via `shortcuts.mineSentence`)
|
||||
- **Mine multiple lines**: `Ctrl/Cmd+Shift+S` followed by a digit 1–9 to select how many recent subtitle lines to combine.
|
||||
- **Mine multiple lines**: `Ctrl/Cmd+Shift+S` followed by a digit 1–9 to select how many recent subtitle lines to combine (the digit selector times out after 3 seconds, configurable via `shortcuts.multiCopyTimeoutMs`).
|
||||
|
||||
The sentence card uses the note type configured in `isLapis.sentenceCardModel` and always maps sentence/audio to `Sentence` and `SentenceAudio`.
|
||||
|
||||
::: warning Requires Lapis/Kiku note type
|
||||
Sentence card creation requires a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type and `ankiConnect.isLapis.enabled: true` in your config. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
|
||||
Sentence card creation requires `ankiConnect.isLapis.sentenceCardModel` to name a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type that exists in Anki (default: `"Lapis"`). See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
|
||||
:::
|
||||
|
||||
### 4. Mark as Audio Card
|
||||
|
||||
After adding a word via Yomitan, press the audio card shortcut to overwrite the audio with a longer clip spanning the full subtitle timing.
|
||||
After adding a word via Yomitan, press the audio card shortcut (`Ctrl/Cmd+Shift+A` by default, `shortcuts.markAudioCard`) to mark the card as an audio card. This sets the audio-card flag and fills sentence, image, and metadata fields alongside the full-subtitle audio clip.
|
||||
|
||||
::: warning Requires Lapis/Kiku note type
|
||||
Audio card marking requires a [Lapis](https://github.com/donkuri/lapis) or [Kiku](https://github.com/youyoumu/kiku) compatible note type and `ankiConnect.isLapis.enabled: true` in your config. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
|
||||
Audio card marking uses the same `ankiConnect.isLapis.sentenceCardModel` note type as sentence cards. See [Anki Integration - Sentence Cards](/anki-integration#sentence-cards-lapis) for setup.
|
||||
:::
|
||||
|
||||
### Field Grouping (Kiku)
|
||||
@@ -172,7 +172,7 @@ Install the sync tools separately - see [Troubleshooting](/troubleshooting#subti
|
||||
|
||||
## Texthooker
|
||||
|
||||
SubMiner runs a local HTTP server at `http://127.0.0.1:5174` (configurable port) that serves a texthooker UI. This allows external tools - such as a browser-based Yomitan instance - to receive subtitle text in real time.
|
||||
SubMiner runs a local HTTP server at `http://127.0.0.1:5174` (fixed default port; overridable only via the mpv plugin's `texthooker_port` script-opt) that serves a texthooker UI. This allows external tools - such as a browser-based Yomitan instance - to receive subtitle text in real time.
|
||||
|
||||
The texthooker page displays the current subtitle and updates as new lines arrive. This is useful if you prefer to do lookups in a browser rather than through the overlay's built-in Yomitan.
|
||||
|
||||
|
||||
+42
-11
@@ -4,7 +4,7 @@
|
||||
|
||||
**Who needs this page:** Most users never touch the plugin directly - SubMiner-managed launches (the app, the `subminer` launcher, or the Windows shortcut) inject the bundled plugin automatically for that session, so there is nothing to install into mpv's global `scripts` directory. Read on if you launch mpv from another tool and want SubMiner's in-player controls, or you want to script mpv against SubMiner.
|
||||
|
||||
The plugin ships as a modular Lua package under `plugin/subminer/` (entry point `init.lua`, which loads `main.lua` and sibling modules). Earlier releases shipped a single global `main.lua`; runtime loading replaces it.
|
||||
The plugin ships as a modular Lua package under `plugin/subminer/` (entry point `main.lua`, which loads `init.lua` and sibling modules). Earlier releases shipped a single global `main.lua`; runtime loading replaces it.
|
||||
|
||||
## Runtime Loading
|
||||
|
||||
@@ -27,6 +27,25 @@ On Windows, use a named pipe instead:
|
||||
input-ipc-server=\\.\pipe\subminer-socket
|
||||
```
|
||||
|
||||
## Configuration (script-opts)
|
||||
|
||||
The plugin reads options from `script-opts` with the `subminer-` prefix (for example `--script-opts=subminer-backend=hyprland`). Managed launches inject these automatically from your SubMiner config; the shipped `subminer.conf` is intentionally empty so command-line opts always win.
|
||||
|
||||
| Option | Default | Description |
|
||||
| -------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------- |
|
||||
| `binary_path` | `""` | Path to the SubMiner binary; empty enables [auto-detection](#binary-auto-detection) |
|
||||
| `socket_path` | platform default | mpv IPC socket path (`/tmp/subminer-socket`, or `\\.\pipe\subminer-socket` on Windows) |
|
||||
| `texthooker_enabled` | `no` | Start the texthooker server with the overlay |
|
||||
| `texthooker_port` | `5174` | Texthooker server port |
|
||||
| `backend` | `auto` | Window backend (`auto`, `hyprland`, `sway`, `x11`, `macos`) |
|
||||
| `auto_start` | `no` | Start the overlay app on `file-loaded` (managed launches set this from `mpv.autoStartSubMiner`) |
|
||||
| `auto_start_visible_overlay` | `no` | Show the visible overlay on auto-start (from `auto_start_overlay` in config) |
|
||||
| `overlay_loading_osd` | `no` | Show an OSD loading spinner while the overlay starts |
|
||||
| `auto_start_pause_until_ready` | `yes` | Keep mpv paused until the overlay reports tokenization-ready |
|
||||
| `auto_start_pause_until_ready_timeout_seconds` | `30` | Timeout before resuming playback anyway |
|
||||
| `osd_messages` | `yes` | Show plugin OSD status messages |
|
||||
| `log_level` | `info` | Plugin log verbosity |
|
||||
|
||||
## Keybindings
|
||||
|
||||
Most plugin actions use a `y` chord prefix - press `y`, then the second key (a "chord"):
|
||||
@@ -44,7 +63,7 @@ Most plugin actions use a `y` chord prefix - press `y`, then the second key (a "
|
||||
| `v` | Toggle primary subtitle bar visibility |
|
||||
| `TAB` (default) | Skip intro (AniSkip) |
|
||||
|
||||
The AniSkip key is **not** a `y` chord and is not bound by the plugin: the SubMiner app binds it over the mpv IPC socket while it is connected. It defaults to `TAB` and is configurable via `mpv.aniskipButtonKey`. The legacy `y-k` chord still works as a fallback unless you remap the AniSkip key onto it. See [AniSkip Integration](/aniskip-integration) for setup and details.
|
||||
The AniSkip key is **not** a `y` chord and is not bound by the plugin: the SubMiner app binds it over the mpv IPC socket while it is connected. It defaults to `TAB` and is configurable via `mpv.aniskipButtonKey`. When a custom key (other than `TAB` or `y-k`) is configured, the legacy `y-k` chord is also bound as a fallback. See [AniSkip Integration](/aniskip-integration) for setup and details.
|
||||
|
||||
The bare `v` binding is a forced mpv binding. It overrides mpv's default primary subtitle visibility toggle and routes the action to SubMiner's primary subtitle bar instead.
|
||||
|
||||
@@ -70,7 +89,7 @@ Notes:
|
||||
|
||||
## Menu
|
||||
|
||||
Press `y-y` to open an interactive menu in mpv's OSD:
|
||||
Press `y-y` to open an interactive menu (rendered with mpv's console selector):
|
||||
|
||||
```text
|
||||
SubMiner:
|
||||
@@ -80,6 +99,7 @@ SubMiner:
|
||||
4. Open options
|
||||
5. Restart overlay
|
||||
6. Check status
|
||||
7. Stats
|
||||
```
|
||||
|
||||
Select an item by pressing its number.
|
||||
@@ -92,8 +112,8 @@ When `binary_path` is empty, the plugin searches platform-specific locations:
|
||||
|
||||
1. `~/.local/bin/SubMiner.AppImage`
|
||||
2. `/opt/SubMiner/SubMiner.AppImage`
|
||||
3. `/usr/local/bin/SubMiner`
|
||||
4. `/usr/bin/SubMiner`
|
||||
3. `/usr/local/bin/SubMiner` / `/usr/local/bin/subminer`
|
||||
4. `/usr/bin/SubMiner` / `/usr/bin/subminer`
|
||||
|
||||
**macOS:**
|
||||
|
||||
@@ -102,11 +122,14 @@ When `binary_path` is empty, the plugin searches platform-specific locations:
|
||||
|
||||
**Windows:**
|
||||
|
||||
1. `C:\Program Files\SubMiner\SubMiner.exe`
|
||||
2. `C:\Program Files (x86)\SubMiner\SubMiner.exe`
|
||||
3. `C:\SubMiner\SubMiner.exe`
|
||||
A PowerShell system lookup runs first (running SubMiner process, registry App Paths, `Get-Command`), then static paths:
|
||||
|
||||
Packaged Windows plugin installs also rewrite `socket_path` to `\\.\pipe\subminer-socket` automatically.
|
||||
1. `%LOCALAPPDATA%\Programs\SubMiner\SubMiner.exe` (the default per-user install location)
|
||||
2. `C:\Program Files\SubMiner\SubMiner.exe`
|
||||
3. `C:\Program Files (x86)\SubMiner\SubMiner.exe`
|
||||
4. `C:\SubMiner\SubMiner.exe`
|
||||
|
||||
On Windows the plugin also normalizes a Unix-style `socket_path` (`/tmp/subminer-socket`) to the named pipe `\\.\pipe\subminer-socket` at runtime.
|
||||
|
||||
## Backend Detection
|
||||
|
||||
@@ -135,8 +158,16 @@ script-message subminer-options
|
||||
script-message subminer-restart
|
||||
script-message subminer-status
|
||||
script-message subminer-autoplay-ready
|
||||
script-message subminer-stats-toggle
|
||||
script-message subminer-visible-overlay-shown
|
||||
script-message subminer-visible-overlay-hidden
|
||||
script-message subminer-managed-subtitles-loading
|
||||
script-message subminer-overlay-loading-ready
|
||||
script-message subminer-reload-session-bindings
|
||||
```
|
||||
|
||||
The last five are primarily used by the SubMiner app to notify the plugin of overlay/loading state and to trigger session-binding reloads.
|
||||
|
||||
The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) still exist, but they are handled by the SubMiner app over the IPC socket rather than by the plugin - see [AniSkip Integration](/aniskip-integration#triggering-from-mpv).
|
||||
|
||||
The `subminer-start` message accepts overrides:
|
||||
@@ -155,8 +186,8 @@ For how the plugin's auto-start fits into the full launch sequence - including w
|
||||
- **File loaded**: If `auto_start=yes`, the plugin starts the overlay.
|
||||
- **Auto-start pause gate**: If `auto_start_visible_overlay=yes` and `auto_start_pause_until_ready=yes`, launcher starts mpv paused. On cold managed background startup, SubMiner opens the tray and visible overlay shell before tokenization warmups finish, then the plugin resumes playback after SubMiner reports tokenization-ready (with a 30-second timeout fallback).
|
||||
- **Duplicate auto-start events**: Repeated `file-loaded` hooks while overlay is already running are ignored for auto-start triggers (prevents duplicate start attempts).
|
||||
- **MPV shutdown**: The plugin sends a stop command to gracefully shut down both the overlay and the texthooker server.
|
||||
- **Texthooker**: Starts as a separate subprocess before the overlay to ensure the app lock is acquired first.
|
||||
- **MPV shutdown**: The plugin clears its hover/OSD/gate state on shutdown; the overlay app notices the closed IPC socket and shuts itself down.
|
||||
- **Texthooker**: When `texthooker_enabled=yes`, the plugin appends `--texthooker` to the overlay start command so the app starts the texthooker server alongside the overlay.
|
||||
|
||||
## Using with the `subminer` Wrapper
|
||||
|
||||
|
||||
+8
-10
@@ -10,17 +10,15 @@ A few terms used throughout:
|
||||
|
||||
All shortcuts are configurable in `config.jsonc` under `shortcuts` and `keybindings`. Set any shortcut to `null` to disable it.
|
||||
|
||||
## Global Shortcuts
|
||||
## App-Wide Shortcuts
|
||||
|
||||
These work system-wide regardless of which window has focus.
|
||||
|
||||
| Shortcut | Action | Configurable |
|
||||
| ------------- | ---------------------- | -------------------------------------- |
|
||||
| `Alt+Shift+O` | Toggle visible overlay | `shortcuts.toggleVisibleOverlayGlobal` |
|
||||
| `Alt+Shift+Y` | Open Yomitan settings | Fixed (not configurable) |
|
||||
| Shortcut | Action | Scope | Configurable |
|
||||
| ------------- | ---------------------- | -------------------------------------------- | -------------------------------------- |
|
||||
| `Alt+Shift+O` | Toggle visible overlay | Works while the overlay or mpv has focus | `shortcuts.toggleVisibleOverlayGlobal` |
|
||||
| `Alt+Shift+Y` | Open Yomitan settings | OS-global (registered with the OS) | Fixed (not configurable) |
|
||||
|
||||
::: tip
|
||||
Global shortcuts are registered with the OS. If they conflict with another application, update them in `shortcuts` config and restart SubMiner.
|
||||
`Alt+Shift+O` is dispatched by the overlay window and the mpv plugin, so it works from either surface without OS registration. Only `Alt+Shift+Y` is registered with the OS; if it conflicts with another application, that binding cannot be changed. All `shortcuts.*` keys hot-reload - no restart needed.
|
||||
:::
|
||||
|
||||
## Mining Shortcuts
|
||||
@@ -88,7 +86,7 @@ Mouse-hover playback behavior is configured separately from shortcuts: `subtitle
|
||||
| `Ctrl/Cmd+N` | Toggle overlay notification history panel | `shortcuts.toggleNotificationHistory` |
|
||||
| `Ctrl+Alt+C` | Open the manual YouTube subtitle picker | `keybindings` |
|
||||
| `Ctrl+Alt+S` | Open subtitle sync (subsync) modal | `shortcuts.triggerSubsync` |
|
||||
| `\` | Toggle subtitle sidebar | `subtitleSidebar.toggleKey` |
|
||||
| `\` | Toggle subtitle sidebar | `subtitleSidebar.toggleKey` (overlay) / `shortcuts.toggleSubtitleSidebar` (mpv session binding) |
|
||||
| `` ` `` | Toggle stats overlay | `stats.toggleKey` |
|
||||
| `W` | Mark current video watched and advance to next in queue | `stats.markWatchedKey` |
|
||||
|
||||
@@ -109,7 +107,7 @@ Controller input only drives the overlay while keyboard-only mode is enabled. Th
|
||||
|
||||
## MPV Plugin Chords
|
||||
|
||||
When the mpv plugin is installed, all commands use a `y` chord prefix - press `y`, then the second key within 1 second.
|
||||
When the mpv plugin is installed, all commands use a `y` chord prefix - press `y`, then the second key (the overlay-side chord times out after 1 second; the mpv plugin uses native mpv key sequences).
|
||||
|
||||
| Chord | Action |
|
||||
| ----- | ---------------------------------------------------------- |
|
||||
|
||||
@@ -64,7 +64,7 @@ For full details on dictionary generation, name variant expansion, auto-sync lif
|
||||
|
||||
## Frequency Highlighting
|
||||
|
||||
Frequency highlighting colors tokens based on how common they are, using dictionary frequency rank data. This helps you spot high-value vocabulary at a glance. Frequency ranks are sourced from the **highest-ranked frequency dictionary** installed in Yomitan - other frequency dictionaries are not consulted.
|
||||
Frequency highlighting colors tokens based on how common they are, using dictionary frequency rank data. This helps you spot high-value vocabulary at a glance. For each token, ranks from the installed Yomitan frequency dictionaries are consulted in priority order: the highest-priority dictionary that has the term wins, lower-priority dictionaries fill in terms it lacks, and occurrence-based dictionaries are skipped.
|
||||
|
||||
**Modes:**
|
||||
|
||||
@@ -126,22 +126,24 @@ All colors are customizable via the `subtitleStyle.jlptColors` object.
|
||||
|
||||
## Runtime Toggles
|
||||
|
||||
All annotation layers can be toggled at runtime via the mpv command menu without restarting:
|
||||
These annotation layers can be toggled at runtime via the runtime options palette (`Ctrl/Cmd+Shift+O`) without restarting:
|
||||
|
||||
- `ankiConnect.knownWords.highlightEnabled` (`On` / `Off`)
|
||||
- `subtitleStyle.nameMatchEnabled` (`On` / `Off`)
|
||||
- `subtitleStyle.nameMatchImagesEnabled` (`On` / `Off`)
|
||||
- `ankiConnect.knownWords.matchMode`
|
||||
- `ankiConnect.nPlusOne.enabled` (`On` / `Off`)
|
||||
- `subtitleStyle.enableJlpt` (`On` / `Off`)
|
||||
- `subtitleStyle.frequencyDictionary.enabled` (`On` / `Off`)
|
||||
|
||||
(Character-name matching, `subtitleStyle.nameMatchEnabled`, is toggled through config or the Settings window, not the runtime palette.)
|
||||
|
||||
Toggles only apply to new subtitle lines after the change - the currently displayed line is not re-tokenized in place.
|
||||
|
||||
## Rendering Priority
|
||||
|
||||
When multiple annotations apply to the same token, the visual priority is:
|
||||
|
||||
1. **N+1 target** (highest) - the single unknown word in an N+1 sentence
|
||||
2. **Character-name match** - dictionary-driven character-name token styling
|
||||
1. **Character-name match** (highest) - dictionary-driven character-name token styling; it clears the token's N+1, frequency, and JLPT annotations
|
||||
2. **N+1 target** - the single unknown word in an N+1 sentence
|
||||
3. **Known-word color** - already-learned token tint
|
||||
4. **Frequency highlight** - common-word coloring (not applied when N+1/character-name/known-word already matched)
|
||||
5. **JLPT underline** - level-based underline (stacks with the above since it uses underline rather than text color)
|
||||
4. **Frequency highlight** - common-word coloring (not applied when a higher layer already matched)
|
||||
5. **JLPT underline** - level-based underline (stacks with N+1/known/frequency since it uses underline rather than text color, but not with a character-name match)
|
||||
|
||||
@@ -164,7 +164,7 @@ Shown when SubMiner tries to update a card that no longer exists, or when AnkiCo
|
||||
**Overlay does not appear**
|
||||
|
||||
- Confirm SubMiner is running: `SubMiner.AppImage --start` or check for the process.
|
||||
- On Linux, the overlay requires a supported window backend. Hyprland and Sway have native Wayland support; all other compositors require both mpv and SubMiner to run under X11 or Xwayland (`xdotool` and `xwininfo` must be installed).
|
||||
- On Linux, the overlay requires a supported window backend. Hyprland and Sway have native Wayland support; all other compositors require both mpv and SubMiner to run under X11 or Xwayland (`xdotool`, `xprop`, and `xwininfo` must be installed).
|
||||
- On macOS, grant Accessibility permission to SubMiner in System Settings > Privacy & Security > Accessibility.
|
||||
|
||||
**Overlay appears but clicks pass through / cannot interact**
|
||||
@@ -177,7 +177,7 @@ Shown when SubMiner tries to update a card that no longer exists, or when AnkiCo
|
||||
|
||||
- Renderer errors now trigger an automatic recovery path. You should see a short toast ("Renderer error recovered. Overlay is still running.").
|
||||
- Recovery closes any open modal and restores click-through/shortcuts automatically without interrupting mpv playback.
|
||||
- If errors keep recurring, toggle the overlay's DevTools using overlay chord `y` then `d` (or global `F12`) and inspect the `renderer overlay recovery` error payload for stack trace + modal/subtitle context.
|
||||
- If errors keep recurring, toggle the overlay's DevTools using overlay chord `y` then `d` (`F12` also works in dev builds) and inspect the `renderer overlay recovery` error payload for stack trace + modal/subtitle context.
|
||||
|
||||
**Overlay is on the wrong monitor or position**
|
||||
|
||||
@@ -258,22 +258,22 @@ Without FFmpeg, card creation still works but audio and image fields will be emp
|
||||
Media generation has a 30-second timeout (60 seconds for animated AVIF). If your video file is on a slow network mount or the codec requires software decoding, generation may time out. Try:
|
||||
|
||||
- Using a local copy of the video file.
|
||||
- Reducing `media.imageQuality` or switching from `avif` to `static` image type.
|
||||
- Checking that `media.maxMediaDuration` is not set too high.
|
||||
- Reducing `ankiConnect.media.imageQuality` or switching from `avif` to `static` image type.
|
||||
- Checking that `ankiConnect.media.maxMediaDuration` is not set too high.
|
||||
|
||||
## Shortcuts
|
||||
|
||||
**"Failed to register global shortcut"**
|
||||
|
||||
Global shortcuts (`Alt+Shift+O`, `Alt+Shift+Y`) may conflict with other applications or desktop environment keybindings.
|
||||
This warning refers to the OS-registered shortcut `Alt+Shift+Y` (Yomitan settings), which is fixed and may conflict with other applications or desktop environment keybindings.
|
||||
|
||||
- Check your DE/WM keybinding settings for conflicts.
|
||||
- Change the shortcut in your config under `shortcuts.toggleVisibleOverlayGlobal`.
|
||||
- Check your DE/WM keybinding settings for conflicts and free up `Alt+Shift+Y` there.
|
||||
- `Alt+Shift+O` (`shortcuts.toggleVisibleOverlayGlobal`) is not OS-registered - it is handled by the overlay window and the mpv plugin, so it does not trigger this warning and only needs those windows focused.
|
||||
- On Wayland, global shortcut registration has limitations depending on the compositor. Only Hyprland and Sway are supported natively - see the [Hyprland](#hyprland) section below for shortcut passthrough rules. Other Wayland compositors require X11/Xwayland.
|
||||
|
||||
**Overlay keybindings not working**
|
||||
|
||||
Overlay-local shortcuts (Space, arrow keys, etc.) only work when the overlay window has focus. Click on the overlay or use the global shortcut to toggle it to give it focus.
|
||||
Overlay-local shortcuts (Space, arrow keys, etc.) only work when the overlay window has focus. Click on the overlay or use `Alt+Shift+O` (with the overlay or mpv focused) to toggle it and give it focus.
|
||||
|
||||
## Subtitle Timing
|
||||
|
||||
@@ -307,9 +307,9 @@ Install ffsubsync or configure the path:
|
||||
- **pip**: `pip install ffsubsync`
|
||||
- Must be on `PATH` or configured via `subsync.ffsubsync_path` in your config.
|
||||
|
||||
**"Subtitle synchronization failed"**
|
||||
**"alass synchronization failed" / "ffsubsync synchronization failed"**
|
||||
|
||||
If subtitle sync fails:
|
||||
If subtitle sync fails (the error message is prefixed with the engine name):
|
||||
|
||||
- Ensure the reference subtitle track exists in the video (alass requires a source track).
|
||||
- Check that `ffmpeg` is available (used to extract the internal subtitle track).
|
||||
|
||||
+37
-15
@@ -60,13 +60,20 @@ The mpv plugin is always available - it's bundled with SubMiner and injected at
|
||||
|
||||
While SubMiner is running, it watches your active config file and applies safe updates automatically.
|
||||
|
||||
Live-updated settings:
|
||||
Live-updated settings include:
|
||||
|
||||
- `subtitleStyle`
|
||||
- `keybindings`
|
||||
- `shortcuts`
|
||||
- `secondarySub.defaultMode`
|
||||
- `ankiConnect.ai`
|
||||
- `subtitleSidebar`
|
||||
- `notifications`
|
||||
- `logging`
|
||||
- `jimaku`, `subsync`
|
||||
- `mpv.aniskipEnabled`, `mpv.aniskipButtonKey`
|
||||
- `stats.toggleKey`, `stats.markWatchedKey`
|
||||
- `youtube.primarySubLanguages`
|
||||
- most `ankiConnect.*` settings (including `ankiConnect.ai`)
|
||||
|
||||
Invalid config edits are rejected; SubMiner keeps the previous valid runtime config and shows an error notification.
|
||||
For restart-required sections, SubMiner shows a restart-needed notification.
|
||||
@@ -82,12 +89,13 @@ subminer -R # Use rofi instead of fzf
|
||||
subminer -d ~/Videos # Specific directory
|
||||
subminer -r -d ~/Anime # Recursive search
|
||||
subminer video.mkv # Play specific file (overlay auto-starts)
|
||||
subminer --start video.mkv # Explicit overlay start (use when auto_start=no in config)
|
||||
subminer -S video.mkv # Same as above via --start-overlay
|
||||
subminer --start video.mkv # Explicit overlay start (use when mpv.autoStartSubMiner is false in config)
|
||||
subminer -S video.mkv # Also force the visible overlay on start (--start-overlay)
|
||||
subminer https://youtu.be/... # Play a YouTube URL
|
||||
subminer ytsearch:"jp news" # Play first YouTube search result
|
||||
subminer --setup # Open first-run setup popup
|
||||
subminer --version # Print installed SubMiner version
|
||||
subminer -H # Browse watch history (replay/continue episodes, fzf or rofi picker)
|
||||
subminer app --setup # Open first-run setup popup
|
||||
subminer --version # Print the launcher's version
|
||||
subminer -v # Same as above
|
||||
subminer --log-level debug video.mkv # Enable verbose logs for launch/debugging
|
||||
subminer --log-level warn video.mkv # Set logging level explicitly
|
||||
@@ -111,11 +119,19 @@ subminer config show # Print active config contents
|
||||
subminer mpv socket # Print active mpv socket path
|
||||
subminer mpv status # Exit 0 if socket is ready, else exit 1
|
||||
subminer mpv idle # Launch detached idle mpv with SubMiner defaults
|
||||
subminer sync media-box # Sync stats/watch history with an SSH host
|
||||
subminer sync media-box --push # Merge this machine's stats into the host only
|
||||
subminer sync media-box --pull # Merge the host's stats into this machine only
|
||||
subminer sync --snapshot ~/subminer-snapshot.sqlite # Write a local DB snapshot
|
||||
subminer sync --merge ~/subminer-snapshot.sqlite # Merge a snapshot into the local DB
|
||||
subminer dictionary /path/to/file-or-directory # Generate character dictionary ZIP from target (manual Yomitan import)
|
||||
subminer dictionary --candidates /path/to/file.mkv
|
||||
subminer dictionary --select 21355 /path/to/file.mkv
|
||||
subminer texthooker # Launch texthooker-only mode
|
||||
subminer texthooker -o # Launch texthooker and open it in your browser
|
||||
subminer stats # Start the local stats server (see Immersion Tracking)
|
||||
subminer stats -b # Start/reuse the background stats daemon
|
||||
subminer stats -s # Stop the background stats daemon
|
||||
subminer app --anilist-setup # Pass args directly to SubMiner binary (example: AniList login flow)
|
||||
|
||||
# Direct packaged app control
|
||||
@@ -129,6 +145,7 @@ SubMiner.AppImage --start --toggle # Start MPV IPC + toggle visibility
|
||||
SubMiner.AppImage --show-visible-overlay # Force show visible overlay
|
||||
SubMiner.AppImage --hide-visible-overlay # Force hide visible overlay
|
||||
SubMiner.AppImage --toggle-primary-subtitle-bar # Toggle primary subtitle bar visibility
|
||||
SubMiner.AppImage --toggle-subtitle-sidebar # Toggle the subtitle sidebar
|
||||
SubMiner.AppImage --start --dev # Enable app/dev mode only
|
||||
SubMiner.AppImage --start --debug # Alias for --dev
|
||||
SubMiner.AppImage --start --log-level debug # Force verbose logging without app/dev mode
|
||||
@@ -194,6 +211,7 @@ This flow requires `mpv.exe` to be discoverable. Leave `mpv.executablePath` blan
|
||||
- `subminer logs -e`: export a sanitized ZIP of today's local-date logs, or the most recent logs when no current-day log exists. The exported copy masks common PII and secrets; on-disk logs are unchanged.
|
||||
- `subminer config`: config file helpers (`path`, `show`).
|
||||
- `subminer mpv`: mpv helpers (`status`, `socket`, `idle`).
|
||||
- `subminer sync <host>`: sync immersion stats and watch history with another machine over SSH. The host is the SSH destination (`user@host` or an SSH config alias). Use `--push` to merge only this machine's data into the host, or `--pull` to merge only the host's data into this machine; both remain insert-only and do not make either database an exact mirror. Remote launcher checks include standard SubMiner and Bun paths even when SSH omits them from `PATH`. Use `--snapshot <file>` to write a consistent local stats DB snapshot, `--merge <file>` to merge a snapshot into the local stats DB, and `--force` to skip the running stats/mpv safety check. Advanced options: `--db <file>` overrides the local stats DB path, and `--remote-cmd <cmd>` overrides the `subminer` command used on the remote host.
|
||||
- `subminer dictionary <path>`: generates a Yomitan-importable character dictionary ZIP from a file/directory target.
|
||||
- Use `subminer dictionary --candidates <path>` and `subminer dictionary --select <id> <path>` to correct AniList character-dictionary matches for a whole series.
|
||||
- `subminer texthooker`: texthooker-only shortcut (same behavior as `--texthooker`). A _texthooker_ is a web page that displays the current subtitle line as selectable text, so browser-based dictionary extensions and other tools can read along with playback.
|
||||
@@ -207,7 +225,7 @@ Setup popup appears on first launch, or when setup has not been completed.
|
||||
You can also open it manually:
|
||||
|
||||
```bash
|
||||
subminer --setup
|
||||
subminer app --setup
|
||||
SubMiner.AppImage --setup
|
||||
```
|
||||
|
||||
@@ -243,6 +261,7 @@ Top-level launcher flags like `--jellyfin-*` are intentionally rejected.
|
||||
- `--sub-file-paths=.;subs;subtitles`
|
||||
- `--sid=auto`
|
||||
- `--secondary-sid=auto`
|
||||
- `--sub-visibility=no` (the overlay renders subtitles instead of mpv)
|
||||
- `--secondary-sub-visibility=no`
|
||||
|
||||
You can append additional MPV arguments with launcher `-a/--args`, for example `--args "--ao=alsa --volume=80"`.
|
||||
@@ -288,8 +307,8 @@ Notes:
|
||||
- Press `Ctrl+Alt+C` during active YouTube playback to open the manual YouTube subtitle picker and retry track selection.
|
||||
- For YouTube URLs, `subminer` probes available YouTube subtitle tracks, reuses existing authoritative tracks when available, and downloads only missing sides.
|
||||
- Native mpv secondary subtitle rendering stays hidden so the overlay remains the visible secondary subtitle surface.
|
||||
- Primary subtitle target languages come from `youtube.primarySubLanguages` (defaults to `["ja","jpn"]`).
|
||||
- Secondary target languages come from `secondarySub.secondarySubLanguages` (empty by default; when empty, no language-based secondary track is auto-selected, though mpv's `--slang` list above still prefers English variants). When multiple matching secondary tracks exist, SubMiner prefers a non-Signs/Songs track.
|
||||
- YouTube auto-selection always targets a Japanese primary track and an English secondary track (manual uploads preferred over auto-generated captions). `youtube.primarySubLanguages` (defaults to `["ja","jpn"]`) defines which loaded track counts as a satisfactory primary for the missing-subtitle notification and for managed local/playlist selection.
|
||||
- When multiple matching secondary tracks exist, SubMiner prefers a non-Signs/Songs track.
|
||||
- Configure defaults in `$XDG_CONFIG_HOME/SubMiner/config.jsonc` (or `~/.config/SubMiner/config.jsonc`) under `youtube` and `secondarySub`.
|
||||
|
||||
For local video files, SubMiner uses the same config-driven language priorities to auto-select the primary and secondary subtitle tracks from internal and external subtitle sources.
|
||||
@@ -324,6 +343,8 @@ By default SubMiner uses the first connected controller after controller support
|
||||
| `Select` / `Minus` | Quit mpv |
|
||||
| `L2` / `R2` | Unbound (available for custom bindings) |
|
||||
|
||||
Note: the default quit binding uses gamepad button index 6. Pads that follow the W3C standard gamepad layout report L2 as index 6 (Select is index 8), so on those controllers the quit action may fire on L2 instead - use `Alt+C` learn mode to remap it for your pad.
|
||||
|
||||
### Analog Controls
|
||||
|
||||
| Input | Action |
|
||||
@@ -341,18 +362,18 @@ All button and axis mappings are configurable under the `controller` config bloc
|
||||
|
||||
See [Keyboard Shortcuts](/shortcuts) for the full reference, including mining shortcuts, overlay controls, and customization.
|
||||
|
||||
**Global shortcuts** (work system-wide):
|
||||
**App-wide shortcuts:**
|
||||
|
||||
| Keybind | Action |
|
||||
| ------------- | ---------------------- |
|
||||
| `Alt+Shift+O` | Toggle visible overlay |
|
||||
| `Alt+Shift+Y` | Open Yomitan settings |
|
||||
| Keybind | Action | Scope |
|
||||
| ------------- | ---------------------- | -------------------------------------------------------------------------------------------------- |
|
||||
| `Alt+Shift+O` | Toggle visible overlay | Works while the overlay or mpv has focus (configurable via `shortcuts.toggleVisibleOverlayGlobal`) |
|
||||
| `Alt+Shift+Y` | Open Yomitan settings | OS-global - registered with the system, works from any window |
|
||||
|
||||
`Alt+Shift+Y` is fixed and not configurable. All other shortcuts can be changed under `shortcuts` in your config.
|
||||
|
||||
Useful overlay-local default keybinding: `Ctrl+Alt+P` opens the playlist browser for the current video's parent directory and the live mpv queue so you can append, reorder, remove, or jump between episodes without leaving playback.
|
||||
|
||||
Press `V` to hide or restore the primary SubMiner subtitle bar. The bundled mpv plugin also binds bare `v` to the same action (injected at runtime).
|
||||
Press `V` to cycle the primary SubMiner subtitle bar through hidden → visible → hover modes. The bundled mpv plugin also binds bare `v` to the same action (injected at runtime).
|
||||
|
||||
`Ctrl/Cmd+/` opens the session help modal with the current overlay and mpv keybindings. The same help view is also available through the `y-h` chord in mpv.
|
||||
|
||||
@@ -362,5 +383,6 @@ Hovering over subtitle text pauses mpv by default; leaving resumes it. Yomitan p
|
||||
|
||||
- Drop video files onto the overlay to replace current playback.
|
||||
- Hold `Shift` while dropping to append to the playlist instead.
|
||||
- Drop subtitle files onto the overlay to load them as a new subtitle track.
|
||||
|
||||
Next: [Mining Workflow](/mining-workflow) - word lookup, card creation, and the full mining loop.
|
||||
|
||||
@@ -265,9 +265,15 @@ script-message subminer-options
|
||||
script-message subminer-restart
|
||||
script-message subminer-status
|
||||
script-message subminer-autoplay-ready
|
||||
script-message subminer-stats-toggle
|
||||
script-message subminer-visible-overlay-shown
|
||||
script-message subminer-visible-overlay-hidden
|
||||
script-message subminer-managed-subtitles-loading
|
||||
script-message subminer-overlay-loading-ready
|
||||
script-message subminer-reload-session-bindings
|
||||
```
|
||||
|
||||
The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) are handled by the SubMiner app over the mpv IPC socket while it is connected.
|
||||
The overlay/loading/session-binding messages are primarily sent by the SubMiner app to keep the plugin's state in sync. The AniSkip messages (`subminer-skip-intro`, `subminer-aniskip-refresh`) are handled by the SubMiner app over the mpv IPC socket while it is connected.
|
||||
|
||||
The start command also accepts inline overrides:
|
||||
|
||||
|
||||
@@ -96,7 +96,7 @@ For auto-generated tracks, SubMiner prefers `srv3` > `srv2` > `srv1` > `vtt` (Ti
|
||||
|
||||
By default, YouTube card audio and screenshots are extracted directly from mpv's active stream URLs. If generated card media fails with YouTube `403` errors, set `youtube.mediaCache.mode` to `"background"`. Background mode starts a separate `yt-dlp` media download after playback loads, including YouTube URLs opened directly in mpv and resolved stream URLs when mpv still exposes the original YouTube playlist entry. It creates text fields immediately, queues audio/image work for mined notes, and fills those fields once the local cache file is ready.
|
||||
|
||||
Background cache downloads use IPv4 and retry flags to reduce YouTube throttling failures. If the background download still fails, SubMiner shows a cache failure notification, shows queued-card failure notifications, and clears those pending updates so cards are not left waiting silently.
|
||||
Background cache downloads are capped at 720p by default (`youtube.mediaCache.maxHeight`; set `0` for unlimited) and use IPv4 and retry flags to reduce YouTube throttling failures. If the background download still fails, SubMiner shows a cache failure notification, shows queued-card failure notifications, and clears those pending updates so cards are not left waiting silently.
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
@@ -112,11 +112,13 @@ Background cache downloads use IPv4 and retry flags to reduce YouTube throttling
|
||||
|
||||
| Option | Type | Description |
|
||||
| --------------------- | ---------- | ------------------------------------------------------------------------------------- |
|
||||
| `primarySubLanguages` | `string[]` | Language priority for YouTube primary subtitle auto-loading (default `["ja", "jpn"]`) |
|
||||
| `primarySubLanguages` | `string[]` | Languages that count as a satisfactory primary subtitle (default `["ja", "jpn"]`). Used by the "primary subtitle missing" notification and by managed local/playlist subtitle selection. |
|
||||
|
||||
YouTube auto-selection itself always picks a Japanese track first (manual over auto), then falls back to any manual track — `primarySubLanguages` does not change which YouTube track is auto-picked.
|
||||
|
||||
### Secondary Subtitle Languages
|
||||
|
||||
Secondary track selection uses the shared `secondarySub` config:
|
||||
YouTube secondary selection is fixed: SubMiner always tries an English track (manual over auto) and loads it when found. The shared `secondarySub` config does not change YouTube track selection — `secondarySubLanguages` and `autoLoadSecondarySub` apply only to local/Jellyfin sidecar selection — but `defaultMode` still controls how the loaded secondary bar is displayed:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -130,11 +132,11 @@ Secondary track selection uses the shared `secondarySub` config:
|
||||
|
||||
| Option | Type | Description |
|
||||
| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `secondarySubLanguages` | `string[]` | Extra language codes (e.g. `["eng", "en"]`) used when auto-selecting a secondary track. Default is empty (`[]`). For YouTube, SubMiner always tries an English track first regardless of this list. |
|
||||
| `autoLoadSecondarySub` | `boolean` | Auto-detect and load a matching secondary track (default: `false`) |
|
||||
| `secondarySubLanguages` | `string[]` | Extra language codes (e.g. `["eng", "en"]`) used when auto-selecting a secondary track for local/Jellyfin sidecar files. Default is empty (`[]`). Not used for YouTube. |
|
||||
| `autoLoadSecondarySub` | `boolean` | Auto-detect and load a matching secondary sidecar track for local files (default: `false`). Not used for YouTube. |
|
||||
| `defaultMode` | `"hidden"` / `"visible"` / `"hover"` | Initial display mode for secondary subtitles (default: `"hover"`) |
|
||||
|
||||
Precedence: CLI flag > environment variable > `config.jsonc` > built-in default.
|
||||
These settings come from `config.jsonc` (or built-in defaults); there are no CLI flags or environment variables for subtitle language selection.
|
||||
|
||||
## Limitations and Troubleshooting
|
||||
|
||||
|
||||
@@ -36,6 +36,14 @@ function createContext(): LauncherCommandContext {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'info',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import type { Args } from '../types.js';
|
||||
import { createEmptyMergeSummary } from '../sync/sync-shared.js';
|
||||
import type { LauncherCommandContext } from './context.js';
|
||||
import { ensureTrackerQuiescent, runSyncCommand, type SyncCommandDeps } from './sync-command.js';
|
||||
|
||||
function makeContext(overrides: Partial<Args>): LauncherCommandContext {
|
||||
return {
|
||||
args: {
|
||||
sync: true,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'warn',
|
||||
...overrides,
|
||||
} as Args,
|
||||
scriptPath: '/tmp/subminer',
|
||||
scriptName: 'subminer',
|
||||
mpvSocketPath: '',
|
||||
pluginRuntimeConfig: {},
|
||||
appPath: null,
|
||||
launcherJellyfinConfig: {},
|
||||
processAdapter: process,
|
||||
} as unknown as LauncherCommandContext;
|
||||
}
|
||||
|
||||
function ok(stdout = ''): { status: number; stdout: string; stderr: string } {
|
||||
return { status: 0, stdout, stderr: '' };
|
||||
}
|
||||
|
||||
test('ensureTrackerQuiescent ignores stale sockets but rejects live sockets', async () => {
|
||||
const context = makeContext({ syncDbPath: '/tmp/local.sqlite' });
|
||||
context.mpvSocketPath = '/tmp/subminer-socket';
|
||||
let socketConnectable = false;
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
realpathSync: (() => '/tracker.sqlite') as unknown as typeof fs.realpathSync,
|
||||
findLiveStatsDaemonPid: () => null,
|
||||
canConnectUnixSocket: async () => socketConnectable,
|
||||
fail: (message: string): never => {
|
||||
throw new Error(message);
|
||||
},
|
||||
};
|
||||
|
||||
await ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps);
|
||||
|
||||
socketConnectable = true;
|
||||
await assert.rejects(
|
||||
async () => ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps),
|
||||
/mpv\/SubMiner session appears to be running/,
|
||||
);
|
||||
});
|
||||
|
||||
test('runSyncCommand dispatches snapshot, merge, host, and missing-target modes', async () => {
|
||||
const calls: string[] = [];
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${dbPath}->${outPath}`);
|
||||
},
|
||||
mergeSnapshotIntoDb: (dbPath: string, snapshotPath: string) => {
|
||||
calls.push(`merge:${dbPath}<-${snapshotPath}`);
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: (host: string) => {
|
||||
calls.push(`host:${host}`);
|
||||
},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
return command.startsWith('mktemp ') ? ok('/tmp/subminer-sync.remote\n') : ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
},
|
||||
fail: (message: string): never => {
|
||||
throw new Error(message);
|
||||
},
|
||||
};
|
||||
|
||||
assert.equal(
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncSnapshotPath: '/tmp/out.sqlite' }),
|
||||
deps,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.ok(calls.includes('snapshot:/tmp/local.sqlite->/tmp/out.sqlite'));
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncMergePath: '/tmp/in.sqlite' }),
|
||||
deps,
|
||||
);
|
||||
assert.ok(calls.includes('quiescent'));
|
||||
assert.ok(calls.includes('merge:/tmp/local.sqlite<-/tmp/in.sqlite'));
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }),
|
||||
deps,
|
||||
);
|
||||
assert.ok(calls.includes('host:media-box'));
|
||||
|
||||
await assert.rejects(
|
||||
() => runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite' }), deps),
|
||||
/sync requires a host, --snapshot <file>, or --merge <file>/,
|
||||
);
|
||||
});
|
||||
|
||||
test('runHostSync keeps tracker quiescent through local and remote merge and cleans up after failure', async () => {
|
||||
const calls: string[] = [];
|
||||
let localTmpDir = '';
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${outPath}`);
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
mergeSnapshotIntoDb: () => {
|
||||
calls.push('local-merge');
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
mkdtempSync: ((prefix: string) => {
|
||||
localTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), path.basename(prefix)));
|
||||
return localTmpDir;
|
||||
}) as typeof fs.mkdtempSync,
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
if (command.includes(' sync --snapshot ')) return ok();
|
||||
if (command.includes(' sync --merge ')) {
|
||||
return { status: 9, stdout: 'remote output', stderr: 'remote merge exploded' };
|
||||
}
|
||||
return ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
|
||||
},
|
||||
};
|
||||
|
||||
await assert.rejects(
|
||||
() =>
|
||||
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
|
||||
/Remote merge failed on media-box[\s\S]*remote merge exploded/,
|
||||
);
|
||||
assert.equal(calls.filter((call) => call === 'quiescent').length, 3);
|
||||
assert.ok(calls.indexOf('quiescent') < calls.findIndex((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(calls.includes('local-merge'));
|
||||
assert.ok(calls.some((call) => call.startsWith('ssh:rm -rf ')));
|
||||
assert.equal(fs.existsSync(localTmpDir), false);
|
||||
});
|
||||
|
||||
test('runHostSync includes remote snapshot stderr in failures', async () => {
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
ensureTrackerQuiescent: async () => {},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
if (command.includes(' sync --snapshot ')) {
|
||||
return { status: 5, stdout: '', stderr: 'snapshot permission denied' };
|
||||
}
|
||||
return ok();
|
||||
},
|
||||
runScp: () => {},
|
||||
};
|
||||
|
||||
await assert.rejects(
|
||||
() =>
|
||||
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
|
||||
/Remote snapshot failed on media-box[\s\S]*snapshot permission denied/,
|
||||
);
|
||||
});
|
||||
|
||||
function makeDirectionDeps(calls: string[]): Partial<SyncCommandDeps> {
|
||||
return {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${outPath}`);
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
mergeSnapshotIntoDb: () => {
|
||||
calls.push('local-merge');
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
return ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test('runHostSync push only snapshots locally and merges remotely', async () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({
|
||||
syncDbPath: '/tmp/local.sqlite',
|
||||
syncHost: 'media-box',
|
||||
syncDirection: 'push',
|
||||
}),
|
||||
makeDirectionDeps(calls),
|
||||
);
|
||||
|
||||
assert.ok(calls.some((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(calls.some((call) => call.includes(' sync --merge ')));
|
||||
assert.ok(calls.some((call) => call.startsWith('scp:') && call.includes('->media-box:')));
|
||||
assert.ok(!calls.some((call) => call.includes(' sync --snapshot ')));
|
||||
assert.ok(!calls.includes('local-merge'));
|
||||
});
|
||||
|
||||
test('runHostSync pull only snapshots remotely and merges locally', async () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({
|
||||
syncDbPath: '/tmp/local.sqlite',
|
||||
syncHost: 'media-box',
|
||||
syncDirection: 'pull',
|
||||
}),
|
||||
makeDirectionDeps(calls),
|
||||
);
|
||||
|
||||
assert.ok(calls.some((call) => call.includes(' sync --snapshot ')));
|
||||
assert.ok(calls.some((call) => call.startsWith('scp:media-box:')));
|
||||
assert.ok(calls.includes('local-merge'));
|
||||
assert.ok(!calls.some((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(!calls.some((call) => call.includes(' sync --merge ')));
|
||||
});
|
||||
@@ -0,0 +1,253 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fail, log } from '../log.js';
|
||||
import { resolveImmersionDbPath } from '../history-db.js';
|
||||
import {
|
||||
createDbSnapshot,
|
||||
findLiveStatsDaemonPid,
|
||||
formatMergeSummary,
|
||||
mergeSnapshotIntoDb,
|
||||
} from '../sync/sync-db.js';
|
||||
import {
|
||||
assertSafeSshHost,
|
||||
resolveRemoteSubminerCommand,
|
||||
runScp,
|
||||
runSsh,
|
||||
shellQuote,
|
||||
} from '../sync/ssh.js';
|
||||
import { resolvePathMaybe } from '../util.js';
|
||||
import { canConnectUnixSocket } from '../mpv.js';
|
||||
import type { LauncherCommandContext } from './context.js';
|
||||
import type { RemoteRunResult } from '../sync/ssh.js';
|
||||
|
||||
export interface SyncCommandDeps {
|
||||
createDbSnapshot: typeof createDbSnapshot;
|
||||
mergeSnapshotIntoDb: typeof mergeSnapshotIntoDb;
|
||||
formatMergeSummary: typeof formatMergeSummary;
|
||||
findLiveStatsDaemonPid: typeof findLiveStatsDaemonPid;
|
||||
assertSafeSshHost: typeof assertSafeSshHost;
|
||||
resolveRemoteSubminerCommand: typeof resolveRemoteSubminerCommand;
|
||||
runScp: typeof runScp;
|
||||
runSsh: typeof runSsh;
|
||||
fail: typeof fail;
|
||||
log: typeof log;
|
||||
canConnectUnixSocket: typeof canConnectUnixSocket;
|
||||
realpathSync: typeof fs.realpathSync;
|
||||
mkdtempSync: typeof fs.mkdtempSync;
|
||||
rmSync: typeof fs.rmSync;
|
||||
consoleLog: typeof console.log;
|
||||
writeStdout: typeof process.stdout.write;
|
||||
ensureTrackerQuiescent: (context: LauncherCommandContext, dbPath: string) => Promise<void>;
|
||||
}
|
||||
|
||||
function resolveDbPath(context: LauncherCommandContext): string {
|
||||
const override = context.args.syncDbPath.trim();
|
||||
return override ? resolvePathMaybe(override) : resolveImmersionDbPath();
|
||||
}
|
||||
|
||||
function isTrackerDb(dbPath: string, deps: SyncCommandDeps): boolean {
|
||||
const trackerDbPath = resolveImmersionDbPath();
|
||||
try {
|
||||
return deps.realpathSync(dbPath) === deps.realpathSync(trackerDbPath);
|
||||
} catch {
|
||||
return dbPath === trackerDbPath;
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensureTrackerQuiescent(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
if (context.args.syncForce) return;
|
||||
// A running SubMiner only holds the tracker's own database; --db pointed
|
||||
// elsewhere needs no guard.
|
||||
if (!isTrackerDb(dbPath, deps)) return;
|
||||
const daemonPid = deps.findLiveStatsDaemonPid(dbPath);
|
||||
if (daemonPid !== null) {
|
||||
deps.fail(
|
||||
`The SubMiner stats server is running (pid ${daemonPid}). Stop it with "subminer stats -s" (or close SubMiner) before syncing, or pass --force.`,
|
||||
);
|
||||
}
|
||||
if (context.mpvSocketPath && (await deps.canConnectUnixSocket(context.mpvSocketPath))) {
|
||||
deps.fail(
|
||||
`An mpv/SubMiner session appears to be running (socket ${context.mpvSocketPath}). Close it before syncing, or pass --force.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const defaultSyncCommandDeps: SyncCommandDeps = {
|
||||
createDbSnapshot,
|
||||
mergeSnapshotIntoDb,
|
||||
formatMergeSummary,
|
||||
findLiveStatsDaemonPid,
|
||||
assertSafeSshHost,
|
||||
resolveRemoteSubminerCommand,
|
||||
runScp,
|
||||
runSsh,
|
||||
fail,
|
||||
log,
|
||||
canConnectUnixSocket,
|
||||
realpathSync: fs.realpathSync,
|
||||
mkdtempSync: fs.mkdtempSync,
|
||||
rmSync: fs.rmSync,
|
||||
consoleLog: console.log,
|
||||
writeStdout: process.stdout.write.bind(process.stdout),
|
||||
ensureTrackerQuiescent: async (context, dbPath) => ensureTrackerQuiescent(context, dbPath),
|
||||
};
|
||||
|
||||
function resolveSyncCommandDeps(inputDeps: Partial<SyncCommandDeps> = {}): SyncCommandDeps {
|
||||
return { ...defaultSyncCommandDeps, ...inputDeps };
|
||||
}
|
||||
|
||||
export function runSnapshotMode(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): void {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const outPath = resolvePathMaybe(context.args.syncSnapshotPath);
|
||||
deps.createDbSnapshot(dbPath, outPath);
|
||||
deps.consoleLog(outPath);
|
||||
}
|
||||
|
||||
export async function runMergeMode(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const snapshotPath = resolvePathMaybe(context.args.syncMergePath);
|
||||
const summary = deps.mergeSnapshotIntoDb(dbPath, snapshotPath);
|
||||
deps.consoleLog(deps.formatMergeSummary(summary));
|
||||
}
|
||||
|
||||
function cleanupRemote(host: string, remoteTmpDir: string, deps: SyncCommandDeps): void {
|
||||
if (!remoteTmpDir.startsWith('/tmp/')) return;
|
||||
deps.runSsh(host, `rm -rf ${shellQuote(remoteTmpDir)}`);
|
||||
}
|
||||
|
||||
function formatRemoteRunError(message: string, run: RemoteRunResult): string {
|
||||
const stderr = run.stderr.trim();
|
||||
return stderr ? `${message}\n${stderr}` : message;
|
||||
}
|
||||
|
||||
export async function runHostSync(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const { args } = context;
|
||||
const host = args.syncHost;
|
||||
const direction = args.syncDirection ?? 'both';
|
||||
const shouldPull = direction !== 'push';
|
||||
const shouldPush = direction !== 'pull';
|
||||
deps.assertSafeSshHost(host);
|
||||
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
|
||||
const remoteCmd = deps.resolveRemoteSubminerCommand(host, args.syncRemoteCmd || null);
|
||||
deps.log('debug', args.logLevel, `Remote subminer command: ${remoteCmd}`);
|
||||
|
||||
const localTmpDir = deps.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-'));
|
||||
let remoteTmpDir = '';
|
||||
try {
|
||||
// Signal failures by throwing (not fail(), which exits synchronously and
|
||||
// would skip the finally cleanup, leaking temp dirs holding snapshot data).
|
||||
// main().catch() reports the message the same way fail() would.
|
||||
const mktemp = deps.runSsh(host, 'mktemp -d /tmp/subminer-sync.XXXXXX');
|
||||
remoteTmpDir = mktemp.stdout.trim();
|
||||
if (mktemp.status !== 0 || !remoteTmpDir.startsWith('/tmp/')) {
|
||||
throw new Error(`Could not create a temporary directory on ${host}.`);
|
||||
}
|
||||
|
||||
const forceFlag = args.syncForce ? ' --force' : '';
|
||||
|
||||
const localSnapshot = path.join(localTmpDir, 'local.sqlite');
|
||||
if (shouldPush) {
|
||||
deps.consoleLog(`Snapshotting local database (${dbPath})...`);
|
||||
deps.createDbSnapshot(dbPath, localSnapshot);
|
||||
}
|
||||
|
||||
const remoteSnapshot = `${remoteTmpDir}/snapshot.sqlite`;
|
||||
if (shouldPull) {
|
||||
deps.consoleLog(`Snapshotting ${host}...`);
|
||||
const snapshotRun = deps.runSsh(
|
||||
host,
|
||||
`${remoteCmd} sync --snapshot ${shellQuote(remoteSnapshot)}${forceFlag}`,
|
||||
);
|
||||
if (snapshotRun.status !== 0) {
|
||||
throw new Error(formatRemoteRunError(`Remote snapshot failed on ${host}.`, snapshotRun));
|
||||
}
|
||||
}
|
||||
|
||||
const pulledSnapshot = path.join(localTmpDir, 'remote.sqlite');
|
||||
if (shouldPull) deps.runScp(`${host}:${remoteSnapshot}`, pulledSnapshot);
|
||||
const incomingSnapshot = `${remoteTmpDir}/incoming.sqlite`;
|
||||
if (shouldPush) deps.runScp(localSnapshot, `${host}:${incomingSnapshot}`);
|
||||
|
||||
if (shouldPull) {
|
||||
deps.consoleLog(`\nMerging ${host} -> local:`);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const summary = deps.mergeSnapshotIntoDb(dbPath, pulledSnapshot);
|
||||
deps.consoleLog(deps.formatMergeSummary(summary));
|
||||
}
|
||||
|
||||
if (shouldPush) {
|
||||
deps.consoleLog(`\nMerging local -> ${host}:`);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const mergeRun = deps.runSsh(
|
||||
host,
|
||||
`${remoteCmd} sync --merge ${shellQuote(incomingSnapshot)}${forceFlag}`,
|
||||
);
|
||||
deps.writeStdout(mergeRun.stdout);
|
||||
if (mergeRun.status !== 0) {
|
||||
const retryCommand =
|
||||
direction === 'push' ? `subminer sync ${host} --push` : `subminer sync ${host}`;
|
||||
const localUpdate = shouldPull ? ' The local database was updated;' : '';
|
||||
throw new Error(
|
||||
formatRemoteRunError(
|
||||
`Remote merge failed on ${host}.${localUpdate} re-run "${retryCommand}" once the remote issue is fixed.`,
|
||||
mergeRun,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
deps.consoleLog('\nSync complete.');
|
||||
} finally {
|
||||
deps.rmSync(localTmpDir, { recursive: true, force: true });
|
||||
if (remoteTmpDir) {
|
||||
try {
|
||||
cleanupRemote(host, remoteTmpDir, deps);
|
||||
} catch {
|
||||
// best effort
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function runSyncCommand(
|
||||
context: LauncherCommandContext,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<boolean> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const { args } = context;
|
||||
if (!args.sync) return false;
|
||||
|
||||
const dbPath = resolveDbPath(context);
|
||||
if (args.syncSnapshotPath) {
|
||||
runSnapshotMode(context, dbPath, deps);
|
||||
} else if (args.syncMergePath) {
|
||||
await runMergeMode(context, dbPath, deps);
|
||||
} else if (args.syncHost) {
|
||||
await runHostSync(context, dbPath, deps);
|
||||
} else {
|
||||
deps.fail('sync requires a host, --snapshot <file>, or --merge <file>.');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -135,6 +135,15 @@ test('applyInvocationsToArgs maps config and jellyfin invocation state', () => {
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -181,6 +190,15 @@ test('applyInvocationsToArgs maps settings invocation to settings window', () =>
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -220,6 +238,15 @@ test('applyInvocationsToArgs fails when config invocation has no action', () =>
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -257,6 +284,15 @@ test('applyInvocationsToArgs maps texthooker browser-open request', () => {
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
|
||||
@@ -199,6 +199,14 @@ export function createDefaultArgs(
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: loggingConfig.level ?? 'warn',
|
||||
logRotation: loggingConfig.rotation ?? 7,
|
||||
passwordStore: '',
|
||||
@@ -264,6 +272,17 @@ export function applyInvocationsToArgs(parsed: Args, invocations: CliInvocations
|
||||
) {
|
||||
fail('Dictionary target path is required.');
|
||||
}
|
||||
if (invocations.syncTriggered) {
|
||||
parsed.sync = true;
|
||||
parsed.syncHost = invocations.syncHost ?? '';
|
||||
parsed.syncSnapshotPath = invocations.syncSnapshotPath ?? '';
|
||||
parsed.syncMergePath = invocations.syncMergePath ?? '';
|
||||
parsed.syncDirection = invocations.syncDirection;
|
||||
parsed.syncRemoteCmd = invocations.syncRemoteCmd ?? '';
|
||||
parsed.syncDbPath = invocations.syncDbPath ?? '';
|
||||
parsed.syncForce = invocations.syncForce;
|
||||
if (invocations.syncLogLevel) parsed.logLevel = parseLogLevel(invocations.syncLogLevel);
|
||||
}
|
||||
if (invocations.doctorTriggered) parsed.doctor = true;
|
||||
if (invocations.doctorRefreshKnownWords) parsed.doctorRefreshKnownWords = true;
|
||||
if (invocations.logsTriggered && !invocations.logsExport) {
|
||||
|
||||
@@ -42,3 +42,22 @@ test('parseCliPrograms captures texthooker browser-open flag', () => {
|
||||
assert.equal(result.invocations.texthookerTriggered, true);
|
||||
assert.equal(result.invocations.texthookerOpenBrowser, true);
|
||||
});
|
||||
|
||||
test('parseCliPrograms captures one-way sync directions', () => {
|
||||
const push = parseCliPrograms(['sync', 'media-box', '--push'], 'subminer');
|
||||
const pull = parseCliPrograms(['sync', 'media-box', '--pull'], 'subminer');
|
||||
|
||||
assert.equal(push.invocations.syncDirection, 'push');
|
||||
assert.equal(pull.invocations.syncDirection, 'pull');
|
||||
});
|
||||
|
||||
test('parseCliPrograms rejects conflicting or hostless one-way sync directions', () => {
|
||||
assert.throws(
|
||||
() => parseCliPrograms(['sync', 'media-box', '--push', '--pull'], 'subminer'),
|
||||
/--push and --pull cannot be combined/,
|
||||
);
|
||||
assert.throws(
|
||||
() => parseCliPrograms(['sync', '--snapshot', '/tmp/stats.sqlite', '--push'], 'subminer'),
|
||||
/--push and --pull require a host/,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -38,6 +38,15 @@ export interface CliInvocations {
|
||||
statsCleanupVocab: boolean;
|
||||
statsCleanupLifetime: boolean;
|
||||
statsLogLevel: string | null;
|
||||
syncTriggered: boolean;
|
||||
syncHost: string | null;
|
||||
syncSnapshotPath: string | null;
|
||||
syncMergePath: string | null;
|
||||
syncDirection: 'both' | 'push' | 'pull';
|
||||
syncRemoteCmd: string | null;
|
||||
syncDbPath: string | null;
|
||||
syncForce: boolean;
|
||||
syncLogLevel: string | null;
|
||||
doctorTriggered: boolean;
|
||||
doctorLogLevel: string | null;
|
||||
doctorRefreshKnownWords: boolean;
|
||||
@@ -98,6 +107,7 @@ function getTopLevelCommand(argv: string[]): { name: string; index: number } | n
|
||||
'dictionary',
|
||||
'dict',
|
||||
'stats',
|
||||
'sync',
|
||||
'texthooker',
|
||||
'app',
|
||||
'bin',
|
||||
@@ -160,6 +170,15 @@ export function parseCliPrograms(
|
||||
let statsCleanupVocab = false;
|
||||
let statsCleanupLifetime = false;
|
||||
let statsLogLevel: string | null = null;
|
||||
let syncTriggered = false;
|
||||
let syncHost: string | null = null;
|
||||
let syncSnapshotPath: string | null = null;
|
||||
let syncMergePath: string | null = null;
|
||||
let syncDirection: 'both' | 'push' | 'pull' = 'both';
|
||||
let syncRemoteCmd: string | null = null;
|
||||
let syncDbPath: string | null = null;
|
||||
let syncForce = false;
|
||||
let syncLogLevel: string | null = null;
|
||||
let doctorLogLevel: string | null = null;
|
||||
let doctorRefreshKnownWords = false;
|
||||
let logsTriggered = false;
|
||||
@@ -289,6 +308,49 @@ export function parseCliPrograms(
|
||||
statsLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
|
||||
});
|
||||
|
||||
commandProgram
|
||||
.command('sync')
|
||||
.description('Sync stats and watch history with another machine over SSH')
|
||||
.argument('[host]', 'SSH destination (user@host or an ssh config alias)')
|
||||
.option('--snapshot <file>', 'Write a consistent snapshot of the local stats database')
|
||||
.option('--merge <file>', 'Merge a snapshot database file into the local stats database')
|
||||
.option('--push', 'Only merge local stats into the SSH host')
|
||||
.option('--pull', 'Only merge stats from the SSH host into the local database')
|
||||
.option('--db <file>', 'Override the local stats database path')
|
||||
.option('--remote-cmd <cmd>', 'subminer command to run on the remote host')
|
||||
.option('-f, --force', 'Skip the running-app safety check')
|
||||
.option('--log-level <level>', 'Log level')
|
||||
.action((rawHost: string | undefined, options: Record<string, unknown>) => {
|
||||
const host = typeof rawHost === 'string' ? rawHost.trim() : '';
|
||||
const snapshot = typeof options.snapshot === 'string' ? options.snapshot.trim() : '';
|
||||
const merge = typeof options.merge === 'string' ? options.merge.trim() : '';
|
||||
const push = options.push === true;
|
||||
const pull = options.pull === true;
|
||||
if (push && pull) {
|
||||
throw new Error('Sync --push and --pull cannot be combined.');
|
||||
}
|
||||
if ((push || pull) && !host) {
|
||||
throw new Error('Sync --push and --pull require a host.');
|
||||
}
|
||||
const modes = [Boolean(host), Boolean(snapshot), Boolean(merge)].filter(Boolean).length;
|
||||
if (modes === 0) {
|
||||
throw new Error('Sync requires a host, --snapshot <file>, or --merge <file>.');
|
||||
}
|
||||
if (modes > 1) {
|
||||
throw new Error('Sync host, --snapshot, and --merge cannot be combined.');
|
||||
}
|
||||
syncTriggered = true;
|
||||
syncHost = host || null;
|
||||
syncSnapshotPath = snapshot || null;
|
||||
syncMergePath = merge || null;
|
||||
syncDirection = push ? 'push' : pull ? 'pull' : 'both';
|
||||
syncRemoteCmd =
|
||||
typeof options.remoteCmd === 'string' ? options.remoteCmd.trim() || null : null;
|
||||
syncDbPath = typeof options.db === 'string' ? options.db.trim() || null : null;
|
||||
syncForce = options.force === true;
|
||||
syncLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
|
||||
});
|
||||
|
||||
commandProgram
|
||||
.command('doctor')
|
||||
.description('Run dependency and environment checks')
|
||||
@@ -400,6 +462,15 @@ export function parseCliPrograms(
|
||||
statsCleanupVocab,
|
||||
statsCleanupLifetime,
|
||||
statsLogLevel,
|
||||
syncTriggered,
|
||||
syncHost,
|
||||
syncSnapshotPath,
|
||||
syncMergePath,
|
||||
syncDirection,
|
||||
syncRemoteCmd,
|
||||
syncDbPath,
|
||||
syncForce,
|
||||
syncLogLevel,
|
||||
doctorTriggered,
|
||||
doctorLogLevel,
|
||||
doctorRefreshKnownWords,
|
||||
|
||||
@@ -30,6 +30,14 @@ function createArgs(): Args {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'info',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
@@ -22,6 +22,7 @@ import { runLogsCommand } from './commands/logs-command.js';
|
||||
import { runStatsCommand } from './commands/stats-command.js';
|
||||
import { runJellyfinCommand } from './commands/jellyfin-command.js';
|
||||
import { runHistoryCommand } from './commands/history-command.js';
|
||||
import { runSyncCommand } from './commands/sync-command.js';
|
||||
import { runPlaybackCommand } from './commands/playback-command.js';
|
||||
import { runUpdateCommand } from './commands/update-command.js';
|
||||
|
||||
@@ -107,6 +108,10 @@ async function main(): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
if (await runSyncCommand(context)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const resolvedAppPath = ensureAppPath(context);
|
||||
state.appPath = resolvedAppPath;
|
||||
log('debug', args.logLevel, `Using SubMiner app binary: ${resolvedAppPath}`);
|
||||
|
||||
@@ -571,6 +571,14 @@ function makeArgs(overrides: Partial<Args> = {}): Args {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'error',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
+1
-1
@@ -1683,7 +1683,7 @@ async function sleepMs(ms: number): Promise<void> {
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
|
||||
export async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
|
||||
return await new Promise<boolean>((resolve) => {
|
||||
const socket = net.createConnection(socketPath);
|
||||
let settled = false;
|
||||
|
||||
@@ -0,0 +1,423 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { insertRow, tableExists, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
const ANIME_COPY_COLUMNS = [
|
||||
'normalized_title_key',
|
||||
'canonical_title',
|
||||
'anilist_id',
|
||||
'title_romaji',
|
||||
'title_english',
|
||||
'title_native',
|
||||
'episodes_total',
|
||||
'description',
|
||||
'metadata_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const VIDEO_COPY_COLUMNS = [
|
||||
'video_key',
|
||||
'canonical_title',
|
||||
'source_type',
|
||||
'source_path',
|
||||
'source_url',
|
||||
'parsed_basename',
|
||||
'parsed_title',
|
||||
'parsed_season',
|
||||
'parsed_episode',
|
||||
'parser_source',
|
||||
'parser_confidence',
|
||||
'parse_metadata_json',
|
||||
'watched',
|
||||
'duration_ms',
|
||||
'file_size_bytes',
|
||||
'codec_id',
|
||||
'container_id',
|
||||
'width_px',
|
||||
'height_px',
|
||||
'fps_x100',
|
||||
'bitrate_kbps',
|
||||
'audio_codec_id',
|
||||
'hash_sha256',
|
||||
'screenshot_path',
|
||||
'metadata_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const MEDIA_ART_COPY_COLUMNS = [
|
||||
'anilist_id',
|
||||
'cover_url',
|
||||
'cover_blob',
|
||||
'cover_blob_hash',
|
||||
'title_romaji',
|
||||
'title_english',
|
||||
'episodes_total',
|
||||
'fetched_at_ms',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const YOUTUBE_COPY_COLUMNS = [
|
||||
'youtube_video_id',
|
||||
'video_url',
|
||||
'video_title',
|
||||
'video_thumbnail_url',
|
||||
'channel_id',
|
||||
'channel_name',
|
||||
'channel_url',
|
||||
'channel_thumbnail_url',
|
||||
'uploader_id',
|
||||
'uploader_url',
|
||||
'description',
|
||||
'metadata_json',
|
||||
'fetched_at_ms',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const WORD_COPY_COLUMNS = [
|
||||
'headword',
|
||||
'word',
|
||||
'reading',
|
||||
'part_of_speech',
|
||||
'pos1',
|
||||
'pos2',
|
||||
'pos3',
|
||||
'first_seen',
|
||||
'last_seen',
|
||||
'frequency',
|
||||
'frequency_rank',
|
||||
] as const;
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
function selectAll(db: Database, sql: string, params: unknown[] = []): SqlRow[] {
|
||||
return db.query<SqlRow>(sql).all(...params);
|
||||
}
|
||||
|
||||
export function mergeAnime(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
summary: SyncMergeSummary,
|
||||
): Map<number, number> {
|
||||
const map = new Map<number, number>();
|
||||
const byAnilist = local.prepare<SqlRow>('SELECT anime_id FROM imm_anime WHERE anilist_id = ?');
|
||||
const byTitleKey = local.prepare<SqlRow>(
|
||||
'SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?',
|
||||
);
|
||||
const fillMissing = local.prepare(
|
||||
`UPDATE imm_anime
|
||||
SET
|
||||
title_romaji = COALESCE(title_romaji, ?),
|
||||
title_english = COALESCE(title_english, ?),
|
||||
title_native = COALESCE(title_native, ?),
|
||||
episodes_total = COALESCE(episodes_total, ?),
|
||||
description = COALESCE(description, ?)
|
||||
WHERE anime_id = ?`,
|
||||
);
|
||||
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
`SELECT anime_id, ${ANIME_COPY_COLUMNS.join(', ')} FROM imm_anime`,
|
||||
)) {
|
||||
const remoteId = Number(row.anime_id);
|
||||
const existing =
|
||||
(row.anilist_id !== null ? byAnilist.get(row.anilist_id) : undefined) ??
|
||||
byTitleKey.get(row.normalized_title_key);
|
||||
if (existing) {
|
||||
const localId = Number(existing.anime_id);
|
||||
map.set(remoteId, localId);
|
||||
fillMissing.run(
|
||||
row.title_romaji,
|
||||
row.title_english,
|
||||
row.title_native,
|
||||
row.episodes_total,
|
||||
row.description,
|
||||
localId,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
// No local row matched by anilist_id (checked first in `existing` above)
|
||||
// or title key, so the remote anilist_id — if any — is free to insert as-is.
|
||||
const values = ANIME_COPY_COLUMNS.map((column) => row[column]);
|
||||
map.set(remoteId, insertRow(local, 'imm_anime', ANIME_COPY_COLUMNS, values));
|
||||
summary.animeAdded += 1;
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
export interface VideoMergeResult {
|
||||
videoIdMap: Map<number, number>;
|
||||
addedVideoIds: Set<number>;
|
||||
}
|
||||
|
||||
export function mergeVideos(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
animeIdMap: Map<number, number>,
|
||||
summary: SyncMergeSummary,
|
||||
): VideoMergeResult {
|
||||
const videoIdMap = new Map<number, number>();
|
||||
const addedVideoIds = new Set<number>();
|
||||
const byKey = local.prepare<SqlRow>(
|
||||
'SELECT video_id, watched FROM imm_videos WHERE video_key = ?',
|
||||
);
|
||||
const setWatched = local.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?');
|
||||
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
`SELECT video_id, anime_id, ${VIDEO_COPY_COLUMNS.join(', ')} FROM imm_videos`,
|
||||
)) {
|
||||
const remoteId = Number(row.video_id);
|
||||
const mappedAnimeId =
|
||||
row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
|
||||
const existing = byKey.get(row.video_key);
|
||||
if (existing) {
|
||||
const localId = Number(existing.video_id);
|
||||
videoIdMap.set(remoteId, localId);
|
||||
if (Number(row.watched) > 0 && Number(existing.watched) <= 0) {
|
||||
setWatched.run(localId);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const columns = ['anime_id', ...VIDEO_COPY_COLUMNS];
|
||||
const values = [mappedAnimeId, ...VIDEO_COPY_COLUMNS.map((column) => row[column])];
|
||||
const localId = insertRow(local, 'imm_videos', columns, values);
|
||||
videoIdMap.set(remoteId, localId);
|
||||
addedVideoIds.add(remoteId);
|
||||
summary.videosAdded += 1;
|
||||
}
|
||||
return { videoIdMap, addedVideoIds };
|
||||
}
|
||||
|
||||
export function mergeMediaMetadata(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
addedVideoIds: Set<number>,
|
||||
): void {
|
||||
if (videoIdMap.size === 0) return;
|
||||
const metadataVideoIds = new Set<number>([...addedVideoIds, ...videoIdMap.keys()]);
|
||||
|
||||
const hasBlobStore =
|
||||
tableExists(local, 'imm_cover_art_blobs') && tableExists(remote, 'imm_cover_art_blobs');
|
||||
const copyBlob = hasBlobStore
|
||||
? local.prepare(
|
||||
`INSERT INTO imm_cover_art_blobs (blob_hash, cover_blob, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(blob_hash) DO NOTHING`,
|
||||
)
|
||||
: null;
|
||||
const readBlob = hasBlobStore
|
||||
? remote.prepare<SqlRow>('SELECT * FROM imm_cover_art_blobs WHERE blob_hash = ?')
|
||||
: null;
|
||||
|
||||
if (tableExists(remote, 'imm_media_art') && tableExists(local, 'imm_media_art')) {
|
||||
const localArtExists = local.prepare<SqlRow>(
|
||||
'SELECT 1 FROM imm_media_art WHERE video_id = ? LIMIT 1',
|
||||
);
|
||||
for (const remoteVideoId of metadataVideoIds) {
|
||||
const localVideoId = videoIdMap.get(remoteVideoId)!;
|
||||
if (localArtExists.get(localVideoId)) continue;
|
||||
const row = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${MEDIA_ART_COPY_COLUMNS.join(', ')} FROM imm_media_art WHERE video_id = ?`,
|
||||
)
|
||||
.get(remoteVideoId);
|
||||
if (!row) continue;
|
||||
if (row.cover_blob_hash && copyBlob && readBlob) {
|
||||
const blob = readBlob.get(row.cover_blob_hash);
|
||||
if (blob) {
|
||||
copyBlob.run(blob.blob_hash, blob.cover_blob, blob.CREATED_DATE, blob.LAST_UPDATE_DATE);
|
||||
}
|
||||
}
|
||||
insertRow(
|
||||
local,
|
||||
'imm_media_art',
|
||||
['video_id', ...MEDIA_ART_COPY_COLUMNS],
|
||||
[localVideoId, ...MEDIA_ART_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (tableExists(remote, 'imm_youtube_videos') && tableExists(local, 'imm_youtube_videos')) {
|
||||
const localYoutubeExists = local.prepare<SqlRow>(
|
||||
'SELECT 1 FROM imm_youtube_videos WHERE video_id = ? LIMIT 1',
|
||||
);
|
||||
for (const remoteVideoId of metadataVideoIds) {
|
||||
const localVideoId = videoIdMap.get(remoteVideoId)!;
|
||||
if (localYoutubeExists.get(localVideoId)) continue;
|
||||
const row = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${YOUTUBE_COPY_COLUMNS.join(', ')} FROM imm_youtube_videos WHERE video_id = ?`,
|
||||
)
|
||||
.get(remoteVideoId);
|
||||
if (!row) continue;
|
||||
insertRow(
|
||||
local,
|
||||
'imm_youtube_videos',
|
||||
['video_id', ...YOUTUBE_COPY_COLUMNS],
|
||||
[localVideoId, ...YOUTUBE_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function mergeExcludedWords(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (
|
||||
!tableExists(remote, 'imm_stats_excluded_words') ||
|
||||
!tableExists(local, 'imm_stats_excluded_words')
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const insert = local.prepare(
|
||||
`INSERT INTO imm_stats_excluded_words (headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(headword, word, reading) DO NOTHING`,
|
||||
);
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
'SELECT headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE FROM imm_stats_excluded_words',
|
||||
)) {
|
||||
const result = insert.run(
|
||||
row.headword,
|
||||
row.word,
|
||||
row.reading,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.excludedWordsAdded += result.changes;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Lazily maps remote imm_words / imm_kanji ids onto local rows by natural key
|
||||
* ((headword, word, reading) / kanji). New rows are copied with the remote's
|
||||
* accumulated frequency; rows that already exist locally get their frequency
|
||||
* incremented later with only the occurrence counts this merge adds (the
|
||||
* remote total would double-count lines merged in earlier syncs).
|
||||
*/
|
||||
export class LexiconResolver {
|
||||
private readonly wordMap = new Map<number, { localId: number; isNew: boolean }>();
|
||||
private readonly kanjiMap = new Map<number, { localId: number; isNew: boolean }>();
|
||||
readonly wordFrequencyDeltas = new Map<number, number>();
|
||||
readonly kanjiFrequencyDeltas = new Map<number, number>();
|
||||
|
||||
constructor(
|
||||
private readonly local: Database,
|
||||
private readonly remote: Database,
|
||||
private readonly summary: SyncMergeSummary,
|
||||
) {}
|
||||
|
||||
resolveWord(remoteWordId: number): number {
|
||||
const cached = this.wordMap.get(remoteWordId);
|
||||
if (cached) return cached.localId;
|
||||
|
||||
const row = this.remote
|
||||
.query<SqlRow>(`SELECT ${WORD_COPY_COLUMNS.join(', ')} FROM imm_words WHERE id = ?`)
|
||||
.get(remoteWordId);
|
||||
if (!row) throw new Error(`Snapshot references missing imm_words row ${remoteWordId}`);
|
||||
|
||||
const existing = this.local
|
||||
.query<SqlRow>('SELECT id FROM imm_words WHERE headword IS ? AND word IS ? AND reading IS ?')
|
||||
.get(row.headword, row.word, row.reading);
|
||||
let entry: { localId: number; isNew: boolean };
|
||||
if (existing) {
|
||||
entry = { localId: Number(existing.id), isNew: false };
|
||||
this.local
|
||||
.prepare(
|
||||
`UPDATE imm_words
|
||||
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
|
||||
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
|
||||
WHERE id = ?`,
|
||||
)
|
||||
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
|
||||
} else {
|
||||
const localId = insertRow(
|
||||
this.local,
|
||||
'imm_words',
|
||||
WORD_COPY_COLUMNS,
|
||||
WORD_COPY_COLUMNS.map((column) => row[column]),
|
||||
);
|
||||
entry = { localId, isNew: true };
|
||||
this.summary.wordsAdded += 1;
|
||||
}
|
||||
this.wordMap.set(remoteWordId, entry);
|
||||
return entry.localId;
|
||||
}
|
||||
|
||||
resolveKanji(remoteKanjiId: number): number {
|
||||
const cached = this.kanjiMap.get(remoteKanjiId);
|
||||
if (cached) return cached.localId;
|
||||
|
||||
const row = this.remote
|
||||
.query<SqlRow>('SELECT kanji, first_seen, last_seen, frequency FROM imm_kanji WHERE id = ?')
|
||||
.get(remoteKanjiId);
|
||||
if (!row) throw new Error(`Snapshot references missing imm_kanji row ${remoteKanjiId}`);
|
||||
|
||||
const existing = this.local
|
||||
.query<SqlRow>('SELECT id FROM imm_kanji WHERE kanji IS ?')
|
||||
.get(row.kanji);
|
||||
let entry: { localId: number; isNew: boolean };
|
||||
if (existing) {
|
||||
entry = { localId: Number(existing.id), isNew: false };
|
||||
this.local
|
||||
.prepare(
|
||||
`UPDATE imm_kanji
|
||||
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
|
||||
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
|
||||
WHERE id = ?`,
|
||||
)
|
||||
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
|
||||
} else {
|
||||
const localId = insertRow(
|
||||
this.local,
|
||||
'imm_kanji',
|
||||
['kanji', 'first_seen', 'last_seen', 'frequency'],
|
||||
[row.kanji, row.first_seen, row.last_seen, row.frequency],
|
||||
);
|
||||
entry = { localId, isNew: true };
|
||||
this.summary.kanjiAdded += 1;
|
||||
}
|
||||
this.kanjiMap.set(remoteKanjiId, entry);
|
||||
return entry.localId;
|
||||
}
|
||||
|
||||
addWordOccurrences(remoteWordId: number, count: number): void {
|
||||
const entry = this.wordMap.get(remoteWordId);
|
||||
if (!entry || entry.isNew) return;
|
||||
this.wordFrequencyDeltas.set(
|
||||
entry.localId,
|
||||
(this.wordFrequencyDeltas.get(entry.localId) ?? 0) + count,
|
||||
);
|
||||
}
|
||||
|
||||
addKanjiOccurrences(remoteKanjiId: number, count: number): void {
|
||||
const entry = this.kanjiMap.get(remoteKanjiId);
|
||||
if (!entry || entry.isNew) return;
|
||||
this.kanjiFrequencyDeltas.set(
|
||||
entry.localId,
|
||||
(this.kanjiFrequencyDeltas.get(entry.localId) ?? 0) + count,
|
||||
);
|
||||
}
|
||||
|
||||
applyFrequencyDeltas(): void {
|
||||
const updateWord = this.local.prepare(
|
||||
'UPDATE imm_words SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
|
||||
);
|
||||
for (const [localId, delta] of this.wordFrequencyDeltas) {
|
||||
updateWord.run(delta, localId);
|
||||
}
|
||||
const updateKanji = this.local.prepare(
|
||||
'UPDATE imm_kanji SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
|
||||
);
|
||||
for (const [localId, delta] of this.kanjiFrequencyDeltas) {
|
||||
updateKanji.run(delta, localId);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { nowDbTimestamp, tableExists, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
const LOCAL_DAY_EXPR = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
|
||||
const LOCAL_MONTH_EXPR = `CAST(strftime('%Y%m', CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER)`;
|
||||
|
||||
// Ported from upsertDailyRollupsForGroups / upsertMonthlyRollupsForGroups in
|
||||
// src/core/services/immersion-tracker/maintenance.ts — must stay in sync.
|
||||
const DAILY_ROLLUP_UPSERT = `
|
||||
WITH matching_sessions AS (
|
||||
SELECT * FROM imm_sessions
|
||||
WHERE ${LOCAL_DAY_EXPR} = ? AND video_id = ?
|
||||
),
|
||||
session_metrics AS (
|
||||
SELECT
|
||||
t.session_id,
|
||||
MAX(t.active_watched_ms) AS max_active_ms,
|
||||
MAX(t.lines_seen) AS max_lines,
|
||||
MAX(t.tokens_seen) AS max_tokens,
|
||||
MAX(t.cards_mined) AS max_cards,
|
||||
MAX(t.lookup_count) AS max_lookups,
|
||||
MAX(t.lookup_hits) AS max_hits
|
||||
FROM imm_session_telemetry t
|
||||
JOIN matching_sessions s ON s.session_id = t.session_id
|
||||
GROUP BY t.session_id
|
||||
)
|
||||
INSERT INTO imm_daily_rollups (
|
||||
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
SELECT
|
||||
${LOCAL_DAY_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_day,
|
||||
s.video_id AS video_id,
|
||||
COUNT(DISTINCT s.session_id) AS total_sessions,
|
||||
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
|
||||
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
|
||||
THEN (COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) * 60.0)
|
||||
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
|
||||
ELSE NULL
|
||||
END AS cards_per_hour,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
|
||||
THEN COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0)
|
||||
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
|
||||
ELSE NULL
|
||||
END AS tokens_per_min,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) > 0
|
||||
THEN CAST(COALESCE(SUM(COALESCE(sm.max_hits, s.lookup_hits)), 0) AS REAL)
|
||||
/ CAST(COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) AS REAL)
|
||||
ELSE NULL
|
||||
END AS lookup_hit_rate,
|
||||
? AS CREATED_DATE,
|
||||
? AS LAST_UPDATE_DATE
|
||||
FROM matching_sessions s
|
||||
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
|
||||
GROUP BY rollup_day, s.video_id
|
||||
ON CONFLICT (rollup_day, video_id) DO UPDATE SET
|
||||
total_sessions = excluded.total_sessions,
|
||||
total_active_min = excluded.total_active_min,
|
||||
total_lines_seen = excluded.total_lines_seen,
|
||||
total_tokens_seen = excluded.total_tokens_seen,
|
||||
total_cards = excluded.total_cards,
|
||||
cards_per_hour = excluded.cards_per_hour,
|
||||
tokens_per_min = excluded.tokens_per_min,
|
||||
lookup_hit_rate = excluded.lookup_hit_rate,
|
||||
CREATED_DATE = COALESCE(imm_daily_rollups.CREATED_DATE, excluded.CREATED_DATE),
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
|
||||
`;
|
||||
|
||||
const MONTHLY_ROLLUP_UPSERT = `
|
||||
WITH matching_sessions AS (
|
||||
SELECT * FROM imm_sessions
|
||||
WHERE ${LOCAL_MONTH_EXPR} = ? AND video_id = ?
|
||||
),
|
||||
session_metrics AS (
|
||||
SELECT
|
||||
t.session_id,
|
||||
MAX(t.active_watched_ms) AS max_active_ms,
|
||||
MAX(t.lines_seen) AS max_lines,
|
||||
MAX(t.tokens_seen) AS max_tokens,
|
||||
MAX(t.cards_mined) AS max_cards
|
||||
FROM imm_session_telemetry t
|
||||
JOIN matching_sessions s ON s.session_id = t.session_id
|
||||
GROUP BY t.session_id
|
||||
)
|
||||
INSERT INTO imm_monthly_rollups (
|
||||
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
SELECT
|
||||
${LOCAL_MONTH_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_month,
|
||||
s.video_id AS video_id,
|
||||
COUNT(DISTINCT s.session_id) AS total_sessions,
|
||||
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
|
||||
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
|
||||
? AS CREATED_DATE,
|
||||
? AS LAST_UPDATE_DATE
|
||||
FROM matching_sessions s
|
||||
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
|
||||
GROUP BY rollup_month, s.video_id
|
||||
ON CONFLICT (rollup_month, video_id) DO UPDATE SET
|
||||
total_sessions = excluded.total_sessions,
|
||||
total_active_min = excluded.total_active_min,
|
||||
total_lines_seen = excluded.total_lines_seen,
|
||||
total_tokens_seen = excluded.total_tokens_seen,
|
||||
total_cards = excluded.total_cards,
|
||||
CREATED_DATE = COALESCE(imm_monthly_rollups.CREATED_DATE, excluded.CREATED_DATE),
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
|
||||
`;
|
||||
|
||||
/**
|
||||
* Recompute daily/monthly rollup groups touched by the newly merged sessions
|
||||
* from the (now merged) local session + telemetry data. The maintenance
|
||||
* watermark is left alone: telemetry newer than it gets recomputed again by
|
||||
* the app later, which is idempotent.
|
||||
*/
|
||||
export function refreshRollupsForNewSessions(
|
||||
local: Database,
|
||||
newSessionIds: number[],
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (newSessionIds.length === 0) return;
|
||||
|
||||
const groups = new Map<string, { day: number; month: number; videoId: number }>();
|
||||
for (let offset = 0; offset < newSessionIds.length; offset += 500) {
|
||||
const chunk = newSessionIds.slice(offset, offset + 500);
|
||||
const rows = local
|
||||
.query<SqlRow>(
|
||||
`SELECT DISTINCT ${LOCAL_DAY_EXPR} AS rollup_day, ${LOCAL_MONTH_EXPR} AS rollup_month, video_id
|
||||
FROM imm_sessions WHERE session_id IN (${chunk.map(() => '?').join(',')})`,
|
||||
)
|
||||
.all(...chunk);
|
||||
for (const row of rows) {
|
||||
const day = Number(row.rollup_day);
|
||||
const month = Number(row.rollup_month);
|
||||
const videoId = Number(row.video_id);
|
||||
groups.set(`${day}-${videoId}`, { day, month, videoId });
|
||||
}
|
||||
}
|
||||
|
||||
const stampMs = nowDbTimestamp();
|
||||
const deleteDaily = local.prepare(
|
||||
'DELETE FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ?',
|
||||
);
|
||||
const deleteMonthly = local.prepare(
|
||||
'DELETE FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ?',
|
||||
);
|
||||
const upsertDaily = local.prepare(DAILY_ROLLUP_UPSERT);
|
||||
const upsertMonthly = local.prepare(MONTHLY_ROLLUP_UPSERT);
|
||||
|
||||
const monthlyGroups = new Set<string>();
|
||||
for (const { day, month, videoId } of groups.values()) {
|
||||
deleteDaily.run(day, videoId);
|
||||
upsertDaily.run(day, videoId, stampMs, stampMs);
|
||||
summary.rollupGroupsRecomputed += 1;
|
||||
const monthKey = `${month}-${videoId}`;
|
||||
if (!monthlyGroups.has(monthKey)) {
|
||||
monthlyGroups.add(monthKey);
|
||||
deleteMonthly.run(month, videoId);
|
||||
upsertMonthly.run(month, videoId, stampMs, stampMs);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sessions are pruned after a retention window, but rollups are kept much
|
||||
* longer — the remote's older rollup history can't be reconstructed from
|
||||
* merged sessions. Copy remote rollup rows for groups where the local DB has
|
||||
* neither a rollup row nor any sessions (i.e. history only the remote knows).
|
||||
* Groups both machines have data for are never summed, to avoid
|
||||
* double-counting sessions that earlier syncs already shared.
|
||||
*/
|
||||
export function copyRemoteOnlyRollups(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (!tableExists(remote, 'imm_daily_rollups') || !tableExists(local, 'imm_daily_rollups')) return;
|
||||
|
||||
const localDailyExists = local.prepare(
|
||||
'SELECT 1 FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ? LIMIT 1',
|
||||
);
|
||||
const localDaySessions = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_DAY_EXPR} = ? LIMIT 1`,
|
||||
);
|
||||
const localMonthSessions = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_MONTH_EXPR} = ? LIMIT 1`,
|
||||
);
|
||||
const localMonthSessionsForDay = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions
|
||||
WHERE video_id = ?
|
||||
AND ${LOCAL_MONTH_EXPR} = CAST(strftime('%Y%m', CAST(? AS INTEGER) * 86400, 'unixepoch', 'localtime') AS INTEGER)
|
||||
LIMIT 1`,
|
||||
);
|
||||
const insertDaily = local.prepare(
|
||||
`INSERT INTO imm_daily_rollups (
|
||||
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
);
|
||||
for (const row of remote.query<SqlRow>('SELECT * FROM imm_daily_rollups').all()) {
|
||||
if (row.video_id === null) continue;
|
||||
const localVideoId = videoIdMap.get(Number(row.video_id));
|
||||
if (localVideoId === undefined) continue;
|
||||
if (localDailyExists.get(row.rollup_day, localVideoId)) continue;
|
||||
if (localDaySessions.get(localVideoId, row.rollup_day)) continue;
|
||||
if (localMonthSessionsForDay.get(localVideoId, row.rollup_day)) continue;
|
||||
insertDaily.run(
|
||||
row.rollup_day,
|
||||
localVideoId,
|
||||
row.total_sessions,
|
||||
row.total_active_min,
|
||||
row.total_lines_seen,
|
||||
row.total_tokens_seen,
|
||||
row.total_cards,
|
||||
row.cards_per_hour,
|
||||
row.tokens_per_min,
|
||||
row.lookup_hit_rate,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.dailyRollupsCopied += 1;
|
||||
}
|
||||
|
||||
const localMonthlyExists = local.prepare(
|
||||
'SELECT 1 FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ? LIMIT 1',
|
||||
);
|
||||
const insertMonthly = local.prepare(
|
||||
`INSERT INTO imm_monthly_rollups (
|
||||
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
);
|
||||
for (const row of remote.query<SqlRow>('SELECT * FROM imm_monthly_rollups').all()) {
|
||||
if (row.video_id === null) continue;
|
||||
const localVideoId = videoIdMap.get(Number(row.video_id));
|
||||
if (localVideoId === undefined) continue;
|
||||
if (localMonthlyExists.get(row.rollup_month, localVideoId)) continue;
|
||||
if (localMonthSessions.get(localVideoId, row.rollup_month)) continue;
|
||||
insertMonthly.run(
|
||||
row.rollup_month,
|
||||
localVideoId,
|
||||
row.total_sessions,
|
||||
row.total_active_min,
|
||||
row.total_lines_seen,
|
||||
row.total_tokens_seen,
|
||||
row.total_cards,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.monthlyRollupsCopied += 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,465 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import type { LexiconResolver } from './merge-catalog.js';
|
||||
import { insertRow, nowDbTimestamp, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
const SESSION_COPY_COLUMNS = [
|
||||
'session_uuid',
|
||||
'started_at_ms',
|
||||
'ended_at_ms',
|
||||
'status',
|
||||
'locale_id',
|
||||
'target_lang_id',
|
||||
'difficulty_tier',
|
||||
'subtitle_mode',
|
||||
'ended_media_ms',
|
||||
'total_watched_ms',
|
||||
'active_watched_ms',
|
||||
'lines_seen',
|
||||
'tokens_seen',
|
||||
'cards_mined',
|
||||
'lookup_count',
|
||||
'lookup_hits',
|
||||
'yomitan_lookup_count',
|
||||
'pause_count',
|
||||
'pause_ms',
|
||||
'seek_forward_count',
|
||||
'seek_backward_count',
|
||||
'media_buffer_events',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const TELEMETRY_COPY_COLUMNS = [
|
||||
'sample_ms',
|
||||
'total_watched_ms',
|
||||
'active_watched_ms',
|
||||
'lines_seen',
|
||||
'tokens_seen',
|
||||
'cards_mined',
|
||||
'lookup_count',
|
||||
'lookup_hits',
|
||||
'yomitan_lookup_count',
|
||||
'pause_count',
|
||||
'pause_ms',
|
||||
'seek_forward_count',
|
||||
'seek_backward_count',
|
||||
'media_buffer_events',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const EVENT_COPY_COLUMNS = [
|
||||
'ts_ms',
|
||||
'event_type',
|
||||
'line_index',
|
||||
'segment_start_ms',
|
||||
'segment_end_ms',
|
||||
'tokens_delta',
|
||||
'cards_delta',
|
||||
'payload_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const LINE_COPY_COLUMNS = [
|
||||
'line_index',
|
||||
'segment_start_ms',
|
||||
'segment_end_ms',
|
||||
'text',
|
||||
'secondary_text',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
export interface SessionMergeResult {
|
||||
newSessionIds: number[];
|
||||
}
|
||||
|
||||
export function mergeSessions(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
animeIdMap: Map<number, number>,
|
||||
lexicon: LexiconResolver,
|
||||
summary: SyncMergeSummary,
|
||||
): SessionMergeResult {
|
||||
const newSessionIds: number[] = [];
|
||||
const uuidExists = local.prepare<SqlRow>(
|
||||
'SELECT session_id FROM imm_sessions WHERE session_uuid = ?',
|
||||
);
|
||||
|
||||
const remoteSessions = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT session_id, video_id, ${SESSION_COPY_COLUMNS.join(', ')}
|
||||
FROM imm_sessions
|
||||
ORDER BY CAST(started_at_ms AS REAL) ASC, session_id ASC`,
|
||||
)
|
||||
.all();
|
||||
|
||||
for (const session of remoteSessions) {
|
||||
if (session.ended_at_ms === null) {
|
||||
// Stale ACTIVE sessions are finalized by the app on its next startup;
|
||||
// they will sync once they carry final numbers.
|
||||
summary.activeSessionsSkipped += 1;
|
||||
continue;
|
||||
}
|
||||
if (uuidExists.get(session.session_uuid)) {
|
||||
summary.sessionsAlreadyPresent += 1;
|
||||
continue;
|
||||
}
|
||||
const localVideoId = videoIdMap.get(Number(session.video_id));
|
||||
if (localVideoId === undefined) {
|
||||
throw new Error(`Snapshot session ${String(session.session_uuid)} references missing video row`);
|
||||
}
|
||||
|
||||
const localSessionId = insertRow(
|
||||
local,
|
||||
'imm_sessions',
|
||||
['video_id', ...SESSION_COPY_COLUMNS],
|
||||
[localVideoId, ...SESSION_COPY_COLUMNS.map((column) => session[column])],
|
||||
);
|
||||
newSessionIds.push(localSessionId);
|
||||
summary.sessionsMerged += 1;
|
||||
|
||||
const remoteSessionId = Number(session.session_id);
|
||||
copyTelemetry(local, remote, remoteSessionId, localSessionId, summary);
|
||||
const eventIdMap = copyEvents(local, remote, remoteSessionId, localSessionId, summary);
|
||||
copySubtitleLines(
|
||||
local,
|
||||
remote,
|
||||
remoteSessionId,
|
||||
localSessionId,
|
||||
localVideoId,
|
||||
animeIdMap,
|
||||
eventIdMap,
|
||||
lexicon,
|
||||
summary,
|
||||
);
|
||||
applyMergedSessionLifetime(local, localSessionId, localVideoId, session);
|
||||
}
|
||||
|
||||
return { newSessionIds };
|
||||
}
|
||||
|
||||
function copyTelemetry(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${TELEMETRY_COPY_COLUMNS.join(', ')} FROM imm_session_telemetry
|
||||
WHERE session_id = ? ORDER BY telemetry_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
for (const row of rows) {
|
||||
insertRow(
|
||||
local,
|
||||
'imm_session_telemetry',
|
||||
['session_id', ...TELEMETRY_COPY_COLUMNS],
|
||||
[localSessionId, ...TELEMETRY_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
summary.telemetryRowsAdded += 1;
|
||||
}
|
||||
}
|
||||
|
||||
function copyEvents(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
summary: SyncMergeSummary,
|
||||
): Map<number, number> {
|
||||
const eventIdMap = new Map<number, number>();
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT event_id, ${EVENT_COPY_COLUMNS.join(', ')} FROM imm_session_events
|
||||
WHERE session_id = ? ORDER BY event_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
for (const row of rows) {
|
||||
const localEventId = insertRow(
|
||||
local,
|
||||
'imm_session_events',
|
||||
['session_id', ...EVENT_COPY_COLUMNS],
|
||||
[localSessionId, ...EVENT_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
eventIdMap.set(Number(row.event_id), localEventId);
|
||||
summary.eventsAdded += 1;
|
||||
}
|
||||
return eventIdMap;
|
||||
}
|
||||
|
||||
function copySubtitleLines(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
localVideoId: number,
|
||||
animeIdMap: Map<number, number>,
|
||||
eventIdMap: Map<number, number>,
|
||||
lexicon: LexiconResolver,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT line_id, event_id, anime_id, ${LINE_COPY_COLUMNS.join(', ')} FROM imm_subtitle_lines
|
||||
WHERE session_id = ? ORDER BY line_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
const wordOccurrences = remote.prepare<SqlRow>(
|
||||
'SELECT word_id, occurrence_count FROM imm_word_line_occurrences WHERE line_id = ?',
|
||||
);
|
||||
const kanjiOccurrences = remote.prepare<SqlRow>(
|
||||
'SELECT kanji_id, occurrence_count FROM imm_kanji_line_occurrences WHERE line_id = ?',
|
||||
);
|
||||
const insertWordOccurrence = local.prepare(
|
||||
`INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)
|
||||
ON CONFLICT(line_id, word_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
|
||||
);
|
||||
const insertKanjiOccurrence = local.prepare(
|
||||
`INSERT INTO imm_kanji_line_occurrences (line_id, kanji_id, occurrence_count) VALUES (?, ?, ?)
|
||||
ON CONFLICT(line_id, kanji_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
|
||||
);
|
||||
|
||||
for (const row of rows) {
|
||||
const localEventId = row.event_id === null ? null : (eventIdMap.get(Number(row.event_id)) ?? null);
|
||||
const localAnimeId = row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
|
||||
const localLineId = insertRow(
|
||||
local,
|
||||
'imm_subtitle_lines',
|
||||
['session_id', 'event_id', 'video_id', 'anime_id', ...LINE_COPY_COLUMNS],
|
||||
[
|
||||
localSessionId,
|
||||
localEventId,
|
||||
localVideoId,
|
||||
localAnimeId,
|
||||
...LINE_COPY_COLUMNS.map((column) => row[column]),
|
||||
],
|
||||
);
|
||||
summary.subtitleLinesAdded += 1;
|
||||
|
||||
for (const occurrence of wordOccurrences.all(row.line_id)) {
|
||||
const localWordId = lexicon.resolveWord(Number(occurrence.word_id));
|
||||
const count = Number(occurrence.occurrence_count);
|
||||
insertWordOccurrence.run(localLineId, localWordId, count);
|
||||
lexicon.addWordOccurrences(Number(occurrence.word_id), count);
|
||||
}
|
||||
for (const occurrence of kanjiOccurrences.all(row.line_id)) {
|
||||
const localKanjiId = lexicon.resolveKanji(Number(occurrence.kanji_id));
|
||||
const count = Number(occurrence.occurrence_count);
|
||||
insertKanjiOccurrence.run(localLineId, localKanjiId, count);
|
||||
lexicon.addKanjiOccurrences(Number(occurrence.kanji_id), count);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Port of applySessionLifetimeSummary (src/core/services/immersion-tracker/
|
||||
* lifetime.ts) for sessions arriving out of chronological order. The
|
||||
* "first session of the day / for this video" checks are order-independent
|
||||
* here (any other session counts, not just earlier ones): the local machine
|
||||
* already credited active_days/episodes_started when its own session was
|
||||
* applied, even if the merged session started earlier that day.
|
||||
*/
|
||||
function applyMergedSessionLifetime(
|
||||
local: Database,
|
||||
sessionId: number,
|
||||
videoId: number,
|
||||
session: SqlRow,
|
||||
): void {
|
||||
const updatedAtMs = nowDbTimestamp();
|
||||
const applied = local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(session_id) DO NOTHING`,
|
||||
)
|
||||
.run(sessionId, session.ended_at_ms, updatedAtMs, updatedAtMs);
|
||||
if (applied.changes <= 0) return;
|
||||
|
||||
const telemetry = local
|
||||
.query<SqlRow>(
|
||||
`SELECT active_watched_ms, cards_mined, lines_seen, tokens_seen
|
||||
FROM imm_session_telemetry
|
||||
WHERE session_id = ?
|
||||
ORDER BY sample_ms DESC, telemetry_id DESC
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get(sessionId);
|
||||
|
||||
const metric = (telemetryValue: unknown, sessionValue: unknown): number => {
|
||||
const fromTelemetry = telemetry ? Number(telemetryValue) : Number.NaN;
|
||||
const value = Number.isFinite(fromTelemetry) ? fromTelemetry : Number(sessionValue);
|
||||
return Math.max(0, Math.floor(Number.isFinite(value) ? value : 0));
|
||||
};
|
||||
const activeMs = metric(telemetry?.active_watched_ms, session.active_watched_ms);
|
||||
const cardsMined = metric(telemetry?.cards_mined, session.cards_mined);
|
||||
const linesSeen = metric(telemetry?.lines_seen, session.lines_seen);
|
||||
const tokensSeen = metric(telemetry?.tokens_seen, session.tokens_seen);
|
||||
|
||||
const video = local
|
||||
.query<SqlRow>('SELECT anime_id, watched FROM imm_videos WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const watched = Number(video?.watched ?? 0);
|
||||
const animeId = video?.anime_id === null || video?.anime_id === undefined ? null : Number(video.anime_id);
|
||||
|
||||
const mediaLifetime = local
|
||||
.query<SqlRow>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const hasOtherSessionForVideo = Boolean(
|
||||
local
|
||||
.query('SELECT 1 FROM imm_sessions WHERE video_id = ? AND session_id != ? LIMIT 1')
|
||||
.get(videoId, sessionId),
|
||||
);
|
||||
const isFirstSessionForVideoRun = !mediaLifetime && !hasOtherSessionForVideo;
|
||||
const isFirstCompletedSessionForVideoRun = watched > 0 && Number(mediaLifetime?.completed ?? 0) <= 0;
|
||||
|
||||
const hasOtherSessionOnDay = Boolean(
|
||||
local
|
||||
.query(
|
||||
`SELECT 1 FROM imm_sessions
|
||||
WHERE session_id != ?
|
||||
AND CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
|
||||
= CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get(sessionId, session.started_at_ms),
|
||||
);
|
||||
|
||||
let animeCompletedDelta = 0;
|
||||
if (animeId !== null && watched > 0 && isFirstCompletedSessionForVideoRun) {
|
||||
const animeLifetime = local
|
||||
.query<SqlRow>('SELECT episodes_completed FROM imm_lifetime_anime WHERE anime_id = ?')
|
||||
.get(animeId);
|
||||
const anime = local
|
||||
.query<SqlRow>('SELECT episodes_total FROM imm_anime WHERE anime_id = ?')
|
||||
.get(animeId);
|
||||
const episodesCompletedBefore = Number(animeLifetime?.episodes_completed ?? 0);
|
||||
const episodesTotal = anime?.episodes_total === null || anime?.episodes_total === undefined
|
||||
? null
|
||||
: Number(anime.episodes_total);
|
||||
if (
|
||||
episodesTotal !== null &&
|
||||
episodesTotal > 0 &&
|
||||
episodesCompletedBefore < episodesTotal &&
|
||||
episodesCompletedBefore + 1 >= episodesTotal
|
||||
) {
|
||||
animeCompletedDelta = 1;
|
||||
}
|
||||
}
|
||||
|
||||
local
|
||||
.prepare(
|
||||
`UPDATE imm_lifetime_global
|
||||
SET total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + ?,
|
||||
total_cards = total_cards + ?,
|
||||
active_days = active_days + ?,
|
||||
episodes_started = episodes_started + ?,
|
||||
episodes_completed = episodes_completed + ?,
|
||||
anime_completed = anime_completed + ?,
|
||||
LAST_UPDATE_DATE = ?
|
||||
WHERE global_id = 1`,
|
||||
)
|
||||
.run(
|
||||
activeMs,
|
||||
cardsMined,
|
||||
hasOtherSessionOnDay ? 0 : 1,
|
||||
isFirstSessionForVideoRun ? 1 : 0,
|
||||
isFirstCompletedSessionForVideoRun ? 1 : 0,
|
||||
animeCompletedDelta,
|
||||
updatedAtMs,
|
||||
);
|
||||
|
||||
local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_media(
|
||||
video_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
|
||||
total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(video_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
completed = MAX(completed, excluded.completed),
|
||||
first_watched_ms = CASE
|
||||
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
|
||||
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
|
||||
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
|
||||
ELSE first_watched_ms
|
||||
END,
|
||||
last_watched_ms = CASE
|
||||
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
|
||||
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
|
||||
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
|
||||
ELSE last_watched_ms
|
||||
END,
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
|
||||
)
|
||||
.run(
|
||||
videoId,
|
||||
activeMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
watched > 0 ? 1 : 0,
|
||||
session.started_at_ms,
|
||||
session.ended_at_ms,
|
||||
updatedAtMs,
|
||||
updatedAtMs,
|
||||
);
|
||||
|
||||
if (animeId !== null) {
|
||||
local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_anime(
|
||||
anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
|
||||
total_tokens_seen, episodes_started, episodes_completed, first_watched_ms,
|
||||
last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(anime_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
episodes_started = episodes_started + excluded.episodes_started,
|
||||
episodes_completed = episodes_completed + excluded.episodes_completed,
|
||||
first_watched_ms = CASE
|
||||
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
|
||||
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
|
||||
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
|
||||
ELSE first_watched_ms
|
||||
END,
|
||||
last_watched_ms = CASE
|
||||
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
|
||||
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
|
||||
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
|
||||
ELSE last_watched_ms
|
||||
END,
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
|
||||
)
|
||||
.run(
|
||||
animeId,
|
||||
activeMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
isFirstSessionForVideoRun ? 1 : 0,
|
||||
isFirstCompletedSessionForVideoRun ? 1 : 0,
|
||||
session.started_at_ms,
|
||||
session.ended_at_ms,
|
||||
updatedAtMs,
|
||||
updatedAtMs,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { assertSafeSshHost, resolveRemoteSubminerCommand, runScp, shellQuote } from './ssh.js';
|
||||
|
||||
test('assertSafeSshHost rejects option-like hosts', () => {
|
||||
assert.throws(() => assertSafeSshHost('-oProxyCommand=touch pwned'), /looks like an option/);
|
||||
assert.throws(() => assertSafeSshHost('-lroot'), /looks like an option/);
|
||||
});
|
||||
|
||||
test('assertSafeSshHost accepts normal destinations', () => {
|
||||
assert.doesNotThrow(() => assertSafeSshHost('macbook'));
|
||||
assert.doesNotThrow(() => assertSafeSshHost('user@192.168.1.20'));
|
||||
assert.doesNotThrow(() => assertSafeSshHost('ssh-alias'));
|
||||
});
|
||||
|
||||
test('shellQuote escapes single quotes and wraps in quotes', () => {
|
||||
assert.equal(shellQuote('subminer'), `'subminer'`);
|
||||
assert.equal(shellQuote(`a'; rm -rf ~; '`), `'a'\\''; rm -rf ~; '\\'''`);
|
||||
});
|
||||
|
||||
test('runScp rejects option-like local endpoints before spawning scp', () => {
|
||||
assert.throws(() => runScp('-oProxyCommand=sh', '/tmp/out.sqlite'), /looks like an option/);
|
||||
assert.throws(() => runScp('/tmp/in.sqlite', '-bad-destination'), /looks like an option/);
|
||||
});
|
||||
|
||||
test('runScp rejects option-like remote host components', () => {
|
||||
assert.throws(
|
||||
() => runScp('-oProxyCommand=sh:/tmp/in.sqlite', '/tmp/out.sqlite'),
|
||||
/SSH host that looks like an option/,
|
||||
);
|
||||
});
|
||||
|
||||
test('resolveRemoteSubminerCommand verifies the launcher under the remote runtime PATH', () => {
|
||||
const calls: Array<{ host: string; remoteCommand: string }> = [];
|
||||
const command = resolveRemoteSubminerCommand('macbook', null, (host, remoteCommand) => {
|
||||
calls.push({ host, remoteCommand });
|
||||
return { status: 0, stdout: '', stderr: '' };
|
||||
});
|
||||
|
||||
assert.equal(
|
||||
command,
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer',
|
||||
);
|
||||
assert.deepEqual(calls, [
|
||||
{
|
||||
host: 'macbook',
|
||||
remoteCommand:
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer --help >/dev/null 2>&1',
|
||||
},
|
||||
]);
|
||||
});
|
||||
@@ -0,0 +1,106 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
export interface RemoteRunResult {
|
||||
status: number;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* ssh/scp have no `--` terminator for the destination, so a host that starts
|
||||
* with `-` (e.g. `-oProxyCommand=...`) is parsed as an option. Reject those
|
||||
* before spawning.
|
||||
*/
|
||||
export function assertSafeSshHost(host: string): void {
|
||||
if (host.startsWith('-')) {
|
||||
throw new Error(`Refusing to use SSH host that looks like an option: ${host}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a command on the SSH host. stdin stays attached so interactive prompts
|
||||
* can still read from the terminal; stdout/stderr are captured for callers
|
||||
* that need actionable remote failure messages.
|
||||
*/
|
||||
export function runSsh(host: string, remoteCommand: string): RemoteRunResult {
|
||||
assertSafeSshHost(host);
|
||||
const result = spawnSync('ssh', [host, remoteCommand], {
|
||||
encoding: 'utf8',
|
||||
stdio: ['inherit', 'pipe', 'pipe'],
|
||||
});
|
||||
if (result.error) {
|
||||
throw new Error(`Failed to run ssh: ${(result.error as Error).message}`);
|
||||
}
|
||||
return { status: result.status ?? 1, stdout: result.stdout ?? '', stderr: result.stderr ?? '' };
|
||||
}
|
||||
|
||||
function assertSafeScpEndpoint(endpoint: string): void {
|
||||
const colon = endpoint.indexOf(':');
|
||||
const slash = endpoint.indexOf('/');
|
||||
if (colon <= 0 || (slash !== -1 && slash < colon)) {
|
||||
if (endpoint.startsWith('-')) {
|
||||
throw new Error(`Refusing to use scp endpoint that looks like an option: ${endpoint}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const host = endpoint.slice(0, colon);
|
||||
const remotePath = endpoint.slice(colon + 1);
|
||||
assertSafeSshHost(host);
|
||||
if (remotePath.startsWith('-')) {
|
||||
throw new Error(`Refusing to use scp remote path that looks like an option: ${remotePath}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function runScp(from: string, to: string): void {
|
||||
assertSafeScpEndpoint(from);
|
||||
assertSafeScpEndpoint(to);
|
||||
const result = spawnSync('scp', ['-q', from, to], {
|
||||
encoding: 'utf8',
|
||||
stdio: ['inherit', 'inherit', 'inherit'],
|
||||
});
|
||||
if (result.error) {
|
||||
throw new Error(`Failed to run scp: ${(result.error as Error).message}`);
|
||||
}
|
||||
if ((result.status ?? 1) !== 0) {
|
||||
throw new Error(`scp failed copying ${from} -> ${to}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function shellQuote(value: string): string {
|
||||
return `'${value.replaceAll("'", `'\\''`)}'`;
|
||||
}
|
||||
|
||||
const REMOTE_RUNTIME_PATH =
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"';
|
||||
|
||||
/**
|
||||
* Non-interactive SSH shells often miss user-installed launchers and Bun.
|
||||
* Probe the launcher under the same deterministic PATH used by sync itself.
|
||||
*/
|
||||
export function resolveRemoteSubminerCommand(
|
||||
host: string,
|
||||
preferred: string | null,
|
||||
runRemote: typeof runSsh = runSsh,
|
||||
): string {
|
||||
// Trusted defaults stay unquoted so the remote shell expands `~`; a
|
||||
// user-supplied override is shell-quoted to prevent command injection.
|
||||
const candidates: Array<{ value: string; invocation: string }> = preferred
|
||||
? [{ value: preferred, invocation: shellQuote(preferred) }]
|
||||
: [
|
||||
{ value: 'subminer', invocation: 'subminer' },
|
||||
{ value: '~/.local/bin/subminer', invocation: '~/.local/bin/subminer' },
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
const command = `${REMOTE_RUNTIME_PATH} ${candidate.invocation}`;
|
||||
const probe = runRemote(host, `${command} --help >/dev/null 2>&1`);
|
||||
if (probe.status === 0) {
|
||||
return command;
|
||||
}
|
||||
}
|
||||
throw new Error(
|
||||
preferred
|
||||
? `Remote command not found on ${host}: ${preferred}`
|
||||
: `subminer not found on ${host} (tried PATH and ~/.local/bin/subminer). Pass --remote-cmd <path>.`,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,598 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { createDbSnapshot, mergeSnapshotIntoDb } from './sync-db.js';
|
||||
import {
|
||||
createImmersionDbFixture,
|
||||
insertFixtureSession,
|
||||
} from '../test-support/immersion-db-fixture.js';
|
||||
|
||||
const DAY_MS = 86_400_000;
|
||||
const BASE_MS = Date.UTC(2026, 5, 1, 12, 0, 0);
|
||||
|
||||
function makeTmpDir(): string {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-test-'));
|
||||
}
|
||||
|
||||
function makeDbPair(): { dir: string; localPath: string; remotePath: string } {
|
||||
const dir = makeTmpDir();
|
||||
const localPath = path.join(dir, 'local.sqlite');
|
||||
const remotePath = path.join(dir, 'remote.sqlite');
|
||||
createImmersionDbFixture(localPath);
|
||||
createImmersionDbFixture(remotePath);
|
||||
return { dir, localPath, remotePath };
|
||||
}
|
||||
|
||||
function queryOne<T extends Record<string, unknown>>(
|
||||
dbPath: string,
|
||||
sql: string,
|
||||
params: unknown[] = [],
|
||||
): T | undefined {
|
||||
const db = new Database(dbPath, { readonly: true });
|
||||
try {
|
||||
return db.query<T>(sql).get(...params) as T | undefined;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function count(dbPath: string, sql: string, params: unknown[] = []): number {
|
||||
return Number(queryOne<{ n: number }>(dbPath, sql, params)?.n ?? 0);
|
||||
}
|
||||
|
||||
function withWritableDb<T>(dbPath: string, fn: (db: Database) => T): T {
|
||||
const db = new Database(dbPath, { readwrite: true });
|
||||
try {
|
||||
return fn(db);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
test('merges remote-only sessions with catalog, lifetime, and rollups', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 2 }],
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
animeTitleKey: 'showb',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
activeWatchedMs: 900_000,
|
||||
cardsMined: 3,
|
||||
applyLifetime: true,
|
||||
words: [
|
||||
{ headword: '見る', word: '見た', reading: 'みた', count: 5 },
|
||||
{ headword: '食べる', word: '食べた', reading: 'たべた', count: 1 },
|
||||
],
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.sessionsMerged, 1);
|
||||
assert.equal(summary.animeAdded, 1);
|
||||
assert.equal(summary.videosAdded, 1);
|
||||
assert.equal(summary.wordsAdded, 1);
|
||||
assert.equal(summary.subtitleLinesAdded, 2);
|
||||
assert.equal(summary.telemetryRowsAdded, 1);
|
||||
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 2);
|
||||
const global = queryOne<{
|
||||
total_sessions: number;
|
||||
total_active_ms: number;
|
||||
total_cards: number;
|
||||
active_days: number;
|
||||
episodes_started: number;
|
||||
}>(
|
||||
localPath,
|
||||
'SELECT total_sessions, total_active_ms, total_cards, active_days, episodes_started FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.total_active_ms, 1_200_000 + 900_000);
|
||||
assert.equal(global?.total_cards, 2 + 3);
|
||||
assert.equal(global?.active_days, 2);
|
||||
assert.equal(global?.episodes_started, 2);
|
||||
|
||||
// Existing word: local 2 + merged 5; new word carries remote frequency.
|
||||
const sharedWord = queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '見た'`,
|
||||
);
|
||||
assert.equal(sharedWord?.frequency, 7);
|
||||
const newWord = queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '食べた'`,
|
||||
);
|
||||
assert.equal(newWord?.frequency, 1);
|
||||
|
||||
// The merged session's rollup group was recomputed.
|
||||
assert.equal(summary.rollupGroupsRecomputed, 1);
|
||||
const mergedVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'showb-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
assert.equal(
|
||||
count(localPath, 'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE video_id = ?', [
|
||||
mergedVideoId,
|
||||
]),
|
||||
1,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('is idempotent: re-merging the same snapshot changes nothing', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 4 }],
|
||||
});
|
||||
|
||||
mergeSnapshotIntoDb(localPath, remotePath);
|
||||
const globalAfterFirst = queryOne<Record<string, unknown>>(
|
||||
localPath,
|
||||
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
|
||||
assert.equal(summary.sessionsMerged, 0);
|
||||
assert.equal(summary.sessionsAlreadyPresent, 1);
|
||||
assert.equal(summary.wordsAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 1);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_subtitle_lines'), 1);
|
||||
const globalAfterSecond = queryOne<Record<string, unknown>>(
|
||||
localPath,
|
||||
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.deepEqual(
|
||||
{ ...globalAfterSecond, LAST_UPDATE_DATE: null },
|
||||
{ ...globalAfterFirst, LAST_UPDATE_DATE: null },
|
||||
);
|
||||
assert.equal(
|
||||
Number(
|
||||
queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '見た'`,
|
||||
)?.frequency,
|
||||
),
|
||||
4,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('preserves anilist_id when inserting a new anime from the snapshot', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
animeTitleKey: 'showb',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 12345 WHERE normalized_title_key = 'showb'`)
|
||||
.run();
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.animeAdded, 1);
|
||||
const anime = queryOne<{ anilist_id: number }>(
|
||||
localPath,
|
||||
`SELECT anilist_id FROM imm_anime WHERE normalized_title_key = 'showb'`,
|
||||
);
|
||||
assert.equal(anime?.anilist_id, 12345);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('matches an existing local anime by anilist_id even when the title key differs', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'show-romaji',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e2',
|
||||
animeTitleKey: 'show-native',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(localPath, (localDb) => {
|
||||
localDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-romaji'`)
|
||||
.run();
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-native'`)
|
||||
.run();
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
// Same anilist_id → one anime, not two.
|
||||
assert.equal(summary.animeAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_anime'), 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('matches shared videos by video_key and merges the watched flag', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
watched: true,
|
||||
applyLifetime: true,
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.videosAdded, 0);
|
||||
assert.equal(summary.animeAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_videos'), 1);
|
||||
const video = queryOne<{ watched: number }>(
|
||||
localPath,
|
||||
`SELECT watched FROM imm_videos WHERE video_key = 'showa-e1'`,
|
||||
);
|
||||
assert.equal(video?.watched, 1);
|
||||
|
||||
// Both sessions now credit the same video; episode was started once and
|
||||
// completed once (by the remote session that watched it to the end).
|
||||
const global = queryOne<{
|
||||
episodes_started: number;
|
||||
episodes_completed: number;
|
||||
total_sessions: number;
|
||||
}>(
|
||||
localPath,
|
||||
'SELECT episodes_started, episodes_completed, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.episodes_started, 1);
|
||||
assert.equal(global?.episodes_completed, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('backfills media metadata for matched videos when local metadata is absent', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const remoteDb = new Database(remotePath, { readwrite: true });
|
||||
try {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_media_art (video_id, anilist_id, cover_url, title_romaji, fetched_at_ms)
|
||||
VALUES (?, 123, 'https://example.test/cover.jpg', 'Show A', ?)`,
|
||||
)
|
||||
.run(remoteVideoId, String(BASE_MS));
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_youtube_videos (video_id, youtube_video_id, video_url, video_title, fetched_at_ms)
|
||||
VALUES (?, 'yt-1', 'https://youtube.test/watch?v=yt-1', 'Remote Video', ?)`,
|
||||
)
|
||||
.run(remoteVideoId, String(BASE_MS));
|
||||
} finally {
|
||||
remoteDb.close();
|
||||
}
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.videosAdded, 0);
|
||||
const localVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
const art = queryOne<{ cover_url: string; title_romaji: string }>(
|
||||
localPath,
|
||||
'SELECT cover_url, title_romaji FROM imm_media_art WHERE video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(art?.cover_url, 'https://example.test/cover.jpg');
|
||||
assert.equal(art?.title_romaji, 'Show A');
|
||||
const youtube = queryOne<{ youtube_video_id: string; video_title: string }>(
|
||||
localPath,
|
||||
'SELECT youtube_video_id, video_title FROM imm_youtube_videos WHERE video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(youtube?.youtube_video_id, 'yt-1');
|
||||
assert.equal(youtube?.video_title, 'Remote Video');
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('does not double-count active_days when a merged session starts earlier on an already-credited day', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS + 3_600_000,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
|
||||
mergeSnapshotIntoDb(localPath, remotePath);
|
||||
const global = queryOne<{ active_days: number; total_sessions: number }>(
|
||||
localPath,
|
||||
'SELECT active_days, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.active_days, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('skips unfinished sessions', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-active',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
endedAtMs: null,
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.sessionsMerged, 0);
|
||||
assert.equal(summary.activeSessionsSkipped, 1);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 0);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('copies remote-only historical rollups but never sums shared groups', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
// Remote has a video plus an old rollup row whose sessions were pruned.
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'old-show-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (10000, ?, 4, 120.5, 400, 3200, 9)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
});
|
||||
|
||||
// Local already has its own rollup row for a shared group.
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'old-show-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const localVideoId = withWritableDb(localPath, (localDb) => {
|
||||
const videoId = Number(
|
||||
localDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
localDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (10000, ?, 2, 60.0, 200, 1600, 4)`,
|
||||
)
|
||||
.run(videoId);
|
||||
return videoId;
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
// Shared group (day 10000) untouched; the remote-only session's own group
|
||||
// was recomputed, not copied.
|
||||
assert.equal(summary.dailyRollupsCopied, 0);
|
||||
const shared = queryOne<{ total_sessions: number }>(
|
||||
localPath,
|
||||
'SELECT total_sessions FROM imm_daily_rollups WHERE rollup_day = 10000 AND video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(shared?.total_sessions, 2);
|
||||
|
||||
// Now a rollup for a video with no local sessions at all gets copied.
|
||||
withWritableDb(remotePath, (remoteDb2) => {
|
||||
const orphanVideoId = Number(
|
||||
remoteDb2
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
|
||||
VALUES ('pruned-show-e1', 'pruned-show-e1', 1, 1, 1440000)`,
|
||||
)
|
||||
.run().lastInsertRowid,
|
||||
);
|
||||
remoteDb2
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (9000, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(orphanVideoId);
|
||||
});
|
||||
|
||||
const summary2 = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary2.dailyRollupsCopied, 1);
|
||||
const copiedVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'pruned-show-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
const copied = queryOne<{ total_sessions: number; total_active_min: number }>(
|
||||
localPath,
|
||||
'SELECT total_sessions, total_active_min FROM imm_daily_rollups WHERE rollup_day = 9000 AND video_id = ?',
|
||||
[copiedVideoId],
|
||||
);
|
||||
assert.equal(copied?.total_sessions, 3);
|
||||
assert.equal(copied?.total_active_min, 90.0);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('does not copy remote-only daily rollups into months with local sessions', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'mixed-month-e1',
|
||||
animeTitleKey: 'mixed-month',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const remoteDb = new Database(remotePath, { readwrite: true });
|
||||
try {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
|
||||
VALUES ('mixed-month-e1', 'mixed-month-e1', 1, 1, 1440000)`,
|
||||
)
|
||||
.run().lastInsertRowid,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (20615, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_monthly_rollups (rollup_month, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (202606, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
} finally {
|
||||
remoteDb.close();
|
||||
}
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.dailyRollupsCopied, 0);
|
||||
assert.equal(summary.monthlyRollupsCopied, 0);
|
||||
const localVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'mixed-month-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
assert.equal(
|
||||
count(
|
||||
localPath,
|
||||
'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE rollup_day = 20615 AND video_id = ?',
|
||||
[localVideoId],
|
||||
),
|
||||
0,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects snapshots at a different schema version', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb.prepare('UPDATE imm_schema_version SET schema_version = 17').run();
|
||||
});
|
||||
|
||||
assert.throws(() => mergeSnapshotIntoDb(localPath, remotePath), /schema version 17/);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('createDbSnapshot produces a mergeable copy', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const snapshotPath = path.join(dir, 'snapshot.sqlite');
|
||||
createDbSnapshot(remotePath, snapshotPath);
|
||||
assert.ok(fs.existsSync(snapshotPath));
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, snapshotPath);
|
||||
assert.equal(summary.sessionsMerged, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,105 @@
|
||||
import fs from 'node:fs';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import {
|
||||
LexiconResolver,
|
||||
mergeAnime,
|
||||
mergeExcludedWords,
|
||||
mergeMediaMetadata,
|
||||
mergeVideos,
|
||||
} from './merge-catalog.js';
|
||||
import { mergeSessions } from './merge-sessions.js';
|
||||
import { copyRemoteOnlyRollups, refreshRollupsForNewSessions } from './merge-rollups.js';
|
||||
import {
|
||||
assertMergeableSchema,
|
||||
createEmptyMergeSummary,
|
||||
type SyncMergeSummary,
|
||||
} from './sync-shared.js';
|
||||
|
||||
export type { SyncMergeSummary } from './sync-shared.js';
|
||||
export { createDbSnapshot, findLiveStatsDaemonPid } from './sync-shared.js';
|
||||
|
||||
/**
|
||||
* Merge a snapshot of another machine's immersion database into the local
|
||||
* one. Insert-only union keyed on natural keys (session_uuid, video_key,
|
||||
* normalized_title_key, word/kanji identity); lifetime and rollup aggregates
|
||||
* are updated incrementally so history older than the session retention
|
||||
* window is preserved on both sides. Idempotent: re-merging the same
|
||||
* snapshot is a no-op.
|
||||
*/
|
||||
export function mergeSnapshotIntoDb(localDbPath: string, snapshotPath: string): SyncMergeSummary {
|
||||
if (!fs.existsSync(localDbPath)) {
|
||||
throw new Error(`Local stats database not found: ${localDbPath}`);
|
||||
}
|
||||
if (!fs.existsSync(snapshotPath)) {
|
||||
throw new Error(`Snapshot database not found: ${snapshotPath}`);
|
||||
}
|
||||
|
||||
const remote = new Database(snapshotPath, { readonly: true });
|
||||
let local: Database;
|
||||
try {
|
||||
local = new Database(localDbPath, { readwrite: true, create: false });
|
||||
} catch (error) {
|
||||
remote.close();
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
assertMergeableSchema(remote, 'Snapshot');
|
||||
assertMergeableSchema(local, 'Local');
|
||||
|
||||
const summary = createEmptyMergeSummary();
|
||||
local.run('PRAGMA foreign_keys = ON');
|
||||
local.run('PRAGMA busy_timeout = 5000');
|
||||
local.run('BEGIN IMMEDIATE');
|
||||
try {
|
||||
const animeIdMap = mergeAnime(local, remote, summary);
|
||||
const { videoIdMap, addedVideoIds } = mergeVideos(local, remote, animeIdMap, summary);
|
||||
mergeMediaMetadata(local, remote, videoIdMap, addedVideoIds);
|
||||
mergeExcludedWords(local, remote, summary);
|
||||
|
||||
const lexicon = new LexiconResolver(local, remote, summary);
|
||||
const { newSessionIds } = mergeSessions(
|
||||
local,
|
||||
remote,
|
||||
videoIdMap,
|
||||
animeIdMap,
|
||||
lexicon,
|
||||
summary,
|
||||
);
|
||||
lexicon.applyFrequencyDeltas();
|
||||
refreshRollupsForNewSessions(local, newSessionIds, summary);
|
||||
copyRemoteOnlyRollups(local, remote, videoIdMap, summary);
|
||||
|
||||
local.run('COMMIT');
|
||||
return summary;
|
||||
} catch (error) {
|
||||
local.run('ROLLBACK');
|
||||
throw error;
|
||||
}
|
||||
} finally {
|
||||
local.close();
|
||||
remote.close();
|
||||
}
|
||||
}
|
||||
|
||||
export function formatMergeSummary(summary: SyncMergeSummary): string {
|
||||
const lines = [
|
||||
`Sessions merged: ${summary.sessionsMerged} (${summary.sessionsAlreadyPresent} already present, ${summary.activeSessionsSkipped} unfinished skipped)`,
|
||||
];
|
||||
const detail: string[] = [];
|
||||
if (summary.animeAdded) detail.push(`${summary.animeAdded} series`);
|
||||
if (summary.videosAdded) detail.push(`${summary.videosAdded} videos`);
|
||||
if (summary.wordsAdded) detail.push(`${summary.wordsAdded} words`);
|
||||
if (summary.kanjiAdded) detail.push(`${summary.kanjiAdded} kanji`);
|
||||
if (summary.subtitleLinesAdded) detail.push(`${summary.subtitleLinesAdded} subtitle lines`);
|
||||
if (summary.excludedWordsAdded) detail.push(`${summary.excludedWordsAdded} excluded words`);
|
||||
if (detail.length > 0) lines.push(`Added: ${detail.join(', ')}`);
|
||||
if (summary.dailyRollupsCopied || summary.monthlyRollupsCopied) {
|
||||
lines.push(
|
||||
`Historical rollups copied: ${summary.dailyRollupsCopied} daily, ${summary.monthlyRollupsCopied} monthly`,
|
||||
);
|
||||
}
|
||||
if (summary.rollupGroupsRecomputed) {
|
||||
lines.push(`Rollup groups recomputed: ${summary.rollupGroupsRecomputed}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
|
||||
import { withReadonlyWalRetry } from '../history-db.js';
|
||||
import { resolveConfigDir } from '../../src/config/path-resolution.js';
|
||||
|
||||
export { SCHEMA_VERSION };
|
||||
|
||||
export interface SyncMergeSummary {
|
||||
sessionsMerged: number;
|
||||
sessionsAlreadyPresent: number;
|
||||
activeSessionsSkipped: number;
|
||||
animeAdded: number;
|
||||
videosAdded: number;
|
||||
wordsAdded: number;
|
||||
kanjiAdded: number;
|
||||
subtitleLinesAdded: number;
|
||||
telemetryRowsAdded: number;
|
||||
eventsAdded: number;
|
||||
excludedWordsAdded: number;
|
||||
dailyRollupsCopied: number;
|
||||
monthlyRollupsCopied: number;
|
||||
rollupGroupsRecomputed: number;
|
||||
}
|
||||
|
||||
export function createEmptyMergeSummary(): SyncMergeSummary {
|
||||
return {
|
||||
sessionsMerged: 0,
|
||||
sessionsAlreadyPresent: 0,
|
||||
activeSessionsSkipped: 0,
|
||||
animeAdded: 0,
|
||||
videosAdded: 0,
|
||||
wordsAdded: 0,
|
||||
kanjiAdded: 0,
|
||||
subtitleLinesAdded: 0,
|
||||
telemetryRowsAdded: 0,
|
||||
eventsAdded: 0,
|
||||
excludedWordsAdded: 0,
|
||||
dailyRollupsCopied: 0,
|
||||
monthlyRollupsCopied: 0,
|
||||
rollupGroupsRecomputed: 0,
|
||||
};
|
||||
}
|
||||
|
||||
export function nowDbTimestamp(): string {
|
||||
return String(Date.now());
|
||||
}
|
||||
|
||||
export function tableExists(db: Database, tableName: string): boolean {
|
||||
return Boolean(
|
||||
db.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?`).get(tableName),
|
||||
);
|
||||
}
|
||||
|
||||
export function readSchemaVersion(db: Database): number | null {
|
||||
if (!tableExists(db, 'imm_schema_version')) return null;
|
||||
const row = db
|
||||
.query<{ schema_version: number }>(
|
||||
'SELECT MAX(schema_version) AS schema_version FROM imm_schema_version',
|
||||
)
|
||||
.get();
|
||||
return typeof row?.schema_version === 'number' ? row.schema_version : null;
|
||||
}
|
||||
|
||||
export function assertMergeableSchema(db: Database, label: string): void {
|
||||
const version = readSchemaVersion(db);
|
||||
if (version === null) {
|
||||
throw new Error(
|
||||
`${label} database has no schema version. Run SubMiner once on that machine so the stats database is initialized.`,
|
||||
);
|
||||
}
|
||||
if (version !== SCHEMA_VERSION) {
|
||||
throw new Error(
|
||||
`${label} database is at schema version ${version} but this launcher expects ${SCHEMA_VERSION}. Update SubMiner on both machines to the same version and run each app once before syncing.`,
|
||||
);
|
||||
}
|
||||
for (const table of ['imm_sessions', 'imm_videos', 'imm_lifetime_global']) {
|
||||
if (!tableExists(db, table)) {
|
||||
throw new Error(`${label} database is missing table ${table}; cannot sync.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function insertRow(
|
||||
db: Database,
|
||||
table: string,
|
||||
columns: readonly string[],
|
||||
values: unknown[],
|
||||
): number {
|
||||
const sql = `INSERT INTO ${table} (${columns.join(', ')}) VALUES (${columns.map(() => '?').join(', ')})`;
|
||||
// db.query() caches the prepared statement per SQL string; this runs once
|
||||
// per copied row, so re-preparing via db.prepare() would dominate merge time.
|
||||
const result = db.query(sql).run(...values);
|
||||
return Number(result.lastInsertRowid);
|
||||
}
|
||||
|
||||
export function createDbSnapshot(dbPath: string, outPath: string): void {
|
||||
if (!fs.existsSync(dbPath)) {
|
||||
throw new Error(`Stats database not found: ${dbPath}`);
|
||||
}
|
||||
fs.rmSync(outPath, { force: true });
|
||||
fs.mkdirSync(path.dirname(outPath), { recursive: true });
|
||||
withReadonlyWalRetry(dbPath, (options) => {
|
||||
const db = new Database(dbPath, options);
|
||||
try {
|
||||
assertMergeableSchema(db, 'Local');
|
||||
db.prepare('VACUUM INTO ?').run(outPath);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
interface DaemonStateFile {
|
||||
pid?: unknown;
|
||||
}
|
||||
|
||||
function isProcessAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
// EPERM means the process exists but we can't signal it → still alive.
|
||||
// Only ESRCH (no such process) means it's actually gone.
|
||||
return (error as NodeJS.ErrnoException)?.code === 'EPERM';
|
||||
}
|
||||
}
|
||||
|
||||
function statsDaemonStateCandidates(dbPath: string): string[] {
|
||||
const homeDir = os.homedir();
|
||||
const candidates = new Set<string>([path.join(path.dirname(dbPath), 'stats-daemon.json')]);
|
||||
const configDir = resolveConfigDir({
|
||||
platform: process.platform,
|
||||
appDataDir: process.env.APPDATA,
|
||||
xdgConfigHome: process.env.XDG_CONFIG_HOME,
|
||||
homeDir,
|
||||
existsSync: fs.existsSync,
|
||||
});
|
||||
candidates.add(path.join(configDir, 'stats-daemon.json'));
|
||||
if (process.platform === 'darwin') {
|
||||
candidates.add(path.join(homeDir, 'Library', 'Application Support', 'SubMiner', 'stats-daemon.json'));
|
||||
}
|
||||
return [...candidates];
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort guard against merging while a SubMiner process holds the
|
||||
* tracker's write queue in memory. Detects the background stats daemon via
|
||||
* its pid state file; the interactive app is caught by the mpv-socket check
|
||||
* in the sync command.
|
||||
*/
|
||||
export function findLiveStatsDaemonPid(dbPath: string): number | null {
|
||||
for (const statePath of statsDaemonStateCandidates(dbPath)) {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = fs.readFileSync(statePath, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as DaemonStateFile;
|
||||
const pid = typeof parsed.pid === 'number' && Number.isInteger(parsed.pid) ? parsed.pid : 0;
|
||||
if (pid > 0 && isProcessAlive(pid)) {
|
||||
return pid;
|
||||
}
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,323 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
|
||||
import { IMMERSION_DB_FIXTURE_DDL } from './immersion-db-schema.js';
|
||||
|
||||
export function createImmersionDbFixture(dbPath: string): void {
|
||||
const db = new Database(dbPath, { create: true });
|
||||
try {
|
||||
db.run('PRAGMA foreign_keys = ON');
|
||||
db.run('PRAGMA journal_mode = WAL');
|
||||
for (const statement of IMMERSION_DB_FIXTURE_DDL.split(';')) {
|
||||
const sql = statement.trim();
|
||||
if (sql) db.run(sql);
|
||||
}
|
||||
db.prepare('INSERT INTO imm_schema_version (schema_version, applied_at_ms) VALUES (?, ?)').run(
|
||||
SCHEMA_VERSION,
|
||||
String(Date.now()),
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0)`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_global(global_id, CREATED_DATE, LAST_UPDATE_DATE) VALUES (1, ?, ?)`,
|
||||
).run(String(Date.now()), String(Date.now()));
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
let uniqueCounter = 0;
|
||||
|
||||
export interface FixtureSessionInput {
|
||||
uuid: string;
|
||||
videoKey: string;
|
||||
animeTitleKey?: string | null;
|
||||
animeEpisodesTotal?: number | null;
|
||||
startedAtMs: number;
|
||||
endedAtMs?: number | null;
|
||||
activeWatchedMs?: number;
|
||||
cardsMined?: number;
|
||||
linesSeen?: number;
|
||||
tokensSeen?: number;
|
||||
watched?: boolean;
|
||||
words?: Array<{ headword: string; word: string; reading: string; count: number }>;
|
||||
applyLifetime?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert a session (with video/anime/subtitle-line/occurrence rows) the way
|
||||
* the app would have recorded it, optionally crediting lifetime aggregates
|
||||
* the way applySessionLifetimeSummary does for locally-recorded sessions.
|
||||
*/
|
||||
export function insertFixtureSession(dbPath: string, input: FixtureSessionInput): void {
|
||||
const db = new Database(dbPath, { readwrite: true });
|
||||
const stamp = String(input.startedAtMs);
|
||||
try {
|
||||
db.run('PRAGMA foreign_keys = ON');
|
||||
let animeId: number | null = null;
|
||||
if (input.animeTitleKey) {
|
||||
const existing = db
|
||||
.query<{
|
||||
anime_id: number;
|
||||
}>('SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?')
|
||||
.get(input.animeTitleKey);
|
||||
animeId = existing
|
||||
? existing.anime_id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_anime (normalized_title_key, canonical_title, episodes_total, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.animeTitleKey,
|
||||
input.animeTitleKey,
|
||||
input.animeEpisodesTotal ?? null,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
}
|
||||
|
||||
const existingVideo = db
|
||||
.query<{ video_id: number }>('SELECT video_id FROM imm_videos WHERE video_key = ?')
|
||||
.get(input.videoKey);
|
||||
const videoId = existingVideo
|
||||
? existingVideo.video_id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, anime_id, canonical_title, source_type, source_path, watched, duration_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, 1, ?, ?, 1440000, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.videoKey,
|
||||
animeId,
|
||||
input.videoKey,
|
||||
`/videos/${input.videoKey}.mkv`,
|
||||
input.watched ? 1 : 0,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
if (input.watched) {
|
||||
db.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?').run(videoId);
|
||||
}
|
||||
|
||||
const endedAtMs =
|
||||
input.endedAtMs === undefined ? input.startedAtMs + 1_500_000 : input.endedAtMs;
|
||||
const activeWatchedMs = input.activeWatchedMs ?? 1_200_000;
|
||||
const cardsMined = input.cardsMined ?? 2;
|
||||
const linesSeen = input.linesSeen ?? 100;
|
||||
const tokensSeen = input.tokensSeen ?? 800;
|
||||
const sessionId = Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_sessions (
|
||||
session_uuid, video_id, started_at_ms, ended_at_ms, status,
|
||||
total_watched_ms, active_watched_ms, lines_seen, tokens_seen, cards_mined,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.uuid,
|
||||
videoId,
|
||||
String(input.startedAtMs),
|
||||
endedAtMs === null ? null : String(endedAtMs),
|
||||
endedAtMs === null ? 1 : 2,
|
||||
activeWatchedMs,
|
||||
activeWatchedMs,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
cardsMined,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO imm_session_telemetry (
|
||||
session_id, sample_ms, total_watched_ms, active_watched_ms, lines_seen, tokens_seen,
|
||||
cards_mined, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
sessionId,
|
||||
String(endedAtMs ?? input.startedAtMs),
|
||||
activeWatchedMs,
|
||||
activeWatchedMs,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
cardsMined,
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
|
||||
for (const word of input.words ?? []) {
|
||||
const existing = db
|
||||
.query<{
|
||||
id: number;
|
||||
}>('SELECT id FROM imm_words WHERE headword = ? AND word = ? AND reading = ?')
|
||||
.get(word.headword, word.word, word.reading);
|
||||
const wordId = existing
|
||||
? existing.id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_words (headword, word, reading, first_seen, last_seen, frequency)
|
||||
VALUES (?, ?, ?, ?, ?, 0)`,
|
||||
)
|
||||
.run(
|
||||
word.headword,
|
||||
word.word,
|
||||
word.reading,
|
||||
Math.floor(input.startedAtMs / 1000),
|
||||
Math.floor(input.startedAtMs / 1000),
|
||||
).lastInsertRowid,
|
||||
);
|
||||
uniqueCounter += 1;
|
||||
const lineId = Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_subtitle_lines (session_id, video_id, anime_id, line_index, text, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
sessionId,
|
||||
videoId,
|
||||
animeId,
|
||||
uniqueCounter,
|
||||
`line ${word.word}`,
|
||||
input.startedAtMs,
|
||||
input.startedAtMs,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
db.prepare(
|
||||
'INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)',
|
||||
).run(lineId, wordId, word.count);
|
||||
db.prepare('UPDATE imm_words SET frequency = frequency + ? WHERE id = ?').run(
|
||||
word.count,
|
||||
wordId,
|
||||
);
|
||||
}
|
||||
|
||||
if (input.applyLifetime && endedAtMs !== null) {
|
||||
applyFixtureLifetime(db, sessionId, videoId, animeId, {
|
||||
endedAtMs,
|
||||
startedAtMs: input.startedAtMs,
|
||||
activeWatchedMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
watched: Boolean(input.watched),
|
||||
episodesTotal: input.animeEpisodesTotal ?? null,
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function applyFixtureLifetime(
|
||||
db: Database,
|
||||
sessionId: number,
|
||||
videoId: number,
|
||||
animeId: number | null,
|
||||
data: {
|
||||
endedAtMs: number;
|
||||
startedAtMs: number;
|
||||
activeWatchedMs: number;
|
||||
cardsMined: number;
|
||||
linesSeen: number;
|
||||
tokensSeen: number;
|
||||
watched: boolean;
|
||||
episodesTotal: number | null;
|
||||
},
|
||||
): void {
|
||||
const stamp = String(data.endedAtMs);
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)`,
|
||||
).run(sessionId, String(data.endedAtMs), stamp, stamp);
|
||||
|
||||
const mediaLifetime = db
|
||||
.query<{ completed: number }>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const isFirstSessionForVideo = !mediaLifetime;
|
||||
const isFirstCompleted = data.watched && Number(mediaLifetime?.completed ?? 0) <= 0;
|
||||
const dayExpr = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
|
||||
const otherOnDay = db
|
||||
.query(
|
||||
`SELECT 1 FROM imm_sessions WHERE session_id != ? AND ${dayExpr} =
|
||||
CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER) LIMIT 1`,
|
||||
)
|
||||
.get(sessionId, String(data.startedAtMs));
|
||||
|
||||
db.prepare(
|
||||
`UPDATE imm_lifetime_global SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + ?,
|
||||
total_cards = total_cards + ?,
|
||||
active_days = active_days + ?,
|
||||
episodes_started = episodes_started + ?,
|
||||
episodes_completed = episodes_completed + ?
|
||||
WHERE global_id = 1`,
|
||||
).run(
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
otherOnDay ? 0 : 1,
|
||||
isFirstSessionForVideo ? 1 : 0,
|
||||
isFirstCompleted ? 1 : 0,
|
||||
);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_media (video_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(video_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
completed = MAX(completed, excluded.completed),
|
||||
last_watched_ms = excluded.last_watched_ms`,
|
||||
).run(
|
||||
videoId,
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
data.linesSeen,
|
||||
data.tokensSeen,
|
||||
data.watched ? 1 : 0,
|
||||
String(data.startedAtMs),
|
||||
String(data.endedAtMs),
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
|
||||
if (animeId !== null) {
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_anime (anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, episodes_started, episodes_completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(anime_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
episodes_started = episodes_started + excluded.episodes_started,
|
||||
episodes_completed = episodes_completed + excluded.episodes_completed,
|
||||
last_watched_ms = excluded.last_watched_ms`,
|
||||
).run(
|
||||
animeId,
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
data.linesSeen,
|
||||
data.tokensSeen,
|
||||
isFirstSessionForVideo ? 1 : 0,
|
||||
isFirstCompleted ? 1 : 0,
|
||||
String(data.startedAtMs),
|
||||
String(data.endedAtMs),
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database as BunDatabase } from 'bun:sqlite';
|
||||
import { ensureSchema } from '../../src/core/services/immersion-tracker/storage.js';
|
||||
import { createImmersionDbFixture, insertFixtureSession } from './immersion-db-fixture.js';
|
||||
|
||||
type SchemaRow = { type: string; name: string; tbl_name: string; sql: string | null };
|
||||
|
||||
const SYNC_SCHEMA_OBJECTS = [
|
||||
'imm_anime',
|
||||
'imm_videos',
|
||||
'imm_sessions',
|
||||
'imm_session_telemetry',
|
||||
'imm_session_events',
|
||||
'imm_daily_rollups',
|
||||
'imm_monthly_rollups',
|
||||
'imm_words',
|
||||
'imm_kanji',
|
||||
'imm_subtitle_lines',
|
||||
'imm_word_line_occurrences',
|
||||
'imm_kanji_line_occurrences',
|
||||
'imm_media_art',
|
||||
'imm_youtube_videos',
|
||||
'imm_cover_art_blobs',
|
||||
'imm_lifetime_global',
|
||||
'imm_lifetime_anime',
|
||||
'imm_lifetime_media',
|
||||
'imm_lifetime_applied_sessions',
|
||||
'imm_stats_excluded_words',
|
||||
'idx_anime_normalized_title',
|
||||
'idx_anime_anilist_id',
|
||||
'idx_videos_anime_id',
|
||||
'idx_sessions_video_started',
|
||||
'idx_sessions_status_started',
|
||||
'idx_sessions_started_at',
|
||||
'idx_sessions_ended_at',
|
||||
'idx_telemetry_session_sample',
|
||||
'idx_telemetry_sample_ms',
|
||||
'idx_events_session_ts',
|
||||
'idx_events_type_ts',
|
||||
'idx_rollups_day_video',
|
||||
'idx_rollups_month_video',
|
||||
'idx_words_headword_word_reading',
|
||||
'idx_words_frequency',
|
||||
'idx_kanji_kanji',
|
||||
'idx_kanji_frequency',
|
||||
'idx_subtitle_lines_session_line',
|
||||
'idx_subtitle_lines_video_line',
|
||||
'idx_subtitle_lines_anime_line',
|
||||
'idx_word_line_occurrences_word',
|
||||
'idx_kanji_line_occurrences_kanji',
|
||||
'idx_media_art_cover_blob_hash',
|
||||
'idx_media_art_anilist_id',
|
||||
'idx_media_art_cover_url',
|
||||
'idx_youtube_videos_channel_id',
|
||||
'idx_youtube_videos_youtube_video_id',
|
||||
] as const;
|
||||
|
||||
function normalizeSql(sql: string | null): string {
|
||||
return (sql ?? '')
|
||||
.replace(/\bIF NOT EXISTS\b/gi, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.replace(/\s+([(),])/g, '$1')
|
||||
.replace(/,\s+/g, ', ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function readSchema(dbPath: string): Map<string, string> {
|
||||
const db = new BunDatabase(dbPath, { readonly: true });
|
||||
try {
|
||||
const rows = db
|
||||
.query<SchemaRow>(
|
||||
`SELECT type, name, tbl_name, sql
|
||||
FROM sqlite_schema
|
||||
WHERE name IN (${SYNC_SCHEMA_OBJECTS.map(() => '?').join(',')})
|
||||
ORDER BY type, name`,
|
||||
)
|
||||
.all(...SYNC_SCHEMA_OBJECTS);
|
||||
return new Map(rows.map((row) => [row.name, normalizeSql(row.sql)]));
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
test('fixture schema stays aligned with production sync-touched tables and indexes', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-schema-'));
|
||||
const fixturePath = path.join(dir, 'fixture.sqlite');
|
||||
const productionPath = path.join(dir, 'production.sqlite');
|
||||
const productionDb = new BunDatabase(productionPath, { create: true });
|
||||
try {
|
||||
createImmersionDbFixture(fixturePath);
|
||||
ensureSchema(productionDb as never);
|
||||
productionDb.close();
|
||||
|
||||
const fixtureSchema = readSchema(fixturePath);
|
||||
const productionSchema = readSchema(productionPath);
|
||||
assert.deepEqual(fixtureSchema, productionSchema);
|
||||
} finally {
|
||||
try {
|
||||
productionDb.close();
|
||||
} catch {
|
||||
// already closed
|
||||
}
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('fixture session inserts enforce foreign keys', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-foreign-keys-'));
|
||||
const fixturePath = path.join(dir, 'fixture.sqlite');
|
||||
try {
|
||||
createImmersionDbFixture(fixturePath);
|
||||
const db = new BunDatabase(fixturePath, { readwrite: true });
|
||||
try {
|
||||
db.run(`
|
||||
CREATE TRIGGER remove_fixture_video
|
||||
BEFORE INSERT ON imm_sessions
|
||||
BEGIN
|
||||
DELETE FROM imm_videos WHERE video_id = NEW.video_id;
|
||||
END
|
||||
`);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
|
||||
assert.throws(
|
||||
() =>
|
||||
insertFixtureSession(fixturePath, {
|
||||
uuid: 'foreign-key-check',
|
||||
videoKey: 'foreign-key-check',
|
||||
startedAtMs: Date.UTC(2026, 6, 9),
|
||||
}),
|
||||
/FOREIGN KEY constraint failed/,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,323 @@
|
||||
// Schema-version-18 shape of the tables the sync merge touches (plus the
|
||||
// app's indexes), mirroring ensureSchema / ensureLifetimeSummaryTables /
|
||||
// ensureStatsExcludedWordsTable in src/core/services/immersion-tracker/storage.ts.
|
||||
export const IMMERSION_DB_FIXTURE_DDL = `
|
||||
CREATE TABLE imm_schema_version (
|
||||
schema_version INTEGER PRIMARY KEY,
|
||||
applied_at_ms TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE imm_rollup_state(
|
||||
state_key TEXT PRIMARY KEY,
|
||||
state_value TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE imm_anime(
|
||||
anime_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
normalized_title_key TEXT NOT NULL UNIQUE,
|
||||
canonical_title TEXT NOT NULL,
|
||||
anilist_id INTEGER UNIQUE,
|
||||
title_romaji TEXT,
|
||||
title_english TEXT,
|
||||
title_native TEXT,
|
||||
episodes_total INTEGER,
|
||||
description TEXT,
|
||||
metadata_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_videos(
|
||||
video_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
video_key TEXT NOT NULL UNIQUE,
|
||||
anime_id INTEGER,
|
||||
canonical_title TEXT NOT NULL,
|
||||
source_type INTEGER NOT NULL,
|
||||
source_path TEXT,
|
||||
source_url TEXT,
|
||||
parsed_basename TEXT,
|
||||
parsed_title TEXT,
|
||||
parsed_season INTEGER,
|
||||
parsed_episode INTEGER,
|
||||
parser_source TEXT,
|
||||
parser_confidence REAL,
|
||||
parse_metadata_json TEXT,
|
||||
watched INTEGER NOT NULL DEFAULT 0,
|
||||
duration_ms INTEGER NOT NULL CHECK(duration_ms>=0),
|
||||
file_size_bytes INTEGER CHECK(file_size_bytes>=0),
|
||||
codec_id INTEGER, container_id INTEGER,
|
||||
width_px INTEGER, height_px INTEGER, fps_x100 INTEGER,
|
||||
bitrate_kbps INTEGER, audio_codec_id INTEGER,
|
||||
hash_sha256 TEXT, screenshot_path TEXT,
|
||||
metadata_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
|
||||
);
|
||||
CREATE TABLE imm_sessions(
|
||||
session_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_uuid TEXT NOT NULL UNIQUE,
|
||||
video_id INTEGER NOT NULL,
|
||||
started_at_ms TEXT NOT NULL, ended_at_ms TEXT,
|
||||
status INTEGER NOT NULL,
|
||||
locale_id INTEGER, target_lang_id INTEGER,
|
||||
difficulty_tier INTEGER, subtitle_mode INTEGER,
|
||||
ended_media_ms INTEGER,
|
||||
total_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
active_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
cards_mined INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_hits INTEGER NOT NULL DEFAULT 0,
|
||||
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_ms INTEGER NOT NULL DEFAULT 0,
|
||||
seek_forward_count INTEGER NOT NULL DEFAULT 0,
|
||||
seek_backward_count INTEGER NOT NULL DEFAULT 0,
|
||||
media_buffer_events INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id)
|
||||
);
|
||||
CREATE TABLE imm_session_telemetry(
|
||||
telemetry_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
sample_ms TEXT NOT NULL,
|
||||
total_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
active_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
cards_mined INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_hits INTEGER NOT NULL DEFAULT 0,
|
||||
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_ms INTEGER NOT NULL DEFAULT 0,
|
||||
seek_forward_count INTEGER NOT NULL DEFAULT 0,
|
||||
seek_backward_count INTEGER NOT NULL DEFAULT 0,
|
||||
media_buffer_events INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_session_events(
|
||||
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
ts_ms TEXT NOT NULL,
|
||||
event_type INTEGER NOT NULL,
|
||||
line_index INTEGER,
|
||||
segment_start_ms INTEGER,
|
||||
segment_end_ms INTEGER,
|
||||
tokens_delta INTEGER NOT NULL DEFAULT 0,
|
||||
cards_delta INTEGER NOT NULL DEFAULT 0,
|
||||
payload_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_daily_rollups(
|
||||
rollup_day INTEGER NOT NULL,
|
||||
video_id INTEGER,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_min REAL NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
cards_per_hour REAL,
|
||||
tokens_per_min REAL,
|
||||
lookup_hit_rate REAL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY (rollup_day, video_id)
|
||||
);
|
||||
CREATE TABLE imm_monthly_rollups(
|
||||
rollup_month INTEGER NOT NULL,
|
||||
video_id INTEGER,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_min REAL NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY (rollup_month, video_id)
|
||||
);
|
||||
CREATE TABLE imm_words(
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
headword TEXT,
|
||||
word TEXT,
|
||||
reading TEXT,
|
||||
part_of_speech TEXT,
|
||||
pos1 TEXT,
|
||||
pos2 TEXT,
|
||||
pos3 TEXT,
|
||||
first_seen REAL,
|
||||
last_seen REAL,
|
||||
frequency INTEGER,
|
||||
frequency_rank INTEGER,
|
||||
UNIQUE(headword, word, reading)
|
||||
);
|
||||
CREATE TABLE imm_kanji(
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
kanji TEXT,
|
||||
first_seen REAL,
|
||||
last_seen REAL,
|
||||
frequency INTEGER,
|
||||
UNIQUE(kanji)
|
||||
);
|
||||
CREATE TABLE imm_subtitle_lines(
|
||||
line_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
event_id INTEGER,
|
||||
video_id INTEGER NOT NULL,
|
||||
anime_id INTEGER,
|
||||
line_index INTEGER NOT NULL,
|
||||
segment_start_ms INTEGER,
|
||||
segment_end_ms INTEGER,
|
||||
text TEXT NOT NULL,
|
||||
secondary_text TEXT,
|
||||
CREATED_DATE INTEGER,
|
||||
LAST_UPDATE_DATE INTEGER,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(event_id) REFERENCES imm_session_events(event_id) ON DELETE SET NULL,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
|
||||
);
|
||||
CREATE TABLE imm_word_line_occurrences(
|
||||
line_id INTEGER NOT NULL,
|
||||
word_id INTEGER NOT NULL,
|
||||
occurrence_count INTEGER NOT NULL,
|
||||
PRIMARY KEY(line_id, word_id),
|
||||
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(word_id) REFERENCES imm_words(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_kanji_line_occurrences(
|
||||
line_id INTEGER NOT NULL,
|
||||
kanji_id INTEGER NOT NULL,
|
||||
occurrence_count INTEGER NOT NULL,
|
||||
PRIMARY KEY(line_id, kanji_id),
|
||||
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(kanji_id) REFERENCES imm_kanji(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_media_art(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
anilist_id INTEGER,
|
||||
cover_url TEXT,
|
||||
cover_blob BLOB,
|
||||
cover_blob_hash TEXT,
|
||||
title_romaji TEXT,
|
||||
title_english TEXT,
|
||||
episodes_total INTEGER,
|
||||
fetched_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_youtube_videos(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
youtube_video_id TEXT NOT NULL,
|
||||
video_url TEXT NOT NULL,
|
||||
video_title TEXT,
|
||||
video_thumbnail_url TEXT,
|
||||
channel_id TEXT,
|
||||
channel_name TEXT,
|
||||
channel_url TEXT,
|
||||
channel_thumbnail_url TEXT,
|
||||
uploader_id TEXT,
|
||||
uploader_url TEXT,
|
||||
description TEXT,
|
||||
metadata_json TEXT,
|
||||
fetched_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_cover_art_blobs(
|
||||
blob_hash TEXT PRIMARY KEY,
|
||||
cover_blob BLOB NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_lifetime_global(
|
||||
global_id INTEGER PRIMARY KEY CHECK(global_id = 1),
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
active_days INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_started INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_completed INTEGER NOT NULL DEFAULT 0,
|
||||
anime_completed INTEGER NOT NULL DEFAULT 0,
|
||||
last_rebuilt_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_lifetime_anime(
|
||||
anime_id INTEGER PRIMARY KEY,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_started INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_completed INTEGER NOT NULL DEFAULT 0,
|
||||
first_watched_ms TEXT,
|
||||
last_watched_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_lifetime_media(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
completed INTEGER NOT NULL DEFAULT 0,
|
||||
first_watched_ms TEXT,
|
||||
last_watched_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_lifetime_applied_sessions(
|
||||
session_id INTEGER PRIMARY KEY,
|
||||
applied_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_stats_excluded_words(
|
||||
headword TEXT NOT NULL,
|
||||
word TEXT NOT NULL,
|
||||
reading TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY(headword, word, reading)
|
||||
);
|
||||
CREATE INDEX idx_anime_normalized_title ON imm_anime(normalized_title_key);
|
||||
CREATE INDEX idx_anime_anilist_id ON imm_anime(anilist_id);
|
||||
CREATE INDEX idx_videos_anime_id ON imm_videos(anime_id);
|
||||
CREATE INDEX idx_sessions_video_started ON imm_sessions(video_id, started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_status_started ON imm_sessions(status, started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_started_at ON imm_sessions(started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_ended_at ON imm_sessions(ended_at_ms DESC);
|
||||
CREATE INDEX idx_telemetry_session_sample ON imm_session_telemetry(session_id, sample_ms DESC);
|
||||
CREATE INDEX idx_telemetry_sample_ms ON imm_session_telemetry(sample_ms DESC);
|
||||
CREATE INDEX idx_events_session_ts ON imm_session_events(session_id, ts_ms DESC);
|
||||
CREATE INDEX idx_events_type_ts ON imm_session_events(event_type, ts_ms DESC);
|
||||
CREATE INDEX idx_rollups_day_video ON imm_daily_rollups(rollup_day, video_id);
|
||||
CREATE INDEX idx_rollups_month_video ON imm_monthly_rollups(rollup_month, video_id);
|
||||
CREATE INDEX idx_words_headword_word_reading ON imm_words(headword, word, reading);
|
||||
CREATE INDEX idx_words_frequency ON imm_words(frequency DESC);
|
||||
CREATE INDEX idx_kanji_kanji ON imm_kanji(kanji);
|
||||
CREATE INDEX idx_kanji_frequency ON imm_kanji(frequency DESC);
|
||||
CREATE INDEX idx_subtitle_lines_session_line ON imm_subtitle_lines(session_id, line_index);
|
||||
CREATE INDEX idx_subtitle_lines_video_line ON imm_subtitle_lines(video_id, line_index);
|
||||
CREATE INDEX idx_subtitle_lines_anime_line ON imm_subtitle_lines(anime_id, line_index);
|
||||
CREATE INDEX idx_word_line_occurrences_word ON imm_word_line_occurrences(word_id, line_id);
|
||||
CREATE INDEX idx_kanji_line_occurrences_kanji ON imm_kanji_line_occurrences(kanji_id, line_id);
|
||||
CREATE INDEX idx_media_art_cover_blob_hash ON imm_media_art(cover_blob_hash);
|
||||
CREATE INDEX idx_media_art_anilist_id ON imm_media_art(anilist_id);
|
||||
CREATE INDEX idx_media_art_cover_url ON imm_media_art(cover_url);
|
||||
CREATE INDEX idx_youtube_videos_channel_id ON imm_youtube_videos(channel_id);
|
||||
CREATE INDEX idx_youtube_videos_youtube_video_id ON imm_youtube_videos(youtube_video_id);
|
||||
`;
|
||||
@@ -113,6 +113,14 @@ export interface Args {
|
||||
texthookerOpenBrowser: boolean;
|
||||
useRofi: boolean;
|
||||
history: boolean;
|
||||
sync: boolean;
|
||||
syncHost: string;
|
||||
syncSnapshotPath: string;
|
||||
syncMergePath: string;
|
||||
syncDirection: 'both' | 'push' | 'pull';
|
||||
syncRemoteCmd: string;
|
||||
syncDbPath: string;
|
||||
syncForce: boolean;
|
||||
logLevel: LogLevel;
|
||||
logRotation: LogRotation;
|
||||
passwordStore: string;
|
||||
|
||||
@@ -13,6 +13,8 @@
|
||||
"get-frequency:electron": "bun run build:yomitan && bun build scripts/get_frequency.ts --format=cjs --target=node --outfile dist/scripts/get_frequency.js --external electron && env -u ELECTRON_RUN_AS_NODE electron dist/scripts/get_frequency.js --pretty --color-top-x 10000 --yomitan-user-data ~/.config/SubMiner --colorized-line",
|
||||
"test-yomitan-parser": "bun run scripts/test-yomitan-parser.ts",
|
||||
"test-yomitan-parser:electron": "bun run build:yomitan && bun build scripts/test-yomitan-parser.ts --format=cjs --target=node --outfile dist/scripts/test-yomitan-parser.js --external electron && env -u ELECTRON_RUN_AS_NODE electron dist/scripts/test-yomitan-parser.js",
|
||||
"record-tokenizer-fixture:electron": "bun run build:yomitan && bun build scripts/record-tokenizer-fixture.ts --format=cjs --target=node --outfile dist/scripts/record-tokenizer-fixture.js --external electron && env -u ELECTRON_RUN_AS_NODE electron dist/scripts/record-tokenizer-fixture.js",
|
||||
"compare-yomitan-api:electron": "bun run build:yomitan && bun build scripts/compare-yomitan-api.ts --format=cjs --target=node --outfile dist/scripts/compare-yomitan-api.js --external electron && env -u ELECTRON_RUN_AS_NODE electron dist/scripts/compare-yomitan-api.js",
|
||||
"build:yomitan": "bun scripts/build-yomitan.mjs",
|
||||
"build:assets": "bun scripts/prepare-build-assets.mjs",
|
||||
"build:launcher": "bun build ./launcher/main.ts --target=bun --packages=bundle --banner='#!/usr/bin/env bun' --outfile=dist/launcher/subminer",
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import test from 'node:test';
|
||||
|
||||
const source = readFileSync('scripts/compare-yomitan-api.ts', 'utf8');
|
||||
|
||||
test('creates the configured Yomitan user-data directory before Electron uses it', () => {
|
||||
const setPathIndex = source.indexOf(
|
||||
"electronModule.app.setPath('userData', options.yomitanUserDataPath)",
|
||||
);
|
||||
assert.notEqual(setPathIndex, -1);
|
||||
|
||||
const mkdirIndex = source.lastIndexOf(
|
||||
'fs.mkdirSync(options.yomitanUserDataPath, { recursive: true })',
|
||||
setPathIndex,
|
||||
);
|
||||
assert.ok(mkdirIndex > -1, 'must create the user-data directory');
|
||||
assert.ok(mkdirIndex < setPathIndex, 'must create the directory before app.setPath');
|
||||
});
|
||||
|
||||
test('lets output drain by using exitCode in the top-level completion handlers', () => {
|
||||
const completionHandlers = source.slice(source.lastIndexOf('\nmain()'));
|
||||
assert.match(completionHandlers, /process\.exitCode = process\.exitCode \?\? 0/);
|
||||
assert.match(completionHandlers, /process\.exitCode = 1/);
|
||||
assert.doesNotMatch(completionHandlers, /process\.exit\(/);
|
||||
});
|
||||
@@ -0,0 +1,589 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import process from 'node:process';
|
||||
|
||||
import { createTokenizerDepsRuntime, tokenizeSubtitle } from '../src/core/services/tokenizer.js';
|
||||
import { selectYomitanParseTokens } from '../src/core/services/tokenizer/parser-selection-stage.js';
|
||||
import { enrichTokensWithMecabPos1 } from '../src/core/services/tokenizer/parser-enrichment-stage.js';
|
||||
import { resolveYomitanExtensionPath as resolveBuiltYomitanExtensionPath } from '../src/core/services/yomitan-extension-paths.js';
|
||||
import { MecabTokenizer } from '../src/mecab-tokenizer.js';
|
||||
import type { MergedToken } from '../src/types.js';
|
||||
|
||||
import { fetchYomitanApi } from './yomitan-api-request.js';
|
||||
|
||||
// Compares SubMiner's full tokenization pipeline against a stock Yomitan
|
||||
// instance reached through the yomitan-api native-messaging bridge
|
||||
// (https://github.com/yomidevs/yomitan-api, default http://127.0.0.1:19633).
|
||||
// The API response is raw parseText output; it is mapped through SubMiner's
|
||||
// own selectYomitanParseTokens so both sides share identical mapping
|
||||
// semantics and the diff isolates real segmentation/headword divergence.
|
||||
|
||||
const DEFAULT_YOMITAN_USER_DATA_PATH = path.join(os.homedir(), '.config', 'SubMiner');
|
||||
const DEFAULT_API_URL = 'http://127.0.0.1:19633';
|
||||
const DEFAULT_SCAN_LENGTH = 40;
|
||||
const FIXTURE_DIR = path.join('src', 'core', 'services', 'tokenizer', '__fixtures__', 'golden');
|
||||
|
||||
interface CliOptions {
|
||||
sentences: string[];
|
||||
fromFixtures: boolean;
|
||||
apiUrl: string;
|
||||
scanLength: number;
|
||||
emitJson: boolean;
|
||||
nameMatch: boolean;
|
||||
yomitanExtensionPath?: string;
|
||||
yomitanUserDataPath: string;
|
||||
mecabCommand?: string;
|
||||
mecabDictionaryPath?: string;
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
process.stdout.write(`Usage:
|
||||
bun run compare-yomitan-api:electron -- [options] [sentence ...]
|
||||
|
||||
Compares SubMiner tokenization against a stock Yomitan instance via the
|
||||
yomitan-api bridge. Without sentences, compares every golden-corpus fixture
|
||||
text plus any --file lines.
|
||||
|
||||
--from-fixtures Include golden-corpus fixture texts (default when
|
||||
no sentences are given).
|
||||
--file <path> Read additional sentences, one per line.
|
||||
--api-url <url> yomitan-api base URL (default: ${DEFAULT_API_URL}).
|
||||
--scan-length <n> Scan length for the API request (default: ${DEFAULT_SCAN_LENGTH}).
|
||||
--name-match Keep SubMiner's character-name pre-pass enabled.
|
||||
Off by default: stock Yomitan has no name regions,
|
||||
so it would only add noise to the diff.
|
||||
--json Emit machine-readable JSON instead of a report.
|
||||
--yomitan-extension <path> Path to built Yomitan extension directory.
|
||||
--yomitan-user-data <path> Electron userData directory (default: ~/.config/SubMiner).
|
||||
--mecab-command <path> MeCab binary (default: mecab).
|
||||
--mecab-dictionary <path> MeCab dictionary directory.
|
||||
-h, --help Show usage.
|
||||
|
||||
Exits 1 when any sentence diverges, 0 when everything matches.
|
||||
`);
|
||||
}
|
||||
|
||||
function requireValue(args: string[], flag: string): string {
|
||||
const next = args.shift();
|
||||
if (!next) {
|
||||
throw new Error(`Missing value for ${flag}`);
|
||||
}
|
||||
return next;
|
||||
}
|
||||
|
||||
function parseCliArgs(argv: string[]): CliOptions {
|
||||
const args = [...argv];
|
||||
const sentences: string[] = [];
|
||||
let fromFixtures = false;
|
||||
let apiUrl = DEFAULT_API_URL;
|
||||
let scanLength = DEFAULT_SCAN_LENGTH;
|
||||
let emitJson = false;
|
||||
let nameMatch = false;
|
||||
let yomitanExtensionPath: string | undefined;
|
||||
let yomitanUserDataPath = DEFAULT_YOMITAN_USER_DATA_PATH;
|
||||
let mecabCommand: string | undefined;
|
||||
let mecabDictionaryPath: string | undefined;
|
||||
|
||||
while (args.length > 0) {
|
||||
const arg = args.shift();
|
||||
if (!arg) break;
|
||||
|
||||
if (arg === '--help' || arg === '-h') {
|
||||
printUsage();
|
||||
process.exit(0);
|
||||
}
|
||||
if (arg === '--from-fixtures') {
|
||||
fromFixtures = true;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--file') {
|
||||
const filePath = requireValue(args, arg);
|
||||
const lines = fs
|
||||
.readFileSync(filePath, 'utf8')
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0);
|
||||
sentences.push(...lines);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--api-url') {
|
||||
apiUrl = requireValue(args, arg).replace(/\/$/, '');
|
||||
continue;
|
||||
}
|
||||
if (arg === '--scan-length') {
|
||||
const value = Number(requireValue(args, arg));
|
||||
if (!Number.isInteger(value) || value <= 0) {
|
||||
throw new Error('Invalid --scan-length value');
|
||||
}
|
||||
scanLength = value;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--json') {
|
||||
emitJson = true;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--name-match') {
|
||||
nameMatch = true;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--yomitan-extension') {
|
||||
yomitanExtensionPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--yomitan-user-data') {
|
||||
yomitanUserDataPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--mecab-command') {
|
||||
mecabCommand = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--mecab-dictionary') {
|
||||
mecabDictionaryPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg.startsWith('-') && arg !== '-') {
|
||||
throw new Error(`Unknown flag: ${arg}`);
|
||||
}
|
||||
sentences.push(arg);
|
||||
}
|
||||
|
||||
if (sentences.length === 0) {
|
||||
fromFixtures = true;
|
||||
}
|
||||
|
||||
return {
|
||||
sentences,
|
||||
fromFixtures,
|
||||
apiUrl,
|
||||
scanLength,
|
||||
emitJson,
|
||||
nameMatch,
|
||||
yomitanExtensionPath,
|
||||
yomitanUserDataPath,
|
||||
mecabCommand,
|
||||
mecabDictionaryPath,
|
||||
};
|
||||
}
|
||||
|
||||
function loadFixtureSentences(): string[] {
|
||||
if (!fs.existsSync(FIXTURE_DIR)) {
|
||||
return [];
|
||||
}
|
||||
return fs
|
||||
.readdirSync(FIXTURE_DIR)
|
||||
.filter((entry) => entry.endsWith('.json'))
|
||||
.sort()
|
||||
.map((entry) => {
|
||||
const fixture = JSON.parse(fs.readFileSync(path.join(FIXTURE_DIR, entry), 'utf8')) as {
|
||||
input?: { text?: string };
|
||||
};
|
||||
return fixture.input?.text ?? '';
|
||||
})
|
||||
.filter((text) => text.length > 0);
|
||||
}
|
||||
|
||||
function normalizeTokenizerText(text: string): string {
|
||||
return text
|
||||
.replace(/\r\n/g, '\n')
|
||||
.replace(/\\N/g, '\n')
|
||||
.replace(/\\n/g, '\n')
|
||||
.trim()
|
||||
.replace(/\n/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
interface FuriganaChunk {
|
||||
startPos: number;
|
||||
endPos: number;
|
||||
reading: string;
|
||||
headword: string;
|
||||
}
|
||||
|
||||
// Walk the raw parseText content the same way the parse-result mapper does
|
||||
// (whole segments, char offsets accumulated across lines) and capture, per
|
||||
// chunk, the effective reading (kana chunks read as themselves) and the first
|
||||
// dictionary headword term. The mapper's token spans are authoritative for
|
||||
// grouping; these chunks supply the complete reading/lemma the legacy mapper
|
||||
// does not synthesize.
|
||||
function collectFuriganaChunks(parseResults: unknown): FuriganaChunk[] {
|
||||
const chunks: FuriganaChunk[] = [];
|
||||
if (!Array.isArray(parseResults)) {
|
||||
return chunks;
|
||||
}
|
||||
const content = (parseResults[0] as { content?: unknown } | undefined)?.content;
|
||||
if (!Array.isArray(content)) {
|
||||
return chunks;
|
||||
}
|
||||
let charOffset = 0;
|
||||
for (const line of content) {
|
||||
if (!Array.isArray(line)) {
|
||||
continue;
|
||||
}
|
||||
for (const segment of line as Array<Record<string, unknown>>) {
|
||||
const text = typeof segment?.text === 'string' ? segment.text : '';
|
||||
if (!text) {
|
||||
continue;
|
||||
}
|
||||
const reading =
|
||||
typeof segment.reading === 'string' && segment.reading.length > 0 ? segment.reading : text;
|
||||
let headword = '';
|
||||
const headwords = segment.headwords;
|
||||
if (Array.isArray(headwords)) {
|
||||
for (const row of headwords) {
|
||||
const first = Array.isArray(row) ? (row[0] as { term?: unknown } | undefined) : undefined;
|
||||
if (first && typeof first.term === 'string' && first.term.trim().length > 0) {
|
||||
headword = first.term.trim();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
chunks.push({ startPos: charOffset, endPos: charOffset + text.length, reading, headword });
|
||||
charOffset += text.length;
|
||||
}
|
||||
}
|
||||
return chunks;
|
||||
}
|
||||
|
||||
function overlayChunkMetadata(tokens: MergedToken[], chunks: FuriganaChunk[]): MergedToken[] {
|
||||
return tokens.map((token) => {
|
||||
const covered = chunks.filter(
|
||||
(chunk) => chunk.startPos >= token.startPos && chunk.endPos <= token.endPos,
|
||||
);
|
||||
if (covered.length === 0) {
|
||||
return token;
|
||||
}
|
||||
const reading = covered.map((chunk) => chunk.reading).join('');
|
||||
const headword = covered.find((chunk) => chunk.headword)?.headword ?? token.surface;
|
||||
return { ...token, reading, headword };
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchApiTokens(
|
||||
apiUrl: string,
|
||||
text: string,
|
||||
scanLength: number,
|
||||
): Promise<MergedToken[] | null> {
|
||||
const response = await fetchYomitanApi(`${apiUrl}/tokenize`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ text, scanLength, parser: 'scanning-parser' }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`yomitan-api /tokenize responded ${response.status}`);
|
||||
}
|
||||
const parseResults = (await response.json()) as unknown;
|
||||
const tokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
if (!tokens) {
|
||||
return tokens;
|
||||
}
|
||||
return overlayChunkMetadata(tokens, collectFuriganaChunks(parseResults));
|
||||
}
|
||||
|
||||
interface ComparedToken {
|
||||
surface: string;
|
||||
reading: string;
|
||||
headword: string;
|
||||
startPos: number;
|
||||
endPos: number;
|
||||
frequencyRank?: number;
|
||||
}
|
||||
|
||||
function toComparedTokens(tokens: MergedToken[] | null): ComparedToken[] {
|
||||
return (tokens ?? []).map((token) => ({
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
frequencyRank: token.frequencyRank,
|
||||
}));
|
||||
}
|
||||
|
||||
interface MismatchRegion {
|
||||
subminer: ComparedToken[];
|
||||
yomitan: ComparedToken[];
|
||||
}
|
||||
|
||||
interface PairDiff {
|
||||
surface: string;
|
||||
field: 'headword' | 'reading';
|
||||
subminer: string;
|
||||
yomitan: string;
|
||||
}
|
||||
|
||||
interface SentenceComparison {
|
||||
text: string;
|
||||
subminerTokens: ComparedToken[];
|
||||
yomitanTokens: ComparedToken[];
|
||||
segmentationMatches: boolean;
|
||||
mismatchRegions: MismatchRegion[];
|
||||
pairDiffs: PairDiff[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
// Walk both token lists in span order. Tokens whose [startPos, endPos) agree
|
||||
// are paired; disagreeing runs are grouped until the boundaries realign.
|
||||
function compareTokenLists(
|
||||
subminer: ComparedToken[],
|
||||
yomitan: ComparedToken[],
|
||||
): Pick<SentenceComparison, 'segmentationMatches' | 'mismatchRegions' | 'pairDiffs'> {
|
||||
const mismatchRegions: MismatchRegion[] = [];
|
||||
const pairDiffs: PairDiff[] = [];
|
||||
let i = 0;
|
||||
let j = 0;
|
||||
|
||||
while (i < subminer.length || j < yomitan.length) {
|
||||
const a = subminer[i];
|
||||
const b = yomitan[j];
|
||||
if (a && b && a.startPos === b.startPos && a.endPos === b.endPos) {
|
||||
if (a.headword !== b.headword) {
|
||||
pairDiffs.push({
|
||||
surface: a.surface,
|
||||
field: 'headword',
|
||||
subminer: a.headword,
|
||||
yomitan: b.headword,
|
||||
});
|
||||
}
|
||||
// An empty reading means "reads as its surface" (kana tokens, unparsed
|
||||
// runs) — normalize before comparing.
|
||||
if ((a.reading || a.surface) !== (b.reading || b.surface)) {
|
||||
pairDiffs.push({
|
||||
surface: a.surface,
|
||||
field: 'reading',
|
||||
subminer: a.reading,
|
||||
yomitan: b.reading,
|
||||
});
|
||||
}
|
||||
i += 1;
|
||||
j += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const region: MismatchRegion = { subminer: [], yomitan: [] };
|
||||
let aEnd = a ? a.startPos : Number.MAX_SAFE_INTEGER;
|
||||
let bEnd = b ? b.startPos : Number.MAX_SAFE_INTEGER;
|
||||
do {
|
||||
if (aEnd <= bEnd && i < subminer.length) {
|
||||
const token = subminer[i];
|
||||
if (token) {
|
||||
region.subminer.push(token);
|
||||
aEnd = token.endPos;
|
||||
}
|
||||
i += 1;
|
||||
} else if (j < yomitan.length) {
|
||||
const token = yomitan[j];
|
||||
if (token) {
|
||||
region.yomitan.push(token);
|
||||
bEnd = token.endPos;
|
||||
}
|
||||
j += 1;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
} while (aEnd !== bEnd && (i < subminer.length || j < yomitan.length));
|
||||
mismatchRegions.push(region);
|
||||
}
|
||||
|
||||
return {
|
||||
segmentationMatches: mismatchRegions.length === 0,
|
||||
mismatchRegions,
|
||||
pairDiffs,
|
||||
};
|
||||
}
|
||||
|
||||
function renderTokenRun(tokens: ComparedToken[]): string {
|
||||
if (tokens.length === 0) {
|
||||
return '(no tokens)';
|
||||
}
|
||||
return tokens.map((token) => token.surface).join('|');
|
||||
}
|
||||
|
||||
function renderReport(comparisons: SentenceComparison[]): void {
|
||||
let divergent = 0;
|
||||
for (const comparison of comparisons) {
|
||||
const clean =
|
||||
!comparison.error && comparison.segmentationMatches && comparison.pairDiffs.length === 0;
|
||||
if (!clean) {
|
||||
divergent += 1;
|
||||
}
|
||||
process.stdout.write(`\n${clean ? 'MATCH' : 'DIFF '} ${comparison.text}\n`);
|
||||
if (comparison.error) {
|
||||
process.stdout.write(` error: ${comparison.error}\n`);
|
||||
continue;
|
||||
}
|
||||
if (clean) {
|
||||
process.stdout.write(
|
||||
` ${comparison.subminerTokens.length} tokens: ${renderTokenRun(comparison.subminerTokens)}\n`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (!comparison.segmentationMatches) {
|
||||
process.stdout.write(' segmentation:\n');
|
||||
for (const region of comparison.mismatchRegions) {
|
||||
process.stdout.write(` subminer: ${renderTokenRun(region.subminer)}\n`);
|
||||
process.stdout.write(` yomitan: ${renderTokenRun(region.yomitan)}\n`);
|
||||
}
|
||||
}
|
||||
for (const diff of comparison.pairDiffs) {
|
||||
process.stdout.write(
|
||||
` ${diff.surface}: ${diff.field} "${diff.subminer}" (subminer) vs "${diff.yomitan}" (yomitan)\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
process.stdout.write(
|
||||
`\n${comparisons.length - divergent}/${comparisons.length} sentences match stock Yomitan\n`,
|
||||
);
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const options = parseCliArgs(process.argv.slice(2));
|
||||
const sentences = [...(options.fromFixtures ? loadFixtureSentences() : []), ...options.sentences];
|
||||
if (sentences.length === 0) {
|
||||
throw new Error('No sentences to compare (no fixtures found and none provided).');
|
||||
}
|
||||
|
||||
// Fail fast with a clear message when the bridge is not running.
|
||||
try {
|
||||
const probe = await fetchYomitanApi(`${options.apiUrl}/serverVersion`, {
|
||||
method: 'POST',
|
||||
body: '{}',
|
||||
});
|
||||
if (!probe.ok) {
|
||||
throw new Error(`status ${probe.status}`);
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`yomitan-api is not reachable at ${options.apiUrl} (${(error as Error).message}). ` +
|
||||
'Ensure the browser is running and "Enable Yomitan API" is on in Yomitan settings.',
|
||||
);
|
||||
}
|
||||
|
||||
const electronModule = await import('electron').catch(() => null);
|
||||
if (!electronModule?.app || !electronModule?.session) {
|
||||
throw new Error('Electron runtime required; run via `bun run compare-yomitan-api:electron`.');
|
||||
}
|
||||
|
||||
const mecabTokenizer = new MecabTokenizer({
|
||||
mecabCommand: options.mecabCommand,
|
||||
dictionaryPath: options.mecabDictionaryPath,
|
||||
});
|
||||
if (!(await mecabTokenizer.checkAvailability())) {
|
||||
throw new Error('MeCab is not available on this system.');
|
||||
}
|
||||
|
||||
if (typeof electronModule.app.setPath === 'function') {
|
||||
fs.mkdirSync(options.yomitanUserDataPath, { recursive: true });
|
||||
electronModule.app.setPath('userData', options.yomitanUserDataPath);
|
||||
}
|
||||
await electronModule.app.whenReady();
|
||||
|
||||
const extensionPath = resolveBuiltYomitanExtensionPath({
|
||||
explicitPath: options.yomitanExtensionPath,
|
||||
cwd: process.cwd(),
|
||||
});
|
||||
if (!extensionPath) {
|
||||
throw new Error('No built Yomitan extension found; run `bun run build:yomitan` first.');
|
||||
}
|
||||
const extension = await electronModule.session.defaultSession.loadExtension(extensionPath, {
|
||||
allowFileAccess: true,
|
||||
});
|
||||
|
||||
let parserWindow: Electron.BrowserWindow | null = null;
|
||||
let parserReadyPromise: Promise<void> | null = null;
|
||||
let parserInitPromise: Promise<boolean> | null = null;
|
||||
|
||||
const runtimeDeps = createTokenizerDepsRuntime({
|
||||
getYomitanExt: () => extension,
|
||||
getYomitanParserWindow: () => parserWindow,
|
||||
setYomitanParserWindow: (window) => {
|
||||
parserWindow = window;
|
||||
},
|
||||
getYomitanParserReadyPromise: () => parserReadyPromise,
|
||||
setYomitanParserReadyPromise: (promise) => {
|
||||
parserReadyPromise = promise;
|
||||
},
|
||||
getYomitanParserInitPromise: () => parserInitPromise,
|
||||
setYomitanParserInitPromise: (promise) => {
|
||||
parserInitPromise = promise;
|
||||
},
|
||||
isKnownWord: () => false,
|
||||
getKnownWordMatchMode: () => 'headword',
|
||||
getJlptLevel: () => null,
|
||||
getNameMatchEnabled: () => options.nameMatch,
|
||||
getMecabTokenizer: () => ({
|
||||
tokenize: (text: string) => mecabTokenizer.tokenize(text),
|
||||
}),
|
||||
});
|
||||
const deps = {
|
||||
...runtimeDeps,
|
||||
enrichTokensWithMecab: async (
|
||||
tokens: Parameters<typeof enrichTokensWithMecabPos1>[0],
|
||||
mecabTokens: Parameters<typeof enrichTokensWithMecabPos1>[1],
|
||||
) => enrichTokensWithMecabPos1(tokens, mecabTokens),
|
||||
};
|
||||
|
||||
const comparisons: SentenceComparison[] = [];
|
||||
try {
|
||||
for (const sentence of sentences) {
|
||||
const text = normalizeTokenizerText(sentence);
|
||||
try {
|
||||
const [subtitleData, apiTokens] = await Promise.all([
|
||||
tokenizeSubtitle(text, deps),
|
||||
fetchApiTokens(options.apiUrl, text, options.scanLength),
|
||||
]);
|
||||
const subminerTokens = toComparedTokens(subtitleData.tokens);
|
||||
const yomitanTokens = toComparedTokens(apiTokens);
|
||||
comparisons.push({
|
||||
text,
|
||||
subminerTokens,
|
||||
yomitanTokens,
|
||||
...compareTokenLists(subminerTokens, yomitanTokens),
|
||||
});
|
||||
} catch (error) {
|
||||
comparisons.push({
|
||||
text,
|
||||
subminerTokens: [],
|
||||
yomitanTokens: [],
|
||||
segmentationMatches: false,
|
||||
mismatchRegions: [],
|
||||
pairDiffs: [],
|
||||
error: (error as Error).message,
|
||||
});
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (parserWindow) {
|
||||
const window = parserWindow as Electron.BrowserWindow;
|
||||
if (!window.isDestroyed()) {
|
||||
window.destroy();
|
||||
}
|
||||
}
|
||||
electronModule.app.quit();
|
||||
}
|
||||
|
||||
if (options.emitJson) {
|
||||
process.stdout.write(`${JSON.stringify(comparisons, null, 2)}\n`);
|
||||
} else {
|
||||
renderReport(comparisons);
|
||||
}
|
||||
|
||||
const anyDivergence = comparisons.some(
|
||||
(comparison) =>
|
||||
comparison.error !== undefined ||
|
||||
!comparison.segmentationMatches ||
|
||||
comparison.pairDiffs.length > 0,
|
||||
);
|
||||
process.exitCode = anyDivergence ? 1 : 0;
|
||||
}
|
||||
|
||||
main()
|
||||
.then(() => {
|
||||
process.exitCode = process.exitCode ?? 0;
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error(`Error: ${(error as Error).message}`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,496 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import process from 'node:process';
|
||||
|
||||
import { createTokenizerDepsRuntime, tokenizeSubtitle } from '../src/core/services/tokenizer.js';
|
||||
import {
|
||||
buildFixtureLookups,
|
||||
classifyInjectedScript,
|
||||
fixtureConfigGetters,
|
||||
hashInjectedScript,
|
||||
projectGoldenTokens,
|
||||
pruneGoldenMessages,
|
||||
} from '../src/core/services/tokenizer/golden-corpus-harness.js';
|
||||
import type {
|
||||
GoldenFixture,
|
||||
GoldenFixtureConfig,
|
||||
GoldenRecordedMessage,
|
||||
GoldenRecordedScript,
|
||||
} from '../src/core/services/tokenizer/golden-corpus-harness.js';
|
||||
import { enrichTokensWithMecabPos1 } from '../src/core/services/tokenizer/parser-enrichment-stage.js';
|
||||
import { resolveYomitanExtensionPath as resolveBuiltYomitanExtensionPath } from '../src/core/services/yomitan-extension-paths.js';
|
||||
import { MecabTokenizer } from '../src/mecab-tokenizer.js';
|
||||
import type { JlptLevel, Token } from '../src/types.js';
|
||||
|
||||
const DEFAULT_YOMITAN_USER_DATA_PATH = path.join(os.homedir(), '.config', 'SubMiner');
|
||||
const DEFAULT_FIXTURE_DIR = path.join(
|
||||
'src',
|
||||
'core',
|
||||
'services',
|
||||
'tokenizer',
|
||||
'__fixtures__',
|
||||
'golden',
|
||||
);
|
||||
const JLPT_LEVELS: ReadonlySet<string> = new Set(['N1', 'N2', 'N3', 'N4', 'N5']);
|
||||
|
||||
interface CliOptions {
|
||||
name: string;
|
||||
text: string;
|
||||
description?: string;
|
||||
issueRefs: string[];
|
||||
config: GoldenFixtureConfig;
|
||||
outDir: string;
|
||||
force: boolean;
|
||||
yomitanExtensionPath?: string;
|
||||
yomitanUserDataPath: string;
|
||||
mecabCommand?: string;
|
||||
mecabDictionaryPath?: string;
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
process.stdout.write(`Usage:
|
||||
bun run record-tokenizer-fixture:electron -- --name <slug> [options] <subtitle text>
|
||||
|
||||
Records a golden tokenizer fixture: raw Yomitan backend responses, raw MeCab
|
||||
tokens, and the annotated tokens the full pipeline produced for the text.
|
||||
Review the "expected" block before committing — it becomes the assertion.
|
||||
|
||||
--name <slug> Fixture name (file becomes <slug>.json). Required.
|
||||
--description <text> What behavior this fixture pins down.
|
||||
--issue <ref> Related issue/PR ref, repeatable (e.g. --issue "#156").
|
||||
--known-word <w[:reading]> Mark a word known, repeatable.
|
||||
--jlpt <term=level> JLPT level for a term, repeatable (level N1..N5).
|
||||
--local-frequency <t=rank> Local frequency-dictionary rank, repeatable.
|
||||
--match-mode <mode> Known-word match mode: headword|surface.
|
||||
--min-sentence-words <n> Minimum sentence words for N+1.
|
||||
--no-known-words Disable known-word annotation.
|
||||
--no-nplusone Disable N+1 targeting.
|
||||
--no-jlpt Disable JLPT annotation.
|
||||
--no-name-match Disable character-name matching.
|
||||
--no-frequency Disable frequency annotation.
|
||||
--out-dir <path> Fixture directory (default: ${DEFAULT_FIXTURE_DIR}).
|
||||
--force Overwrite an existing fixture file.
|
||||
--yomitan-extension <path> Path to built Yomitan extension directory.
|
||||
--yomitan-user-data <path> Electron userData directory (default: ~/.config/SubMiner).
|
||||
--mecab-command <path> MeCab binary (default: mecab).
|
||||
--mecab-dictionary <path> MeCab dictionary directory.
|
||||
-h, --help Show usage.
|
||||
`);
|
||||
}
|
||||
|
||||
function requireValue(args: string[], flag: string): string {
|
||||
const next = args.shift();
|
||||
if (!next) {
|
||||
throw new Error(`Missing value for ${flag}`);
|
||||
}
|
||||
return next;
|
||||
}
|
||||
|
||||
function parseCliArgs(argv: string[]): CliOptions {
|
||||
const args = [...argv];
|
||||
const inputParts: string[] = [];
|
||||
const issueRefs: string[] = [];
|
||||
const knownWords: GoldenFixtureConfig['knownWords'] = [];
|
||||
const jlptLevels: Record<string, JlptLevel> = {};
|
||||
const localFrequencyRanks: Record<string, number> = {};
|
||||
const config: GoldenFixtureConfig = {};
|
||||
let name: string | undefined;
|
||||
let description: string | undefined;
|
||||
let outDir = DEFAULT_FIXTURE_DIR;
|
||||
let force = false;
|
||||
let yomitanExtensionPath: string | undefined;
|
||||
let yomitanUserDataPath = DEFAULT_YOMITAN_USER_DATA_PATH;
|
||||
let mecabCommand: string | undefined;
|
||||
let mecabDictionaryPath: string | undefined;
|
||||
|
||||
while (args.length > 0) {
|
||||
const arg = args.shift();
|
||||
if (!arg) break;
|
||||
|
||||
if (arg === '--help' || arg === '-h') {
|
||||
printUsage();
|
||||
process.exit(0);
|
||||
}
|
||||
if (arg === '--name') {
|
||||
name = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--description') {
|
||||
description = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--issue') {
|
||||
issueRefs.push(requireValue(args, arg));
|
||||
continue;
|
||||
}
|
||||
if (arg === '--known-word') {
|
||||
const value = requireValue(args, arg);
|
||||
const separator = value.indexOf(':');
|
||||
knownWords.push(
|
||||
separator === -1
|
||||
? value
|
||||
: { text: value.slice(0, separator), reading: value.slice(separator + 1) },
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--jlpt') {
|
||||
const value = requireValue(args, arg);
|
||||
const separator = value.indexOf('=');
|
||||
const level = separator === -1 ? '' : value.slice(separator + 1);
|
||||
if (separator === -1 || !JLPT_LEVELS.has(level)) {
|
||||
throw new Error(`Invalid --jlpt value "${value}"; expected <term>=N1..N5`);
|
||||
}
|
||||
jlptLevels[value.slice(0, separator)] = level as JlptLevel;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--local-frequency') {
|
||||
const value = requireValue(args, arg);
|
||||
const separator = value.indexOf('=');
|
||||
const rank = separator === -1 ? Number.NaN : Number(value.slice(separator + 1));
|
||||
if (!Number.isInteger(rank) || rank <= 0) {
|
||||
throw new Error(`Invalid --local-frequency value "${value}"; expected <term>=<rank>`);
|
||||
}
|
||||
localFrequencyRanks[value.slice(0, separator)] = rank;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--match-mode') {
|
||||
const value = requireValue(args, arg);
|
||||
if (value !== 'headword' && value !== 'surface') {
|
||||
throw new Error(`Invalid --match-mode "${value}"; expected headword|surface`);
|
||||
}
|
||||
config.knownWordMatchMode = value;
|
||||
config.frequencyMatchMode = value;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--min-sentence-words') {
|
||||
const value = Number(requireValue(args, arg));
|
||||
if (!Number.isInteger(value) || value < 0) {
|
||||
throw new Error('Invalid --min-sentence-words value');
|
||||
}
|
||||
config.minSentenceWordsForNPlusOne = value;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--no-known-words') {
|
||||
config.knownWordsEnabled = false;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--no-nplusone') {
|
||||
config.nPlusOneEnabled = false;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--no-jlpt') {
|
||||
config.jlptEnabled = false;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--no-name-match') {
|
||||
config.nameMatchEnabled = false;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--no-frequency') {
|
||||
config.frequencyEnabled = false;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--out-dir') {
|
||||
outDir = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--force') {
|
||||
force = true;
|
||||
continue;
|
||||
}
|
||||
if (arg === '--yomitan-extension') {
|
||||
yomitanExtensionPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--yomitan-user-data') {
|
||||
yomitanUserDataPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--mecab-command') {
|
||||
mecabCommand = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg === '--mecab-dictionary') {
|
||||
mecabDictionaryPath = requireValue(args, arg);
|
||||
continue;
|
||||
}
|
||||
if (arg.startsWith('-') && arg !== '-') {
|
||||
throw new Error(`Unknown flag: ${arg}`);
|
||||
}
|
||||
inputParts.push(arg);
|
||||
}
|
||||
|
||||
if (!name || !/^[a-z0-9][a-z0-9-]*$/.test(name)) {
|
||||
throw new Error('A kebab-case --name is required (e.g. --name kureru-lexical-keep)');
|
||||
}
|
||||
const text = inputParts.join(' ').trim();
|
||||
if (!text) {
|
||||
throw new Error('Provide the subtitle text to record as positional arguments.');
|
||||
}
|
||||
|
||||
if (knownWords.length > 0) config.knownWords = knownWords;
|
||||
if (Object.keys(jlptLevels).length > 0) config.jlptLevels = jlptLevels;
|
||||
if (Object.keys(localFrequencyRanks).length > 0) {
|
||||
config.localFrequencyRanks = localFrequencyRanks;
|
||||
}
|
||||
|
||||
return {
|
||||
name,
|
||||
text,
|
||||
description,
|
||||
issueRefs,
|
||||
config,
|
||||
outDir,
|
||||
force,
|
||||
yomitanExtensionPath,
|
||||
yomitanUserDataPath,
|
||||
mecabCommand,
|
||||
mecabDictionaryPath,
|
||||
};
|
||||
}
|
||||
|
||||
const RECORDER_SHIM_SCRIPT = `
|
||||
(() => {
|
||||
const g = globalThis;
|
||||
if (g.__subminerGoldenRecorderInstalled) { return true; }
|
||||
g.__subminerGoldenRecorderInstalled = true;
|
||||
g.__subminerGoldenMessages = [];
|
||||
const runtime = chrome.runtime;
|
||||
const original = runtime.sendMessage.bind(runtime);
|
||||
runtime.sendMessage = (payload, callback) => {
|
||||
return original(payload, (response) => {
|
||||
try {
|
||||
g.__subminerGoldenMessages.push(JSON.parse(JSON.stringify({
|
||||
action: payload && payload.action ? payload.action : '',
|
||||
params: payload && typeof payload.params !== 'undefined' ? payload.params : null,
|
||||
response: typeof response !== 'undefined' ? response : null,
|
||||
})));
|
||||
} catch {}
|
||||
if (callback) { callback(response); }
|
||||
});
|
||||
};
|
||||
return true;
|
||||
})();
|
||||
`;
|
||||
|
||||
const RECORDER_DRAIN_SCRIPT = `
|
||||
(() => {
|
||||
const g = globalThis;
|
||||
const log = g.__subminerGoldenMessages || [];
|
||||
g.__subminerGoldenMessages = [];
|
||||
return JSON.stringify(log);
|
||||
})();
|
||||
`;
|
||||
|
||||
interface RecordingSink {
|
||||
messages: GoldenRecordedMessage[];
|
||||
scripts: GoldenRecordedScript[];
|
||||
mecab: Record<string, Token[] | null>;
|
||||
}
|
||||
|
||||
function patchParserWindowForRecording(
|
||||
window: Electron.BrowserWindow | null,
|
||||
patched: WeakSet<object>,
|
||||
sink: RecordingSink,
|
||||
): void {
|
||||
if (!window || window.isDestroyed()) {
|
||||
return;
|
||||
}
|
||||
const webContents = window.webContents as unknown as {
|
||||
executeJavaScript: (script: string, userGesture?: boolean) => Promise<unknown>;
|
||||
};
|
||||
if (patched.has(webContents)) {
|
||||
return;
|
||||
}
|
||||
patched.add(webContents);
|
||||
|
||||
const original = webContents.executeJavaScript.bind(webContents);
|
||||
let shimReady: Promise<unknown> | null = null;
|
||||
const drain = async (): Promise<void> => {
|
||||
try {
|
||||
const raw = (await original(RECORDER_DRAIN_SCRIPT, true)) as string;
|
||||
const drained = JSON.parse(raw) as GoldenRecordedMessage[];
|
||||
sink.messages.push(...drained);
|
||||
} catch (error) {
|
||||
process.stderr.write(
|
||||
`Warning: failed to drain recorded messages: ${(error as Error).message}\n`,
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
webContents.executeJavaScript = async (script: string, userGesture?: boolean) => {
|
||||
if (!shimReady) {
|
||||
shimReady = original(RECORDER_SHIM_SCRIPT, true);
|
||||
}
|
||||
try {
|
||||
await shimReady;
|
||||
} catch {
|
||||
shimReady = null;
|
||||
}
|
||||
let result: unknown;
|
||||
try {
|
||||
result = await original(script, userGesture);
|
||||
} finally {
|
||||
await drain();
|
||||
}
|
||||
sink.scripts.push({
|
||||
sha256: hashInjectedScript(script),
|
||||
marker: classifyInjectedScript(script),
|
||||
result: result === undefined ? null : result,
|
||||
});
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const options = parseCliArgs(process.argv.slice(2));
|
||||
const outPath = path.join(options.outDir, `${options.name}.json`);
|
||||
if (fs.existsSync(outPath) && !options.force) {
|
||||
throw new Error(`${outPath} already exists; pass --force to overwrite.`);
|
||||
}
|
||||
|
||||
const electronModule = await import('electron').catch(() => null);
|
||||
if (!electronModule?.app || !electronModule?.session) {
|
||||
throw new Error(
|
||||
'Electron runtime is required; run via `bun run record-tokenizer-fixture:electron`.',
|
||||
);
|
||||
}
|
||||
|
||||
const mecabTokenizer = new MecabTokenizer({
|
||||
mecabCommand: options.mecabCommand,
|
||||
dictionaryPath: options.mecabDictionaryPath,
|
||||
});
|
||||
if (!(await mecabTokenizer.checkAvailability())) {
|
||||
throw new Error('MeCab is not available on this system.');
|
||||
}
|
||||
|
||||
if (typeof electronModule.app.setPath === 'function') {
|
||||
electronModule.app.setPath('userData', options.yomitanUserDataPath);
|
||||
}
|
||||
await electronModule.app.whenReady();
|
||||
|
||||
const extensionPath = resolveBuiltYomitanExtensionPath({
|
||||
explicitPath: options.yomitanExtensionPath,
|
||||
cwd: process.cwd(),
|
||||
});
|
||||
if (!extensionPath) {
|
||||
throw new Error('No built Yomitan extension found; run `bun run build:yomitan` first.');
|
||||
}
|
||||
const extension = await electronModule.session.defaultSession.loadExtension(extensionPath, {
|
||||
allowFileAccess: true,
|
||||
});
|
||||
|
||||
const sink: RecordingSink = { messages: [], scripts: [], mecab: {} };
|
||||
const patched = new WeakSet<object>();
|
||||
let parserWindow: Electron.BrowserWindow | null = null;
|
||||
let parserReadyPromise: Promise<void> | null = null;
|
||||
let parserInitPromise: Promise<boolean> | null = null;
|
||||
|
||||
const lookups = buildFixtureLookups(options.config);
|
||||
const runtimeDeps = createTokenizerDepsRuntime({
|
||||
getYomitanExt: () => extension,
|
||||
getYomitanParserWindow: () => {
|
||||
patchParserWindowForRecording(parserWindow, patched, sink);
|
||||
return parserWindow;
|
||||
},
|
||||
setYomitanParserWindow: (window) => {
|
||||
parserWindow = window;
|
||||
patchParserWindowForRecording(parserWindow, patched, sink);
|
||||
},
|
||||
getYomitanParserReadyPromise: () => parserReadyPromise,
|
||||
setYomitanParserReadyPromise: (promise) => {
|
||||
parserReadyPromise = promise;
|
||||
},
|
||||
getYomitanParserInitPromise: () => parserInitPromise,
|
||||
setYomitanParserInitPromise: (promise) => {
|
||||
parserInitPromise = promise;
|
||||
},
|
||||
isKnownWord: lookups.isKnownWord,
|
||||
getJlptLevel: lookups.getJlptLevel,
|
||||
getFrequencyRank: lookups.getFrequencyRank,
|
||||
...fixtureConfigGetters(options.config),
|
||||
getMecabTokenizer: () => ({
|
||||
tokenize: async (text: string) => {
|
||||
const rawTokens = await mecabTokenizer.tokenize(text);
|
||||
sink.mecab[text] = rawTokens;
|
||||
return rawTokens;
|
||||
},
|
||||
}),
|
||||
});
|
||||
const deps = {
|
||||
...runtimeDeps,
|
||||
// Same pinned enrichment implementation the replay harness uses.
|
||||
enrichTokensWithMecab: async (
|
||||
tokens: Parameters<typeof enrichTokensWithMecabPos1>[0],
|
||||
mecabTokens: Parameters<typeof enrichTokensWithMecabPos1>[1],
|
||||
) => enrichTokensWithMecabPos1(tokens, mecabTokens),
|
||||
};
|
||||
|
||||
try {
|
||||
const subtitleData = await tokenizeSubtitle(options.text, deps);
|
||||
const expectedTokens = projectGoldenTokens(subtitleData.tokens);
|
||||
|
||||
const prunedMessages = pruneGoldenMessages(sink.messages);
|
||||
const fixture: GoldenFixture = {
|
||||
name: options.name,
|
||||
...(options.description ? { description: options.description } : {}),
|
||||
...(options.issueRefs.length > 0 ? { issueRefs: options.issueRefs } : {}),
|
||||
recordedAt: new Date().toISOString(),
|
||||
input: { text: options.text },
|
||||
config: options.config,
|
||||
recording: {
|
||||
messages: prunedMessages,
|
||||
scripts: sink.scripts,
|
||||
mecab: sink.mecab,
|
||||
},
|
||||
expected: { tokens: expectedTokens },
|
||||
};
|
||||
|
||||
fs.mkdirSync(options.outDir, { recursive: true });
|
||||
fs.writeFileSync(outPath, `${JSON.stringify(fixture, null, 2)}\n`);
|
||||
|
||||
process.stdout.write(`Recorded ${outPath}\n`);
|
||||
process.stdout.write(
|
||||
` messages=${prunedMessages.length} scripts=${sink.scripts.length} mecabTexts=${Object.keys(sink.mecab).length}\n`,
|
||||
);
|
||||
process.stdout.write('\nExpected tokens (review before committing):\n');
|
||||
if (!expectedTokens || expectedTokens.length === 0) {
|
||||
process.stdout.write(' (none)\n');
|
||||
} else {
|
||||
for (const token of expectedTokens) {
|
||||
const flags = [
|
||||
token.isKnown ? 'known' : null,
|
||||
token.isNPlusOneTarget ? 'n+1' : null,
|
||||
token.isNameMatch ? 'name' : null,
|
||||
token.isUnparsedRun ? 'unparsed' : null,
|
||||
token.jlptLevel ?? null,
|
||||
token.frequencyRank !== undefined ? `freq=${token.frequencyRank}` : null,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(', ');
|
||||
process.stdout.write(
|
||||
` ${token.surface} -> ${token.headword} (${token.reading})${flags ? ` [${flags}]` : ''}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
process.stdout.write(
|
||||
'\nVerify replay with: bun test src/core/services/tokenizer/golden-corpus.test.ts\n',
|
||||
);
|
||||
} finally {
|
||||
if (parserWindow) {
|
||||
const window = parserWindow as Electron.BrowserWindow;
|
||||
if (!window.isDestroyed()) {
|
||||
window.destroy();
|
||||
}
|
||||
}
|
||||
electronModule.app.quit();
|
||||
}
|
||||
}
|
||||
|
||||
main()
|
||||
.then(() => process.exit(0))
|
||||
.catch((error) => {
|
||||
console.error(`Error: ${(error as Error).message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,69 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import { fetchYomitanApi } from './yomitan-api-request.js';
|
||||
|
||||
function createStalledServer(): ReturnType<typeof Bun.serve> {
|
||||
return Bun.serve({
|
||||
hostname: '127.0.0.1',
|
||||
port: 0,
|
||||
fetch: () => new Promise<Response>(() => {}),
|
||||
});
|
||||
}
|
||||
|
||||
test('fetchYomitanApi reports a clear timeout when the bridge stalls', async () => {
|
||||
const server = createStalledServer();
|
||||
|
||||
try {
|
||||
await assert.rejects(
|
||||
() => fetchYomitanApi(`${server.url}tokenize`, { method: 'POST' }, 20),
|
||||
/Yomitan API request timed out after 20ms/,
|
||||
);
|
||||
} finally {
|
||||
server.stop(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('fetchYomitanApi preserves an init cancellation signal', async () => {
|
||||
const server = createStalledServer();
|
||||
const controller = new AbortController();
|
||||
const reason = new Error('cancelled by caller');
|
||||
|
||||
try {
|
||||
const request = fetchYomitanApi(`${server.url}tokenize`, { signal: controller.signal }, 50);
|
||||
controller.abort(reason);
|
||||
await assert.rejects(request, (error) => error === reason);
|
||||
} finally {
|
||||
server.stop(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('fetchYomitanApi preserves a Request input cancellation signal', async () => {
|
||||
const server = createStalledServer();
|
||||
const controller = new AbortController();
|
||||
const reason = new Error('cancelled by request');
|
||||
const input = new Request(`${server.url}tokenize`, { signal: controller.signal });
|
||||
|
||||
try {
|
||||
const request = fetchYomitanApi(input, undefined, 50);
|
||||
controller.abort(reason);
|
||||
await assert.rejects(request, (error) => error === reason);
|
||||
} finally {
|
||||
server.stop(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('fetchYomitanApi accepts a null init cancellation signal', async () => {
|
||||
const server = Bun.serve({
|
||||
hostname: '127.0.0.1',
|
||||
port: 0,
|
||||
fetch: () => new Response('ok'),
|
||||
});
|
||||
|
||||
try {
|
||||
const response = await fetchYomitanApi(`${server.url}tokenize`, { signal: null }, 50);
|
||||
assert.equal(response.status, 200);
|
||||
} finally {
|
||||
server.stop(true);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,26 @@
|
||||
const DEFAULT_YOMITAN_API_TIMEOUT_MS = 10_000;
|
||||
|
||||
export async function fetchYomitanApi(
|
||||
input: RequestInfo | URL,
|
||||
init?: RequestInit,
|
||||
timeoutMs = DEFAULT_YOMITAN_API_TIMEOUT_MS,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const requestSignal = input instanceof Request ? input.signal : undefined;
|
||||
const signals = [controller.signal, init?.signal, requestSignal].filter(
|
||||
(signal): signal is AbortSignal => signal != null,
|
||||
);
|
||||
const signal = signals.length === 1 ? controller.signal : AbortSignal.any(signals);
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
return await fetch(input, { ...init, signal });
|
||||
} catch (error) {
|
||||
if (controller.signal.aborted) {
|
||||
throw new Error(`Yomitan API request timed out after ${timeoutMs}ms`, { cause: error });
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
@@ -762,7 +762,10 @@ test('KnownWordCacheManager suppresses reading-only matches when disallowed', as
|
||||
// Reading-only match stays available for kana subtitle text…
|
||||
assert.equal(manager.isKnownWord('けいこく'), true);
|
||||
// …but a kanji token's reading (渓谷/けいこく) must not borrow 警告's.
|
||||
assert.equal(manager.isKnownWord('けいこく', undefined, { allowReadingOnlyMatch: false }), false);
|
||||
assert.equal(
|
||||
manager.isKnownWord('けいこく', undefined, { allowReadingOnlyMatch: false }),
|
||||
false,
|
||||
);
|
||||
// Mined word texts still match regardless of the flag.
|
||||
assert.equal(manager.isKnownWord('警告', undefined, { allowReadingOnlyMatch: false }), true);
|
||||
} finally {
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Tokenizer golden corpus
|
||||
|
||||
End-to-end regression fixtures for the tokenizer/annotation pipeline. Each
|
||||
`.json` file captures one real subtitle line together with everything the
|
||||
pipeline consumed while tokenizing it live:
|
||||
|
||||
- `recording.messages` — the raw Yomitan backend responses
|
||||
(`chrome.runtime.sendMessage` level: `optionsGetFull`, `getDictionaryInfo`,
|
||||
`parseText`, `termsFind`, `getTermFrequencies`), pruned to the fields the
|
||||
injected scanning helpers actually read.
|
||||
- `recording.scripts` — sha256 → result pairs for each injected script, used
|
||||
only as a replay fallback if a script cannot run in the vm.
|
||||
- `recording.mecab` — raw MeCab tokens keyed by the tokenized text.
|
||||
- `config` — annotation toggles plus fixture-local known words, JLPT levels,
|
||||
and local frequency ranks (see `golden-corpus-harness.ts` for the simplified
|
||||
known-word semantics shared by recorder and replay).
|
||||
- `expected.tokens` — the annotated tokens the full pipeline produced at
|
||||
record time. **This is the assertion.** Review it before committing; it is a
|
||||
characterization of current behavior, not a statement that the behavior is
|
||||
ideal.
|
||||
|
||||
`golden-corpus.test.ts` replays every fixture through the real
|
||||
`tokenizeSubtitle` (scan-token merge, MeCab enrichment, frequency ranks,
|
||||
annotation stage, noise suppression) by executing the real injected scripts in
|
||||
a `node:vm` sandbox against the recorded responses — no Electron, no
|
||||
dictionaries, no network.
|
||||
|
||||
## Recording a fixture
|
||||
|
||||
Requires a built Yomitan extension (`bun run build:yomitan`), your SubMiner
|
||||
Yomitan profile (`~/.config/SubMiner`), and MeCab:
|
||||
|
||||
```sh
|
||||
bun run record-tokenizer-fixture:electron -- \
|
||||
--name my-regression-case \
|
||||
--issue "#123" \
|
||||
--description "what behavior this pins down" \
|
||||
--known-word 私 --known-word 要る:いる \
|
||||
--jlpt 美しい=N4 \
|
||||
そのまま字幕の一行
|
||||
```
|
||||
|
||||
Run with `--help` for all options (annotation toggles, match mode, overwrite
|
||||
with `--force`, ...). The recorder prints the expected tokens for review and
|
||||
the fixture replays immediately via:
|
||||
|
||||
```sh
|
||||
bun test src/core/services/tokenizer/golden-corpus.test.ts
|
||||
```
|
||||
|
||||
## When a fixture fails
|
||||
|
||||
A failure means the pipeline now produces different annotated tokens for that
|
||||
line. If the change is intentional, re-record the fixture with `--force`
|
||||
(same flags — they are stored in the fixture's `config`) and review the diff
|
||||
of `expected.tokens`; the diff _is_ the behavior change. If the change is not
|
||||
intentional, you found the regression before shipping it.
|
||||
|
||||
Fixture dictionaries reflect whatever was installed in the recording profile,
|
||||
so re-recorded fixtures may differ in frequency ranks if dictionaries changed.
|
||||
|
||||
## Comparing against stock Yomitan
|
||||
|
||||
`bun run compare-yomitan-api:electron` diffs SubMiner's tokenization against a
|
||||
stock Yomitan instance reached through the
|
||||
[yomitan-api](https://github.com/yomidevs/yomitan-api) bridge
|
||||
(`http://127.0.0.1:19633`, enable "Yomitan API" in the browser extension's
|
||||
settings). Without arguments it compares every fixture text; pass sentences or
|
||||
`--file <path>` for ad-hoc checks. It reports segmentation, reading, and
|
||||
headword-form divergence and exits non-zero on any difference — useful for
|
||||
spot-checking that the pipeline still matches what Yomitan itself would
|
||||
produce, with your real browser profile and dictionaries as the reference.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,475 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import vm from 'node:vm';
|
||||
import type {
|
||||
FrequencyDictionaryMatchMode,
|
||||
JlptLevel,
|
||||
MergedToken,
|
||||
NPlusOneMatchMode,
|
||||
PartOfSpeech,
|
||||
Token,
|
||||
} from '../../../types';
|
||||
import { createTokenizerDepsRuntime } from '../tokenizer';
|
||||
import type { TokenizerServiceDeps } from '../tokenizer';
|
||||
import { enrichTokensWithMecabPos1 } from './parser-enrichment-stage';
|
||||
|
||||
// Golden-corpus fixtures capture a real subtitle line, the raw Yomitan
|
||||
// backend responses (chrome.runtime.sendMessage level) and raw MeCab tokens
|
||||
// observed while tokenizing it live, plus the annotated tokens the full
|
||||
// pipeline produced. Replay runs the real injected scanning scripts in a vm
|
||||
// against the recorded responses, so merge/enrichment/annotation/filtering
|
||||
// are exercised end-to-end without Electron or dictionaries.
|
||||
|
||||
export interface GoldenFixtureKnownWord {
|
||||
text: string;
|
||||
reading?: string;
|
||||
}
|
||||
|
||||
export interface GoldenFixtureConfig {
|
||||
knownWords?: Array<string | GoldenFixtureKnownWord>;
|
||||
knownWordMatchMode?: NPlusOneMatchMode;
|
||||
jlptLevels?: Record<string, JlptLevel>;
|
||||
localFrequencyRanks?: Record<string, number>;
|
||||
knownWordsEnabled?: boolean;
|
||||
nPlusOneEnabled?: boolean;
|
||||
jlptEnabled?: boolean;
|
||||
nameMatchEnabled?: boolean;
|
||||
frequencyEnabled?: boolean;
|
||||
frequencyMatchMode?: FrequencyDictionaryMatchMode;
|
||||
minSentenceWordsForNPlusOne?: number;
|
||||
}
|
||||
|
||||
export interface GoldenRecordedMessage {
|
||||
action: string;
|
||||
params: unknown;
|
||||
/** Raw response object the page callback received: { result } or { error }. */
|
||||
response: unknown;
|
||||
}
|
||||
|
||||
export interface GoldenRecordedScript {
|
||||
sha256: string;
|
||||
/** Short classification of the script, for debugging only. */
|
||||
marker: string;
|
||||
result: unknown;
|
||||
}
|
||||
|
||||
export interface GoldenFixtureRecording {
|
||||
messages: GoldenRecordedMessage[];
|
||||
scripts: GoldenRecordedScript[];
|
||||
/** Raw MeCab tokens keyed by the exact text passed to tokenizeWithMecab. */
|
||||
mecab: Record<string, Token[] | null>;
|
||||
}
|
||||
|
||||
export interface GoldenExpectedToken {
|
||||
surface: string;
|
||||
reading: string;
|
||||
headword: string;
|
||||
headwordReading?: string;
|
||||
startPos: number;
|
||||
endPos: number;
|
||||
partOfSpeech: PartOfSpeech;
|
||||
pos1?: string;
|
||||
pos2?: string;
|
||||
pos3?: string;
|
||||
isKnown: boolean;
|
||||
isNPlusOneTarget: boolean;
|
||||
isNameMatch?: true;
|
||||
isUnparsedRun?: true;
|
||||
jlptLevel?: JlptLevel;
|
||||
frequencyRank?: number;
|
||||
}
|
||||
|
||||
export interface GoldenFixture {
|
||||
name: string;
|
||||
description?: string;
|
||||
issueRefs?: string[];
|
||||
recordedAt?: string;
|
||||
input: { text: string };
|
||||
config: GoldenFixtureConfig;
|
||||
recording: GoldenFixtureRecording;
|
||||
expected: { tokens: GoldenExpectedToken[] | null };
|
||||
}
|
||||
|
||||
// Only these backend actions are requested by the scanning/frequency scripts;
|
||||
// everything else in the recorded page traffic is Yomitan's own chatter.
|
||||
const GOLDEN_MESSAGE_ACTIONS: ReadonlySet<string> = new Set([
|
||||
'optionsGetFull',
|
||||
'getDictionaryInfo',
|
||||
'parseText',
|
||||
'termsFind',
|
||||
'getTermFrequencies',
|
||||
]);
|
||||
|
||||
// Field candidates appendDictionaryNames() in the injected helpers reads off
|
||||
// definitions/pronunciations — the only reason those arrays are consulted.
|
||||
const DICTIONARY_NAME_FIELDS = [
|
||||
'dictionary',
|
||||
'dictionaryName',
|
||||
'name',
|
||||
'title',
|
||||
'dictionaryTitle',
|
||||
'dictionaryAlias',
|
||||
] as const;
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return Boolean(value) && typeof value === 'object' && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function pruneToDictionaryNameFields(value: unknown): unknown {
|
||||
if (!isRecord(value)) {
|
||||
return {};
|
||||
}
|
||||
const pruned: Record<string, unknown> = {};
|
||||
for (const field of DICTIONARY_NAME_FIELDS) {
|
||||
if (typeof value[field] === 'string') {
|
||||
pruned[field] = value[field];
|
||||
}
|
||||
}
|
||||
return pruned;
|
||||
}
|
||||
|
||||
// Fields getBestFrequencyRank() and the frequency grouping helpers read off a
|
||||
// termsFind frequency item, beyond the dictionary-name candidates.
|
||||
const FREQUENCY_ITEM_FIELDS = [
|
||||
'headwordIndex',
|
||||
'displayValue',
|
||||
'displayValueParsed',
|
||||
'frequency',
|
||||
'dictionaryIndex',
|
||||
'term',
|
||||
'reading',
|
||||
] as const;
|
||||
|
||||
function pruneFrequencyItem(item: unknown): unknown {
|
||||
if (!isRecord(item)) {
|
||||
return item;
|
||||
}
|
||||
const pruned = pruneToDictionaryNameFields(item) as Record<string, unknown>;
|
||||
for (const field of FREQUENCY_ITEM_FIELDS) {
|
||||
if (item[field] !== undefined) {
|
||||
pruned[field] = item[field];
|
||||
}
|
||||
}
|
||||
return pruned;
|
||||
}
|
||||
|
||||
function pruneTermsFindEntry(entry: unknown): unknown {
|
||||
if (!isRecord(entry)) {
|
||||
return entry;
|
||||
}
|
||||
const pruned: Record<string, unknown> = { ...entry };
|
||||
if (Array.isArray(entry.definitions)) {
|
||||
pruned.definitions = entry.definitions.map(pruneToDictionaryNameFields);
|
||||
}
|
||||
if (Array.isArray(entry.pronunciations)) {
|
||||
pruned.pronunciations = entry.pronunciations.map(pruneToDictionaryNameFields);
|
||||
}
|
||||
if (Array.isArray(entry.frequencies)) {
|
||||
pruned.frequencies = entry.frequencies.map(pruneFrequencyItem);
|
||||
}
|
||||
return pruned;
|
||||
}
|
||||
|
||||
function pruneMessageResult(action: string, result: unknown): unknown {
|
||||
if (action === 'optionsGetFull' && isRecord(result)) {
|
||||
const profiles = Array.isArray(result.profiles) ? result.profiles : [];
|
||||
return {
|
||||
profileCurrent: result.profileCurrent,
|
||||
profiles: profiles.map((profile) => {
|
||||
if (!isRecord(profile) || !isRecord(profile.options)) {
|
||||
return {};
|
||||
}
|
||||
const options = profile.options;
|
||||
return {
|
||||
options: {
|
||||
scanning: isRecord(options.scanning) ? { length: options.scanning.length } : {},
|
||||
dictionaries: Array.isArray(options.dictionaries)
|
||||
? options.dictionaries.map((dictionary) =>
|
||||
isRecord(dictionary)
|
||||
? {
|
||||
name: dictionary.name,
|
||||
enabled: dictionary.enabled,
|
||||
id: dictionary.id,
|
||||
}
|
||||
: {},
|
||||
)
|
||||
: [],
|
||||
},
|
||||
};
|
||||
}),
|
||||
};
|
||||
}
|
||||
if (action === 'getDictionaryInfo' && Array.isArray(result)) {
|
||||
return result.map((entry) =>
|
||||
isRecord(entry) ? { title: entry.title, frequencyMode: entry.frequencyMode } : {},
|
||||
);
|
||||
}
|
||||
if (action === 'termsFind' && isRecord(result) && Array.isArray(result.dictionaryEntries)) {
|
||||
return {
|
||||
...result,
|
||||
dictionaryEntries: result.dictionaryEntries.map(pruneTermsFindEntry),
|
||||
};
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Strip recorded page traffic down to what replay can ever request: drop
|
||||
// Yomitan's own background messages and the heavyweight response fields
|
||||
// (definition glossaries, full options blobs) the injected helpers never read.
|
||||
// Prune safety is verified by the replay tests themselves — a pruned fixture
|
||||
// must still reproduce its recorded expectation.
|
||||
export function pruneGoldenMessages(messages: GoldenRecordedMessage[]): GoldenRecordedMessage[] {
|
||||
return messages
|
||||
.filter((message) => GOLDEN_MESSAGE_ACTIONS.has(message.action))
|
||||
.map((message) => {
|
||||
if (isRecord(message.response) && 'result' in message.response) {
|
||||
return {
|
||||
...message,
|
||||
response: {
|
||||
...message.response,
|
||||
result: pruneMessageResult(message.action, message.response.result),
|
||||
},
|
||||
};
|
||||
}
|
||||
return message;
|
||||
});
|
||||
}
|
||||
|
||||
export function hashInjectedScript(script: string): string {
|
||||
return createHash('sha256').update(script).digest('hex');
|
||||
}
|
||||
|
||||
export function classifyInjectedScript(script: string): string {
|
||||
const markers = [
|
||||
'optionsGetFull',
|
||||
'parseText',
|
||||
'termsFind',
|
||||
'getTermFrequencies',
|
||||
'setAllSettings',
|
||||
];
|
||||
const found = markers.filter((marker) => script.includes(marker));
|
||||
return found.length > 0 ? found.join('+') : 'unknown';
|
||||
}
|
||||
|
||||
function stableStringify(value: unknown): string {
|
||||
// Recorded params pass through JSON, which turns undefined into null.
|
||||
return JSON.stringify(value ?? null, (_key, entry: unknown) => {
|
||||
if (entry && typeof entry === 'object' && !Array.isArray(entry)) {
|
||||
const record = entry as Record<string, unknown>;
|
||||
const sorted: Record<string, unknown> = {};
|
||||
for (const key of Object.keys(record).sort()) {
|
||||
sorted[key] = record[key];
|
||||
}
|
||||
return sorted;
|
||||
}
|
||||
return entry;
|
||||
});
|
||||
}
|
||||
|
||||
export interface FixtureLookups {
|
||||
isKnownWord: (
|
||||
text: string,
|
||||
reading?: string,
|
||||
options?: { allowReadingOnlyMatch?: boolean },
|
||||
) => boolean;
|
||||
getJlptLevel: (text: string) => JlptLevel | null;
|
||||
getFrequencyRank: (term: string) => number | null;
|
||||
}
|
||||
|
||||
// Deliberately simple known-word semantics (exact text, optional reading
|
||||
// constraint) shared by the recorder and the replay harness so both sides of
|
||||
// a fixture agree. The production reading-aware cache is exercised by its own
|
||||
// unit tests, not by the golden corpus.
|
||||
export function buildFixtureLookups(config: GoldenFixtureConfig): FixtureLookups {
|
||||
const readingsByText = new Map<string, Set<string> | null>();
|
||||
for (const entry of config.knownWords ?? []) {
|
||||
const text = typeof entry === 'string' ? entry : entry.text;
|
||||
const reading = typeof entry === 'string' ? undefined : entry.reading;
|
||||
if (!reading) {
|
||||
readingsByText.set(text, null);
|
||||
continue;
|
||||
}
|
||||
const existing = readingsByText.get(text);
|
||||
if (existing === null) {
|
||||
continue;
|
||||
}
|
||||
const readings = existing ?? new Set<string>();
|
||||
readings.add(reading);
|
||||
readingsByText.set(text, readings);
|
||||
}
|
||||
|
||||
return {
|
||||
isKnownWord: (text, reading) => {
|
||||
if (!readingsByText.has(text)) {
|
||||
return false;
|
||||
}
|
||||
const readings = readingsByText.get(text);
|
||||
if (readings === null || readings === undefined) {
|
||||
return true;
|
||||
}
|
||||
return reading === undefined || readings.has(reading);
|
||||
},
|
||||
getJlptLevel: (text) => config.jlptLevels?.[text] ?? null,
|
||||
getFrequencyRank: (term) => config.localFrequencyRanks?.[term] ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
// Config-driven deps getters shared by the recorder and the replay harness so
|
||||
// a fixture is tokenized under identical toggles in both directions.
|
||||
export function fixtureConfigGetters(config: GoldenFixtureConfig) {
|
||||
return {
|
||||
getKnownWordMatchMode: () => config.knownWordMatchMode ?? ('headword' as NPlusOneMatchMode),
|
||||
getKnownWordsEnabled: () => config.knownWordsEnabled !== false,
|
||||
getNPlusOneEnabled: () => config.nPlusOneEnabled !== false,
|
||||
getJlptEnabled: () => config.jlptEnabled !== false,
|
||||
getNameMatchEnabled: () => config.nameMatchEnabled !== false,
|
||||
getNameMatchImagesEnabled: () => false,
|
||||
getFrequencyDictionaryEnabled: () => config.frequencyEnabled !== false,
|
||||
getFrequencyDictionaryMatchMode: () =>
|
||||
config.frequencyMatchMode ?? ('headword' as FrequencyDictionaryMatchMode),
|
||||
getMinSentenceWordsForNPlusOne: () => config.minSentenceWordsForNPlusOne ?? 3,
|
||||
};
|
||||
}
|
||||
|
||||
interface ReplayMessageEntry extends GoldenRecordedMessage {
|
||||
used: boolean;
|
||||
paramsKey: string;
|
||||
}
|
||||
|
||||
export interface ReplayMessageStore {
|
||||
handle: (action: string, params: unknown) => unknown;
|
||||
unusedCount: () => number;
|
||||
}
|
||||
|
||||
export function createReplayMessageStore(messages: GoldenRecordedMessage[]): ReplayMessageStore {
|
||||
const entries: ReplayMessageEntry[] = messages.map((message) => ({
|
||||
...message,
|
||||
used: false,
|
||||
paramsKey: stableStringify(message.params),
|
||||
}));
|
||||
|
||||
return {
|
||||
handle: (action, params) => {
|
||||
const paramsKey = stableStringify(params);
|
||||
const match =
|
||||
entries.find((e) => !e.used && e.action === action && e.paramsKey === paramsKey) ??
|
||||
entries.find((e) => !e.used && e.action === action) ??
|
||||
entries.find((e) => e.action === action && e.paramsKey === paramsKey);
|
||||
if (!match) {
|
||||
throw new Error(
|
||||
`golden fixture replay: no recorded response for action "${action}" params=${paramsKey}`,
|
||||
);
|
||||
}
|
||||
match.used = true;
|
||||
return match.response;
|
||||
},
|
||||
unusedCount: () => entries.filter((e) => !e.used).length,
|
||||
};
|
||||
}
|
||||
|
||||
async function runInjectedScriptInVm(script: string, store: ReplayMessageStore): Promise<unknown> {
|
||||
return await vm.runInNewContext(script, {
|
||||
chrome: {
|
||||
runtime: {
|
||||
lastError: null,
|
||||
sendMessage: (
|
||||
payload: { action?: string; params?: unknown },
|
||||
callback: (response: unknown) => void,
|
||||
) => {
|
||||
callback(store.handle(payload.action ?? '', payload.params));
|
||||
},
|
||||
},
|
||||
},
|
||||
Array,
|
||||
Boolean,
|
||||
Date,
|
||||
Error,
|
||||
JSON,
|
||||
Map,
|
||||
Math,
|
||||
Number,
|
||||
Object,
|
||||
Promise,
|
||||
RegExp,
|
||||
Set,
|
||||
String,
|
||||
});
|
||||
}
|
||||
|
||||
export function createReplayTokenizerDeps(fixture: GoldenFixture): TokenizerServiceDeps {
|
||||
const store = createReplayMessageStore(fixture.recording.messages);
|
||||
const scriptResults = new Map(
|
||||
fixture.recording.scripts.map((entry) => [entry.sha256, entry] as const),
|
||||
);
|
||||
|
||||
const parserWindow = {
|
||||
isDestroyed: () => false,
|
||||
webContents: {
|
||||
executeJavaScript: async (script: string) => {
|
||||
try {
|
||||
return await runInjectedScriptInVm(script, store);
|
||||
} catch (vmError) {
|
||||
const recorded = scriptResults.get(hashInjectedScript(script));
|
||||
if (recorded) {
|
||||
return recorded.result;
|
||||
}
|
||||
throw new Error(
|
||||
`golden fixture "${fixture.name}": injected script (${classifyInjectedScript(script)}) failed in vm replay and has no recorded script-level result: ${(vmError as Error).message}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const lookups = buildFixtureLookups(fixture.config);
|
||||
const deps = createTokenizerDepsRuntime({
|
||||
getYomitanExt: () => ({ id: 'golden-fixture-extension' }) as never,
|
||||
getYomitanParserWindow: () => parserWindow as never,
|
||||
setYomitanParserWindow: () => undefined,
|
||||
getYomitanParserReadyPromise: () => null,
|
||||
setYomitanParserReadyPromise: () => undefined,
|
||||
getYomitanParserInitPromise: () => null,
|
||||
setYomitanParserInitPromise: () => undefined,
|
||||
isKnownWord: lookups.isKnownWord,
|
||||
getJlptLevel: lookups.getJlptLevel,
|
||||
getFrequencyRank: lookups.getFrequencyRank,
|
||||
...fixtureConfigGetters(fixture.config),
|
||||
getMecabTokenizer: () => ({
|
||||
tokenize: async (text: string) => fixture.recording.mecab[text] ?? null,
|
||||
}),
|
||||
});
|
||||
|
||||
return {
|
||||
...deps,
|
||||
// Pin the synchronous enrichment stage so record and replay share one
|
||||
// implementation and no worker spawns inside bun test.
|
||||
enrichTokensWithMecab: async (tokens, mecabTokens) =>
|
||||
enrichTokensWithMecabPos1(tokens, mecabTokens),
|
||||
};
|
||||
}
|
||||
|
||||
export function projectGoldenTokens(tokens: MergedToken[] | null): GoldenExpectedToken[] | null {
|
||||
if (!tokens) {
|
||||
return null;
|
||||
}
|
||||
return tokens.map((token) => {
|
||||
const projected: GoldenExpectedToken = {
|
||||
surface: token.surface,
|
||||
reading: token.reading,
|
||||
headword: token.headword,
|
||||
startPos: token.startPos,
|
||||
endPos: token.endPos,
|
||||
partOfSpeech: token.partOfSpeech,
|
||||
isKnown: token.isKnown,
|
||||
isNPlusOneTarget: token.isNPlusOneTarget,
|
||||
};
|
||||
if (token.headwordReading !== undefined) projected.headwordReading = token.headwordReading;
|
||||
if (token.pos1 !== undefined) projected.pos1 = token.pos1;
|
||||
if (token.pos2 !== undefined) projected.pos2 = token.pos2;
|
||||
if (token.pos3 !== undefined) projected.pos3 = token.pos3;
|
||||
if (token.isNameMatch === true) projected.isNameMatch = true;
|
||||
if (token.isUnparsedRun === true) projected.isUnparsedRun = true;
|
||||
if (token.jlptLevel !== undefined) projected.jlptLevel = token.jlptLevel;
|
||||
if (token.frequencyRank !== undefined) projected.frequencyRank = token.frequencyRank;
|
||||
return projected;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
import { tokenizeSubtitle } from '../tokenizer';
|
||||
import {
|
||||
createReplayTokenizerDeps,
|
||||
projectGoldenTokens,
|
||||
type GoldenFixture,
|
||||
} from './golden-corpus-harness';
|
||||
|
||||
// End-to-end regression corpus: each fixture replays recorded Yomitan backend
|
||||
// responses and raw MeCab tokens through the real tokenizeSubtitle pipeline
|
||||
// (scan-token merge, MeCab enrichment, frequency ranks, annotation stage,
|
||||
// noise suppression) and asserts the final annotated tokens.
|
||||
//
|
||||
// Record new fixtures with:
|
||||
// bun run record-tokenizer-fixture:electron -- --name <slug> <text>
|
||||
// See __fixtures__/golden/README.md for the format and options.
|
||||
|
||||
const FIXTURE_DIR = path.join(__dirname, '__fixtures__', 'golden');
|
||||
|
||||
function loadFixtures(): GoldenFixture[] {
|
||||
if (!fs.existsSync(FIXTURE_DIR)) {
|
||||
return [];
|
||||
}
|
||||
return fs
|
||||
.readdirSync(FIXTURE_DIR)
|
||||
.filter((entry) => entry.endsWith('.json'))
|
||||
.sort()
|
||||
.map((entry) => {
|
||||
const filePath = path.join(FIXTURE_DIR, entry);
|
||||
const fixture = JSON.parse(fs.readFileSync(filePath, 'utf8')) as GoldenFixture;
|
||||
assert.ok(fixture.name, `${entry}: fixture is missing "name"`);
|
||||
assert.equal(
|
||||
`${fixture.name}.json`,
|
||||
entry,
|
||||
`${entry}: fixture "name" must match its file name`,
|
||||
);
|
||||
return fixture;
|
||||
});
|
||||
}
|
||||
|
||||
const fixtures = loadFixtures();
|
||||
|
||||
// Fixtures are not copied into dist by tsc; the corpus only runs from src.
|
||||
test('golden corpus has fixtures', { skip: !fs.existsSync(FIXTURE_DIR) }, () => {
|
||||
assert.ok(
|
||||
fixtures.length > 0,
|
||||
`no golden fixtures found in ${FIXTURE_DIR}; record one with record-tokenizer-fixture:electron`,
|
||||
);
|
||||
});
|
||||
|
||||
for (const fixture of fixtures) {
|
||||
const label = fixture.issueRefs?.length
|
||||
? `${fixture.name} (${fixture.issueRefs.join(', ')})`
|
||||
: fixture.name;
|
||||
test(`golden: ${label}`, async () => {
|
||||
const deps = createReplayTokenizerDeps(fixture);
|
||||
const subtitleData = await tokenizeSubtitle(fixture.input.text, deps);
|
||||
assert.deepEqual(
|
||||
projectGoldenTokens(subtitleData.tokens),
|
||||
fixture.expected.tokens,
|
||||
fixture.description
|
||||
? `${fixture.name}: ${fixture.description}`
|
||||
: `${fixture.name}: annotated tokens diverged from recorded expectation`,
|
||||
);
|
||||
});
|
||||
}
|
||||
Reference in New Issue
Block a user