mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-05 07:21:34 -07:00
Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
afa66ee508
|
|||
|
3c24597724
|
|||
|
d878d8bf4f
|
|||
|
8683961967
|
|||
|
c9baaeea17
|
|||
|
2003efa235
|
|||
|
f43674cc39
|
|||
|
b0a2ce6e8a
|
|||
|
030c94934e
|
|||
|
e7cef039f3
|
|||
| fe4dacc1e7 | |||
| b08cd0db35 | |||
| bffb1c5982 | |||
| 5b8848518a | |||
|
176edd67f1
|
|||
| 4d65dec340 | |||
|
6607c333bc
|
|||
|
b2bbf1ae12
|
|||
| b204d4dd6e | |||
| 89ed675935 | |||
|
1995200e76
|
@@ -1,5 +1,50 @@
|
||||
# Changelog
|
||||
|
||||
## v0.19.1 (2026-08-01)
|
||||
|
||||
### Added
|
||||
- Word Card Type: Adds a setting (Settings > Mining/Anki > Kiku/Lapis Features > "Word Card Type") to choose which card-type flag SubMiner marks on Kiku/Lapis word cards — `word-and-sentence` (default), `click`, `sentence`, `audio`, or `none`. Click cards (`IsClickCard`) can now be flagged, and setting any card-type flag clears the others so a note can't claim two types at once.
|
||||
|
||||
### Fixed
|
||||
- Yomitan Popup: Fixes the macOS Yomitan popup going inert after mining a card — clicks outside the popup no longer pass through to mpv, and scrolling over the popup scrolls its definitions instead of seeking playback.
|
||||
- YouTube Playlist Links: Fixes opening a video from a playlist URL (e.g. a Watch Later link with `list=`/`index=`) timing out while probing subtitles, metadata, or the playback URL.
|
||||
|
||||
## v0.19.0 (2026-07-29)
|
||||
|
||||
### Added
|
||||
- Anki Maturity Highlighting: Known-word subtitle highlights can now be colored by Anki card maturity (new, learning, young, mature), similar to asbplayer. Tier thresholds and colors are configurable, with a runtime toggle and an updated help legend.
|
||||
- Post-Playback Menu: After a watch-history episode ends, the fzf/rofi launcher returns to that series with options to play the previous or next episode, rewatch, pick another episode, or quit. The pre-playback series menu now offers the previous episode too.
|
||||
- Delete Library Entries: The stats Library detail view can now delete an entire title in one step (episodes, sessions, subtitle lines, rollups, cover art, and vocabulary counts). Delete progress is now shown app-wide via a progress bar and status toast instead of disappearing when you switch tabs.
|
||||
- Cross-Machine Sync: Added SSH-based syncing of stats and watch history between machines, available from the tray ("Sync Stats & History") or `subminer sync`, with saved devices, per-host sync direction, background auto-sync, connection testing, manual snapshots, and support for Windows remotes.
|
||||
- TsukiHime Subtitle Downloads: Added subtitle downloads for the current video via TsukiHime, loading Japanese as the primary track and your configured secondary language directly into mpv.
|
||||
|
||||
### Changed
|
||||
- Clipboard-Video Shortcut: The "append clipboard video to queue" shortcut is now configurable.
|
||||
|
||||
### Fixed
|
||||
- AniList Season Resolution: Season 2+ files now resolve to the correct AniList entry instead of silently falling back to season 1 (which mismatched character dictionaries and watch progress). Manual overrides now stay scoped per season, fix both the dictionary and progress tracking together, and also correct per-season cover art.
|
||||
- Subtitle Annotation Accuracy: Fixed several annotation edge cases, including inconsistent POS exclusions on merged quote-particle tokens, dropped annotations on supplementary-plane kanji, katakana punctuation wrongly treated as noise, and certain kanji vocabulary losing N+1 highlighting eligibility.
|
||||
- AnkiConnect Proxy Port Conflict: Video startup no longer crashes when another process already holds the configured AnkiConnect proxy port; a notification now explains how to resolve it.
|
||||
- AppImage Quit Crash: Fixed a "Service Crash" desktop notification appearing after closing a video when running the Linux AppImage.
|
||||
- Autoplay Pause Timing: Fixed playback resuming a few seconds before subtitle tokenization warmup finished, most noticeable when resuming mid-episode or when a cue starts within the first two seconds.
|
||||
- Stats Known-Word Count: Fixed stats reporting 0 known words for every session after the known-word cache format changed.
|
||||
- Stats Library Cover Art: Relinking a title to a different AniList entry now updates its cover in the Library grid immediately instead of leaving a stale, mismatched cover cached.
|
||||
- Rofi Prompt Spacing: Rofi menu prompts now keep a space before the input field instead of running into the placeholder text.
|
||||
- Stats Settings & Reliability: Hardened stats settings validation (nested/legacy AnkiConnect config now falls back safely instead of breaking) and stats routes against malformed requests and other edge cases.
|
||||
- Stats Delete Performance: Deleting sessions, episodes, and library entries is now dramatically faster and no longer stalls playback (e.g. a 12-episode title dropped from about a minute to under a second on a large library); the Vocabulary tab also loads much faster. The first launch after upgrading runs a one-time database migration.
|
||||
|
||||
<details>
|
||||
<summary>Internal changes</summary>
|
||||
|
||||
### Internal
|
||||
- Added a golden-file regression test corpus for the tokenizer/annotation pipeline, plus scripts to record new fixtures and diff against stock Yomitan.
|
||||
- Consolidated renderer modal state handling into a descriptor registry.
|
||||
- Consolidated CI quality checks (PR, stable, and prerelease) into one reusable workflow with mpv plugin tests and dependency audits.
|
||||
- Removed the unused stats IPC transport and unified stats dashboard HTTP types with the backend contract.
|
||||
- Added a script to verify known-word highlight tiers against live Anki data outside of playback.
|
||||
|
||||
</details>
|
||||
|
||||
## v0.18.0 (2026-07-10)
|
||||
|
||||
### Added
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
"eslint": "^10.8.0",
|
||||
"prettier": "^3.8.1",
|
||||
"typescript": "^5.9.3",
|
||||
"undici": "7.28.0",
|
||||
"undici": "7.29.0",
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -36,8 +36,9 @@
|
||||
"overrides": {
|
||||
"@xmldom/xmldom": "0.8.13",
|
||||
"app-builder-lib": "26.15.3",
|
||||
"brace-expansion": "5.0.8",
|
||||
"brace-expansion": "5.0.9",
|
||||
"electron-builder-squirrel-windows": "26.15.3",
|
||||
"fast-uri": "3.1.5",
|
||||
"form-data": "4.0.6",
|
||||
"ip-address": "10.2.0",
|
||||
"js-yaml": "4.3.0",
|
||||
@@ -46,6 +47,7 @@
|
||||
"picomatch": "4.0.4",
|
||||
"tar": "7.5.21",
|
||||
"tmp": "0.2.7",
|
||||
"undici": "7.29.0",
|
||||
},
|
||||
"packages": {
|
||||
"@discordjs/collection": ["@discordjs/collection@2.1.1", "", {}, "sha512-LiSusze9Tc7qF03sLCujF5iZp7K+vRNEDBZ86FT9aQAv3vxMLihUvKvpsCWiQ2DJq1tVckopKm1rxomgNUc9hg=="],
|
||||
@@ -266,7 +268,7 @@
|
||||
|
||||
"boolean": ["boolean@3.2.0", "", {}, "sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw=="],
|
||||
|
||||
"brace-expansion": ["brace-expansion@5.0.8", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-JZyDyq3D4AUifKTPOB7DELf6XsB3WdPuNxCtob1vFXPsSXhdAiHBWJ/tJ8HAc9aH84BK+5JFZLNkJKx3G9kzQg=="],
|
||||
"brace-expansion": ["brace-expansion@5.0.9", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg=="],
|
||||
|
||||
"buffer-from": ["buffer-from@1.1.2", "", {}, "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ=="],
|
||||
|
||||
@@ -404,7 +406,7 @@
|
||||
|
||||
"fast-levenshtein": ["fast-levenshtein@2.0.6", "", {}, "sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw=="],
|
||||
|
||||
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
|
||||
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
|
||||
|
||||
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
|
||||
|
||||
@@ -714,7 +716,7 @@
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici": ["undici@7.28.0", "", {}, "sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA=="],
|
||||
"undici": ["undici@7.29.0", "", {}, "sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw=="],
|
||||
|
||||
"undici-types": ["undici-types@7.16.0", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
|
||||
|
||||
@@ -754,8 +756,6 @@
|
||||
|
||||
"@discordjs/rest/discord-api-types": ["discord-api-types@0.38.43", "", {}, "sha512-sSoBf/nK6m7BGtw65mi+QBuvEWaHE8MMziFLqWL+gT6ME/BLg34dRSVKS3Husx40uU06bvxUc3/X+D9Y6/zAbw=="],
|
||||
|
||||
"@discordjs/rest/undici": ["undici@6.27.0", "", {}, "sha512-YmfV3YnEDzXRC5lZ2jWtWWHKGUm1zIt8AhesR1tens+HTNv+YZlN/dp6G727LOvMJ8xjP9Be7Y2Sdr96LDm+pg=="],
|
||||
|
||||
"@discordjs/util/discord-api-types": ["discord-api-types@0.38.43", "", {}, "sha512-sSoBf/nK6m7BGtw65mi+QBuvEWaHE8MMziFLqWL+gT6ME/BLg34dRSVKS3Husx40uU06bvxUc3/X+D9Y6/zAbw=="],
|
||||
|
||||
"@electron/asar/commander": ["commander@5.1.0", "", {}, "sha512-P0CysNDQ7rtVw4QIQtm+MRxV66vKFSvlsQvGYXZWR3qFU0jlMKHZZZgw8e+8DSah4UDKMqnknRDQz+xuQXQ/Zg=="],
|
||||
@@ -808,8 +808,6 @@
|
||||
|
||||
"node-gyp/env-paths": ["env-paths@2.2.1", "", {}, "sha512-+h1lkLKhZMTYjog1VEpJNG7NZJWcuc2DDk/qsqSTRRCOXiLjeQ1d1/udrUGhqMxUgAlwKNZ0cf2uqan5GLuS2A=="],
|
||||
|
||||
"node-gyp/undici": ["undici@6.27.0", "", {}, "sha512-YmfV3YnEDzXRC5lZ2jWtWWHKGUm1zIt8AhesR1tens+HTNv+YZlN/dp6G727LOvMJ8xjP9Be7Y2Sdr96LDm+pg=="],
|
||||
|
||||
"node-gyp/which": ["which@6.0.1", "", { "dependencies": { "isexe": "^4.0.0" }, "bin": { "node-which": "bin/which.js" } }, "sha512-oGLe46MIrCRqX7ytPUf66EAYvdeMIZYn3WaocqqKZAxrBpkqHfL/qvTyJ/bTk5+AqHCjXmrv3CEWgy368zhRUg=="],
|
||||
|
||||
"pkijs/@noble/hashes": ["@noble/hashes@1.4.0", "", {}, "sha512-V1JJ1WTRUqHHrOSh597hURcMqVKVGL/ea3kv0gSnEdsEZ0/+VyPghM1lMNGc00z7CIQorSvbKpuJkxvuHbvdbg=="],
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
type: added
|
||||
area: overlay
|
||||
|
||||
- Known-word subtitle highlights can now be colored by Anki card maturity (new, learning, young, mature) like asbplayer. Enable with `ankiConnect.knownWords.maturityEnabled`; the mature interval threshold (`matureThresholdDays`, default 21) and the four tier colors (`subtitleStyle.knownWordMaturityColors`) are configurable, and a runtime option toggles it in-session. The session help color legend shows the four tier colors while maturity highlighting is on.
|
||||
- Tiers follow Anki's own card counts: the interval tiers exclude cards in the learning/relearning queue, so a lapsed card shows the learning color instead of young (its interval is reset to at least 1 day, which previously made the learning tier unreachable). A note with a mature card alongside a relearning card still shows mature.
|
||||
@@ -1,5 +0,0 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Applied configured primary POS exclusions consistently to merged trailing quote-particle tokens, preserved annotations for supplementary-plane kanji, and stopped treating katakana punctuation as kana-only annotation noise.
|
||||
- Kept kanji vocabulary tagged `名詞/非自立` eligible for N+1 highlighting, consistent with frequency, JLPT, and vocabulary persistence.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: logging
|
||||
|
||||
- Background startup now respects the configured logging level when `--log-level` is not explicitly provided.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: changed
|
||||
area: shortcuts
|
||||
|
||||
- Made the clipboard-video playlist shortcut configurable through `shortcuts.appendClipboardVideoToQueue`.
|
||||
@@ -1,8 +0,0 @@
|
||||
type: fixed
|
||||
area: anilist
|
||||
|
||||
- Season 2 and later files now resolve to the right AniList entry. AniList has no numbered seasons (sequels are separate entries titled `Zoku`, `Kan`, `2nd Season`), so the old `<title> Season N` search returned nothing and silently fell back to the season 1 entry: an Oregairu S03 episode loaded the season 1 character dictionary and pushed watch progress onto season 1. SubMiner now searches the base title and walks AniList `SEQUEL` relations to reach the requested season, ordering the franchise's TV entries by air date when the relation chain is incomplete.
|
||||
- When the season still cannot be located, SubMiner no longer writes progress to the season 1 entry; it skips the update and points at the manual AniList override instead. The character dictionary logs the same warning and states which entry it fell back to.
|
||||
- Character dictionary overrides are now scoped by detected season as well as directory, so seasons kept in one flat folder no longer share (or overwrite) each other's override, and an override now also pins the entry used for AniList watch progress: correcting a wrong match once fixes both the dictionary and progress tracking.
|
||||
- Cover art lookup uses the same season-aware resolution, so per-season art no longer falls back to the season 1 cover.
|
||||
- An unresolvable season is no longer cached as if it were a normal match, so the fallback cannot silently stick for every later episode of that series.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: anki
|
||||
|
||||
- Prevented video startup from crashing when another process already owns the configured AnkiConnect proxy port, and added a notification explaining how to resolve the conflict.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: app
|
||||
|
||||
- Fixed "Service Crash" desktop notifications (KDE DrKonqi) after closing a video when running the Linux AppImage: the short-lived background bootstrap spawned a Chromium GPU child that outlived it (surviving `app.exit`) and died with SIGBUS at session end when the bootstrap's FUSE mount was finally released. The bootstrap now runs with the GPU in-process so it leaves no children behind, and the detached app's mount remains supervised until its Chromium children finish. Set `SUBMINER_NO_APPIMAGE_MOUNT_KEEPALIVE=1` to disable the detached-app mount supervisor.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: character dictionary
|
||||
|
||||
- Kept manual AniList overrides active across episodes in the same season directory when filename guesses differ.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Fixed `mpv.pauseUntilOverlayReady` releasing playback seconds before tokenization warmup finished: startup subtitle priming emits the current cue untokenized so the overlay can paint early, and that emission was treated as the autoplay-readiness signal as soon as the overlay window loaded. The autoplay gate now ignores untokenized subtitle payloads while tokenization warmup is pending, so playback resumes only after the first tokenized delivery (or the post-warmup release). Most visible when resuming mid-episode or when a subtitle cue starts within the first two seconds.
|
||||
@@ -1,5 +0,0 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Stats reported 0 known words for every session after the known-word cache gained maturity tiers. The stats server carried its own copy of the cache parser that only recognized versions up to 3, so the new v4 file was read as "no cache" rather than as a format it should understand.
|
||||
- The cache format, its parser, and the derived known-word set now live in one module that both the cache manager and the stats server read, and the version dispatch ends in an exhaustive check so a future format bump fails the build instead of silently reporting zero. A cache that exists but does not parse now logs a warning rather than passing for an empty one.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Relinking a title to a different AniList entry now updates its cover in the stats Library grid too. The detail view picked up the new art immediately while the grid kept the previous entry's image, so several unrelated titles could end up sharing one wrong cover. The library list is refetched after a relink, and cover responses now carry an ETag and revalidate instead of being cached for a day, so a stale image can no longer be served from the browser cache.
|
||||
@@ -1,7 +0,0 @@
|
||||
type: internal
|
||||
area: tokenizer
|
||||
|
||||
- Added a golden-file regression corpus for the tokenizer/annotation pipeline: recorded Yomitan backend responses and MeCab tokens replay through the real tokenizeSubtitle pipeline in bun tests without Electron or dictionaries.
|
||||
- Added `record-tokenizer-fixture:electron` script to capture new fixtures from a live Yomitan/MeCab session, with flags for known words, JLPT levels, and annotation toggles.
|
||||
- Seeded eleven fixtures covering the #147–#156 regression classes (grammar-helper suppression, lexical くれる, kanji non-independent nouns, N+1 targeting, reading collisions, unparsed runs, ordinal/honorific prefixes).
|
||||
- Added `compare-yomitan-api:electron` script that diffs SubMiner tokenization against a stock Yomitan instance via the yomitan-api bridge (segmentation, readings, headword forms).
|
||||
@@ -1,5 +0,0 @@
|
||||
type: added
|
||||
area: launcher
|
||||
|
||||
- After a watch-history episode ends or mpv closes, the fzf or rofi launcher returns to that series with options to play the previous episode, rewatch, play the next episode, select another episode, or quit SubMiner. Previous and Next continue across season directories.
|
||||
- The action menu shown right after picking a series from `subminer -H` now also offers the previous episode, matching the menu shown after playback.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: internal
|
||||
area: overlay
|
||||
|
||||
- Consolidated renderer modal state handling into a descriptor registry.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: internal
|
||||
area: release
|
||||
|
||||
- Consolidated pull request, stable release, and prerelease quality checks in one reusable workflow, with Lua mpv plugin tests and blocking high-severity dependency audits running in every gate.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: fixed
|
||||
area: launcher
|
||||
|
||||
- Rofi menu prompts now keep a space between the prompt text and the input field instead of running into the search placeholder.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: internal
|
||||
area: dependencies
|
||||
|
||||
- Patched three high-severity dependency advisories flagged by `bun audit`: `undici` (cross-user information disclosure via degenerate private cache directives), `brace-expansion` (denial of service via unbounded intermediate arrays), and `fast-uri` (host confusion via backslash authority introducer).
|
||||
@@ -1,5 +0,0 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Validated nested and legacy AnkiConnect settings after splitting the resolver, preserving valid modern overrides while warning and falling back for invalid primitive values.
|
||||
- Hardened stats routes against malformed IDs and static paths, stalled AniList searches, word-mining media collisions, missing Yomitan bridges, and throwing timing observers.
|
||||
@@ -1,5 +0,0 @@
|
||||
type: added
|
||||
area: stats
|
||||
|
||||
- Added a "Delete Entry" action to the stats Library detail view. It removes the whole title in one step — every episode, session, subtitle line, rollup, cover art and vocabulary counts derived from them — and the title disappears from the Library grid. Previously only individual episodes and sessions could be deleted, so a mistakenly created entry had to be cleared episode by episode and still lingered in the library.
|
||||
- Deletion progress is now shown app-wide instead of only on Overview and Sessions. Every delete (session, session group, episode, and library entry) drives a sweeping bar at the top of the window plus a bottom-right status toast, and both stay visible across tab switches, detail views, and the in-app stats overlay window. The per-tab indicators previously went blank the moment you left the tab that started the delete.
|
||||
@@ -1,7 +0,0 @@
|
||||
type: fixed
|
||||
area: stats
|
||||
|
||||
- Made deleting stats data much faster and stopped it stalling playback. Every delete used to recompute each affected word's totals by re-reading that word's whole occurrence history across the library, which meant one random read into the largest table per occurrence. On a 960-session library that cost 3.8s to delete a single session and over a minute to clear a 12-episode title, and because the stats server runs in the app process when the app owns it, that time was spent blocking mpv.
|
||||
- Word and kanji occurrences now store the subtitle line's timestamp alongside the count, so those totals are answered from a covering index instead of the subtitle-line table, and a delete only subtracts what it actually removed rather than recomputing untouched entries. Measured on the same 960-session library: deleting one session 3832ms to 190ms, a ten-session day group 6453ms to 433ms, one episode 5437ms to 263ms, and a whole 12-episode title 60556ms to 621ms.
|
||||
- Opening the Vocabulary tab no longer stalls either. It was computing "seen in N titles" for every word in the library before ordering by frequency and keeping the top 100, so the whole occurrence history was walked to produce one page. It now picks the page first and counts only those rows: 2136ms to 62ms on the same library.
|
||||
- The first launch after upgrading migrates the existing stats database in place (about 3s for a 288k-line library) and it grows roughly 20% from the added index. Vocabulary dates recorded before the upgrade keep working while the migration runs.
|
||||
@@ -0,0 +1,7 @@
|
||||
type: fixed
|
||||
area: streaming
|
||||
|
||||
- Jellyfin playback now seeds the subtitle tokenization prefetch straight from the subtitle file it downloads, instead of waiting on an mpv track-selection event that could be missed or coalesced and leave a whole episode tokenizing line by line.
|
||||
- Streamed media no longer drops its parsed subtitle cues when the active subtitle track briefly cannot be resolved, such as when cycling onto a subtitle track embedded in the stream.
|
||||
- Subtitle prefetching now runs to the end of a file instead of stopping as soon as the tokenization cache fills, so the back half of an episode no longer gets tokenized line by line during playback. Previously the cache was also never cleared between episodes, so the stall carried over to every later title in a session.
|
||||
- Raised the tokenization cache from 256 to 2500 lines. It is now purely a memory bound rather than a limit on how much gets prefetched, and it leaves room for lines that repeat across episodes so openings and endings stay warm between titles.
|
||||
@@ -0,0 +1,7 @@
|
||||
type: changed
|
||||
area: subsync
|
||||
|
||||
- The subsync modal now lets you pick both sides of an alass run: the reference subtitle (correct timing, defaults to the loaded secondary subtitle track) and the out-of-sync subtitle that gets retimed (defaults to the active primary track).
|
||||
- alass can now use the loaded video file itself as the reference (audio-based, local files only). It is offered in the reference list but is never the default.
|
||||
- The out-of-sync subtitle picker also applies to ffsubsync, so a track other than the active primary one can be retimed.
|
||||
- Retiming the secondary subtitle track now reloads the synced result back into the secondary slot and leaves the primary track selected, instead of replacing the primary subtitle.
|
||||
@@ -0,0 +1,4 @@
|
||||
type: fixed
|
||||
area: overlay
|
||||
|
||||
- Subtitle lines no longer wait for tokenization to finish before appearing, even when the previous line is still being processed. On a tokenization cache miss, the plain line is shown immediately at its cue time and upgrades in place once tokens and annotations are ready; stale results cannot replace newer cues, and the basic plain-text websocket no longer receives a duplicate event for the annotation-only upgrade. A failed tokenization is no longer cached as the plain line, so a repeated line gets another chance at annotations instead of staying plain for the rest of the session.
|
||||
@@ -0,0 +1,18 @@
|
||||
type: changed
|
||||
area: subtitles
|
||||
|
||||
- Subtitle tokenization no longer runs a duplicate full `parseText` pass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line.
|
||||
- The Yomitan scanning helpers are now installed once per parser window (`__subminerYomitanScan`) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call.
|
||||
- termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
|
||||
- The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ). The shrinking-window retry ladder keeps following the consumed lengths the backend reports, and only blind guesses (windows the backend consumed whole, which tell it nothing) are capped at four per position. A line that hits that cap escalates to a single `parseText` for the whole line, so a hard line still resolves to dictionary tokens instead of an unparsed run, without letting the ladder run to one lookup per window length.
|
||||
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a `which mecab` availability check that re-ran synchronously on every line when MeCab is absent.
|
||||
- Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused for the whole time the subtitle processing controller is working on the line, including the provisional raw emit that precedes tokenization, so it never competes with the on-screen line for the parser window. The pause is released when the controller reports it has settled, which also covers the lines that finish without an emit (a suppressed duplicate or a failed tokenization) and used to leave prefetching paused indefinitely.
|
||||
- Added per-stage debug timings (`scanMs`, `mecabMs`, `frequencyMs`, `annotateMs`) to the subtitle tokenization pipeline log.
|
||||
- Fixed a reading that stopped covering its surface when an unmatched kana run extended the preceding token (for example a trailing る on 待ち合わせ), which silently disabled the known-word reading fallback for those tokens.
|
||||
- Subtitle prefetching no longer stays paused for the rest of a cue when the same subtitle text is reported twice and there is nothing to tokenize. This covers the startup and overlay priming paths as well as ordinary subtitle changes.
|
||||
- Character name and image lookups are now refreshed centrally whenever a character dictionary sync changes its content, so a newly added name can no longer be skipped by a stale candidate list.
|
||||
- A subtitle that was on screen when its annotations were invalidated (by mining a card, for example) is now re-annotated instead of staying plain for the rest of the line.
|
||||
- Character name annotations no longer cost a dictionary lookup at every position in a line. The scanner now knows which name forms the current title's character dictionary actually contains and only checks where one can start, which removes the whole overhead of having the character dictionary enabled (measured: 21 lookups per line down to 10, the same as with it disabled). Titles with no cached character data keep the previous exhaustive scan, so a missing snapshot costs speed rather than a missing name.
|
||||
- The cross-line termsFind cache is now bounded by the number of retained dictionary entries as well as by key count, so a run of lookups that each carry hundreds of entries with full glossaries cannot grow the parser window's memory without limit. The budget is re-checked when a lookup resolves, so a single oversized response is dropped rather than parked in the cache and reused.
|
||||
- The unnamed-mob disambiguator filter (Girl A / Girl B) now targets the letters those labels are split into, instead of every one-character term: a character whose name really is one character (𠮷, or a single kana) keeps it. The character dictionary and the scanner's name pre-pass also share one Han code-point table now, so a name the dictionary accepts is a name the scanner will look for.
|
||||
- A character name written in halfwidth katakana takes part in the greedy name pre-pass again, so a longer generic word can no longer swallow the start of it.
|
||||
@@ -1,8 +0,0 @@
|
||||
type: added
|
||||
area: sync
|
||||
|
||||
- Added cross-machine immersion sync for stats and watch history over SSH, available as a window (**Sync Stats & History** in the tray menu, or `subminer sync --ui`) and as a command (`subminer sync <host>`, with `--push` / `--pull` for one-way insert-only transfers). The window keeps saved devices with per-host direction, one-click sync with live stage-by-stage progress and separate merge summaries for each machine, connection testing for first-time setup, cancellable runs while the app/stats server/playback is active, and manual database snapshots (create/merge/reveal/delete, stored in `/tmp/subminer-db-snapshots/` by default). Hosts with auto-sync enabled sync in the background on a configurable interval, including during playback, with results reported as overlay notifications; hosts synced from the CLI are remembered in `<config dir>/sync-hosts.json` and appear in the window automatically.
|
||||
- Merges are an insert-only union keyed on session UUIDs / video keys / series title keys, so re-syncing is idempotent and nothing is double-counted: each side snapshots its database (`VACUUM INTO`) from a consistent WAL point, snapshots are exchanged with `scp`, and each machine merges the other's data transactionally. Lifetime totals and daily/monthly rollups are updated incrementally (history older than the session retention window is preserved), unfinished sessions are excluded until a later sync sees them finalized, and remote-only historical rollups are copied only when they do not conflict with retained local session history. Sync aborts on stats schema version mismatches and refuses to run while the stats daemon or a live mpv session is active (`--force` overrides).
|
||||
- The sync engine runs only inside the app: the sync window and the `subminer sync` command both delegate to `SubMiner --sync-cli` (headless, works over SSH with no display), so neither machine needs bun or the command-line launcher. A remote machine only needs SubMiner itself, found automatically as the app binary or via the launcher proxy.
|
||||
- Windows remotes are supported: sync detects the remote shell (POSIX, cmd, or PowerShell) and manages remote temp files through SubMiner itself (`sync --make-temp` / `--remove-temp`) instead of `mktemp` / `rm`, so a Windows machine with the built-in OpenSSH Server works as a sync remote, found in its default Windows install location automatically.
|
||||
- Added supporting flags: `subminer sync <host> --check` tests the SSH connection and remote launcher availability without syncing, `subminer sync --snapshot <file>` and `--merge <file>` expose the underlying steps for manual transfers, and `subminer sync --json` emits machine-readable NDJSON progress (the protocol the sync window consumes).
|
||||
@@ -1,4 +0,0 @@
|
||||
type: added
|
||||
area: overlay
|
||||
|
||||
- Added TsukiHime subtitle downloads for the current video, with Japanese primary and configured secondary-language tracks loaded directly into mpv.
|
||||
@@ -1,4 +0,0 @@
|
||||
type: internal
|
||||
area: stats
|
||||
|
||||
- Removed the unused stats IPC data transport and unified the stats dashboard's HTTP wire types with the backend contract.
|
||||
@@ -1,7 +0,0 @@
|
||||
type: internal
|
||||
area: tokenizer
|
||||
|
||||
- Added `verify-known-word-highlights:electron` script: tokenizes a real subtitle file through the app's Yomitan/MeCab pipeline with the live known-word cache, prints each line in the configured tier colors, and summarizes the tier counts so highlighting can be checked outside of playback.
|
||||
- Added `--audit`, which re-derives every highlighted tier from live Anki card data (`notesInfo` + `cardsInfo` intervals) and reports each token whose rendered tier disagrees, catching both stale cache entries and tier-classification bugs.
|
||||
- Added `--profile-copy` so the check can run while SubMiner is open (Electron locks the Yomitan userData dir), plus `--refresh`, `--limit`, `--json`, and `--quiet`.
|
||||
- Added `KnownWordCacheManager.getKnownWordMatchNoteIds`, exposing the note ids behind a known-word match so an audit can trace a rendered tier back to the exact Anki notes.
|
||||
@@ -523,7 +523,7 @@
|
||||
// ==========================================
|
||||
// AnkiConnect Integration
|
||||
// Automatic Anki updates and media generation options.
|
||||
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
|
||||
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, isKiku.fieldGrouping, and lapisKiku.wordCardKind update live while SubMiner is running.
|
||||
// Shared AI provider transport settings are read from top-level ai and typically require restart.
|
||||
// Most other AnkiConnect settings still require restart.
|
||||
// ==========================================
|
||||
@@ -605,7 +605,10 @@
|
||||
"enabled": false, // Enable Kiku-specific mining behaviors (duplicate handling, field grouping). Values: true | false
|
||||
"fieldGrouping": "disabled", // Kiku duplicate-card field grouping mode. Values: auto | manual | disabled
|
||||
"deleteDuplicateInAuto": true // When Kiku field grouping is "auto", delete the duplicate source card after grouping completes. Values: true | false
|
||||
} // Is kiku setting.
|
||||
}, // Is kiku setting.
|
||||
"lapisKiku": {
|
||||
"wordCardKind": "word-and-sentence" // Card-type flag SubMiner marks on Kiku/Lapis word cards. Only one flag is set at a time; the others are cleared. Requires isKiku.enabled or isLapis.enabled. Values: word-and-sentence | click | sentence | audio | none
|
||||
} // Lapis kiku setting.
|
||||
}, // Automatic Anki updates and media generation options.
|
||||
|
||||
// ==========================================
|
||||
|
||||
@@ -289,6 +289,21 @@ Trigger with the mine sentence shortcut (`Ctrl/Cmd+S` by default). The card is c
|
||||
|
||||
To mine multiple subtitle lines as one sentence card, use `Ctrl/Cmd+Shift+S` followed by a digit (1–9) to select how many recent lines to combine.
|
||||
|
||||
## Word Card Type (Kiku/Lapis)
|
||||
|
||||
Word cards get a card-type flag when SubMiner fills their sentence, whether that comes from Yomitan auto-enrichment, a manual clipboard update, or stats-dashboard word mining. By default the flag is `IsWordAndSentenceCard`; pick a different one with `ankiConnect.lapisKiku.wordCardKind`.
|
||||
|
||||
```jsonc
|
||||
"ankiConnect": {
|
||||
"isKiku": { "enabled": true },
|
||||
"lapisKiku": {
|
||||
"wordCardKind": "click" // word-and-sentence (default), click, sentence, audio, none
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`click` marks `IsClickCard`, `sentence` marks `IsSentenceCard`, `audio` marks `IsAudioCard`, and `none` leaves the flags untouched for templates that manage them elsewhere. Whichever flag is chosen, the other card-type flags are cleared so the note never claims two card types. The setting is only read when `isKiku` or `isLapis` is enabled, and cards mined with Mine Sentence or Mine Audio keep their own flag.
|
||||
|
||||
## Field Grouping (Kiku)
|
||||
|
||||
When you mine the same word multiple times, SubMiner can merge the cards instead of creating duplicates. This is designed for note types like [Kiku](https://github.com/youyoumu/kiku) that support grouped sentence/audio/image fields.
|
||||
@@ -314,7 +329,7 @@ When you mine the same word multiple times, SubMiner can merge the cards instead
|
||||
### What Gets Merged
|
||||
|
||||
| Field | Merge behavior |
|
||||
| -------- | ---------------------------------------- |
|
||||
| -------- | --------------------------------------------- |
|
||||
| Sentence | Both cards' sentences kept as grouped entries |
|
||||
| Audio | Both cards' `[sound:...]` entries kept |
|
||||
| Image | Both cards' images kept |
|
||||
|
||||
+52
-2
@@ -1,6 +1,56 @@
|
||||
# Changelog
|
||||
|
||||
## v0.18.0 (2026-07-10)
|
||||
## v0.19.1 (2026-08-01)
|
||||
|
||||
**Added**
|
||||
- Word Card Type: Adds a setting (Settings > Mining/Anki > Kiku/Lapis Features > "Word Card Type") to choose which card-type flag SubMiner marks on Kiku/Lapis word cards — `word-and-sentence` (default), `click`, `sentence`, `audio`, or `none`. Click cards (`IsClickCard`) can now be flagged, and setting any card-type flag clears the others so a note can't claim two types at once.
|
||||
|
||||
**Fixed**
|
||||
- Yomitan Popup: Fixes the macOS Yomitan popup going inert after mining a card — clicks outside the popup no longer pass through to mpv, and scrolling over the popup scrolls its definitions instead of seeking playback.
|
||||
- YouTube Playlist Links: Fixes opening a video from a playlist URL (e.g. a Watch Later link with `list=`/`index=`) timing out while probing subtitles, metadata, or the playback URL.
|
||||
|
||||
## v0.19.0 (2026-07-29)
|
||||
|
||||
**Added**
|
||||
- Anki Maturity Highlighting: Known-word subtitle highlights can now be colored by Anki card maturity (new, learning, young, mature), similar to asbplayer. Tier thresholds and colors are configurable, with a runtime toggle and an updated help legend.
|
||||
- Post-Playback Menu: After a watch-history episode ends, the fzf/rofi launcher returns to that series with options to play the previous or next episode, rewatch, pick another episode, or quit. The pre-playback series menu now offers the previous episode too.
|
||||
- Delete Library Entries: The stats Library detail view can now delete an entire title in one step (episodes, sessions, subtitle lines, rollups, cover art, and vocabulary counts). Delete progress is now shown app-wide via a progress bar and status toast instead of disappearing when you switch tabs.
|
||||
- Cross-Machine Sync: Added SSH-based syncing of stats and watch history between machines, available from the tray ("Sync Stats & History") or `subminer sync`, with saved devices, per-host sync direction, background auto-sync, connection testing, manual snapshots, and support for Windows remotes.
|
||||
- TsukiHime Subtitle Downloads: Added subtitle downloads for the current video via TsukiHime, loading Japanese as the primary track and your configured secondary language directly into mpv.
|
||||
|
||||
**Changed**
|
||||
- Clipboard-Video Shortcut: The "append clipboard video to queue" shortcut is now configurable.
|
||||
|
||||
**Fixed**
|
||||
- AniList Season Resolution: Season 2+ files now resolve to the correct AniList entry instead of silently falling back to season 1 (which mismatched character dictionaries and watch progress). Manual overrides now stay scoped per season, fix both the dictionary and progress tracking together, and also correct per-season cover art.
|
||||
- Subtitle Annotation Accuracy: Fixed several annotation edge cases, including inconsistent POS exclusions on merged quote-particle tokens, dropped annotations on supplementary-plane kanji, katakana punctuation wrongly treated as noise, and certain kanji vocabulary losing N+1 highlighting eligibility.
|
||||
- AnkiConnect Proxy Port Conflict: Video startup no longer crashes when another process already holds the configured AnkiConnect proxy port; a notification now explains how to resolve it.
|
||||
- AppImage Quit Crash: Fixed a "Service Crash" desktop notification appearing after closing a video when running the Linux AppImage.
|
||||
- Autoplay Pause Timing: Fixed playback resuming a few seconds before subtitle tokenization warmup finished, most noticeable when resuming mid-episode or when a cue starts within the first two seconds.
|
||||
- Stats Known-Word Count: Fixed stats reporting 0 known words for every session after the known-word cache format changed.
|
||||
- Stats Library Cover Art: Relinking a title to a different AniList entry now updates its cover in the Library grid immediately instead of leaving a stale, mismatched cover cached.
|
||||
- Rofi Prompt Spacing: Rofi menu prompts now keep a space before the input field instead of running into the placeholder text.
|
||||
- Stats Settings & Reliability: Hardened stats settings validation (nested/legacy AnkiConnect config now falls back safely instead of breaking) and stats routes against malformed requests and other edge cases.
|
||||
- Stats Delete Performance: Deleting sessions, episodes, and library entries is now dramatically faster and no longer stalls playback (e.g. a 12-episode title dropped from about a minute to under a second on a large library); the Vocabulary tab also loads much faster. The first launch after upgrading runs a one-time database migration.
|
||||
|
||||
<details>
|
||||
<summary>Internal changes</summary>
|
||||
|
||||
**Internal**
|
||||
- Added a golden-file regression test corpus for the tokenizer/annotation pipeline, plus scripts to record new fixtures and diff against stock Yomitan.
|
||||
- Consolidated renderer modal state handling into a descriptor registry.
|
||||
- Consolidated CI quality checks (PR, stable, and prerelease) into one reusable workflow with mpv plugin tests and dependency audits.
|
||||
- Removed the unused stats IPC transport and unified stats dashboard HTTP types with the backend contract.
|
||||
- Added a script to verify known-word highlight tiers against live Anki data outside of playback.
|
||||
|
||||
</details>
|
||||
|
||||
## Previous Versions
|
||||
|
||||
<details>
|
||||
<summary>v0.18.x</summary>
|
||||
|
||||
<h2>v0.18.0 (2026-07-10)</h2>
|
||||
|
||||
**Added**
|
||||
- Sentence Audio Normalization: Generated sentence audio is now normalized to -23 LUFS by default, and clips mined from playback mirror mpv's software volume curve with a limiter to prevent clipping. Both behaviors are configurable independently.
|
||||
@@ -31,7 +81,7 @@
|
||||
|
||||
</details>
|
||||
|
||||
## Previous Versions
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>v0.17.x</summary>
|
||||
|
||||
@@ -399,7 +399,7 @@ See `config.example.jsonc` for detailed configuration options.
|
||||
```
|
||||
|
||||
| Option | Values | Description |
|
||||
| ---------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ---------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `primaryDefaultMode` | string | Default primary subtitle bar visibility mode: `"hidden"`, `"visible"`, or `"hover"` (default: `"visible"`) |
|
||||
| `subtitleStyle.css` | object | CSS declaration object applied to primary subtitles after normal style defaults. Use CSS property names such as `font-size`. |
|
||||
| `secondary.css` | object | CSS declaration object applied to secondary subtitles after normal secondary style defaults. |
|
||||
@@ -556,7 +556,7 @@ Secondary subtitles do **not** auto-load by default. To turn them on for local a
|
||||
```
|
||||
|
||||
| Option | Values | Description |
|
||||
| ----------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ----------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `secondarySubLanguages` | string[] | Language codes to auto-load (e.g., `["eng", "en"]`); non-Signs/Songs tracks are preferred when several tracks match. Default is empty (`[]`). |
|
||||
| `autoLoadSecondarySub` | `true`, `false` | Auto-detect and load a matching secondary subtitle track for local/Jellyfin sidecar files (default: `false`) |
|
||||
| `defaultMode` | `"hidden"`, `"visible"`, `"hover"` | Initial display mode (default: `"hover"`) |
|
||||
@@ -1069,6 +1069,9 @@ SubMiner is intentionally built for [Kiku](https://kiku.youyoumu.my.id/) and [La
|
||||
"enabled": true,
|
||||
"fieldGrouping": "manual",
|
||||
"deleteDuplicateInAuto": true
|
||||
},
|
||||
"lapisKiku": {
|
||||
"wordCardKind": "word-and-sentence"
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -1077,6 +1080,21 @@ SubMiner is intentionally built for [Kiku](https://kiku.youyoumu.my.id/) and [La
|
||||
- Enable `isKiku` to turn on duplicate merge behavior for mined Word/Expression hits.
|
||||
- When both are enabled, Kiku behavior is applied for grouping while sentence-card model settings are still read from `isLapis`.
|
||||
- `isKiku.fieldGrouping` supports `disabled`, `auto`, and `manual` merge modes; see [Field Grouping Modes](#field-grouping-modes).
|
||||
- `lapisKiku.wordCardKind` picks the card-type flag set on word cards; see [Word Card Type](#word-card-type). It is read only while `isLapis` or `isKiku` is enabled.
|
||||
|
||||
### Word Card Type
|
||||
|
||||
When SubMiner fills the sentence on a mined word card - from Yomitan auto-enrichment, a manual clipboard update, or stats-dashboard word mining - it marks which card that note should generate. `ankiConnect.lapisKiku.wordCardKind` chooses the flag:
|
||||
|
||||
| Value | Flag set |
|
||||
| ----------------------------- | ----------------------- |
|
||||
| `word-and-sentence` (default) | `IsWordAndSentenceCard` |
|
||||
| `click` | `IsClickCard` |
|
||||
| `sentence` | `IsSentenceCard` |
|
||||
| `audio` | `IsAudioCard` |
|
||||
| `none` | none; flags left as-is |
|
||||
|
||||
The other card-type flags are cleared so a note never claims two card types at once. Notes are skipped when the note type has no field for the chosen flag, and when the note was already mined as a sentence or audio card. Cards created by Mine Sentence and Mine Audio keep their own flag regardless of this setting.
|
||||
|
||||
### N+1 Word Highlighting
|
||||
|
||||
@@ -1168,7 +1186,7 @@ TsukiHime subtitle search works out of the box and needs no account or API key.
|
||||
```
|
||||
|
||||
| Option | Values | Description |
|
||||
| ---------------------------- | ------------ | ---------------------------------------------------------------------------------------------------- |
|
||||
| ---------------------------- | ------------ | ----------------------------------------------------------------------------------------------------- |
|
||||
| `tsukihime.apiBaseUrl` | string (URL) | Base URL of the TsukiHime API (default: `https://api.tsukihime.org/v1`). Only change it for a mirror. |
|
||||
| `tsukihime.maxSearchResults` | number | Maximum releases returned per search (default: `10`; the API caps this at 100) |
|
||||
|
||||
@@ -1178,9 +1196,9 @@ See [TsukiHime Integration](/tsukihime-integration) for the modal workflow, lang
|
||||
|
||||
### Subtitle Sync
|
||||
|
||||
Sync the active subtitle track from the overlay picker using `alass` or `ffsubsync`. Both are **optional external tools** that must be installed separately and available on your `PATH` (or configured via the path options below).
|
||||
Sync a subtitle track from the overlay picker using `alass` or `ffsubsync`. The picker lets you choose which track gets retimed (the active primary track by default) and, for alass, which reference it is aligned against (the secondary subtitle track by default). Both are **optional external tools** that must be installed separately and available on your `PATH` (or configured via the path options below).
|
||||
|
||||
- [`alass`](https://github.com/kaegi/alass) - fast, audio-independent sync using a secondary subtitle as reference
|
||||
- [`alass`](https://github.com/kaegi/alass) - fast, audio-independent sync using another subtitle as reference; it can also take the local video file as reference (alass extracts the audio itself)
|
||||
- [`ffsubsync`](https://github.com/smacke/ffsubsync) - audio-based sync using the video file as reference
|
||||
|
||||
```json
|
||||
@@ -1229,7 +1247,7 @@ AniList integration is opt-in and disabled by default. Enable it to allow SubMin
|
||||
```
|
||||
|
||||
| Option | Values | Description |
|
||||
| -------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| -------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `anilist.enabled` | `true`, `false` | Enable AniList post-watch progress updates (default: `false`) |
|
||||
| `accessToken` | string | Optional explicit AniList access token override (default: empty string) |
|
||||
| `characterDictionary.maxLoaded` | number | Maximum number of most-recently-used AniList media snapshots included in the merged dictionary (default: `3`) |
|
||||
@@ -1540,7 +1558,7 @@ Configure the mpv executable, profile, and window state for SubMiner-managed mpv
|
||||
```
|
||||
|
||||
| Option | Values | Description |
|
||||
| ------------------------ | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ------------------------ | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `executablePath` | string | Absolute path to `mpv.exe` for Windows launch flows. Leave empty to auto-discover from `SUBMINER_MPV_PATH` or `PATH` (default `""`) |
|
||||
| `profile` | string | mpv profile name passed as `--profile=<name>`. Leave empty to pass no profile (default `""`) |
|
||||
| `launchMode` | `"normal"` \| `"maximized"` \| `"fullscreen"` | Window state when SubMiner spawns mpv (default `"normal"`) |
|
||||
|
||||
@@ -161,10 +161,13 @@ If your subtitle file is out of sync with the audio, SubMiner can resynchronize
|
||||
|
||||
1. Open the subsync modal from the overlay.
|
||||
2. Select the sync engine (alass or ffsubsync).
|
||||
3. For alass, select a reference subtitle track from the video.
|
||||
4. SubMiner runs the sync and reloads the corrected subtitle.
|
||||
3. For alass, pick the **reference** - the subtitle with correct timing. This defaults to the secondary subtitle track. The loaded video file can also be used as the reference (alass extracts the audio itself), but it is never the default.
|
||||
4. Pick the **out-of-sync subtitle** - the track that gets retimed. This defaults to the active primary subtitle track and applies to both engines.
|
||||
5. SubMiner runs the sync and reloads the corrected subtitle into the slot the out-of-sync track came from: retiming the secondary track keeps it secondary and leaves the primary track selected.
|
||||
|
||||
For remote streams, including Jellyfin playback, the modal only offers alass. Jellyfin subtitle URLs are cached as temporary subtitle files so alass can read them, but the video stream is not downloaded. ffsubsync needs direct access to the local media file and is unavailable for stream URLs.
|
||||
The reference and the out-of-sync subtitle must be different tracks; the reference list hides whichever track is selected as the target.
|
||||
|
||||
For remote streams, including Jellyfin playback, the modal only offers alass with a subtitle reference. Jellyfin subtitle URLs are cached as temporary subtitle files so alass can read them, but the video stream is not downloaded. ffsubsync and the video-file reference need direct access to the local media file and are unavailable for stream URLs.
|
||||
|
||||
When you mine a sentence card from the stats dashboard, SubMiner can also use `alass` automatically to align a local English sidecar against the matching local Japanese sidecar before filling the card translation field. The source subtitle files are not modified; SubMiner writes a temporary retimed copy and reuses it while the stats server is running.
|
||||
|
||||
|
||||
@@ -523,7 +523,7 @@
|
||||
// ==========================================
|
||||
// AnkiConnect Integration
|
||||
// Automatic Anki updates and media generation options.
|
||||
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.
|
||||
// Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, isKiku.fieldGrouping, and lapisKiku.wordCardKind update live while SubMiner is running.
|
||||
// Shared AI provider transport settings are read from top-level ai and typically require restart.
|
||||
// Most other AnkiConnect settings still require restart.
|
||||
// ==========================================
|
||||
@@ -605,7 +605,10 @@
|
||||
"enabled": false, // Enable Kiku-specific mining behaviors (duplicate handling, field grouping). Values: true | false
|
||||
"fieldGrouping": "disabled", // Kiku duplicate-card field grouping mode. Values: auto | manual | disabled
|
||||
"deleteDuplicateInAuto": true // When Kiku field grouping is "auto", delete the duplicate source card after grouping completes. Values: true | false
|
||||
} // Is kiku setting.
|
||||
}, // Is kiku setting.
|
||||
"lapisKiku": {
|
||||
"wordCardKind": "word-and-sentence" // Card-type flag SubMiner marks on Kiku/Lapis word cards. Only one flag is set at a time; the others are cleared. Requires isKiku.enabled or isLapis.enabled. Values: word-and-sentence | click | sentence | audio | none
|
||||
} // Lapis kiku setting.
|
||||
}, // Automatic Anki updates and media generation options.
|
||||
|
||||
// ==========================================
|
||||
|
||||
@@ -227,7 +227,7 @@ Install ffsubsync or configure the path:
|
||||
|
||||
If subtitle sync fails (the error message is prefixed with the engine name):
|
||||
|
||||
- Ensure the reference subtitle track exists in the video (alass requires a source track).
|
||||
- Ensure a reference is selected (alass needs either a second subtitle track or the local video file, and it cannot be the same track that is being retimed).
|
||||
- Check that `ffmpeg` is available (used to extract the internal subtitle track).
|
||||
- Try running the sync tool manually to see detailed error output.
|
||||
- ffsubsync requires local files and cannot handle remote media streams (e.g., streaming URLs).
|
||||
|
||||
+1
-1
@@ -149,7 +149,7 @@ Once Jellyfin is configured, the tray menu includes `Jellyfin Discovery` for sta
|
||||
|
||||
- `--log-level` controls logger verbosity.
|
||||
- `--dev` and `--debug` are app/dev-mode switches; they are not log-level aliases.
|
||||
- `--background` defaults to quieter logging (`warn`) unless `--log-level` is set.
|
||||
- `--background` starts at the default quieter logging level (`warn`), then follows `logging.level` after config loads. An explicit `--log-level` remains the override.
|
||||
- `--background` launched from a terminal detaches and returns the prompt; stop it with tray Quit or `SubMiner.AppImage --stop` (`SubMiner.exe --stop` on Windows).
|
||||
- Linux desktop launcher starts SubMiner with `--background` by default (via electron-builder `linux.executableArgs`).
|
||||
- On Hyprland and other Wayland compositors, the tray icon appears only when your panel provides a StatusNotifier/AppIndicator tray host.
|
||||
|
||||
@@ -64,7 +64,7 @@ Use the basic subtitle websocket when you only need the current subtitle line as
|
||||
- **Client auth:** none
|
||||
- **Reconnects:** client-managed
|
||||
|
||||
When a client connects, SubMiner immediately sends the latest subtitle payload if one is available. After that, it pushes a new message each time the current subtitle changes.
|
||||
When a client connects, SubMiner immediately sends the latest subtitle payload if one is available. After that, it pushes a new message each time the current subtitle changes. Annotation-only upgrades do not repeat the same line on this basic stream.
|
||||
|
||||
#### Message shape
|
||||
|
||||
@@ -96,6 +96,8 @@ Use the annotation websocket for custom clients that want the same structured to
|
||||
|
||||
In practice, if you are building a new client, prefer `annotationWebsocket` unless you specifically need compatibility with an existing `websocket` consumer.
|
||||
|
||||
On a tokenization cache miss, this stream first sends the cue as plain text with an empty `tokens` array, then sends the annotated replacement when tokenization finishes. Treat each message as the complete current state, replacing the previous payload.
|
||||
|
||||
#### Message shape
|
||||
|
||||
```json
|
||||
|
||||
@@ -11,7 +11,7 @@ SubMiner auto-loads Japanese subtitles when you play a YouTube URL, giving you t
|
||||
|
||||
When SubMiner detects a YouTube URL (or `ytsearch:` target), it pauses mpv at startup and runs a subtitle pipeline before resuming playback:
|
||||
|
||||
1. **Probe** --- `yt-dlp --dump-single-json` extracts all available subtitle tracks (manual uploads and auto-generated captions) along with video metadata.
|
||||
1. **Probe** --- `yt-dlp --dump-single-json` extracts all available subtitle tracks (manual uploads and auto-generated captions) along with video metadata. Every yt-dlp call passes `--no-playlist`, so playlist links (for example a Watch Later URL with `list=`/`index=`) resolve to the single video instead of the whole playlist.
|
||||
2. **Discover** --- Each track is normalized into a `YoutubeTrackOption` with language code, kind (`manual` or `auto`), display label, and direct download URL.
|
||||
3. **Select** --- SubMiner picks the best primary track (Japanese, preferring manual over auto) and secondary track (English, preferring manual over auto).
|
||||
4. **Download** --- Selected tracks are fetched via direct URL when available, falling back to `yt-dlp --write-subs` / `--write-auto-subs`. YouTube TimedText XML formats (`srv1`/`srv2`/`srv3`) are converted to VTT on the fly. Auto-generated VTT captions are normalized to remove rolling-caption duplication.
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# Subtitle Overlay Priming
|
||||
|
||||
Status: active
|
||||
Last verified: 2026-06-14
|
||||
Last verified: 2026-08-04
|
||||
Owner: Kyle Yasuda
|
||||
Read when: debugging subtitle state or blank Linux/X11 overlay windows when the visible overlay is shown or recreated
|
||||
|
||||
@@ -47,18 +47,39 @@ subtitles do not draw.
|
||||
`emitSubtitle(payload)` and `refreshCurrentSubtitle(text)`, then prime secondary subtitles.
|
||||
6. Tokenization cache hit: call `consumeCachedSubtitle(text)`, `onSubtitleChange(text)`, and
|
||||
`emitSubtitle(cachedPayload)`, then prime secondary subtitles.
|
||||
7. Cache miss: call `refreshCurrentSubtitle(text)` and let normal tokenization emit the final
|
||||
payload.
|
||||
7. Cache miss: call `refreshCurrentSubtitle(text)`. Normal processing emits a plain payload
|
||||
synchronously, then replaces it with the tokenized payload when ready.
|
||||
|
||||
In `src/main.ts`, both `onSubtitleChange` and `refreshCurrentSubtitle` pause
|
||||
`subtitlePrefetchService`, notify it with `onSeek(lastObservedTimePos)`, and then call the matching
|
||||
`subtitleProcessingController` method. This gives the visible overlay priority over background
|
||||
prefetch work and re-centers prefetch around the live playback time.
|
||||
Both `onSubtitleChange` and `refreshCurrentSubtitle` pause `subtitlePrefetchService` and then call
|
||||
the matching `subtitleProcessingController` method, giving the visible overlay priority over
|
||||
background prefetch work. Prefetch is not re-centered here: restarting the run per line
|
||||
(`onSeek`) discarded the in-flight tokenization every time the subtitle changed, so only real
|
||||
seeks restart it (see `onTimePosUpdate` in `src/main.ts`).
|
||||
|
||||
The pause is released by the controller's `onProcessingSettled` callback, which fires once it has
|
||||
no work left. Emits do not release it: the first emit for an uncached line is the plain payload
|
||||
that precedes tokenization, and a run can finish without emitting at all (a suppressed duplicate,
|
||||
a failed tokenization). Both controller methods return whether processing is now pending, and the
|
||||
caller resumes immediately when it is not — a repeated subtitle schedules no work, so no settle is
|
||||
coming and prefetching would otherwise idle for the rest of the cue.
|
||||
|
||||
## Live Cue Delivery
|
||||
|
||||
- A tokenization cache miss emits the plain cue synchronously. Tokenization remains serialized so
|
||||
live work does not contend for Yomitan state.
|
||||
- If a newer cue arrives while an older line is still tokenizing, the newer plain cue or empty
|
||||
clear payload is emitted immediately. The older tokenization result is dropped before it can
|
||||
replace the current cue.
|
||||
- The current cue upgrades in place when its tokens and annotations are ready. This can reflow text
|
||||
or character images, but cue visibility does not wait for that work.
|
||||
|
||||
## Emitted State
|
||||
|
||||
- `emitSubtitle(payload)` maps to `emitSubtitlePayload(payload)`, which sends the normal
|
||||
annotated subtitle payload to overlay windows and subtitle websocket listeners.
|
||||
- `emitSubtitle(payload)` maps to `emitSubtitlePayload(payload)`. Overlay windows and annotation
|
||||
websocket listeners receive both the immediate plain cue and its later annotation upgrade.
|
||||
- The basic subtitle websocket receives the immediate plain cue only. Because its serialized
|
||||
payload discards annotations, the later upgrade would be an identical duplicate and is skipped
|
||||
when text and cue timing match.
|
||||
- Secondary priming reads mpv `secondary-sub-text`, stores it in
|
||||
`mpvClient.currentSecondarySubText`, and broadcasts `secondary-subtitle:set` to overlay windows.
|
||||
- If secondary `requestProperty` fails, the primary flow stays complete and only a debug line is
|
||||
|
||||
+6
-4
@@ -2,7 +2,7 @@
|
||||
"name": "subminer",
|
||||
"productName": "SubMiner",
|
||||
"desktopName": "SubMiner.desktop",
|
||||
"version": "0.19.0-beta.5",
|
||||
"version": "0.19.1",
|
||||
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
|
||||
"packageManager": "bun@1.3.5",
|
||||
"main": "dist/main-entry.js",
|
||||
@@ -84,8 +84,9 @@
|
||||
"overrides": {
|
||||
"@xmldom/xmldom": "0.8.13",
|
||||
"app-builder-lib": "26.15.3",
|
||||
"brace-expansion": "5.0.8",
|
||||
"brace-expansion": "5.0.9",
|
||||
"electron-builder-squirrel-windows": "26.15.3",
|
||||
"fast-uri": "3.1.5",
|
||||
"form-data": "4.0.6",
|
||||
"ip-address": "10.2.0",
|
||||
"js-yaml": "4.3.0",
|
||||
@@ -93,7 +94,8 @@
|
||||
"minimatch": "10.2.5",
|
||||
"picomatch": "4.0.4",
|
||||
"tar": "7.5.21",
|
||||
"tmp": "0.2.7"
|
||||
"tmp": "0.2.7",
|
||||
"undici": "7.29.0"
|
||||
},
|
||||
"keywords": [
|
||||
"anki",
|
||||
@@ -125,7 +127,7 @@
|
||||
"@types/ws": "^8.18.1",
|
||||
"electron": "42.6.0",
|
||||
"electron-builder": "26.15.3",
|
||||
"undici": "7.28.0",
|
||||
"undici": "7.29.0",
|
||||
"esbuild": "^0.25.12",
|
||||
"eslint": "^10.8.0",
|
||||
"prettier": "^3.8.1",
|
||||
|
||||
+7
-74
@@ -22,10 +22,12 @@ import { MediaGenerator } from './media-generator';
|
||||
import path from 'path';
|
||||
import {
|
||||
AnkiConnectConfig,
|
||||
type CardKind,
|
||||
KikuDuplicateCardInfo,
|
||||
KikuFieldGroupingChoice,
|
||||
KikuMergePreviewResponse,
|
||||
NotificationOptions,
|
||||
type WordCardKind,
|
||||
} from './types/anki';
|
||||
import { AiConfig } from './types/integrations';
|
||||
import type { KnownWordMaturityTier } from './types/subtitle';
|
||||
@@ -50,6 +52,7 @@ import {
|
||||
withUpdateProgress,
|
||||
UiFeedbackState,
|
||||
} from './anki-integration/ui-feedback';
|
||||
import { applyCardKindFlagFields, resolveWordCardKindSetting } from './anki-integration/card-kinds';
|
||||
import { KnownWordCacheManager } from './anki-integration/known-word-cache';
|
||||
import { PollingRunner } from './anki-integration/polling';
|
||||
import type { AnkiConnectProxyServer } from './anki-integration/anki-connect-proxy';
|
||||
@@ -83,8 +86,6 @@ interface NoteInfo {
|
||||
fields: Record<string, { value: string }>;
|
||||
}
|
||||
|
||||
type CardKind = 'sentence' | 'audio' | 'word-and-sentence';
|
||||
|
||||
function trimToNonEmptyString(value: unknown): string | null {
|
||||
if (typeof value !== 'string') return null;
|
||||
const trimmed = value.trim();
|
||||
@@ -840,6 +841,7 @@ export class AnkiIntegration {
|
||||
kikuEnabled: boolean;
|
||||
kikuFieldGrouping: 'auto' | 'manual' | 'disabled';
|
||||
kikuDeleteDuplicateInAuto: boolean;
|
||||
wordCardKind: WordCardKind;
|
||||
} {
|
||||
const lapis = this.getLapisConfig();
|
||||
const kiku = this.getKikuConfig();
|
||||
@@ -852,6 +854,7 @@ export class AnkiIntegration {
|
||||
kikuEnabled: kiku.enabled,
|
||||
kikuFieldGrouping: (kiku.fieldGrouping || 'disabled') as 'auto' | 'manual' | 'disabled',
|
||||
kikuDeleteDuplicateInAuto: kiku.deleteDuplicateInAuto !== false,
|
||||
wordCardKind: resolveWordCardKindSetting(this.config.lapisKiku?.wordCardKind),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1315,79 +1318,9 @@ export class AnkiIntegration {
|
||||
availableFieldNames: string[],
|
||||
cardKind: CardKind,
|
||||
): void {
|
||||
const audioFlagNames = ['IsAudioCard'];
|
||||
|
||||
if (cardKind === 'word-and-sentence') {
|
||||
const wordAndSentenceFlag = this.resolveFieldName(
|
||||
availableFieldNames,
|
||||
'IsWordAndSentenceCard',
|
||||
applyCardKindFlagFields(updatedFields, cardKind, (preferredName) =>
|
||||
this.resolveFieldName(availableFieldNames, preferredName),
|
||||
);
|
||||
if (!wordAndSentenceFlag) {
|
||||
return;
|
||||
}
|
||||
updatedFields[wordAndSentenceFlag] = 'x';
|
||||
|
||||
const sentenceFlag = this.resolveFieldName(availableFieldNames, 'IsSentenceCard');
|
||||
if (sentenceFlag && sentenceFlag !== wordAndSentenceFlag) {
|
||||
updatedFields[sentenceFlag] = '';
|
||||
}
|
||||
|
||||
for (const audioFlagName of audioFlagNames) {
|
||||
const resolved = this.resolveFieldName(availableFieldNames, audioFlagName);
|
||||
if (resolved && resolved !== wordAndSentenceFlag) {
|
||||
updatedFields[resolved] = '';
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (cardKind === 'sentence') {
|
||||
const sentenceFlag = this.resolveFieldName(availableFieldNames, 'IsSentenceCard');
|
||||
if (sentenceFlag) {
|
||||
updatedFields[sentenceFlag] = 'x';
|
||||
}
|
||||
|
||||
for (const audioFlagName of audioFlagNames) {
|
||||
const resolved = this.resolveFieldName(availableFieldNames, audioFlagName);
|
||||
if (resolved && resolved !== sentenceFlag) {
|
||||
updatedFields[resolved] = '';
|
||||
}
|
||||
}
|
||||
|
||||
const wordAndSentenceFlag = this.resolveFieldName(
|
||||
availableFieldNames,
|
||||
'IsWordAndSentenceCard',
|
||||
);
|
||||
if (wordAndSentenceFlag && wordAndSentenceFlag !== sentenceFlag) {
|
||||
updatedFields[wordAndSentenceFlag] = '';
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const resolvedAudioFlags = Array.from(
|
||||
new Set(
|
||||
audioFlagNames
|
||||
.map((name) => this.resolveFieldName(availableFieldNames, name))
|
||||
.filter((name): name is string => Boolean(name)),
|
||||
),
|
||||
);
|
||||
const audioFlagName = resolvedAudioFlags[0] || null;
|
||||
if (audioFlagName) {
|
||||
updatedFields[audioFlagName] = 'x';
|
||||
}
|
||||
for (const extraAudioFlag of resolvedAudioFlags.slice(1)) {
|
||||
updatedFields[extraAudioFlag] = '';
|
||||
}
|
||||
|
||||
const sentenceFlag = this.resolveFieldName(availableFieldNames, 'IsSentenceCard');
|
||||
if (sentenceFlag && sentenceFlag !== audioFlagName) {
|
||||
updatedFields[sentenceFlag] = '';
|
||||
}
|
||||
|
||||
const wordAndSentenceFlag = this.resolveFieldName(availableFieldNames, 'IsWordAndSentenceCard');
|
||||
if (wordAndSentenceFlag && wordAndSentenceFlag !== audioFlagName) {
|
||||
updatedFields[wordAndSentenceFlag] = '';
|
||||
}
|
||||
}
|
||||
|
||||
private async showNotification(
|
||||
|
||||
@@ -4,29 +4,23 @@ import test from 'node:test';
|
||||
import { CardCreationService } from './card-creation';
|
||||
import { toMpvEdlValue } from './mpv-edl-test-utils';
|
||||
import type { MediaInput } from '../media-generator';
|
||||
import type { AnkiConnectConfig } from '../types/anki';
|
||||
import type { AnkiConnectConfig, CardKind } from '../types/anki';
|
||||
import { applyCardKindFlagFields } from './card-kinds';
|
||||
|
||||
type CardCreationDeps = ConstructorParameters<typeof CardCreationService>[0];
|
||||
|
||||
function setWordAndSentenceCardTypeFields(
|
||||
function setCardTypeFields(
|
||||
updatedFields: Record<string, string>,
|
||||
availableFieldNames: string[],
|
||||
cardKind: 'sentence' | 'audio' | 'word-and-sentence',
|
||||
cardKind: CardKind,
|
||||
): void {
|
||||
if (cardKind !== 'word-and-sentence') return;
|
||||
|
||||
const resolveFieldName = (preferredName: string): string | null =>
|
||||
availableFieldNames.find((name) => name.toLowerCase() === preferredName.toLowerCase()) ?? null;
|
||||
const wordAndSentenceFlag = resolveFieldName('IsWordAndSentenceCard');
|
||||
if (!wordAndSentenceFlag) return;
|
||||
|
||||
updatedFields[wordAndSentenceFlag] = 'x';
|
||||
for (const flagName of ['IsSentenceCard', 'IsAudioCard']) {
|
||||
const resolved = resolveFieldName(flagName);
|
||||
if (resolved && resolved !== wordAndSentenceFlag) {
|
||||
updatedFields[resolved] = '';
|
||||
}
|
||||
}
|
||||
applyCardKindFlagFields(
|
||||
updatedFields,
|
||||
cardKind,
|
||||
(preferredName) =>
|
||||
availableFieldNames.find((name) => name.toLowerCase() === preferredName.toLowerCase()) ??
|
||||
null,
|
||||
);
|
||||
}
|
||||
|
||||
function createManualUpdateService(overrides: Partial<CardCreationDeps> = {}): {
|
||||
@@ -217,7 +211,7 @@ test('manual clipboard subtitle update marks Kiku word cards as word-and-sentenc
|
||||
kikuFieldGrouping: 'disabled',
|
||||
kikuDeleteDuplicateInAuto: false,
|
||||
}),
|
||||
setCardTypeFields: setWordAndSentenceCardTypeFields,
|
||||
setCardTypeFields,
|
||||
});
|
||||
|
||||
await service.updateLastAddedFromClipboard('字幕');
|
||||
|
||||
@@ -3,7 +3,7 @@ import {
|
||||
getConfiguredWordFieldName,
|
||||
getPreferredWordValueFromExtractedFields,
|
||||
} from '../anki-field-config';
|
||||
import { AnkiConnectConfig } from '../types/anki';
|
||||
import { AnkiConnectConfig, type CardKind, type WordCardKind } from '../types/anki';
|
||||
import { createLogger } from '../logger';
|
||||
import type { MediaInput } from '../media-input';
|
||||
import { SubtitleTimingTracker } from '../subtitle-timing-tracker';
|
||||
@@ -15,7 +15,7 @@ import {
|
||||
resolveAudioStreamIndexForMediaGeneration,
|
||||
type MediaGenerationInputResolverOptions,
|
||||
} from './media-source';
|
||||
import { shouldMarkWordAndSentenceCard } from './note-field-utils';
|
||||
import { resolveWordCardKind } from './note-field-utils';
|
||||
import type { PendingYoutubeMediaUpdate } from './pending-youtube-media';
|
||||
import { resolveMpvVolumeScale } from './mpv-volume';
|
||||
|
||||
@@ -42,8 +42,6 @@ export interface CardCreationNoteInfo {
|
||||
fields: Record<string, { value: string }>;
|
||||
}
|
||||
|
||||
type CardKind = 'sentence' | 'audio' | 'word-and-sentence';
|
||||
|
||||
interface CardCreationClient {
|
||||
addNote(
|
||||
deck: string,
|
||||
@@ -136,6 +134,7 @@ interface CardCreationDeps {
|
||||
kikuEnabled: boolean;
|
||||
kikuFieldGrouping: 'auto' | 'manual' | 'disabled';
|
||||
kikuDeleteDuplicateInAuto: boolean;
|
||||
wordCardKind?: WordCardKind;
|
||||
};
|
||||
getFallbackDurationSeconds: () => number;
|
||||
appendKnownWordsFromNoteInfo: (noteInfo: CardCreationNoteInfo) => void;
|
||||
@@ -274,12 +273,9 @@ export class CardCreationService {
|
||||
if (sentenceField) {
|
||||
const processedSentence = this.deps.processSentence(sentence, fields);
|
||||
updatedFields[sentenceField] = processedSentence;
|
||||
if (shouldMarkWordAndSentenceCard(noteInfo, sentenceCardConfig)) {
|
||||
this.deps.setCardTypeFields(
|
||||
updatedFields,
|
||||
Object.keys(noteInfo.fields),
|
||||
'word-and-sentence',
|
||||
);
|
||||
const wordCardKind = resolveWordCardKind(noteInfo, sentenceCardConfig);
|
||||
if (wordCardKind) {
|
||||
this.deps.setCardTypeFields(updatedFields, Object.keys(noteInfo.fields), wordCardKind);
|
||||
}
|
||||
updatePerformed = true;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import { applyCardKindFlagFields } from './card-kinds';
|
||||
|
||||
function resolverFor(availableFieldNames: string[]) {
|
||||
return (preferredName: string): string | null =>
|
||||
availableFieldNames.find((name) => name.toLowerCase() === preferredName.toLowerCase()) ?? null;
|
||||
}
|
||||
|
||||
const KIKU_FLAG_FIELDS = ['IsWordAndSentenceCard', 'IsClickCard', 'IsSentenceCard', 'IsAudioCard'];
|
||||
|
||||
test('flags the requested card kind and clears the others', () => {
|
||||
const fields: Record<string, string> = {};
|
||||
|
||||
applyCardKindFlagFields(fields, 'click', resolverFor(KIKU_FLAG_FIELDS));
|
||||
|
||||
assert.deepEqual(fields, {
|
||||
IsClickCard: 'x',
|
||||
IsWordAndSentenceCard: '',
|
||||
IsSentenceCard: '',
|
||||
IsAudioCard: '',
|
||||
});
|
||||
});
|
||||
|
||||
test('matches flag fields case-insensitively', () => {
|
||||
const fields: Record<string, string> = {};
|
||||
|
||||
applyCardKindFlagFields(fields, 'word-and-sentence', resolverFor(['iswordandsentencecard']));
|
||||
|
||||
assert.deepEqual(fields, { iswordandsentencecard: 'x' });
|
||||
});
|
||||
|
||||
test('leaves flags untouched when the note type has no flag for a word card kind', () => {
|
||||
const fields: Record<string, string> = {};
|
||||
|
||||
applyCardKindFlagFields(
|
||||
fields,
|
||||
'click',
|
||||
resolverFor(['IsWordAndSentenceCard', 'IsSentenceCard']),
|
||||
);
|
||||
|
||||
assert.deepEqual(fields, {});
|
||||
});
|
||||
|
||||
test('clears stale flags for explicit mine actions even without the target flag', () => {
|
||||
const fields: Record<string, string> = {};
|
||||
|
||||
applyCardKindFlagFields(
|
||||
fields,
|
||||
'audio',
|
||||
resolverFor(['IsWordAndSentenceCard', 'IsSentenceCard']),
|
||||
);
|
||||
|
||||
assert.deepEqual(fields, { IsWordAndSentenceCard: '', IsSentenceCard: '' });
|
||||
});
|
||||
|
||||
test('does not blank the target flag it just set', () => {
|
||||
const fields: Record<string, string> = {};
|
||||
|
||||
applyCardKindFlagFields(fields, 'sentence', resolverFor(['IsSentenceCard']));
|
||||
|
||||
assert.deepEqual(fields, { IsSentenceCard: 'x' });
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
import type { CardKind, WordCardKind } from '../types/anki';
|
||||
|
||||
/**
|
||||
* Kiku/Lapis note types decide which card a note generates from mutually exclusive
|
||||
* `Is...Card` flag fields. Setting one always means clearing the others.
|
||||
*/
|
||||
export const CARD_KIND_FLAG_FIELDS: Record<CardKind, string> = {
|
||||
'word-and-sentence': 'IsWordAndSentenceCard',
|
||||
click: 'IsClickCard',
|
||||
sentence: 'IsSentenceCard',
|
||||
audio: 'IsAudioCard',
|
||||
};
|
||||
|
||||
export const WORD_CARD_KINDS: readonly WordCardKind[] = [
|
||||
'word-and-sentence',
|
||||
'click',
|
||||
'sentence',
|
||||
'audio',
|
||||
'none',
|
||||
];
|
||||
|
||||
export const DEFAULT_WORD_CARD_KIND: WordCardKind = 'word-and-sentence';
|
||||
|
||||
/**
|
||||
* Card kinds SubMiner marks on its own initiative (word cards). They are only applied
|
||||
* when the note type actually carries the matching flag field, so plain note types keep
|
||||
* their fields untouched.
|
||||
*/
|
||||
const IMPLICIT_CARD_KINDS = new Set<CardKind>(['word-and-sentence', 'click']);
|
||||
|
||||
export function isWordCardKind(value: unknown): value is WordCardKind {
|
||||
return typeof value === 'string' && WORD_CARD_KINDS.includes(value as WordCardKind);
|
||||
}
|
||||
|
||||
export function resolveWordCardKindSetting(value: unknown): WordCardKind {
|
||||
return isWordCardKind(value) ? value : DEFAULT_WORD_CARD_KIND;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flags `cardKind` on the note and clears every other card-kind flag it has, so the note
|
||||
* never ends up claiming to be two kinds of card at once.
|
||||
*/
|
||||
export function applyCardKindFlagFields(
|
||||
updatedFields: Record<string, string>,
|
||||
cardKind: CardKind,
|
||||
resolveFieldName: (preferredName: string) => string | null,
|
||||
): void {
|
||||
const targetFlag = resolveFieldName(CARD_KIND_FLAG_FIELDS[cardKind]);
|
||||
if (!targetFlag && IMPLICIT_CARD_KINDS.has(cardKind)) {
|
||||
return;
|
||||
}
|
||||
if (targetFlag) {
|
||||
updatedFields[targetFlag] = 'x';
|
||||
}
|
||||
|
||||
for (const [kind, flagName] of Object.entries(CARD_KIND_FLAG_FIELDS)) {
|
||||
if (kind === cardKind) continue;
|
||||
const resolved = resolveFieldName(flagName);
|
||||
if (resolved && resolved !== targetFlag) {
|
||||
updatedFields[resolved] = '';
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import { resolveWordCardKind, type NoteFieldValueInfo } from './note-field-utils';
|
||||
|
||||
function kikuNote(values: Record<string, string> = {}): NoteFieldValueInfo {
|
||||
const defaults: Record<string, string> = {
|
||||
Expression: '単語',
|
||||
Sentence: '',
|
||||
IsWordAndSentenceCard: '',
|
||||
IsClickCard: '',
|
||||
IsSentenceCard: '',
|
||||
IsAudioCard: '',
|
||||
};
|
||||
return {
|
||||
fields: Object.fromEntries(
|
||||
Object.entries({ ...defaults, ...values }).map(([name, value]) => [name, { value }]),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
test('marks word-and-sentence cards by default when Kiku is enabled', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote(), { lapisEnabled: false, kikuEnabled: true }),
|
||||
'word-and-sentence',
|
||||
);
|
||||
});
|
||||
|
||||
test('honors the configured word card kind', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote(), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
wordCardKind: 'click',
|
||||
}),
|
||||
'click',
|
||||
);
|
||||
});
|
||||
|
||||
test('marks nothing when neither Kiku nor Lapis is enabled', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote(), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: false,
|
||||
wordCardKind: 'click',
|
||||
}),
|
||||
null,
|
||||
);
|
||||
});
|
||||
|
||||
test('marks nothing when the word card kind is "none"', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote(), {
|
||||
lapisEnabled: true,
|
||||
kikuEnabled: false,
|
||||
wordCardKind: 'none',
|
||||
}),
|
||||
null,
|
||||
);
|
||||
});
|
||||
|
||||
test('falls back to the default kind for an unrecognized setting', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote(), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
wordCardKind: 'bogus' as never,
|
||||
}),
|
||||
'word-and-sentence',
|
||||
);
|
||||
});
|
||||
|
||||
test('marks nothing when the note type lacks the configured flag field', () => {
|
||||
const note: NoteFieldValueInfo = {
|
||||
fields: { Expression: { value: '単語' }, Sentence: { value: '' } },
|
||||
};
|
||||
|
||||
assert.equal(
|
||||
resolveWordCardKind(note, { lapisEnabled: false, kikuEnabled: true, wordCardKind: 'click' }),
|
||||
null,
|
||||
);
|
||||
});
|
||||
|
||||
test('leaves cards already mined as sentence or audio cards alone', () => {
|
||||
for (const flagField of ['IsSentenceCard', 'IsAudioCard']) {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote({ [flagField]: 'x' }), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
wordCardKind: 'click',
|
||||
}),
|
||||
null,
|
||||
flagField,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('re-affirms the configured kind when the note already carries its flag', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote({ IsSentenceCard: 'x' }), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
wordCardKind: 'sentence',
|
||||
}),
|
||||
'sentence',
|
||||
);
|
||||
});
|
||||
|
||||
test('overrides a differently flagged word card', () => {
|
||||
assert.equal(
|
||||
resolveWordCardKind(kikuNote({ IsWordAndSentenceCard: 'x' }), {
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
wordCardKind: 'click',
|
||||
}),
|
||||
'click',
|
||||
);
|
||||
});
|
||||
@@ -1,3 +1,13 @@
|
||||
import type { CardKind, WordCardKind } from '../types/anki';
|
||||
import { createLogger } from '../logger';
|
||||
import {
|
||||
CARD_KIND_FLAG_FIELDS,
|
||||
DEFAULT_WORD_CARD_KIND,
|
||||
resolveWordCardKindSetting,
|
||||
} from './card-kinds';
|
||||
|
||||
const log = createLogger('anki').child('integration.note-fields');
|
||||
|
||||
export interface NoteFieldValueInfo {
|
||||
fields: Record<string, { value: string }>;
|
||||
}
|
||||
@@ -16,22 +26,57 @@ export function hasNoteFieldValue(noteInfo: NoteFieldValueInfo, preferredName: s
|
||||
return (getNoteFieldValue(noteInfo, preferredName) ?? '').trim().length > 0;
|
||||
}
|
||||
|
||||
export function shouldMarkWordAndSentenceCard(
|
||||
noteInfo: NoteFieldValueInfo,
|
||||
sentenceCardConfig: { lapisEnabled: boolean; kikuEnabled: boolean },
|
||||
): boolean {
|
||||
if (!sentenceCardConfig.lapisEnabled && !sentenceCardConfig.kikuEnabled) {
|
||||
return false;
|
||||
}
|
||||
/** Flags set only by an explicit mine action; a note carrying one is not a word card. */
|
||||
const EXPLICIT_CARD_FLAG_FIELDS = [CARD_KIND_FLAG_FIELDS.sentence, CARD_KIND_FLAG_FIELDS.audio];
|
||||
|
||||
const wordAndSentenceValue = getNoteFieldValue(noteInfo, 'IsWordAndSentenceCard');
|
||||
if (wordAndSentenceValue === null) {
|
||||
return false;
|
||||
const warnedMissingFlagFields = new Set<CardKind>();
|
||||
|
||||
function warnMissingFlagFieldOnce(wordCardKind: CardKind, flagField: string): void {
|
||||
if (wordCardKind === DEFAULT_WORD_CARD_KIND || warnedMissingFlagFields.has(wordCardKind)) {
|
||||
// The default kind is also the fallback for plain note types, so its absence is expected.
|
||||
return;
|
||||
}
|
||||
if (wordAndSentenceValue.trim().length > 0) {
|
||||
return true;
|
||||
}
|
||||
return (
|
||||
!hasNoteFieldValue(noteInfo, 'IsSentenceCard') && !hasNoteFieldValue(noteInfo, 'IsAudioCard')
|
||||
warnedMissingFlagFields.add(wordCardKind);
|
||||
log.warn(
|
||||
`Word card type "${wordCardKind}" is configured but the note has no ${flagField} field; leaving card type flags unchanged.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Card kind to flag when SubMiner fills a word card's sentence, or null to leave the
|
||||
* card-kind flags alone. Kiku/Lapis only: other note types have no such fields.
|
||||
*/
|
||||
export function resolveWordCardKind(
|
||||
noteInfo: NoteFieldValueInfo,
|
||||
sentenceCardConfig: {
|
||||
lapisEnabled: boolean;
|
||||
kikuEnabled: boolean;
|
||||
wordCardKind?: WordCardKind;
|
||||
},
|
||||
): CardKind | null {
|
||||
if (!sentenceCardConfig.lapisEnabled && !sentenceCardConfig.kikuEnabled) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const wordCardKind = resolveWordCardKindSetting(sentenceCardConfig.wordCardKind);
|
||||
if (wordCardKind === 'none') {
|
||||
return null;
|
||||
}
|
||||
|
||||
const flagField = CARD_KIND_FLAG_FIELDS[wordCardKind];
|
||||
const flagValue = getNoteFieldValue(noteInfo, flagField);
|
||||
if (flagValue === null) {
|
||||
// Note type has no flag field for the configured kind.
|
||||
warnMissingFlagFieldOnce(wordCardKind, flagField);
|
||||
return null;
|
||||
}
|
||||
if (flagValue.trim().length > 0) {
|
||||
return wordCardKind;
|
||||
}
|
||||
|
||||
const alreadyExplicitCard = EXPLICIT_CARD_FLAG_FIELDS.some(
|
||||
(fieldName) =>
|
||||
fieldName.toLowerCase() !== flagField.toLowerCase() && hasNoteFieldValue(noteInfo, fieldName),
|
||||
);
|
||||
return alreadyExplicitCard ? null : wordCardKind;
|
||||
}
|
||||
|
||||
@@ -6,26 +6,21 @@ import {
|
||||
type NoteUpdateWorkflowNoteInfo,
|
||||
} from './note-update-workflow';
|
||||
import type { SubtitleMiningContext } from '../types/subtitle';
|
||||
import type { CardKind } from '../types/anki';
|
||||
import { applyCardKindFlagFields } from './card-kinds';
|
||||
|
||||
function setWordAndSentenceCardTypeFields(
|
||||
function setCardTypeFields(
|
||||
updatedFields: Record<string, string>,
|
||||
availableFieldNames: string[],
|
||||
cardKind: 'word-and-sentence',
|
||||
cardKind: CardKind,
|
||||
): void {
|
||||
assert.equal(cardKind, 'word-and-sentence');
|
||||
const resolveFieldName = (preferredName: string): string | null =>
|
||||
availableFieldNames.find((name) => name.toLowerCase() === preferredName.toLowerCase()) ?? null;
|
||||
|
||||
const wordAndSentenceFlag = resolveFieldName('IsWordAndSentenceCard');
|
||||
if (!wordAndSentenceFlag) return;
|
||||
|
||||
updatedFields[wordAndSentenceFlag] = 'x';
|
||||
for (const flagName of ['IsSentenceCard', 'IsAudioCard']) {
|
||||
const resolved = resolveFieldName(flagName);
|
||||
if (resolved && resolved !== wordAndSentenceFlag) {
|
||||
updatedFields[resolved] = '';
|
||||
}
|
||||
}
|
||||
applyCardKindFlagFields(
|
||||
updatedFields,
|
||||
cardKind,
|
||||
(preferredName) =>
|
||||
availableFieldNames.find((name) => name.toLowerCase() === preferredName.toLowerCase()) ??
|
||||
null,
|
||||
);
|
||||
}
|
||||
|
||||
function createWorkflowHarness() {
|
||||
@@ -79,7 +74,7 @@ function createWorkflowHarness() {
|
||||
handleFieldGroupingManual: async (_originalNoteId, _newNoteId, _newNoteInfo, _expression) =>
|
||||
false,
|
||||
processSentence: (text: string, _noteFields: Record<string, string>) => text,
|
||||
setCardTypeFields: setWordAndSentenceCardTypeFields,
|
||||
setCardTypeFields,
|
||||
resolveConfiguredFieldName: (noteInfo: NoteUpdateWorkflowNoteInfo, preferred?: string) => {
|
||||
if (!preferred) return null;
|
||||
const names = Object.keys(noteInfo.fields);
|
||||
@@ -183,6 +178,73 @@ test('NoteUpdateWorkflow marks enriched Kiku word cards as word-and-sentence car
|
||||
});
|
||||
});
|
||||
|
||||
test('NoteUpdateWorkflow marks the configured word card kind instead of word-and-sentence', async () => {
|
||||
const harness = createWorkflowHarness();
|
||||
harness.deps.getEffectiveSentenceCardConfig = () => ({
|
||||
sentenceField: 'Sentence',
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
kikuFieldGrouping: 'manual',
|
||||
wordCardKind: 'click',
|
||||
});
|
||||
harness.deps.client.notesInfo = async () =>
|
||||
[
|
||||
{
|
||||
noteId: 42,
|
||||
fields: {
|
||||
Expression: { value: 'taberu' },
|
||||
Sentence: { value: '' },
|
||||
IsWordAndSentenceCard: { value: 'x' },
|
||||
IsClickCard: { value: '' },
|
||||
IsSentenceCard: { value: '' },
|
||||
IsAudioCard: { value: '' },
|
||||
},
|
||||
},
|
||||
] satisfies NoteUpdateWorkflowNoteInfo[];
|
||||
|
||||
await harness.workflow.execute(42);
|
||||
|
||||
assert.equal(harness.updates.length, 1);
|
||||
assert.deepEqual(harness.updates[0]?.fields, {
|
||||
Sentence: 'subtitle-text',
|
||||
IsClickCard: 'x',
|
||||
IsWordAndSentenceCard: '',
|
||||
IsSentenceCard: '',
|
||||
IsAudioCard: '',
|
||||
});
|
||||
});
|
||||
|
||||
test('NoteUpdateWorkflow leaves card type flags alone when the word card kind is none', async () => {
|
||||
const harness = createWorkflowHarness();
|
||||
harness.deps.getEffectiveSentenceCardConfig = () => ({
|
||||
sentenceField: 'Sentence',
|
||||
lapisEnabled: false,
|
||||
kikuEnabled: true,
|
||||
kikuFieldGrouping: 'manual',
|
||||
wordCardKind: 'none',
|
||||
});
|
||||
harness.deps.client.notesInfo = async () =>
|
||||
[
|
||||
{
|
||||
noteId: 42,
|
||||
fields: {
|
||||
Expression: { value: 'taberu' },
|
||||
Sentence: { value: '' },
|
||||
IsWordAndSentenceCard: { value: '' },
|
||||
IsSentenceCard: { value: '' },
|
||||
IsAudioCard: { value: '' },
|
||||
},
|
||||
},
|
||||
] satisfies NoteUpdateWorkflowNoteInfo[];
|
||||
|
||||
await harness.workflow.execute(42);
|
||||
|
||||
assert.equal(harness.updates.length, 1);
|
||||
assert.deepEqual(harness.updates[0]?.fields, {
|
||||
Sentence: 'subtitle-text',
|
||||
});
|
||||
});
|
||||
|
||||
test('NoteUpdateWorkflow does not set Kiku card flags when Lapis and Kiku are disabled', async () => {
|
||||
const harness = createWorkflowHarness();
|
||||
harness.deps.client.notesInfo = async () =>
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { DEFAULT_ANKI_CONNECT_CONFIG } from '../config';
|
||||
import { getPreferredWordValueFromExtractedFields } from '../anki-field-config';
|
||||
import type { SubtitleMiningContext } from '../types/subtitle';
|
||||
import { shouldMarkWordAndSentenceCard } from './note-field-utils';
|
||||
import type { CardKind, WordCardKind } from '../types/anki';
|
||||
import { resolveWordCardKind } from './note-field-utils';
|
||||
|
||||
export interface NoteUpdateWorkflowNoteInfo {
|
||||
noteId: number;
|
||||
@@ -39,6 +40,7 @@ export interface NoteUpdateWorkflowDeps {
|
||||
lapisEnabled: boolean;
|
||||
kikuEnabled: boolean;
|
||||
kikuFieldGrouping: 'auto' | 'manual' | 'disabled';
|
||||
wordCardKind?: WordCardKind;
|
||||
};
|
||||
appendKnownWordsFromNoteInfo: (noteInfo: NoteUpdateWorkflowNoteInfo) => void;
|
||||
extractFields: (fields: Record<string, { value: string }>) => Record<string, string>;
|
||||
@@ -67,7 +69,7 @@ export interface NoteUpdateWorkflowDeps {
|
||||
setCardTypeFields: (
|
||||
updatedFields: Record<string, string>,
|
||||
availableFieldNames: string[],
|
||||
cardKind: 'word-and-sentence',
|
||||
cardKind: CardKind,
|
||||
) => void;
|
||||
resolveConfiguredFieldName: (
|
||||
noteInfo: NoteUpdateWorkflowNoteInfo,
|
||||
@@ -207,12 +209,9 @@ export class NoteUpdateWorkflow {
|
||||
if (sentenceField && currentSubtitleText) {
|
||||
const processedSentence = this.deps.processSentence(currentSubtitleText, fields);
|
||||
updatedFields[sentenceField] = processedSentence;
|
||||
if (shouldMarkWordAndSentenceCard(noteInfo, sentenceCardConfig)) {
|
||||
this.deps.setCardTypeFields(
|
||||
updatedFields,
|
||||
Object.keys(noteInfo.fields),
|
||||
'word-and-sentence',
|
||||
);
|
||||
const wordCardKind = resolveWordCardKind(noteInfo, sentenceCardConfig);
|
||||
if (wordCardKind) {
|
||||
this.deps.setCardTypeFields(updatedFields, Object.keys(noteInfo.fields), wordCardKind);
|
||||
}
|
||||
updatePerformed = true;
|
||||
}
|
||||
|
||||
@@ -116,6 +116,10 @@ export function normalizeAnkiIntegrationConfig(config: AnkiConnectConfig): AnkiC
|
||||
...DEFAULT_ANKI_CONNECT_CONFIG.isKiku,
|
||||
...(config.isKiku ?? {}),
|
||||
},
|
||||
lapisKiku: {
|
||||
...DEFAULT_ANKI_CONNECT_CONFIG.lapisKiku,
|
||||
...(config.lapisKiku ?? {}),
|
||||
},
|
||||
} as AnkiConnectConfig;
|
||||
}
|
||||
|
||||
@@ -205,6 +209,10 @@ export class AnkiIntegrationRuntime {
|
||||
patch.isKiku !== undefined
|
||||
? { ...this.config.isKiku, ...patch.isKiku }
|
||||
: this.config.isKiku,
|
||||
lapisKiku:
|
||||
patch.lapisKiku !== undefined
|
||||
? { ...this.config.lapisKiku, ...patch.lapisKiku }
|
||||
: this.config.lapisKiku,
|
||||
};
|
||||
this.config = normalizeAnkiIntegrationConfig(mergedConfig);
|
||||
this.deps.onConfigChanged?.(this.config);
|
||||
|
||||
@@ -2738,6 +2738,43 @@ test('ignores deprecated isLapis sentence-card field overrides', () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('accepts a Kiku/Lapis word card kind and warns on an unknown one', () => {
|
||||
const dir = makeTempDir();
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'config.jsonc'),
|
||||
`{
|
||||
"ankiConnect": {
|
||||
"isKiku": { "enabled": true },
|
||||
"lapisKiku": { "wordCardKind": "click" }
|
||||
}
|
||||
}`,
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
const service = new ConfigService(dir);
|
||||
assert.equal(service.getConfig().ankiConnect.lapisKiku.wordCardKind, 'click');
|
||||
assert.equal(service.getWarnings().length, 0);
|
||||
|
||||
const invalidDir = makeTempDir();
|
||||
fs.writeFileSync(
|
||||
path.join(invalidDir, 'config.jsonc'),
|
||||
`{
|
||||
"ankiConnect": {
|
||||
"lapisKiku": { "wordCardKind": "isClickCard" }
|
||||
}
|
||||
}`,
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
const invalidService = new ConfigService(invalidDir);
|
||||
assert.equal(invalidService.getConfig().ankiConnect.lapisKiku.wordCardKind, 'word-and-sentence');
|
||||
assert.ok(
|
||||
invalidService
|
||||
.getWarnings()
|
||||
.some((warning) => warning.path === 'ankiConnect.lapisKiku.wordCardKind'),
|
||||
);
|
||||
});
|
||||
|
||||
test('accepts valid ankiConnect knownWords deck object', () => {
|
||||
const dir = makeTempDir();
|
||||
fs.writeFileSync(
|
||||
|
||||
@@ -91,6 +91,9 @@ export const INTEGRATIONS_DEFAULT_CONFIG: Pick<
|
||||
fieldGrouping: 'disabled',
|
||||
deleteDuplicateInAuto: true,
|
||||
},
|
||||
lapisKiku: {
|
||||
wordCardKind: 'word-and-sentence',
|
||||
},
|
||||
},
|
||||
jimaku: {
|
||||
apiBaseUrl: 'https://jimaku.cc',
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { ResolvedConfig } from '../../types/config';
|
||||
import { WORD_CARD_KINDS } from '../../anki-integration/card-kinds';
|
||||
import { MPV_LAUNCH_MODE_VALUES } from '../../shared/mpv-launch-mode';
|
||||
import {
|
||||
NOTIFICATION_TYPE_VALUES,
|
||||
@@ -374,6 +375,21 @@ export function buildIntegrationConfigOptionRegistry(
|
||||
defaultValue: defaultConfig.ankiConnect.isLapis.sentenceCardModel,
|
||||
description: 'Note type name used by Lapis sentence cards.',
|
||||
},
|
||||
{
|
||||
path: 'ankiConnect.lapisKiku.wordCardKind',
|
||||
kind: 'enum',
|
||||
enumValues: WORD_CARD_KINDS,
|
||||
enumLabels: {
|
||||
'word-and-sentence': 'Word and sentence card (IsWordAndSentenceCard)',
|
||||
click: 'Click card (IsClickCard)',
|
||||
sentence: 'Sentence card (IsSentenceCard)',
|
||||
audio: 'Audio card (IsAudioCard)',
|
||||
none: 'Leave card type flags untouched',
|
||||
},
|
||||
defaultValue: defaultConfig.ankiConnect.lapisKiku.wordCardKind,
|
||||
description:
|
||||
'Card-type flag SubMiner marks on Kiku/Lapis word cards. Only one flag is set at a time; the others are cleared. Requires isKiku.enabled or isLapis.enabled.',
|
||||
},
|
||||
{
|
||||
path: 'ankiConnect.metadata.pattern',
|
||||
kind: 'string',
|
||||
|
||||
@@ -135,7 +135,7 @@ const INTEGRATION_TEMPLATE_SECTIONS: ConfigTemplateSection[] = [
|
||||
title: 'AnkiConnect Integration',
|
||||
description: ['Automatic Anki updates and media generation options.'],
|
||||
notes: [
|
||||
'Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, and isKiku.fieldGrouping update live while SubMiner is running.',
|
||||
'Hot-reload: ankiConnect.ai.enabled, media.normalizeAudio/mirrorMpvVolume, knownWords, nPlusOne, fields.word/audio/image/sentence/miscInfo, behavior.autoUpdateNewCards, isLapis.sentenceCardModel, isKiku.fieldGrouping, and lapisKiku.wordCardKind update live while SubMiner is running.',
|
||||
'Shared AI provider transport settings are read from top-level ai and typically require restart.',
|
||||
'Most other AnkiConnect settings still require restart.',
|
||||
],
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { ResolveContext } from './context';
|
||||
import { initializeAnkiConnectResolution } from './anki-connect/initialize';
|
||||
import { applyAnkiKikuResolution } from './anki-connect/kiku';
|
||||
import { applyAnkiLapisKikuResolution } from './anki-connect/lapis-kiku';
|
||||
import { applyAnkiKnownWordsResolution } from './anki-connect/known-words';
|
||||
import { applyAnkiLegacyResolution } from './anki-connect/legacy';
|
||||
import { applyAnkiModernResolution } from './anki-connect/modern';
|
||||
@@ -22,4 +23,5 @@ export function applyAnkiConnectResolution(context: ResolveContext): void {
|
||||
applyAnkiLegacyResolution(context, ankiConnect, behavior, fields, media, metadata);
|
||||
applyAnkiKnownWordsResolution(context, ankiConnect, behavior);
|
||||
applyAnkiKikuResolution(context);
|
||||
applyAnkiLapisKikuResolution(context, ankiConnect);
|
||||
}
|
||||
|
||||
@@ -77,5 +77,8 @@ export function initializeAnkiConnectResolution(
|
||||
? (ankiConnect.isKiku as (typeof context.resolved)['ankiConnect']['isKiku'])
|
||||
: {}),
|
||||
},
|
||||
lapisKiku: {
|
||||
...context.resolved.ankiConnect.lapisKiku,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
import { isWordCardKind, WORD_CARD_KINDS } from '../../../anki-integration/card-kinds';
|
||||
import { DEFAULT_CONFIG } from '../../definitions';
|
||||
import type { ResolveContext } from '../context';
|
||||
import { isObject } from '../shared';
|
||||
|
||||
export function applyAnkiLapisKikuResolution(
|
||||
context: ResolveContext,
|
||||
ankiConnect: Record<string, unknown>,
|
||||
): void {
|
||||
if (!isObject(ankiConnect.lapisKiku)) {
|
||||
if (ankiConnect.lapisKiku !== undefined) {
|
||||
context.warn(
|
||||
'ankiConnect.lapisKiku',
|
||||
ankiConnect.lapisKiku,
|
||||
context.resolved.ankiConnect.lapisKiku,
|
||||
'Expected object.',
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const wordCardKind = ankiConnect.lapisKiku.wordCardKind;
|
||||
if (wordCardKind === undefined) {
|
||||
return;
|
||||
}
|
||||
if (isWordCardKind(wordCardKind)) {
|
||||
context.resolved.ankiConnect.lapisKiku.wordCardKind = wordCardKind;
|
||||
return;
|
||||
}
|
||||
|
||||
context.warn(
|
||||
'ankiConnect.lapisKiku.wordCardKind',
|
||||
wordCardKind,
|
||||
DEFAULT_CONFIG.ankiConnect.lapisKiku.wordCardKind,
|
||||
`Expected one of ${WORD_CARD_KINDS.join(', ')}.`,
|
||||
);
|
||||
context.resolved.ankiConnect.lapisKiku.wordCardKind =
|
||||
DEFAULT_CONFIG.ankiConnect.lapisKiku.wordCardKind;
|
||||
}
|
||||
@@ -221,6 +221,7 @@ const LABEL_OVERRIDES: Record<string, string> = {
|
||||
'ankiConnect.nPlusOne.enabled': 'Enabled',
|
||||
'ankiConnect.isLapis.enabled': 'Enable Lapis Features',
|
||||
'ankiConnect.isKiku.enabled': 'Enable Kiku Features',
|
||||
'ankiConnect.lapisKiku.wordCardKind': 'Word Card Type',
|
||||
'stats.toggleKey': 'Toggle Stats Overlay',
|
||||
'shortcuts.openCharacterDictionaryManager': 'Open Character Dictionary Manager',
|
||||
'subtitleSidebar.pauseVideoOnHover': 'Pause Video On Hover - Sidebar',
|
||||
@@ -255,6 +256,8 @@ const DESCRIPTION_OVERRIDES: Record<string, string> = {
|
||||
'Enable Lapis-specific mining behavior and sentence-card model targeting. When Kiku is enabled, Lapis features still work and Kiku-specific features are added on top.',
|
||||
'ankiConnect.isLapis.sentenceCardModel':
|
||||
'Anki note type used for Lapis sentence cards. Select from note types reported by AnkiConnect.',
|
||||
'ankiConnect.lapisKiku.wordCardKind':
|
||||
'Card-type flag marked on mined word cards. Only one flag is set at a time; the others are cleared. Requires Kiku or Lapis to be enabled.',
|
||||
'subtitleStyle.css':
|
||||
'CSS declarations applied to primary subtitles. Includes color, background-color, and all font properties.',
|
||||
'subtitleStyle.secondary.css':
|
||||
@@ -401,7 +404,11 @@ function categoryAndSection(path: string): { category: ConfigSettingsCategory; s
|
||||
if (path.startsWith('ankiConnect.media.')) {
|
||||
return { category: 'mining-anki', section: 'Media Capture' };
|
||||
}
|
||||
if (path.startsWith('ankiConnect.isKiku.') || path.startsWith('ankiConnect.isLapis.')) {
|
||||
if (
|
||||
path.startsWith('ankiConnect.isKiku.') ||
|
||||
path.startsWith('ankiConnect.isLapis.') ||
|
||||
path.startsWith('ankiConnect.lapisKiku.')
|
||||
) {
|
||||
return { category: 'mining-anki', section: 'Kiku/Lapis Features' };
|
||||
}
|
||||
if (path.startsWith('ankiConnect.ai.')) {
|
||||
@@ -702,6 +709,7 @@ function restartBehaviorForPath(path: string): ConfigSettingsRestartBehavior {
|
||||
path === 'ankiConnect.fields.miscInfo' ||
|
||||
path === 'ankiConnect.isLapis.sentenceCardModel' ||
|
||||
path === 'ankiConnect.isKiku.fieldGrouping' ||
|
||||
path === 'ankiConnect.lapisKiku.wordCardKind' ||
|
||||
path === 'mpv.aniskipEnabled' ||
|
||||
path === 'mpv.aniskipButtonKey' ||
|
||||
path === 'stats.toggleKey' ||
|
||||
|
||||
@@ -2454,6 +2454,80 @@ Aligned English subtitle
|
||||
});
|
||||
});
|
||||
|
||||
it('POST /api/stats/mine-card marks the configured Kiku word card kind', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
const sourcePath = path.join(dir, 'episode.mkv');
|
||||
fs.writeFileSync(sourcePath, 'fake media');
|
||||
|
||||
await withFakeAnkiConnect(
|
||||
async (requests, url) => {
|
||||
const app = createStatsApp(createMockTracker(), {
|
||||
addYomitanNote: async () => 777,
|
||||
createMediaGenerator: () => ({
|
||||
generateAudio: async () => null,
|
||||
generateScreenshot: async () => null,
|
||||
generateAnimatedImage: async () => null,
|
||||
}),
|
||||
ankiConnectConfig: {
|
||||
url,
|
||||
deck: 'Mining',
|
||||
fields: {
|
||||
image: 'Picture',
|
||||
sentence: 'Sentence',
|
||||
},
|
||||
media: {
|
||||
generateAudio: false,
|
||||
generateImage: false,
|
||||
},
|
||||
isKiku: {
|
||||
enabled: true,
|
||||
fieldGrouping: 'disabled',
|
||||
deleteDuplicateInAuto: true,
|
||||
},
|
||||
lapisKiku: {
|
||||
wordCardKind: 'click',
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const res = await app.request('/api/stats/mine-card?mode=word', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
sourcePath,
|
||||
startMs: 1_000,
|
||||
endMs: 2_000,
|
||||
sentence: '猫を見た',
|
||||
word: '猫',
|
||||
videoTitle: 'Episode 1',
|
||||
}),
|
||||
});
|
||||
|
||||
const body = await res.json();
|
||||
assert.equal(res.status, 200, JSON.stringify(body));
|
||||
|
||||
const updateRequest = requests.find((request) => request.action === 'updateNoteFields');
|
||||
const fields = updateRequest?.params?.note?.fields ?? {};
|
||||
assert.equal(fields.IsClickCard, 'x');
|
||||
assert.equal(fields.IsWordAndSentenceCard, '');
|
||||
assert.equal(fields.IsSentenceCard, '');
|
||||
assert.equal(fields.IsAudioCard, '');
|
||||
},
|
||||
{
|
||||
notesInfoFields: {
|
||||
Expression: { value: '猫' },
|
||||
Sentence: { value: '' },
|
||||
Picture: { value: '' },
|
||||
IsWordAndSentenceCard: { value: '' },
|
||||
IsClickCard: { value: '' },
|
||||
IsSentenceCard: { value: '' },
|
||||
IsAudioCard: { value: '' },
|
||||
},
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it('POST /api/stats/mine-card writes word mining sentence audio and image together', async () => {
|
||||
await withTempDir(async (dir) => {
|
||||
const sourcePath = path.join(dir, 'episode.mkv');
|
||||
|
||||
@@ -85,6 +85,7 @@ const HOT_RELOAD_EXACT_OR_PREFIX_PATHS = [
|
||||
'ankiConnect.fields.miscInfo',
|
||||
'ankiConnect.isLapis.sentenceCardModel',
|
||||
'ankiConnect.isKiku.fieldGrouping',
|
||||
'ankiConnect.lapisKiku.wordCardKind',
|
||||
] as const;
|
||||
|
||||
function hotReloadFieldForChangedPath(path: string): string | null {
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
export { Texthooker } from './texthooker';
|
||||
export { hasMpvWebsocketPlugin, SubtitleWebSocket } from './subtitle-ws';
|
||||
export {
|
||||
hasMpvWebsocketPlugin,
|
||||
isSubtitleAnnotationUpgrade,
|
||||
SubtitleWebSocket,
|
||||
} from './subtitle-ws';
|
||||
export { registerGlobalShortcuts } from './shortcut';
|
||||
export { createIpcDepsRuntime, registerIpcHandlers } from './ipc';
|
||||
export { shortcutMatchesInputForLocalFallback } from './shortcut-fallback';
|
||||
|
||||
@@ -205,7 +205,7 @@ test('runStartupBootstrapRuntime skips lifecycle when generate-config flow handl
|
||||
assert.deepEqual(calls, ['setLog:warn:cli', 'forceX11', 'enforceWayland']);
|
||||
});
|
||||
|
||||
test('runStartupBootstrapRuntime enables quiet background mode by default', () => {
|
||||
test('runStartupBootstrapRuntime lets config govern background log level by default', () => {
|
||||
const calls: string[] = [];
|
||||
const args = makeArgs({ background: true });
|
||||
|
||||
@@ -222,7 +222,7 @@ test('runStartupBootstrapRuntime enables quiet background mode by default', () =
|
||||
});
|
||||
|
||||
assert.equal(result.backgroundMode, true);
|
||||
assert.deepEqual(calls, ['setLog:warn:cli', 'forceX11', 'enforceWayland', 'startLifecycle']);
|
||||
assert.deepEqual(calls, ['forceX11', 'enforceWayland', 'startLifecycle']);
|
||||
});
|
||||
|
||||
test('runStartupBootstrapRuntime enables quiet update mode by default', () => {
|
||||
|
||||
@@ -45,7 +45,7 @@ export function runStartupBootstrapRuntime(
|
||||
|
||||
if (initialArgs.logLevel) {
|
||||
deps.setLogLevel(initialArgs.logLevel, 'cli');
|
||||
} else if (initialArgs.background || initialArgs.update) {
|
||||
} else if (initialArgs.update) {
|
||||
deps.setLogLevel('warn', 'cli');
|
||||
}
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ import {
|
||||
resolveSecondarySubtitleTextFromSidecar,
|
||||
} from '../secondary-subtitle-sidecar.js';
|
||||
import {
|
||||
applyStatsWordAndSentenceCardFields,
|
||||
applyStatsWordCardFields,
|
||||
createStatsMiningContext,
|
||||
getStatsDirectMiningAudioFieldNames,
|
||||
getStatsWordMiningAudioFieldName,
|
||||
@@ -272,7 +272,7 @@ export function registerStatsMiningRoutes(app: Hono, options?: StatsMiningRouteO
|
||||
const imageFieldName = ankiConfig.fields?.image ?? 'Picture';
|
||||
|
||||
mediaFields[sentenceFieldName] = highlightedSentence;
|
||||
applyStatsWordAndSentenceCardFields(mediaFields, noteInfo, ankiConfig);
|
||||
applyStatsWordCardFields(mediaFields, noteInfo, ankiConfig);
|
||||
|
||||
if (audioBuffer) {
|
||||
const audioFilename = `subminer_audio_${timestamp}_${noteId}.mp3`;
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import type { MediaGenerator } from '../../../media-generator.js';
|
||||
import type { AnkiConnectConfig } from '../../../types.js';
|
||||
import { applyCardKindFlagFields } from '../../../anki-integration/card-kinds.js';
|
||||
import { resolveWordCardKind } from '../../../anki-integration/note-field-utils.js';
|
||||
import { createLogger } from '../../../logger.js';
|
||||
import type { RetimedSecondarySubtitleInput } from '../secondary-subtitle-sidecar.js';
|
||||
|
||||
@@ -94,20 +96,22 @@ export function shouldUseStatsLapisKikuCardFields(ankiConfig: AnkiConnectConfig)
|
||||
return ankiConfig.isLapis?.enabled === true || ankiConfig.isKiku?.enabled === true;
|
||||
}
|
||||
|
||||
export function applyStatsWordAndSentenceCardFields(
|
||||
export function applyStatsWordCardFields(
|
||||
fields: Record<string, string>,
|
||||
noteInfo: StatsServerNoteInfo | null,
|
||||
ankiConfig: AnkiConnectConfig,
|
||||
): void {
|
||||
if (!shouldUseStatsLapisKikuCardFields(ankiConfig) || !noteInfo) return;
|
||||
const wordAndSentenceFlag = resolveStatsNoteFieldName(noteInfo, 'IsWordAndSentenceCard');
|
||||
if (!wordAndSentenceFlag) return;
|
||||
if (!noteInfo) return;
|
||||
const cardKind = resolveWordCardKind(noteInfo, {
|
||||
lapisEnabled: ankiConfig.isLapis?.enabled === true,
|
||||
kikuEnabled: ankiConfig.isKiku?.enabled === true,
|
||||
wordCardKind: ankiConfig.lapisKiku?.wordCardKind,
|
||||
});
|
||||
if (!cardKind) return;
|
||||
|
||||
fields[wordAndSentenceFlag] = 'x';
|
||||
for (const flagName of ['IsSentenceCard', 'IsAudioCard']) {
|
||||
const resolved = resolveStatsNoteFieldName(noteInfo, flagName);
|
||||
if (resolved && resolved !== wordAndSentenceFlag) fields[resolved] = '';
|
||||
}
|
||||
applyCardKindFlagFields(fields, cardKind, (preferredName) =>
|
||||
resolveStatsNoteFieldName(noteInfo, preferredName),
|
||||
);
|
||||
}
|
||||
|
||||
export function getStatsDirectMiningAudioFieldNames(
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
runSubsyncManual,
|
||||
triggerSubsyncFromConfig,
|
||||
} from './subsync';
|
||||
import type { SubsyncManualPayload } from '../../types';
|
||||
|
||||
function makeDeps(
|
||||
overrides: Partial<TriggerSubsyncFromConfigDeps> = {},
|
||||
@@ -76,7 +77,7 @@ test('triggerSubsyncFromConfig opens manual picker', async () => {
|
||||
await triggerSubsyncFromConfig(
|
||||
makeDeps({
|
||||
openManualPicker: (payload) => {
|
||||
payloadTrackCount = payload.sourceTracks.length;
|
||||
payloadTrackCount = payload.subtitleTracks.length;
|
||||
ffsubsyncAvailable = payload.ffsubsyncAvailable;
|
||||
},
|
||||
showMpvOsd: (text) => {
|
||||
@@ -88,9 +89,9 @@ test('triggerSubsyncFromConfig opens manual picker', async () => {
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(payloadTrackCount, 1);
|
||||
assert.equal(payloadTrackCount, 2);
|
||||
assert.equal(ffsubsyncAvailable, true);
|
||||
assert.ok(osd.includes('Subsync: choose engine and source'));
|
||||
assert.ok(osd.includes('Subsync: choose engine and subtitles'));
|
||||
assert.equal(inProgressState, false);
|
||||
});
|
||||
|
||||
@@ -140,7 +141,7 @@ test('triggerSubsyncFromConfig does not run automatic sync', async () => {
|
||||
await triggerSubsyncFromConfig(
|
||||
makeDeps({
|
||||
openManualPicker: (payload) => {
|
||||
payloadTrackCount = payload.sourceTracks.length;
|
||||
payloadTrackCount = payload.subtitleTracks.length;
|
||||
},
|
||||
showMpvOsd: (text) => {
|
||||
osd.push(text);
|
||||
@@ -152,9 +153,9 @@ test('triggerSubsyncFromConfig does not run automatic sync', async () => {
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(payloadTrackCount, 1);
|
||||
assert.equal(payloadTrackCount, 2);
|
||||
assert.equal(spinnerRan, false);
|
||||
assert.deepEqual(osd, ['Subsync: choose engine and source']);
|
||||
assert.deepEqual(osd, ['Subsync: choose engine and subtitles']);
|
||||
});
|
||||
|
||||
test('triggerSubsyncFromConfig dedupes repeated subtitle source tracks', async () => {
|
||||
@@ -195,12 +196,71 @@ test('triggerSubsyncFromConfig dedupes repeated subtitle source tracks', async (
|
||||
},
|
||||
}),
|
||||
openManualPicker: (payload) => {
|
||||
payloadTrackCount = payload.sourceTracks.length;
|
||||
payloadTrackCount = payload.subtitleTracks.length;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(payloadTrackCount, 1);
|
||||
assert.equal(payloadTrackCount, 2);
|
||||
});
|
||||
|
||||
test('triggerSubsyncFromConfig keeps both active tracks when they share a file', async () => {
|
||||
let payload: SubsyncManualPayload | null = null;
|
||||
|
||||
await triggerSubsyncFromConfig(
|
||||
makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: () => {},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return '/tmp/video.mkv';
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return 2;
|
||||
if (name === 'track-list') {
|
||||
// mpv appends a duplicate entry when the same file is re-added, so
|
||||
// the primary and secondary slots can point at one path.
|
||||
return [
|
||||
{
|
||||
id: 1,
|
||||
type: 'sub',
|
||||
selected: true,
|
||||
external: true,
|
||||
'external-filename': '/tmp/ref.srt',
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
type: 'sub',
|
||||
selected: true,
|
||||
external: true,
|
||||
'external-filename': '/tmp/ref.srt',
|
||||
},
|
||||
{
|
||||
id: 3,
|
||||
type: 'sub',
|
||||
selected: false,
|
||||
external: true,
|
||||
'external-filename': '/tmp/ref.srt',
|
||||
},
|
||||
];
|
||||
}
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
openManualPicker: (nextPayload) => {
|
||||
payload = nextPayload;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(payload);
|
||||
const resolved = payload as SubsyncManualPayload;
|
||||
assert.deepEqual(
|
||||
resolved.subtitleTracks.map((track) => track.id),
|
||||
[1, 2],
|
||||
);
|
||||
assert.equal(resolved.defaultReferenceTrackId, 2);
|
||||
assert.equal(resolved.defaultTargetTrackId, 1);
|
||||
});
|
||||
|
||||
test('triggerSubsyncFromConfig reports failures to OSD', async () => {
|
||||
@@ -217,15 +277,157 @@ test('triggerSubsyncFromConfig reports failures to OSD', async () => {
|
||||
assert.ok(osd.some((line) => line.startsWith('Subsync failed: MPV not connected')));
|
||||
});
|
||||
|
||||
test('runSubsyncManual requires a source track for alass', async () => {
|
||||
const result = await runSubsyncManual({ engine: 'alass', sourceTrackId: null }, makeDeps());
|
||||
test('runSubsyncManual requires a reference track for alass', async () => {
|
||||
const result = await runSubsyncManual({ engine: 'alass', referenceTrackId: null }, makeDeps());
|
||||
|
||||
assert.deepEqual(result, {
|
||||
ok: false,
|
||||
message: 'Select a subtitle source track for alass',
|
||||
message: 'Select a reference subtitle track for alass',
|
||||
});
|
||||
});
|
||||
|
||||
test('runSubsyncManual rejects alass when reference and target are the same track', async () => {
|
||||
const result = await runSubsyncManual(
|
||||
{ engine: 'alass', referenceTrackId: 2, targetTrackId: 2 },
|
||||
makeDeps(),
|
||||
);
|
||||
|
||||
assert.deepEqual(result, {
|
||||
ok: false,
|
||||
message: 'Reference and out-of-sync subtitles must be different tracks',
|
||||
});
|
||||
});
|
||||
|
||||
test('runSubsyncManual rejects an unknown target track', async () => {
|
||||
const result = await runSubsyncManual(
|
||||
{ engine: 'alass', referenceTrackId: 2, targetTrackId: 99 },
|
||||
makeDeps(),
|
||||
);
|
||||
|
||||
assert.deepEqual(result, {
|
||||
ok: false,
|
||||
message: 'Select the out-of-sync subtitle track to retime',
|
||||
});
|
||||
});
|
||||
|
||||
test('runSubsyncManual rejects the video reference for remote media', async () => {
|
||||
const result = await runSubsyncManual(
|
||||
{ engine: 'alass', referenceMode: 'video' },
|
||||
makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: () => {},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return 'https://jellyfin.example/Videos/movie/stream.mkv';
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return null;
|
||||
if (name === 'track-list') {
|
||||
return [{ id: 1, type: 'sub', selected: true, lang: 'jpn' }];
|
||||
}
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.message, /cannot use a stream URL as reference/);
|
||||
});
|
||||
|
||||
test('openSubsyncManualPicker defaults the reference to the secondary subtitle track', async () => {
|
||||
let payload: SubsyncManualPayload | null = null;
|
||||
|
||||
await triggerSubsyncFromConfig(
|
||||
makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: () => {},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return '/tmp/video.mkv';
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return 3;
|
||||
if (name === 'track-list') {
|
||||
return [
|
||||
{ id: 1, type: 'sub', selected: true, lang: 'jpn' },
|
||||
{
|
||||
id: 2,
|
||||
type: 'sub',
|
||||
selected: false,
|
||||
external: true,
|
||||
lang: 'eng',
|
||||
'external-filename': '/tmp/other.srt',
|
||||
},
|
||||
{
|
||||
id: 3,
|
||||
type: 'sub',
|
||||
selected: true,
|
||||
external: true,
|
||||
lang: 'eng',
|
||||
'external-filename': '/tmp/secondary.srt',
|
||||
},
|
||||
];
|
||||
}
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
openManualPicker: (nextPayload) => {
|
||||
payload = nextPayload;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(payload);
|
||||
const resolved = payload as SubsyncManualPayload;
|
||||
assert.deepEqual(
|
||||
resolved.subtitleTracks.map((track) => track.id),
|
||||
[1, 2, 3],
|
||||
);
|
||||
assert.equal(resolved.defaultReferenceTrackId, 3);
|
||||
assert.equal(resolved.defaultTargetTrackId, 1);
|
||||
assert.equal(resolved.videoReferenceAvailable, true);
|
||||
});
|
||||
|
||||
test('openSubsyncManualPicker never defaults to a reference missing from the track list', async () => {
|
||||
let payload: SubsyncManualPayload | null = null;
|
||||
|
||||
await triggerSubsyncFromConfig(
|
||||
makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: () => {},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return '/tmp/video.mkv';
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return 2;
|
||||
if (name === 'track-list') {
|
||||
return [
|
||||
{ id: 1, type: 'sub', selected: true, lang: 'jpn' },
|
||||
// Secondary track with no usable file path: filtered out of the picker.
|
||||
{ id: 2, type: 'sub', selected: true, external: true, 'external-filename': '' },
|
||||
{ id: 3, type: 'sub', selected: false, lang: 'eng' },
|
||||
];
|
||||
}
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
openManualPicker: (nextPayload) => {
|
||||
payload = nextPayload;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(payload);
|
||||
const resolved = payload as SubsyncManualPayload;
|
||||
assert.deepEqual(
|
||||
resolved.subtitleTracks.map((track) => track.id),
|
||||
[1, 3],
|
||||
);
|
||||
assert.equal(resolved.defaultReferenceTrackId, 3);
|
||||
});
|
||||
|
||||
test('triggerSubsyncFromConfig does not validate sync tool paths before manual selection', async () => {
|
||||
const osd: string[] = [];
|
||||
const inProgress: boolean[] = [];
|
||||
@@ -242,7 +444,7 @@ test('triggerSubsyncFromConfig does not validate sync tool paths before manual s
|
||||
inProgress.push(value);
|
||||
},
|
||||
openManualPicker: (payload) => {
|
||||
payloadTrackCount = payload.sourceTracks.length;
|
||||
payloadTrackCount = payload.subtitleTracks.length;
|
||||
},
|
||||
showMpvOsd: (text) => {
|
||||
osd.push(text);
|
||||
@@ -251,8 +453,8 @@ test('triggerSubsyncFromConfig does not validate sync tool paths before manual s
|
||||
);
|
||||
|
||||
assert.deepEqual(inProgress, [false]);
|
||||
assert.equal(payloadTrackCount, 1);
|
||||
assert.deepEqual(osd, ['Subsync: choose engine and source']);
|
||||
assert.equal(payloadTrackCount, 2);
|
||||
assert.deepEqual(osd, ['Subsync: choose engine and subtitles']);
|
||||
});
|
||||
|
||||
function writeExecutableScript(filePath: string, content: string): void {
|
||||
@@ -333,7 +535,7 @@ test('runSubsyncManual constructs ffsubsync command and returns success', async
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync', sourceTrackId: null }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync' }, deps);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
assert.equal(result.message, 'Subtitle synchronized with ffsubsync');
|
||||
@@ -346,7 +548,7 @@ test('runSubsyncManual constructs ffsubsync command and returns success', async
|
||||
const ffOutputFlagIndex = ffArgs.indexOf('-o');
|
||||
assert.equal(ffOutputFlagIndex >= 0, true);
|
||||
assert.equal(ffArgs[ffOutputFlagIndex + 1], toShellPath(primaryPath));
|
||||
assert.equal(sentCommands[0]?.[0], 'sub_add');
|
||||
assert.equal(sentCommands[0]?.[0], 'sub-add');
|
||||
assert.deepEqual(sentCommands[1], ['set_property', 'sub-delay', 0]);
|
||||
});
|
||||
|
||||
@@ -399,7 +601,7 @@ test('runSubsyncManual writes deterministic _retimed filename when replace is fa
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync', sourceTrackId: null }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync' }, deps);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
const ffArgs = fs.readFileSync(ffsubsyncLogPath, 'utf8').trim().split('\n');
|
||||
@@ -453,7 +655,7 @@ test('runSubsyncManual reports ffsubsync command failures with details', async (
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync', sourceTrackId: null }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync' }, deps);
|
||||
|
||||
assert.equal(result.ok, false);
|
||||
assert.equal(result.message.startsWith('ffsubsync synchronization failed'), true);
|
||||
@@ -518,7 +720,7 @@ test('runSubsyncManual constructs alass command and returns failure on non-zero
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'alass', sourceTrackId: 2 }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'alass', referenceTrackId: 2 }, deps);
|
||||
|
||||
assert.equal(result.ok, false);
|
||||
assert.equal(typeof result.message, 'string');
|
||||
@@ -528,6 +730,179 @@ test('runSubsyncManual constructs alass command and returns failure on non-zero
|
||||
assert.equal(alassArgs[1], toShellPath(primaryPath));
|
||||
});
|
||||
|
||||
function makeAlassSelectionDeps(tmpDir: string): {
|
||||
deps: TriggerSubsyncFromConfigDeps;
|
||||
alassLogPath: string;
|
||||
videoPath: string;
|
||||
primaryPath: string;
|
||||
sourcePath: string;
|
||||
sentCommands: Array<Array<string | number>>;
|
||||
} {
|
||||
const alassLogPath = path.join(tmpDir, 'alass-args.log');
|
||||
const alassPath = path.join(tmpDir, 'alass.sh');
|
||||
const ffmpegPath = path.join(tmpDir, 'ffmpeg.sh');
|
||||
const ffsubsyncPath = path.join(tmpDir, 'ffsubsync.sh');
|
||||
const videoPath = path.join(tmpDir, 'video.mkv');
|
||||
const primaryPath = path.join(tmpDir, 'primary.srt');
|
||||
const sourcePath = path.join(tmpDir, 'source.srt');
|
||||
|
||||
fs.writeFileSync(videoPath, 'video');
|
||||
fs.writeFileSync(primaryPath, 'sub');
|
||||
fs.writeFileSync(sourcePath, 'sub2');
|
||||
writeExecutableScript(ffmpegPath, '#!/bin/sh\nexit 0\n');
|
||||
writeExecutableScript(ffsubsyncPath, '#!/bin/sh\nexit 0\n');
|
||||
writeExecutableScript(
|
||||
alassPath,
|
||||
`#!/bin/sh\n: > "${toShellPath(alassLogPath)}"\nfor arg in "$@"; do printf '%s\\n' "$arg" >> "${toShellPath(alassLogPath)}"; done\n: > "$3"\nexit 0\n`,
|
||||
);
|
||||
|
||||
const trackList: Array<Record<string, unknown>> = [
|
||||
{ id: 1, type: 'sub', selected: true, external: true, 'external-filename': primaryPath },
|
||||
{ id: 2, type: 'sub', selected: true, external: true, 'external-filename': sourcePath },
|
||||
];
|
||||
const sentCommands: Array<Array<string | number>> = [];
|
||||
const deps = makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: (payload) => {
|
||||
sentCommands.push(payload.command);
|
||||
if (payload.command[0] === 'sub-add' || payload.command[0] === 'sub_add') {
|
||||
trackList.push({
|
||||
id: trackList.length + 1,
|
||||
type: 'sub',
|
||||
selected: false,
|
||||
external: true,
|
||||
'external-filename': payload.command[1],
|
||||
});
|
||||
}
|
||||
},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return videoPath;
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return 2;
|
||||
if (name === 'track-list') return trackList;
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
getResolvedConfig: () => ({
|
||||
alassPath,
|
||||
ffsubsyncPath,
|
||||
ffmpegPath,
|
||||
replace: false,
|
||||
}),
|
||||
});
|
||||
|
||||
return { deps, alassLogPath, videoPath, primaryPath, sourcePath, sentCommands };
|
||||
}
|
||||
|
||||
test('runSubsyncManual uses the video file as alass reference when requested', async () => {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subsync-alass-video-ref-'));
|
||||
const { deps, alassLogPath, videoPath, primaryPath } = makeAlassSelectionDeps(tmpDir);
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'alass', referenceMode: 'video' }, deps);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
const alassArgs = fs.readFileSync(alassLogPath, 'utf8').trim().split('\n');
|
||||
assert.equal(alassArgs[0], toShellPath(videoPath));
|
||||
assert.equal(alassArgs[1], toShellPath(primaryPath));
|
||||
});
|
||||
|
||||
test('runSubsyncManual retimes the selected target track instead of the primary', async () => {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subsync-alass-target-'));
|
||||
const { deps, alassLogPath, primaryPath, sourcePath, sentCommands } =
|
||||
makeAlassSelectionDeps(tmpDir);
|
||||
|
||||
const result = await runSubsyncManual(
|
||||
{ engine: 'alass', referenceTrackId: 1, targetTrackId: 2 },
|
||||
deps,
|
||||
);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
const alassArgs = fs.readFileSync(alassLogPath, 'utf8').trim().split('\n');
|
||||
assert.equal(alassArgs[0], toShellPath(primaryPath));
|
||||
assert.equal(alassArgs[1], toShellPath(sourcePath));
|
||||
assert.equal(sentCommands[0]?.[0], 'sub-add');
|
||||
assert.equal(sentCommands[0]?.[2], 'auto');
|
||||
});
|
||||
|
||||
test('runSubsyncManual keeps a retimed secondary track in the secondary slot', async () => {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subsync-alass-secondary-slot-'));
|
||||
const alassPath = path.join(tmpDir, 'alass.sh');
|
||||
const ffmpegPath = path.join(tmpDir, 'ffmpeg.sh');
|
||||
const ffsubsyncPath = path.join(tmpDir, 'ffsubsync.sh');
|
||||
const videoPath = path.join(tmpDir, 'video.mkv');
|
||||
const primaryPath = path.join(tmpDir, 'ja.srt');
|
||||
const secondaryPath = path.join(tmpDir, 'en.srt');
|
||||
const retimedPath = path.join(tmpDir, 'en_retimed.srt');
|
||||
|
||||
fs.writeFileSync(videoPath, 'video');
|
||||
fs.writeFileSync(primaryPath, 'ja');
|
||||
fs.writeFileSync(secondaryPath, 'en');
|
||||
writeExecutableScript(ffmpegPath, '#!/bin/sh\nexit 0\n');
|
||||
writeExecutableScript(ffsubsyncPath, '#!/bin/sh\nexit 0\n');
|
||||
writeExecutableScript(alassPath, '#!/bin/sh\n: > "$3"\nexit 0\n');
|
||||
|
||||
const trackList: Array<Record<string, unknown>> = [
|
||||
{ id: 1, type: 'sub', selected: true, external: true, 'external-filename': primaryPath },
|
||||
{ id: 2, type: 'sub', selected: true, external: true, 'external-filename': secondaryPath },
|
||||
];
|
||||
const sentCommands: Array<Array<string | number>> = [];
|
||||
const deps = makeDeps({
|
||||
getMpvClient: () => ({
|
||||
connected: true,
|
||||
currentAudioStreamIndex: null,
|
||||
send: (payload) => {
|
||||
sentCommands.push(payload.command);
|
||||
if (payload.command[0] === 'sub-add') {
|
||||
trackList.push({
|
||||
id: 3,
|
||||
type: 'sub',
|
||||
selected: false,
|
||||
external: true,
|
||||
'external-filename': payload.command[1],
|
||||
});
|
||||
}
|
||||
},
|
||||
requestProperty: async (name: string) => {
|
||||
if (name === 'path') return videoPath;
|
||||
if (name === 'sid') return 1;
|
||||
if (name === 'secondary-sid') return 2;
|
||||
if (name === 'track-list') return trackList;
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
getResolvedConfig: () => ({
|
||||
alassPath,
|
||||
ffsubsyncPath,
|
||||
ffmpegPath,
|
||||
replace: false,
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual(
|
||||
{ engine: 'alass', referenceTrackId: 1, targetTrackId: 2 },
|
||||
deps,
|
||||
);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
assert.deepEqual(sentCommands[0], ['sub-add', retimedPath, 'auto']);
|
||||
assert.deepEqual(sentCommands[1], ['set_property', 'secondary-sub-delay', 0]);
|
||||
assert.deepEqual(sentCommands[2], ['set_property', 'secondary-sid', 3]);
|
||||
assert.equal(
|
||||
sentCommands.some((command) => command[1] === 'sub-delay'),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
sentCommands.some((command) => command[1] === 'sid'),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
sentCommands.some((command) => command[1] === 'sid'),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('runSubsyncManual keeps internal alass source file alive until sync finishes', async () => {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subsync-alass-internal-source-'));
|
||||
const alassPath = path.join(tmpDir, 'alass.sh');
|
||||
@@ -589,11 +964,11 @@ test('runSubsyncManual keeps internal alass source file alive until sync finishe
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'alass', sourceTrackId: 2 }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'alass', referenceTrackId: 2 }, deps);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
assert.equal(result.message, 'Subtitle synchronized with alass');
|
||||
assert.equal(sentCommands[0]?.[0], 'sub_add');
|
||||
assert.equal(sentCommands[0]?.[0], 'sub-add');
|
||||
assert.deepEqual(sentCommands[1], ['set_property', 'sub-delay', 0]);
|
||||
});
|
||||
|
||||
@@ -645,7 +1020,7 @@ test('runSubsyncManual resolves string sid values from mpv stream properties', a
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync', sourceTrackId: null }, deps);
|
||||
const result = await runSubsyncManual({ engine: 'ffsubsync' }, deps);
|
||||
|
||||
assert.equal(result.ok, true);
|
||||
assert.equal(result.message, 'Subtitle synchronized with ffsubsync');
|
||||
|
||||
+195
-34
@@ -21,6 +21,11 @@ interface FileExtractionResult {
|
||||
temporary: boolean;
|
||||
}
|
||||
|
||||
type SubtitleSlot = 'primary' | 'secondary';
|
||||
|
||||
const SYNCED_TRACK_LOOKUP_ATTEMPTS = 5;
|
||||
const SYNCED_TRACK_LOOKUP_RETRY_MS = 100;
|
||||
|
||||
function summarizeCommandFailure(command: string, result: CommandResult): string {
|
||||
const parts = [
|
||||
`code=${result.code ?? 'n/a'}`,
|
||||
@@ -90,16 +95,28 @@ function getSourceTrackIdentity(track: MpvTrack): string {
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
function dedupeSourceTracks(tracks: MpvTrack[]): MpvTrack[] {
|
||||
const deduped = new Map<string, MpvTrack>();
|
||||
function isPinned(track: MpvTrack, pinnedIds: Set<number>): boolean {
|
||||
return typeof track.id === 'number' && pinnedIds.has(track.id);
|
||||
}
|
||||
|
||||
// Pinned tracks (the active primary/secondary) always survive, even when two of
|
||||
// them point at the same file; only unpinned duplicates are collapsed.
|
||||
function dedupeSubtitleTracks(tracks: MpvTrack[], pinnedIds: Set<number>): MpvTrack[] {
|
||||
const pinnedIdentities = new Set(
|
||||
tracks.filter((track) => isPinned(track, pinnedIds)).map(getSourceTrackIdentity),
|
||||
);
|
||||
const winners = new Map<string, MpvTrack>();
|
||||
for (const track of tracks) {
|
||||
if (isPinned(track, pinnedIds)) continue;
|
||||
const identity = getSourceTrackIdentity(track);
|
||||
const existing = deduped.get(identity);
|
||||
if (pinnedIdentities.has(identity)) continue;
|
||||
const existing = winners.get(identity);
|
||||
if (!existing || (track.selected && !existing.selected)) {
|
||||
deduped.set(identity, track);
|
||||
winners.set(identity, track);
|
||||
}
|
||||
}
|
||||
return [...deduped.values()];
|
||||
const kept = new Set(winners.values());
|
||||
return tracks.filter((track) => isPinned(track, pinnedIds) || kept.has(track));
|
||||
}
|
||||
|
||||
export interface TriggerSubsyncFromConfigDeps extends SubsyncCoreDeps {
|
||||
@@ -142,20 +159,21 @@ async function gatherSubsyncContext(client: MpvClientLike): Promise<SubsyncConte
|
||||
}
|
||||
|
||||
const secondaryTrack = subtitleTracks.find((track) => track.id === secondarySid) ?? null;
|
||||
const sourceTracks = subtitleTracks
|
||||
.filter((track) => track.id !== sid)
|
||||
.filter((track) => {
|
||||
const usableTracks = subtitleTracks.filter((track) => {
|
||||
if (typeof track.id !== 'number') return false;
|
||||
if (!track.external) return true;
|
||||
const filename = track['external-filename'];
|
||||
return typeof filename === 'string' && filename.length > 0;
|
||||
});
|
||||
const uniqueSourceTracks = dedupeSourceTracks(sourceTracks);
|
||||
|
||||
return {
|
||||
videoPath,
|
||||
primaryTrack,
|
||||
secondaryTrack,
|
||||
sourceTracks: uniqueSourceTracks,
|
||||
subtitleTracks: dedupeSubtitleTracks(
|
||||
usableTracks,
|
||||
new Set([sid, secondarySid].filter((id): id is number => typeof id === 'number')),
|
||||
),
|
||||
audioStreamIndex: client.currentAudioStreamIndex,
|
||||
};
|
||||
}
|
||||
@@ -271,41 +289,104 @@ async function runFfsubsyncSync(
|
||||
return runCommand(ffsubsyncPath, args);
|
||||
}
|
||||
|
||||
function loadSyncedSubtitle(client: MpvClientLike, pathToLoad: string): void {
|
||||
function delay(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// mpv may echo the path back with different separators, and Windows paths are
|
||||
// case-insensitive, so compare normalized forms instead of raw strings.
|
||||
function normalizeSubtitlePathForCompare(value: string): string {
|
||||
const normalized = value.replace(/\\/g, '/');
|
||||
return process.platform === 'win32' ? normalized.toLowerCase() : normalized;
|
||||
}
|
||||
|
||||
async function findAddedSubtitleTrackId(
|
||||
client: MpvClientLike,
|
||||
pathToLoad: string,
|
||||
): Promise<number | null> {
|
||||
const wanted = normalizeSubtitlePathForCompare(pathToLoad);
|
||||
// sub-add is queued, so the track may not appear in the first track-list reply.
|
||||
for (let attempt = 0; attempt < SYNCED_TRACK_LOOKUP_ATTEMPTS; attempt += 1) {
|
||||
let tracks: MpvTrack[] = [];
|
||||
try {
|
||||
const trackListRaw = await client.requestProperty('track-list');
|
||||
tracks = Array.isArray(trackListRaw) ? normalizeTrackIds(trackListRaw as MpvTrack[]) : [];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
// Re-adding a file mpv already knows appends a duplicate entry; the newest
|
||||
// one holds the retimed content, so prefer the last match.
|
||||
const matches = tracks.filter((track) => {
|
||||
if (track.type !== 'sub') return false;
|
||||
const filename = track['external-filename'];
|
||||
return typeof filename === 'string' && normalizeSubtitlePathForCompare(filename) === wanted;
|
||||
});
|
||||
const added = matches[matches.length - 1];
|
||||
if (added && typeof added.id === 'number') {
|
||||
return added.id;
|
||||
}
|
||||
if (attempt < SYNCED_TRACK_LOOKUP_ATTEMPTS - 1) {
|
||||
await delay(SYNCED_TRACK_LOOKUP_RETRY_MS);
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function loadSyncedSubtitle(
|
||||
client: MpvClientLike,
|
||||
pathToLoad: string,
|
||||
slot: SubtitleSlot,
|
||||
): Promise<void> {
|
||||
if (!client.connected) {
|
||||
throw new Error('MPV disconnected while loading subtitle');
|
||||
}
|
||||
client.send({ command: ['sub_add', pathToLoad] });
|
||||
|
||||
if (slot === 'secondary') {
|
||||
// Keep the primary track untouched: load without selecting, then point
|
||||
// secondary-sid at the freshly added track.
|
||||
client.send({ command: ['sub-add', pathToLoad, 'auto'] });
|
||||
client.send({ command: ['set_property', 'secondary-sub-delay', 0] });
|
||||
const addedTrackId = await findAddedSubtitleTrackId(client, pathToLoad);
|
||||
if (addedTrackId === null) {
|
||||
throw new Error('Synchronized subtitle did not appear in the mpv track list');
|
||||
}
|
||||
client.send({ command: ['set_property', 'secondary-sid', addedTrackId] });
|
||||
return;
|
||||
}
|
||||
|
||||
client.send({ command: ['sub-add', pathToLoad] });
|
||||
client.send({ command: ['set_property', 'sub-delay', 0] });
|
||||
}
|
||||
|
||||
async function subsyncToReference(
|
||||
engine: 'alass' | 'ffsubsync',
|
||||
referenceFilePath: string,
|
||||
targetTrack: MpvTrack,
|
||||
context: SubsyncContext,
|
||||
resolved: SubsyncResolvedConfig,
|
||||
client: MpvClientLike,
|
||||
slot: SubtitleSlot,
|
||||
): Promise<SubsyncResult> {
|
||||
const ffmpegPath = ensureExecutablePath(resolved.ffmpegPath, 'ffmpeg');
|
||||
const primaryExtraction = await extractSubtitleTrackToFile(
|
||||
const targetExtraction = await extractSubtitleTrackToFile(
|
||||
ffmpegPath,
|
||||
context.videoPath,
|
||||
context.primaryTrack,
|
||||
targetTrack,
|
||||
);
|
||||
const replacePrimary = resolved.replace !== false && !primaryExtraction.temporary;
|
||||
const outputPath = buildRetimedPath(primaryExtraction.path, replacePrimary);
|
||||
const replaceTarget = resolved.replace !== false && !targetExtraction.temporary;
|
||||
const outputPath = buildRetimedPath(targetExtraction.path, replaceTarget);
|
||||
|
||||
try {
|
||||
let result: CommandResult;
|
||||
if (engine === 'alass') {
|
||||
const alassPath = ensureExecutablePath(resolved.alassPath, 'alass');
|
||||
result = await runAlassSync(alassPath, referenceFilePath, primaryExtraction.path, outputPath);
|
||||
result = await runAlassSync(alassPath, referenceFilePath, targetExtraction.path, outputPath);
|
||||
} else {
|
||||
const ffsubsyncPath = ensureExecutablePath(resolved.ffsubsyncPath, 'ffsubsync');
|
||||
result = await runFfsubsyncSync(
|
||||
ffsubsyncPath,
|
||||
context.videoPath,
|
||||
primaryExtraction.path,
|
||||
targetExtraction.path,
|
||||
outputPath,
|
||||
context.audioStreamIndex,
|
||||
);
|
||||
@@ -319,13 +400,13 @@ async function subsyncToReference(
|
||||
};
|
||||
}
|
||||
|
||||
loadSyncedSubtitle(client, outputPath);
|
||||
await loadSyncedSubtitle(client, outputPath, slot);
|
||||
return {
|
||||
ok: true,
|
||||
message: `Subtitle synchronized with ${engine}`,
|
||||
};
|
||||
} finally {
|
||||
cleanupTemporaryFile(primaryExtraction);
|
||||
cleanupTemporaryFile(targetExtraction);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -337,6 +418,25 @@ function validateFfsubsyncReference(videoPath: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
function resolveTargetTrack(
|
||||
request: SubsyncManualRunRequest,
|
||||
context: SubsyncContext,
|
||||
): MpvTrack | null {
|
||||
if (request.targetTrackId === undefined || request.targetTrackId === null) {
|
||||
return context.primaryTrack;
|
||||
}
|
||||
return getTrackById(context.subtitleTracks, request.targetTrackId);
|
||||
}
|
||||
|
||||
// Retiming the secondary track must not steal the primary slot: the synced file
|
||||
// goes back where the out-of-sync one was.
|
||||
function resolveTargetSlot(targetTrack: MpvTrack, context: SubsyncContext): SubtitleSlot {
|
||||
if (typeof targetTrack.id !== 'number') return 'primary';
|
||||
if (targetTrack.id === context.primaryTrack.id) return 'primary';
|
||||
if (context.secondaryTrack && targetTrack.id === context.secondaryTrack.id) return 'secondary';
|
||||
return 'primary';
|
||||
}
|
||||
|
||||
export async function runSubsyncManual(
|
||||
request: SubsyncManualRunRequest,
|
||||
deps: SubsyncCoreDeps,
|
||||
@@ -345,6 +445,12 @@ export async function runSubsyncManual(
|
||||
const context = await gatherSubsyncContext(client);
|
||||
const resolved = deps.getResolvedConfig();
|
||||
|
||||
const targetTrack = resolveTargetTrack(request, context);
|
||||
if (!targetTrack) {
|
||||
return { ok: false, message: 'Select the out-of-sync subtitle track to retime' };
|
||||
}
|
||||
const targetSlot = resolveTargetSlot(targetTrack, context);
|
||||
|
||||
if (request.engine === 'ffsubsync') {
|
||||
try {
|
||||
validateFfsubsyncReference(context.videoPath);
|
||||
@@ -354,22 +460,64 @@ export async function runSubsyncManual(
|
||||
message: `ffsubsync synchronization failed: ${(error as Error).message}`,
|
||||
};
|
||||
}
|
||||
return subsyncToReference('ffsubsync', context.videoPath, context, resolved, client);
|
||||
return subsyncToReference(
|
||||
'ffsubsync',
|
||||
context.videoPath,
|
||||
targetTrack,
|
||||
context,
|
||||
resolved,
|
||||
client,
|
||||
targetSlot,
|
||||
);
|
||||
}
|
||||
|
||||
const sourceTrack = getTrackById(context.sourceTracks, request.sourceTrackId ?? null);
|
||||
if (!sourceTrack) {
|
||||
return { ok: false, message: 'Select a subtitle source track for alass' };
|
||||
if (request.referenceMode === 'video') {
|
||||
if (isRemoteMediaPath(context.videoPath)) {
|
||||
return {
|
||||
ok: false,
|
||||
message:
|
||||
'alass cannot use a stream URL as reference. Pick a reference subtitle track instead.',
|
||||
};
|
||||
}
|
||||
return subsyncToReference(
|
||||
'alass',
|
||||
context.videoPath,
|
||||
targetTrack,
|
||||
context,
|
||||
resolved,
|
||||
client,
|
||||
targetSlot,
|
||||
);
|
||||
}
|
||||
|
||||
const referenceTrack = getTrackById(context.subtitleTracks, request.referenceTrackId ?? null);
|
||||
if (!referenceTrack) {
|
||||
return { ok: false, message: 'Select a reference subtitle track for alass' };
|
||||
}
|
||||
if (referenceTrack.id === targetTrack.id) {
|
||||
return { ok: false, message: 'Reference and out-of-sync subtitles must be different tracks' };
|
||||
}
|
||||
|
||||
const ffmpegPath = ensureExecutablePath(resolved.ffmpegPath, 'ffmpeg');
|
||||
let sourceExtraction: FileExtractionResult | null = null;
|
||||
let referenceExtraction: FileExtractionResult | null = null;
|
||||
try {
|
||||
sourceExtraction = await extractSubtitleTrackToFile(ffmpegPath, context.videoPath, sourceTrack);
|
||||
return await subsyncToReference('alass', sourceExtraction.path, context, resolved, client);
|
||||
referenceExtraction = await extractSubtitleTrackToFile(
|
||||
ffmpegPath,
|
||||
context.videoPath,
|
||||
referenceTrack,
|
||||
);
|
||||
return await subsyncToReference(
|
||||
'alass',
|
||||
referenceExtraction.path,
|
||||
targetTrack,
|
||||
context,
|
||||
resolved,
|
||||
client,
|
||||
targetSlot,
|
||||
);
|
||||
} finally {
|
||||
if (sourceExtraction) {
|
||||
cleanupTemporaryFile(sourceExtraction);
|
||||
if (referenceExtraction) {
|
||||
cleanupTemporaryFile(referenceExtraction);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -377,14 +525,27 @@ export async function runSubsyncManual(
|
||||
export async function openSubsyncManualPicker(deps: TriggerSubsyncFromConfigDeps): Promise<void> {
|
||||
const client = getMpvClientForSubsync(deps);
|
||||
const context = await gatherSubsyncContext(client);
|
||||
const payload: SubsyncManualPayload = {
|
||||
ffsubsyncAvailable: !isRemoteMediaPath(context.videoPath),
|
||||
sourceTracks: context.sourceTracks
|
||||
const subtitleTracks = context.subtitleTracks
|
||||
.filter((track) => typeof track.id === 'number')
|
||||
.map((track) => ({
|
||||
id: track.id as number,
|
||||
label: formatTrackLabel(track),
|
||||
})),
|
||||
}));
|
||||
const primaryTrackId =
|
||||
typeof context.primaryTrack.id === 'number' ? context.primaryTrack.id : null;
|
||||
const secondaryTrackId =
|
||||
typeof context.secondaryTrack?.id === 'number' ? context.secondaryTrack.id : null;
|
||||
const payload: SubsyncManualPayload = {
|
||||
subtitleTracks,
|
||||
// The secondary track can be filtered or deduped out of the emitted list,
|
||||
// so only default to it when the picker actually offers it.
|
||||
defaultReferenceTrackId:
|
||||
subtitleTracks.find((track) => track.id === secondaryTrackId)?.id ??
|
||||
subtitleTracks.find((track) => track.id !== primaryTrackId)?.id ??
|
||||
null,
|
||||
defaultTargetTrackId: primaryTrackId,
|
||||
videoReferenceAvailable: !isRemoteMediaPath(context.videoPath),
|
||||
ffsubsyncAvailable: !isRemoteMediaPath(context.videoPath),
|
||||
};
|
||||
deps.openManualPicker(payload);
|
||||
}
|
||||
@@ -397,7 +558,7 @@ export async function triggerSubsyncFromConfig(deps: TriggerSubsyncFromConfigDep
|
||||
|
||||
try {
|
||||
await openSubsyncManualPicker(deps);
|
||||
deps.showMpvOsd('Subsync: choose engine and source');
|
||||
deps.showMpvOsd('Subsync: choose engine and subtitles');
|
||||
} catch (error) {
|
||||
deps.showMpvOsd(`Subsync failed: ${(error as Error).message}`);
|
||||
} finally {
|
||||
|
||||
@@ -74,7 +74,6 @@ test('prefetch service tokenizes priority window cues and caches them', async ()
|
||||
preCacheTokenization: (text, data) => {
|
||||
cached.set(text, data);
|
||||
},
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -91,32 +90,38 @@ test('prefetch service tokenizes priority window cues and caches them', async ()
|
||||
assert.ok(cached.has('line-2'));
|
||||
});
|
||||
|
||||
test('prefetch service stops when cache is full', async () => {
|
||||
test('prefetch service warms every cue even when the cache evicts along the way', async () => {
|
||||
const cues = makeCues(20);
|
||||
let tokenizeCalls = 0;
|
||||
let cacheSize = 0;
|
||||
const tokenized: string[] = [];
|
||||
// Stand-in for the LRU: only the last 5 entries survive, so later cues evict earlier ones.
|
||||
const cache = new Set<string>();
|
||||
|
||||
const service = createSubtitlePrefetchService({
|
||||
cues,
|
||||
tokenizeSubtitle: async (text) => {
|
||||
tokenizeCalls += 1;
|
||||
tokenized.push(text);
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
preCacheTokenization: () => {
|
||||
cacheSize += 1;
|
||||
preCacheTokenization: (text) => {
|
||||
cache.add(text);
|
||||
while (cache.size > 5) {
|
||||
const oldest = cache.values().next().value;
|
||||
if (oldest === undefined) break;
|
||||
cache.delete(oldest);
|
||||
}
|
||||
},
|
||||
isCacheFull: () => cacheSize >= 5,
|
||||
hasCachedTokenization: (text) => cache.has(text),
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
service.start(0);
|
||||
for (let i = 0; i < 30; i += 1) {
|
||||
for (let i = 0; i < 60; i += 1) {
|
||||
await flushMicrotasks();
|
||||
}
|
||||
service.stop();
|
||||
|
||||
// Should have stopped at 5 (cache full), not tokenized all 20
|
||||
assert.ok(tokenizeCalls <= 6, `Expected <= 6 tokenize calls, got ${tokenizeCalls}`);
|
||||
assert.equal(tokenized.length, 20, `Expected all 20 cues warmed, got ${tokenized.length}`);
|
||||
assert.equal(new Set(tokenized).size, 20, 'Each cue is tokenized at most once per run');
|
||||
});
|
||||
|
||||
test('prefetch service can be stopped mid-flight', async () => {
|
||||
@@ -130,7 +135,6 @@ test('prefetch service can be stopped mid-flight', async () => {
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
preCacheTokenization: () => {},
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -159,7 +163,6 @@ test('prefetch service onSeek re-prioritizes from new position', async () => {
|
||||
preCacheTokenization: (text) => {
|
||||
cachedTexts.push(text);
|
||||
},
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -183,7 +186,7 @@ test('prefetch service onSeek re-prioritizes from new position', async () => {
|
||||
assert.ok(hasPostSeekCue, 'Should have cached cues after seek position');
|
||||
});
|
||||
|
||||
test('prefetch service still warms the priority window when cache is full', async () => {
|
||||
test('prefetch service warms the priority window ahead of the rest of the file', async () => {
|
||||
const cues = makeCues(20);
|
||||
const cachedTexts: string[] = [];
|
||||
|
||||
@@ -193,7 +196,6 @@ test('prefetch service still warms the priority window when cache is full', asyn
|
||||
preCacheTokenization: (text) => {
|
||||
cachedTexts.push(text);
|
||||
},
|
||||
isCacheFull: () => true,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -217,7 +219,6 @@ test('prefetch service pause/resume halts and continues tokenization', async ()
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
preCacheTokenization: () => {},
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -255,7 +256,6 @@ test('prefetch service skips cues already present in tokenization cache', async
|
||||
},
|
||||
preCacheTokenization: () => {},
|
||||
hasCachedTokenization: (text) => text === 'line-0' || text === 'line-1',
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
@@ -285,7 +285,6 @@ test('prefetch service deduplicates repeated cue text within a run', async () =>
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
preCacheTokenization: () => {},
|
||||
isCacheFull: () => false,
|
||||
priorityWindowSize: 3,
|
||||
});
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ export interface SubtitlePrefetchServiceDeps {
|
||||
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
||||
preCacheTokenization: (text: string, data: SubtitleData) => void;
|
||||
hasCachedTokenization?: (text: string) => boolean;
|
||||
isCacheFull: () => boolean;
|
||||
priorityWindowSize?: number;
|
||||
}
|
||||
|
||||
@@ -57,11 +56,14 @@ export function createSubtitlePrefetchService(
|
||||
let paused = false;
|
||||
let currentRunId = 0;
|
||||
|
||||
// A run is a single bounded pass over one file's cues, deduped by `warmedKeys` and by
|
||||
// `hasCachedTokenization`, so the worst case is one tokenization per cue. The cache is
|
||||
// an LRU and bounds its own memory, so a full cache is not a reason to stop warming;
|
||||
// stopping there used to leave the tail of longer media permanently uncached.
|
||||
async function tokenizeCueList(
|
||||
cuesToProcess: SubtitleCue[],
|
||||
runId: number,
|
||||
warmedKeys: Set<string>,
|
||||
options: { allowWhenCacheFull?: boolean } = {},
|
||||
): Promise<void> {
|
||||
for (const cue of cuesToProcess) {
|
||||
if (stopped || runId !== currentRunId) {
|
||||
@@ -77,10 +79,6 @@ export function createSubtitlePrefetchService(
|
||||
return;
|
||||
}
|
||||
|
||||
if (!options.allowWhenCacheFull && deps.isCacheFull()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const cacheKey = normalizeSubtitleCacheKey(cue.text);
|
||||
if (!cacheKey || warmedKeys.has(cacheKey) || deps.hasCachedTokenization?.(cue.text)) {
|
||||
if (cacheKey) {
|
||||
@@ -110,7 +108,7 @@ export function createSubtitlePrefetchService(
|
||||
|
||||
// Phase 1: Priority window
|
||||
const priorityCues = computePriorityWindow(cues, currentTimeSeconds, windowSize);
|
||||
await tokenizeCueList(priorityCues, runId, warmedKeys, { allowWhenCacheFull: true });
|
||||
await tokenizeCueList(priorityCues, runId, warmedKeys);
|
||||
|
||||
if (stopped || runId !== currentRunId) {
|
||||
return;
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import type { SubtitleData } from '../../types';
|
||||
import { createSubtitleProcessingController } from './subtitle-processing-controller';
|
||||
|
||||
function flushMicrotasks(): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
|
||||
test('new subtitle emits plain immediately without parallel tokenization or a stale overwrite', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const resolvers = new Map<string, (value: SubtitleData | null) => void>();
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) =>
|
||||
await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolvers.set(text, resolve);
|
||||
}),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('first');
|
||||
controller.onSubtitleChange('second');
|
||||
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: 'second', tokens: null },
|
||||
]);
|
||||
assert.equal(resolvers.has('second'), false);
|
||||
|
||||
const resolveFirst = resolvers.get('first');
|
||||
assert.ok(resolveFirst);
|
||||
resolveFirst({ text: 'first', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: 'second', tokens: null },
|
||||
]);
|
||||
assert.equal(resolvers.has('second'), true);
|
||||
|
||||
const resolveSecond = resolvers.get('second');
|
||||
assert.ok(resolveSecond);
|
||||
resolveSecond({ text: 'second', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: 'second', tokens: null },
|
||||
{ text: 'second', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle clears immediately while previous tokenization remains pending', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveTokenization: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async () =>
|
||||
await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveTokenization = resolve;
|
||||
}),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('first');
|
||||
controller.onSubtitleChange('');
|
||||
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: '', tokens: null },
|
||||
]);
|
||||
|
||||
assert.ok(resolveTokenization);
|
||||
resolveTokenization({ text: 'first', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: '', tokens: null },
|
||||
]);
|
||||
});
|
||||
|
||||
test('returning to an uncached completed line emits it while another line is pending', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolvePending: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
if (text === 'A') {
|
||||
return { text, tokens: [] };
|
||||
}
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolvePending = resolve;
|
||||
});
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('A');
|
||||
await flushMicrotasks();
|
||||
controller.invalidateTokenizationCache();
|
||||
controller.onSubtitleChange('B');
|
||||
controller.onSubtitleChange('A');
|
||||
|
||||
assert.deepEqual(emitted.at(-1), { text: 'A', tokens: null });
|
||||
assert.ok(resolvePending);
|
||||
resolvePending({ text: 'B', tokens: [] });
|
||||
});
|
||||
|
||||
test('ABA subtitle changes reuse the matching first tokenization only after A is current again', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const tokenizeCalls: string[] = [];
|
||||
const resolvers: Array<(value: SubtitleData | null) => void> = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
tokenizeCalls.push(text);
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolvers.push(resolve);
|
||||
});
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('A');
|
||||
controller.onSubtitleChange('B');
|
||||
controller.onSubtitleChange('A');
|
||||
const resolveFirst = resolvers[0];
|
||||
assert.ok(resolveFirst);
|
||||
resolveFirst({ text: 'A', tokens: [{ value: 1 } as never] });
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(tokenizeCalls, ['A']);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'A', tokens: null },
|
||||
{ text: 'B', tokens: null },
|
||||
{ text: 'A', tokens: null },
|
||||
{ text: 'A', tokens: [{ value: 1 } as never] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('cached next subtitle does not downgrade to plain while processing is busy', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveTokenization: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) =>
|
||||
await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveTokenization = () => resolve({ text, tokens: [] });
|
||||
}),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.preCacheTokenization('cached', { text: 'cached', tokens: [] });
|
||||
controller.onSubtitleChange('pending');
|
||||
controller.onSubtitleChange('cached');
|
||||
|
||||
assert.deepEqual(emitted, [{ text: 'pending', tokens: null }]);
|
||||
assert.ok(resolveTokenization);
|
||||
resolveTokenization({ text: 'pending', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'pending', tokens: null },
|
||||
{ text: 'cached', tokens: [] },
|
||||
]);
|
||||
});
|
||||
@@ -7,18 +7,138 @@ function flushMicrotasks(): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
|
||||
test('subtitle processing emits tokenized payload when tokenization succeeds', async () => {
|
||||
test('subtitle processing emits plain payload immediately on cache miss, then tokenized payload', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('字幕');
|
||||
assert.deepEqual(emitted, [{ text: '字幕', tokens: null }]);
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: '字幕', tokens: null },
|
||||
{ text: '字幕', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('cache invalidation during pending tokenization does not re-emit the plain payload', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const resolvers: Array<(value: SubtitleData | null) => void> = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) =>
|
||||
await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolvers.push(() => resolve({ text, tokens: [{ value: resolvers.length } as never] }));
|
||||
}),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('行');
|
||||
assert.deepEqual(emitted, [{ text: '行', tokens: null }]);
|
||||
|
||||
controller.invalidateTokenizationCache();
|
||||
resolvers[0]?.({ text: '行', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
// Retry for the new generation is now pending; still no duplicate plain emit.
|
||||
assert.deepEqual(emitted, [{ text: '行', tokens: null }]);
|
||||
|
||||
resolvers[1]?.({ text: '行', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: '行', tokens: null },
|
||||
{ text: '行', tokens: [{ value: 2 } as never] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('failed refresh does not downgrade an already emitted tokenized subtitle', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let tokenizeCalls = 0;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
tokenizeCalls += 1;
|
||||
if (tokenizeCalls > 1) {
|
||||
throw new Error('tokenizer gone');
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('行');
|
||||
await flushMicrotasks();
|
||||
controller.invalidateTokenizationCache();
|
||||
controller.refreshCurrentSubtitle();
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(tokenizeCalls, 2);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: '行', tokens: null },
|
||||
{ text: '行', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('null-tokenization refresh does not downgrade an already emitted tokenized subtitle', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let tokenizeCalls = 0;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
tokenizeCalls += 1;
|
||||
return tokenizeCalls > 1 ? null : { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('行');
|
||||
await flushMicrotasks();
|
||||
controller.invalidateTokenizationCache();
|
||||
controller.refreshCurrentSubtitle();
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(tokenizeCalls, 2);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: '行', tokens: null },
|
||||
{ text: '行', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing does not emit plain payload for cached lines', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.preCacheTokenization('字幕', { text: '字幕', tokens: [] });
|
||||
controller.onSubtitleChange('字幕');
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [{ text: '字幕', tokens: [] }]);
|
||||
});
|
||||
|
||||
test('subtitle processing shows plain line while tokenization is still pending', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveTokenization: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) =>
|
||||
await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveTokenization = () => resolve({ text, tokens: [] });
|
||||
}),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('遅い行');
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [{ text: '遅い行', tokens: null }]);
|
||||
|
||||
assert.ok(resolveTokenization);
|
||||
resolveTokenization({ text: '遅い行', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: '遅い行', tokens: null },
|
||||
{ text: '遅い行', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing drops stale tokenization and delivers latest subtitle only once', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let firstResolve: ((value: SubtitleData | null) => void) | undefined;
|
||||
@@ -41,7 +161,11 @@ test('subtitle processing drops stale tokenization and delivers latest subtitle
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(emitted, [{ text: 'second', tokens: [] }]);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: 'second', tokens: null },
|
||||
{ text: 'second', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing skips duplicate subtitle emission', async () => {
|
||||
@@ -60,7 +184,10 @@ test('subtitle processing skips duplicate subtitle emission', async () => {
|
||||
controller.onSubtitleChange('same');
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(emitted.length, 1);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'same', tokens: null },
|
||||
{ text: 'same', tokens: [] },
|
||||
]);
|
||||
assert.equal(tokenizeCalls, 1);
|
||||
});
|
||||
|
||||
@@ -84,7 +211,9 @@ test('subtitle processing reuses cached tokenization for repeated subtitle text'
|
||||
|
||||
assert.equal(tokenizeCalls, 2);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'first', tokens: null },
|
||||
{ text: 'first', tokens: [] },
|
||||
{ text: 'second', tokens: null },
|
||||
{ text: 'second', tokens: [] },
|
||||
{ text: 'first', tokens: [] },
|
||||
]);
|
||||
@@ -100,7 +229,48 @@ test('subtitle processing falls back to plain subtitle when tokenization returns
|
||||
controller.onSubtitleChange('fallback');
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(
|
||||
emitted,
|
||||
[{ text: 'fallback', tokens: null }],
|
||||
'plain payload should not be re-emitted when tokenization yields nothing new',
|
||||
);
|
||||
});
|
||||
|
||||
test('null tokenization is not cached and a later cue retries tokenization', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const callsByText = new Map<string, number>();
|
||||
let failNext = true;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
callsByText.set(text, (callsByText.get(text) ?? 0) + 1);
|
||||
if (text === 'fallback' && failNext) {
|
||||
failNext = false;
|
||||
return null;
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('fallback');
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(callsByText.get('fallback'), 1);
|
||||
assert.equal(
|
||||
controller.hasCachedSubtitle('fallback'),
|
||||
false,
|
||||
'plain fallback must not be cached when tokenization yields nothing',
|
||||
);
|
||||
assert.deepEqual(emitted, [{ text: 'fallback', tokens: null }]);
|
||||
|
||||
controller.onSubtitleChange('other');
|
||||
await flushMicrotasks();
|
||||
controller.onSubtitleChange('fallback');
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(callsByText.get('fallback'), 2, 'later cue should retry tokenization');
|
||||
assert.equal(controller.hasCachedSubtitle('fallback'), true);
|
||||
assert.deepEqual(emitted.at(-1), { text: 'fallback', tokens: [] });
|
||||
});
|
||||
|
||||
test('subtitle processing ignores duplicate current subtitle refresh without cache invalidation', async () => {
|
||||
@@ -120,7 +290,10 @@ test('subtitle processing ignores duplicate current subtitle refresh without cac
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(tokenizeCalls, 1);
|
||||
assert.deepEqual(emitted, [{ text: 'same', tokens: [] }]);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'same', tokens: null },
|
||||
{ text: 'same', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing coalesces refresh requests while current subtitle is processing', async () => {
|
||||
@@ -146,7 +319,10 @@ test('subtitle processing coalesces refresh requests while current subtitle is p
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(tokenizeCalls, 1);
|
||||
assert.deepEqual(emitted, [{ text: 'same', tokens: [] }]);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'same', tokens: null },
|
||||
{ text: 'same', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing refresh re-tokenizes after cache invalidation', async () => {
|
||||
@@ -168,6 +344,7 @@ test('subtitle processing refresh re-tokenizes after cache invalidation', async
|
||||
|
||||
assert.equal(tokenizeCalls, 2);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'same', tokens: null },
|
||||
{ text: 'same', tokens: [{ value: 1 } as never] },
|
||||
{ text: 'same', tokens: [{ value: 2 } as never] },
|
||||
]);
|
||||
@@ -183,7 +360,10 @@ test('subtitle processing refresh can use explicit text override', async () => {
|
||||
controller.refreshCurrentSubtitle('initial');
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(emitted, [{ text: 'initial', tokens: [] }]);
|
||||
assert.deepEqual(emitted, [
|
||||
{ text: 'initial', tokens: null },
|
||||
{ text: 'initial', tokens: [] },
|
||||
]);
|
||||
});
|
||||
|
||||
test('subtitle processing cache invalidation only affects future subtitle events', async () => {
|
||||
@@ -205,10 +385,10 @@ test('subtitle processing cache invalidation only affects future subtitle events
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.equal(callsByText.get('same'), 1);
|
||||
assert.equal(emitted.length, 3);
|
||||
assert.equal(emitted.length, 5);
|
||||
|
||||
controller.invalidateTokenizationCache();
|
||||
assert.equal(emitted.length, 3);
|
||||
assert.equal(emitted.length, 5);
|
||||
|
||||
controller.onSubtitleChange('different');
|
||||
await flushMicrotasks();
|
||||
@@ -308,25 +488,161 @@ test('hasCachedSubtitle checks prefetched entries without consuming them', async
|
||||
assert.equal(controller.hasCachedSubtitle('猫\nです'), false);
|
||||
});
|
||||
|
||||
test('isCacheFull returns false when cache is below limit', () => {
|
||||
test('cache keeps every entry while below the limit', () => {
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: null }),
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: () => {},
|
||||
cacheLimit: 8,
|
||||
});
|
||||
|
||||
assert.equal(controller.isCacheFull(), false);
|
||||
for (let i = 0; i < 8; i += 1) {
|
||||
controller.preCacheTokenization(`line-${i}`, { text: `line-${i}`, tokens: [] });
|
||||
}
|
||||
|
||||
assert.deepEqual(
|
||||
Array.from({ length: 8 }, (_, i) => controller.hasCachedSubtitle(`line-${i}`)),
|
||||
Array.from({ length: 8 }, () => true),
|
||||
);
|
||||
});
|
||||
|
||||
test('isCacheFull returns true when cache reaches limit', async () => {
|
||||
test('cache evicts least recently used entries once the limit is reached', () => {
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: () => {},
|
||||
cacheLimit: 3,
|
||||
});
|
||||
|
||||
for (const line of ['a', 'b', 'c']) {
|
||||
controller.preCacheTokenization(line, { text: line, tokens: [] });
|
||||
}
|
||||
// Touching 'a' makes 'b' the eviction candidate.
|
||||
controller.consumeCachedSubtitle('a');
|
||||
controller.preCacheTokenization('d', { text: 'd', tokens: [] });
|
||||
|
||||
assert.equal(controller.hasCachedSubtitle('b'), false);
|
||||
assert.deepEqual(
|
||||
['a', 'c', 'd'].map((line) => controller.hasCachedSubtitle(line)),
|
||||
[true, true, true],
|
||||
);
|
||||
});
|
||||
|
||||
test('default cache limit covers a full-length title without evicting', () => {
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: () => {},
|
||||
});
|
||||
|
||||
// Fill cache to the 256 limit
|
||||
for (let i = 0; i < 256; i += 1) {
|
||||
for (let i = 0; i < 2000; i += 1) {
|
||||
controller.preCacheTokenization(`line-${i}`, { text: `line-${i}`, tokens: [] });
|
||||
}
|
||||
|
||||
assert.equal(controller.isCacheFull(), true);
|
||||
assert.equal(controller.hasCachedSubtitle('line-0'), true);
|
||||
assert.equal(controller.hasCachedSubtitle('line-1999'), true);
|
||||
});
|
||||
|
||||
test('onSubtitleChange reports whether processing was scheduled', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
// New text schedules work, so an emit (and anything gated on it) will follow.
|
||||
assert.equal(controller.onSubtitleChange('字幕'), true);
|
||||
await flushMicrotasks();
|
||||
|
||||
// A repeat emits nothing, so callers must not wait on an emit that is never
|
||||
// coming (subtitle prefetching would stay paused for the rest of the cue).
|
||||
const emittedCount = emitted.length;
|
||||
assert.equal(controller.onSubtitleChange('字幕'), false);
|
||||
await flushMicrotasks();
|
||||
assert.equal(emitted.length, emittedCount);
|
||||
});
|
||||
|
||||
test('refreshCurrentSubtitle reports the empty-text emit that an in-flight run will deliver', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
let resolveFirst: ((value: SubtitleData | null) => void) | undefined;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
if (text === '字幕') {
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveFirst = resolve;
|
||||
});
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('字幕');
|
||||
await flushMicrotasks();
|
||||
|
||||
// Clearing the subtitle while tokenization is in flight: the running loop
|
||||
// picks the empty text up and emits it, so callers gated on that emit (the
|
||||
// prefetch pause) must be told one is coming.
|
||||
assert.equal(controller.refreshCurrentSubtitle(''), true);
|
||||
|
||||
resolveFirst?.({ text: '字幕', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
// '字幕' is the provisional plain emit the in-flight run already made before
|
||||
// the refresh; '' is the emit the refresh promised.
|
||||
assert.deepEqual(
|
||||
emitted.map((payload) => payload.text),
|
||||
['字幕', ''],
|
||||
);
|
||||
});
|
||||
|
||||
test('onProcessingSettled fires once after the queue drains, including runs that emit nothing', async () => {
|
||||
const events: string[] = [];
|
||||
let resolveFirst: ((value: SubtitleData | null) => void) | undefined;
|
||||
let tokenizationFails = false;
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => {
|
||||
if (tokenizationFails) {
|
||||
return null;
|
||||
}
|
||||
if (text === '一行目') {
|
||||
return await new Promise<SubtitleData | null>((resolve) => {
|
||||
resolveFirst = resolve;
|
||||
});
|
||||
}
|
||||
return { text, tokens: [] };
|
||||
},
|
||||
emitSubtitle: (payload) => events.push(`emit:${payload.text}`),
|
||||
onProcessingSettled: () => events.push('settled'),
|
||||
});
|
||||
|
||||
controller.onSubtitleChange('一行目');
|
||||
await flushMicrotasks();
|
||||
// A second line arrives before the first finishes: the controller still has
|
||||
// work, so it must not report itself settled between the two.
|
||||
controller.onSubtitleChange('二行目');
|
||||
resolveFirst?.({ text: '一行目', tokens: [] });
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
|
||||
assert.deepEqual(events, ['emit:一行目', 'emit:二行目', 'emit:二行目', 'settled']);
|
||||
|
||||
// Tokenization failure on a line already shown plain: nothing is emitted, and
|
||||
// the settle signal is the only way a caller learns the work is over.
|
||||
events.length = 0;
|
||||
tokenizationFails = true;
|
||||
controller.invalidateTokenizationCache();
|
||||
assert.equal(controller.refreshCurrentSubtitle('二行目'), true);
|
||||
await flushMicrotasks();
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(events, ['settled']);
|
||||
});
|
||||
|
||||
test('refreshCurrentSubtitle reports no emit for empty text when nothing is running', async () => {
|
||||
const emitted: SubtitleData[] = [];
|
||||
const controller = createSubtitleProcessingController({
|
||||
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
|
||||
emitSubtitle: (payload) => emitted.push(payload),
|
||||
});
|
||||
|
||||
assert.equal(controller.refreshCurrentSubtitle(''), false);
|
||||
await flushMicrotasks();
|
||||
assert.deepEqual(emitted, []);
|
||||
});
|
||||
|
||||
@@ -3,18 +3,41 @@ import type { SubtitleData } from '../../types';
|
||||
export interface SubtitleProcessingControllerDeps {
|
||||
tokenizeSubtitle: (text: string) => Promise<SubtitleData | null>;
|
||||
emitSubtitle: (payload: SubtitleData) => void;
|
||||
/**
|
||||
* Fires when the controller runs out of work: every scheduled line has been
|
||||
* processed, whether it ended in an emit, a suppressed duplicate, or a
|
||||
* tokenizer failure. Callers that hold a resource for the duration of
|
||||
* processing (prefetch pausing) release it here rather than on an emit,
|
||||
* which is not guaranteed to happen.
|
||||
*/
|
||||
onProcessingSettled?: () => void;
|
||||
logDebug?: (message: string) => void;
|
||||
now?: () => number;
|
||||
cacheLimit?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure memory bound on the LRU, not a coverage limit: prefetching runs to the end of a
|
||||
* file regardless of cache pressure. Sized to hold a feature-length title (a 24-minute
|
||||
* episode runs 300-400 lines, a 2-hour film ~2000) plus room for lines that repeat across
|
||||
* episodes of a series, so openings and endings stay warm between titles.
|
||||
*/
|
||||
export const DEFAULT_SUBTITLE_TOKENIZATION_CACHE_LIMIT = 2500;
|
||||
|
||||
export interface SubtitleProcessingController {
|
||||
onSubtitleChange: (text: string) => void;
|
||||
refreshCurrentSubtitle: (textOverride?: string) => void;
|
||||
/**
|
||||
* Returns whether processing is now scheduled or already in flight for this
|
||||
* event. A false return means the controller is idle and will do nothing, so
|
||||
* onProcessingSettled will not fire; callers that pause work for the duration
|
||||
* of processing (such as subtitle prefetching) must release it themselves.
|
||||
*/
|
||||
onSubtitleChange: (text: string) => boolean;
|
||||
/** Same contract as onSubtitleChange: whether processing is pending. */
|
||||
refreshCurrentSubtitle: (textOverride?: string) => boolean;
|
||||
invalidateTokenizationCache: () => void;
|
||||
preCacheTokenization: (text: string, data: SubtitleData) => void;
|
||||
consumeCachedSubtitle: (text: string) => SubtitleData | null;
|
||||
hasCachedSubtitle: (text: string) => boolean;
|
||||
isCacheFull: () => boolean;
|
||||
}
|
||||
|
||||
export function normalizeSubtitleCacheKey(text: string): string {
|
||||
@@ -24,9 +47,15 @@ export function normalizeSubtitleCacheKey(text: string): string {
|
||||
export function createSubtitleProcessingController(
|
||||
deps: SubtitleProcessingControllerDeps,
|
||||
): SubtitleProcessingController {
|
||||
const SUBTITLE_TOKENIZATION_CACHE_LIMIT = 256;
|
||||
const SUBTITLE_TOKENIZATION_CACHE_LIMIT =
|
||||
deps.cacheLimit && deps.cacheLimit > 0
|
||||
? deps.cacheLimit
|
||||
: DEFAULT_SUBTITLE_TOKENIZATION_CACHE_LIMIT;
|
||||
let latestText = '';
|
||||
let lastEmittedText = '';
|
||||
// Tracks the latest provisional plain emit across rapid changes and loop retries
|
||||
// so the same line is never shown plain twice.
|
||||
let lastPlainEmittedText: string | null = null;
|
||||
let cacheGeneration = 0;
|
||||
let lastEmittedGeneration = 0;
|
||||
let processing = false;
|
||||
@@ -70,9 +99,12 @@ export function createSubtitleProcessingController(
|
||||
const startedAtMs = now();
|
||||
|
||||
if (!text.trim()) {
|
||||
if (lastPlainEmittedText !== text) {
|
||||
deps.emitSubtitle({ text, tokens: null });
|
||||
}
|
||||
lastEmittedText = text;
|
||||
lastEmittedGeneration = generation;
|
||||
lastPlainEmittedText = null;
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -82,11 +114,25 @@ export function createSubtitleProcessingController(
|
||||
if (cachedTokenized) {
|
||||
output = cachedTokenized;
|
||||
} else {
|
||||
// Cache miss: show the plain line on time; the tokenized payload
|
||||
// upgrades it once ready. Skipped on refreshes of an already
|
||||
// emitted line so downstream consumers never see a downgrade.
|
||||
if (text !== lastEmittedText && text !== lastPlainEmittedText) {
|
||||
deps.emitSubtitle({ text, tokens: null });
|
||||
lastPlainEmittedText = text;
|
||||
}
|
||||
const tokenized = await deps.tokenizeSubtitle(text);
|
||||
// A null result is a transient tokenizer failure, not a verdict on
|
||||
// the line: caching the plain fallback would pin it untokenized for
|
||||
// every later occurrence.
|
||||
if (tokenized) {
|
||||
output = tokenized;
|
||||
// A result computed before an invalidation must not repopulate the
|
||||
// fresh cache, or the retry below would serve the stale entry.
|
||||
if (generation === cacheGeneration) {
|
||||
setCachedTokenization(text, tokenized);
|
||||
}
|
||||
}
|
||||
setCachedTokenization(text, output);
|
||||
}
|
||||
} catch (error) {
|
||||
deps.logDebug?.(`Subtitle tokenization failed: ${(error as Error).message}`);
|
||||
@@ -107,9 +153,16 @@ export function createSubtitleProcessingController(
|
||||
continue;
|
||||
}
|
||||
|
||||
// An untokenized result adds nothing when this line was already shown,
|
||||
// either provisionally or as an earlier full emit (failed refresh) —
|
||||
// emitting it would duplicate or downgrade what is on screen.
|
||||
const plainAlreadyShown = lastPlainEmittedText === text || lastEmittedText === text;
|
||||
if (!(output.tokens === null && output.text === text && plainAlreadyShown)) {
|
||||
deps.emitSubtitle(output);
|
||||
}
|
||||
lastEmittedText = text;
|
||||
lastEmittedGeneration = generation;
|
||||
lastPlainEmittedText = null;
|
||||
deps.logDebug?.(
|
||||
`Subtitle tokenization delivered; elapsed=${now() - startedAtMs}ms, staleDrops=${staleDropCount}`,
|
||||
);
|
||||
@@ -126,32 +179,50 @@ export function createSubtitleProcessingController(
|
||||
(latestText.trim() && cacheGeneration !== lastEmittedGeneration)
|
||||
) {
|
||||
processLatest();
|
||||
return;
|
||||
}
|
||||
// Nothing left to do: signal completion even when this run emitted
|
||||
// nothing (suppressed duplicate, tokenizer failure), or callers waiting
|
||||
// on the controller would wait forever.
|
||||
deps.onProcessingSettled?.();
|
||||
});
|
||||
};
|
||||
|
||||
return {
|
||||
onSubtitleChange: (text: string) => {
|
||||
if (text === latestText) {
|
||||
return;
|
||||
// A run already in flight for this text will still emit for it.
|
||||
return processing;
|
||||
}
|
||||
latestText = text;
|
||||
if (
|
||||
processing &&
|
||||
text !== lastPlainEmittedText &&
|
||||
!tokenizationCache.has(normalizeSubtitleCacheKey(text))
|
||||
) {
|
||||
deps.emitSubtitle({ text, tokens: null });
|
||||
lastPlainEmittedText = text;
|
||||
}
|
||||
processLatest();
|
||||
return true;
|
||||
},
|
||||
refreshCurrentSubtitle: (textOverride?: string) => {
|
||||
if (typeof textOverride === 'string') {
|
||||
latestText = textOverride;
|
||||
}
|
||||
if (!latestText.trim()) {
|
||||
return;
|
||||
// A run in flight will pick this up and emit the empty subtitle, so
|
||||
// the caller is still waiting on an emit.
|
||||
return processing;
|
||||
}
|
||||
if (
|
||||
processing ||
|
||||
(latestText === lastEmittedText && cacheGeneration === lastEmittedGeneration)
|
||||
) {
|
||||
return;
|
||||
if (processing) {
|
||||
return true;
|
||||
}
|
||||
if (latestText === lastEmittedText && cacheGeneration === lastEmittedGeneration) {
|
||||
return false;
|
||||
}
|
||||
processLatest();
|
||||
return true;
|
||||
},
|
||||
invalidateTokenizationCache: () => {
|
||||
tokenizationCache.clear();
|
||||
@@ -169,13 +240,11 @@ export function createSubtitleProcessingController(
|
||||
latestText = text;
|
||||
lastEmittedText = text;
|
||||
lastEmittedGeneration = cacheGeneration;
|
||||
lastPlainEmittedText = null;
|
||||
return cached;
|
||||
},
|
||||
hasCachedSubtitle: (text: string) => {
|
||||
return tokenizationCache.has(normalizeSubtitleCacheKey(text));
|
||||
},
|
||||
isCacheFull: () => {
|
||||
return tokenizationCache.size >= SUBTITLE_TOKENIZATION_CACHE_LIMIT;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {
|
||||
isSubtitleAnnotationUpgrade,
|
||||
serializeInitialSubtitleWebsocketMessage,
|
||||
serializeSubtitleMarkup,
|
||||
serializeSubtitleWebsocketMessage,
|
||||
@@ -13,6 +14,40 @@ const frequencyOptions = {
|
||||
mode: 'banded' as const,
|
||||
};
|
||||
|
||||
test('annotation upgrade requires matching text and cue timing', () => {
|
||||
const current: SubtitleData = {
|
||||
text: '字幕',
|
||||
tokens: null,
|
||||
startTime: 10,
|
||||
endTime: 12,
|
||||
};
|
||||
|
||||
assert.equal(
|
||||
isSubtitleAnnotationUpgrade(current, {
|
||||
...current,
|
||||
tokens: [],
|
||||
}),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
isSubtitleAnnotationUpgrade(current, {
|
||||
...current,
|
||||
tokens: [],
|
||||
startTime: 11,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
isSubtitleAnnotationUpgrade(current, {
|
||||
...current,
|
||||
text: '次の字幕',
|
||||
tokens: [],
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(isSubtitleAnnotationUpgrade(current, current), false);
|
||||
});
|
||||
|
||||
test('serializeSubtitleMarkup escapes plain text and preserves line breaks', () => {
|
||||
const payload: SubtitleData = {
|
||||
text: 'a < b\nx & y',
|
||||
|
||||
@@ -20,6 +20,20 @@ export type SubtitleWebsocketFrequencyOptions = {
|
||||
|
||||
export type SubtitleWebsocketPayloadMode = 'plain' | 'annotated';
|
||||
|
||||
export function isSubtitleAnnotationUpgrade(
|
||||
current: SubtitleData | null,
|
||||
next: SubtitleData,
|
||||
): boolean {
|
||||
return (
|
||||
current !== null &&
|
||||
current.tokens === null &&
|
||||
next.tokens !== null &&
|
||||
current.text === next.text &&
|
||||
current.startTime === next.startTime &&
|
||||
current.endTime === next.endTime
|
||||
);
|
||||
}
|
||||
|
||||
type SubtitleWebsocketMessageOptions = {
|
||||
payloadMode?: SubtitleWebsocketPayloadMode;
|
||||
};
|
||||
|
||||
@@ -2934,44 +2934,12 @@ test('tokenizeSubtitle preserves Yomitan compound token when MeCab components ar
|
||||
return [];
|
||||
}
|
||||
|
||||
if (script.includes('parseText')) {
|
||||
return [
|
||||
{
|
||||
source: 'scanning-parser',
|
||||
index: 0,
|
||||
content: [
|
||||
[
|
||||
{
|
||||
text: '取り組んで',
|
||||
surface: '取り組んで',
|
||||
reading: 'とりくんで',
|
||||
headwords: [[{ term: '取り組む' }]],
|
||||
},
|
||||
],
|
||||
[
|
||||
{
|
||||
text: 'もらいます',
|
||||
reading: 'もらいます',
|
||||
headwords: [[{ term: 'もらう' }]],
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
surface: '取り',
|
||||
reading: 'とり',
|
||||
headword: '取る',
|
||||
headword: '取り組む',
|
||||
startPos: 0,
|
||||
endPos: 2,
|
||||
},
|
||||
{
|
||||
surface: '組んで',
|
||||
reading: 'くんで',
|
||||
headword: '組む',
|
||||
startPos: 2,
|
||||
endPos: 5,
|
||||
},
|
||||
{
|
||||
|
||||
@@ -70,6 +70,7 @@ export interface TokenizerServiceDeps {
|
||||
getNameMatchImagesEnabled?: () => boolean;
|
||||
getCharacterNameImage?: (term: string) => CharacterNameImage | null;
|
||||
getCurrentCharacterDictionaryMediaId?: () => number | null;
|
||||
getCharacterNameCandidates?: () => { key: string; forms: string[] } | null;
|
||||
getFrequencyDictionaryEnabled?: () => boolean;
|
||||
getFrequencyDictionaryMatchMode?: () => FrequencyDictionaryMatchMode;
|
||||
getFrequencyRank?: FrequencyDictionaryLookup;
|
||||
@@ -106,6 +107,7 @@ export interface TokenizerDepsRuntimeOptions {
|
||||
getNameMatchImagesEnabled?: () => boolean;
|
||||
getCharacterNameImage?: (term: string) => CharacterNameImage | null;
|
||||
getCurrentCharacterDictionaryMediaId?: () => number | null;
|
||||
getCharacterNameCandidates?: () => { key: string; forms: string[] } | null;
|
||||
getFrequencyDictionaryEnabled?: () => boolean;
|
||||
getFrequencyDictionaryMatchMode?: () => FrequencyDictionaryMatchMode;
|
||||
getFrequencyRank?: FrequencyDictionaryLookup;
|
||||
@@ -266,6 +268,7 @@ export function createTokenizerDepsRuntime(
|
||||
getNameMatchImagesEnabled: options.getNameMatchImagesEnabled,
|
||||
getCharacterNameImage: options.getCharacterNameImage,
|
||||
getCurrentCharacterDictionaryMediaId: options.getCurrentCharacterDictionaryMediaId,
|
||||
getCharacterNameCandidates: options.getCharacterNameCandidates,
|
||||
getFrequencyDictionaryEnabled: options.getFrequencyDictionaryEnabled,
|
||||
getFrequencyDictionaryMatchMode: options.getFrequencyDictionaryMatchMode ?? (() => 'headword'),
|
||||
getFrequencyRank: options.getFrequencyRank,
|
||||
@@ -716,15 +719,30 @@ function getAnnotationOptions(deps: TokenizerServiceDeps): TokenizerAnnotationOp
|
||||
};
|
||||
}
|
||||
|
||||
// Per-line stage durations for the pipeline debug log; every field is filled in
|
||||
// by the stage that awaits the corresponding work.
|
||||
interface TokenizationStageTimings {
|
||||
scanMs?: number;
|
||||
mecabMs?: number;
|
||||
frequencyMs?: number;
|
||||
annotateMs?: number;
|
||||
}
|
||||
|
||||
async function parseWithYomitanInternalParser(
|
||||
text: string,
|
||||
deps: TokenizerServiceDeps,
|
||||
options: TokenizerAnnotationOptions,
|
||||
stageTimings?: TokenizationStageTimings,
|
||||
): Promise<MergedToken[] | null> {
|
||||
const scanStartedAtMs = Date.now();
|
||||
const selectedTokens = await requestYomitanScanTokens(text, deps, logger, {
|
||||
includeNameMatchMetadata: options.nameMatchEnabled,
|
||||
currentCharacterDictionaryMediaId: deps.getCurrentCharacterDictionaryMediaId?.() ?? null,
|
||||
nameCandidates: deps.getCharacterNameCandidates?.() ?? null,
|
||||
});
|
||||
if (stageTimings) {
|
||||
stageTimings.scanMs = Date.now() - scanStartedAtMs;
|
||||
}
|
||||
if (!selectedTokens || selectedTokens.length === 0) {
|
||||
return null;
|
||||
}
|
||||
@@ -757,6 +775,7 @@ async function parseWithYomitanInternalParser(
|
||||
|
||||
const frequencyRankPromise: Promise<YomitanFrequencyIndex> = options.frequencyEnabled
|
||||
? (async () => {
|
||||
const frequencyStartedAtMs = Date.now();
|
||||
const frequencyMatchMode = options.frequencyMatchMode;
|
||||
const termReadingList = buildYomitanFrequencyTermReadingList(
|
||||
normalizedSelectedTokens,
|
||||
@@ -767,12 +786,17 @@ async function parseWithYomitanInternalParser(
|
||||
deps,
|
||||
logger,
|
||||
);
|
||||
return buildYomitanFrequencyIndex(yomitanFrequencies);
|
||||
const frequencyIndex = buildYomitanFrequencyIndex(yomitanFrequencies);
|
||||
if (stageTimings) {
|
||||
stageTimings.frequencyMs = Date.now() - frequencyStartedAtMs;
|
||||
}
|
||||
return frequencyIndex;
|
||||
})()
|
||||
: Promise.resolve({ byPair: new Map(), byTerm: new Map() });
|
||||
|
||||
const mecabEnrichmentPromise: Promise<MergedToken[]> = needsMecabPosEnrichment(options)
|
||||
? (async () => {
|
||||
const mecabStartedAtMs = Date.now();
|
||||
try {
|
||||
const mecabTokens = await deps.tokenizeWithMecab(text);
|
||||
const enrichTokensWithMecab = deps.enrichTokensWithMecab ?? enrichTokensWithMecabAsync;
|
||||
@@ -786,6 +810,10 @@ async function parseWithYomitanInternalParser(
|
||||
`textLength=${text.length}`,
|
||||
);
|
||||
return normalizedSelectedTokens;
|
||||
} finally {
|
||||
if (stageTimings) {
|
||||
stageTimings.mecabMs = Date.now() - mecabStartedAtMs;
|
||||
}
|
||||
}
|
||||
})()
|
||||
: Promise.resolve(normalizedSelectedTokens);
|
||||
@@ -876,15 +904,35 @@ export async function tokenizeSubtitle(
|
||||
const annotationOptions = getAnnotationOptions(deps);
|
||||
annotationOptions.sourceText = tokenizeText;
|
||||
|
||||
const yomitanTokens = await parseWithYomitanInternalParser(tokenizeText, deps, annotationOptions);
|
||||
const stageTimings: TokenizationStageTimings = {};
|
||||
const startedAtMs = Date.now();
|
||||
const logStageTimings = (tokenCount: number): void => {
|
||||
logger.debug(
|
||||
`Subtitle tokenization stages; textLength=${tokenizeText.length}, tokenCount=${tokenCount}, ` +
|
||||
`scanMs=${stageTimings.scanMs ?? '-'}, mecabMs=${stageTimings.mecabMs ?? '-'}, ` +
|
||||
`frequencyMs=${stageTimings.frequencyMs ?? '-'}, annotateMs=${stageTimings.annotateMs ?? '-'}, ` +
|
||||
`totalMs=${Date.now() - startedAtMs}`,
|
||||
);
|
||||
};
|
||||
|
||||
const yomitanTokens = await parseWithYomitanInternalParser(
|
||||
tokenizeText,
|
||||
deps,
|
||||
annotationOptions,
|
||||
stageTimings,
|
||||
);
|
||||
if (yomitanTokens && yomitanTokens.length > 0) {
|
||||
const annotateStartedAtMs = Date.now();
|
||||
const annotatedTokens = await applyAnnotationStage(yomitanTokens, deps, annotationOptions);
|
||||
stageTimings.annotateMs = Date.now() - annotateStartedAtMs;
|
||||
const renderedTokens = applyCharacterNameImages(annotatedTokens, deps, annotationOptions);
|
||||
logStageTimings(renderedTokens.length);
|
||||
return {
|
||||
text: displayText,
|
||||
tokens: renderedTokens.length > 0 ? renderedTokens : null,
|
||||
};
|
||||
}
|
||||
|
||||
logStageTimings(0);
|
||||
return { text: displayText, tokens: null };
|
||||
}
|
||||
|
||||
@@ -366,8 +366,11 @@ export function createReplayMessageStore(messages: GoldenRecordedMessage[]): Rep
|
||||
};
|
||||
}
|
||||
|
||||
async function runInjectedScriptInVm(script: string, store: ReplayMessageStore): Promise<unknown> {
|
||||
return await vm.runInNewContext(script, {
|
||||
// One persistent context per fixture, matching the real parser window: the
|
||||
// scan runtime installs itself once into globalThis and later per-line call
|
||||
// scripts reuse it.
|
||||
function createInjectedScriptVm(store: ReplayMessageStore): (script: string) => Promise<unknown> {
|
||||
const context = vm.createContext({
|
||||
chrome: {
|
||||
runtime: {
|
||||
lastError: null,
|
||||
@@ -393,6 +396,7 @@ async function runInjectedScriptInVm(script: string, store: ReplayMessageStore):
|
||||
Set,
|
||||
String,
|
||||
});
|
||||
return async (script: string) => await vm.runInContext(script, context);
|
||||
}
|
||||
|
||||
export function createReplayTokenizerDeps(fixture: GoldenFixture): TokenizerServiceDeps {
|
||||
@@ -400,13 +404,14 @@ export function createReplayTokenizerDeps(fixture: GoldenFixture): TokenizerServ
|
||||
const scriptResults = new Map(
|
||||
fixture.recording.scripts.map((entry) => [entry.sha256, entry] as const),
|
||||
);
|
||||
const runInjectedScriptInVm = createInjectedScriptVm(store);
|
||||
|
||||
const parserWindow = {
|
||||
isDestroyed: () => false,
|
||||
webContents: {
|
||||
executeJavaScript: async (script: string) => {
|
||||
try {
|
||||
return await runInjectedScriptInVm(script, store);
|
||||
return await runInjectedScriptInVm(script);
|
||||
} catch (vmError) {
|
||||
const recorded = scriptResults.get(hashInjectedScript(script));
|
||||
if (recorded) {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3,6 +3,14 @@ import * as fs from 'fs';
|
||||
import * as http from 'http';
|
||||
import * as path from 'path';
|
||||
import { selectYomitanParseTokens } from './parser-selection-stage';
|
||||
import {
|
||||
buildYomitanScanCallScript,
|
||||
buildYomitanScanNameCandidatesScript,
|
||||
CHARACTER_DICTIONARY_TITLE_PREFIX,
|
||||
YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT,
|
||||
YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL,
|
||||
type YomitanFrequencyMode,
|
||||
} from './yomitan-scan-runtime-script';
|
||||
|
||||
interface LoggerLike {
|
||||
error: (message: string, ...args: unknown[]) => void;
|
||||
@@ -22,8 +30,6 @@ interface YomitanParserRuntimeDeps {
|
||||
createYomitanExtensionWindow?: (pageName: string) => Promise<BrowserWindow | null>;
|
||||
}
|
||||
|
||||
type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
||||
|
||||
export interface YomitanDictionaryInfo {
|
||||
title: string;
|
||||
revision?: string | number;
|
||||
@@ -74,13 +80,19 @@ export interface YomitanAddNoteResult {
|
||||
}
|
||||
|
||||
const DEFAULT_YOMITAN_SCAN_LENGTH = 40;
|
||||
const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
const yomitanProfileMetadataByWindow = new WeakMap<BrowserWindow, YomitanProfileMetadata>();
|
||||
const yomitanProfileDiagnosticsLoggedByWindow = new WeakSet<BrowserWindow>();
|
||||
const yomitanFrequencyCacheByWindow = new WeakMap<
|
||||
BrowserWindow,
|
||||
Map<string, YomitanTermFrequency[]>
|
||||
>();
|
||||
// Epoch passed with every scan request; the in-window termsFind cache clears
|
||||
// itself when the epoch changes (dictionary imports, settings changes).
|
||||
const yomitanScanCacheEpochByWindow = new WeakMap<BrowserWindow, number>();
|
||||
|
||||
function getYomitanScanCacheEpoch(window: BrowserWindow): number {
|
||||
return yomitanScanCacheEpochByWindow.get(window) ?? 0;
|
||||
}
|
||||
|
||||
function isObject(value: unknown): value is Record<string, unknown> {
|
||||
return Boolean(value && typeof value === 'object');
|
||||
@@ -99,6 +111,7 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
typeof entry.startPos === 'number' &&
|
||||
typeof entry.endPos === 'number' &&
|
||||
(entry.isNameMatch === undefined || typeof entry.isNameMatch === 'boolean') &&
|
||||
(entry.isUnparsedRun === undefined || typeof entry.isUnparsedRun === 'boolean') &&
|
||||
(entry.frequencyRank === undefined || typeof entry.frequencyRank === 'number') &&
|
||||
(entry.wordClasses === undefined ||
|
||||
(Array.isArray(entry.wordClasses) &&
|
||||
@@ -107,13 +120,9 @@ function isScanTokenArray(value: unknown): value is YomitanScanToken[] {
|
||||
);
|
||||
}
|
||||
|
||||
function scanTokenSpanKey(token: YomitanScanToken): string {
|
||||
return `${token.startPos}:${token.endPos}:${token.surface}`;
|
||||
}
|
||||
|
||||
// Maps a parse-selected token to the scanner-token shape carried out of the
|
||||
// parser runtime. Shared by both selectYomitanParseTokens fallback paths so the
|
||||
// projected fields stay in sync as the shape changes.
|
||||
// parser runtime, used by the parseText fallback path when the in-window
|
||||
// scanner is unavailable.
|
||||
function toYomitanScanToken(token: {
|
||||
surface: string;
|
||||
reading: string;
|
||||
@@ -132,66 +141,6 @@ function toYomitanScanToken(token: {
|
||||
};
|
||||
}
|
||||
|
||||
// parseText segmentation is authoritative (it emits filler chunks for text the
|
||||
// termsFind scanner skips), but only the termsFind scanner carries annotation
|
||||
// metadata (isNameMatch, frequencyRank, headwordReading, wordClasses). Graft
|
||||
// scanner tokens onto the parseText segmentation per matching span so one
|
||||
// unmatched chunk degrades only itself instead of dropping the whole line's
|
||||
// metadata.
|
||||
//
|
||||
// Exception: character-name tokens. The greedy name scan can re-segment text
|
||||
// around a name (e.g. とヨータ → と + ヨータ instead of とヨー + タ), so
|
||||
// parseText segmentation cannot be authoritative there. Each name span is
|
||||
// expanded until it aligns with token boundaries in both segmentations, then
|
||||
// the parse tokens inside are replaced with the scanner tokens.
|
||||
function mergeScannerTokensIntoParseTokens(
|
||||
parseScanTokens: YomitanScanToken[],
|
||||
scannerTokens: YomitanScanToken[],
|
||||
): YomitanScanToken[] {
|
||||
const scannerTokensBySpan = new Map<string, YomitanScanToken>();
|
||||
for (const token of scannerTokens) {
|
||||
scannerTokensBySpan.set(scanTokenSpanKey(token), token);
|
||||
}
|
||||
const graftedTokens = parseScanTokens.map(
|
||||
(token) => scannerTokensBySpan.get(scanTokenSpanKey(token)) ?? token,
|
||||
);
|
||||
|
||||
const nameTokens = scannerTokens.filter((token) => token.isNameMatch === true);
|
||||
if (nameTokens.length === 0) {
|
||||
return graftedTokens;
|
||||
}
|
||||
|
||||
const regions = nameTokens.map((token) => ({ start: token.startPos, end: token.endPos }));
|
||||
const allTokens = [...parseScanTokens, ...scannerTokens];
|
||||
let expanded = true;
|
||||
while (expanded) {
|
||||
expanded = false;
|
||||
for (const region of regions) {
|
||||
for (const token of allTokens) {
|
||||
const overlaps = token.startPos < region.end && token.endPos > region.start;
|
||||
const extendsBeyond = token.startPos < region.start || token.endPos > region.end;
|
||||
if (overlaps && extendsBeyond) {
|
||||
region.start = Math.min(region.start, token.startPos);
|
||||
region.end = Math.max(region.end, token.endPos);
|
||||
expanded = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isInsideNameRegion = (token: YomitanScanToken): boolean =>
|
||||
regions.some((region) => token.startPos >= region.start && token.endPos <= region.end);
|
||||
|
||||
const merged = graftedTokens.filter((token) => !isInsideNameRegion(token));
|
||||
for (const token of scannerTokens) {
|
||||
if (isInsideNameRegion(token)) {
|
||||
merged.push(token);
|
||||
}
|
||||
}
|
||||
merged.sort((a, b) => a.startPos - b.startPos || a.endPos - b.endPos);
|
||||
return merged;
|
||||
}
|
||||
|
||||
function makeTermReadingCacheKey(term: string, reading: string | null): string {
|
||||
return `${term}\u0000${reading ?? ''}`;
|
||||
}
|
||||
@@ -208,6 +157,7 @@ function getWindowFrequencyCache(window: BrowserWindow): Map<string, YomitanTerm
|
||||
function clearWindowCaches(window: BrowserWindow): void {
|
||||
yomitanProfileMetadataByWindow.delete(window);
|
||||
yomitanFrequencyCacheByWindow.delete(window);
|
||||
yomitanScanCacheEpochByWindow.set(window, getYomitanScanCacheEpoch(window) + 1);
|
||||
}
|
||||
export function clearYomitanParserCachesForWindow(window: BrowserWindow): void {
|
||||
clearWindowCaches(window);
|
||||
@@ -704,6 +654,10 @@ async function ensureYomitanParserWindow(
|
||||
if (readyPromise) {
|
||||
await readyPromise;
|
||||
}
|
||||
// Eagerly install the scan runtime so the first subtitle line does not
|
||||
// pay the install round trip; failures fall back to the per-request
|
||||
// install-and-retry path.
|
||||
await installYomitanScanRuntime(parserWindow).catch(() => {});
|
||||
|
||||
return true;
|
||||
} catch (err) {
|
||||
@@ -877,668 +831,42 @@ async function serveDictionaryZipOnce<T>(
|
||||
}
|
||||
}
|
||||
|
||||
const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
||||
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
||||
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
||||
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
||||
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
||||
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
||||
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0x3400, 0x9fff]];
|
||||
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
||||
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
||||
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
||||
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
||||
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
||||
function getSegmentReadingContribution(segment) {
|
||||
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
||||
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
||||
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
return isKanaOnly ? segmentText : "";
|
||||
async function installYomitanScanRuntime(parserWindow: BrowserWindow): Promise<void> {
|
||||
await parserWindow.webContents.executeJavaScript(YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT, true);
|
||||
// A fresh runtime has no candidate list; force the next scan to reinstall it.
|
||||
yomitanScanNameCandidateKeyByWindow.delete(parserWindow);
|
||||
}
|
||||
function getProlongedHiragana(previousCharacter) {
|
||||
switch (previousCharacter) {
|
||||
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
||||
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
||||
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
||||
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
||||
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
||||
default: return null;
|
||||
}
|
||||
}
|
||||
function getFuriganaKanaSegments(text, reading) {
|
||||
const newSegments = [];
|
||||
let start = 0;
|
||||
let state = (reading[0] === text[0]);
|
||||
for (let i = 1; i < text.length; ++i) {
|
||||
const newState = (reading[i] === text[i]);
|
||||
if (state === newState) { continue; }
|
||||
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
||||
state = newState;
|
||||
start = i;
|
||||
}
|
||||
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
||||
return newSegments;
|
||||
}
|
||||
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
||||
let result = '';
|
||||
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
||||
for (let char of text) {
|
||||
const codePoint = char.codePointAt(0);
|
||||
switch (codePoint) {
|
||||
case KATAKANA_SMALL_KA_CODE_POINT:
|
||||
case KATAKANA_SMALL_KE_CODE_POINT:
|
||||
break;
|
||||
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
||||
if (!keepProlongedSoundMarks && result.length > 0) {
|
||||
const char2 = getProlongedHiragana(result[result.length - 1]);
|
||||
if (char2 !== null) { char = char2; }
|
||||
}
|
||||
break;
|
||||
default:
|
||||
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
||||
char = String.fromCodePoint(codePoint + offset);
|
||||
}
|
||||
break;
|
||||
}
|
||||
result += char;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
||||
const groupCount = groups.length - groupsStart;
|
||||
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
||||
const group = groups[groupsStart];
|
||||
const {isKana, text} = group;
|
||||
if (isKana) {
|
||||
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
||||
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
||||
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
||||
return segments;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let result = null;
|
||||
for (let i = reading.length; i >= text.length; --i) {
|
||||
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (result !== null) { return null; }
|
||||
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
||||
result = segments;
|
||||
}
|
||||
if (groupCount === 1) { break; }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function distributeFurigana(term, reading) {
|
||||
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
||||
const groups = [];
|
||||
let groupPre = null;
|
||||
let isKanaPre = null;
|
||||
for (const c of term) {
|
||||
const isKana = isCodePointKana(c.codePointAt(0));
|
||||
if (isKana === isKanaPre) { groupPre.text += c; }
|
||||
else {
|
||||
groupPre = {isKana, text: c, textNormalized: null};
|
||||
groups.push(groupPre);
|
||||
isKanaPre = isKana;
|
||||
}
|
||||
}
|
||||
for (const group of groups) {
|
||||
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
||||
}
|
||||
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
||||
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
||||
}
|
||||
function getStemLength(text1, text2) {
|
||||
const minLength = Math.min(text1.length, text2.length);
|
||||
if (minLength === 0) { return 0; }
|
||||
let i = 0;
|
||||
while (true) {
|
||||
const char1 = text1.codePointAt(i);
|
||||
const char2 = text2.codePointAt(i);
|
||||
if (char1 !== char2) { break; }
|
||||
const charLength = String.fromCodePoint(char1).length;
|
||||
i += charLength;
|
||||
if (i >= minLength) {
|
||||
if (i > minLength) { i -= charLength; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
return i;
|
||||
}
|
||||
function distributeFuriganaInflected(term, reading, source) {
|
||||
const termNormalized = convertKatakanaToHiragana(term);
|
||||
const readingNormalized = convertKatakanaToHiragana(reading);
|
||||
const sourceNormalized = convertKatakanaToHiragana(source);
|
||||
let mainText = term;
|
||||
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
||||
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
||||
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
||||
mainText = reading;
|
||||
stemLength = readingStemLength;
|
||||
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
||||
}
|
||||
const segments = [];
|
||||
if (stemLength > 0) {
|
||||
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
||||
const segments2 = distributeFurigana(mainText, reading);
|
||||
let consumed = 0;
|
||||
for (const segment of segments2) {
|
||||
const start = consumed;
|
||||
consumed += segment.text.length;
|
||||
if (consumed < stemLength) { segments.push(segment); }
|
||||
else if (consumed === stemLength) { segments.push(segment); break; }
|
||||
else {
|
||||
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (stemLength < source.length) {
|
||||
const remainder = source.substring(stemLength);
|
||||
const last = segments[segments.length - 1];
|
||||
if (last && last.reading.length === 0) { last.text += remainder; }
|
||||
else { segments.push(createFuriganaSegment(remainder, '')); }
|
||||
}
|
||||
return segments;
|
||||
}
|
||||
function parsePositiveFrequencyNumber(value) {
|
||||
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||
return Math.max(1, Math.floor(value));
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
||||
if (!numericMatch) { return null; }
|
||||
const parsed = Number.parseFloat(numericMatch);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
||||
return Math.max(1, Math.floor(parsed));
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) {
|
||||
const parsed = parsePositiveFrequencyNumber(item);
|
||||
if (parsed !== null) { return parsed; }
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseDisplayFrequencyNumber(value) {
|
||||
if (typeof value === 'string') {
|
||||
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
||||
if (!leadingDigits) { return null; }
|
||||
const parsed = Number.parseInt(leadingDigits, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
||||
}
|
||||
return parsePositiveFrequencyNumber(value);
|
||||
}
|
||||
function getFrequencyDictionaryName(frequency) {
|
||||
const candidates = [
|
||||
frequency?.dictionary,
|
||||
frequency?.dictionaryName,
|
||||
frequency?.name,
|
||||
frequency?.title,
|
||||
frequency?.dictionaryTitle,
|
||||
frequency?.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
return candidate.trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
||||
for (const frequency of dictionaryEntry?.frequencies || []) {
|
||||
if (!frequency || typeof frequency !== 'object') { continue; }
|
||||
const frequencyHeadwordIndex = frequency.headwordIndex;
|
||||
if (typeof frequencyHeadwordIndex === 'number') {
|
||||
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
||||
} else if (headwordCount > 1) {
|
||||
continue;
|
||||
}
|
||||
const dictionary = getFrequencyDictionaryName(frequency);
|
||||
if (!dictionary) { continue; }
|
||||
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
||||
const rank =
|
||||
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
||||
parsePositiveFrequencyNumber(frequency.frequency);
|
||||
if (rank === null) { continue; }
|
||||
const priorityRaw = dictionaryPriorityByName[dictionary];
|
||||
const fallbackPriority =
|
||||
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
||||
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
||||
: Number.MAX_SAFE_INTEGER;
|
||||
const priority =
|
||||
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
||||
? Math.max(0, Math.floor(priorityRaw))
|
||||
: fallbackPriority;
|
||||
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
||||
best = { priority, rank };
|
||||
}
|
||||
}
|
||||
return best?.rank ?? null;
|
||||
}
|
||||
function hasExactSource(headword, token, requirePrimary) {
|
||||
for (const src of headword.sources || []) {
|
||||
if (src.originalText !== token) { continue; }
|
||||
if (requirePrimary && !src.isPrimary) { continue; }
|
||||
if (src.matchType !== 'exact') { continue; }
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
||||
const matches = [];
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
||||
matches.push({ dictionaryEntry, headword, headwordIndex });
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
function sameHeadword(match, preferredMatch) {
|
||||
if (!match || !preferredMatch) {
|
||||
return false;
|
||||
}
|
||||
if (match.headword?.term !== preferredMatch.headword?.term) {
|
||||
return false;
|
||||
}
|
||||
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
||||
const preferredReading =
|
||||
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
||||
if (!matchReading || !preferredReading) {
|
||||
return true;
|
||||
}
|
||||
return matchReading === preferredReading;
|
||||
}
|
||||
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
for (const match of matches) {
|
||||
const rank = getBestFrequencyRank(
|
||||
match.dictionaryEntry,
|
||||
match.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (rank === null) { continue; }
|
||||
if (best === null || rank < best) {
|
||||
best = rank;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
}
|
||||
function appendDictionaryNames(target, value) {
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
const candidates = [
|
||||
value.dictionary,
|
||||
value.dictionaryName,
|
||||
value.name,
|
||||
value.title,
|
||||
value.dictionaryTitle,
|
||||
value.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
target.push(candidate.trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
function getDictionaryEntryNames(entry) {
|
||||
const names = [];
|
||||
appendDictionaryNames(names, entry);
|
||||
for (const definition of entry?.definitions || []) {
|
||||
appendDictionaryNames(names, definition);
|
||||
}
|
||||
for (const frequency of entry?.frequencies || []) {
|
||||
appendDictionaryNames(names, frequency);
|
||||
}
|
||||
for (const pronunciation of entry?.pronunciations || []) {
|
||||
appendDictionaryNames(names, pronunciation);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
function isNameDictionaryEntry(entry) {
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
if (imageMatch) {
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseSubMinerMediaIdCandidate(value) {
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
||||
const parsed = Number.parseInt(value.trim(), 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function collectSubMinerMediaIds(value, target) {
|
||||
if (typeof value === 'string') {
|
||||
const parsed = parseSubMinerMediaIdFromString(value);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
||||
return;
|
||||
}
|
||||
const mediaIdCandidates = [
|
||||
value.subminerMediaId,
|
||||
value.subMinerMediaId,
|
||||
value.characterDictionaryMediaId,
|
||||
value.data?.subminerMediaId,
|
||||
value.data?.subMinerMediaId,
|
||||
value.data?.characterDictionaryMediaId
|
||||
];
|
||||
for (const candidate of mediaIdCandidates) {
|
||||
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
collectSubMinerMediaIds(child, target);
|
||||
}
|
||||
}
|
||||
function getSubMinerMediaIds(entry) {
|
||||
const mediaIds = new Set();
|
||||
collectSubMinerMediaIds(entry, mediaIds);
|
||||
return mediaIds;
|
||||
}
|
||||
function isCurrentMediaNameDictionaryEntry(entry) {
|
||||
if (!isNameDictionaryEntry(entry)) {
|
||||
return false;
|
||||
}
|
||||
if (currentCharacterDictionaryMediaId === null) {
|
||||
return true;
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
: (dictionaryEntries || []).filter((entry) => {
|
||||
if (!isNameDictionaryEntry(entry)) { return true; }
|
||||
return isCurrentMediaNameDictionaryEntry(entry);
|
||||
});
|
||||
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
||||
let matchedNameDictionary = false;
|
||||
if (includeNameMatchMetadata) {
|
||||
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
for (const match of exactPrimaryMatches) {
|
||||
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
||||
matchedNameDictionary = true;
|
||||
break;
|
||||
}
|
||||
if (matchedNameDictionary) { break; }
|
||||
}
|
||||
}
|
||||
const preferredMatch = exactPrimaryMatches[0];
|
||||
if (preferredMatch) {
|
||||
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
||||
.filter((match) => sameHeadword(match, preferredMatch));
|
||||
return {
|
||||
term: preferredMatch.headword.term,
|
||||
reading: preferredMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
||||
isNameMatch:
|
||||
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
||||
frequencyRank: getBestFrequencyRankForMatches(
|
||||
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
`;
|
||||
|
||||
function buildYomitanScanningScript(
|
||||
text: string,
|
||||
profileIndex: number,
|
||||
scanLength: number,
|
||||
includeNameMatchMetadata: boolean,
|
||||
greedyNameScanEnabled: boolean,
|
||||
currentCharacterDictionaryMediaId: number | null,
|
||||
dictionaryPriorityByName: Record<string, number>,
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>,
|
||||
): string {
|
||||
return `
|
||||
(async () => {
|
||||
const invoke = (action, params) =>
|
||||
new Promise((resolve, reject) => {
|
||||
chrome.runtime.sendMessage({ action, params }, (response) => {
|
||||
if (chrome.runtime.lastError) {
|
||||
reject(new Error(chrome.runtime.lastError.message));
|
||||
// Key of the character-name candidate list currently installed in each parser
|
||||
// window, so an unchanged list costs nothing per line.
|
||||
const yomitanScanNameCandidateKeyByWindow = new WeakMap<BrowserWindow, string>();
|
||||
|
||||
async function ensureYomitanScanNameCandidates(
|
||||
parserWindow: BrowserWindow,
|
||||
nameCandidates: { key: string; forms: string[] } | null,
|
||||
logger: LoggerLike,
|
||||
): Promise<void> {
|
||||
const installedKey = yomitanScanNameCandidateKeyByWindow.get(parserWindow);
|
||||
const nextKey = nameCandidates?.key ?? '';
|
||||
if (installedKey === nextKey) {
|
||||
return;
|
||||
}
|
||||
if (!response || typeof response !== "object") {
|
||||
reject(new Error("Invalid response from Yomitan backend"));
|
||||
return;
|
||||
}
|
||||
if (response.error) {
|
||||
reject(new Error(response.error.message || "Yomitan backend error"));
|
||||
return;
|
||||
}
|
||||
resolve(response.result);
|
||||
});
|
||||
});
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const includeNameMatchMetadata = ${includeNameMatchMetadata ? 'true' : 'false'};
|
||||
const greedyNameScanEnabled = ${greedyNameScanEnabled ? 'true' : 'false'};
|
||||
const currentCharacterDictionaryMediaId = ${
|
||||
currentCharacterDictionaryMediaId !== null
|
||||
? String(currentCharacterDictionaryMediaId)
|
||||
: 'null'
|
||||
};
|
||||
const dictionaryPriorityByName = ${JSON.stringify(dictionaryPriorityByName)};
|
||||
const dictionaryFrequencyModeByName = ${JSON.stringify(dictionaryFrequencyModeByName)};
|
||||
const text = ${JSON.stringify(text)};
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
const termsFindCache = new Map();
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const cacheKey = position + ":" + windowLength;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached) { return cached; }
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const result = await invoke("termsFind", { text: substring, details, optionsContext: { index: ${profileIndex} } });
|
||||
termsFindCache.set(cacheKey, result);
|
||||
return result;
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
surface: segments.map((segment) => segment.text).join("") || source,
|
||||
reading: segments.map(getSegmentReadingContribution).join(""),
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
? Math.max(1, Math.floor(preferredHeadword.frequencyRank))
|
||||
: undefined,
|
||||
};
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return tokenPayload;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
|
||||
try {
|
||||
await parserWindow.webContents.executeJavaScript(
|
||||
buildYomitanScanNameCandidatesScript(nameCandidates),
|
||||
true,
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
yomitanScanNameCandidateKeyByWindow.set(parserWindow, nextKey);
|
||||
} catch (err) {
|
||||
// The scan falls back to checking every position when the list is absent,
|
||||
// so a failed install costs speed, never a missed name.
|
||||
logger.warn?.(
|
||||
'Failed to install Yomitan character-name scan candidates:',
|
||||
(err as Error).message,
|
||||
);
|
||||
yomitanScanNameCandidateKeyByWindow.delete(parserWindow);
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, ${scanLength});
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + ${scanLength});
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
while (i < text.length) {
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
// Cap the window at the next reserved name span so a generic match
|
||||
// cannot consume into it.
|
||||
const windowLength = nextNameToken ? Math.min(${scanLength}, nextNameToken.startPos - i) : ${scanLength};
|
||||
let attempt = await findTokenAt(i, windowLength);
|
||||
// Yomitan text normalization can consume characters (whitespace,
|
||||
// punctuation) beyond the matched term, leaving no headword whose
|
||||
// source equals the consumed text. Retry with shorter windows so a
|
||||
// valid prefix term (e.g. a character name before a paren) still
|
||||
// tokenizes instead of the position being skipped.
|
||||
let retryLength = Math.min(attempt.matchedLength, windowLength) - 1;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
const retry = await findTokenAt(i, retryLength);
|
||||
if (retry.token) {
|
||||
attempt = retry;
|
||||
break;
|
||||
}
|
||||
retryLength = Math.min(retryLength - 1, retry.matchedLength - 1);
|
||||
}
|
||||
if (attempt.token) {
|
||||
tokens.push(attempt.token);
|
||||
i += attempt.matchedLength;
|
||||
continue;
|
||||
}
|
||||
i += String.fromCodePoint(text.codePointAt(i)).length;
|
||||
}
|
||||
return tokens;
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
export async function requestYomitanParseResults(
|
||||
@@ -1635,6 +963,20 @@ export async function requestYomitanParseResults(
|
||||
}
|
||||
}
|
||||
|
||||
// parseText fallback for when the in-window scanner cannot run (script eval
|
||||
// failure, unexpected payload). The scanner walk is the primary tokenizer and
|
||||
// emits its own filler runs, so this extra full parse only happens on errors.
|
||||
async function requestYomitanParseFallbackTokens(
|
||||
text: string,
|
||||
deps: YomitanParserRuntimeDeps,
|
||||
logger: LoggerLike,
|
||||
): Promise<YomitanScanToken[] | null> {
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens = selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
return parseScanTokens && parseScanTokens.length > 0 ? parseScanTokens : null;
|
||||
}
|
||||
|
||||
export async function requestYomitanScanTokens(
|
||||
text: string,
|
||||
deps: YomitanParserRuntimeDeps,
|
||||
@@ -1642,6 +984,7 @@ export async function requestYomitanScanTokens(
|
||||
options?: {
|
||||
includeNameMatchMetadata?: boolean;
|
||||
currentCharacterDictionaryMediaId?: number | null;
|
||||
nameCandidates?: { key: string; forms: string[] } | null;
|
||||
},
|
||||
): Promise<YomitanScanToken[] | null> {
|
||||
const yomitanExt = deps.getYomitanExt();
|
||||
@@ -1655,10 +998,6 @@ export async function requestYomitanScanTokens(
|
||||
return null;
|
||||
}
|
||||
|
||||
const parseResults = await requestYomitanParseResults(text, deps, logger);
|
||||
const selectedParseTokens = selectYomitanParseTokens(parseResults, () => false, 'headword');
|
||||
const parseScanTokens = selectedParseTokens?.map(toYomitanScanToken) ?? null;
|
||||
|
||||
const metadata = await requestYomitanProfileMetadata(parserWindow, logger);
|
||||
const profileIndex = metadata?.profileIndex ?? 0;
|
||||
const scanLength = metadata?.scanLength ?? DEFAULT_YOMITAN_SCAN_LENGTH;
|
||||
@@ -1669,44 +1008,63 @@ export async function requestYomitanScanTokens(
|
||||
name.startsWith(CHARACTER_DICTIONARY_TITLE_PREFIX),
|
||||
);
|
||||
|
||||
try {
|
||||
const rawResult = await parserWindow.webContents.executeJavaScript(
|
||||
buildYomitanScanningScript(
|
||||
// Candidate name forms let the in-page pre-pass skip positions where no
|
||||
// character name can start. Installed only when it changes (per media), so
|
||||
// the per-line call stays a single tiny script.
|
||||
const nameCandidates = greedyNameScanEnabled ? (options?.nameCandidates ?? null) : null;
|
||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||
|
||||
const callScript = buildYomitanScanCallScript({
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
currentCharacterDictionaryMediaId:
|
||||
typeof options?.currentCharacterDictionaryMediaId === 'number' &&
|
||||
Number.isFinite(options.currentCharacterDictionaryMediaId) &&
|
||||
options.currentCharacterDictionaryMediaId > 0
|
||||
? Math.floor(options.currentCharacterDictionaryMediaId)
|
||||
: null,
|
||||
metadata?.dictionaryPriorityByName ?? {},
|
||||
metadata?.dictionaryFrequencyModeByName ?? {},
|
||||
),
|
||||
true,
|
||||
);
|
||||
dictionaryPriorityByName: metadata?.dictionaryPriorityByName ?? {},
|
||||
dictionaryFrequencyModeByName: metadata?.dictionaryFrequencyModeByName ?? {},
|
||||
cacheEpoch: getYomitanScanCacheEpoch(parserWindow),
|
||||
nameCandidateKey: nameCandidates?.key ?? null,
|
||||
});
|
||||
|
||||
try {
|
||||
let rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||
if (rawResult === YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL) {
|
||||
// First request for this window, or the page reloaded and dropped the
|
||||
// installed runtime: install and retry once. The candidate list lives in
|
||||
// the same page state, so it has to be reinstalled alongside it.
|
||||
await installYomitanScanRuntime(parserWindow);
|
||||
await ensureYomitanScanNameCandidates(parserWindow, nameCandidates, logger);
|
||||
rawResult = await parserWindow.webContents.executeJavaScript(callScript, true);
|
||||
}
|
||||
// The scanner reports a line where a position ran out of shrinking-window
|
||||
// retries: it stopped short of windows an uncapped ladder would have tried,
|
||||
// so a real term may be sitting in an unparsed run. One parseText for the
|
||||
// line is the bounded way to get the exhaustive answer back (this is the
|
||||
// parse the scanner replaced, and it only runs for these rare lines).
|
||||
if (isObject(rawResult) && rawResult.retryBudgetExhausted === true) {
|
||||
logger.info?.('Yomitan scanner exhausted its retry budget; parsing the line as a fallback.');
|
||||
const fallbackTokens = await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
if (fallbackTokens) {
|
||||
return fallbackTokens;
|
||||
}
|
||||
rawResult = rawResult.tokens;
|
||||
}
|
||||
if (isScanTokenArray(rawResult)) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return mergeScannerTokensIntoParseTokens(parseScanTokens, rawResult);
|
||||
// Filler-only results carry no dictionary match; keep the historical
|
||||
// contract of returning null so callers fall back to raw text.
|
||||
return rawResult.some((token) => token.isUnparsedRun !== true) ? rawResult : null;
|
||||
}
|
||||
return rawResult;
|
||||
}
|
||||
if (Array.isArray(rawResult)) {
|
||||
const selectedTokens = selectYomitanParseTokens(rawResult, () => false, 'headword');
|
||||
return selectedTokens?.map(toYomitanScanToken) ?? null;
|
||||
}
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
}
|
||||
return null;
|
||||
logger.error('Yomitan scanner returned an unexpected payload; using parseText fallback.');
|
||||
return await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
} catch (err) {
|
||||
if (parseScanTokens && parseScanTokens.length > 0) {
|
||||
return parseScanTokens;
|
||||
}
|
||||
logger.error('Yomitan scanner request failed:', (err as Error).message);
|
||||
return null;
|
||||
return await requestYomitanParseFallbackTokens(text, deps, logger);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,513 @@
|
||||
// In-page Yomitan scan runtime: the scan walk that gets installed once per
|
||||
// parser window as globalThis.__subminerYomitanScan, plus the tiny per-line
|
||||
// call script. Kept separate from the host runtime module so the injected
|
||||
// script text (which is data, not executed here) does not dominate that file;
|
||||
// the helper bundle it embeds lives in yomitan-scanning-helpers-script.ts.
|
||||
import { YOMITAN_SCANNING_HELPERS } from './yomitan-scanning-helpers-script';
|
||||
|
||||
export { CHARACTER_DICTIONARY_TITLE_PREFIX } from './yomitan-scanning-helpers-script';
|
||||
|
||||
export type YomitanFrequencyMode = 'occurrence-based' | 'rank-based';
|
||||
|
||||
// Bump whenever the install script below changes so already-loaded parser
|
||||
// windows re-install the new scan runtime instead of running the stale one.
|
||||
export const YOMITAN_SCAN_RUNTIME_VERSION = 6;
|
||||
export const YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL = '__subminer-yomitan-scan-runtime-missing__';
|
||||
|
||||
export interface YomitanScanRequestParams {
|
||||
text: string;
|
||||
profileIndex: number;
|
||||
scanLength: number;
|
||||
includeNameMatchMetadata: boolean;
|
||||
greedyNameScanEnabled: boolean;
|
||||
currentCharacterDictionaryMediaId: number | null;
|
||||
dictionaryPriorityByName: Record<string, number>;
|
||||
dictionaryFrequencyModeByName: Partial<Record<string, YomitanFrequencyMode>>;
|
||||
cacheEpoch: number;
|
||||
/**
|
||||
* Key of the character-name candidate list installed for the current media,
|
||||
* or null to scan every Japanese position (see the pre-pass prefilter).
|
||||
*/
|
||||
nameCandidateKey: string | null;
|
||||
}
|
||||
|
||||
// Installed once per parser window (and re-installed after in-page reloads):
|
||||
// keeps V8 from re-parsing the helper bundle on every subtitle line, and hosts
|
||||
// the cross-line termsFind cache. Each subtitle line then only evaluates a tiny
|
||||
// call into globalThis.__subminerYomitanScan.
|
||||
export const YOMITAN_SCAN_RUNTIME_INSTALL_SCRIPT = String.raw`
|
||||
(() => {
|
||||
if (globalThis.__subminerYomitanScanVersion === ${YOMITAN_SCAN_RUNTIME_VERSION}) {
|
||||
return true;
|
||||
}
|
||||
const invoke = (action, params) =>
|
||||
new Promise((resolve, reject) => {
|
||||
chrome.runtime.sendMessage({ action, params }, (response) => {
|
||||
if (chrome.runtime.lastError) {
|
||||
reject(new Error(chrome.runtime.lastError.message));
|
||||
return;
|
||||
}
|
||||
if (!response || typeof response !== "object") {
|
||||
reject(new Error("Invalid response from Yomitan backend"));
|
||||
return;
|
||||
}
|
||||
if (response.error) {
|
||||
reject(new Error(response.error.message || "Yomitan backend error"));
|
||||
return;
|
||||
}
|
||||
resolve(response.result);
|
||||
});
|
||||
});
|
||||
// Cross-line termsFind LRU keyed by profile + substring: subtitle lines
|
||||
// repeat particles and inflections constantly, so most lookups hit here.
|
||||
// Entries hold in-flight promises so concurrent identical lookups dedupe.
|
||||
const termsFindCache = new Map();
|
||||
// Two bounds. The key count keeps the map itself small; the accumulated
|
||||
// dictionary-entry count stands in for retained bytes, because a single
|
||||
// lookup over a common prefix can hold hundreds of entries with their full
|
||||
// glossaries and a key-count cap alone would not bound that.
|
||||
const TERMS_FIND_CACHE_LIMIT = 2000;
|
||||
const TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT = 20000;
|
||||
let termsFindCacheDictionaryEntries = 0;
|
||||
let termsFindCacheEpoch = -1;
|
||||
function dropCachedTermsFind(cacheKey, entry) {
|
||||
if (termsFindCache.get(cacheKey) !== entry) { return; }
|
||||
termsFindCache.delete(cacheKey);
|
||||
termsFindCacheDictionaryEntries -= entry.dictionaryEntryCount;
|
||||
}
|
||||
// Runs on insert and again once a lookup resolves: an entry is only worth
|
||||
// its estimated weight of 1 until then, so a single oversized response
|
||||
// would otherwise sit in the cache forever, over the limit and reused.
|
||||
function evictOverflowingTermsFindEntries() {
|
||||
while (
|
||||
termsFindCache.size > TERMS_FIND_CACHE_LIMIT ||
|
||||
termsFindCacheDictionaryEntries > TERMS_FIND_CACHE_DICTIONARY_ENTRY_LIMIT
|
||||
) {
|
||||
const oldest = termsFindCache.entries().next().value;
|
||||
if (oldest === undefined) { break; }
|
||||
dropCachedTermsFind(oldest[0], oldest[1]);
|
||||
}
|
||||
}
|
||||
// Only blind ladder steps are capped (see the retry loop): those are the
|
||||
// ones that would otherwise degrade into O(scanLength) lookups at a single
|
||||
// position. Steps the backend guides by reporting a shorter consumed length
|
||||
// stay uncapped, so a valid prefix term is still found on lines where
|
||||
// normalization eats a long tail.
|
||||
const MAX_BLIND_SHRINKING_WINDOW_RETRIES = 4;
|
||||
// Character-name candidate forms for the current media, installed
|
||||
// separately from the per-line scan call so the per-line script stays tiny.
|
||||
// Stored raw here; the normalized lookup index is built inside the scan,
|
||||
// where the kana-normalization helper is in scope, and reused by key.
|
||||
let rawNameCandidates = null;
|
||||
let nameCandidateIndex = null;
|
||||
globalThis.__subminerYomitanScanSetNameCandidates = (key, forms) => {
|
||||
if (!key || !Array.isArray(forms) || forms.length === 0) {
|
||||
rawNameCandidates = null;
|
||||
nameCandidateIndex = null;
|
||||
return false;
|
||||
}
|
||||
rawNameCandidates = { key, forms };
|
||||
nameCandidateIndex = null;
|
||||
return true;
|
||||
};
|
||||
globalThis.__subminerYomitanScanVersion = ${YOMITAN_SCAN_RUNTIME_VERSION};
|
||||
globalThis.__subminerYomitanScan = async (scanParams) => {
|
||||
const {
|
||||
text,
|
||||
profileIndex,
|
||||
scanLength,
|
||||
includeNameMatchMetadata,
|
||||
greedyNameScanEnabled,
|
||||
currentCharacterDictionaryMediaId,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName,
|
||||
cacheEpoch,
|
||||
nameCandidateKey
|
||||
} = scanParams;
|
||||
if (cacheEpoch !== termsFindCacheEpoch) {
|
||||
termsFindCache.clear();
|
||||
termsFindCacheDictionaryEntries = 0;
|
||||
termsFindCacheEpoch = cacheEpoch;
|
||||
}
|
||||
${YOMITAN_SCANNING_HELPERS}
|
||||
const CAPTION_OPENING_BRACKETS = new Set(["(", "(", "[", "[", "{", "{", "「", "『", "【", "〈", "《", "≪", "<", "<"]);
|
||||
function shouldEmitUnparsedRunAsToken(runText) {
|
||||
if (!/[\p{L}\p{N}]/u.test(runText)) { return false; }
|
||||
const firstChar = Array.from(runText.trim())[0];
|
||||
return firstChar !== undefined && !CAPTION_OPENING_BRACKETS.has(firstChar);
|
||||
}
|
||||
function isLookupWorthyCodePoint(codePoint) {
|
||||
if (isCodePointJapanese(codePoint)) { return true; }
|
||||
return /[\p{L}\p{N}]/u.test(String.fromCodePoint(codePoint));
|
||||
}
|
||||
function isKanaOnlyRunText(runText) {
|
||||
const chars = Array.from(runText);
|
||||
return chars.length > 0 && chars.every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
}
|
||||
const details = {matchType: "exact", deinflect: true};
|
||||
const tokens = [];
|
||||
async function termsFindAt(position, windowLength) {
|
||||
const substring = text.substring(position, position + windowLength);
|
||||
const cacheKey = profileIndex + "\u0000" + substring;
|
||||
const cached = termsFindCache.get(cacheKey);
|
||||
if (cached !== undefined) {
|
||||
termsFindCache.delete(cacheKey);
|
||||
termsFindCache.set(cacheKey, cached);
|
||||
return await cached.promise;
|
||||
}
|
||||
// An in-flight lookup counts as one entry until it resolves; the real
|
||||
// weight replaces that estimate once the result is known.
|
||||
const entry = { promise: null, dictionaryEntryCount: 1 };
|
||||
entry.promise = invoke("termsFind", { text: substring, details, optionsContext: { index: profileIndex } })
|
||||
.then((result) => {
|
||||
const resolvedCount =
|
||||
1 + (Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries.length : 0);
|
||||
const isCached = termsFindCache.get(cacheKey) === entry;
|
||||
if (isCached) {
|
||||
termsFindCacheDictionaryEntries += resolvedCount - entry.dictionaryEntryCount;
|
||||
}
|
||||
entry.dictionaryEntryCount = resolvedCount;
|
||||
// The real weight can push the cache over its budget, and a single
|
||||
// response can exceed it on its own, so re-check here.
|
||||
if (isCached) { evictOverflowingTermsFindEntries(); }
|
||||
return result;
|
||||
});
|
||||
termsFindCache.set(cacheKey, entry);
|
||||
termsFindCacheDictionaryEntries += entry.dictionaryEntryCount;
|
||||
evictOverflowingTermsFindEntries();
|
||||
try {
|
||||
return await entry.promise;
|
||||
} catch (error) {
|
||||
dropCachedTermsFind(cacheKey, entry);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
// Text the walk skips accumulates into unparsed runs, mirroring the
|
||||
// filler chunks the parseText segmentation used to provide: runs stay
|
||||
// hoverable (flagged isUnparsedRun) unless they are punctuation-only or
|
||||
// caption-style asides, and kana continuations of a longer headword
|
||||
// extend the previous token instead.
|
||||
function flushUnparsedRun(runStart, runEnd) {
|
||||
if (runStart === null || runEnd <= runStart) { return; }
|
||||
const runText = text.substring(runStart, runEnd);
|
||||
const previousToken = tokens[tokens.length - 1];
|
||||
if (
|
||||
previousToken &&
|
||||
previousToken.endPos === runStart &&
|
||||
isKanaOnlyRunText(runText) &&
|
||||
typeof previousToken.headword === "string" &&
|
||||
previousToken.headword.length > previousToken.surface.length &&
|
||||
previousToken.headword.startsWith(previousToken.surface + runText)
|
||||
) {
|
||||
previousToken.surface += runText;
|
||||
// The run is kana-only, so its reading is itself: append it or the
|
||||
// reading stops covering the surface, which disables the known-word
|
||||
// reading fallback (isCompleteReadingForSurface) downstream.
|
||||
previousToken.reading += runText;
|
||||
// The run is kana-only, so its reading is itself: append it or the
|
||||
// reading stops covering the surface, which disables the known-word
|
||||
// reading fallback (isCompleteReadingForSurface) downstream.
|
||||
previousToken.endPos = runEnd;
|
||||
return;
|
||||
}
|
||||
if (!shouldEmitUnparsedRunAsToken(runText)) { return; }
|
||||
tokens.push({
|
||||
surface: runText,
|
||||
reading: "",
|
||||
headword: runText,
|
||||
startPos: runStart,
|
||||
endPos: runEnd,
|
||||
isUnparsedRun: true
|
||||
});
|
||||
}
|
||||
function buildScanToken(position, source, preferredHeadword) {
|
||||
const reading = typeof preferredHeadword.reading === "string" ? preferredHeadword.reading : "";
|
||||
const segments = distributeFuriganaInflected(preferredHeadword.term, reading, source);
|
||||
const tokenPayload = {
|
||||
surface: segments.map((segment) => segment.text).join("") || source,
|
||||
reading: segments.map(getSegmentReadingContribution).join(""),
|
||||
headword: preferredHeadword.term,
|
||||
headwordReading: reading || undefined,
|
||||
startPos: position,
|
||||
endPos: position + source.length,
|
||||
isNameMatch: includeNameMatchMetadata && preferredHeadword.isNameMatch === true,
|
||||
frequencyRank:
|
||||
typeof preferredHeadword.frequencyRank === "number" && Number.isFinite(preferredHeadword.frequencyRank)
|
||||
? Math.max(1, Math.floor(preferredHeadword.frequencyRank))
|
||||
: undefined,
|
||||
};
|
||||
if (Array.isArray(preferredHeadword.wordClasses) && preferredHeadword.wordClasses.length > 0) {
|
||||
tokenPayload.wordClasses = preferredHeadword.wordClasses;
|
||||
}
|
||||
return tokenPayload;
|
||||
}
|
||||
// findTokenAt plus the shrinking-window ladder below it: Yomitan text
|
||||
// normalization can consume characters (whitespace, punctuation) beyond
|
||||
// the matched term, leaving no headword whose source equals the consumed
|
||||
// text. Retry with shorter windows so a valid prefix term (e.g. a
|
||||
// character name before a paren) still tokenizes instead of the position
|
||||
// being skipped.
|
||||
// Every window at or above the consumed length repeats the same result,
|
||||
// so the next informative window sits just below it. A lookup that
|
||||
// consumed its whole window reports nothing to aim at, and the step down
|
||||
// from it is a blind guess: only those are budgeted.
|
||||
// The window can run past the end of the line, so blindness is judged
|
||||
// against the text the lookup actually saw.
|
||||
// Set when a position stopped short of windows an uncapped ladder would
|
||||
// still have tried; the line then escalates to parseText at the end.
|
||||
let blindRetryBudgetExhausted = false;
|
||||
async function resolveTokenAt(position, windowLength) {
|
||||
let attempt = await findTokenAt(position, windowLength);
|
||||
const scannedLength = Math.min(windowLength, text.length - position);
|
||||
let retryLength = Math.min(attempt.matchedLength, scannedLength) - 1;
|
||||
let stepIsBlind = attempt.matchedLength >= scannedLength;
|
||||
let blindRetriesRemaining = MAX_BLIND_SHRINKING_WINDOW_RETRIES;
|
||||
while (!attempt.token && retryLength >= 1) {
|
||||
if (stepIsBlind) {
|
||||
if (blindRetriesRemaining <= 0) {
|
||||
blindRetryBudgetExhausted = true;
|
||||
break;
|
||||
}
|
||||
blindRetriesRemaining -= 1;
|
||||
}
|
||||
const retry = await findTokenAt(position, retryLength);
|
||||
if (retry.token) { return retry; }
|
||||
const guidedLength = retry.matchedLength - 1;
|
||||
stepIsBlind = guidedLength >= retryLength - 1;
|
||||
retryLength = Math.min(retryLength - 1, guidedLength);
|
||||
}
|
||||
return attempt;
|
||||
}
|
||||
async function findTokenAt(position, windowLength) {
|
||||
const codePoint = text.codePointAt(position);
|
||||
const character = String.fromCodePoint(codePoint);
|
||||
const result = await termsFindAt(position, windowLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const originalTextLength = typeof result?.originalTextLength === "number" ? result.originalTextLength : 0;
|
||||
if (dictionaryEntries.length === 0 || originalTextLength <= 0 || (originalTextLength === character.length && !isCodePointJapanese(codePoint))) {
|
||||
return { token: null, matchedLength: 0 };
|
||||
}
|
||||
const source = text.substring(position, position + originalTextLength);
|
||||
const preferredHeadword = getPreferredHeadword(
|
||||
dictionaryEntries,
|
||||
source,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (!preferredHeadword || typeof preferredHeadword.term !== "string") {
|
||||
return { token: null, matchedLength: originalTextLength };
|
||||
}
|
||||
return { token: buildScanToken(position, source, preferredHeadword), matchedLength: originalTextLength };
|
||||
}
|
||||
// Halfwidth katakana survives kana normalization unchanged, so a name
|
||||
// written that way would not prefix-match a candidate form. Those
|
||||
// positions bypass the prefilter rather than risk a missed name.
|
||||
function isHalfwidthKatakanaCodePoint(codePoint) {
|
||||
return codePoint >= 0xff66 && codePoint <= 0xff9f;
|
||||
}
|
||||
// Build (once per candidate list) a first-character bucket index of the
|
||||
// normalized name forms, so the pre-pass can reject a position with a
|
||||
// single map hit instead of a backend round trip.
|
||||
if (rawNameCandidates && nameCandidateIndex?.key !== rawNameCandidates.key) {
|
||||
const byFirstChar = new Map();
|
||||
for (const form of rawNameCandidates.forms) {
|
||||
const normalized = typeof form === "string" ? convertKatakanaToHiragana(form.trim()) : "";
|
||||
if (!normalized) { continue; }
|
||||
const bucket = byFirstChar.get(normalized[0]);
|
||||
if (bucket) { bucket.push(normalized); } else { byFirstChar.set(normalized[0], [normalized]); }
|
||||
}
|
||||
nameCandidateIndex = byFirstChar.size > 0 ? { key: rawNameCandidates.key, byFirstChar } : null;
|
||||
} else if (!rawNameCandidates) {
|
||||
nameCandidateIndex = null;
|
||||
}
|
||||
// Only meaningful when the installed list matches the media this scan is
|
||||
// for; otherwise fall back to scanning every position.
|
||||
const activeNameCandidateIndex =
|
||||
nameCandidateKey !== null && nameCandidateIndex?.key === nameCandidateKey
|
||||
? nameCandidateIndex
|
||||
: null;
|
||||
const normalizedText = activeNameCandidateIndex ? convertKatakanaToHiragana(text) : "";
|
||||
// Yomitan collapses emphatic sequences before matching (すっっごーーい →
|
||||
// すごい), so a stretched name still resolves to its entry. Skipping these
|
||||
// characters keeps such spellings candidates; the filter only ever grows
|
||||
// the probe set, so a false positive costs one lookup, never a name.
|
||||
const EMPHATIC_SKIP_CHARS = new Set(["ぁ", "ぃ", "ぅ", "ぇ", "ぉ", "っ", "ゃ", "ゅ", "ょ", "ー"]);
|
||||
function matchesCandidateFormAt(form, position) {
|
||||
let textIndex = position;
|
||||
for (let formIndex = 0; formIndex < form.length; formIndex += 1) {
|
||||
while (
|
||||
textIndex < normalizedText.length &&
|
||||
normalizedText[textIndex] !== form[formIndex] &&
|
||||
EMPHATIC_SKIP_CHARS.has(normalizedText[textIndex])
|
||||
) {
|
||||
textIndex += 1;
|
||||
}
|
||||
if (normalizedText[textIndex] !== form[formIndex]) { return false; }
|
||||
textIndex += 1;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
function couldNameStartAt(position, codePoint) {
|
||||
if (!activeNameCandidateIndex) { return true; }
|
||||
if (isHalfwidthKatakanaCodePoint(codePoint)) { return true; }
|
||||
const bucket = activeNameCandidateIndex.byFirstChar.get(normalizedText[position]);
|
||||
if (!bucket) { return false; }
|
||||
for (const form of bucket) {
|
||||
if (matchesCandidateFormAt(form, position)) { return true; }
|
||||
}
|
||||
return false;
|
||||
}
|
||||
// Greedy name pre-pass: character-name matches claim their spans before
|
||||
// the left-to-right walk, so a longer generic match starting earlier
|
||||
// (e.g. とヨー → 渡洋) cannot swallow the start of a name (ヨータ).
|
||||
const nameTokens = [];
|
||||
if (greedyNameScanEnabled) {
|
||||
let namePos = 0;
|
||||
while (namePos < text.length) {
|
||||
const codePoint = text.codePointAt(namePos);
|
||||
if (!isCodePointJapanese(codePoint) || !couldNameStartAt(namePos, codePoint)) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const result = await termsFindAt(namePos, scanLength);
|
||||
const dictionaryEntries = Array.isArray(result?.dictionaryEntries) ? result.dictionaryEntries : [];
|
||||
const textWindow = text.substring(namePos, namePos + scanLength);
|
||||
const nameMatch = findLongestNameMatch(dictionaryEntries, textWindow);
|
||||
// A name only claims its span when no strictly longer generic word
|
||||
// starts at the same position (a character named 空 must not split
|
||||
// 空気). Ties go to the name. Generic matches that start earlier and
|
||||
// overlap the name are still blocked by the reservation.
|
||||
if (
|
||||
!nameMatch ||
|
||||
findLongestGenericMatchLength(dictionaryEntries, textWindow) > nameMatch.sourceLength
|
||||
) {
|
||||
namePos += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
const source = text.substring(namePos, namePos + nameMatch.sourceLength);
|
||||
nameTokens.push(buildScanToken(namePos, source, {
|
||||
term: nameMatch.headword.term,
|
||||
reading: nameMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(nameMatch.headword),
|
||||
isNameMatch: true,
|
||||
frequencyRank: getBestFrequencyRank(
|
||||
nameMatch.dictionaryEntry,
|
||||
nameMatch.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
}));
|
||||
namePos += nameMatch.sourceLength;
|
||||
}
|
||||
}
|
||||
// First reserved name span that a match ending at endPos would leave
|
||||
// half-consumed. Spans the match covers entirely are not returned: those
|
||||
// lose to the longer word instead of splitting it.
|
||||
function findSplitNameToken(startIndex, endPos) {
|
||||
for (let index = startIndex; index < nameTokens.length; index += 1) {
|
||||
const nameToken = nameTokens[index];
|
||||
if (nameToken.startPos >= endPos) { return null; }
|
||||
if (nameToken.endPos > endPos) { return nameToken; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let i = 0;
|
||||
let nameIndex = 0;
|
||||
let unparsedRunStart = null;
|
||||
while (i < text.length) {
|
||||
while (nameIndex < nameTokens.length && nameTokens[nameIndex].startPos < i) { nameIndex += 1; }
|
||||
const nextNameToken = nameIndex < nameTokens.length ? nameTokens[nameIndex] : null;
|
||||
if (nextNameToken && nextNameToken.startPos === i) {
|
||||
flushUnparsedRun(unparsedRunStart, i);
|
||||
unparsedRunStart = null;
|
||||
tokens.push(nextNameToken);
|
||||
i = nextNameToken.endPos;
|
||||
nameIndex += 1;
|
||||
continue;
|
||||
}
|
||||
const codePoint = text.codePointAt(i);
|
||||
// Punctuation and whitespace can never start a token: skip the backend
|
||||
// round trip entirely. Latin letters and digits stay lookup-worthy
|
||||
// (terms like Tシャツ start on an ASCII letter).
|
||||
if (!isLookupWorthyCodePoint(codePoint)) {
|
||||
if (unparsedRunStart === null) { unparsedRunStart = i; }
|
||||
i += String.fromCodePoint(codePoint).length;
|
||||
continue;
|
||||
}
|
||||
// A reservation only outranks generic matches that would cut into it.
|
||||
// Look the position up unrestricted first: a generic word that starts
|
||||
// earlier and covers the whole name span (写真 over a character named
|
||||
// 真) is the better reading, so the reservation yields rather than
|
||||
// splitting the word. Only a match that ends inside a name span gets
|
||||
// re-run against a window capped at that span.
|
||||
let attempt = await resolveTokenAt(i, scanLength);
|
||||
if (attempt.token) {
|
||||
const splitNameToken = findSplitNameToken(nameIndex, attempt.token.endPos);
|
||||
if (splitNameToken) {
|
||||
attempt = await resolveTokenAt(i, splitNameToken.startPos - i);
|
||||
}
|
||||
}
|
||||
if (attempt.token) {
|
||||
flushUnparsedRun(unparsedRunStart, i);
|
||||
unparsedRunStart = null;
|
||||
tokens.push(attempt.token);
|
||||
i += attempt.matchedLength;
|
||||
continue;
|
||||
}
|
||||
if (unparsedRunStart === null) { unparsedRunStart = i; }
|
||||
i += String.fromCodePoint(text.codePointAt(i)).length;
|
||||
}
|
||||
flushUnparsedRun(unparsedRunStart, text.length);
|
||||
if (blindRetryBudgetExhausted) {
|
||||
// A position gave up with shorter windows still worth trying. The walk
|
||||
// is the only tokenizer now, so stopping there would leave a real term
|
||||
// as an unparsed run; report it so the host can spend one parseText on
|
||||
// the line instead of letting the ladder run to O(scanLength) lookups.
|
||||
return { tokens, retryBudgetExhausted: true };
|
||||
}
|
||||
return tokens;
|
||||
};
|
||||
return true;
|
||||
})();
|
||||
`;
|
||||
|
||||
// Installs (or clears) the character-name candidate forms for the current
|
||||
// media. Runs only when the list changes, not per line. Passing null restores
|
||||
// the exhaustive every-position pre-pass.
|
||||
export function buildYomitanScanNameCandidatesScript(
|
||||
nameCandidates: { key: string; forms: string[] } | null,
|
||||
): string {
|
||||
if (!nameCandidates) {
|
||||
return `
|
||||
(() => {
|
||||
if (typeof globalThis.__subminerYomitanScanSetNameCandidates !== "function") {
|
||||
return false;
|
||||
}
|
||||
return globalThis.__subminerYomitanScanSetNameCandidates(null, null);
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
return `
|
||||
(() => {
|
||||
if (typeof globalThis.__subminerYomitanScanSetNameCandidates !== "function") {
|
||||
return false;
|
||||
}
|
||||
return globalThis.__subminerYomitanScanSetNameCandidates(
|
||||
${JSON.stringify(nameCandidates.key)},
|
||||
${JSON.stringify(nameCandidates.forms)}
|
||||
);
|
||||
})();
|
||||
`;
|
||||
}
|
||||
|
||||
export function buildYomitanScanCallScript(params: YomitanScanRequestParams): string {
|
||||
return `
|
||||
(async () => {
|
||||
if (typeof globalThis.__subminerYomitanScan !== "function") {
|
||||
return ${JSON.stringify(YOMITAN_SCAN_RUNTIME_MISSING_SENTINEL)};
|
||||
}
|
||||
return await globalThis.__subminerYomitanScan(${JSON.stringify(params)});
|
||||
})();
|
||||
`;
|
||||
}
|
||||
@@ -0,0 +1,496 @@
|
||||
// Helper bundle for the in-page Yomitan scan runtime: kana/furigana handling,
|
||||
// headword preference, and frequency-rank resolution. Injected as text into the
|
||||
// parser window by yomitan-scan-runtime-script.ts, so it is data here, not code
|
||||
// this process runs.
|
||||
import { HAN_CODE_POINT_RANGES } from '../../text/han-code-points';
|
||||
|
||||
export const CHARACTER_DICTIONARY_TITLE_PREFIX = 'SubMiner Character Dictionary';
|
||||
|
||||
export const YOMITAN_SCANNING_HELPERS = String.raw`
|
||||
const HIRAGANA_CONVERSION_RANGE = [0x3041, 0x3096];
|
||||
const KATAKANA_CONVERSION_RANGE = [0x30a1, 0x30f6];
|
||||
const KANA_PROLONGED_SOUND_MARK_CODE_POINT = 0x30fc;
|
||||
const KATAKANA_SMALL_KA_CODE_POINT = 0x30f5;
|
||||
const KATAKANA_SMALL_KE_CODE_POINT = 0x30f6;
|
||||
const KANA_RANGES = [[0x3040, 0x309f], [0x30a0, 0x30ff]];
|
||||
// Han ranges come from the shared table so the scan walk and the character
|
||||
// dictionary agree on what a kanji is (supplementary planes included).
|
||||
// Halfwidth katakana counts as Japanese text: a name written that way has
|
||||
// to reach the greedy pre-pass, which has its own handling for it.
|
||||
const JAPANESE_RANGES = [[0x3040, 0x30ff], [0xff66, 0xff9f], ...${JSON.stringify(HAN_CODE_POINT_RANGES)}];
|
||||
function isCodePointInRange(codePoint, range) { return codePoint >= range[0] && codePoint <= range[1]; }
|
||||
function isCodePointInRanges(codePoint, ranges) { return ranges.some((range) => isCodePointInRange(codePoint, range)); }
|
||||
function isCodePointKana(codePoint) { return isCodePointInRanges(codePoint, KANA_RANGES); }
|
||||
function isCodePointJapanese(codePoint) { return isCodePointInRanges(codePoint, JAPANESE_RANGES); }
|
||||
function createFuriganaSegment(text, reading) { return {text, reading}; }
|
||||
function getSegmentReadingContribution(segment) {
|
||||
if (typeof segment.reading === "string" && segment.reading.length > 0) { return segment.reading; }
|
||||
const segmentText = typeof segment.text === "string" ? segment.text : "";
|
||||
const isKanaOnly = segmentText.length > 0 && [...segmentText].every((char) => isCodePointKana(char.codePointAt(0)));
|
||||
return isKanaOnly ? segmentText : "";
|
||||
}
|
||||
function getProlongedHiragana(previousCharacter) {
|
||||
switch (previousCharacter) {
|
||||
case "あ": case "か": case "が": case "さ": case "ざ": case "た": case "だ": case "な": case "は": case "ば": case "ぱ": case "ま": case "や": case "ら": case "わ": case "ぁ": case "ゃ": case "ゎ": return "あ";
|
||||
case "い": case "き": case "ぎ": case "し": case "じ": case "ち": case "ぢ": case "に": case "ひ": case "び": case "ぴ": case "み": case "り": case "ぃ": return "い";
|
||||
case "う": case "く": case "ぐ": case "す": case "ず": case "つ": case "づ": case "ぬ": case "ふ": case "ぶ": case "ぷ": case "む": case "ゆ": case "る": case "ぅ": case "ゅ": return "う";
|
||||
case "え": case "け": case "げ": case "せ": case "ぜ": case "て": case "で": case "ね": case "へ": case "べ": case "ぺ": case "め": case "れ": case "ぇ": return "え";
|
||||
case "お": case "こ": case "ご": case "そ": case "ぞ": case "と": case "ど": case "の": case "ほ": case "ぼ": case "ぽ": case "も": case "よ": case "ろ": case "を": case "ぉ": case "ょ": return "う";
|
||||
default: return null;
|
||||
}
|
||||
}
|
||||
function getFuriganaKanaSegments(text, reading) {
|
||||
const newSegments = [];
|
||||
let start = 0;
|
||||
let state = (reading[0] === text[0]);
|
||||
for (let i = 1; i < text.length; ++i) {
|
||||
const newState = (reading[i] === text[i]);
|
||||
if (state === newState) { continue; }
|
||||
newSegments.push(createFuriganaSegment(text.substring(start, i), state ? '' : reading.substring(start, i)));
|
||||
state = newState;
|
||||
start = i;
|
||||
}
|
||||
newSegments.push(createFuriganaSegment(text.substring(start), state ? '' : reading.substring(start)));
|
||||
return newSegments;
|
||||
}
|
||||
function convertKatakanaToHiragana(text, keepProlongedSoundMarks = false) {
|
||||
let result = '';
|
||||
const offset = (HIRAGANA_CONVERSION_RANGE[0] - KATAKANA_CONVERSION_RANGE[0]);
|
||||
for (let char of text) {
|
||||
const codePoint = char.codePointAt(0);
|
||||
switch (codePoint) {
|
||||
case KATAKANA_SMALL_KA_CODE_POINT:
|
||||
case KATAKANA_SMALL_KE_CODE_POINT:
|
||||
break;
|
||||
case KANA_PROLONGED_SOUND_MARK_CODE_POINT:
|
||||
if (!keepProlongedSoundMarks && result.length > 0) {
|
||||
const char2 = getProlongedHiragana(result[result.length - 1]);
|
||||
if (char2 !== null) { char = char2; }
|
||||
}
|
||||
break;
|
||||
default:
|
||||
if (isCodePointInRange(codePoint, KATAKANA_CONVERSION_RANGE)) {
|
||||
char = String.fromCodePoint(codePoint + offset);
|
||||
}
|
||||
break;
|
||||
}
|
||||
result += char;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function segmentizeFurigana(reading, readingNormalized, groups, groupsStart) {
|
||||
const groupCount = groups.length - groupsStart;
|
||||
if (groupCount <= 0) { return reading.length === 0 ? [] : null; }
|
||||
const group = groups[groupsStart];
|
||||
const {isKana, text} = group;
|
||||
if (isKana) {
|
||||
if (group.textNormalized !== null && readingNormalized.startsWith(group.textNormalized)) {
|
||||
const segments = segmentizeFurigana(reading.substring(text.length), readingNormalized.substring(text.length), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (reading.startsWith(text)) { segments.unshift(createFuriganaSegment(text, '')); }
|
||||
else { segments.unshift(...getFuriganaKanaSegments(text, reading)); }
|
||||
return segments;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
let result = null;
|
||||
for (let i = reading.length; i >= text.length; --i) {
|
||||
const segments = segmentizeFurigana(reading.substring(i), readingNormalized.substring(i), groups, groupsStart + 1);
|
||||
if (segments !== null) {
|
||||
if (result !== null) { return null; }
|
||||
segments.unshift(createFuriganaSegment(text, reading.substring(0, i)));
|
||||
result = segments;
|
||||
}
|
||||
if (groupCount === 1) { break; }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function distributeFurigana(term, reading) {
|
||||
if (reading === term) { return [createFuriganaSegment(term, '')]; }
|
||||
const groups = [];
|
||||
let groupPre = null;
|
||||
let isKanaPre = null;
|
||||
for (const c of term) {
|
||||
const isKana = isCodePointKana(c.codePointAt(0));
|
||||
if (isKana === isKanaPre) { groupPre.text += c; }
|
||||
else {
|
||||
groupPre = {isKana, text: c, textNormalized: null};
|
||||
groups.push(groupPre);
|
||||
isKanaPre = isKana;
|
||||
}
|
||||
}
|
||||
for (const group of groups) {
|
||||
if (group.isKana) { group.textNormalized = convertKatakanaToHiragana(group.text); }
|
||||
}
|
||||
const segments = segmentizeFurigana(reading, convertKatakanaToHiragana(reading), groups, 0);
|
||||
return segments !== null ? segments : [createFuriganaSegment(term, reading)];
|
||||
}
|
||||
function getStemLength(text1, text2) {
|
||||
const minLength = Math.min(text1.length, text2.length);
|
||||
if (minLength === 0) { return 0; }
|
||||
let i = 0;
|
||||
while (true) {
|
||||
const char1 = text1.codePointAt(i);
|
||||
const char2 = text2.codePointAt(i);
|
||||
if (char1 !== char2) { break; }
|
||||
const charLength = String.fromCodePoint(char1).length;
|
||||
i += charLength;
|
||||
if (i >= minLength) {
|
||||
if (i > minLength) { i -= charLength; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
return i;
|
||||
}
|
||||
function distributeFuriganaInflected(term, reading, source) {
|
||||
const termNormalized = convertKatakanaToHiragana(term);
|
||||
const readingNormalized = convertKatakanaToHiragana(reading);
|
||||
const sourceNormalized = convertKatakanaToHiragana(source);
|
||||
let mainText = term;
|
||||
let stemLength = getStemLength(termNormalized, sourceNormalized);
|
||||
const readingStemLength = getStemLength(readingNormalized, sourceNormalized);
|
||||
if (readingStemLength > 0 && readingStemLength >= stemLength) {
|
||||
mainText = reading;
|
||||
stemLength = readingStemLength;
|
||||
reading = source.substring(0, stemLength) + reading.substring(stemLength);
|
||||
}
|
||||
const segments = [];
|
||||
if (stemLength > 0) {
|
||||
mainText = source.substring(0, stemLength) + mainText.substring(stemLength);
|
||||
const segments2 = distributeFurigana(mainText, reading);
|
||||
let consumed = 0;
|
||||
for (const segment of segments2) {
|
||||
const start = consumed;
|
||||
consumed += segment.text.length;
|
||||
if (consumed < stemLength) { segments.push(segment); }
|
||||
else if (consumed === stemLength) { segments.push(segment); break; }
|
||||
else {
|
||||
if (start < stemLength) { segments.push(createFuriganaSegment(mainText.substring(start, stemLength), '')); }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (stemLength < source.length) {
|
||||
const remainder = source.substring(stemLength);
|
||||
const last = segments[segments.length - 1];
|
||||
if (last && last.reading.length === 0) { last.text += remainder; }
|
||||
else { segments.push(createFuriganaSegment(remainder, '')); }
|
||||
}
|
||||
return segments;
|
||||
}
|
||||
function parsePositiveFrequencyNumber(value) {
|
||||
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||
return Math.max(1, Math.floor(value));
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numericMatch = value.trim().match(/[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?/)?.[0];
|
||||
if (!numericMatch) { return null; }
|
||||
const parsed = Number.parseFloat(numericMatch);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) { return null; }
|
||||
return Math.max(1, Math.floor(parsed));
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) {
|
||||
const parsed = parsePositiveFrequencyNumber(item);
|
||||
if (parsed !== null) { return parsed; }
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseDisplayFrequencyNumber(value) {
|
||||
if (typeof value === 'string') {
|
||||
const leadingDigits = value.trim().match(/^\d+/)?.[0];
|
||||
if (!leadingDigits) { return null; }
|
||||
const parsed = Number.parseInt(leadingDigits, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
|
||||
}
|
||||
return parsePositiveFrequencyNumber(value);
|
||||
}
|
||||
function getFrequencyDictionaryName(frequency) {
|
||||
const candidates = [
|
||||
frequency?.dictionary,
|
||||
frequency?.dictionaryName,
|
||||
frequency?.name,
|
||||
frequency?.title,
|
||||
frequency?.dictionaryTitle,
|
||||
frequency?.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
return candidate.trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function getBestFrequencyRank(dictionaryEntry, headwordIndex, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
const headwordCount = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords.length : 0;
|
||||
for (const frequency of dictionaryEntry?.frequencies || []) {
|
||||
if (!frequency || typeof frequency !== 'object') { continue; }
|
||||
const frequencyHeadwordIndex = frequency.headwordIndex;
|
||||
if (typeof frequencyHeadwordIndex === 'number') {
|
||||
if (frequencyHeadwordIndex !== headwordIndex) { continue; }
|
||||
} else if (headwordCount > 1) {
|
||||
continue;
|
||||
}
|
||||
const dictionary = getFrequencyDictionaryName(frequency);
|
||||
if (!dictionary) { continue; }
|
||||
if (dictionaryFrequencyModeByName[dictionary] === 'occurrence-based') { continue; }
|
||||
const rank =
|
||||
parseDisplayFrequencyNumber(frequency.displayValue) ??
|
||||
parsePositiveFrequencyNumber(frequency.frequency);
|
||||
if (rank === null) { continue; }
|
||||
const priorityRaw = dictionaryPriorityByName[dictionary];
|
||||
const fallbackPriority =
|
||||
typeof frequency.dictionaryIndex === 'number' && Number.isFinite(frequency.dictionaryIndex)
|
||||
? Math.max(0, Math.floor(frequency.dictionaryIndex))
|
||||
: Number.MAX_SAFE_INTEGER;
|
||||
const priority =
|
||||
typeof priorityRaw === 'number' && Number.isFinite(priorityRaw)
|
||||
? Math.max(0, Math.floor(priorityRaw))
|
||||
: fallbackPriority;
|
||||
if (best === null || priority < best.priority || (priority === best.priority && rank < best.rank)) {
|
||||
best = { priority, rank };
|
||||
}
|
||||
}
|
||||
return best?.rank ?? null;
|
||||
}
|
||||
function hasExactSource(headword, token, requirePrimary) {
|
||||
for (const src of headword.sources || []) {
|
||||
if (src.originalText !== token) { continue; }
|
||||
if (requirePrimary && !src.isPrimary) { continue; }
|
||||
if (src.matchType !== 'exact') { continue; }
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function collectExactHeadwordMatches(dictionaryEntries, token, requirePrimary) {
|
||||
const matches = [];
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
if (!hasExactSource(headword, token, requirePrimary)) { continue; }
|
||||
matches.push({ dictionaryEntry, headword, headwordIndex });
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
function sameHeadword(match, preferredMatch) {
|
||||
if (!match || !preferredMatch) {
|
||||
return false;
|
||||
}
|
||||
if (match.headword?.term !== preferredMatch.headword?.term) {
|
||||
return false;
|
||||
}
|
||||
const matchReading = typeof match.headword?.reading === 'string' ? match.headword.reading : '';
|
||||
const preferredReading =
|
||||
typeof preferredMatch.headword?.reading === 'string' ? preferredMatch.headword.reading : '';
|
||||
if (!matchReading || !preferredReading) {
|
||||
return true;
|
||||
}
|
||||
return matchReading === preferredReading;
|
||||
}
|
||||
function getBestFrequencyRankForMatches(matches, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
let best = null;
|
||||
for (const match of matches) {
|
||||
const rank = getBestFrequencyRank(
|
||||
match.dictionaryEntry,
|
||||
match.headwordIndex,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
);
|
||||
if (rank === null) { continue; }
|
||||
if (best === null || rank < best) {
|
||||
best = rank;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function normalizeWordClasses(headword) {
|
||||
if (!Array.isArray(headword?.wordClasses)) { return undefined; }
|
||||
const classes = headword.wordClasses.filter((wordClass) => typeof wordClass === "string" && wordClass.trim().length > 0);
|
||||
return classes.length > 0 ? classes : undefined;
|
||||
}
|
||||
function appendDictionaryNames(target, value) {
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
const candidates = [
|
||||
value.dictionary,
|
||||
value.dictionaryName,
|
||||
value.name,
|
||||
value.title,
|
||||
value.dictionaryTitle,
|
||||
value.dictionaryAlias
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim().length > 0) {
|
||||
target.push(candidate.trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
function getDictionaryEntryNames(entry) {
|
||||
const names = [];
|
||||
appendDictionaryNames(names, entry);
|
||||
for (const definition of entry?.definitions || []) {
|
||||
appendDictionaryNames(names, definition);
|
||||
}
|
||||
for (const frequency of entry?.frequencies || []) {
|
||||
appendDictionaryNames(names, frequency);
|
||||
}
|
||||
for (const pronunciation of entry?.pronunciations || []) {
|
||||
appendDictionaryNames(names, pronunciation);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
function isNameDictionaryEntry(entry) {
|
||||
if (!includeNameMatchMetadata || !entry || typeof entry !== 'object') {
|
||||
return false;
|
||||
}
|
||||
return getDictionaryEntryNames(entry).some((name) => name.startsWith(${JSON.stringify(CHARACTER_DICTIONARY_TITLE_PREFIX)}));
|
||||
}
|
||||
function parseSubMinerMediaIdFromString(value) {
|
||||
const imageMatch = value.match(/\bimg\/m(\d+)-/i);
|
||||
if (imageMatch) {
|
||||
const parsed = Number.parseInt(imageMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
const titleMatch = value.match(/${CHARACTER_DICTIONARY_TITLE_PREFIX}[^\d]*(?:AniList\s*)?(\d+)/i);
|
||||
if (titleMatch) {
|
||||
const parsed = Number.parseInt(titleMatch[1], 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function parseSubMinerMediaIdCandidate(value) {
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value > 0) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string' && /^\d+$/.test(value.trim())) {
|
||||
const parsed = Number.parseInt(value.trim(), 10);
|
||||
if (Number.isSafeInteger(parsed) && parsed > 0) { return parsed; }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
function collectSubMinerMediaIds(value, target) {
|
||||
if (typeof value === 'string') {
|
||||
const parsed = parseSubMinerMediaIdFromString(value);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') {
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) { collectSubMinerMediaIds(item, target); }
|
||||
return;
|
||||
}
|
||||
const mediaIdCandidates = [
|
||||
value.subminerMediaId,
|
||||
value.subMinerMediaId,
|
||||
value.characterDictionaryMediaId,
|
||||
value.data?.subminerMediaId,
|
||||
value.data?.subMinerMediaId,
|
||||
value.data?.characterDictionaryMediaId
|
||||
];
|
||||
for (const candidate of mediaIdCandidates) {
|
||||
const parsed = parseSubMinerMediaIdCandidate(candidate);
|
||||
if (parsed !== null) { target.add(parsed); }
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
collectSubMinerMediaIds(child, target);
|
||||
}
|
||||
}
|
||||
function getSubMinerMediaIds(entry) {
|
||||
const mediaIds = new Set();
|
||||
collectSubMinerMediaIds(entry, mediaIds);
|
||||
return mediaIds;
|
||||
}
|
||||
function isCurrentMediaNameDictionaryEntry(entry) {
|
||||
if (!isNameDictionaryEntry(entry)) {
|
||||
return false;
|
||||
}
|
||||
if (currentCharacterDictionaryMediaId === null) {
|
||||
return true;
|
||||
}
|
||||
const mediaIds = getSubMinerMediaIds(entry);
|
||||
return mediaIds.size === 0 || mediaIds.has(currentCharacterDictionaryMediaId);
|
||||
}
|
||||
function findLongestNameMatch(dictionaryEntries, textWindow) {
|
||||
let best = null;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (let headwordIndex = 0; headwordIndex < headwords.length; headwordIndex += 1) {
|
||||
const headword = headwords[headwordIndex];
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (best === null || originalText.length > best.sourceLength) {
|
||||
best = { dictionaryEntry, headword, headwordIndex, sourceLength: originalText.length };
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function findLongestGenericMatchLength(dictionaryEntries, textWindow) {
|
||||
let best = 0;
|
||||
for (const dictionaryEntry of dictionaryEntries || []) {
|
||||
if (isNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
const headwords = Array.isArray(dictionaryEntry?.headwords) ? dictionaryEntry.headwords : [];
|
||||
for (const headword of headwords) {
|
||||
for (const src of headword?.sources || []) {
|
||||
if (src.matchType !== 'exact' || src.isPrimary !== true) { continue; }
|
||||
const originalText = typeof src.originalText === 'string' ? src.originalText : '';
|
||||
if (!originalText || !textWindow.startsWith(originalText)) { continue; }
|
||||
if (originalText.length > best) { best = originalText.length; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function getPreferredHeadword(dictionaryEntries, token, dictionaryPriorityByName, dictionaryFrequencyModeByName) {
|
||||
const currentMediaDictionaryEntries =
|
||||
currentCharacterDictionaryMediaId === null
|
||||
? (dictionaryEntries || [])
|
||||
: (dictionaryEntries || []).filter((entry) => {
|
||||
if (!isNameDictionaryEntry(entry)) { return true; }
|
||||
return isCurrentMediaNameDictionaryEntry(entry);
|
||||
});
|
||||
const exactPrimaryMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, true);
|
||||
let matchedNameDictionary = false;
|
||||
if (includeNameMatchMetadata) {
|
||||
for (const dictionaryEntry of currentMediaDictionaryEntries || []) {
|
||||
if (!isCurrentMediaNameDictionaryEntry(dictionaryEntry)) { continue; }
|
||||
for (const match of exactPrimaryMatches) {
|
||||
if (match.dictionaryEntry !== dictionaryEntry) { continue; }
|
||||
matchedNameDictionary = true;
|
||||
break;
|
||||
}
|
||||
if (matchedNameDictionary) { break; }
|
||||
}
|
||||
}
|
||||
const preferredMatch = exactPrimaryMatches[0];
|
||||
if (preferredMatch) {
|
||||
const exactFrequencyMatches = collectExactHeadwordMatches(currentMediaDictionaryEntries, token, false)
|
||||
.filter((match) => sameHeadword(match, preferredMatch));
|
||||
return {
|
||||
term: preferredMatch.headword.term,
|
||||
reading: preferredMatch.headword.reading,
|
||||
wordClasses: normalizeWordClasses(preferredMatch.headword),
|
||||
isNameMatch:
|
||||
matchedNameDictionary || isCurrentMediaNameDictionaryEntry(preferredMatch.dictionaryEntry),
|
||||
frequencyRank: getBestFrequencyRankForMatches(
|
||||
exactFrequencyMatches.length > 0 ? exactFrequencyMatches : exactPrimaryMatches,
|
||||
dictionaryPriorityByName,
|
||||
dictionaryFrequencyModeByName
|
||||
)
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
`;
|
||||
@@ -6,7 +6,7 @@ import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import type { YoutubeMediaCacheMode } from '../../../types/integrations';
|
||||
import { getYoutubeYtDlpCommand } from './ytdlp-command';
|
||||
import { getYoutubeYtDlpCommand, YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
|
||||
type MediaCacheSessionState = 'running' | 'ready' | 'failed';
|
||||
|
||||
@@ -88,7 +88,7 @@ function normalizeMaxHeight(maxHeight: number | undefined): number {
|
||||
|
||||
function createYtDlpArgs(url: string, outputTemplate: string, maxHeight?: number): string[] {
|
||||
return [
|
||||
'--no-playlist',
|
||||
YTDLP_SINGLE_VIDEO_ARG,
|
||||
'--no-warnings',
|
||||
'--force-ipv4',
|
||||
'--retries',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import type { YoutubeVideoMetadata } from '../immersion-tracker/types';
|
||||
import { getYoutubeYtDlpCommand } from './ytdlp-command';
|
||||
import { getYoutubeYtDlpCommand, YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
|
||||
const YOUTUBE_METADATA_PROBE_TIMEOUT_MS = 15_000;
|
||||
|
||||
@@ -85,15 +85,23 @@ function pickChannelThumbnail(thumbnails: YtDlpThumbnail[] | undefined): string
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function probeYoutubeVideoMetadata(
|
||||
targetUrl: string,
|
||||
): Promise<YoutubeVideoMetadata | null> {
|
||||
const { stdout } = await runCapture(getYoutubeYtDlpCommand(), [
|
||||
export function buildYoutubeMetadataProbeArgs(targetUrl: string): string[] {
|
||||
return [
|
||||
YTDLP_SINGLE_VIDEO_ARG,
|
||||
'--dump-single-json',
|
||||
'--no-warnings',
|
||||
'--skip-download',
|
||||
targetUrl,
|
||||
]);
|
||||
];
|
||||
}
|
||||
|
||||
export async function probeYoutubeVideoMetadata(
|
||||
targetUrl: string,
|
||||
): Promise<YoutubeVideoMetadata | null> {
|
||||
const { stdout } = await runCapture(
|
||||
getYoutubeYtDlpCommand(),
|
||||
buildYoutubeMetadataProbeArgs(targetUrl),
|
||||
);
|
||||
let info: YtDlpYoutubeMetadata;
|
||||
try {
|
||||
info = JSON.parse(stdout) as YtDlpYoutubeMetadata;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { getYoutubeYtDlpCommand } from './ytdlp-command';
|
||||
import { getYoutubeYtDlpCommand, YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
|
||||
const YOUTUBE_PLAYBACK_RESOLVE_TIMEOUT_MS = 15_000;
|
||||
const DEFAULT_PLAYBACK_FORMAT = 'b';
|
||||
@@ -85,17 +85,18 @@ function runCapture(
|
||||
});
|
||||
}
|
||||
|
||||
export function buildYoutubePlaybackResolveArgs(targetUrl: string, format: string): string[] {
|
||||
return [YTDLP_SINGLE_VIDEO_ARG, '--get-url', '--no-warnings', '-f', format, targetUrl];
|
||||
}
|
||||
|
||||
export async function resolveYoutubePlaybackUrl(
|
||||
targetUrl: string,
|
||||
format = DEFAULT_PLAYBACK_FORMAT,
|
||||
): Promise<string> {
|
||||
const { stdout } = await runCapture(getYoutubeYtDlpCommand(), [
|
||||
'--get-url',
|
||||
'--no-warnings',
|
||||
'-f',
|
||||
format,
|
||||
targetUrl,
|
||||
]);
|
||||
const { stdout } = await runCapture(
|
||||
getYoutubeYtDlpCommand(),
|
||||
buildYoutubePlaybackResolveArgs(targetUrl, format),
|
||||
);
|
||||
const playbackUrl =
|
||||
stdout
|
||||
.split(/\r?\n/)
|
||||
|
||||
@@ -2,7 +2,7 @@ import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { spawn } from 'node:child_process';
|
||||
import type { YoutubeTrackOption } from './track-probe';
|
||||
import { getYoutubeYtDlpCommand } from './ytdlp-command';
|
||||
import { getYoutubeYtDlpCommand, YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
import {
|
||||
convertYoutubeTimedTextToVtt,
|
||||
isYoutubeTimedTextExtension,
|
||||
@@ -126,14 +126,14 @@ function pickLatestSubtitleFileForLanguage(
|
||||
return candidates[0] ?? null;
|
||||
}
|
||||
|
||||
function buildDownloadArgs(input: {
|
||||
export function buildDownloadArgs(input: {
|
||||
targetUrl: string;
|
||||
outputTemplate: string;
|
||||
sourceLanguages: string[];
|
||||
includeAutoSubs: boolean;
|
||||
includeManualSubs: boolean;
|
||||
}): string[] {
|
||||
const args = ['--skip-download', '--no-warnings'];
|
||||
const args = [YTDLP_SINGLE_VIDEO_ARG, '--skip-download', '--no-warnings'];
|
||||
if (input.includeAutoSubs) {
|
||||
args.push('--write-auto-subs');
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import type { YoutubeTrackOption } from '../../../types';
|
||||
import { formatYoutubeTrackLabel, normalizeYoutubeLangCode, type YoutubeTrackKind } from './labels';
|
||||
import { getYoutubeYtDlpCommand } from './ytdlp-command';
|
||||
import { getYoutubeYtDlpCommand, YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
|
||||
const YOUTUBE_TRACK_PROBE_TIMEOUT_MS = 15_000;
|
||||
|
||||
@@ -111,12 +111,15 @@ function toTracks(entries: Record<string, YtDlpSubtitleEntry> | undefined, kind:
|
||||
|
||||
export type { YoutubeTrackOption };
|
||||
|
||||
export function buildYoutubeTrackProbeArgs(targetUrl: string): string[] {
|
||||
return [YTDLP_SINGLE_VIDEO_ARG, '--dump-single-json', '--no-warnings', targetUrl];
|
||||
}
|
||||
|
||||
export async function probeYoutubeTracks(targetUrl: string): Promise<YoutubeTrackProbeResult> {
|
||||
const { stdout } = await runCapture(getYoutubeYtDlpCommand(), [
|
||||
'--dump-single-json',
|
||||
'--no-warnings',
|
||||
targetUrl,
|
||||
]);
|
||||
const { stdout } = await runCapture(
|
||||
getYoutubeYtDlpCommand(),
|
||||
buildYoutubeTrackProbeArgs(targetUrl),
|
||||
);
|
||||
const trimmedStdout = stdout.trim();
|
||||
if (!trimmedStdout) {
|
||||
throw new Error('yt-dlp returned empty output while probing subtitle tracks');
|
||||
|
||||
@@ -4,6 +4,13 @@ import path from 'node:path';
|
||||
const DEFAULT_YTDLP_COMMAND = 'yt-dlp';
|
||||
const WINDOWS_YTDLP_COMMANDS = ['yt-dlp.cmd', 'yt-dlp.exe', 'yt-dlp'];
|
||||
|
||||
/**
|
||||
* yt-dlp expands `list=`/`index=` URL params into the whole playlist unless told not to, which
|
||||
* makes single-video extraction hang (e.g. a full Watch Later list) until our timeouts fire.
|
||||
* Every yt-dlp invocation targeting one video must include this.
|
||||
*/
|
||||
export const YTDLP_SINGLE_VIDEO_ARG = '--no-playlist';
|
||||
|
||||
function resolveFromPath(commandName: string): string | null {
|
||||
if (!process.env.PATH) {
|
||||
return null;
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { buildYoutubeMetadataProbeArgs } from './metadata-probe';
|
||||
import { buildYoutubePlaybackResolveArgs } from './playback-resolve';
|
||||
import { buildDownloadArgs } from './track-download';
|
||||
import { buildYoutubeTrackProbeArgs } from './track-probe';
|
||||
import { YTDLP_SINGLE_VIDEO_ARG } from './ytdlp-command';
|
||||
|
||||
// Regression guard for issue #179: a `list=`/`index=` URL made yt-dlp enumerate the whole
|
||||
// playlist (e.g. Watch Later) and blow past our 15s timeouts on every single-video call.
|
||||
const PLAYLIST_URL = 'https://www.youtube.com/watch?v=LKfWC6CgFng&list=WL&index=3';
|
||||
|
||||
const cases: Array<{ name: string; args: string[] }> = [
|
||||
{ name: 'track probe', args: buildYoutubeTrackProbeArgs(PLAYLIST_URL) },
|
||||
{ name: 'metadata probe', args: buildYoutubeMetadataProbeArgs(PLAYLIST_URL) },
|
||||
{ name: 'playback resolve', args: buildYoutubePlaybackResolveArgs(PLAYLIST_URL, 'b') },
|
||||
{
|
||||
name: 'subtitle download',
|
||||
args: buildDownloadArgs({
|
||||
targetUrl: PLAYLIST_URL,
|
||||
outputTemplate: '/tmp/out.%(ext)s',
|
||||
sourceLanguages: ['ja'],
|
||||
includeAutoSubs: true,
|
||||
includeManualSubs: false,
|
||||
}),
|
||||
},
|
||||
];
|
||||
|
||||
test('YTDLP_SINGLE_VIDEO_ARG is the yt-dlp flag that disables playlist expansion', () => {
|
||||
assert.equal(YTDLP_SINGLE_VIDEO_ARG, '--no-playlist');
|
||||
});
|
||||
|
||||
for (const { name, args } of cases) {
|
||||
test(`${name} passes --no-playlist for playlist-scoped URLs`, () => {
|
||||
assert.ok(args.includes('--no-playlist'), `${name} args: ${args.join(' ')}`);
|
||||
assert.equal(args.at(-1), PLAYLIST_URL);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { HAN_CODE_POINT_RANGES, HAN_REGEXP_CLASS_BODY, isHanCodePoint } from './han-code-points';
|
||||
|
||||
test('every range boundary is inside the table', () => {
|
||||
for (const [start, end] of HAN_CODE_POINT_RANGES) {
|
||||
for (const codePoint of [start, end]) {
|
||||
assert.ok(isHanCodePoint(codePoint), `expected U+${codePoint.toString(16)} to be Han`);
|
||||
}
|
||||
}
|
||||
|
||||
// Extension J (Unicode 17) and the Compatibility blocks are the ones a
|
||||
// BMP-only table used to miss.
|
||||
assert.ok(isHanCodePoint(0x323b0));
|
||||
assert.ok(isHanCodePoint(0x33479));
|
||||
assert.ok(isHanCodePoint(0xf900));
|
||||
assert.ok(isHanCodePoint(0x2f800));
|
||||
});
|
||||
|
||||
test('no unified ideograph the runtime knows about falls outside the table', () => {
|
||||
// One direction only: a runtime with older Unicode data simply checks fewer
|
||||
// code points, where asserting the reverse would fail on Extension J.
|
||||
const unifiedIdeograph = /\p{Unified_Ideograph}/u;
|
||||
|
||||
for (let codePoint = 0x3000; codePoint <= 0x40000; codePoint += 1) {
|
||||
if (unifiedIdeograph.test(String.fromCodePoint(codePoint))) {
|
||||
assert.ok(
|
||||
isHanCodePoint(codePoint),
|
||||
`expected unified ideograph U+${codePoint.toString(16)} to be in the table`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('code points just outside the table are rejected', () => {
|
||||
for (const codePoint of [0x33ff, 0x4dc0, 0xa000, 0x1f000, 0x3347a]) {
|
||||
assert.equal(
|
||||
isHanCodePoint(codePoint),
|
||||
false,
|
||||
`expected U+${codePoint.toString(16)} not to be Han`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('the regexp class body matches the same code points as the predicate', () => {
|
||||
const classRegExp = new RegExp(`^[${HAN_REGEXP_CLASS_BODY}]$`, 'u');
|
||||
|
||||
for (const codePoint of [0x3400, 0x4e00, 0x9fff, 0xf900, 0x20000, 0x323b0, 0x33479]) {
|
||||
assert.match(String.fromCodePoint(codePoint), classRegExp);
|
||||
}
|
||||
for (const codePoint of [0x3040, 0x30ff, 0x33fa, 0x3347a]) {
|
||||
assert.doesNotMatch(String.fromCodePoint(codePoint), classRegExp);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,31 @@
|
||||
// Single source of truth for "this code point is a Han character", shared by
|
||||
// the main-process character dictionary and the in-page Yomitan scan runtime.
|
||||
// The two used to carry separate range lists, and they drifted: a name written
|
||||
// with a supplementary-plane kanji could enter the generated dictionary while
|
||||
// the scanner's greedy name pre-pass refused to probe the position.
|
||||
//
|
||||
// Ranges rather than \p{Script=Han}: the scan walk tests one code point per
|
||||
// character of every subtitle line, where an integer compare beats building a
|
||||
// string for a regex, and the script is injected as text into a page where a
|
||||
// shared helper cannot be imported.
|
||||
export const HAN_CODE_POINT_RANGES: ReadonlyArray<readonly [number, number]> = [
|
||||
[0x3400, 0x4dbf], // Extension A
|
||||
[0x4e00, 0x9fff], // CJK Unified Ideographs
|
||||
[0xf900, 0xfaff], // Compatibility Ideographs
|
||||
[0x20000, 0x2a6df], // Extension B
|
||||
[0x2a700, 0x2ebef], // Extensions C-F
|
||||
[0x2ebf0, 0x2ee5f], // Extension I
|
||||
[0x2f800, 0x2fa1f], // Compatibility Ideographs Supplement
|
||||
[0x30000, 0x3134f], // Extension G
|
||||
[0x31350, 0x323af], // Extension H
|
||||
[0x323b0, 0x33479], // Extension J (Unicode 17)
|
||||
];
|
||||
|
||||
export function isHanCodePoint(codePoint: number): boolean {
|
||||
return HAN_CODE_POINT_RANGES.some(([start, end]) => codePoint >= start && codePoint <= end);
|
||||
}
|
||||
|
||||
/** The same ranges as a regular expression character class body (needs the `u` flag). */
|
||||
export const HAN_REGEXP_CLASS_BODY = HAN_CODE_POINT_RANGES.map(
|
||||
([start, end]) => `\\u{${start.toString(16)}}-\\u{${end.toString(16)}}`,
|
||||
).join('');
|
||||
+60
-11
@@ -295,6 +295,7 @@ import {
|
||||
importYomitanDictionaryFromZip,
|
||||
initializeOverlayAnkiIntegration as initializeOverlayAnkiIntegrationCore,
|
||||
initializeOverlayRuntime as initializeOverlayRuntimeCore,
|
||||
isSubtitleAnnotationUpgrade,
|
||||
isOverlayWindowContentReady,
|
||||
jellyfinTicksToSecondsRuntime,
|
||||
listJellyfinItemsRuntime,
|
||||
@@ -486,6 +487,7 @@ import { createOverlayVisibilityRuntimeService } from './main/overlay-visibility
|
||||
import { createDiscordPresenceRuntime } from './main/runtime/discord-presence-runtime';
|
||||
import { createCharacterDictionaryRuntimeService } from './main/character-dictionary-runtime';
|
||||
import { createCharacterDictionaryImageLookup } from './main/character-dictionary-runtime/image-lookup';
|
||||
import { createCharacterNameCandidateLookup } from './main/character-dictionary-runtime/name-candidates';
|
||||
import {
|
||||
createCharacterDictionaryAutoSyncRuntimeService,
|
||||
getCharacterDictionaryManagerSnapshot,
|
||||
@@ -1815,8 +1817,10 @@ function withCurrentSubtitleTiming(payload: SubtitleData): SubtitleData {
|
||||
endTime: appState.mpvClient?.currentSubEnd ?? null,
|
||||
};
|
||||
}
|
||||
function emitSubtitlePayload(payload: SubtitleData): void {
|
||||
function emitSubtitlePayload(payload: SubtitleData, options?: { resumePrefetch?: boolean }): void {
|
||||
const timedPayload = withCurrentSubtitleTiming(payload);
|
||||
const currentSubtitleData = appState.currentSubtitleData;
|
||||
const isAnnotationUpgrade = isSubtitleAnnotationUpgrade(currentSubtitleData, timedPayload);
|
||||
const frequencyDictionary = configService.getConfig().subtitleStyle.frequencyDictionary;
|
||||
const frequencyOptions = {
|
||||
enabled: frequencyDictionary.enabled,
|
||||
@@ -1825,11 +1829,19 @@ function emitSubtitlePayload(payload: SubtitleData): void {
|
||||
};
|
||||
appState.currentSubtitleData = timedPayload;
|
||||
overlayManager.broadcastToOverlayWindows('subtitle:set', timedPayload);
|
||||
if (!isAnnotationUpgrade) {
|
||||
subtitleWsService.broadcast(timedPayload, frequencyOptions);
|
||||
}
|
||||
annotationSubtitleWsService.broadcast(timedPayload, frequencyOptions);
|
||||
autoplayReadyGate.maybeSignalPluginAutoplayReady(timedPayload, { forceWhilePaused: true });
|
||||
// resumePrefetch: false marks an emit that is not the end of the work for
|
||||
// this line; prefetch stays paused until the subtitle processing controller
|
||||
// settles so it does not compete with the on-screen line for the single
|
||||
// Yomitan parser window.
|
||||
if (options?.resumePrefetch !== false) {
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
}
|
||||
function getCurrentAutoplaySubtitlePayload(): SubtitleData | null {
|
||||
const payload = appState.currentSubtitleData;
|
||||
if (!payload?.text.trim()) {
|
||||
@@ -1884,7 +1896,17 @@ const buildSubtitleProcessingControllerMainDepsHandler =
|
||||
createBuildSubtitleProcessingControllerMainDepsHandler({
|
||||
tokenizeSubtitle: async (text: string) =>
|
||||
tokenizeSubtitleDeferred ? await tokenizeSubtitleDeferred(text) : { text, tokens: null },
|
||||
emitSubtitle: (payload) => emitSubtitlePayload(payload),
|
||||
// Controller emits never release the prefetch pause: the first emit for an
|
||||
// uncached line is the provisional plain payload, sent before tokenization
|
||||
// starts, so resuming on it would put prefetch back in contention with the
|
||||
// on-screen line for the single parser window.
|
||||
emitSubtitle: (payload) => emitSubtitlePayload(payload, { resumePrefetch: false }),
|
||||
// The pause is released once the controller has no work left, which covers
|
||||
// the runs that end without an emit (suppressed duplicate, failed
|
||||
// tokenization) as well as the ones that deliver a payload.
|
||||
onProcessingSettled: () => {
|
||||
subtitlePrefetchService?.resume();
|
||||
},
|
||||
logDebug: (message) => {
|
||||
logger.debug(`[subtitle-processing] ${message}`);
|
||||
},
|
||||
@@ -1921,7 +1943,7 @@ const autoplaySubtitlePrimingRuntime = createAutoplaySubtitlePrimingRuntime({
|
||||
appState.activeParsedSubtitleMediaPath = mediaPath;
|
||||
},
|
||||
subtitleProcessingController,
|
||||
emitSubtitlePayload: (payload) => emitSubtitlePayload(payload),
|
||||
emitSubtitlePayload: (payload, options) => emitSubtitlePayload(payload, options),
|
||||
getSubtitlePrefetchService: () => subtitlePrefetchService,
|
||||
getLastObservedTimePos: () => lastObservedTimePos,
|
||||
getVisibleOverlayVisible: () => overlayManager.getVisibleOverlayVisible(),
|
||||
@@ -1955,7 +1977,6 @@ const subtitlePrefetchInitController = createSubtitlePrefetchInitController({
|
||||
subtitleProcessingController.preCacheTokenization(text, data);
|
||||
},
|
||||
hasCachedTokenization: (text) => subtitleProcessingController.hasCachedSubtitle(text),
|
||||
isCacheFull: () => subtitleProcessingController.isCacheFull(),
|
||||
logInfo: (message) => logger.info(message),
|
||||
logWarn: (message) => logger.warn(message),
|
||||
onParsedSubtitleCuesChanged: (cues, sourceKey) => {
|
||||
@@ -1982,15 +2003,22 @@ const resolveActiveSubtitleSidebarSourceHandler = createResolveActiveSubtitleSid
|
||||
getFfmpegPath: () => configService.getConfig().subsync.ffmpeg_path.trim() || 'ffmpeg',
|
||||
extractInternalSubtitleTrack: (ffmpegPath, videoPath, track) =>
|
||||
extractInternalSubtitleTrackToTempFile(ffmpegPath, videoPath, track),
|
||||
logDebug: (message) => logger.debug(message),
|
||||
});
|
||||
|
||||
const refreshSubtitlePrefetchFromActiveTrackHandler =
|
||||
createRefreshSubtitlePrefetchFromActiveTrackHandler({
|
||||
getMpvClient: () => appState.mpvClient,
|
||||
getLastObservedTimePos: () => lastObservedTimePos,
|
||||
shouldKeepExistingCuesOnMissingSource: (videoPath) => isYoutubeMediaPath(videoPath),
|
||||
// Remote media has no extractable on-disk track to fall back to, so a transient
|
||||
// resolve miss (sid briefly 'no', a cycle onto an embedded stream track) would
|
||||
// otherwise drop a working cue list for the rest of the episode.
|
||||
shouldKeepExistingCuesOnMissingSource: (videoPath) =>
|
||||
isYoutubeMediaPath(videoPath) || isRemoteMediaPath(videoPath),
|
||||
subtitlePrefetchInitController,
|
||||
resolveActiveSubtitleSidebarSource: (input) => resolveActiveSubtitleSidebarSourceHandler(input),
|
||||
logDebug: (message) => logger.debug(message),
|
||||
logWarn: (message) => logger.warn(message),
|
||||
});
|
||||
|
||||
const subtitlePrefetchRuntime = {
|
||||
@@ -2521,6 +2549,10 @@ const characterDictionaryAutoSyncRuntime = createCharacterDictionaryAutoSyncRunt
|
||||
},
|
||||
{
|
||||
hasParserWindow: () => Boolean(appState.yomitanParserWindow),
|
||||
invalidateCharacterDictionaryLookups: () => {
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
characterNameCandidateLookup.invalidate();
|
||||
},
|
||||
clearParserCaches: () => {
|
||||
if (appState.yomitanParserWindow) {
|
||||
clearYomitanParserCachesForWindow(appState.yomitanParserWindow);
|
||||
@@ -2546,6 +2578,13 @@ const characterDictionaryImageLookup = createCharacterDictionaryImageLookup({
|
||||
getCurrentMediaId: () => characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
});
|
||||
|
||||
// Lets the Yomitan scan runtime skip name lookups at positions where no
|
||||
// character name can start; absent candidates just mean the exhaustive scan.
|
||||
const characterNameCandidateLookup = createCharacterNameCandidateLookup({
|
||||
userDataPath: USER_DATA_PATH,
|
||||
getCurrentMediaId: () => characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
});
|
||||
|
||||
const overlayVisibilityRuntime = createOverlayVisibilityRuntimeService(
|
||||
createBuildOverlayVisibilityRuntimeMainDepsHandler({
|
||||
getMainWindow: () => overlayManager.getMainWindow(),
|
||||
@@ -2994,6 +3033,8 @@ const {
|
||||
streamIndex,
|
||||
delaySeconds,
|
||||
}),
|
||||
initSubtitlePrefetch: (sourcePath) =>
|
||||
subtitlePrefetchRuntime.refreshSubtitleSidebarFromSource(sourcePath),
|
||||
logDebug: (message, error) => {
|
||||
logger.debug(message, error);
|
||||
},
|
||||
@@ -3958,7 +3999,10 @@ const refreshCurrentSubtitleAfterKnownWordUpdate = (): void => {
|
||||
}
|
||||
subtitleProcessingController.invalidateTokenizationCache();
|
||||
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
||||
subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);
|
||||
if (!subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)) {
|
||||
// Idle controller: no settle is coming to release the pause above.
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
};
|
||||
let hasAttemptedImmersionTrackerStartup = false;
|
||||
const ensureImmersionTrackerStarted = (): void => {
|
||||
@@ -4359,13 +4403,20 @@ const {
|
||||
emitSubtitlePayload(payload);
|
||||
},
|
||||
onSubtitleChange: (text) => {
|
||||
// Pause only; restarting the prefetch run here would discard in-flight
|
||||
// tokenization work on every line. Real seeks restart via onTimePosUpdate.
|
||||
subtitlePrefetchService?.pause();
|
||||
subtitlePrefetchService?.onSeek(lastObservedTimePos);
|
||||
subtitleProcessingController.onSubtitleChange(text);
|
||||
if (!subtitleProcessingController.onSubtitleChange(text)) {
|
||||
// Repeat of the current text: the controller is idle, so no settle is
|
||||
// coming to release the pause. Resume now instead of idling prefetch
|
||||
// for the rest of the cue.
|
||||
subtitlePrefetchService?.resume();
|
||||
}
|
||||
},
|
||||
refreshDiscordPresence: () => {
|
||||
discordPresenceRuntime.publishDiscordPresence();
|
||||
},
|
||||
logSubtitleProcessingDebug: (message: string) => logger.debug(message),
|
||||
ensureImmersionTrackerInitialized: () => {
|
||||
ensureImmersionTrackerStarted();
|
||||
},
|
||||
@@ -4604,6 +4655,7 @@ const {
|
||||
getCharacterNameImage: (term) => characterDictionaryImageLookup.get(term),
|
||||
getCurrentCharacterDictionaryMediaId: () =>
|
||||
characterDictionaryAutoSyncRuntime.getCurrentMediaId(),
|
||||
getCharacterNameCandidates: () => characterNameCandidateLookup.get(),
|
||||
getFrequencyDictionaryEnabled: () =>
|
||||
getRuntimeBooleanOption(
|
||||
'subtitle.annotation.frequency',
|
||||
@@ -5658,7 +5710,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager override:', error);
|
||||
}
|
||||
@@ -5689,7 +5740,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager removal:', error);
|
||||
}
|
||||
@@ -5706,7 +5756,6 @@ const { registerIpcRuntimeHandlers } = composeIpcRuntimeHandlers({
|
||||
if (result.ok && result.rebuildRequired) {
|
||||
try {
|
||||
await characterDictionaryAutoSyncRuntime.runSyncNow();
|
||||
characterDictionaryImageLookup.invalidate();
|
||||
} catch (error) {
|
||||
logger.warn('Failed to rebuild character dictionary after manager reorder:', error);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export const ANILIST_GRAPHQL_URL = 'https://graphql.anilist.co';
|
||||
export const ANILIST_REQUEST_DELAY_MS = 2000;
|
||||
export const CHARACTER_IMAGE_DOWNLOAD_DELAY_MS = 250;
|
||||
export const CHARACTER_DICTIONARY_FORMAT_VERSION = 19;
|
||||
export const CHARACTER_DICTIONARY_FORMAT_VERSION = 20;
|
||||
export const CHARACTER_DICTIONARY_MERGED_TITLE = 'SubMiner Character Dictionary';
|
||||
|
||||
export const HONORIFIC_SUFFIXES = [
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import test from 'node:test';
|
||||
import { CHARACTER_DICTIONARY_FORMAT_VERSION } from './constants';
|
||||
import { createCharacterNameCandidateLookup } from './name-candidates';
|
||||
|
||||
function writeSnapshot(outputDir: string, mediaId: number, entries: Array<[string, string]>): void {
|
||||
const snapshotsDir = path.join(outputDir, 'snapshots');
|
||||
fs.mkdirSync(snapshotsDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(snapshotsDir, `anilist-${mediaId}.json`),
|
||||
JSON.stringify({
|
||||
formatVersion: CHARACTER_DICTIONARY_FORMAT_VERSION,
|
||||
mediaId,
|
||||
mediaTitle: `title-${mediaId}`,
|
||||
entryCount: entries.length,
|
||||
updatedAt: 1,
|
||||
termEntries: entries.map(([term, reading]) => [
|
||||
term,
|
||||
reading,
|
||||
'name main',
|
||||
'',
|
||||
100,
|
||||
[],
|
||||
0,
|
||||
'',
|
||||
]),
|
||||
images: [],
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function withTempDir<T>(run: (dir: string) => T): T {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-name-candidates-'));
|
||||
try {
|
||||
return run(dir);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
test('collects terms and readings for the current media', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['湊', 'みなと'],
|
||||
]);
|
||||
writeSnapshot(dir, 2, [['カズマ', 'かずま']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const candidates = lookup.get();
|
||||
|
||||
assert.ok(candidates);
|
||||
assert.deepEqual([...candidates.forms].sort(), ['みなと', 'ミナト', '湊'].sort());
|
||||
// Deduplicated: both entries share the みなと reading.
|
||||
assert.equal(candidates.forms.length, 3);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null without a media scope so the scanner stays exhaustive', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => null,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('returns null for a media with no cached snapshot', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 999,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get(), null);
|
||||
});
|
||||
});
|
||||
|
||||
test('key changes when the snapshot content changes', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
});
|
||||
const first = lookup.get();
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
lookup.invalidate();
|
||||
const second = lookup.get();
|
||||
|
||||
assert.ok(first && second);
|
||||
assert.notEqual(first.key, second.key);
|
||||
assert.equal(second.forms.length, 4);
|
||||
});
|
||||
});
|
||||
|
||||
// The lookup runs once per subtitle line, so it must not stat the snapshot
|
||||
// directory every call. Asserted behaviorally: an unannounced on-disk change is
|
||||
// invisible until the recheck interval elapses, which can only be true if the
|
||||
// filesystem is not consulted per lookup.
|
||||
test('does not re-read the snapshot directory on every lookup', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
|
||||
nowMs += 1000;
|
||||
assert.equal(lookup.get()?.forms.length, 2, 'expected the cached list within the interval');
|
||||
|
||||
nowMs += 10_000;
|
||||
assert.equal(lookup.get()?.forms.length, 4, 'expected a refresh past the interval');
|
||||
});
|
||||
});
|
||||
|
||||
test('invalidate picks up a snapshot change immediately', () => {
|
||||
withTempDir((dir) => {
|
||||
writeSnapshot(dir, 1, [['ミナト', 'みなと']]);
|
||||
let nowMs = 1_000_000;
|
||||
const lookup = createCharacterNameCandidateLookup({
|
||||
outputDir: dir,
|
||||
getCurrentMediaId: () => 1,
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 2);
|
||||
|
||||
writeSnapshot(dir, 1, [
|
||||
['ミナト', 'みなと'],
|
||||
['アクア', 'あくあ'],
|
||||
]);
|
||||
nowMs += 1;
|
||||
lookup.invalidate();
|
||||
|
||||
assert.equal(lookup.get()?.forms.length, 4);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,159 @@
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { readCachedSnapshots } from './cache';
|
||||
import type { CharacterDictionarySnapshot } from './types';
|
||||
|
||||
// Candidate name forms for the greedy name pre-pass in the Yomitan scan
|
||||
// runtime. The scanner otherwise has to ask the backend at every Japanese
|
||||
// position, because a character name can start mid-token; knowing which forms
|
||||
// exist lets it look up only where a name can actually begin.
|
||||
//
|
||||
// A form is any string Yomitan could match a character entry by: the term and
|
||||
// its reading. Both come from the dictionary SubMiner generated, so the pair is
|
||||
// the complete matchable set for an entry. Callers treat a missing list as
|
||||
// "scan every position", so a stale or absent snapshot costs speed, never a
|
||||
// missed name.
|
||||
|
||||
function getSnapshotsDir(outputDir: string): string {
|
||||
return path.join(outputDir, 'snapshots');
|
||||
}
|
||||
|
||||
function collectSnapshotNameForms(snapshot: CharacterDictionarySnapshot): string[] {
|
||||
const forms = new Set<string>();
|
||||
for (const entry of snapshot.termEntries) {
|
||||
const term = typeof entry[0] === 'string' ? entry[0].trim() : '';
|
||||
if (term) {
|
||||
forms.add(term);
|
||||
}
|
||||
const reading = typeof entry[1] === 'string' ? entry[1].trim() : '';
|
||||
if (reading) {
|
||||
forms.add(reading);
|
||||
}
|
||||
}
|
||||
return [...forms];
|
||||
}
|
||||
|
||||
// The signature grows with the size of the dictionary library, and it rides
|
||||
// along in every per-line scan call, so it is folded into a fixed-width digest
|
||||
// first. Collisions only matter against the immediately previous signature (the
|
||||
// runtime compares keys for equality), and FNV-1a over the file list is far
|
||||
// beyond what that needs.
|
||||
function digestSnapshotDirectorySignature(signature: string): string {
|
||||
let hash = 0x811c9dc5;
|
||||
for (let index = 0; index < signature.length; index += 1) {
|
||||
hash ^= signature.charCodeAt(index);
|
||||
hash = Math.imul(hash, 0x01000193);
|
||||
}
|
||||
return (hash >>> 0).toString(36);
|
||||
}
|
||||
|
||||
function getSnapshotDirectorySignature(outputDir: string): string {
|
||||
let entries: fs.Dirent[] = [];
|
||||
try {
|
||||
entries = fs.readdirSync(getSnapshotsDir(outputDir), { withFileTypes: true });
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
|
||||
const parts: string[] = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isFile() || !/^anilist-\d+\.json$/.test(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const stat = fs.statSync(path.join(getSnapshotsDir(outputDir), entry.name));
|
||||
parts.push(`${entry.name}:${stat.mtimeMs}:${stat.size}`);
|
||||
} catch {
|
||||
// Ignore files that disappear during a refresh; the next lookup rebuilds.
|
||||
}
|
||||
}
|
||||
return parts.sort().join('|');
|
||||
}
|
||||
|
||||
export interface CharacterNameCandidateSet {
|
||||
/** Identifies this exact form list, so the scan runtime can cache it. */
|
||||
key: string;
|
||||
forms: string[];
|
||||
}
|
||||
|
||||
// This lookup is consulted once per subtitle line, so it must not stat the
|
||||
// snapshot directory every time. Dictionary writes are rare and always call
|
||||
// invalidate(), which forces the next lookup to re-read; the interval only
|
||||
// bounds staleness from changes made behind our back.
|
||||
const SNAPSHOT_SIGNATURE_RECHECK_INTERVAL_MS = 5000;
|
||||
|
||||
export function createCharacterNameCandidateLookup(deps: {
|
||||
userDataPath?: string;
|
||||
outputDir?: string;
|
||||
getCurrentMediaId?: () => number | null | undefined;
|
||||
now?: () => number;
|
||||
}): {
|
||||
get: (mediaId?: number | null) => CharacterNameCandidateSet | null;
|
||||
invalidate: () => void;
|
||||
} {
|
||||
const outputDir =
|
||||
deps.outputDir ??
|
||||
(deps.userDataPath ? path.join(deps.userDataPath, 'character-dictionaries') : '');
|
||||
const now = deps.now ?? (() => Date.now());
|
||||
let signature: string | null = null;
|
||||
let lastSignatureCheckAtMs = 0;
|
||||
let formsByMediaId = new Map<number, string[]>();
|
||||
|
||||
function refreshIfNeeded(): void {
|
||||
if (!outputDir) {
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
signature = '';
|
||||
return;
|
||||
}
|
||||
const nowMs = now();
|
||||
if (
|
||||
signature !== null &&
|
||||
nowMs - lastSignatureCheckAtMs < SNAPSHOT_SIGNATURE_RECHECK_INTERVAL_MS
|
||||
) {
|
||||
return;
|
||||
}
|
||||
lastSignatureCheckAtMs = nowMs;
|
||||
const nextSignature = getSnapshotDirectorySignature(outputDir);
|
||||
if (nextSignature === signature) {
|
||||
return;
|
||||
}
|
||||
signature = nextSignature;
|
||||
formsByMediaId = new Map<number, string[]>();
|
||||
for (const snapshot of readCachedSnapshots(outputDir)) {
|
||||
const forms = collectSnapshotNameForms(snapshot);
|
||||
if (forms.length > 0) {
|
||||
formsByMediaId.set(snapshot.mediaId, forms);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
get(mediaId?: number | null): CharacterNameCandidateSet | null {
|
||||
refreshIfNeeded();
|
||||
const rawMediaId = mediaId ?? deps.getCurrentMediaId?.() ?? null;
|
||||
const normalizedMediaId =
|
||||
typeof rawMediaId === 'number' && Number.isFinite(rawMediaId) && rawMediaId > 0
|
||||
? Math.floor(rawMediaId)
|
||||
: null;
|
||||
|
||||
// Without a media scope the pre-pass would need every character of every
|
||||
// cached title, which is both slow to match and pointless: report no
|
||||
// candidates so the scanner keeps its exhaustive behavior.
|
||||
if (normalizedMediaId === null) {
|
||||
return null;
|
||||
}
|
||||
const forms = formsByMediaId.get(normalizedMediaId);
|
||||
if (!forms || forms.length === 0) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
key: `${digestSnapshotDirectorySignature(signature ?? '')}:${normalizedMediaId}`,
|
||||
forms,
|
||||
};
|
||||
},
|
||||
invalidate(): void {
|
||||
signature = null;
|
||||
lastSignatureCheckAtMs = 0;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isHanCodePoint } from '../../core/text/han-code-points';
|
||||
import { HONORIFIC_SUFFIXES } from './constants';
|
||||
import type { JapaneseNameParts, NameReadings, ResolvedNameSplits } from './types';
|
||||
|
||||
@@ -26,10 +27,12 @@ export function buildReading(term: string): string {
|
||||
return katakanaToHiragana(compact);
|
||||
}
|
||||
|
||||
// Code points, not code units: a supplementary-plane kanji (𠮷, U+20BB7) is a
|
||||
// surrogate pair, and reading only the high surrogate would classify a real
|
||||
// single-character name as non-kanji and drop it.
|
||||
export function containsKanji(value: string): boolean {
|
||||
for (const char of value) {
|
||||
const code = char.charCodeAt(0);
|
||||
if ((code >= 0x4e00 && code <= 0x9fff) || (code >= 0x3400 && code <= 0x4dbf)) {
|
||||
if (isHanCodePoint(char.codePointAt(0) ?? 0)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,3 +36,75 @@ test('buildNameTerms adds surname honorifics from Japanese localized aliases', (
|
||||
assert.ok(terms.includes('馬渕さん'));
|
||||
assert.ok(!terms.includes('송치'));
|
||||
});
|
||||
|
||||
test('buildNameTerms drops the disambiguator letter of a mob character name', () => {
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: '',
|
||||
lastNameHint: '',
|
||||
fullName: 'Joshi A',
|
||||
nativeName: '女子A',
|
||||
}),
|
||||
);
|
||||
|
||||
// ア would match every あ〜 in the subtitles; the letter is a disambiguator
|
||||
// (Girl A / Girl B), not a name.
|
||||
assert.ok(!terms.includes('ア'));
|
||||
assert.ok(!terms.includes('アさん'));
|
||||
assert.ok(terms.includes('女子A'));
|
||||
assert.ok(terms.includes('ジョシア'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a character whose whole name is one kana', () => {
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: '',
|
||||
lastNameHint: '',
|
||||
fullName: 'A',
|
||||
nativeName: 'あ',
|
||||
}),
|
||||
);
|
||||
|
||||
// The mob-disambiguator filter targets letters split off a longer name; an
|
||||
// explicit one-character name is the character's actual name.
|
||||
assert.ok(terms.includes('あ'));
|
||||
assert.ok(terms.includes('あさん'));
|
||||
// The romanized "A" is still a label, so it contributes neither itself nor
|
||||
// its single-kana alias.
|
||||
assert.ok(!terms.includes('A'));
|
||||
assert.ok(!terms.includes('ア'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a single-kanji name part', () => {
|
||||
// The name is an alias, not the native name, so the parts come from the
|
||||
// space split rather than from the native-name split.
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: 'Sora',
|
||||
lastNameHint: 'Yamada',
|
||||
fullName: 'Sora Yamada',
|
||||
nativeName: '',
|
||||
alternativeNames: ['山田 空'],
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(terms.includes('山田'));
|
||||
assert.ok(terms.includes('空'));
|
||||
});
|
||||
|
||||
test('buildNameTerms keeps a single supplementary-plane kanji name part', () => {
|
||||
// 𠮷 (U+20BB7) is a surrogate pair: a code-unit kanji check reads only the
|
||||
// high surrogate and drops the part as if it were a mob disambiguator.
|
||||
const terms = buildNameTerms(
|
||||
characterRecord({
|
||||
firstNameHint: 'Tsukasa',
|
||||
lastNameHint: 'Yoshi',
|
||||
fullName: 'Tsukasa Yoshi',
|
||||
nativeName: '',
|
||||
alternativeNames: ['𠮷 司'],
|
||||
}),
|
||||
);
|
||||
|
||||
assert.ok(terms.includes('𠮷'));
|
||||
assert.ok(terms.includes('司'));
|
||||
});
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { HAN_REGEXP_CLASS_BODY } from '../../core/text/han-code-points';
|
||||
import { HONORIFIC_SUFFIXES } from './constants';
|
||||
import {
|
||||
addRomanizedKanaAliases,
|
||||
@@ -42,11 +43,34 @@ export function expandRawNameVariants(rawName: string): string[] {
|
||||
return [...variants];
|
||||
}
|
||||
|
||||
// Kana, halfwidth included: one of these can stand alone as a name, where a
|
||||
// latin letter or a digit cannot.
|
||||
const SINGLE_KANA_CHARACTER = /^[\u3040-\u30ff\u31f0-\u31ff\uff66-\uff9f]$/u;
|
||||
|
||||
// AniList disambiguates unnamed mob characters with a trailing letter (女子A /
|
||||
// "Joshi A"), and a lone letter romanizes into a single-kana alias (A → ア)
|
||||
// that collides with interjections (あ〜 matching ア). That letter is a label,
|
||||
// not a name, so it is dropped where a name splits into it and before it can
|
||||
// become a kana alias. A name that is genuinely one character, a character
|
||||
// actually called あ or a single kanji, is a real lookup target and is kept.
|
||||
function isNameDisambiguatorLetter(name: string): boolean {
|
||||
return [...name].length === 1 && !containsKanji(name) && !SINGLE_KANA_CHARACTER.test(name);
|
||||
}
|
||||
|
||||
function isUsableNameTerm(name: string): boolean {
|
||||
return !isNameDisambiguatorLetter(name);
|
||||
}
|
||||
|
||||
// Kana, Han (shared ranges), and the marks that only ever appear inside a
|
||||
// Japanese name: iteration marks and the small ka/ke used in place names.
|
||||
const JAPANESE_NAME_CHARACTERS = new RegExp(
|
||||
`^[\\u3040-\\u30ff${HAN_REGEXP_CLASS_BODY}\u3005\u3006\u30f5\u30f6\u30fc]+$`,
|
||||
'u',
|
||||
);
|
||||
|
||||
export function isJapaneseNameSplitCandidate(name: string): boolean {
|
||||
const compact = name.replace(/[\s\u3000・・·•]/g, '');
|
||||
return (
|
||||
containsKanji(compact) && /^[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff々〆ヵヶー]+$/.test(compact)
|
||||
);
|
||||
return containsKanji(compact) && JAPANESE_NAME_CHARACTERS.test(compact);
|
||||
}
|
||||
|
||||
function addJapaneseNameParts(
|
||||
@@ -97,8 +121,11 @@ export function buildNameTerms(
|
||||
|
||||
const split = name.split(/[\s\u3000]+/).filter((part) => part.trim().length > 0);
|
||||
if (split.length === 2) {
|
||||
target.add(split[0]!);
|
||||
target.add(split[1]!);
|
||||
for (const part of split) {
|
||||
if (isUsableNameTerm(part)) {
|
||||
target.add(part);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const splitByMiddleDot = name
|
||||
@@ -107,9 +134,11 @@ export function buildNameTerms(
|
||||
.filter((part) => part.length > 0);
|
||||
if (splitByMiddleDot.length >= 2) {
|
||||
for (const part of splitByMiddleDot) {
|
||||
if (isUsableNameTerm(part)) {
|
||||
target.add(part);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (target === base) {
|
||||
addJapaneseNameParts(character, name, base, resolvedSplits);
|
||||
@@ -117,7 +146,10 @@ export function buildNameTerms(
|
||||
}
|
||||
}
|
||||
|
||||
for (const alias of addRomanizedKanaAliases(romanizedBase)) {
|
||||
// Romanized forms that are a bare letter would become a single-kana alias.
|
||||
for (const alias of addRomanizedKanaAliases(
|
||||
[...romanizedBase].filter((entry) => !isNameDisambiguatorLetter(entry)),
|
||||
)) {
|
||||
base.add(alias);
|
||||
}
|
||||
|
||||
@@ -136,6 +168,9 @@ export function buildNameTerms(
|
||||
|
||||
const withHonorifics = new Set<string>();
|
||||
for (const entry of base) {
|
||||
// Only labels split off a longer name are filtered (see above); an explicit
|
||||
// one-character name reaches this point intact.
|
||||
if (isNameDisambiguatorLetter(entry)) continue;
|
||||
withHonorifics.add(entry);
|
||||
for (const suffix of HONORIFIC_SUFFIXES) {
|
||||
withHonorifics.add(`${entry}${suffix.term}`);
|
||||
|
||||
@@ -176,6 +176,29 @@ test('subtitle sidebar media path tag is assigned after prefetch succeeds', () =
|
||||
);
|
||||
});
|
||||
|
||||
test('remote media keeps parsed cues when the active subtitle source cannot be resolved', () => {
|
||||
const source = readMainSource();
|
||||
const actionBlock = source.match(
|
||||
/createRefreshSubtitlePrefetchFromActiveTrackHandler\(\{(?<body>[\s\S]*?)\n \}\);/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(actionBlock);
|
||||
assert.match(actionBlock, /isYoutubeMediaPath\(videoPath\) \|\| isRemoteMediaPath\(videoPath\)/);
|
||||
});
|
||||
|
||||
test('jellyfin subtitle preload seeds the tokenization prefetch directly', () => {
|
||||
const source = readMainSource();
|
||||
const actionBlock = source.match(
|
||||
/preloadJellyfinExternalSubtitlesMainDeps:\s*\{(?<body>[\s\S]*?)\n \},/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(actionBlock);
|
||||
assert.match(
|
||||
actionBlock,
|
||||
/initSubtitlePrefetch: \(sourcePath\) =>\s*subtitlePrefetchRuntime\.refreshSubtitleSidebarFromSource\(sourcePath\),/,
|
||||
);
|
||||
});
|
||||
|
||||
test('update overlay notification action triggers install flow', () => {
|
||||
const source = readMainSource();
|
||||
const runtimeSource = readSource('src/main/runtime/overlay-notifications-runtime.ts');
|
||||
@@ -200,7 +223,7 @@ test('update overlay notification action triggers install flow', () => {
|
||||
assert.match(runtimeSource, /fallbackClient\.openNoteInBrowser\(noteId\)/);
|
||||
});
|
||||
|
||||
test('subtitle change re-prioritizes prefetch around live playback before tokenizing current line', () => {
|
||||
test('subtitle change pauses prefetch without restarting its run before tokenizing current line', () => {
|
||||
const source = readMainSource();
|
||||
const actionBlock = source.match(
|
||||
/onSubtitleChange:\s*\(text\)\s*=>\s*\{(?<body>[\s\S]*?)\n \},\n refreshDiscordPresence:/,
|
||||
@@ -208,15 +231,19 @@ test('subtitle change re-prioritizes prefetch around live playback before tokeni
|
||||
|
||||
assert.ok(actionBlock);
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.pause\(\);/);
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
||||
assert.match(actionBlock, /subtitleProcessingController\.onSubtitleChange\(text\);/);
|
||||
// Restarting the run per line (onSeek) discards in-flight prefetch work;
|
||||
// only real seeks restart via onTimePosUpdate.
|
||||
assert.doesNotMatch(actionBlock, /subtitlePrefetchService\?\.onSeek\(/);
|
||||
assert.match(actionBlock, /subtitleProcessingController\.onSubtitleChange\(text\)/);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitlePrefetchService?.pause();') <
|
||||
actionBlock.indexOf('subtitlePrefetchService?.onSeek(lastObservedTimePos);'),
|
||||
actionBlock.indexOf('subtitleProcessingController.onSubtitleChange(text)'),
|
||||
);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitlePrefetchService?.onSeek(lastObservedTimePos);') <
|
||||
actionBlock.indexOf('subtitleProcessingController.onSubtitleChange(text);'),
|
||||
// A repeated subtitle emits nothing, so the pause has to be released here or
|
||||
// prefetching idles until the next distinct line.
|
||||
assert.match(
|
||||
actionBlock,
|
||||
/if \(!subtitleProcessingController\.onSubtitleChange\(text\)\) \{[\s\S]*?subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -466,16 +493,35 @@ test('known-word updates invalidate prefetched tokenizations before refreshing c
|
||||
assert.match(actionBlock, /subtitlePrefetchService\?\.onSeek\(lastObservedTimePos\);/);
|
||||
assert.match(
|
||||
actionBlock,
|
||||
/subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\);/,
|
||||
/if \(!subtitleProcessingController\.refreshCurrentSubtitle\(appState\.currentSubText\)\) \{[\s\S]*?subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
assert.ok(
|
||||
actionBlock.indexOf('subtitleProcessingController.invalidateTokenizationCache();') <
|
||||
actionBlock.indexOf(
|
||||
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);',
|
||||
'subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText)',
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
test('subtitle processing controller resumes prefetch on settle, not on its emits', () => {
|
||||
const source = readMainSource();
|
||||
const depsBlock = source.match(
|
||||
/createBuildSubtitleProcessingControllerMainDepsHandler\(\{(?<body>[\s\S]*?)\n \}\);/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(depsBlock);
|
||||
// A controller emit can be the provisional plain payload sent before the
|
||||
// scan runs, so it must not release the prefetch pause.
|
||||
assert.match(
|
||||
depsBlock,
|
||||
/emitSubtitle: \(payload\) => emitSubtitlePayload\(payload, \{ resumePrefetch: false \}\),/,
|
||||
);
|
||||
assert.match(
|
||||
depsBlock,
|
||||
/onProcessingSettled: \(\) => \{\s+subtitlePrefetchService\?\.resume\(\);/,
|
||||
);
|
||||
});
|
||||
|
||||
test('manual visible overlay changes notify mpv plugin visibility state', () => {
|
||||
const source = readMainSource();
|
||||
const setBlock = source.match(
|
||||
@@ -570,7 +616,7 @@ test('YouTube media cache lifecycle routes through configured status notificatio
|
||||
test('subtitle broadcasts share one frequency options snapshot per emitted payload', () => {
|
||||
const source = readMainSource();
|
||||
const emitBlock = source.match(
|
||||
/function emitSubtitlePayload\(payload: SubtitleData\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
/function emitSubtitlePayload\([\s\S]*?\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
)?.groups?.body;
|
||||
const frequencyOptionsSnapshot = emitBlock?.match(
|
||||
/const frequencyDictionary = configService\.getConfig\(\)\.subtitleStyle\.frequencyDictionary;(?<body>[\s\S]*?)\n \};/,
|
||||
@@ -590,6 +636,35 @@ test('subtitle broadcasts share one frequency options snapshot per emitted paylo
|
||||
);
|
||||
});
|
||||
|
||||
test('annotation upgrades skip the duplicate basic websocket event', () => {
|
||||
const source = readMainSource();
|
||||
const emitBlock = source.match(
|
||||
/function emitSubtitlePayload\([\s\S]*?\): void \{(?<body>[\s\S]*?)\n\}/,
|
||||
)?.groups?.body;
|
||||
|
||||
assert.ok(emitBlock);
|
||||
assert.match(
|
||||
emitBlock,
|
||||
/const isAnnotationUpgrade = isSubtitleAnnotationUpgrade\(currentSubtitleData, timedPayload\);/,
|
||||
);
|
||||
assert.match(
|
||||
emitBlock,
|
||||
/if \(!isAnnotationUpgrade\) \{\s+subtitleWsService\.broadcast\(timedPayload, frequencyOptions\);\s+\}/,
|
||||
);
|
||||
assert.equal(
|
||||
(emitBlock.match(/overlayManager\.broadcastToOverlayWindows\('subtitle:set'/g) ?? []).length,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
(
|
||||
emitBlock.match(
|
||||
/annotationSubtitleWsService\.broadcast\(timedPayload, frequencyOptions\)/g,
|
||||
) ?? []
|
||||
).length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('websocket frequency options callbacks each read one configuration snapshot', () => {
|
||||
const source = readMainSource();
|
||||
const subtitleBlock = source.match(
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user