Compare commits

...
Author SHA1 Message Date
sudacode db96659b14 test(release): treat escaped shell separators as literal text
The scanner honoured backslash escapes inside double quotes but not
outside them, so an escaped separator such as find's -exec ... \; still
broke a line into a bare command position.

Consume an unquoted backslash and the character after it before the
comment and separator checks.
2026-08-23 04:31:52 -07:00
sudacode 107c6d1b06 test(release): split shell lines outside quotes and comments
Splitting on separators anywhere in a line broke quoted strings apart, so
echo 'note; bun run changelog:check-prerelease-notes ...' produced a bare
command position and satisfied the ordering assertion.

Track quote state while scanning, stop at an unquoted inline comment, and
keep redirects from being read as pipes. Cover quoted separators, inline
comments, trailing comments, and pipe-vs-redirect.
2026-08-23 04:20:26 -07:00
sudacode 2bfb72a09e test(release): match workflow commands at command positions only
An unanchored pattern matched the command text anywhere on an executable
line, so wrapping it in echo or printf satisfied the ordering assertion
while the workflow ran no validation at all.

Split each line on shell separators, strip control-flow prefixes, and
match anchored patterns against those command positions. Cover the helper
directly with cases for echo, printf, comments, and quoted mentions.
2026-08-23 04:09:08 -07:00
sudacode 94643ffceb test(release): ignore shell comments when matching workflow commands
Matching a step's whole run body meant a commented-out command satisfied
the ordering assertion, so commenting out the prerelease notes check left
the test green while the workflow no longer validated anything.

Match executable lines only, and fold the version argument into the same
match so dropping it also fails.
2026-08-23 03:57:21 -07:00
sudacode 615a785703 test(release): assert workflow injection safety against parsed YAML
The line-based guards allow-listed known-safe spellings, so bracket forms
like ${{ steps.version.outputs['VERSION'] }} never entered the candidate
list and passed, and the ordering check read indexOf over raw text, which
a comment naming the step could satisfy.

Parse the workflows instead and assert on step structure: no template
expression may appear in any run body, every step reading $RELEASE_VERSION
or $TAG_NAME must declare it in env, and the prerelease notes check must
precede the publishing step in the release job's step list.

Verified each guard fails on the three evasions it now covers.
2026-08-23 03:45:29 -07:00
sudacode 7b403bf8ad test(release): cover workflow injection safety and prerelease note check
The AUR test pinned the literal ${{ steps.version.outputs.VERSION }}
interpolation that moved into an env block, so it failed once the tag
value started reaching the shell as $RELEASE_VERSION.

Update that assertion and add guards for the invariant behind the move:
no tag-derived value may be interpolated into a run body, where GitHub
substitutes it before the shell parses the line. Also assert the
prerelease workflow runs the committed-notes check before it creates or
edits the release.

Document that the fragment delta resolves git paths against cwd, which
is both the project root and the repository root.
2026-08-23 03:17:06 -07:00
sudacode 1cbc5f3853 feat(release): track prerelease deltas and validate committed notes
- Compare fragment changes with previous beta/RC tags
- Reject stale prerelease notes in CI
2026-08-23 03:02:09 -07:00
sudacode 0a0aa3ec98 chore(release): prepare v0.19.4-beta.4 2026-08-23 01:23:16 -07:00
sudacode b87cc3cfdd chore(workflow): add release skill 2026-08-23 01:14:03 -07:00
sudacode 8cb3c8c90a fix(macos): pin window helper deployment target to fix older systems
- Build the Swift window-tracking helper with an explicit `-target ...-apple-macos12.0` instead of letting swiftc stamp the build machine's OS, so the overlay attaches to mpv on older macOS (e.g. Ventura) instead of crashing at load
- Fold the standalone build-macos-helper.sh into prepare-build-assets.mjs and update docs/tests to match
2026-08-23 00:58:24 -07:00
sudacode 03ea903927 fix(subtitles): preserve authored spaces in fragmented ASS karaoke
- Only trim indentation before slicing ASS event fields so a trailing authored space at a fragment's end is kept, preventing words from being joined together when fragmented karaoke lines are reconstructed
- Add regression test covering event-boundary word spacing
2026-08-22 23:32:35 -07:00
sudacode 3aea42e6f8 chore(release): prepare v0.19.4-beta.3 2026-08-21 23:25:37 -07:00
sudacode 88bb3edfa4 fix(subtitles): preserve lyric selection and deduplicate secondary text
- Seek sidebar selections past overlapping karaoke exit spans
- Collapse duplicate long ASS lines and omit reconstructed sign grids
2026-08-21 01:28:00 -07:00
sudacode 9445aef004 fix(subtitles): navigate by sanitized cues across ASS animation
- Route subtitle navigation through parsed cue boundaries
- Treat explicit short seeks as subtitle priming events
2026-08-20 00:20:51 -07:00
sudacode c01bcd9d0f fix(subtitles): collapse layered ASS lyrics across startup and seeks
- Reconstruct fragment-only karaoke while preserving canonical animated signs
- Reprocess live subtitles after initial playback and seek updates
- Add a development launcher for playable media files
2026-08-19 22:55:10 -07:00
sudacode 695613b1e7 chore(release): prepare v0.19.4-beta.2 2026-08-19 02:19:40 -07:00
sudacode 20797772f8 fix(character-dictionary): cache completed MeCab refreshes (#212) 2026-08-19 01:49:15 -07:00
sudacode 72c312810a docs: link README demo to GitHub video playback
- Use GitHub-hosted video playback for the README demo
- Document the animated fallback behavior
2026-08-19 01:02:05 -07:00
sudacode b61f050e78 docs: update mining demo and scale video crops
- Refresh mining demo assets and documentation references
- Scale mpv crop coordinates for resized OBS recordings
2026-08-19 00:55:31 -07:00
sudacode 2a77ba9dd4 fix(launcher): restore Matroska thumbnails in Linux rofi picker (#210) 2026-08-18 20:30:58 -07:00
sudacode 4ed878270f fix(stats): accept legacy lexical rollup state schemas 2026-08-18 02:24:49 -07:00
sudacode 1d1b0c7bb6 fix(subtitles): collapse duplicate primary ASS style layers
- Use parsed cues when they fully explain mpv live text
- Preserve unmatched overlapping dialogue and signs
2026-08-18 01:51:19 -07:00
sudacode db61ce358d fix(stats): preserve session rollups across schema upgrades
- Stop deleting imm_daily_rollups/imm_monthly_rollups on unrelated schema version bumps; their source session/telemetry rows may already be pruned, so deleted buckets could not be rebuilt
- Run startup session-rollup maintenance before the lexical rollup backfill takes the write lock, so recovery no longer races playback writes
- Update the vocabulary summary totals changelog fragment to reflect that watch-time, activity, efficiency, and library charts are no longer cleared during rebuilds
2026-08-18 01:12:58 -07:00
sudacode e9778a945a fix(overlay): deduplicate secondary subtitle rendering (#208) 2026-08-18 00:44:02 -07:00
sudacode 7de73e16a1 fix(stats): report complete vocabulary totals and new-word history (#202) 2026-08-18 00:42:46 -07:00
sudacode 273652f781 fix(subtitles): recover canonical lines from ASS animation (#207) 2026-08-18 00:30:48 -07:00
sudacode 14c0086520 chore(stats): reword duplicates popup 2026-08-17 18:42:58 -07:00
sudacode ff0bc0af6d fix(dictionary): prevent freezes and restore AppImage notifications (#205) 2026-08-17 18:41:03 -07:00
sudacode 00b1b79bf4 fix(mpv): recover from stalled IPC connects (#204) 2026-08-16 22:58:28 -07:00
180 changed files with 10030 additions and 693 deletions
+34
View File
@@ -0,0 +1,34 @@
---
name: subminer-release
description: Prepare, cut, publish, or repair SubMiner stable and prerelease releases. Use for hands-on release work; do not use for general release questions.
---
# SubMiner release
Carry out the requested release phase using the repository's current release process.
## Source of truth
Read `docs/RELEASING.md` completely before changing files or release state. Treat it as canonical. Read `changes/README.md` when the work touches change fragments or generated release notes.
Do not copy release commands or policy into this skill. If this skill disagrees with the release guide, follow the guide and reconcile the skill before handoff.
## Workflow
1. Identify whether the request is for a stable release, prerelease, release preparation, publication, or repair.
2. Inspect the current branch, worktree status, package version, pending change fragments, relevant tags, and latest CI state before making changes.
3. Follow the matching procedure in `docs/RELEASING.md` in order. Review generated changelog and release-note Markdown before it can be committed or published.
4. Run every required gate for the requested release phase. Do not treat a cheaper test lane as a substitute for the documented release gate.
5. Before a stable tag, confirm the package and tag versions match and no pending `changes/*.md` fragments remain. Preserve fragments for prereleases as documented.
6. Report the resulting version, completed checks, local commit and tag state, remote publication state, skipped platform checks, and any remaining manual work.
## Authorization boundaries
- A request to prepare a release stops before commit, tag, push, or remote publication unless the user also authorizes those actions.
- A clear request to cut or publish a release includes the documented commit, tag, and push steps. Ask before the first remote mutation when the wording is ambiguous.
- Do not edit an existing GitHub release, publish to the AUR, change secrets, or alter signing configuration unless the user explicitly requests that operation.
- Do not switch branches without consent.
## Stop conditions
Stop and report the blocker when required CI or a release gate fails, authentication is missing, versions disagree, required artifacts are absent, or the worktree contains unexpected changes that overlap the release. Do not tag or publish a partially verified release.
+4 -2
View File
@@ -32,9 +32,11 @@ jobs:
- name: Guard stable docs tag shape
id: tag_guard
if: github.ref_type == 'tag'
env:
TAG_NAME: ${{ github.ref_name }}
run: |
if [[ ! "${{ github.ref_name }}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "::notice::Skipping non-stable docs tag ${{ github.ref_name }}"
if [[ ! "$TAG_NAME" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "::notice::Skipping non-stable docs tag $TAG_NAME"
echo "stable_tag=false" >> "$GITHUB_OUTPUT"
exit 0
fi
+17 -9
View File
@@ -274,7 +274,8 @@ jobs:
config.example.jsonc \
plugin/subminer \
plugin/subminer.conf \
assets/themes/subminer.rasi
assets/themes/subminer.rasi \
assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer
- name: Generate checksums
run: |
@@ -296,15 +297,22 @@ jobs:
run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
- name: Verify committed prerelease notes
env:
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
if [ ! -s release/prerelease-notes.md ]; then
echo "::error::release/prerelease-notes.md is missing or empty. Run 'bun run changelog:prerelease-notes --version <version>' locally and commit the file before tagging."
exit 1
fi
if ! bun run changelog:check-prerelease-notes --version "$RELEASE_VERSION"; then
echo "::error::release/prerelease-notes.md was not generated for $RELEASE_VERSION. Rerun 'bun run changelog:prerelease-notes --version $RELEASE_VERSION' locally, commit, and retag."
exit 1
fi
- name: Publish Prerelease
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
set -euo pipefail
@@ -326,27 +334,27 @@ jobs:
exit 1
fi
if gh release view "${{ steps.version.outputs.VERSION }}" >/dev/null 2>&1; then
gh release edit "${{ steps.version.outputs.VERSION }}" \
if gh release view "$RELEASE_VERSION" >/dev/null 2>&1; then
gh release edit "$RELEASE_VERSION" \
--draft \
--prerelease \
--title "${{ steps.version.outputs.VERSION }}" \
--title "$RELEASE_VERSION" \
--notes-file release/prerelease-notes.md
else
gh release create "${{ steps.version.outputs.VERSION }}" \
gh release create "$RELEASE_VERSION" \
--draft \
--latest=false \
--prerelease \
--title "${{ steps.version.outputs.VERSION }}" \
--title "$RELEASE_VERSION" \
--notes-file release/prerelease-notes.md
fi
for asset in "${artifacts[@]}"; do
gh release upload "${{ steps.version.outputs.VERSION }}" "$asset" --clobber
gh release upload "$RELEASE_VERSION" "$asset" --clobber
done
gh release edit "${{ steps.version.outputs.VERSION }}" \
gh release edit "$RELEASE_VERSION" \
--draft=false \
--prerelease \
--title "${{ steps.version.outputs.VERSION }}" \
--title "$RELEASE_VERSION" \
--notes-file release/prerelease-notes.md
+26 -14
View File
@@ -273,7 +273,8 @@ jobs:
config.example.jsonc \
plugin/subminer \
plugin/subminer.conf \
assets/themes/subminer.rasi
assets/themes/subminer.rasi \
assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer
- name: Generate checksums
run: |
@@ -295,33 +296,40 @@ jobs:
run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
- name: Guard against pending changelog fragments
env:
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
if find changes -maxdepth 1 -name '*.md' -not -name README.md -print -quit | grep -q .; then
echo "::error::Pending changelog fragments detected. Run 'bun run changelog:build --version ${{ steps.version.outputs.VERSION }}' locally and commit the polished CHANGELOG.md before tagging. CI no longer auto-builds the changelog because the polish step requires the local 'claude' CLI."
echo "::error::Pending changelog fragments detected. Run 'bun run changelog:build --version $RELEASE_VERSION' locally and commit the polished CHANGELOG.md before tagging. CI no longer auto-builds the changelog because the polish step requires the local 'claude' CLI."
exit 1
fi
- name: Verify changelog is ready for tagged release
run: bun run changelog:check --version "${{ steps.version.outputs.VERSION }}"
env:
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: bun run changelog:check --version "$RELEASE_VERSION"
- name: Generate release notes from changelog
run: bun run changelog:release-notes --version "${{ steps.version.outputs.VERSION }}"
env:
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: bun run changelog:release-notes --version "$RELEASE_VERSION"
- name: Publish Release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
set -euo pipefail
if gh release view "${{ steps.version.outputs.VERSION }}" >/dev/null 2>&1; then
if gh release view "$RELEASE_VERSION" >/dev/null 2>&1; then
# Do not pass the prerelease flag here; gh defaults to a normal release.
gh release edit "${{ steps.version.outputs.VERSION }}" \
gh release edit "$RELEASE_VERSION" \
--draft=false \
--title "${{ steps.version.outputs.VERSION }}" \
--title "$RELEASE_VERSION" \
--notes-file release/release-notes.md
else
gh release create "${{ steps.version.outputs.VERSION }}" \
--title "${{ steps.version.outputs.VERSION }}" \
gh release create "$RELEASE_VERSION" \
--title "$RELEASE_VERSION" \
--notes-file release/release-notes.md
fi
@@ -344,7 +352,7 @@ jobs:
fi
for asset in "${artifacts[@]}"; do
gh release upload "${{ steps.version.outputs.VERSION }}" "$asset" --clobber
gh release upload "$RELEASE_VERSION" "$asset" --clobber
done
aur-publish:
@@ -420,9 +428,10 @@ jobs:
if: steps.aur_prereqs.outputs.skip != 'true' && steps.aur_ssh.outputs.skip != 'true' && steps.aur_clone.outputs.skip != 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
set -euo pipefail
version="${{ steps.version.outputs.VERSION }}"
version="$RELEASE_VERSION"
install -dm755 .tmp/aur-release-assets
gh release download "$version" \
--dir .tmp/aur-release-assets \
@@ -432,15 +441,17 @@ jobs:
- name: Update AUR packaging metadata
if: steps.aur_prereqs.outputs.skip != 'true' && steps.aur_ssh.outputs.skip != 'true' && steps.aur_clone.outputs.skip != 'true'
env:
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
set -euo pipefail
version_no_v="${{ steps.version.outputs.VERSION }}"
version_no_v="$RELEASE_VERSION"
version_no_v="${version_no_v#v}"
cp packaging/aur/subminer-bin/PKGBUILD aur-subminer-bin/PKGBUILD
cp packaging/aur/subminer-bin/.SRCINFO aur-subminer-bin/.SRCINFO
bash scripts/update-aur-package.sh \
--pkg-dir aur-subminer-bin \
--version "${{ steps.version.outputs.VERSION }}" \
--version "$RELEASE_VERSION" \
--appimage ".tmp/aur-release-assets/SubMiner-${version_no_v}.AppImage" \
--wrapper ".tmp/aur-release-assets/subminer" \
--assets ".tmp/aur-release-assets/subminer-assets.tar.gz"
@@ -450,6 +461,7 @@ jobs:
working-directory: aur-subminer-bin
env:
GIT_SSH_COMMAND: ssh -i ~/.ssh/aur -o IdentitiesOnly=yes
RELEASE_VERSION: ${{ steps.version.outputs.VERSION }}
run: |
set -euo pipefail
if git diff --quiet -- PKGBUILD .SRCINFO; then
@@ -459,7 +471,7 @@ jobs:
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add PKGBUILD .SRCINFO
git commit -m "Update to ${{ steps.version.outputs.VERSION }}"
git commit -m "Update to $RELEASE_VERSION"
attempts=3
for attempt in $(seq 1 "$attempts"); do
+3
View File
@@ -49,6 +49,7 @@ tests/*
!.agents/skills/
.agents/skills/*
!.agents/skills/subminer-change-verification/
!.agents/skills/subminer-release/
!.agents/skills/subminer-scrum-master/
.agents/skills/subminer-change-verification/*
!.agents/skills/subminer-change-verification/SKILL.md
@@ -56,6 +57,8 @@ tests/*
.agents/skills/subminer-change-verification/scripts/*
!.agents/skills/subminer-change-verification/scripts/classify_subminer_diff.sh
!.agents/skills/subminer-change-verification/scripts/verify_subminer_change.sh
.agents/skills/subminer-release/*
!.agents/skills/subminer-release/SKILL.md
.agents/skills/subminer-scrum-master/*
!.agents/skills/subminer-scrum-master/SKILL.md
favicon.png
+8 -3
View File
@@ -2,8 +2,10 @@
APP_NAME := subminer
THEME_SOURCE := assets/themes/subminer.rasi
THUMBNAILER_SOURCE := assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer
LAUNCHER_OUT := dist/launcher/$(APP_NAME)
THEME_FILE := subminer.rasi
THUMBNAILER_FILE := subminer-ffmpegthumbnailer.thumbnailer
# Default install prefix for the wrapper script.
PREFIX ?= $(HOME)/.local
@@ -221,11 +223,13 @@ docs-dev: ensure-bun
install-linux: build-launcher
@printf '%s\n' "[INFO] Installing Linux wrapper/theme artifacts"
@printf '%s\n' "[INFO] Installing Linux wrapper/support artifacts"
@install -d "$(BINDIR)"
@install -m 0755 "$(LAUNCHER_OUT)" "$(BINDIR)/$(APP_NAME)"
@install -d "$(LINUX_DATA_DIR)/themes"
@install -m 0644 "./$(THEME_SOURCE)" "$(LINUX_DATA_DIR)/themes/$(THEME_FILE)"
@install -d "$(LINUX_DATA_DIR)/thumbnailers"
@install -m 0644 "./$(THUMBNAILER_SOURCE)" "$(LINUX_DATA_DIR)/thumbnailers/$(THUMBNAILER_FILE)"
@install -d "$(LINUX_DATA_DIR)/plugin/subminer"
@cp -R ./plugin/subminer/. "$(LINUX_DATA_DIR)/plugin/subminer/"
@if [ -n "$(APPIMAGE_SRC)" ]; then \
@@ -234,7 +238,7 @@ install-linux: build-launcher
printf '%s\n' "[WARN] No release/SubMiner-*.AppImage found; skipping AppImage install"; \
printf '%s\n' " Build one with: make build"; \
fi
@printf '%s\n' "Installed to:" " $(BINDIR)/subminer" " $(LINUX_DATA_DIR)/themes/$(THEME_FILE)"
@printf '%s\n' "Installed to:" " $(BINDIR)/subminer" " $(LINUX_DATA_DIR)/themes/$(THEME_FILE)" " $(LINUX_DATA_DIR)/thumbnailers/$(THUMBNAILER_FILE)"
install-macos: build-launcher
@printf '%s\n' "[INFO] Installing macOS wrapper/theme/app artifacts"
@@ -275,8 +279,9 @@ uninstall:
uninstall-linux:
@rm -f "$(BINDIR)/subminer" "$(BINDIR)/SubMiner.AppImage"
@rm -f "$(LINUX_DATA_DIR)/themes/$(THEME_FILE)"
@rm -f "$(LINUX_DATA_DIR)/thumbnailers/$(THUMBNAILER_FILE)"
@rm -rf "$(LINUX_DATA_DIR)/plugin/subminer"
@printf '%s\n' "Removed:" " $(BINDIR)/subminer" " $(BINDIR)/SubMiner.AppImage" " $(LINUX_DATA_DIR)/themes/$(THEME_FILE)" " $(LINUX_DATA_DIR)/plugin/subminer"
@printf '%s\n' "Removed:" " $(BINDIR)/subminer" " $(BINDIR)/SubMiner.AppImage" " $(LINUX_DATA_DIR)/themes/$(THEME_FILE)" " $(LINUX_DATA_DIR)/thumbnailers/$(THUMBNAILER_FILE)" " $(LINUX_DATA_DIR)/plugin/subminer"
uninstall-macos:
@rm -f "$(BINDIR)/subminer"
+1 -1
View File
@@ -15,7 +15,7 @@ Integrates Yomitan and mpv - on-screen lookups, mine to Anki, and track immersio
[![License](https://img.shields.io/github/license/ksyasuda/SubMiner?style=flat-square&color=1a1a2e)](https://www.gnu.org/licenses/gpl-3.0)
[![TypeScript](https://img.shields.io/badge/TypeScript-1a1a2e?style=flat-square&logo=typescript&logoColor=3178c6)](https://www.typescriptlang.org)
[![SubMiner demo](./assets/minecard.webp)](https://github.com/user-attachments/assets/89e61895-e2b7-4b47-8d50-a35afe4132b2)
[![SubMiner demo](./assets/minecard.webp)](https://github.com/user-attachments/assets/7abab8a9-4e4e-4f06-9f3c-9783e15a3807)
</div>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 303 KiB

Binary file not shown.
Binary file not shown.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.0 MiB

After

Width:  |  Height:  |  Size: 3.8 MiB

@@ -0,0 +1,4 @@
[Thumbnailer Entry]
TryExec=ffmpegthumbnailer
Exec=ffmpegthumbnailer -i %i -o %o -s %s -f
MimeType=video/matroska;video/matroska-3d;video/x-matroska;video/x-matroska-3d;
+1
View File
@@ -49,6 +49,7 @@ How fragments turn into a release:
Prerelease notes:
- prerelease tags like `v0.11.3-beta.1` and `v0.11.3-rc.1` reuse the current pending fragments to generate `release/prerelease-notes.md`
- from the second prerelease of a base version onward, the notes also open with a `## Changes since <previous tag>` section generated from the fragment diff against the previous beta/RC tag; keep fragment edits meaningful — editorial-only rewording is filtered out of that section, while genuinely changed behavior and deleted fragments (reverted changes) are reported
- existing prerelease notes are a reviewed baseline; later prerelease runs should replace stale beta/RC wording with the current outcome instead of appending fix churn
- prerelease note generation does not consume fragments and does not update `CHANGELOG.md` or `docs-site/changelog.md`
- the final stable release is the point where `bun run changelog:build` consumes fragments into the stable changelog and release notes
+5
View File
@@ -0,0 +1,5 @@
type: fixed
area: subtitles
- Typeset ASS karaoke and animated signs no longer flood the primary overlay, subtitle sidebar, immersion history, or sentence mining with repeated glyph fragments or full-line color phases. Matching timed comments and full-line boundary events recover the complete authored line without merging ordinary repeated dialogue or separately positioned signs, and dialogue spoken while a song's animation is on screen is kept intact instead of being replaced by the lyric. Entrance and exit frames that run past the authored line timing still resolve to the clean line during lyric transitions, and dialogue spoken while a song's animation is on screen enters immersion and subtitle history without the fragment lines beside it.
- The secondary subtitle overlay drops layered duplicate lines from animated tracks, so a short stack of repeated words collapses to its distinct lines even when the full karaoke heuristic does not apply.
@@ -0,0 +1,5 @@
type: fixed
area: character dictionary
- Reuse character dictionaries after MeCab completes without finding any name splits instead of regenerating character data and portraits on every launch.
- Restore inline character portraits when a cached portrait index finishes loading after subtitles have already been tokenized.
@@ -0,0 +1,5 @@
type: fixed
area: dictionary
- Character dictionary generation, merged rebuilds, and imports no longer freeze the app (and trigger the compositor's "application not responding" dialog) on large dictionaries; snapshot reads/writes, archive building, and the character image/name lookup caches now do their heavy work off the UI's critical path.
- Desktop progress notifications now update in place on Linux AppImage installs too: the AppImage's bundled libraries broke the system notify-send helper, which silently forced the flickering close-and-reopen notification fallback.
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- The macOS window-tracking helper is now built for macOS 12.0+, so the overlay attaches to mpv on older systems (previously the helper required the macOS version of the build machine and crashed on e.g. Ventura, leaving the overlay stuck on "Overlay loading").
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Fixed the overlay getting stuck on "Overlay loading" forever when startup stalls: mpv IPC connection attempts now time out and retry, switching sockets aborts obsolete attempts, and the plugin replaces its spinner with an actionable error if overlay content is still not ready after 30 seconds.
@@ -0,0 +1,4 @@
type: fixed
area: subtitles
- Primary and secondary ASS subtitles now collapse layered and whitespace variants of full-span lyrics, including when playback starts or seeks into a line, reconstruct fragment-only karaoke per style, preserve authored stack order, keep canonical signs visible for their complete generated animation, navigate song lyrics by sanitized lines instead of generated animation events, and keep sidebar selections on the requested overlapping lyric while preserving unmatched dialogue and signs.
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Secondary subtitles now parse the selected ASS/SRT/VTT source with the primary subtitle deduplication pipeline, preventing layered animation text from appearing several times in the overlay, mined cards, and statistics. Fragmented ASS karaoke keeps spaces authored at event boundaries instead of joining every word together. Long ASS lines repeated as dialogue and positioned signs are also collapsed when they differ only in whitespace or terminal punctuation. Dense multi-row sign layouts no longer become one concatenated secondary line. Live mpv text remains the fallback for unreadable tracks and applies full-line duplicate filtering before display.
-4
View File
@@ -1,4 +0,0 @@
type: docs
area: documentation
- Hid the unfinished feature demos page from the documentation sidebar while keeping its direct URL available.
+5
View File
@@ -0,0 +1,5 @@
type: changed
area: release
- Prerelease notes now open with a "Changes since" section that lists only what changed compared to the previous beta/RC of the same version, above the cumulative highlights.
- CI now rejects prerelease tags whose committed notes were generated for a different beta/RC, instead of silently shipping stale notes.
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: launcher
- Fixed missing MKV thumbnails in the Linux rofi picker when system thumbnailer registrations only advertise legacy Matroska MIME aliases.
@@ -0,0 +1,8 @@
type: fixed
area: stats
- Fixed Vocabulary totals and charts counting only the first browsing page instead of all tracked vocabulary, without delaying the rest of the page.
- New-word history now uses permanent daily lexical rollups that apply the same vocabulary filters as the totals and normalize legacy second/millisecond timestamps; versioned background rebuilds repair existing history across legacy rollup-state schemas without dropping playback writes or clearing watch-time, activity, efficiency, and library charts.
- Calendar-day chart labels now preserve the recorded local date in time zones west of UTC.
- Vocabulary summary cards and charts refresh automatically after the word exclusion list changes, and failed or unfinished loads use bounded retries before showing an inline error with a Retry control.
- Rapid exclusion edits no longer race each other; writes are sent in order so a slower earlier save cannot overwrite a newer list.
+1 -1
View File
@@ -5,7 +5,7 @@ Short recordings of SubMiner's key features and integrations from real playback
<script setup>
import { withBase } from 'vitepress';
const v = '20260301-1';
const v = '20260819-1';
</script>
## Anki Card Mining & Enrichment
+1 -1
View File
@@ -6,7 +6,7 @@ For internal architecture/workflow guidance, use `docs/README.md` at the repo ro
- [Bun](https://bun.sh)
- A system `lua` interpreter for `bun run test:launcher` / `bun run test:plugin:src`
- macOS builds compile a Swift helper via `scripts/build-macos-helper.sh` (skip with `SUBMINER_SKIP_MACOS_HELPER_BUILD=1`)
- macOS builds compile a Swift helper via `scripts/prepare-build-assets.mjs` (skip with `SUBMINER_SKIP_MACOS_HELPER_BUILD=1`)
## Setup
+5 -1
View File
@@ -82,7 +82,7 @@ Expandable session history with new-word activity, cumulative totals, and pause/
#### Vocabulary
Top repeated words (click a bar to open the word), new-word timeline, cross-title and frequency rank tables with Hide Known / Hide Kana filters, kanji breakdown, word exclusion list, and click-through occurrence drilldown with Mine Word / Mine Sentence / Mine Audio buttons.
The summary cards show all unique vocabulary and kanji recorded in the local tracking database; **New This Week** is the only weekly figure and uses a rolling seven-day window. The word and kanji tables load first while those complete totals calculate separately. Top Repeated Words and New Words by Day use complete tracking history rather than the table's browsing page. New-word history is maintained as a permanent daily lexical rollup using the same token-visibility rules as the totals, including normalization of older timestamps stored in either seconds or milliseconds and retroactive corrections when tracked material is removed or reprocessed. On the first launch after an applicable upgrade, that history is version-rebuilt in the background and the chart refreshes when it is ready; if it remains unavailable, polling stops and an inline Retry control appears. The cards and charts also refresh automatically after the word exclusion list changes. The rest of the tab includes cross-title and frequency rank tables with Hide Known / Hide Kana filters, kanji breakdown, word exclusion list, and click-through occurrence drilldown with Mine Word / Mine Sentence / Mine Audio buttons.
![Stats Vocabulary](/screenshots/stats-vocabulary.png)
@@ -138,6 +138,8 @@ Karaoke openings and animated signs are authored as one subtitle event per anima
Recording now collapses those runs as they happen, matching what the subtitle sidebar shows:
- When a typeset ASS file stores a clean lyric or sign in a timed authoring comment, or in full-line events surrounding generated fragments, the matching complete line is recorded once. The repeated glyph or clip-animation frames are not recorded. Dialogue spoken while such an animation is on screen records as itself, without the fragment lines beside it.
- When karaoke styling redraws the same complete lyric across consecutive color or highlight phases, those phases are combined into one line with their full timing. Repeated ordinary dialogue remains separate.
- When the active subtitle source has been parsed, its cue list has already had duplicate events and animation bursts merged. A line landing inside a surviving cue but after that cue's start is a frame the sidebar merged away, and is not recorded.
- When no parsed cue covers the live timing, including while a subtitle source is changing or shifted, the strict metadata-free rule applies: a run of identical, contiguous lines each shorter than 0.1s stops being recorded after a few frames. Runs are tracked per line of text, so dual-line karaoke (a kanji and a romaji line frame-flipped together) collapses both lines. Ordinary repeated dialogue, and lines held for a normal beat, always record.
@@ -180,6 +182,7 @@ In practice:
- Anime and episode pages keep lifetime totals from summary tables while session drill-down still reads retained sessions directly. With the current defaults, both are kept forever.
- Trends can read the full available history because daily/monthly rollups are also kept forever by default.
- Vocabulary and kanji totals are cumulative and not bounded by the raw session retention knobs.
- New-word charts use their own permanent lexical daily rollups, which are not pruned by activity-rollup retention.
## Storage / Performance Model
@@ -349,6 +352,7 @@ Rollup tables:
- `imm_daily_rollups`
- `imm_monthly_rollups`
- `imm_lexical_daily_rollups` - permanent first-discovery counts for vocabulary and kanji chart history
- `imm_rollup_state` - incremental rollup progress bookkeeping
Vocabulary tables:
+1 -1
View File
@@ -88,7 +88,7 @@ features:
<script setup>
import { withBase } from 'vitepress';
const demoAssetVersion = '20260223-2';
const demoAssetVersion = '20260819-1';
</script>
<div class="landing-shell">
+6 -4
View File
@@ -392,7 +392,7 @@ subminer -u
subminer --update
```
SubMiner verifies AppImage, launcher, and Linux support-asset downloads against `SHA256SUMS.txt`. On Linux those support assets include the launcher-managed runtime plugin copy under `SubMiner/plugin/subminer` plus the rofi theme at `SubMiner/themes/subminer.rasi`. If the binary is in a protected path, SubMiner shows the exact command to run rather than elevating itself.
SubMiner verifies AppImage, launcher, and Linux support-asset downloads against `SHA256SUMS.txt`. On Linux those support assets include the launcher-managed runtime plugin copy under `SubMiner/plugin/subminer`, the rofi theme at `SubMiner/themes/subminer.rasi`, and the scoped Matroska thumbnailer registration under `SubMiner/thumbnailers`. If the binary is in a protected path, SubMiner shows the exact command to run rather than elevating itself.
The tray "Check for Updates" entry installs the new app automatically on Linux, macOS, and Windows. On Linux it replaces the running `.AppImage` in place via `electron-updater` and refreshes the managed support assets from `subminer-assets.tar.gz`; AppImages managed by a system package (for example the AUR `/opt/SubMiner/SubMiner.AppImage`) are skipped so the package manager stays in charge.
@@ -404,7 +404,7 @@ SubMiner is an overlay that sits on top of mpv. It connects to mpv through an IP
The `subminer` launcher handles mpv IPC socket setup automatically. If you launch mpv yourself or from another tool, you must pass `--input-ipc-server=/tmp/subminer-socket` (or `\\.\pipe\subminer-socket` on Windows) - without it the overlay starts but subtitles won't appear.
The bundled mpv plugin is injected at runtime automatically - you don't need to install it separately. On Linux, the `subminer` launcher now checks for its managed runtime plugin copy and rofi theme before every mpv-managed launch and installs those support assets from the bundled app automatically if either one is missing. It provides in-player keybindings (the `y` chord) for controlling the overlay from within mpv. See [MPV Plugin](/mpv-plugin) for the full keybinding and configuration reference.
The bundled mpv plugin is injected at runtime automatically - you don't need to install it separately. On Linux, the `subminer` launcher checks for its managed runtime plugin copy, rofi theme, and scoped thumbnailer registration before every mpv-managed launch and installs those support assets from the bundled app automatically if one is missing. For a rofi picker launch, this check runs before the picker opens. It provides in-player keybindings (the `y` chord) for controlling the overlay from within mpv. See [MPV Plugin](/mpv-plugin) for the full keybinding and configuration reference.
## Platform Notes
@@ -456,18 +456,20 @@ sudo chmod +x /usr/local/bin/subminer
### Linux Support Assets
SubMiner ships the Linux rofi theme plus the launcher-managed runtime plugin copy in `subminer-assets.tar.gz`:
SubMiner ships the Linux rofi theme, scoped Matroska thumbnailer registration, and launcher-managed runtime plugin copy in `subminer-assets.tar.gz`:
```bash
wget https://github.com/ksyasuda/SubMiner/releases/latest/download/subminer-assets.tar.gz -O /tmp/subminer-assets.tar.gz
tar -xzf /tmp/subminer-assets.tar.gz -C /tmp
mkdir -p ~/.local/share/SubMiner/themes
cp /tmp/assets/themes/subminer.rasi ~/.local/share/SubMiner/themes/subminer.rasi
mkdir -p ~/.local/share/SubMiner/thumbnailers
cp /tmp/assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer ~/.local/share/SubMiner/thumbnailers/
mkdir -p ~/.local/share/SubMiner/plugin
cp -R /tmp/plugin/subminer ~/.local/share/SubMiner/plugin/subminer
```
`subminer -u` and the tray updater keep those Linux support assets in sync automatically once the `SubMiner` data dir exists. Normal Linux launcher playback also auto-installs the managed runtime plugin copy and rofi theme from the bundled app if either support asset is missing, so manual extraction is mainly useful for pre-seeding or custom setups.
`subminer -u` and the tray updater keep those Linux support assets in sync automatically once the `SubMiner` data dir exists. Normal Linux launcher playback also auto-installs all three assets from the bundled app if one is missing, so manual extraction is mainly useful for pre-seeding or custom setups. Rofi receives the SubMiner data path through its process-local `XDG_DATA_DIRS`, so the thumbnailer registration does not change the desktop-wide configuration.
Override the theme path with `SUBMINER_ROFI_THEME=/absolute/path/to/theme.rasi`.
+6 -2
View File
@@ -34,18 +34,22 @@ subminer -R -r -d ~/Anime # rofi picker, recursive
subminer -R /directory # rofi picker, directory shortcut
```
rofi shows a GUI menu with icon thumbnails when available. SubMiner ships the rofi theme plus the Linux launcher-managed runtime plugin copy in the release assets tarball:
rofi shows a GUI menu with icon thumbnails when available. SubMiner ships the rofi theme, a scoped `ffmpegthumbnailer` MIME registration, and the Linux launcher-managed runtime plugin copy in the release assets tarball:
```bash
wget https://github.com/ksyasuda/SubMiner/releases/latest/download/subminer-assets.tar.gz -O /tmp/subminer-assets.tar.gz
tar -xzf /tmp/subminer-assets.tar.gz -C /tmp
mkdir -p ~/.local/share/SubMiner/themes
cp /tmp/assets/themes/subminer.rasi ~/.local/share/SubMiner/themes/subminer.rasi
mkdir -p ~/.local/share/SubMiner/thumbnailers
cp /tmp/assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer ~/.local/share/SubMiner/thumbnailers/
mkdir -p ~/.local/share/SubMiner/plugin
cp -R /tmp/plugin/subminer ~/.local/share/SubMiner/plugin/subminer
```
Once the `SubMiner` data dir exists, `subminer -u` refreshes both assets automatically. Normal Linux launcher playback also checks for the managed runtime plugin copy and rofi theme before mpv launch and installs them from the bundled app automatically if either one is missing.
Once the `SubMiner` data dir exists, `subminer -u` refreshes these assets automatically. Normal Linux launcher playback checks for all three assets and installs them from the bundled app when one is missing. For `subminer -R`, this repair runs before rofi opens.
When `ffmpegthumbnailer` is installed, SubMiner prepends its own data directory to `XDG_DATA_DIRS` for the rofi process only. This lets rofi recognize the canonical Matroska MIME types used by newer GLib versions without changing the desktop-wide MIME or thumbnailer configuration. An existing registration in your own `$XDG_DATA_HOME/thumbnailers` still takes priority.
The theme is auto-detected from these paths (first match wins):
+2
View File
@@ -110,6 +110,8 @@ The secondary bar is a compact top-strip region in the same overlay window. It s
It is controlled by `secondarySub` configuration and shares its lifecycle with the main overlay window. Cycle which track feeds it with `Shift+J`.
SubMiner collapses duplicate ASS layers in parsed secondary tracks. Long lines repeated as dialogue and positioned signs are treated as the same line when they differ only in whitespace or terminal punctuation. Dense multi-row sign layouts, such as translated timetables, are excluded instead of being concatenated into the secondary bar. When SubMiner must use mpv's live text as a fallback, it still filters full-line duplicates while preserving short repeated dialogue.
### Display Modes
Both the primary and secondary subtitle bars share the same three visibility modes, and each can be changed independently at runtime:
Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 160 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 303 KiB

Binary file not shown.
Binary file not shown.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 523 KiB

Binary file not shown.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.0 MiB

After

Width:  |  Height:  |  Size: 3.8 MiB

+3 -1
View File
@@ -9,9 +9,11 @@ The sidebar is enabled by default. Set `subtitleSidebar.enabled` to `false` if y
When SubMiner parses the active subtitle source into a cue list, the sidebar becomes available. Toggle it with the `\` key (configurable via `subtitleSidebar.toggleKey`). While open:
- The active cue is highlighted and kept in view as playback advances (when `autoScroll` is `true`).
- Clicking any cue seeks mpv to that timestamp.
- Clicking any cue seeks mpv into that line. For overlapping ASS karaoke, SubMiner moves past the previous line's exit animation when the selected cue has enough time remaining.
- The sidebar stays synchronized with the overlay - media transitions and subtitle source changes update both simultaneously.
For typeset ASS karaoke and animated signs, SubMiner collapses generated animation frames and repeated full-line color phases before they reach the sidebar. It recovers a clean complete line from a matching timed authoring comment or from full-line events surrounding generated fragments. Ordinary ASS comments, editor notes, alternate lines, repeated dialogue, and separately positioned signs remain distinct.
The sidebar only appears when a parsed cue list is available. External subtitle sources that SubMiner cannot parse (for example, embedded ASS tracks rendered directly by mpv) will not populate the sidebar.
## Layout Modes
+11 -6
View File
@@ -58,12 +58,15 @@
`latest*.yml` and `*.blockmap` files under `release/`.
5. Commit the prerelease prep (package.json version bump + the generated
`release/prerelease-notes.md`). CI does not regenerate notes — it uses the
committed file — so review it before committing. If you add more
`changes/*.md` fragments for a later beta/RC, rerun
`bun run changelog:prerelease-notes --version <version>`; the generator uses
the existing prerelease notes as the baseline only when their hidden
`prerelease-base-version` marker matches the current base version, and asks
Claude to merge only the new fragment material. Do not run
committed file — so review it before committing. Rerun
`bun run changelog:prerelease-notes --version <version>` for every later
beta/RC, even if no fragments changed: the notes carry a hidden
`prerelease-version` marker and CI rejects the tag when the marker does not
match it (verify locally with
`bun run changelog:check-prerelease-notes --version <version>`). The
generator reuses the existing notes as the cumulative baseline when their
marker (or legacy `prerelease-base-version` marker) matches the current base
version, and asks Claude to merge only the new fragment material. Do not run
`bun run changelog:build`.
6. Tag the commit: `git tag v<version>`.
7. Push commit + tag.
@@ -78,6 +81,8 @@ Notes:
- Pass `--date` explicitly when you want the release stamped with the local cut date; otherwise the generator uses the current ISO date, which can roll over to the next UTC day late at night.
- `changelog:check` now rejects tag/package version mismatches.
- `changelog:prerelease-notes` also rejects tag/package version mismatches and writes `release/prerelease-notes.md` without mutating tracked changelog files. When that file already exists, the generator includes it in the Claude prompt so later beta/RC notes reuse the reviewed text instead of starting over.
- From the second prerelease of a base version onward, the notes open with a `## Changes since <previous tag>` section above the cumulative `## Highlights`. The generator locates the newest preceding beta/RC tag for the same base version (semver order: all betas before all RCs), diffs `changes/*.md` between that tag and the working tree, and asks Claude to describe only the behavioral beta-to-beta differences — added fragments as new changes, modified fragments by their before/after difference (editorial-only edits are dropped), deleted fragments as removed/reverted changes. If no fragments changed (for example a packaging-only rebuild), the section states that explicitly without a Claude call. The delta section carries no separate contributor attribution; `## What's Changed` stays cumulative like `## Highlights`.
- `changelog:check-prerelease-notes --version <version>` verifies the committed notes' `prerelease-version` marker matches the version being tagged; the prerelease workflow runs it and fails the release on stale notes.
- `changelog:build` generates `CHANGELOG.md` + `release/release-notes.md` (both polished by `claude -p`) and removes the released `changes/*.md` fragments. The CHANGELOG keeps internal notes inside a `<details><summary>Internal changes</summary>` collapse; the release notes drop them entirely.
- `release/release-notes.md` (and `release/prerelease-notes.md`) include GitHub-style attribution after `## Highlights`: a `## What's Changed` list crediting each released fragment as `by @<author> in #<pr>`, plus a `## New Contributors` section for first-time authors. Attribution is resolved per fragment via `git log` (the commit that added the fragment) + `gh api .../commits/<sha>/pulls`, with one `gh` search per author for the first-contribution check. It needs `gh` installed and authenticated; if `gh` is unavailable or a lookup fails, the generator warns and emits notes without the attribution sections rather than failing. The CHANGELOG itself stays attribution-free.
- The release workflow no longer auto-runs `changelog:build`. If pending `changes/*.md` fragments are present on a tag-based run, CI exits with a clear `::error::` pointing at the local fix. Run `bun run changelog:build --version <version>` locally, commit the polished output, then tag.
@@ -70,18 +70,25 @@ interface SubtitleCue {
startTime: number; // seconds
endTime: number; // seconds
text: string; // plain text, decoded from the source format
source?: 'canonical-ass'; // recovered authored text for generated ASS animation
animationStartTime?: number; // full generated-frame envelope; entrance/exit frames
animationEndTime?: number; // run past the authored timing, live matching uses this
}
```
**Supported formats:**
- SRT/VTT: Regex-based parsing of timing lines + text content between timing blocks.
- ASS: Parse `[Events]` section, extract `Dialogue:` lines, read the field order from the `Format:` row, and take everything after the Text field index as the text (Text can itself contain commas).
- ASS: Parse the `[Events]` section, read the field order from the `Format:` row, and extract timed `Dialogue:` lines. Timed `Comment:` lines are normally ignored, but can supply canonical authored text when they match a nearby generated animation from the same style and actor. Text can itself contain commas.
**ASS decoding.** The parser is where ASS text is decoded, once, via `assToPlainText()` in `src/core/services/ass-text.ts`. That decoder mirrors mpv's `ass_to_plaintext` so a cue read from a file reads identically to the same line arriving live on `sub-text`: `{...}` override blocks are markup, `\pN … \p0` vector drawing runs are dropped rather than shown as text, `\N`/`\n`/`\h` are the only escapes (`\{`, `\}` and `\\` are not), and an unclosed `{` is rendered verbatim. Every layer downstream — renderer, timing tracker, tokenizer, tokenization cache keys — receives plain text and uses `normalizePlainSubtitleText()` for whitespace only, so nothing decodes the same string twice and one authored line always maps to one cache key.
**Duplicate collapsing.** Typeset scripts emit one `Dialogue:` event per animation frame, plus layered copies of the same line. The parser collapses identical text over an identical span unconditionally, and collapses contiguous same-text runs of at least three events when the run looks like an animation. For ASS that means shared style and actor plus authoring evidence: a temporal tag (`\t`, `\move`, `\k`/`\kf`/`\ko`/`\K`, or anything wrapped in `\t(...)`), an animated `Effect` column (`Karaoke`, `Banner`, `Scroll`), or override values that change across the run. Static tags shared by every event (`\pos`, an identical `\clip`) are not evidence. SRT/VTT carry no such metadata, so there collapsing needs at least five contiguous events all under 0.1s — the frame timing left behind by ASS-to-SRT conversion. The parser keeps this authoring metadata (style, actor, layer, `Effect`, parsed override commands, source order) private; `parseSubtitleCues()` returns only `SubtitleCue`.
ASS scripts can also redraw one complete lyric for two or more long color/highlight phases. Those flush-timed phases collapse separately from short animation frames when they share text, style, actor, and layer and carry direct animation evidence, such as temporal tags or changing non-spatial overrides. Spatial command changes do not prove a phase, so separately positioned signs remain distinct.
**Canonical animation recovery.** Some ASS producers keep the readable lyric or sign as a timed `Comment:` and generate hundreds of `Dialogue:` frames containing repeated glyphs or changing clip regions. Others retain the complete line as brief `Dialogue:` events around the generated fragments. A complete event is promoted only when nearby dialogue from the same style and actor forms a proven animation cluster and reconstructs its entire text in source order. The generated frames are then replaced by one cue marked `source: 'canonical-ass'`. This source marker lets the live primary-subtitle path prefer the clean authored text and timing for display, sidebar history, immersion recording, and mining, while unmatched editor notes and alternative translations remain ignored.
#### Prefetch Service Lifecycle
1. **Activation trigger:** When a subtitle track is activated (or changes), check if it's external via MPV's `track-list` property. If `external === true`, read the file via `external-filename` using the existing `loadSubtitleSourceText` infrastructure.
+3 -1
View File
@@ -23,7 +23,9 @@ Trend charts now consume one chart-oriented backend payload from `/api/stats/tre
- lookup rate trends
- watch-time by day-of-week/hour
- vocabulary-backed:
- new-words trend
- new-words trend reads permanent daily lexical rollups
- rollup rows count only vocabulary-visible tokens and normalize mixed legacy timestamp units
- a persisted rollup version invalidates stale materializations and triggers an atomic background rebuild
## Metric Semantics
+58 -3
View File
@@ -3,7 +3,7 @@
# Subtitle Overlay Priming
Status: active
Last verified: 2026-08-04
Last verified: 2026-08-19
Owner: Kyle Yasuda
Read when: debugging subtitle state or blank Linux/X11 overlay windows when the visible overlay is shown or recreated
@@ -69,14 +69,69 @@ coming and prefetching would otherwise idle for the rest of the cue.
## Live Cue Delivery
- Primary live text first resolves recovered canonical ASS animations. Otherwise, when
every live mpv line matches an active parsed cue, it uses the parsed cue text so exact
full-span style layers appear once instead of repeating for fill, border, blur, shadow,
or equivalent whitespace variants. Any unmatched live line keeps the complete live
stack, preserving dialogue or signs that overlap a lyric.
- A tokenization cache miss emits the plain cue synchronously. Tokenization remains serialized so
live work does not contend for Yomitan state.
- The initial `time-pos`, explicit renderer seeks, and later seek-like jumps reprocess mpv's
current raw `sub-text` after the new playback time is stored. Explicit intent matters because
adjacent subtitle jumps can be shorter than the general seek-distance threshold. This corrects
ASS cleanup when mpv delivered the destination subtitle before the destination timestamp.
- Renderer `sub-seek` commands use the active parsed cue list when available. Simultaneous cues
share one boundary, overlapping lyrics advance from the latest active boundary, and mpv's native
command remains the fallback when no parsed destination exists. This prevents generated karaoke
frames from consuming next/previous subtitle presses.
- Subtitle sidebar selections seek past the preceding sanitized cue's overlapping exit span when
the selected cue has enough time remaining. This keeps direct row selection on the requested
karaoke line while clamping the seek inside that cue.
- If startup paints raw text before embedded ASS parsing finishes, parsed cue arrival may replace
that provisional line. The one-prime-per-media guard still suppresses identical repeats.
- If a newer cue arrives while an older line is still tokenizing, the newer plain cue or empty
clear payload is emitted immediately. The older tokenization result is dropped before it can
replace the current cue.
- The current cue upgrades in place when its tokens and annotations are ready. This can reflow text
or character images, but cue visibility does not wait for that work.
## Secondary Subtitle Flow
- `secondary-sub-text` remains the immediate fallback, so unreadable and remote subtitle sources
still appear without waiting for file resolution.
- Parsed secondary text and the live fallback share a flattened-line identity for long lines. This
removes dialogue/sign repetitions that differ only in whitespace or terminal punctuation while
retaining short repeated lines that can represent authored dialogue without source metadata.
- `secondary-subtitle-track.ts` resolves `secondary-sid` against mpv's track list. External tracks
are read directly; supported embedded text tracks are extracted through the same ffmpeg-backed
source resolver used by primary subtitle prefetching.
- The selected source is parsed with `parseSubtitleCues()`, including metadata-aware ASS duplicate
and animation collapse. Playback `time-pos` selects the active parsed cue after applying
`secondary-sub-delay`.
- Fragment reconstruction marks positioned parts that span multiple vertical rows as a grid.
Secondary text omits those grids instead of flattening a translated table or schedule into one
synthetic line. Reconstructed single-line karaoke remains eligible for display.
- The resolved text is stored in `mpvClient.currentSecondarySubText` before it is broadcast. The
overlay, mining, timing tracker, and immersion statistics therefore consume the same secondary
text when a readable source is available.
- Simultaneous parsed cues use whitespace-insensitive identity, so ASS layers that vary only
between ordinary, hard, or ideographic spaces appear once.
- Simultaneous ASS lines are flattened in top-to-bottom positioned order, falling back to their
authored source order when no usable position exists.
- Fragment-only ASS karaoke is reconstructed per style before publication. Explicit spaces
survive concatenation, while scripts that discarded their word boundaries remain compact
instead of gaining false spaces between syllables. Short runs qualify only when overlapping
positioned events also show changing overrides or repeated layer copies; an English or romaji
style name alone never turns ordinary dialogue into a lyric.
- Recovered canonical ASS text remains active for the generated animation envelope. For
reconstructed lyric styles, the longest-lived active line wins over brief entrance and exit
fragments from the same style.
- Media and `secondary-sid` changes clear the previous parsed state before refreshing the source;
track-list changes refresh without discarding an unchanged source. Observed
`secondary-sub-delay` changes retime the active parsed cue without rereading the file. If loading,
extraction, or parsing fails, the controller returns to live mpv text and the renderer's
conservative short stack heuristic remains the final display fallback.
## Emitted State
- `emitSubtitle(payload)` maps to `emitSubtitlePayload(payload)`. Overlay windows and annotation
@@ -84,8 +139,8 @@ coming and prefetching would otherwise idle for the rest of the cue.
- The basic subtitle websocket receives the immediate plain cue only. Because its serialized
payload discards annotations, the later upgrade would be an identical duplicate and is skipped
when text and cue timing match.
- Secondary priming reads mpv `secondary-sub-text`, stores it in
`mpvClient.currentSecondarySubText`, and broadcasts `secondary-subtitle:set` to overlay windows.
- Secondary priming reads mpv `secondary-sub-text` and routes it through the secondary track
controller. A parsed active cue replaces the live text when the selected source is readable.
- If secondary `requestProperty` fails, the primary flow stays complete and only a debug line is
written.
+1 -1
View File
@@ -19,7 +19,7 @@ Read when: finding internal docs or checking verification status
| Quality scorecard | `docs/knowledge-base/quality.md` | active | 2026-03-13 | quality grades and gaps |
| Workflow index | `docs/workflow/README.md` | active | 2026-08-13 | execution map |
| Planning guide | `docs/workflow/planning.md` | active | 2026-05-23 | lightweight vs execution plans |
| Agent skills | `docs/workflow/agent-skills.md` | active | 2026-08-13 | repo-local workflow skill ownership |
| Agent skills | `docs/workflow/agent-skills.md` | active | 2026-08-23 | repo-local workflow skill ownership |
| Verification guide | `docs/workflow/verification.md` | active | 2026-08-13 | maintained verification lanes |
| Release guide | `docs/RELEASING.md` | active | 2026-05-23 | release checklist |
+4 -1
View File
@@ -3,7 +3,7 @@
# Agent Skills
Status: active
Last verified: 2026-08-13
Last verified: 2026-08-23
Owner: Kyle Yasuda
Read when: using, adding, or changing a repo-local agent workflow skill
@@ -12,6 +12,9 @@ Read when: using, adding, or changing a repo-local agent workflow skill
- `.agents/skills/subminer-change-verification/`
- Selects the cheapest sufficient repo-native verification lane.
- Defers command ownership to `package.json` and `docs/workflow/verification.md`.
- `.agents/skills/subminer-release/`
- Prepares, cuts, publishes, or repairs stable and prerelease releases.
- Defers release procedure and policy to `docs/RELEASING.md`.
Repo-local workflows stay as standalone skills. Do not add plugin packaging, marketplace metadata, or compatibility shims unless the workflow is intentionally being distributed beyond this repository.
+8
View File
@@ -23,6 +23,7 @@ import {
type HistorySeriesEntry,
} from '../history.js';
import type { Args } from '../types.js';
import { ensureLinuxRuntimePluginAvailable } from '../runtime-plugin-preflight.js';
import type { LauncherCommandContext } from './context.js';
export type HistorySessionAction = 'previous' | 'replay' | 'next' | 'browse' | 'quit';
@@ -333,6 +334,13 @@ export async function runHistoryCommand(
const { args, scriptPath } = context;
checkPickerDependencies(args);
if (args.useRofi) {
await ensureLinuxRuntimePluginAvailable({
appPath: context.appPath ?? undefined,
scriptPath,
logLevel: args.logLevel,
});
}
const themePath = args.useRofi ? findRofiTheme(scriptPath) : null;
const dbPath = resolveImmersionDbPath();
+8
View File
@@ -2,6 +2,7 @@ import { fail } from '../log.js';
import { runAppCommandWithInherit } from '../mpv.js';
import { commandExists } from '../util.js';
import { runJellyfinPlayMenu } from '../jellyfin.js';
import { ensureLinuxRuntimePluginAvailable } from '../runtime-plugin-preflight.js';
import { shouldForwardLogLevel } from '../types.js';
import type { LauncherCommandContext } from './context.js';
@@ -64,6 +65,13 @@ export async function runJellyfinCommand(context: LauncherCommandContext): Promi
if (args.useRofi && !commandExists('rofi')) {
fail('rofi not found. Install rofi or omit -R for fzf.');
}
if (args.useRofi) {
await ensureLinuxRuntimePluginAvailable({
appPath,
scriptPath,
logLevel: args.logLevel,
});
}
await runJellyfinPlayMenu(appPath, args, scriptPath, mpvSocketPath);
return true;
}
@@ -496,3 +496,39 @@ test('playback command ensures Linux runtime plugin before mpv launch', async ()
assert.deepEqual(calls, ['plugin', 'startMpv']);
});
test('rofi playback repairs support assets before opening the picker', async () => {
const context = createContext();
context.args = {
...context.args,
target: '',
targetKind: '',
useRofi: true,
};
const calls: string[] = [];
await runPlaybackCommandWithDeps(context, {
ensurePlaybackSetupReady: async () => {},
ensureRuntimePluginReady: async () => {
calls.push('assets');
},
chooseTarget: async () => {
calls.push('picker');
return { target: '/tmp/movie.mkv', kind: 'file' };
},
checkPickerDependencies: () => {},
checkDependencies: () => {},
registerCleanup: () => {},
startMpv: async () => {
calls.push('startMpv');
},
waitForUnixSocketReady: async () => true,
startOverlay: async () => {},
launchAppCommandDetached: () => {},
log: () => {},
cleanupPlaybackSession: async () => {},
getMpvProc: () => null,
});
assert.deepEqual(calls, ['assets', 'picker', 'startMpv']);
});
+15 -2
View File
@@ -157,6 +157,7 @@ export async function runPlaybackCommand(context: LauncherCommandContext): Promi
});
},
chooseTarget,
checkPickerDependencies,
checkDependencies,
registerCleanup,
startMpv,
@@ -177,6 +178,7 @@ type PlaybackCommandDeps = {
args: Args,
scriptPath: string,
) => Promise<{ target: string; kind: 'file' | 'url' } | null>;
checkPickerDependencies?: (args: Args) => void;
checkDependencies: (args: Args) => void;
registerCleanup: (context: LauncherCommandContext) => void;
startMpv: typeof startMpv;
@@ -201,7 +203,18 @@ export async function runPlaybackCommandWithDeps(
await deps.ensurePlaybackSetupReady(context);
if (!args.target) {
checkPickerDependencies(args);
(deps.checkPickerDependencies ?? checkPickerDependencies)(args);
}
let runtimeAssetsReady = false;
const ensureRuntimeAssetsReady = async (): Promise<void> => {
if (runtimeAssetsReady) return;
await deps.ensureRuntimePluginReady(context);
runtimeAssetsReady = true;
};
if (!args.target && args.useRofi) {
await ensureRuntimeAssetsReady();
}
const targetChoice = await deps.chooseTarget(args, scriptPath);
@@ -266,7 +279,7 @@ export async function runPlaybackCommandWithDeps(
);
}
await deps.ensureRuntimePluginReady(context);
await ensureRuntimeAssetsReady();
await deps.startMpv(
selectedTarget.target,
+6
View File
@@ -36,6 +36,11 @@ test('runUpdateCommand updates directly on Linux without launching Electron', as
launcher: { status: 'updated' },
supportAssets: [
{ status: 'updated', component: 'theme', message: 'Installed theme.' },
{
status: 'updated',
component: 'thumbnailer',
message: 'Installed rofi thumbnailer.',
},
{ status: 'skipped', component: 'plugin', message: 'Plugin already up to date.' },
],
};
@@ -52,6 +57,7 @@ test('runUpdateCommand updates directly on Linux without launching Electron', as
'info:AppImage update: updated',
'info:Launcher update: updated',
'info:Support assets (theme) update: updated - Installed theme.',
'info:Support assets (thumbnailer) update: updated - Installed rofi thumbnailer.',
'info:Support assets (plugin) update: skipped - Plugin already up to date.',
]);
});
+10 -13
View File
@@ -21,7 +21,10 @@ import {
parseSha256Sums,
type FetchLike,
} from '../../src/main/runtime/update/release-assets.js';
import { updateSupportAssetsFromRelease } from '../../src/main/runtime/update/support-assets.js';
import {
updateSupportAssetsFromRelease,
type SupportAssetsUpdateResult,
} from '../../src/main/runtime/update/support-assets.js';
type UpdateCommandResponse = {
ok: boolean;
@@ -36,15 +39,14 @@ type DirectReleaseUpdateRequest = {
channel: UpdateChannel;
};
type DirectSupportAssetsUpdateResult = Omit<SupportAssetsUpdateResult, 'status'> & {
status: string;
};
type DirectReleaseUpdateResult = {
appImage: { status: string; command?: string; message?: string };
launcher: { status: string; command?: string; message?: string };
supportAssets: Array<{
status: string;
component?: 'theme' | 'plugin';
command?: string;
message?: string;
}>;
supportAssets: DirectSupportAssetsUpdateResult[];
};
type UpdateCommandDeps = {
@@ -129,12 +131,7 @@ function readUpdateChannel(root: Record<string, unknown> | null): UpdateChannel
function logUpdateResult(
label: string,
result: {
status: string;
component?: 'theme' | 'plugin';
command?: string;
message?: string;
},
result: DirectSupportAssetsUpdateResult,
configuredLogLevel: NonNullable<LauncherCommandContext['args']['logLevel']>,
deps: Pick<UpdateCommandDeps, 'log'>,
): void {
+6 -5
View File
@@ -73,20 +73,21 @@ function makeTestEnv(homeDir: string, xdgConfigHome: string): NodeJS.ProcessEnv
};
}
// On Linux the playback path runs `ensureLinuxRuntimePluginAvailable`, which
// when the runtime plugin/theme are missing — spawns the app with
// `--ensure-linux-runtime-plugin-assets` and polls up to 30s
// On Linux the playback path runs `ensureLinuxRuntimePluginAvailable`, which
// spawns the app with `--ensure-linux-runtime-plugin-assets` when managed
// support assets are missing and polls up to 30s
// (RESPONSE_TIMEOUT_MS) for an install response. A fake app that just exits
// never writes that response, so the launcher hangs and the test times out on
// Linux CI (the preflight is a no-op on macOS/Windows). This shell prelude makes
// the fake app install the managed plugin/theme and write the response, matching
// the fake app install the managed support assets and write the response, matching
// launcher/smoke.e2e.test.ts. Prepend it to each fake app that reaches playback.
const RUNTIME_PLUGIN_PREFLIGHT_SH = `if [ "$1" = "--ensure-linux-runtime-plugin-assets" ]; then
data="\${XDG_DATA_HOME:-$HOME/.local/share}/SubMiner"
mkdir -p "$data/plugin/subminer" "$data/themes"
mkdir -p "$data/plugin/subminer" "$data/themes" "$data/thumbnailers"
printf -- '-- test plugin\\n' > "$data/plugin/subminer/main.lua"
printf 'test=true\\n' > "$data/plugin/subminer.conf"
printf '/* test theme */\\n' > "$data/themes/subminer.rasi"
printf '[Thumbnailer Entry]\\n' > "$data/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer"
if [ "$2" = "--ensure-linux-runtime-plugin-assets-response-path" ] && [ -n "$3" ]; then
mkdir -p "$(dirname "$3")"
printf '{"ok":true,"status":"installed","path":"%s"}' "$data/plugin/subminer/main.lua" > "$3"
+46 -1
View File
@@ -3,7 +3,12 @@ import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import os from 'node:os';
import { findRofiTheme, formatRofiPrompt } from './picker';
import {
findRofiTheme,
findRofiThumbnailerDataRoot,
formatRofiPrompt,
prependXdgDataDir,
} from './picker';
// ── formatRofiPrompt: spacing between prompt and input field ──────────────────
@@ -23,6 +28,7 @@ test('formatRofiPrompt leaves an empty prompt empty', () => {
// ── findRofiTheme: Linux packaged path discovery ──────────────────────────────
const ROFI_THEME_FILE = 'subminer.rasi';
const ROFI_THUMBNAILER_FILE = 'subminer-ffmpegthumbnailer.thumbnailer';
function makeFile(filePath: string): void {
fs.mkdirSync(path.dirname(filePath), { recursive: true });
@@ -121,3 +127,42 @@ test('findRofiTheme resolves ~/.local/share/SubMiner/themes/subminer.rasi when X
fs.rmSync(baseDir, { recursive: true, force: true });
}
});
test('findRofiThumbnailerDataRoot resolves the managed XDG data root', () => {
const xdgDataHome = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-test-xdg-'));
const originalXdgDataHome = process.env.XDG_DATA_HOME;
try {
process.env.XDG_DATA_HOME = xdgDataHome;
const dataRoot = path.join(xdgDataHome, 'SubMiner');
makeFile(path.join(dataRoot, 'thumbnailers', ROFI_THUMBNAILER_FILE));
const result = withPlatform('linux', () => findRofiThumbnailerDataRoot('/usr/bin/subminer'));
assert.equal(result, dataRoot);
} finally {
if (originalXdgDataHome === undefined) {
delete process.env.XDG_DATA_HOME;
} else {
process.env.XDG_DATA_HOME = originalXdgDataHome;
}
fs.rmSync(xdgDataHome, { recursive: true, force: true });
}
});
test('findRofiThumbnailerDataRoot is Linux-only', () => {
assert.equal(
withPlatform('darwin', () => findRofiThumbnailerDataRoot('/usr/bin/subminer')),
null,
);
});
test('prependXdgDataDir preserves existing roots and avoids duplicates', () => {
const root = '/tmp/subminer-data';
assert.equal(
prependXdgDataDir(root, `/opt/share${path.delimiter}${root}${path.delimiter}/usr/share`),
`${root}${path.delimiter}/opt/share${path.delimiter}/usr/share`,
);
assert.equal(
prependXdgDataDir(root),
`${root}${path.delimiter}/usr/local/share${path.delimiter}/usr/share`,
);
});
+45
View File
@@ -159,6 +159,9 @@ interface RofiIconEntry {
iconPath?: string;
}
const ROFI_THUMBNAILER_FILE = 'subminer-ffmpegthumbnailer.thumbnailer';
const DEFAULT_XDG_DATA_DIRS = ['/usr/local/share', '/usr/share'];
function showRofiIconMenu(
entries: RofiIconEntry[],
prompt: string,
@@ -389,6 +392,47 @@ export function findRofiTheme(scriptPath: string): string | null {
return null;
}
export function findRofiThumbnailerDataRoot(scriptPath: string): string | null {
if (process.platform !== 'linux') return null;
const scriptDir = path.dirname(realpathMaybe(scriptPath));
const xdgDataHome = process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local/share');
const roots = [
path.join(xdgDataHome, 'SubMiner'),
path.posix.join('/usr/local/share/SubMiner'),
path.posix.join('/usr/share/SubMiner'),
path.join(scriptDir, 'assets'),
path.join(scriptDir, '..', 'assets'),
];
for (const root of roots) {
if (fs.existsSync(path.join(root, 'thumbnailers', ROFI_THUMBNAILER_FILE))) {
return root;
}
}
return null;
}
export function prependXdgDataDir(dataRoot: string, currentValue?: string): string {
const currentDirs = currentValue
? currentValue.split(path.delimiter).filter(Boolean)
: DEFAULT_XDG_DATA_DIRS;
return [dataRoot, ...currentDirs.filter((candidate) => candidate !== dataRoot)].join(
path.delimiter,
);
}
function buildRofiThumbnailEnvironment(scriptPath: string): NodeJS.ProcessEnv {
if (!commandExists('ffmpegthumbnailer')) return process.env;
const dataRoot = findRofiThumbnailerDataRoot(scriptPath);
if (!dataRoot) return process.env;
return {
...process.env,
XDG_DATA_DIRS: prependXdgDataDir(dataRoot, process.env.XDG_DATA_DIRS),
};
}
export function showRofiMenu(
videos: string[],
dir: string,
@@ -420,6 +464,7 @@ export function showRofiMenu(
const result = spawnSync('rofi', args, {
input: buildRofiMenu(videos, dir, recursive),
encoding: 'utf8',
env: buildRofiThumbnailEnvironment(scriptPath),
stdio: ['pipe', 'pipe', 'ignore'],
});
if (result.error) {
+136 -9
View File
@@ -1,6 +1,8 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import {
ensureLinuxRuntimePluginAvailable,
installManagedPluginAssetsViaApp,
@@ -31,7 +33,7 @@ test('ensureLinuxRuntimePluginAvailable is a no-op on non-Linux platforms', asyn
assert.deepEqual(calls, []);
});
test('ensureLinuxRuntimePluginAvailable skips install when installed global plugin and managed theme exist', async () => {
test('ensureLinuxRuntimePluginAvailable skips install when plugin, theme, and thumbnailer exist', async () => {
const calls: string[] = [];
await ensureLinuxRuntimePluginAvailable({
@@ -52,13 +54,17 @@ test('ensureLinuxRuntimePluginAvailable skips install when installed global plug
calls.push('theme');
return true;
},
isManagedThumbnailerAvailable: () => {
calls.push('thumbnailer');
return true;
},
log: () => {},
});
assert.deepEqual(calls, ['detect', 'theme']);
assert.deepEqual(calls, ['detect', 'theme', 'thumbnailer']);
});
test('ensureLinuxRuntimePluginAvailable skips install when managed runtime path and theme already resolve', async () => {
test('ensureLinuxRuntimePluginAvailable skips install when all managed assets resolve', async () => {
const calls: string[] = [];
await ensureLinuxRuntimePluginAvailable({
@@ -80,14 +86,19 @@ test('ensureLinuxRuntimePluginAvailable skips install when managed runtime path
calls.push('theme');
return true;
},
isManagedThumbnailerAvailable: () => {
calls.push('thumbnailer');
return true;
},
log: () => {},
});
assert.deepEqual(calls, ['detect', 'resolve', 'theme']);
assert.deepEqual(calls, ['detect', 'resolve', 'theme', 'thumbnailer']);
});
test('ensureLinuxRuntimePluginAvailable installs managed assets when rofi theme is missing', async () => {
const calls: string[] = [];
let themeAvailable = false;
await ensureLinuxRuntimePluginAvailable({
platform: 'linux',
@@ -102,10 +113,15 @@ test('ensureLinuxRuntimePluginAvailable installs managed assets when rofi theme
},
isManagedThemeAvailable: () => {
calls.push('theme');
return false;
return themeAvailable;
},
isManagedThumbnailerAvailable: () => {
calls.push('thumbnailer');
return true;
},
installManagedPluginAssets: async () => {
calls.push('install');
themeAvailable = true;
return { ok: true, status: 'installed', path: '/tmp/plugin/main.lua' };
},
log: (level, _configured, message) => {
@@ -117,13 +133,68 @@ test('ensureLinuxRuntimePluginAvailable installs managed assets when rofi theme
'detect',
'resolve',
'theme',
'info:Linux runtime support assets missing; installing managed plugin/theme assets.',
'info:Linux runtime support assets missing; installing managed plugin/theme/thumbnailer assets.',
'install',
'info:Managed Linux runtime support assets installed: plugin=/tmp/plugin/main.lua theme=/tmp/xdg-data/SubMiner/themes/subminer.rasi',
'info:Managed Linux runtime support assets installed: plugin=/tmp/plugin/main.lua theme=/tmp/xdg-data/SubMiner/themes/subminer.rasi thumbnailer=/tmp/xdg-data/SubMiner/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer',
'resolve',
'theme',
'thumbnailer',
]);
});
test('ensureLinuxRuntimePluginAvailable installs managed assets when thumbnailer is missing', async () => {
const calls: string[] = [];
let thumbnailerAvailable = false;
await ensureLinuxRuntimePluginAvailable({
platform: 'linux',
xdgDataHome: '/tmp/xdg-data',
detectInstalledPlugin: () => true,
resolveRuntimePluginPath: () => '/tmp/plugin/main.lua',
isManagedThemeAvailable: () => true,
isManagedThumbnailerAvailable: () => thumbnailerAvailable,
installManagedPluginAssets: async () => {
calls.push('install');
thumbnailerAvailable = true;
return { ok: true, status: 'installed', path: '/tmp/plugin/main.lua' };
},
log: (_level, _configured, message) => {
calls.push(message);
},
});
assert.deepEqual(calls, [
'Linux runtime support assets missing; installing managed plugin/theme/thumbnailer assets.',
'install',
'Managed Linux runtime support assets installed: plugin=/tmp/plugin/main.lua theme=/tmp/xdg-data/SubMiner/themes/subminer.rasi thumbnailer=/tmp/xdg-data/SubMiner/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer',
]);
});
test('ensureLinuxRuntimePluginAvailable retains an installed plugin after installing support assets', async () => {
const calls: string[] = [];
let thumbnailerAvailable = false;
await ensureLinuxRuntimePluginAvailable({
platform: 'linux',
xdgDataHome: '/tmp/xdg-data',
detectInstalledPlugin: () => true,
resolveRuntimePluginPath: () => {
calls.push('resolve');
return null;
},
isManagedThemeAvailable: () => true,
isManagedThumbnailerAvailable: () => thumbnailerAvailable,
installManagedPluginAssets: async () => {
calls.push('install');
thumbnailerAvailable = true;
return { ok: true, status: 'installed', path: '/tmp/plugin/main.lua' };
},
log: () => {},
});
assert.deepEqual(calls, ['install']);
});
test('ensureLinuxRuntimePluginAvailable installs managed assets and re-resolves plugin path', async () => {
const calls: string[] = [];
let resolveCount = 0;
@@ -137,6 +208,8 @@ test('ensureLinuxRuntimePluginAvailable installs managed assets and re-resolves
calls.push(`resolve:${resolveCount}`);
return resolveCount === 1 ? null : '/tmp/plugin/main.lua';
},
isManagedThemeAvailable: () => true,
isManagedThumbnailerAvailable: () => true,
installManagedPluginAssets: async () => {
calls.push('install');
return { ok: true, status: 'installed', path: '/tmp/plugin/main.lua' };
@@ -148,9 +221,9 @@ test('ensureLinuxRuntimePluginAvailable installs managed assets and re-resolves
assert.deepEqual(calls, [
'resolve:1',
'info:Linux runtime support assets missing; installing managed plugin/theme assets.',
'info:Linux runtime support assets missing; installing managed plugin/theme/thumbnailer assets.',
'install',
'info:Managed Linux runtime support assets installed: plugin=/tmp/plugin/main.lua theme=/tmp/xdg-data/SubMiner/themes/subminer.rasi',
'info:Managed Linux runtime support assets installed: plugin=/tmp/plugin/main.lua theme=/tmp/xdg-data/SubMiner/themes/subminer.rasi thumbnailer=/tmp/xdg-data/SubMiner/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer',
'resolve:2',
]);
});
@@ -191,6 +264,60 @@ test('ensureLinuxRuntimePluginAvailable fails when runtime path remains unresolv
);
});
test('ensureLinuxRuntimePluginAvailable fails when thumbnailer remains missing after install', async () => {
await assert.rejects(
() =>
ensureLinuxRuntimePluginAvailable({
platform: 'linux',
xdgDataHome: '/tmp/xdg-data',
detectInstalledPlugin: () => true,
resolveRuntimePluginPath: () => '/tmp/plugin/main.lua',
isManagedThemeAvailable: () => true,
isManagedThumbnailerAvailable: () => false,
installManagedPluginAssets: async () => ({
ok: true,
status: 'installed',
path: '/tmp/plugin/main.lua',
}),
log: () => {},
}),
/thumbnailer=.*subminer-ffmpegthumbnailer\.thumbnailer/i,
);
});
test('ensureLinuxRuntimePluginAvailable rejects a thumbnailer directory before and after install', async () => {
const xdgDataHome = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-thumbnailer-directory-'));
const thumbnailerPath = path.join(
xdgDataHome,
'SubMiner',
'thumbnailers',
'subminer-ffmpegthumbnailer.thumbnailer',
);
fs.mkdirSync(thumbnailerPath, { recursive: true });
const calls: string[] = [];
try {
await assert.rejects(
() =>
ensureLinuxRuntimePluginAvailable({
platform: 'linux',
xdgDataHome,
detectInstalledPlugin: () => true,
isManagedThemeAvailable: () => true,
installManagedPluginAssets: async () => {
calls.push('install');
return { ok: true, status: 'installed', path: '/tmp/plugin/main.lua' };
},
log: () => {},
}),
/thumbnailer=.*subminer-ffmpegthumbnailer\.thumbnailer/i,
);
assert.deepEqual(calls, ['install']);
} finally {
fs.rmSync(xdgDataHome, { recursive: true, force: true });
}
});
test('installManagedPluginAssetsViaApp returns launch errors without waiting for a response file', async () => {
let waited = false;
+22 -6
View File
@@ -31,6 +31,7 @@ type EnsureLinuxRuntimePluginAvailableOptions = {
detectInstalledPlugin?: () => boolean;
resolveRuntimePluginPath?: () => string | null;
isManagedThemeAvailable?: () => boolean;
isManagedThumbnailerAvailable?: () => boolean;
installManagedPluginAssets?: () => Promise<EnsureLinuxRuntimePluginAssetsResult>;
log?: PreflightLog;
};
@@ -48,6 +49,14 @@ function resolveConfiguredLogLevel(
return logLevel ?? 'warn';
}
function isRegularFile(filePath: string): boolean {
try {
return fs.statSync(filePath).isFile();
} catch {
return false;
}
}
async function waitForInstallResponse(
responsePath: string,
): Promise<RuntimePluginPreflightResponse | null> {
@@ -170,15 +179,17 @@ export async function ensureLinuxRuntimePluginAvailable(
});
const isManagedThemeAvailable =
options.isManagedThemeAvailable ?? (() => fs.existsSync(managedPaths.themePath));
const isManagedThumbnailerAvailable =
options.isManagedThumbnailerAvailable ?? (() => isRegularFile(managedPaths.thumbnailerPath));
const runtimePluginAvailable = installedPluginAvailable || Boolean(resolveRuntimePluginPath());
if (runtimePluginAvailable && isManagedThemeAvailable()) {
if (runtimePluginAvailable && isManagedThemeAvailable() && isManagedThumbnailerAvailable()) {
return;
}
log(
'info',
configuredLogLevel,
'Linux runtime support assets missing; installing managed plugin/theme assets.',
'Linux runtime support assets missing; installing managed plugin/theme/thumbnailer assets.',
);
const installManagedPluginAssets =
options.installManagedPluginAssets ??
@@ -207,16 +218,21 @@ export async function ensureLinuxRuntimePluginAvailable(
log(
'info',
configuredLogLevel,
`Managed Linux runtime support assets installed: plugin=${installResult.path ?? 'unknown path'} theme=${managedPaths.themePath}`,
`Managed Linux runtime support assets installed: plugin=${installResult.path ?? 'unknown path'} theme=${managedPaths.themePath} thumbnailer=${managedPaths.thumbnailerPath}`,
);
const runtimePluginPath = resolveRuntimePluginPath();
if (runtimePluginPath) {
const runtimePluginAvailableAfterInstall =
installedPluginAvailable || Boolean(resolveRuntimePluginPath());
if (
runtimePluginAvailableAfterInstall &&
isManagedThemeAvailable() &&
isManagedThumbnailerAvailable()
) {
return;
}
const message =
`Linux managed runtime plugin assets could not be installed. ` +
`Checked path: ${managedPaths.pluginEntrypointPath}. ` +
`Checked paths: plugin=${managedPaths.pluginEntrypointPath} theme=${managedPaths.themePath} thumbnailer=${managedPaths.thumbnailerPath}. ` +
'Launch aborted before starting mpv.';
log('warn', configuredLogLevel, message);
throw new Error(message);
+15 -1
View File
@@ -165,11 +165,14 @@ if (entry.argv.includes('--ensure-linux-runtime-plugin-assets')) {
const pluginDir = path.join(dataDir, 'plugin', 'subminer');
const pluginConfigPath = path.join(dataDir, 'plugin', 'subminer.conf');
const themePath = path.join(dataDir, 'themes', 'subminer.rasi');
const thumbnailerPath = path.join(dataDir, 'thumbnailers', 'subminer-ffmpegthumbnailer.thumbnailer');
fs.mkdirSync(pluginDir, { recursive: true });
fs.mkdirSync(path.dirname(themePath), { recursive: true });
fs.mkdirSync(path.dirname(thumbnailerPath), { recursive: true });
fs.writeFileSync(path.join(pluginDir, 'main.lua'), '-- smoke plugin\\n');
fs.writeFileSync(pluginConfigPath, 'smoke=true\\n');
fs.writeFileSync(themePath, '/* smoke theme */\\n');
fs.writeFileSync(thumbnailerPath, '[Thumbnailer Entry]\\n');
if (responsePath) {
fs.mkdirSync(path.dirname(responsePath), { recursive: true });
fs.writeFileSync(responsePath, JSON.stringify({ ok: true, status: 'installed', path: path.join(pluginDir, 'main.lua') }));
@@ -620,11 +623,22 @@ test(
);
assert.match(result.stdout, /pause mpv until overlay and tokenization are ready/i);
if (process.platform === 'linux') {
assert.match(result.stdout, /managed plugin\/theme assets/i);
assert.match(result.stdout, /managed plugin\/theme\/thumbnailer assets/i);
assert.equal(
fs.existsSync(path.join(smokeCase.xdgDataHome, 'SubMiner', 'themes', 'subminer.rasi')),
true,
);
assert.equal(
fs.existsSync(
path.join(
smokeCase.xdgDataHome,
'SubMiner',
'thumbnailers',
'subminer-ffmpegthumbnailer.thumbnailer',
),
),
true,
);
}
});
},
@@ -18,6 +18,9 @@ export function createImmersionDbFixture(dbPath: string): void {
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0)`,
).run();
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('lexical_daily_rollups_version', 0)`,
).run();
db.prepare(
`INSERT INTO imm_lifetime_global(global_id, CREATED_DATE, LAST_UPDATE_DATE) VALUES (1, ?, ?)`,
).run(String(Date.now()), String(Date.now()));
@@ -108,6 +108,36 @@ test('fixture schema stays aligned with production sync-touched tables and index
}
});
test('fixture leaves lexical rollups pending when their table is absent', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-rollup-state-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
try {
createImmersionDbFixture(fixturePath);
const db = new BunDatabase(fixturePath, { readonly: true });
try {
const state = db
.query<{ state_value: string }>(
`SELECT state_value FROM imm_rollup_state
WHERE state_key = 'lexical_daily_rollups_version'`,
)
.get();
const rollupTable = db
.query<{ name: string }>(
`SELECT name FROM sqlite_schema
WHERE type = 'table' AND name = 'imm_lexical_daily_rollups'`,
)
.get();
assert.equal(state?.state_value, '0');
assert.equal(rollupTable, null);
} finally {
db.close();
}
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
});
test('fixture session inserts enforce foreign keys', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-foreign-keys-'));
const fixturePath = path.join(dir, 'fixture.sqlite');
@@ -154,6 +154,7 @@ export const IMMERSION_DB_FIXTURE_DDL = `
last_seen REAL,
frequency INTEGER,
frequency_rank INTEGER,
vocabulary_visible INTEGER NOT NULL DEFAULT 1 CHECK(vocabulary_visible IN (0, 1)),
UNIQUE(headword, word, reading)
);
CREATE TABLE imm_kanji(
+2 -1
View File
@@ -2,7 +2,7 @@
"name": "subminer",
"productName": "SubMiner",
"desktopName": "SubMiner.desktop",
"version": "0.19.4-beta.1",
"version": "0.19.4-beta.4",
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
"packageManager": "bun@1.3.5",
"main": "dist/main-entry.js",
@@ -32,6 +32,7 @@
"changelog:pr-check": "bun run scripts/build-changelog.ts pr-check",
"changelog:release-notes": "bun run scripts/build-changelog.ts release-notes",
"changelog:prerelease-notes": "bun run scripts/build-changelog.ts prerelease-notes",
"changelog:check-prerelease-notes": "bun run scripts/build-changelog.ts check-prerelease-notes",
"format": "prettier --write .",
"format:check": "prettier --check .",
"format:src": "bash scripts/prettier-scope.sh --write",
+2
View File
@@ -58,6 +58,8 @@ package() {
"${pkgdir}/usr/share/SubMiner/plugin/subminer.conf"
install -Dm644 "${srcdir}/assets/themes/subminer.rasi" \
"${pkgdir}/usr/share/SubMiner/themes/subminer.rasi"
install -Dm644 "${srcdir}/assets/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer" \
"${pkgdir}/usr/share/SubMiner/thumbnailers/subminer-ffmpegthumbnailer.thumbnailer"
install -dm755 "${pkgdir}/usr/share/SubMiner/plugin/subminer"
cp -a "${srcdir}/plugin/subminer/." "${pkgdir}/usr/share/SubMiner/plugin/subminer/"
+26
View File
@@ -7,6 +7,8 @@ local OVERLAY_RESTART_PING_MAX_ATTEMPTS = 20
local OVERLAY_LOADING_OSD_PREFIX = "Overlay loading "
local OVERLAY_LOADING_OSD_FRAMES = { "|", "/", "-", "\\" }
local OVERLAY_LOADING_OSD_REFRESH_SECONDS = 0.18
local OVERLAY_LOADING_OSD_DEADLINE_SECONDS = 30
local OVERLAY_LOADING_OSD_TIMEOUT_MESSAGE = "Overlay did not become ready; check SubMiner logs"
local AUTO_PLAY_READY_LOADING_OSD = "Loading subtitle tokenization..."
local AUTO_PLAY_READY_READY_OSD = "Subtitle tokenization ready"
local DEFAULT_AUTO_PLAY_READY_TIMEOUT_SECONDS = 30
@@ -265,10 +267,19 @@ function M.create(ctx)
state.overlay_loading_osd_timer = nil
end
local function clear_overlay_loading_osd_deadline()
local timeout = state.overlay_loading_osd_deadline
if timeout and timeout.kill then
timeout:kill()
end
state.overlay_loading_osd_deadline = nil
end
local function stop_overlay_loading_osd()
state.overlay_loading_osd_active = false
state.overlay_loading_osd_frame = 1
clear_overlay_loading_osd_timer()
clear_overlay_loading_osd_deadline()
end
local function start_overlay_loading_osd()
@@ -291,6 +302,21 @@ function M.create(ctx)
end
end)
end
if type(mp.add_timeout) == "function" then
state.overlay_loading_osd_deadline = mp.add_timeout(OVERLAY_LOADING_OSD_DEADLINE_SECONDS, function()
if not state.overlay_loading_osd_active then
return
end
state.overlay_loading_osd_deadline = nil
stop_overlay_loading_osd()
subminer_log(
"warn",
"process",
"Overlay loading deadline expired before the app reported content ready"
)
show_osd(OVERLAY_LOADING_OSD_TIMEOUT_MESSAGE, { force = true })
end)
end
end
local function disarm_auto_play_ready_gate(options)
+1
View File
@@ -26,6 +26,7 @@ function M.new()
auto_play_ready_initial_pause_ownership_consumed = false,
overlay_loading_osd_active = false,
overlay_loading_osd_timer = nil,
overlay_loading_osd_deadline = nil,
overlay_loading_osd_frame = 1,
pending_visible_overlay_hide_timer = nil,
pending_visible_overlay_hide_generation = 0,
+44 -28
View File
@@ -4,37 +4,46 @@
## Highlights
### Added
- **Library Merge and Move**
- Duplicate library cards for the same show can now be combined: select cards in the library grid, choose "Merge Selected," and pick which entry to keep. Sessions, mined cards, and watch time move over, and future episodes stay matched to the merged card.
- Episodes can be reassigned to a different library entry with a "→" button on the episode row, fixing cases where a file lands under the wrong title. Manual assignments survive later filename parsing, Jellyfin refreshes, and season repair.
- Exact AniList matches with compatible seasons now merge automatically, while fuzzy matches show up as a dismissible "Possible duplicate" prompt instead of merging without confirmation.
- Library Merge & Reassignment
- Duplicate library cards for the same show can be combined: select entries in "Select" mode and use "Merge Selected" to combine their sessions, mined cards, and watch time onto one card.
- Episodes can be moved to a different entry with a per-episode "→" button, fixing stray files that split off their own entry; manual assignments now survive later filename parsing, Jellyfin refreshes, and season repair.
- Exact AniList matches with compatible seasons merge automatically, while likely (fuzzy) matches surface as a dismissible "Possible duplicate" suggestion instead of merging silently.
- Duplicate Line Cleanup
- The Vocabulary tab's new **Duplicates** button scans a chosen time window for the repeated-line bursts described under Fixed below and collapses each burst to a single line once confirmed.
- A matching `subminer stats cleanup --duplicate-lines` command (with `--dry-run` and `--lookback-days <n>`) is available from the terminal.
- Only the affected subtitle lines and the vocabulary counts they inflated are touched; watch time and lines-seen totals are left as recorded.
### Fixed
- **Anki Audio Generation on Network Drives**
- Fixed sentence-audio generation timing out on slow network-mounted video files with many subtitle and font-attachment streams.
- Extraction now uses bounded probing and a two-minute budget, and failures show a clear error instead of a cryptic one.
- **Duplicate Subtitle Line Stats**
- Fixed karaoke openings and animated signs (which record one subtitle event per animation frame) inflating word and kanji counts and skewing "Top Repeated Words." Ordinary repeated dialogue and rewatches are unaffected.
- Already-inflated stats can be cleaned up with the new "Duplicates" button in the Vocabulary tab, or `subminer stats cleanup --duplicate-lines` (supports `--dry-run` and `--lookback-days`). Only the affected subtitle lines and vocabulary counts are touched; watch time and lines-seen totals are untouched.
- **Overlay Modals on macOS and Windows**
- Fixed overlay modals and the stats window opening on the wrong macOS Space, or forcing a Space switch, when mpv is fullscreen. They now open above fullscreen mpv on its current Space.
- Modals are now prewarmed on macOS and Windows so shortcuts open them promptly, and Windows keeps the hidden modal responsive between sessions.
- **Wayland File Drag-and-Drop**
- Fixed dragging subtitle and video files from file managers like Thunar onto the overlay on native Wayland; dropped files are now resolved and sent to mpv.
- **Windows Mouse Lag**
- Fixed system-wide mouse lag while SubMiner is running on Windows, caused by a global mouse hook and blocking window lookups during click-through tracking.
- **Mining Clip Accuracy**
- Fixed mined audio and animated image clips sometimes capturing the wrong subtitle line when audio extraction was slow. The clip range is now locked in at the moment of lookup, so audio and image clips always match.
- **Linux Notifications**
- Character dictionary progress notifications on Linux now update in place instead of flickering off and back on with every status change.
- **Stats Delete Performance**
- Fixed stats deletes freezing the dashboard; deletes now reliably run off the main thread, with automatic retry if the delete worker crashes.
- Deletes, library merges/moves, and AniList reassignments are now much faster because totals are updated incrementally instead of rebuilt from scratch, and no longer erase lifetime totals older than the recent session history.
- Session deletes on large libraries dropped from minutes to milliseconds.
### Docs
- **Feature Demos Page**
- Hidden the unfinished feature demos page from the documentation sidebar; it's still reachable by direct URL.
- Subtitle Duplication from Karaoke & Animated Signs
- Typeset ASS karaoke and animated signs no longer flood the overlay, subtitle sidebar, immersion history, mined cards, or stats with repeated glyph fragments or per-frame duplicates; the complete authored line is recovered instead, without merging genuinely repeated dialogue or separately positioned signs.
- Fragmented karaoke now preserves the spaces the author placed between words instead of joining them together, and lyric transitions (including seeking into the middle of a line) resolve to the clean line instead of a stray entrance or exit frame.
- The secondary overlay shares the same deduplication logic as the primary overlay, including collapsing lines that differ only by whitespace or trailing punctuation, and sidebar navigation moves between clean lyric lines while keeping the right line selected.
- Anki Media Generation
- Sentence-audio generation no longer times out on slow network-mounted video files with many subtitle and font streams, and a failed extraction now reports a clear error instead of a raw `ENOENT`.
- Mined audio and animated AVIF clips now capture the subtitle line you actually mined, instead of whatever line happened to be on screen once slow audio extraction finished.
- Character Dictionary Performance & Notifications
- Character dictionary generation, merged rebuilds, and imports no longer freeze the app on large dictionaries, and cached results (including character portraits) are reused across launches instead of regenerating everything every time.
- Portraits also now display correctly if their cache finishes loading after subtitles have already started showing.
- Desktop progress notifications, including on Linux AppImage installs, now update in place instead of flickering closed and reopening.
- Overlay Reliability
- Overlay modals (settings, stats, etc.) now open promptly on the first shortcut press, including on repeated sessions on Windows, and appear above fullscreen mpv on macOS instead of switching Spaces or opening off-screen.
- The macOS window-tracking helper is now built for macOS 12.0+, so the overlay attaches to mpv on older systems like Ventura instead of crashing and getting stuck on "Overlay loading."
- The overlay no longer gets stuck on "Overlay loading" indefinitely if mpv's connection stalls; it now retries and shows an actionable error after 30 seconds.
- Fixed native Wayland drag-and-drop from file managers like Thunar, and fixed system-wide mouse lag on Windows caused by the overlay's click-through handling.
- Stats Dashboard
- Deletes, library merges, video moves, and AniList reassignments no longer freeze the stats dashboard or rebuild lifetime totals from scratch; large deletes that used to take minutes now finish in milliseconds.
- Vocabulary totals and charts now count all tracked vocabulary instead of just the first page, and new-word history uses corrected daily rollups.
- Calendar labels respect time zones west of UTC, and vocabulary cards refresh automatically after editing the word exclusion list (with a Retry option if a load fails).
- Linux Launcher Thumbnails
- Fixed missing MKV thumbnails in the Linux rofi picker when the system thumbnailer only registers legacy Matroska MIME aliases.
## What's Changed
@@ -47,6 +56,13 @@
- fix(overlay): support native Wayland file drag-and-drop by @ksyasuda in #199
- fix(overlay): keep macOS modal windows on fullscreen Spaces by @ksyasuda in #200
- fix(overlay): prevent Windows mouse lag during click-through tracking by @ksyasuda in #201
- fix(stats): report complete vocabulary totals and new-word history by @ksyasuda in #202
- fix(mpv): recover from stalled IPC connects by @ksyasuda in #204
- fix(dictionary): prevent freezes and restore AppImage notifications by @ksyasuda in #205
- fix(subtitles): recover canonical lines from ASS animation by @ksyasuda in #207
- fix(overlay): deduplicate secondary subtitle rendering by @ksyasuda in #208
- fix(launcher): restore Matroska thumbnails in Linux rofi picker by @ksyasuda in #210
- fix(character-dictionary): cache completed MeCab refreshes by @ksyasuda in #212
## Installation
+378 -6
View File
@@ -1,4 +1,5 @@
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
import test from 'node:test';
@@ -583,7 +584,7 @@ test('writePrereleaseNotesForVersion writes cumulative beta notes without mutati
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.11.3-beta.1',
deps: { runClaude: stub.runClaude },
deps: { runClaude: stub.runClaude, listPrereleaseTags: () => [] },
});
assert.equal(outputPath, path.join(projectRoot, 'release', 'prerelease-notes.md'));
@@ -605,7 +606,8 @@ test('writePrereleaseNotesForVersion writes cumulative beta notes without mutati
const prereleaseNotes = fs.readFileSync(outputPath, 'utf8');
assert.match(prereleaseNotes, /^> This is a prerelease build for testing\./m);
assert.match(prereleaseNotes, /<!-- prerelease-base-version: 0\.11\.3 -->/);
assert.match(prereleaseNotes, /<!-- prerelease-version: 0\.11\.3-beta\.1 -->/);
assert.doesNotMatch(prereleaseNotes, /## Changes since /);
assert.match(prereleaseNotes, /## Highlights\n### Added\n- Polished: added entry\./);
assert.match(prereleaseNotes, /### Fixed\n- Polished: fixed entry\./);
assert.match(prereleaseNotes, /## Installation\n\nSee the README and docs\/installation guide/);
@@ -668,7 +670,7 @@ test('writePrereleaseNotesForVersion reuses existing prerelease notes when addin
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.11.3-beta.2',
deps: { runClaude: stub.runClaude },
deps: { runClaude: stub.runClaude, listPrereleaseTags: () => [] },
});
assert.equal(stub.calls.length, 1, 'prerelease should issue exactly one Claude call');
@@ -723,7 +725,7 @@ test('writePrereleaseNotesForVersion ignores unmarked prerelease notes from an o
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.17.0-beta.1',
deps: { runClaude: stub.runClaude },
deps: { runClaude: stub.runClaude, listPrereleaseTags: () => [] },
});
assert.equal(stub.calls.length, 1, 'prerelease should issue exactly one Claude call');
@@ -790,7 +792,7 @@ test('writePrereleaseNotesForVersion prompts Claude to revise stale prerelease b
writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.12.0-beta.2',
deps: { runClaude: stub.runClaude },
deps: { runClaude: stub.runClaude, listPrereleaseTags: () => [] },
});
assert.equal(stub.calls.length, 1, 'prerelease should issue exactly one Claude call');
@@ -830,7 +832,7 @@ test('writePrereleaseNotesForVersion supports rc prereleases', async () => {
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.11.3-rc.1',
deps: { runClaude: stub.runClaude },
deps: { runClaude: stub.runClaude, listPrereleaseTags: () => [] },
});
const prereleaseNotes = fs.readFileSync(outputPath, 'utf8');
@@ -1447,3 +1449,373 @@ test('writeChangelogArtifacts strips <details> blocks from release notes when re
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('selectPreviousPrereleaseTag orders betas before rcs and filters other base versions', async () => {
const { selectPreviousPrereleaseTag } = await loadModule();
const tags = [
'v0.19.4-beta.1',
'v0.19.4-beta.3',
'v0.19.4-beta.2',
'v0.19.3-beta.9',
'v0.19.4-rc.1',
'not-a-tag',
];
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-beta.1'), null);
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-beta.2'), 'v0.19.4-beta.1');
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-beta.4'), 'v0.19.4-beta.3');
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-rc.1'), 'v0.19.4-beta.3');
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-rc.2'), 'v0.19.4-rc.1');
// Regenerating notes for an already-tagged version must not pick itself.
assert.equal(selectPreviousPrereleaseTag(tags, '0.19.4-beta.3'), 'v0.19.4-beta.2');
assert.equal(selectPreviousPrereleaseTag(['v0.19.3-beta.1'], '0.19.4-beta.2'), null);
});
test('writePrereleaseNotesForVersion adds a delta section generated from fragment diffs', async () => {
const { writePrereleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('prerelease-delta-section');
const projectRoot = path.join(workspace, 'SubMiner');
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.12.0-beta.2' }, null, 2),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: fixed', 'area: overlay', '', '- Fixed overlay focus and macOS helper.'].join('\n'),
'utf8',
);
try {
const stub = recordingRunClaude((input) =>
input.includes('MODIFIED FRAGMENT')
? '- Fixed the macOS helper deployment target for older systems.'
: '### Fixed\n- Overlay: cumulative fixed entry.',
);
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.12.0-beta.2',
deps: {
runClaude: stub.runClaude,
listPrereleaseTags: () => ['v0.12.0-beta.1'],
resolveFragmentDelta: (_cwd, previousTag) => {
assert.equal(previousTag, 'v0.12.0-beta.1');
return [
{
path: 'changes/002.md',
status: 'added',
after: 'type: fixed\narea: macos\n\n- Fixed helper deployment target.',
},
{
path: 'changes/001.md',
status: 'modified',
before: '- Fixed overlay focus.',
after: '- Fixed overlay focus and macOS helper.',
},
{
path: 'changes/003.md',
status: 'deleted',
before: 'type: added\narea: stats\n\n- Reverted experimental stats view.',
},
];
},
},
});
assert.equal(stub.calls.length, 2, 'delta and cumulative polish are separate Claude calls');
const deltaPrompt = stub.calls[0]!.input;
assert.match(deltaPrompt, /ADDED FRAGMENT changes\/002\.md/);
assert.match(deltaPrompt, /MODIFIED FRAGMENT changes\/001\.md/);
assert.match(deltaPrompt, /BEFORE:\n- Fixed overlay focus\./);
assert.match(deltaPrompt, /AFTER:\n- Fixed overlay focus and macOS helper\./);
assert.match(deltaPrompt, /DELETED FRAGMENT changes\/003\.md/);
assert.match(deltaPrompt, /If the edit is editorial/);
assert.match(deltaPrompt, /removed or reverted/);
assert.match(deltaPrompt, /No user-facing changes since v0\.12\.0-beta\.1\./);
assert.equal(modeFromPrompt(stub.calls[1]!.input), 'release-notes');
const prereleaseNotes = fs.readFileSync(outputPath, 'utf8');
assert.match(
prereleaseNotes,
/<!-- prerelease-version: 0\.12\.0-beta\.2; since: v0\.12\.0-beta\.1 -->/,
);
const deltaIndex = prereleaseNotes.indexOf('## Changes since v0.12.0-beta.1');
const highlightsIndex = prereleaseNotes.indexOf('## Highlights');
assert.ok(deltaIndex !== -1, 'delta section heading should be present');
assert.ok(deltaIndex < highlightsIndex, 'delta section should precede Highlights');
assert.match(prereleaseNotes, /- Fixed the macOS helper deployment target for older systems\./);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('writePrereleaseNotesForVersion renders a fallback delta line when no fragments changed', async () => {
const { writePrereleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('prerelease-empty-delta');
const projectRoot = path.join(workspace, 'SubMiner');
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.12.0-beta.3' }, null, 2),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: fixed', 'area: overlay', '', '- Fixed overlay focus.'].join('\n'),
'utf8',
);
try {
const stub = defaultStubClaude();
const outputPath = writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.12.0-beta.3',
deps: {
runClaude: stub.runClaude,
listPrereleaseTags: () => ['v0.12.0-beta.1', 'v0.12.0-beta.2'],
resolveFragmentDelta: () => [],
},
});
assert.equal(stub.calls.length, 1, 'empty delta must not spend a Claude call');
const prereleaseNotes = fs.readFileSync(outputPath, 'utf8');
assert.match(
prereleaseNotes,
/## Changes since v0\.12\.0-beta\.2\n\n- No changelog fragment changes since v0\.12\.0-beta\.2; this build contains packaging or internal-only updates\./,
);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('writePrereleaseNotesForVersion rejects non-bullet delta output from Claude', async () => {
const { writePrereleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('prerelease-delta-invalid-output');
const projectRoot = path.join(workspace, 'SubMiner');
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.12.0-beta.2' }, null, 2),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: fixed', 'area: overlay', '', '- Fixed overlay focus.'].join('\n'),
'utf8',
);
try {
const stub = recordingRunClaude(() => 'Here are the changes:\n- One change.');
assert.throws(
() =>
writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.12.0-beta.2',
deps: {
runClaude: stub.runClaude,
listPrereleaseTags: () => ['v0.12.0-beta.1'],
resolveFragmentDelta: () => [
{ path: 'changes/001.md', status: 'added', after: '- Fixed overlay focus.' },
],
},
}),
/delta output must contain only Markdown bullets/,
);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('writePrereleaseNotesForVersion strips the stale delta section from the reused baseline', async () => {
const { writePrereleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('prerelease-reuse-strips-delta');
const projectRoot = path.join(workspace, 'SubMiner');
const existingNotes = [
'> This is a prerelease build for testing. Stable changelog and docs-site updates remain pending until the final stable release.',
'',
'<!-- prerelease-version: 0.12.0-beta.2; since: v0.12.0-beta.1 -->',
'',
'## Changes since v0.12.0-beta.1',
'',
'- Stale beta-to-beta delta bullet.',
'',
'## Highlights',
'### Added',
'- Overlay: Previous beta entry.',
'',
'## Installation',
'',
'See the README and docs/installation guide for full setup steps.',
'',
].join('\n');
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.mkdirSync(path.join(projectRoot, 'release'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.12.0-beta.3' }, null, 2),
'utf8',
);
fs.writeFileSync(path.join(projectRoot, 'release', 'prerelease-notes.md'), existingNotes, 'utf8');
fs.writeFileSync(
path.join(projectRoot, 'changes', '001.md'),
['type: added', 'area: overlay', '', '- Added overlay coverage.'].join('\n'),
'utf8',
);
try {
const stub = defaultStubClaude();
writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.12.0-beta.3',
deps: {
runClaude: stub.runClaude,
listPrereleaseTags: () => [],
resolveFragmentDelta: () => [],
},
});
assert.equal(stub.calls.length, 1);
const prompt = stub.calls[0]!.input;
assert.match(prompt, /EXISTING PRERELEASE NOTES/);
assert.match(prompt, /Overlay: Previous beta entry\./);
assert.doesNotMatch(prompt, /Stale beta-to-beta delta bullet\./);
assert.doesNotMatch(prompt, /## Changes since /);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('verifyPrereleaseNotesMatchVersion accepts matching notes and rejects stale or legacy markers', async () => {
const { verifyPrereleaseNotesMatchVersion } = await loadModule();
const workspace = createWorkspace('verify-prerelease-notes');
const projectRoot = path.join(workspace, 'SubMiner');
const notesPath = path.join(projectRoot, 'release', 'prerelease-notes.md');
fs.mkdirSync(path.join(projectRoot, 'release'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.12.0-beta.2' }, null, 2),
'utf8',
);
try {
assert.throws(
() => verifyPrereleaseNotesMatchVersion({ cwd: projectRoot, version: '0.12.0-beta.2' }),
/Missing .*prerelease-notes\.md/,
);
fs.writeFileSync(
notesPath,
'<!-- prerelease-version: 0.12.0-beta.2; since: v0.12.0-beta.1 -->\n\n## Highlights\n',
'utf8',
);
verifyPrereleaseNotesMatchVersion({ cwd: projectRoot, version: '0.12.0-beta.2' });
verifyPrereleaseNotesMatchVersion({ cwd: projectRoot, version: 'v0.12.0-beta.2' });
fs.writeFileSync(
notesPath,
'<!-- prerelease-version: 0.12.0-beta.1 -->\n\n## Highlights\n',
'utf8',
);
assert.throws(
() => verifyPrereleaseNotesMatchVersion({ cwd: projectRoot, version: '0.12.0-beta.2' }),
/generated for 0\.12\.0-beta\.1 but this release is 0\.12\.0-beta\.2/,
);
fs.writeFileSync(
notesPath,
'<!-- prerelease-base-version: 0.12.0 -->\n\n## Highlights\n',
'utf8',
);
assert.throws(
() => verifyPrereleaseNotesMatchVersion({ cwd: projectRoot, version: '0.12.0-beta.2' }),
/missing or legacy prerelease-version marker/,
);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
test('default git tag listing and fragment delta resolution work against a real repository', async () => {
const { writePrereleaseNotesForVersion } = await loadModule();
const workspace = createWorkspace('prerelease-git-defaults');
const projectRoot = path.join(workspace, 'SubMiner');
const git = (...args: string[]): void => {
execFileSync('git', args, { cwd: projectRoot, stdio: 'ignore' });
};
fs.mkdirSync(path.join(projectRoot, 'changes'), { recursive: true });
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.11.3-beta.1' }, null, 2),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', 'kept.md'),
['type: added', 'area: overlay', '', '- Kept change.'].join('\n'),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', 'edited.md'),
['type: fixed', 'area: launcher', '', '- Original launcher fix.'].join('\n'),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'changes', 'removed.md'),
['type: added', 'area: stats', '', '- Reverted stats change.'].join('\n'),
'utf8',
);
try {
git('init', '--quiet');
git('-c', 'user.email=test@example.com', '-c', 'user.name=Test', 'add', '.');
git('-c', 'user.email=test@example.com', '-c', 'user.name=Test', 'commit', '-m', 'beta.1');
git('tag', 'v0.11.3-beta.1');
fs.writeFileSync(
path.join(projectRoot, 'changes', 'edited.md'),
['type: fixed', 'area: launcher', '', '- Broader launcher fix.'].join('\n'),
'utf8',
);
fs.rmSync(path.join(projectRoot, 'changes', 'removed.md'));
fs.writeFileSync(
path.join(projectRoot, 'changes', 'new.md'),
['type: added', 'area: anki', '', '- New anki change.'].join('\n'),
'utf8',
);
fs.writeFileSync(
path.join(projectRoot, 'package.json'),
JSON.stringify({ name: 'subminer', version: '0.11.3-beta.2' }, null, 2),
'utf8',
);
const stub = recordingRunClaude((input) =>
input.includes('PREVIOUS_TAG:') ? '- Delta bullet.' : defaultPolishedBody(input),
);
writePrereleaseNotesForVersion({
cwd: projectRoot,
version: '0.11.3-beta.2',
deps: { runClaude: stub.runClaude },
});
assert.equal(stub.calls.length, 2);
const deltaPrompt = stub.calls[0]!.input;
assert.match(deltaPrompt, /PREVIOUS_TAG: v0\.11\.3-beta\.1/);
assert.match(deltaPrompt, /ADDED FRAGMENT changes\/new\.md/);
assert.match(deltaPrompt, /- New anki change\./);
assert.match(deltaPrompt, /MODIFIED FRAGMENT changes\/edited\.md/);
assert.match(deltaPrompt, /- Original launcher fix\./);
assert.match(deltaPrompt, /- Broader launcher fix\./);
assert.match(deltaPrompt, /DELETED FRAGMENT changes\/removed\.md/);
assert.match(deltaPrompt, /- Reverted stats change\./);
assert.doesNotMatch(deltaPrompt, /kept\.md/);
} finally {
fs.rmSync(workspace, { recursive: true, force: true });
}
});
+304 -6
View File
@@ -18,6 +18,15 @@ type Contribution = {
// and the GitHub API.
type ResolveContributions = (fragmentPaths: string[], cwd: string) => Contribution[];
// One changelog fragment's change between the previous prerelease tag and the
// working tree. `before` is the content at the tag, `after` the current content.
export type FragmentDeltaEntry = {
path: string;
status: 'added' | 'modified' | 'deleted';
before?: string;
after?: string;
};
type ChangelogFsDeps = {
existsSync?: (candidate: string) => boolean;
mkdirSync?: (candidate: string, options: { recursive: true }) => void;
@@ -28,6 +37,8 @@ type ChangelogFsDeps = {
log?: (message: string) => void;
runClaude?: RunClaude;
resolveContributions?: ResolveContributions;
listPrereleaseTags?: (cwd: string, baseVersion: string) => string[];
resolveFragmentDelta?: (cwd: string, previousTag: string) => FragmentDeltaEntry[];
};
type PolishMode = 'changelog' | 'release-notes';
@@ -103,16 +114,57 @@ function resolvePrereleaseBaseVersion(version: string): string {
return match[1]!;
}
function renderPrereleaseBaseVersionMarker(version: string): string {
return `<!-- prerelease-base-version: ${resolvePrereleaseBaseVersion(version)} -->`;
// The marker records which exact prerelease the committed notes were generated
// for (and which prior tag the delta section compares against), so CI can
// reject notes that were prepared for a different beta/RC.
function renderPrereleaseVersionMarker(version: string, previousTag: string | null): string {
const since = previousTag ? `; since: ${previousTag}` : '';
return `<!-- prerelease-version: ${normalizeVersion(version)}${since} -->`;
}
export function extractPrereleaseVersionMarker(notes: string): string | null {
return (
/<!--\s*prerelease-version:\s*(\d+\.\d+\.\d+-(?:beta|rc)\.\d+)(?:;\s*since:\s*\S+)?\s*-->/u.exec(
notes,
)?.[1] ?? null
);
}
// Legacy marker written before the per-version marker existed. Still accepted
// when deciding whether existing notes can seed the cumulative baseline.
function extractPrereleaseBaseVersionMarker(notes: string): string | null {
const fullVersion = extractPrereleaseVersionMarker(notes);
if (fullVersion) {
return resolvePrereleaseBaseVersion(fullVersion);
}
return /<!--\s*prerelease-base-version:\s*(\d+\.\d+\.\d+)\s*-->/u.exec(notes)?.[1] ?? null;
}
const DELTA_SECTION_HEADING_PREFIX = '## Changes since ';
// Removes the previous run's "Changes since" section so the cumulative baseline
// fed back to Claude never carries a stale beta-to-beta delta.
function stripDeltaSection(notes: string): string {
const lines = notes.split(/\r?\n/);
const start = lines.findIndex((line) => line.startsWith(DELTA_SECTION_HEADING_PREFIX));
if (start === -1) {
return notes;
}
let end = lines.length;
for (let index = start + 1; index < lines.length; index += 1) {
if (lines[index]!.startsWith('## ')) {
end = index;
break;
}
}
return [...lines.slice(0, start), ...lines.slice(end)].join('\n');
}
function stripPrereleaseMetadata(notes: string): string {
return notes.replace(/<!--\s*prerelease-base-version:\s*\d+\.\d+\.\d+\s*-->\s*/u, '').trim();
return notes
.replace(/<!--\s*prerelease-version:[^>]*-->\s*/u, '')
.replace(/<!--\s*prerelease-base-version:\s*\d+\.\d+\.\d+\s*-->\s*/u, '')
.trim();
}
function resolveReusablePrereleaseNotes(notes: string, version: string): string | undefined {
@@ -120,7 +172,124 @@ function resolveReusablePrereleaseNotes(notes: string, version: string): string
if (existingBaseVersion !== resolvePrereleaseBaseVersion(version)) {
return undefined;
}
return stripPrereleaseMetadata(notes);
return stripPrereleaseMetadata(stripDeltaSection(notes));
}
type ParsedPrereleaseTag = {
tag: string;
base: string;
channel: 'beta' | 'rc';
iteration: number;
};
function parsePrereleaseTag(tag: string): ParsedPrereleaseTag | null {
const match = /^v?(\d+\.\d+\.\d+)-(beta|rc)\.(\d+)$/u.exec(tag.trim());
if (!match) {
return null;
}
return {
tag: tag.trim(),
base: match[1]!,
channel: match[2] as 'beta' | 'rc',
iteration: Number.parseInt(match[3]!, 10),
};
}
// Semver prerelease order: every beta sorts before every rc, then numerically.
function comparePrereleaseTags(a: ParsedPrereleaseTag, b: ParsedPrereleaseTag): number {
if (a.channel !== b.channel) {
return a.channel === 'beta' ? -1 : 1;
}
return a.iteration - b.iteration;
}
// Picks the newest prerelease tag for the same base version that strictly
// precedes the version being released. Returns null for the first prerelease.
export function selectPreviousPrereleaseTag(tags: string[], version: string): string | null {
const current = parsePrereleaseTag(normalizeVersion(version));
if (!current) {
return null;
}
const candidates = tags
.map(parsePrereleaseTag)
.filter((parsed): parsed is ParsedPrereleaseTag => parsed !== null)
.filter((parsed) => parsed.base === current.base)
.filter((parsed) => comparePrereleaseTags(parsed, current) < 0)
.sort(comparePrereleaseTags);
return candidates[candidates.length - 1]?.tag ?? null;
}
function defaultListPrereleaseTags(cwd: string, baseVersion: string): string[] {
return execFileSync('git', ['tag', '--list', `v${baseVersion}-beta.*`, `v${baseVersion}-rc.*`], {
cwd,
encoding: 'utf8',
})
.split(/\r?\n/)
.map((line) => line.trim())
.filter(Boolean);
}
// Diffs changes/*.md between the previous prerelease tag and the working tree.
// Renamed fragments are treated as modifications of the new path.
//
// Like every other path in this script, git paths are resolved against `cwd`,
// which is the project root and also the repository root. Callers that point
// `cwd` elsewhere already fail earlier and loudly, when package.json and
// changes/ come back missing.
function defaultResolveFragmentDelta(cwd: string, previousTag: string): FragmentDeltaEntry[] {
const output = execFileSync(
'git',
['diff', '--name-status', '--find-renames', previousTag, '--', 'changes'],
{ cwd, encoding: 'utf8' },
);
const showAtTag = (fragmentPath: string): string =>
execFileSync('git', ['show', `${previousTag}:${fragmentPath}`], { cwd, encoding: 'utf8' });
const readCurrent = (fragmentPath: string): string =>
fs.readFileSync(path.join(cwd, fragmentPath), 'utf8');
const entries: FragmentDeltaEntry[] = [];
for (const line of output.split(/\r?\n/)) {
if (!line.trim()) {
continue;
}
const [status = '', ...paths] = line.split('\t');
const oldPath = paths[0] ?? '';
const newPath = paths[paths.length - 1] ?? '';
if (!isFragmentPath(newPath) && !isFragmentPath(oldPath)) {
continue;
}
if (status.startsWith('A')) {
entries.push({ path: newPath, status: 'added', after: readCurrent(newPath) });
} else if (status.startsWith('D')) {
entries.push({ path: oldPath, status: 'deleted', before: showAtTag(oldPath) });
} else if (status.startsWith('M') || status.startsWith('R')) {
entries.push({
path: newPath,
status: 'modified',
before: showAtTag(oldPath),
after: readCurrent(newPath),
});
}
}
// git diff misses fragments that exist only in the working tree; treat
// untracked fragments as additions so a pre-commit run still sees them.
const untracked = execFileSync(
'git',
['ls-files', '--others', '--exclude-standard', '--', 'changes'],
{ cwd, encoding: 'utf8' },
)
.split(/\r?\n/)
.map((line) => line.trim())
.filter((candidate) => candidate && isFragmentPath(candidate));
for (const fragmentPath of untracked) {
entries.push({ path: fragmentPath, status: 'added', after: readCurrent(fragmentPath) });
}
return entries;
}
function verifyRequestedVersionMatchesPackageVersion(
@@ -615,7 +784,7 @@ function polishFragmentsWithClaude(
? [
'## Existing Prerelease Notes',
'',
'The input includes EXISTING PRERELEASE NOTES before the fragment list. Existing prerelease notes are a baseline, not an immutable changelog. Reuse reviewed highlight bullets when they still describe the current outcome, but replace stale beta or RC wording when new fragments supersede it. Merge in only new or changed fragment material, and deduplicate instead of restating existing bullets. Output only the final highlights body using the section headings above; do not include the prerelease disclaimer, Installation, or Assets sections.',
'The input includes EXISTING PRERELEASE NOTES before the fragment list. Existing prerelease notes are a baseline, not an immutable changelog. Reuse reviewed highlight bullets when they still describe the current outcome, but replace stale beta or RC wording when new fragments supersede it. Merge in only new or changed fragment material, and deduplicate instead of restating existing bullets. Output only the final highlights body using the section headings above; do not include the prerelease disclaimer, any "Changes since" section, or the Installation or Assets sections.',
'',
].join('\n')
: '';
@@ -627,6 +796,75 @@ function polishFragmentsWithClaude(
return validatePolishedOutput(output, mode, hasInternalFragments);
}
const DELTA_PROMPT_INSTRUCTIONS = `You are writing the "changes since the previous prerelease" section of a prerelease notes file for SubMiner, an Electron app for Japanese sentence mining.
You will receive changelog fragment diffs between the previous prerelease tag and the current build. Fragments are engineer-written release-note sources; a fragment diff is a proxy for what changed, not proof of a behavior change.
Rules:
1. Output Markdown bullets ONLY. No headings, no preamble, no commentary. Every line must be a top-level "- " bullet or an indented nested bullet.
2. Describe only what changed for users between the two prerelease builds, in user-facing language. Drop implementation jargon, file paths, and PR numbers.
3. ADDED fragments describe changes that are new in this build; summarize them.
4. MODIFIED fragments include BEFORE and AFTER content. Describe only the behavioral difference between them. If the edit is editorial (rewording, deduplication, reformatting, reconciling stale phrasing) with no user-visible behavior change, omit it entirely.
5. DELETED fragments mean the described change was removed or reverted before this build; say so explicitly.
6. Keep bullets short and concrete. Use nested bullets sparingly.
7. Do not invent changes. Every bullet must be grounded in the diffs.
8. If no bullet survives rules 2-5, output exactly this single line:
- No user-facing changes since PREVIOUS_TAG.
The input begins below.
`;
function serializeFragmentDeltaForPrompt(
delta: FragmentDeltaEntry[],
version: string,
previousTag: string,
): string {
const header = [`VERSION: ${version}`, `PREVIOUS_TAG: ${previousTag}`];
const blocks = delta.map((entry) => {
if (entry.status === 'added') {
return [`ADDED FRAGMENT ${entry.path}`, entry.after ?? ''].join('\n');
}
if (entry.status === 'deleted') {
return [`DELETED FRAGMENT ${entry.path}`, entry.before ?? ''].join('\n');
}
return [
`MODIFIED FRAGMENT ${entry.path}`,
'BEFORE:',
entry.before ?? '',
'AFTER:',
entry.after ?? '',
].join('\n');
});
return [...header, '', ...blocks].join('\n\n');
}
function validateDeltaOutput(output: string): string {
const trimmed = output.trim();
if (!trimmed) {
throw new Error('claude returned empty output for the prerelease delta section.');
}
const invalidLine = trimmed.split(/\r?\n/).find((line) => line.trim() && !/^\s*- /.test(line));
if (invalidLine !== undefined) {
throw new Error(
`claude delta output must contain only Markdown bullets. Offending line:\n${invalidLine}`,
);
}
return trimmed;
}
function buildDeltaSectionWithClaude(
delta: FragmentDeltaEntry[],
options: { version: string; previousTag: string; deps?: ChangelogFsDeps },
): string {
const runClaude = options.deps?.runClaude ?? defaultRunClaude;
const prompt =
DELTA_PROMPT_INSTRUCTIONS.replace('PREVIOUS_TAG', options.previousTag) +
serializeFragmentDeltaForPrompt(delta, options.version, options.previousTag);
return validateDeltaOutput(runClaude(prompt, CLAUDE_CLI_ARGS));
}
function stripDetailsBlocks(body: string): string {
return body.replace(/<details>[\s\S]*?<\/details>\s*/gm, '').trim();
}
@@ -709,15 +947,18 @@ function renderReleaseNotes(
contributions?: Contribution[];
contributorSections?: string[];
metadata?: string[];
deltaSection?: string[];
},
): string {
const prefix = options?.disclaimer ? [options.disclaimer, ''] : [];
const metadata = options?.metadata?.length ? [...options.metadata, ''] : [];
const deltaSection = options?.deltaSection?.length ? [...options.deltaSection, ''] : [];
const contributorSections =
options?.contributorSections ?? renderContributorsSections(options?.contributions ?? []);
return [
...prefix,
...metadata,
...deltaSection,
'## Highlights',
changes,
'',
@@ -748,6 +989,7 @@ function writeReleaseNotesFile(
contributions?: Contribution[];
contributorSections?: string[];
metadata?: string[];
deltaSection?: string[];
},
): string {
const mkdirSync = deps?.mkdirSync ?? fs.mkdirSync;
@@ -1079,6 +1321,26 @@ export function writePrereleaseNotesForVersion(options?: ChangelogOptions): stri
throw new Error('No changelog fragments found in changes/.');
}
const listPrereleaseTags = options?.deps?.listPrereleaseTags ?? defaultListPrereleaseTags;
const previousTag = selectPreviousPrereleaseTag(
listPrereleaseTags(cwd, resolvePrereleaseBaseVersion(version)),
version,
);
// Later betas/RCs get a "Changes since <previous tag>" section on top of the
// cumulative Highlights, generated from the fragment diff between the
// previous prerelease tag and the working tree.
let deltaSection: string[] = [];
if (previousTag) {
const resolveFragmentDelta = options?.deps?.resolveFragmentDelta ?? defaultResolveFragmentDelta;
const delta = resolveFragmentDelta(cwd, previousTag);
const deltaBody =
delta.length === 0
? `- No changelog fragment changes since ${previousTag}; this build contains packaging or internal-only updates.`
: buildDeltaSectionWithClaude(delta, { version, previousTag, deps: options?.deps });
deltaSection = [`${DELTA_SECTION_HEADING_PREFIX}${previousTag}`, '', deltaBody];
}
const prereleaseNotesPath = path.join(cwd, PRERELEASE_NOTES_PATH);
const existingReleaseNotes = existsSync(prereleaseNotesPath)
? resolveReusablePrereleaseNotes(readFileSync(prereleaseNotesPath, 'utf8'), version)
@@ -1095,10 +1357,41 @@ export function writePrereleaseNotesForVersion(options?: ChangelogOptions): stri
'> This is a prerelease build for testing. Stable changelog and docs-site updates remain pending until the final stable release.',
outputPath: PRERELEASE_NOTES_PATH,
contributions,
metadata: [renderPrereleaseBaseVersionMarker(version)],
metadata: [renderPrereleaseVersionMarker(version, previousTag)],
deltaSection,
});
}
// CI gate: the committed prerelease notes must carry a marker generated for
// exactly the version being tagged, so stale beta.N-1 notes can't ship.
export function verifyPrereleaseNotesMatchVersion(options?: ChangelogOptions): void {
verifyRequestedVersionMatchesPackageVersion(options ?? {});
const cwd = options?.cwd ?? process.cwd();
const existsSync = options?.deps?.existsSync ?? fs.existsSync;
const readFileSync = options?.deps?.readFileSync ?? fs.readFileSync;
const version = resolveVersion(options ?? {});
if (!isSupportedPrereleaseVersion(version)) {
throw new Error(
`Unsupported prerelease version (${version}). Expected x.y.z-beta.N or x.y.z-rc.N.`,
);
}
const prereleaseNotesPath = path.join(cwd, PRERELEASE_NOTES_PATH);
if (!existsSync(prereleaseNotesPath)) {
throw new Error(
`Missing ${prereleaseNotesPath}. Run 'bun run changelog:prerelease-notes --version ${version}' and commit the file before tagging.`,
);
}
const markerVersion = extractPrereleaseVersionMarker(readFileSync(prereleaseNotesPath, 'utf8'));
if (markerVersion !== version) {
throw new Error(
`release/prerelease-notes.md was generated for ${markerVersion ?? 'an unknown version (missing or legacy prerelease-version marker)'} but this release is ${version}. Rerun 'bun run changelog:prerelease-notes --version ${version}' and commit the result.`,
);
}
}
function parseCliArgs(argv: string[]): {
baseRef?: string;
cwd?: string;
@@ -1206,6 +1499,11 @@ function main(): void {
return;
}
if (command === 'check-prerelease-notes') {
verifyPrereleaseNotesMatchVersion(options);
return;
}
if (command === 'docs') {
generateDocsChangelog(options);
return;
-54
View File
@@ -1,54 +0,0 @@
#!/bin/bash
# Build macOS window tracking helper binary
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SWIFT_SOURCE="$SCRIPT_DIR/get-mpv-window-macos.swift"
OUTPUT_DIR="$SCRIPT_DIR/../dist/scripts"
OUTPUT_BINARY="$OUTPUT_DIR/get-mpv-window-macos"
OUTPUT_SOURCE_COPY="$OUTPUT_DIR/get-mpv-window-macos.swift"
fallback_to_source() {
echo "Falling back to source fallback: $OUTPUT_SOURCE_COPY"
mkdir -p "$OUTPUT_DIR"
cp "$SWIFT_SOURCE" "$OUTPUT_SOURCE_COPY"
}
build_swift_helper() {
echo "Compiling macOS window tracking helper..."
if ! command -v swiftc >/dev/null 2>&1; then
echo "swiftc not found in PATH; skipping compilation."
return 1
fi
if ! swiftc -O "$SWIFT_SOURCE" -o "$OUTPUT_BINARY"; then
return 1
fi
chmod +x "$OUTPUT_BINARY"
echo "✓ Built $OUTPUT_BINARY"
return 0
}
# Optional skip flag for non-macOS CI/dev environments
if [[ "${SUBMINER_SKIP_MACOS_HELPER_BUILD:-}" == "1" ]]; then
echo "Skipping macOS helper build (SUBMINER_SKIP_MACOS_HELPER_BUILD=1)"
fallback_to_source
exit 0
fi
# Only build on macOS
if [[ "$(uname)" != "Darwin" ]]; then
echo "Skipping macOS helper build (not on macOS)"
fallback_to_source
exit 0
fi
# Create output directory
mkdir -p "$OUTPUT_DIR"
# Compile Swift script to binary, fallback to source if unavailable or compilation fails
if ! build_swift_helper; then
fallback_to_source
fi
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
FILE="${1:-}"
if [[ ! -f "$FILE" ]]; then
printf 'Not a file: %s\n' "${FILE:-<missing>}" >&2
exit 1
fi
if ! mpv --no-config --no-terminal --msg-level=all=no --vo=null --ao=null --frames=1 -- "$FILE"; then
printf 'Not playable by mpv: %s\n' "$FILE" >&2
exit 1
fi
exec subminer app --dev --launch-mpv "$FILE"
+4 -2
View File
@@ -19,7 +19,8 @@ Options:
-w, --webp Generate animated WebP preview
Encoding profile:
- Crop: 1920x1080 at x=760 y=200
- Crop: mpv region at 1920x1080, x=760 y=205 on a 3440x1440 canvas
- Output size: 1920x1080
- MP4: H.264 + AAC
- WebM: AV1/VP9 + Opus at 30 fps
USAGE
@@ -148,7 +149,8 @@ pick_webp_encoder() {
return 1
}
crop_vf="crop=1920:1080:760:205"
# OBS may resize the 3440x1440 canvas, so scale the mpv bounds with the input.
crop_vf="crop=1920*iw/3440:1080*ih/1440:760*iw/3440:205*ih/1440,scale=1920:1080:flags=lanczos"
webm_vf="${crop_vf},fps=30"
echo "Generating MP4: $mp4_out"
+5 -1
View File
@@ -40,7 +40,7 @@ function toBashPath(filePath: string): string {
return `${drive.toUpperCase()}:/${rest}`;
}
test('mkv-to-readme-video accepts libwebp_anim when libwebp is unavailable', () => {
test('mkv-to-readme-video builds every output with the scaled mpv crop', () => {
withTempDir((root) => {
const binDir = path.join(root, 'bin');
const inputPath = path.join(root, 'sample.mkv');
@@ -104,5 +104,9 @@ touch "$output"
const ffmpegLog = fs.readFileSync(ffmpegLogPath, 'utf8');
assert.match(ffmpegLog, /-c:v libwebp_anim/);
const scaledCropUses = ffmpegLog.match(
/-vf crop=1920\*iw\/3440:1080\*ih\/1440:760\*iw\/3440:205\*ih\/1440,scale=1920:1080:flags=lanczos/g,
);
assert.equal(scaledCropUses?.length, 4);
});
});
+18 -3
View File
@@ -1,4 +1,5 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { execFileSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
@@ -52,6 +53,16 @@ function fallbackToMacosSource() {
process.stdout.write(`Staged macOS helper source fallback: ${macosHelperSourceCopyPath}\n`);
}
// Pin the minimum macOS to the app's own floor (Electron's `minos`). Without an
// explicit target, swiftc stamps the build machine's OS version as the binary's
// minimum and the helper fails to load on older systems (#213). The arch stays
// the host's, matching the single-arch app electron-builder packages here.
const MACOS_HELPER_DEPLOYMENT_TARGET = '12.0';
function macosHelperTarget() {
return `${os.arch() === 'x64' ? 'x86_64' : 'arm64'}-apple-macos${MACOS_HELPER_DEPLOYMENT_TARGET}`;
}
function shouldSkipMacosHelperBuild() {
return process.env.SUBMINER_SKIP_MACOS_HELPER_BUILD === '1';
}
@@ -72,9 +83,13 @@ function buildMacosHelper() {
ensureDir(scriptsOutputDir);
try {
execFileSync('swiftc', ['-O', macosHelperSourcePath, '-o', macosHelperBinaryPath], {
stdio: 'inherit',
});
execFileSync(
'swiftc',
['-O', '-target', macosHelperTarget(), macosHelperSourcePath, '-o', macosHelperBinaryPath],
{
stdio: 'inherit',
},
);
fs.chmodSync(macosHelperBinaryPath, 0o755);
process.stdout.write(`Built macOS helper: ${macosHelperBinaryPath}\n`);
} catch (error) {
+8 -1
View File
@@ -8,7 +8,7 @@ test('macOS helper build creates dist scripts directory before swiftc output', (
const buildFunctionIndex = source.indexOf('function buildMacosHelper()');
assert.notEqual(buildFunctionIndex, -1);
const swiftcIndex = source.indexOf("execFileSync('swiftc'", buildFunctionIndex);
const swiftcIndex = source.indexOf("'swiftc'", buildFunctionIndex);
assert.notEqual(swiftcIndex, -1);
const ensureDirIndex = source.lastIndexOf('ensureDir(scriptsOutputDir)', swiftcIndex);
@@ -18,3 +18,10 @@ test('macOS helper build creates dist scripts directory before swiftc output', (
'buildMacosHelper must create dist/scripts before swiftc writes the helper binary',
);
});
// Regression guard for #213: an untargeted swiftc stamps the build machine's OS
// version as the helper's minimum, so released builds refuse to load on older macOS.
test('macOS helper is compiled with an explicit deployment target', () => {
assert.match(source, /-target/);
assert.match(source, /apple-macos\$\{MACOS_HELPER_DEPLOYMENT_TARGET\}/);
});
+53 -1
View File
@@ -130,7 +130,9 @@ local function run_plugin_scenario(config)
function mp.add_timeout(seconds, callback)
recorded.timeouts[#recorded.timeouts + 1] = seconds
local delay = tonumber(seconds) or 0
local timeout = {
seconds = delay,
killed = false,
callback = callback,
}
@@ -138,7 +140,6 @@ local function run_plugin_scenario(config)
self.killed = true
end
local delay = tonumber(seconds) or 0
if callback and delay < 5 and not config.defer_timeouts then
callback()
end
@@ -514,6 +515,15 @@ local function has_timeout(timeouts, target)
return false
end
local function find_timeout_handle(recorded, target)
for _, timeout in ipairs(recorded.timeout_handles) do
if math.abs(timeout.seconds - target) < 0.0001 then
return timeout
end
end
return nil
end
local function env_has(call, target)
local env = (call and call.env) or {}
for _, value in ipairs(env) do
@@ -1636,6 +1646,8 @@ do
#recorded.periodic_timers == 1,
"auto-start visible overlay should refresh the early overlay loading OSD"
)
local overlay_loading_deadline = find_timeout_handle(recorded, 30)
assert_true(overlay_loading_deadline ~= nil, "overlay loading OSD should have a bounded deadline")
local overlay_loading_timer = recorded.periodic_timers[1]
recorded.periodic_timers[1].callback()
assert_true(
@@ -1670,6 +1682,46 @@ do
recorded.periodic_timers[1].killed == true,
"overlay loading ready should stop the early overlay loading OSD refresher"
)
assert_true(
overlay_loading_deadline.killed == true,
"overlay loading ready should cancel the bounded loading deadline"
)
end
do
local recorded, err = run_plugin_scenario({
defer_timeouts = true,
process_list = "",
option_overrides = {
binary_path = binary_path,
auto_start = "yes",
auto_start_visible_overlay = "yes",
osd_messages = false,
socket_path = "/tmp/subminer-socket",
},
input_ipc_server = "/tmp/subminer-socket",
media_title = "Random Movie",
files = {
[binary_path] = true,
},
})
assert_true(recorded ~= nil, "plugin failed to load for overlay loading deadline scenario: " .. tostring(err))
fire_event(recorded, "start-file")
local overlay_loading_deadline = find_timeout_handle(recorded, 30)
assert_true(overlay_loading_deadline ~= nil, "overlay loading deadline should be scheduled")
overlay_loading_deadline.callback()
assert_true(
recorded.periodic_timers[1].killed == true,
"overlay loading deadline should stop the loading spinner"
)
assert_true(
has_osd_message(recorded.osd, "SubMiner: Overlay did not become ready; check SubMiner logs"),
"overlay loading deadline should replace the spinner with actionable feedback"
)
assert_true(
has_log_containing(recorded.logs, "Overlay loading deadline expired"),
"overlay loading deadline should leave a diagnostic log entry"
)
end
do
+1
View File
@@ -82,6 +82,7 @@ test('update-aur-package updates PKGBUILD and .SRCINFO without makepkg', () => {
pkgbuild,
/^\s*install -Dm755 "\$\{srcdir\}\/subminer-\$\{pkgver\}" "\$\{pkgdir\}\/usr\/bin\/subminer"$/m,
);
assert.match(pkgbuild, /assets\/thumbnailers\/subminer-ffmpegthumbnailer\.thumbnailer/);
assert.match(srcinfo, /^\tpkgver = 0\.6\.3$/m);
assert.match(srcinfo, /^\tprovides = subminer=0\.6\.3$/m);
assert.match(
@@ -284,6 +284,22 @@ function createMockTracker(
getSessionTimeline: async () => [],
getSessionEvents: async () => [],
getVocabularyStats: async () => VOCABULARY_STATS,
getVocabularySummary: async () => ({
uniqueWords: 501,
uniqueWordsWithoutNames: 500,
uniqueKanji: 201,
newThisWeek: 7,
newThisWeekWithoutNames: 6,
knownWordCount: 250,
knownWordCountWithoutNames: 249,
}),
getVocabularyChartData: async () => ({
ready: true,
topWords: [{ wordId: 1, headword: 'する', frequency: 50 }],
topWordsWithoutNames: [{ wordId: 1, headword: 'する', frequency: 50 }],
newWordsTimeline: [{ epochDay: 20_000, wordCount: 3 }],
newWordsTimelineWithoutNames: [{ epochDay: 20_000, wordCount: 3 }],
}),
getStatsExcludedWords: async () => [],
replaceStatsExcludedWords: async () => {},
getKanjiStats: async () => KANJI_STATS,
@@ -711,6 +727,38 @@ describe('stats server API routes', () => {
assert.equal(body[0].headword, 'する');
});
it('GET /api/stats/vocabulary/summary returns database-wide card totals', async () => {
const app = createStatsApp(createMockTracker());
const res = await app.request('/api/stats/vocabulary/summary');
assert.equal(res.status, 200);
assert.deepEqual(await res.json(), {
uniqueWords: 501,
uniqueWordsWithoutNames: 500,
uniqueKanji: 201,
newThisWeek: 7,
newThisWeekWithoutNames: 6,
knownWordCount: 250,
knownWordCountWithoutNames: 249,
});
});
it('GET /api/stats/vocabulary/charts returns complete chart datasets', async () => {
const app = createStatsApp(createMockTracker());
const res = await app.request('/api/stats/vocabulary/charts');
assert.equal(res.status, 200);
assert.deepEqual(await res.json(), {
ready: true,
topWords: [{ wordId: 1, headword: 'する', frequency: 50 }],
topWordsWithoutNames: [{ wordId: 1, headword: 'する', frequency: 50 }],
newWordsTimeline: [{ epochDay: 20_000, wordCount: 3 }],
newWordsTimelineWithoutNames: [{ epochDay: 20_000, wordCount: 3 }],
});
});
it('GET /api/stats/kanji returns kanji frequency data', async () => {
const app = createStatsApp(createMockTracker());
const res = await app.request('/api/stats/kanji');
@@ -559,6 +559,241 @@ test('fresh tracker DB creates lifetime summary tables', async () => {
}
});
test('fresh tracker DB skips lexical rollup backfill work', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let backfillRuns = 0;
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor({ dbPath }, {
runLexicalRollupBackfillTask: async () => {
backfillRuns += 1;
},
} as never);
assert.equal(backfillRuns, 0);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('tracker starts the injected lexical rollup backfill when it is pending', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let backfillRuns = 0;
try {
const setupDb = new Database(dbPath);
const { ensureSchema } = await import('./immersion-tracker/storage');
ensureSchema(setupDb);
setupDb
.prepare(
`UPDATE imm_rollup_state SET state_value = '0' WHERE state_key = 'lexical_daily_rollups_version'`,
)
.run();
setupDb.close();
const Ctor = await loadTrackerCtor();
tracker = new Ctor({ dbPath }, {
runLexicalRollupBackfillTask: async () => {
backfillRuns += 1;
},
} as never);
assert.equal(backfillRuns, 1);
await waitForCondition(
() => !(tracker as unknown as { writeLock: { locked: boolean } }).writeLock.locked,
);
assert.equal(
(tracker as unknown as { preserveWriteQueueUntilDrained: boolean })
.preserveWriteQueueUntilDrained,
false,
);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('tracker runs startup session-rollup maintenance before lexical backfill locks writes', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let releaseBackfill = (): void => {};
const heldBackfill = new Promise<void>((resolve) => {
releaseBackfill = resolve;
});
try {
const startedAtMs = trackerNowMs() - 60_000;
const endedAtMs = trackerNowMs();
const setupDb = new Database(dbPath);
const { ensureSchema } = await import('./immersion-tracker/storage');
ensureSchema(setupDb);
setupDb.exec(`
INSERT INTO imm_videos (
video_id, video_key, canonical_title, source_type, duration_ms, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (1, 'local:/tmp/rollup-recovery.mkv', 'Rollup Recovery', 1, 0, '1', '1');
INSERT INTO imm_sessions (
session_id, session_uuid, video_id, started_at_ms, ended_at_ms, status,
active_watched_ms, lines_seen, tokens_seen, cards_mined, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (
1, 'rollup-recovery', 1, '${startedAtMs}', '${endedAtMs}', 2,
60000, 10, 20, 2, '${startedAtMs}', '${endedAtMs}'
);
INSERT INTO imm_session_telemetry (
session_id, sample_ms, total_watched_ms, active_watched_ms, lines_seen,
tokens_seen, cards_mined, lookup_count, lookup_hits, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (
1, '${endedAtMs}', 60000, 60000, 10, 20, 2, 0, 0,
'${endedAtMs}', '${endedAtMs}'
);
DELETE FROM imm_daily_rollups;
DELETE FROM imm_monthly_rollups;
UPDATE imm_rollup_state SET state_value = '0';
`);
setupDb.close();
const Ctor = await loadTrackerCtor();
tracker = new Ctor({ dbPath }, {
runLexicalRollupBackfillTask: async () => heldBackfill,
} as never);
const privateApi = tracker as unknown as {
db: DatabaseSync;
writeLock: { locked: boolean };
};
assert.equal(privateApi.writeLock.locked, true);
assert.equal(
(
privateApi.db.prepare('SELECT COUNT(*) AS total FROM imm_daily_rollups').get() as {
total: number;
}
).total,
1,
);
assert.equal(
(
privateApi.db.prepare('SELECT COUNT(*) AS total FROM imm_monthly_rollups').get() as {
total: number;
}
).total,
1,
);
} finally {
releaseBackfill();
if (tracker) {
await waitForCondition(
() => !(tracker as unknown as { writeLock: { locked: boolean } }).writeLock.locked,
);
}
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('tracker queues playback writes until lexical rollup backfill settles', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let startBackfill = (): void => {};
let releaseBackfill = (): void => {};
let markBackfillStarted = (): void => {};
const backfillStartGate = new Promise<void>((resolve) => {
startBackfill = resolve;
});
const heldBackfill = new Promise<void>((resolve) => {
releaseBackfill = resolve;
});
const backfillStarted = new Promise<void>((resolve) => {
markBackfillStarted = resolve;
});
try {
const setupDb = new Database(dbPath);
const { ensureSchema } = await import('./immersion-tracker/storage');
ensureSchema(setupDb);
setupDb
.prepare(
`UPDATE imm_rollup_state SET state_value = '0' WHERE state_key = 'lexical_daily_rollups_version'`,
)
.run();
setupDb.close();
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath, policy: { queueCap: 100 } },
{
runLexicalRollupBackfillTask: async (workerDbPath) => {
await backfillStartGate;
const workerDb = new Database(workerDbPath);
try {
workerDb.exec('BEGIN IMMEDIATE');
markBackfillStarted();
await heldBackfill;
workerDb.exec('COMMIT');
} catch (error) {
try {
workerDb.exec('ROLLBACK');
} catch {
// Preserve the original worker failure.
}
throw error;
} finally {
workerDb.close();
}
},
},
);
tracker.handleMediaChange('https://example.com/backfill-test.mp4', 'Backfill Test');
startBackfill();
await backfillStarted;
for (let index = 0; index < 125; index += 1) tracker.recordCardsMined(1);
const privateApi = tracker as unknown as {
db: DatabaseSync;
queue: unknown[];
droppedWriteCount: number;
flushNow: () => void;
writeLock: { locked: boolean };
};
assert.equal(privateApi.writeLock.locked, true);
privateApi.flushNow();
assert.ok(privateApi.queue.length > 100, 'the protected queue may grow past its normal cap');
assert.equal(privateApi.droppedWriteCount, 0, 'backfill must not discard playback writes');
assert.equal(
(
privateApi.db.prepare('SELECT COUNT(*) AS total FROM imm_session_events').get() as {
total: number;
}
).total,
0,
);
releaseBackfill();
await waitForCondition(() => privateApi.queue.length === 0, 5_000);
assert.equal(
(
privateApi.db.prepare('SELECT COUNT(*) AS total FROM imm_session_events').get() as {
total: number;
}
).total,
125,
);
} finally {
releaseBackfill();
if (tracker) {
await waitForCondition(
() => !(tracker as unknown as { writeLock: { locked: boolean } }).writeLock.locked,
5_000,
);
}
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('startup backfills lifetime summaries when retained sessions exist but summary tables are empty', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
@@ -4909,3 +5144,149 @@ test('ensureAnimeCoverArt fetches art via the latest video of the anime', async
cleanupDbPath(dbPath);
}
});
test('getVocabularySummary coalesces concurrent requests into one worker task', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let taskRuns = 0;
let releaseTask: (() => void) | null = null;
const seenKnownWords: Array<ReadonlySet<string> | null> = [];
const summary = {
uniqueWords: 1,
uniqueWordsWithoutNames: 1,
uniqueKanji: 0,
newThisWeek: 0,
newThisWeekWithoutNames: 0,
knownWordCount: null,
knownWordCountWithoutNames: null,
};
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runVocabularySummaryTask: async (_dbPath, knownWords) => {
taskRuns += 1;
seenKnownWords.push(knownWords);
await new Promise<void>((resolve) => {
releaseTask = resolve;
});
return summary;
},
destroyVocabularySummaryRunner: () => {},
},
);
const knownWordsSnapshot = new Set(['猫']);
const first = tracker.getVocabularySummary(knownWordsSnapshot);
const second = tracker.getVocabularySummary(knownWordsSnapshot);
await waitForCondition(() => releaseTask !== null);
let release = releaseTask as (() => void) | null;
assert.ok(release);
release();
assert.deepEqual(await first, summary);
assert.equal(await second, await first);
assert.equal(taskRuns, 1);
assert.deepEqual(seenKnownWords, [knownWordsSnapshot]);
releaseTask = null;
const third = tracker.getVocabularySummary(null);
await waitForCondition(() => releaseTask !== null);
release = releaseTask as (() => void) | null;
assert.ok(release);
release();
assert.deepEqual(await third, summary);
assert.equal(taskRuns, 2);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('getVocabularySummary coalesces equivalent known-word snapshots by value', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
let taskRuns = 0;
const releases: Array<() => void> = [];
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runVocabularySummaryTask: async () => {
taskRuns += 1;
await new Promise<void>((resolve) => releases.push(resolve));
return {
uniqueWords: 2,
uniqueWordsWithoutNames: 2,
uniqueKanji: 2,
newThisWeek: 0,
newThisWeekWithoutNames: 0,
knownWordCount: 2,
knownWordCountWithoutNames: 2,
};
},
destroyVocabularySummaryRunner: () => {},
},
);
const first = tracker.getVocabularySummary(new Set(['猫', '犬']));
const second = tracker.getVocabularySummary(new Set(['犬', '猫']));
await waitForCondition(() => releases.length > 0);
const observedTaskRuns = taskRuns;
for (const release of releases) release();
await Promise.all([first, second]);
assert.equal(observedTaskRuns, 1);
const third = tracker.getVocabularySummary(new Set(['猫', '犬']));
await waitForCondition(() => releases.length === 2);
releases[1]!();
await third;
assert.equal(taskRuns, 2, 'a settled snapshot must be evicted from the in-flight map');
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('getVocabularySummary keeps different known-word snapshots independent', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
const releases: Array<() => void> = [];
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor(
{ dbPath },
{
runVocabularySummaryTask: async (_dbPath, knownWords) => {
await new Promise<void>((resolve) => releases.push(resolve));
return {
uniqueWords: 1,
uniqueWordsWithoutNames: 1,
uniqueKanji: 0,
newThisWeek: 0,
newThisWeekWithoutNames: 0,
knownWordCount: knownWords?.size ?? null,
knownWordCountWithoutNames: knownWords?.size ?? null,
};
},
destroyVocabularySummaryRunner: () => {},
},
);
const withoutKnownWords = tracker.getVocabularySummary(null);
const withKnownWords = tracker.getVocabularySummary(new Set(['猫']));
await waitForCondition(() => releases.length === 2);
for (const release of releases) release();
assert.equal((await withoutKnownWords).knownWordCount, null);
assert.equal((await withKnownWords).knownWordCount, 1);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
+106 -6
View File
@@ -58,6 +58,7 @@ import {
getSessionEvents,
getSimilarWords,
getStatsExcludedWords,
getVocabularyChartData,
getVocabularyStats,
replaceStatsExcludedWords,
searchSubtitleSentences,
@@ -96,6 +97,12 @@ import {
DeleteMaintenanceWorkerRuntime,
type RunDeleteMaintenanceTask,
} from './immersion-tracker/delete-maintenance-worker-runtime';
import {
VocabularySummaryWorkerRuntime,
type RunVocabularySummaryTask,
} from './immersion-tracker/vocabulary-summary-worker-runtime';
import { LexicalRollupWorkerRuntime } from './immersion-tracker/lexical-rollup-worker-runtime';
import { areLexicalDailyRollupsReady } from './immersion-tracker/lexical-rollups';
import { DeleteMaintenanceScheduler } from './immersion-tracker/delete-maintenance-scheduler';
import {
cleanupDuplicateSubtitleLines,
@@ -185,6 +192,7 @@ import {
type StatsExcludedWordRow,
type StreakCalendarRow,
type VocabularyCleanupSummary,
type VocabularyStatsSummary,
type WatchTimePerAnimeRow,
type WordAnimeAppearanceRow,
type WordDetailRow,
@@ -405,13 +413,24 @@ export class ImmersionTrackerService {
private readonly monthlyRollupRetentionMs: number;
private readonly vacuumIntervalMs: number;
private readonly dbPath: string;
private readonly writeLock = { locked: false };
private readonly writeLock = {
locked: false,
reasons: new Set<'flush' | 'delete-maintenance' | 'lexical-rollup-backfill'>(),
};
private readonly destroyDeleteMaintenanceRunner: () => void;
private readonly runVocabularySummaryTask: (
knownWords: ReadonlySet<string> | null,
) => Promise<VocabularyStatsSummary>;
private readonly vocabularySummariesInFlight = new Map<string, Promise<VocabularyStatsSummary>>();
private readonly destroyVocabularySummaryRunner: () => void;
private readonly runLexicalRollupBackfillTask: () => Promise<void>;
private readonly destroyLexicalRollupBackfillRunner: () => void;
private readonly deleteMaintenanceScheduler: DeleteMaintenanceScheduler;
private flushTimer: ReturnType<typeof setTimeout> | null = null;
private maintenanceTimer: ReturnType<typeof setInterval> | null = null;
private flushScheduled = false;
private droppedWriteCount = 0;
private preserveWriteQueueUntilDrained = false;
private lastVacuumMs = 0;
private isDestroyed = false;
private sessionState: SessionState | null = null;
@@ -434,6 +453,10 @@ export class ImmersionTrackerService {
dependencies: {
runDeleteMaintenanceTask?: RunDeleteMaintenanceTask;
destroyDeleteMaintenanceRunner?: () => void;
runVocabularySummaryTask?: RunVocabularySummaryTask;
destroyVocabularySummaryRunner?: () => void;
runLexicalRollupBackfillTask?: (dbPath: string) => Promise<void>;
destroyLexicalRollupBackfillRunner?: () => void;
} = {},
) {
this.dbPath = options.dbPath;
@@ -453,13 +476,34 @@ export class ImmersionTrackerService {
runTask: (task) => runDeleteMaintenanceTask(this.dbPath, task),
onBusy: () => {
this.requireWriteQueueDrained('delete maintenance');
this.writeLock.locked = true;
this.setWriteLock('delete-maintenance', true);
},
onIdle: () => {
this.writeLock.locked = false;
this.setWriteLock('delete-maintenance', false);
if (!this.isDestroyed && this.queue.length > 0) this.scheduleFlush(0);
},
});
if (dependencies.runVocabularySummaryTask) {
this.runVocabularySummaryTask = (knownWords) =>
dependencies.runVocabularySummaryTask!(this.dbPath, knownWords);
this.destroyVocabularySummaryRunner =
dependencies.destroyVocabularySummaryRunner ?? (() => {});
} else {
const vocabularySummaryRuntime = new VocabularySummaryWorkerRuntime();
this.runVocabularySummaryTask = (knownWords) =>
vocabularySummaryRuntime.run(this.dbPath, knownWords);
this.destroyVocabularySummaryRunner = () => vocabularySummaryRuntime.destroy();
}
if (dependencies.runLexicalRollupBackfillTask) {
this.runLexicalRollupBackfillTask = () =>
dependencies.runLexicalRollupBackfillTask!(this.dbPath);
this.destroyLexicalRollupBackfillRunner =
dependencies.destroyLexicalRollupBackfillRunner ?? (() => {});
} else {
const lexicalRollupRuntime = new LexicalRollupWorkerRuntime();
this.runLexicalRollupBackfillTask = () => lexicalRollupRuntime.run(this.dbPath);
this.destroyLexicalRollupBackfillRunner = () => lexicalRollupRuntime.destroy();
}
const parentDir = path.dirname(this.dbPath);
if (!fs.existsSync(parentDir)) {
fs.mkdirSync(parentDir, { recursive: true });
@@ -548,6 +592,7 @@ export class ImmersionTrackerService {
}
this.preparedStatements = createTrackerPreparedStatements(this.db);
this.scheduleMaintenance();
if (!areLexicalDailyRollupsReady(this.db)) this.startLexicalRollupBackfill();
this.scheduleFlush();
}
@@ -565,6 +610,8 @@ export class ImmersionTrackerService {
this.isDestroyed = true;
this.deleteMaintenanceScheduler.destroy();
this.destroyDeleteMaintenanceRunner();
this.destroyVocabularySummaryRunner();
this.destroyLexicalRollupBackfillRunner();
this.db.close();
}
@@ -634,6 +681,25 @@ export class ImmersionTrackerService {
return getVocabularyStats(this.db, limit, excludePos);
}
async getVocabularySummary(knownWords: ReadonlySet<string> | null) {
const key = knownWords ? JSON.stringify([...knownWords].sort()) : 'null';
const inFlight = this.vocabularySummariesInFlight.get(key);
if (inFlight) return inFlight;
const task = this.runVocabularySummaryTask(knownWords);
this.vocabularySummariesInFlight.set(key, task);
try {
return await task;
} finally {
if (this.vocabularySummariesInFlight.get(key) === task) {
this.vocabularySummariesInFlight.delete(key);
}
}
}
async getVocabularyChartData() {
return getVocabularyChartData(this.db);
}
async getStatsExcludedWords(): Promise<StatsExcludedWordRow[]> {
return getStatsExcludedWords(this.db);
}
@@ -910,6 +976,33 @@ export class ImmersionTrackerService {
}
}
private setWriteLock(
reason: 'flush' | 'delete-maintenance' | 'lexical-rollup-backfill',
active: boolean,
): void {
if (active) this.writeLock.reasons.add(reason);
else this.writeLock.reasons.delete(reason);
this.writeLock.locked = this.writeLock.reasons.size > 0;
}
private startLexicalRollupBackfill(): void {
this.requireWriteQueueDrained('lexical rollup backfill');
this.preserveWriteQueueUntilDrained = true;
this.setWriteLock('lexical-rollup-backfill', true);
void this.runLexicalRollupBackfillTask()
.catch((error: unknown) => {
this.logger.warn(
'Lexical daily rollup backfill failed; it will retry on next startup',
error,
);
})
.finally(() => {
this.setWriteLock('lexical-rollup-backfill', false);
if (this.queue.length === 0) this.preserveWriteQueueUntilDrained = false;
else if (!this.isDestroyed) this.scheduleFlush(0);
});
}
async reassignAnimeAnilist(
animeId: number,
info: {
@@ -1906,7 +1999,12 @@ export class ImmersionTrackerService {
private recordWrite(write: QueuedWrite): void {
if (this.isDestroyed) return;
const { dropped } = enqueueWrite(this.queue, write, this.queueCap);
// A lexical migration owns the database write lock, so dropping the oldest
// entry cannot relieve pressure: nothing can flush until the worker exits.
// Preserve that finite startup burst and drain it as soon as the lock lifts.
const { dropped } = this.preserveWriteQueueUntilDrained
? (this.queue.push(write), { dropped: 0 })
: enqueueWrite(this.queue, write, this.queueCap);
if (dropped > 0) {
this.droppedWriteCount += dropped;
this.logger.warn(`Immersion tracker queue overflow; dropped ${dropped} oldest writes`);
@@ -1954,6 +2052,7 @@ export class ImmersionTrackerService {
private flushNow(): void {
if (this.writeLock.locked || this.isDestroyed) return;
if (this.queue.length === 0) {
this.preserveWriteQueueUntilDrained = false;
this.flushScheduled = false;
return;
}
@@ -1965,7 +2064,7 @@ export class ImmersionTrackerService {
}
const batch = this.queue.splice(0, Math.min(this.batchSize, this.queue.length));
this.writeLock.locked = true;
this.setWriteLock('flush', true);
try {
this.db.exec('BEGIN IMMEDIATE');
for (const write of batch) {
@@ -1977,8 +2076,9 @@ export class ImmersionTrackerService {
this.queue.unshift(...batch);
this.logger.warn('Immersion tracker flush failed, retrying later', error as Error);
} finally {
this.writeLock.locked = false;
this.setWriteLock('flush', false);
this.flushScheduled = false;
if (this.queue.length === 0) this.preserveWriteQueueUntilDrained = false;
if (this.queue.length > 0) {
this.scheduleFlush(this.flushIntervalMs);
}
@@ -1,7 +1,4 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import { Database } from '../sqlite.js';
import type { DatabaseSync } from '../sqlite.js';
@@ -21,17 +18,6 @@ interface SeedLine {
createdMs?: number;
}
function makeDbPath(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-duplicate-line-test-'));
return path.join(dir, 'immersion.sqlite');
}
function cleanupDbPath(dbPath: string): void {
const dir = path.dirname(dbPath);
if (!fs.existsSync(dir)) return;
fs.rmSync(dir, { recursive: true, force: true });
}
/** One episode, two sessions of it, and one word occurrence per seeded line. */
function seed(db: DatabaseSync, lines: SeedLine[]): void {
db.exec(`
@@ -82,12 +68,16 @@ function seed(db: DatabaseSync, lines: SeedLine[]): void {
`);
}
function createDb(lines: SeedLine[]): { db: DatabaseSync; dbPath: string } {
const dbPath = makeDbPath();
const db = new Database(dbPath);
/**
* These tests exercise the cleanup SQL, not durability. A fresh on-disk database per
* test pays a schema-creation fsync that is cheap on a local NVMe but slow enough on CI
* runners to blow the 5s per-test timeout, so the database stays in memory.
*/
function createDb(lines: SeedLine[]): { db: DatabaseSync } {
const db = new Database(':memory:');
ensureSchema(db);
seed(db, lines);
return { db, dbPath };
return { db };
}
/** A typeset line mpv reported once per animation frame. */
@@ -119,7 +109,7 @@ function wordFrequency(db: DatabaseSync): number {
}
test('a karaoke burst collapses to one line and gives back its word counts', () => {
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 40, 40),
{ session: 1, text: 'おはよう', startMs: 20_000, endMs: 22_000 },
]);
@@ -148,7 +138,6 @@ test('a karaoke burst collapses to one line and gives back its word counts', ()
assert.equal(summary.samples[0]!.videoTitle, 'Ep 1');
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
@@ -160,7 +149,7 @@ test('ordinary repeated dialogue survives', () => {
startMs: 5_000 + index * 800,
endMs: 5_000 + (index + 1) * 800,
}));
const { db, dbPath } = createDb(lines);
const { db } = createDb(lines);
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -171,14 +160,13 @@ test('ordinary repeated dialogue survives', () => {
assert.equal(wordFrequency(db), 6);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a long run of quarter-second frames is still a burst', () => {
// Between the timing-only bound (0.1s) and the animation-frame bound (0.3s): heavier
// typesetting lands here, and the run length is what makes it conclusive.
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -189,12 +177,11 @@ test('a long run of quarter-second frames is still a burst', () => {
assert.equal(wordFrequency(db), 1);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a qualifying short-frame burst may end with one long hold frame', () => {
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 8, 40),
{ session: 1, text: '飛び上がる', startMs: 10_320, endMs: 12_320 },
]);
@@ -208,12 +195,11 @@ test('a qualifying short-frame burst may end with one long hold frame', () => {
assert.equal(wordFrequency(db), 1);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a long event before the final frame prevents burst cleanup', () => {
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 5, 40),
{ session: 1, text: '飛び上がる', startMs: 10_200, endMs: 12_200 },
{ session: 1, text: '飛び上がる', startMs: 12_200, endMs: 12_240 },
@@ -226,12 +212,11 @@ test('a long event before the final frame prevents burst cleanup', () => {
assert.equal(countLines(db), 7);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a run of frames longer than the animation bound survives', () => {
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 400));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 400));
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -240,7 +225,6 @@ test('a run of frames longer than the animation bound survives', () => {
assert.equal(countLines(db), 6);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
@@ -248,7 +232,7 @@ test('the four-frame residue the live gate stores is cleaned up', () => {
// The streaming gate records the first four frames of a burst before the run is long
// enough to recognise. Four contiguous identical events under the strict timing-only
// bound are that residue, and no real dialogue.
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 40));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 40));
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -259,14 +243,13 @@ test('the four-frame residue the live gate stores is cleaned up', () => {
assert.equal(wordFrequency(db), 1);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a four-frame run above the strict frame bound survives', () => {
// Long enough per event to be plausible dialogue; only a five-event run may use the
// looser animation-frame bound.
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 250));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 4, 250));
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -275,14 +258,13 @@ test('a four-frame run above the strict frame bound survives', () => {
assert.equal(countLines(db), 4);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('an explicit minRunLength raises the bar', () => {
// Five quarter-second frames qualify under the defaults; a cautious run asking for six
// leaves them alone. Above the strict bound, so the residue rule stays out of it.
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 5, 250));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 5, 250));
try {
const preview = cleanupDuplicateSubtitleLines(db, { dryRun: true });
@@ -293,12 +275,11 @@ test('an explicit minRunLength raises the bar', () => {
assert.equal(countLines(db), 5);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('an explicit maxFrameSeconds tightens the frame bound', () => {
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 250));
try {
const summary = cleanupDuplicateSubtitleLines(db, { maxFrameSeconds: 0.2 });
@@ -307,13 +288,12 @@ test('an explicit maxFrameSeconds tightens the frame bound', () => {
assert.equal(countLines(db), 6);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a non-finite maxFrameSeconds falls back to the default bound', () => {
// Six normal-beat lines: Infinity must not turn every event into a "short frame".
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 800));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 6, 800));
try {
const summary = cleanupDuplicateSubtitleLines(db, { maxFrameSeconds: Infinity });
@@ -322,12 +302,11 @@ test('a non-finite maxFrameSeconds falls back to the default bound', () => {
assert.equal(countLines(db), 6);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('sampleLimit zero removes bursts but reports no samples', () => {
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
try {
const summary = cleanupDuplicateSubtitleLines(db, { sampleLimit: 0 });
@@ -337,12 +316,11 @@ test('sampleLimit zero removes bursts but reports no samples', () => {
assert.equal(countLines(db), 1);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a short run below every threshold survives', () => {
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 3, 40));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 1_000, 3, 40));
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -351,7 +329,6 @@ test('a short run below every threshold survives', () => {
assert.equal(countLines(db), 3);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
@@ -361,7 +338,7 @@ test('interleaved dual-line karaoke collapses each line to one row', () => {
const kanji = karaokeFrames(1, '飛び上がる', 10_000, 20, 60);
const romaji = karaokeFrames(1, 'tobiagaru', 10_001, 20, 60);
const interleaved = [...kanji, ...romaji].sort((a, b) => a.startMs - b.startMs);
const { db, dbPath } = createDb(interleaved);
const { db } = createDb(interleaved);
try {
const summary = cleanupDuplicateSubtitleLines(db);
@@ -372,12 +349,11 @@ test('interleaved dual-line karaoke collapses each line to one row', () => {
assert.equal(wordFrequency(db), 2);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('the same line in a rewatch session is never merged into the first watch', () => {
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40),
...karaokeFrames(2, '飛び上がる', 10_000, 6, 40),
]);
@@ -392,12 +368,11 @@ test('the same line in a rewatch session is never merged into the first watch',
assert.equal(wordFrequency(db), 2);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a gap between runs splits them', () => {
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40),
...karaokeFrames(1, '飛び上がる', 60_000, 6, 40),
]);
@@ -409,12 +384,11 @@ test('a gap between runs splits them', () => {
assert.equal(countLines(db), 2);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('a dry run reports what an apply would do and writes nothing', () => {
const { db, dbPath } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
const { db } = createDb(karaokeFrames(1, '飛び上がる', 10_000, 40, 40));
try {
const preview = cleanupDuplicateSubtitleLines(db, { dryRun: true });
@@ -430,14 +404,13 @@ test('a dry run reports what an apply would do and writes nothing', () => {
assert.equal(countLines(db), 1);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('the lookback window leaves older bursts alone', () => {
const recentMs = BASE_MS;
const oldMs = BASE_MS - 40 * DAY_MS;
const { db, dbPath } = createDb([
const { db } = createDb([
...karaokeFrames(1, '飛び上がる', 10_000, 6, 40).map((line) => ({
...line,
createdMs: oldMs,
@@ -462,6 +435,5 @@ test('the lookback window leaves older bursts alone', () => {
} finally {
globalThis.__subminerTestNowMs = undefined;
db.close();
cleanupDbPath(dbPath);
}
});
@@ -31,6 +31,7 @@ import {
getKanjiOccurrences,
getSessionSummaries,
getVocabularyStats,
getVocabularySummary,
getKanjiStats,
getSessionEvents,
getSessionTimeline,
@@ -1875,6 +1876,115 @@ test('getVocabularyStats returns rows ordered by frequency descending', () => {
}
});
test('getVocabularySummary counts every tracked vocabulary row instead of a display page', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
try {
ensureSchema(db);
const nowSec = Math.floor(Date.now() / 1000);
const insertWord = db.prepare(`
INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3,
first_seen, last_seen, frequency
) VALUES (?, ?, '', 'noun', '名詞', '一般', '', ?, ?, 1)
`);
const insertKanji = db.prepare(`
INSERT INTO imm_kanji (kanji, first_seen, last_seen, frequency)
VALUES (?, ?, ?, 1)
`);
for (let index = 0; index < 501; index += 1) {
insertWord.run(`単語${index}`, `単語${index}`, nowSec - 8 * 86_400, nowSec - 8 * 86_400);
}
for (let index = 0; index < 201; index += 1) {
insertKanji.run(
String.fromCodePoint(0x4e00 + index),
nowSec - 8 * 86_400,
nowSec - 8 * 86_400,
);
}
insertWord.run('今週', '今週', nowSec - 86_400, nowSec - 86_400);
assert.deepEqual(getVocabularySummary(db, new Set(['単語0', '今週']), nowSec * 1000), {
uniqueWords: 502,
uniqueWordsWithoutNames: 502,
uniqueKanji: 201,
newThisWeek: 1,
newThisWeekWithoutNames: 1,
knownWordCount: 2,
knownWordCountWithoutNames: 2,
});
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('getVocabularySummary applies vocabulary exclusions and Hide Names totals', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
try {
ensureSchema(db);
const insertWord = db.prepare(`
INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3,
first_seen, last_seen, frequency
) VALUES (?, ?, '', 'noun', '名詞', ?, '', 1, 1, 1)
`);
insertWord.run('猫', '猫', '一般');
insertWord.run('太郎', '太郎', '固有名詞');
insertWord.run('東京', '東京都', '一般');
db.prepare(
`
INSERT INTO imm_stats_excluded_words (headword, word, reading)
VALUES ('東京', '東京', '')
`,
).run();
assert.deepEqual(getVocabularySummary(db, new Set(['猫', '太郎', '東京']), 9 * 86_400_000), {
uniqueWords: 2,
uniqueWordsWithoutNames: 1,
uniqueKanji: 0,
newThisWeek: 0,
newThisWeekWithoutNames: 0,
knownWordCount: 2,
knownWordCountWithoutNames: 1,
});
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('getVocabularySummary counts identically across id-keyed scan batches', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
try {
ensureSchema(db);
const insertWord = db.prepare(`
INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3,
first_seen, last_seen, frequency
) VALUES (?, ?, '', 'noun', '名詞', '一般', '', 1, 1, 1)
`);
for (let index = 0; index < 5; index += 1) {
insertWord.run(`単語${index}`, `単語${index}`);
}
const fullScan = getVocabularySummary(db, new Set(['単語0']), 9 * 86_400_000);
const batchedScan = getVocabularySummary(db, new Set(['単語0']), 9 * 86_400_000, 2);
assert.equal(fullScan.uniqueWords, 5);
assert.deepEqual(batchedScan, fullScan);
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('getVocabularyStats filters rows that fail tokenizer vocabulary rules', () => {
const dbPath = makeDbPath();
const db = openTestDb(dbPath);
@@ -0,0 +1,136 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import {
LexicalRollupWorkerRuntime,
resolveLexicalRollupWorkerPath,
} from './lexical-rollup-worker-runtime';
import { areLexicalDailyRollupsReady } from './lexical-rollups';
import { Database } from './sqlite';
import { applyPragmas, ensureSchema } from './storage';
test('lexical rollup worker backfills without using the tracker connection', async () => {
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-lexical-rollup-runtime-'));
const dbPath = path.join(directory, 'immersion.sqlite');
const runtime = new LexicalRollupWorkerRuntime();
const db = new Database(dbPath);
try {
applyPragmas(db);
ensureSchema(db);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES ('鳥', '鳥', 'とり', 1700000000, 1700000000, 1)`,
).run();
db.exec('DELETE FROM imm_lexical_daily_rollups');
db.prepare(`UPDATE imm_rollup_state SET state_value = '0' WHERE state_key = ?`).run(
'lexical_daily_rollups_version',
);
db.close();
await runtime.run(dbPath);
const checkDb = new Database(dbPath);
try {
assert.equal(areLexicalDailyRollupsReady(checkDb), true);
} finally {
checkDb.close();
}
} finally {
runtime.destroy();
try {
db.close();
} catch {
// Closed before the worker starts.
}
fs.rmSync(directory, { recursive: true, force: true });
}
});
test('lexical rollup worker module resolves in the current layout', () => {
const workerPath = resolveLexicalRollupWorkerPath();
assert.ok(workerPath, 'expected the lexical rollup worker module to resolve');
assert.ok(workerPath.endsWith(__filename.endsWith('.ts') ? '.ts' : '.js'));
});
test('lexical rollup worker leaves a backfill pending when no worker can start', async () => {
const runtime = new LexicalRollupWorkerRuntime({
resolveWorkerPath: () => null,
warn: () => {},
} as never);
try {
await assert.doesNotReject(runtime.run('/tmp/not-used.sqlite'));
} finally {
runtime.destroy();
}
});
test('lexical rollup worker absorbs termination failures after settling', async () => {
let sendMessage: ((message: { ok: boolean }) => void) | null = null;
const runtime = new LexicalRollupWorkerRuntime({
resolveWorkerPath: () => '/tmp/fake-worker.js',
createWorker: async () => ({
once(event: string, listener: (value: never) => void) {
if (event === 'message') sendMessage = listener as (message: { ok: boolean }) => void;
return this;
},
terminate: async () => {
throw new Error('termination failed');
},
}),
warn: () => {},
} as never);
const unhandled: unknown[] = [];
const captureUnhandled = (reason: unknown) => unhandled.push(reason);
process.on('unhandledRejection', captureUnhandled);
try {
const task = runtime.run('/tmp/not-used.sqlite');
await new Promise((resolve) => setImmediate(resolve));
const notify = sendMessage as ((message: { ok: boolean }) => void) | null;
assert.ok(notify);
notify({ ok: true });
await task;
await new Promise((resolve) => setImmediate(resolve));
assert.deepEqual(unhandled, []);
} finally {
process.off('unhandledRejection', captureUnhandled);
runtime.destroy();
}
});
test('lexical rollup worker times out when it never responds', async () => {
let terminated = false;
const runtime = new LexicalRollupWorkerRuntime({
resolveWorkerPath: () => '/tmp/fake-worker.js',
createWorker: async () => ({
once() {
return this;
},
terminate: async () => {
terminated = true;
return 0;
},
}),
timeoutMs: 1,
warn: () => {},
} as never);
try {
const outcome = await Promise.race([
runtime.run('/tmp/not-used.sqlite').then(
() => 'resolved',
(error: unknown) => String(error),
),
new Promise<string>((resolve) => setTimeout(() => resolve('still pending'), 50)),
]);
assert.match(outcome, /timed out/);
assert.equal(terminated, true);
} finally {
runtime.destroy();
}
});
@@ -0,0 +1,117 @@
import fs from 'node:fs';
import path from 'node:path';
import { createLogger } from '../../../logger';
interface WorkerResponse {
ok?: boolean;
error?: unknown;
}
interface WorkerHandle {
once(event: 'message', listener: (message: WorkerResponse) => void): this;
once(event: 'error', listener: (error: Error) => void): this;
once(event: 'exit', listener: (code: number) => void): this;
terminate(): Promise<number>;
}
interface LexicalRollupWorkerRuntimeOptions {
resolveWorkerPath?: () => string | null;
createWorker?: (workerPath: string, workerData: { dbPath: string }) => Promise<WorkerHandle>;
timeoutMs?: number;
warn?: (message: string, ...meta: unknown[]) => void;
}
const logger = createLogger('main:immersion-tracker:lexical-rollup-worker');
const DEFAULT_WORKER_TIMEOUT_MS = 5 * 60 * 1_000;
export function resolveLexicalRollupWorkerPath(): string | null {
const fileName = __filename.endsWith('.ts')
? 'lexical-rollup-worker-thread.ts'
: 'lexical-rollup-worker-thread.js';
const workerPath = path.join(__dirname, fileName);
return fs.existsSync(workerPath) ? workerPath : null;
}
export class LexicalRollupWorkerRuntime {
private readonly activeWorkers = new Set<WorkerHandle>();
private destroyed = false;
constructor(private readonly options: LexicalRollupWorkerRuntimeOptions = {}) {}
async run(dbPath: string): Promise<void> {
if (this.destroyed) throw new Error('Lexical rollup worker is shut down');
let worker: WorkerHandle;
try {
const workerPath = (this.options.resolveWorkerPath ?? resolveLexicalRollupWorkerPath)();
if (!workerPath) throw new Error('Emitted lexical rollup worker module was not found');
const createWorker =
this.options.createWorker ??
(async (resolvedPath, workerData) => {
const { Worker } = await import('node:worker_threads');
return new Worker(resolvedPath, { workerData });
});
worker = await createWorker(workerPath, { dbPath });
} catch (error) {
if (this.destroyed) throw new Error('Lexical rollup worker is shut down');
(this.options.warn ?? logger.warn)(
'Lexical rollup worker unavailable; leaving backfill pending for a later startup',
error,
);
return;
}
if (this.destroyed) {
await worker.terminate().catch(() => undefined);
throw new Error('Lexical rollup worker is shut down');
}
return new Promise<void>((resolve, reject) => {
let settled = false;
let timeout: ReturnType<typeof setTimeout> | null = null;
this.activeWorkers.add(worker);
const settle = (error?: Error) => {
if (settled) return;
settled = true;
if (timeout) clearTimeout(timeout);
this.activeWorkers.delete(worker);
void worker.terminate().catch(() => undefined);
if (error) reject(error);
else resolve();
};
timeout = setTimeout(
() => settle(new Error('Lexical rollup worker timed out')),
this.options.timeoutMs ?? DEFAULT_WORKER_TIMEOUT_MS,
);
worker.once('message', (message) => {
if (message.ok) settle();
else
settle(
new Error(
`Lexical rollup backfill failed: ${String(message.error ?? 'unknown error')}`,
),
);
});
worker.once('error', (error) => settle(error));
worker.once('exit', (code) => {
if (!settled) {
settle(
new Error(
code === 0
? 'Lexical rollup worker exited without a response'
: `Lexical rollup worker exited with code ${code}`,
),
);
}
});
});
}
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
for (const worker of this.activeWorkers) {
void worker.terminate().catch(() => undefined);
}
this.activeWorkers.clear();
}
}
@@ -0,0 +1,11 @@
import { parentPort, workerData } from 'node:worker_threads';
import { executeLexicalRollupBackfillTask } from './lexical-rollup-worker';
if (!parentPort) throw new Error('lexical rollup worker missing parent port');
try {
executeLexicalRollupBackfillTask((workerData as { dbPath: string }).dbPath);
parentPort.postMessage({ ok: true });
} catch (error) {
parentPort.postMessage({ error: error instanceof Error ? error.message : String(error) });
}
@@ -0,0 +1,35 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import { areLexicalDailyRollupsReady, getLexicalDailyRollups } from './lexical-rollups';
import { executeLexicalRollupBackfillTask } from './lexical-rollup-worker';
import { Database } from './sqlite';
import { ensureSchema } from './storage';
test('lexical rollup backfill materializes pre-existing vocabulary off the caller DB connection', () => {
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-lexical-rollup-worker-'));
const dbPath = path.join(directory, 'immersion.sqlite');
const db = new Database(dbPath);
try {
ensureSchema(db);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 1)`,
).run('犬', '犬', 'いぬ', 1_700_000_000, 1_700_000_000);
db.exec('DELETE FROM imm_lexical_daily_rollups');
db.prepare(`UPDATE imm_rollup_state SET state_value = '0' WHERE state_key = ?`).run(
'lexical_daily_rollups_version',
);
executeLexicalRollupBackfillTask(dbPath);
assert.equal(areLexicalDailyRollupsReady(db), true);
assert.equal(getLexicalDailyRollups(db)[0]?.wordCount, 1);
} finally {
db.close();
fs.rmSync(directory, { recursive: true, force: true });
}
});
@@ -0,0 +1,15 @@
import { areLexicalDailyRollupsReady, rebuildLexicalDailyRollups } from './lexical-rollups';
import { Database } from './sqlite';
import { applyPragmas } from './storage';
export function executeLexicalRollupBackfillTask(dbPath: string): void {
const db = new Database(dbPath);
try {
applyPragmas(db);
if (!areLexicalDailyRollupsReady(db)) {
rebuildLexicalDailyRollups(db);
}
} finally {
db.close();
}
}
@@ -0,0 +1,421 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import {
areLexicalDailyRollupsReady,
getLexicalDailyRollups,
rebuildLexicalDailyRollups,
} from './lexical-rollups';
import { getTrendsDashboard } from './query-trends';
import {
getVocabularyChartData,
getVocabularySummary,
replaceStatsExcludedWords,
} from './query-lexical';
import { Database } from './sqlite';
import type { DatabaseSync } from './sqlite';
import { ensureSchema } from './storage';
function makeDbPath(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-lexical-rollups-'));
return path.join(dir, 'immersion.sqlite');
}
test('lexical daily rollups follow first-seen corrections and deletions', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const firstDay = 19_500;
const correctedDay = firstDay + 2;
const firstSeen = firstDay * 86_400 + 43_200;
const correctedSeen = correctedDay * 86_400 + 43_200;
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 1)`,
).run('猫', '猫', 'ねこ', firstSeen, firstSeen);
db.prepare(
`INSERT INTO imm_kanji(kanji, first_seen, last_seen, frequency)
VALUES (?, ?, ?, 1)`,
).run('猫', firstSeen, firstSeen);
assert.deepEqual(getLexicalDailyRollups(db), [
{ epochDay: firstDay, wordCount: 1, wordCountWithoutNames: 1, kanjiCount: 1 },
]);
db.prepare(`UPDATE imm_words SET first_seen = ? WHERE headword = ?`).run(correctedSeen, '猫');
db.prepare(`DELETE FROM imm_kanji WHERE kanji = ?`).run('猫');
assert.deepEqual(getLexicalDailyRollups(db), [
{ epochDay: correctedDay, wordCount: 1, wordCountWithoutNames: 1, kanjiCount: 0 },
]);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('lexical daily rollups normalize second and millisecond timestamps', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const epochDay = 19_500;
const timestampSeconds = epochDay * 86_400 + 43_200;
const timestampMilliseconds = timestampSeconds * 1_000;
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 1)`,
).run('猫', '猫', 'ねこ', timestampSeconds, timestampSeconds);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, ?, ?, ?, 1)`,
).run('犬', '犬', 'いぬ', timestampMilliseconds, timestampMilliseconds);
db.prepare(
`INSERT INTO imm_kanji(kanji, first_seen, last_seen, frequency)
VALUES (?, ?, ?, 1)`,
).run('猫', timestampSeconds, timestampSeconds);
db.prepare(
`INSERT INTO imm_kanji(kanji, first_seen, last_seen, frequency)
VALUES (?, ?, ?, 1)`,
).run('犬', timestampMilliseconds, timestampMilliseconds);
assert.deepEqual(getLexicalDailyRollups(db), [
{ epochDay, wordCount: 2, wordCountWithoutNames: 2, kanjiCount: 2 },
]);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('lexical rollup rebuild excludes rows hidden by vocabulary persistence rules', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const epochDay = 19_500;
const firstSeen = epochDay * 86_400 + 43_200;
db.prepare(
`INSERT INTO imm_words(
headword, word, reading, part_of_speech, first_seen, last_seen, frequency
) VALUES (?, ?, ?, ?, ?, ?, 1)`,
).run('猫', '猫', 'ねこ', 'noun', firstSeen, firstSeen);
db.prepare(
`INSERT INTO imm_words(
headword, word, reading, part_of_speech, first_seen, last_seen, frequency
) VALUES (?, ?, ?, ?, ?, ?, 1)`,
).run('は', 'は', 'は', 'particle', firstSeen, firstSeen);
rebuildLexicalDailyRollups(db);
assert.deepEqual(getLexicalDailyRollups(db), [
{ epochDay, wordCount: 1, wordCountWithoutNames: 1, kanjiCount: 0 },
]);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('lexical rollup rebuild tolerates nullable legacy vocabulary text', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (NULL, NULL, NULL, 1700000000, 1700000000, 1)`,
).run();
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (NULL, '猫', 'ねこ', 1700000000, 1700000000, 1)`,
).run();
assert.doesNotThrow(() => rebuildLexicalDailyRollups(db));
assert.equal(areLexicalDailyRollupsReady(db), true);
assert.equal(getVocabularySummary(db, null).uniqueWords, 1);
assert.equal(getVocabularySummary(db, new Set(['猫'])).knownWordCount, 1);
assert.equal(getVocabularyChartData(db).topWords[0]?.headword, '猫');
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('lexical rollup rebuild scans vocabulary visibility in bounded id batches', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
const expectedBatchSize = 5_000;
try {
ensureSchema(db);
const insertWord = db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, '', 1700000000, 1700000000, 1)`,
);
db.exec('BEGIN');
for (let index = 0; index <= expectedBatchSize; index += 1) {
insertWord.run(`${index}`, `${index}`);
}
db.exec('COMMIT');
const scanPageSizes: number[] = [];
const instrumentedDb: DatabaseSync = {
prepare(source) {
const statement = db.prepare(source);
if (!source.includes('WHERE id > ?') || !source.includes('ORDER BY id')) {
return statement;
}
return {
run: (...params) => statement.run(...params),
get: (...params) => statement.get(...params),
all: (...params) => {
const rows = statement.all(...params);
scanPageSizes.push(rows.length);
return rows;
},
};
},
exec(source) {
db.exec(source);
return instrumentedDb;
},
close() {
return instrumentedDb;
},
};
rebuildLexicalDailyRollups(instrumentedDb);
assert.deepEqual(scanPageSizes, [expectedBatchSize, 1]);
assert.equal(getVocabularySummary(db, null).uniqueWords, expectedBatchSize + 1);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('chart exclusions do not subtract vocabulary rows already hidden from the rollup', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const epochDay = 19_500;
const firstSeen = epochDay * 86_400 + 43_200;
db.prepare(
`INSERT INTO imm_words(
headword, word, reading, part_of_speech, first_seen, last_seen, frequency
) VALUES (?, ?, ?, ?, ?, ?, 1)`,
).run('猫', '猫', 'ねこ', 'noun', firstSeen, firstSeen);
db.prepare(
`INSERT INTO imm_words(
headword, word, reading, part_of_speech, first_seen, last_seen, frequency
) VALUES (?, ?, ?, ?, ?, ?, 1)`,
).run('は', 'は', 'は', 'particle', firstSeen, firstSeen);
rebuildLexicalDailyRollups(db);
replaceStatsExcludedWords(db, [{ headword: 'は', word: 'は', reading: 'は' }]);
const charts = getVocabularyChartData(db);
assert.deepEqual(charts.newWordsTimeline, [{ epochDay, wordCount: 1 }]);
assert.deepEqual(charts.newWordsTimelineWithoutNames, [{ epochDay, wordCount: 1 }]);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('legacy lexical rollup readiness does not satisfy the current rollup version', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value)
VALUES ('lexical_daily_rollups_ready', '1')
ON CONFLICT(state_key) DO UPDATE SET state_value = excluded.state_value`,
).run();
db.prepare(
`DELETE FROM imm_rollup_state WHERE state_key = 'lexical_daily_rollups_version'`,
).run();
assert.equal(areLexicalDailyRollupsReady(db), false);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('current lexical rollup readiness accepts legacy integer state storage', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
db.exec(`
CREATE TABLE imm_rollup_state(
state_key TEXT PRIMARY KEY,
state_value INTEGER NOT NULL
);
INSERT INTO imm_rollup_state(state_key, state_value)
VALUES ('lexical_daily_rollups_version', 2);
`);
assert.equal(areLexicalDailyRollupsReady(db), true);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('imm_words persists vocabulary visibility for rollup maintenance', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const columns = db.prepare(`PRAGMA table_info(imm_words)`).all() as Array<{ name: string }>;
assert.equal(
columns.some((column) => column.name === 'vocabulary_visible'),
true,
);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('vocabulary charts use complete top-word and lexical rollup data', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const insertWord = db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES (?, ?, '', 1700000000, 1700000000, ?)`,
);
for (let index = 0; index < 501; index += 1) {
insertWord.run(`${index}`, `${index}`, index === 500 ? 10_000 : 1);
}
const charts = getVocabularyChartData(db);
assert.equal(charts.topWords[0]?.headword, '語500');
assert.equal(charts.topWords[0]?.frequency, 10_000);
assert.equal(charts.newWordsTimeline[0]?.wordCount, 501);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('vocabulary charts find full top-word sets beyond excluded and name rows', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const insertWord = db.prepare(
`INSERT INTO imm_words(headword, word, reading, pos2, first_seen, last_seen, frequency)
VALUES (?, ?, '', ?, 1700000000, 1700000000, ?)`,
);
const exclusions = [];
for (let index = 0; index < 100; index += 1) {
const headword = `${index}`;
insertWord.run(
headword,
headword,
index < 80 && index >= 60 ? '固有名詞' : '一般',
100 - index,
);
if (index < 60) exclusions.push({ headword, word: headword, reading: '' });
}
replaceStatsExcludedWords(db, exclusions);
const charts = getVocabularyChartData(db);
assert.equal(charts.topWords.length, 12);
assert.equal(charts.topWords[0]?.headword, '語60');
assert.equal(charts.topWordsWithoutNames.length, 12);
assert.equal(charts.topWordsWithoutNames[0]?.headword, '語80');
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('vocabulary charts handle exclusion lists above one SQLite variable batch', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES ('語0', '語0', '', 1700000000, 1700000000, 1)`,
).run();
const exclusions = Array.from({ length: 10_923 }, (_, index) => ({
headword: `${index}`,
word: `${index}`,
reading: '',
}));
replaceStatsExcludedWords(db, exclusions);
const charts = getVocabularyChartData(db);
assert.deepEqual(charts.topWords, []);
assert.deepEqual(charts.newWordsTimeline, []);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
test('lexical rollup rebuild preserves the original error when rollback also fails', () => {
const originalError = new Error('rebuild failed');
const db = {
exec(sql: string) {
if (sql === 'BEGIN IMMEDIATE') return;
if (sql === 'ROLLBACK') throw new Error('rollback failed');
throw originalError;
},
prepare() {
return { all: () => [], run: () => undefined };
},
} as unknown as DatabaseSync;
assert.throws(() => rebuildLexicalDailyRollups(db), originalError);
});
test('trends read historical new-word buckets from lexical rollups', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
db.prepare(
`INSERT INTO imm_words(headword, word, reading, first_seen, last_seen, frequency)
VALUES ('海', '海', 'うみ', 1700000000, 1700000000, 1)`,
).run();
db.prepare(`UPDATE imm_lexical_daily_rollups SET word_count = 9`).run();
const dashboard = getTrendsDashboard(db, 'all', 'day', false);
assert.equal(dashboard.progress.newWords[0]?.value, 9);
} finally {
db.close();
fs.rmSync(path.dirname(dbPath), { recursive: true, force: true });
}
});
@@ -0,0 +1,229 @@
import type { DatabaseSync } from './sqlite';
import { isVocabularyStatsRowVisible, type VocabularyVisibilityRow } from './vocabulary-visibility';
export interface LexicalDailyRollup {
epochDay: number;
wordCount: number;
wordCountWithoutNames: number;
kanjiCount: number;
}
const LOCAL_EPOCH_DAY_SQL = `
CAST(julianday(
CASE
WHEN ABS(CAST(%VALUE% AS REAL)) >= 10000000000 THEN CAST(%VALUE% AS REAL) / 1000
ELSE CAST(%VALUE% AS REAL)
END,
'unixepoch', 'localtime'
) - 2440587.5 AS INTEGER)
`;
const LEXICAL_DAILY_ROLLUP_VERSION = '2';
const LEXICAL_DAILY_ROLLUP_VERSION_KEY = 'lexical_daily_rollups_version';
const VOCABULARY_VISIBILITY_SCAN_BATCH_SIZE = 5_000;
export function localEpochDaySql(value: string): string {
return LOCAL_EPOCH_DAY_SQL.replaceAll('%VALUE%', value);
}
function createWordRollupTriggers(db: DatabaseSync): void {
const dayForNew = localEpochDaySql('NEW.first_seen');
const dayForOld = localEpochDaySql('OLD.first_seen');
db.exec(`
DROP TRIGGER IF EXISTS imm_words_lexical_rollup_insert;
DROP TRIGGER IF EXISTS imm_words_lexical_rollup_delete;
DROP TRIGGER IF EXISTS imm_words_lexical_rollup_first_seen_update;
CREATE TRIGGER imm_words_lexical_rollup_insert
AFTER INSERT ON imm_words
WHEN NEW.first_seen IS NOT NULL AND NEW.vocabulary_visible = 1
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
VALUES (${dayForNew}, 1, CASE WHEN NEW.pos2 = '固有名詞' THEN 0 ELSE 1 END, 0)
ON CONFLICT(epoch_day) DO UPDATE SET
word_count = word_count + 1,
word_count_without_names = word_count_without_names + excluded.word_count_without_names;
END;
CREATE TRIGGER imm_words_lexical_rollup_delete
AFTER DELETE ON imm_words
WHEN OLD.first_seen IS NOT NULL AND OLD.vocabulary_visible = 1
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
VALUES (${dayForOld}, -1, CASE WHEN OLD.pos2 = '固有名詞' THEN 0 ELSE -1 END, 0)
ON CONFLICT(epoch_day) DO UPDATE SET
word_count = word_count - 1,
word_count_without_names = word_count_without_names + excluded.word_count_without_names;
DELETE FROM imm_lexical_daily_rollups
WHERE epoch_day = ${dayForOld} AND word_count = 0 AND kanji_count = 0;
END;
CREATE TRIGGER imm_words_lexical_rollup_first_seen_update
AFTER UPDATE OF first_seen, pos2, vocabulary_visible ON imm_words
WHEN OLD.first_seen IS NOT NEW.first_seen
OR OLD.pos2 IS NOT NEW.pos2
OR OLD.vocabulary_visible IS NOT NEW.vocabulary_visible
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${dayForOld}, -1, CASE WHEN OLD.pos2 = '固有名詞' THEN 0 ELSE -1 END, 0
WHERE OLD.first_seen IS NOT NULL AND OLD.vocabulary_visible = 1
ON CONFLICT(epoch_day) DO UPDATE SET
word_count = word_count - 1,
word_count_without_names = word_count_without_names + excluded.word_count_without_names;
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${dayForNew}, 1, CASE WHEN NEW.pos2 = '固有名詞' THEN 0 ELSE 1 END, 0
WHERE NEW.first_seen IS NOT NULL AND NEW.vocabulary_visible = 1
ON CONFLICT(epoch_day) DO UPDATE SET
word_count = word_count + 1,
word_count_without_names = word_count_without_names + excluded.word_count_without_names;
DELETE FROM imm_lexical_daily_rollups
WHERE word_count = 0 AND kanji_count = 0;
END;
`);
}
function createKanjiRollupTriggers(db: DatabaseSync): void {
const dayForNew = localEpochDaySql('NEW.first_seen');
const dayForOld = localEpochDaySql('OLD.first_seen');
db.exec(`
DROP TRIGGER IF EXISTS imm_kanji_lexical_rollup_insert;
DROP TRIGGER IF EXISTS imm_kanji_lexical_rollup_delete;
DROP TRIGGER IF EXISTS imm_kanji_lexical_rollup_first_seen_update;
CREATE TRIGGER imm_kanji_lexical_rollup_insert
AFTER INSERT ON imm_kanji WHEN NEW.first_seen IS NOT NULL
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
VALUES (${dayForNew}, 0, 0, 1)
ON CONFLICT(epoch_day) DO UPDATE SET kanji_count = kanji_count + 1;
END;
CREATE TRIGGER imm_kanji_lexical_rollup_delete
AFTER DELETE ON imm_kanji WHEN OLD.first_seen IS NOT NULL
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
VALUES (${dayForOld}, 0, 0, -1)
ON CONFLICT(epoch_day) DO UPDATE SET kanji_count = kanji_count - 1;
DELETE FROM imm_lexical_daily_rollups
WHERE epoch_day = ${dayForOld} AND word_count = 0 AND kanji_count = 0;
END;
CREATE TRIGGER imm_kanji_lexical_rollup_first_seen_update
AFTER UPDATE OF first_seen ON imm_kanji WHEN OLD.first_seen IS NOT NEW.first_seen
BEGIN
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${dayForOld}, 0, 0, -1 WHERE OLD.first_seen IS NOT NULL
ON CONFLICT(epoch_day) DO UPDATE SET kanji_count = kanji_count - 1;
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${dayForNew}, 0, 0, 1 WHERE NEW.first_seen IS NOT NULL
ON CONFLICT(epoch_day) DO UPDATE SET kanji_count = kanji_count + 1;
DELETE FROM imm_lexical_daily_rollups WHERE word_count = 0 AND kanji_count = 0;
END;
`);
}
export function ensureLexicalDailyRollupTables(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS imm_lexical_daily_rollups(
epoch_day INTEGER PRIMARY KEY,
word_count INTEGER NOT NULL DEFAULT 0,
word_count_without_names INTEGER NOT NULL DEFAULT 0,
kanji_count INTEGER NOT NULL DEFAULT 0
);
INSERT INTO imm_rollup_state(state_key, state_value)
VALUES ('${LEXICAL_DAILY_ROLLUP_VERSION_KEY}', '0')
ON CONFLICT(state_key) DO NOTHING;
`);
createWordRollupTriggers(db);
createKanjiRollupTriggers(db);
}
export function areLexicalDailyRollupsReady(db: DatabaseSync): boolean {
const row = db
.prepare(`SELECT state_value AS value FROM imm_rollup_state WHERE state_key = ?`)
.get(LEXICAL_DAILY_ROLLUP_VERSION_KEY) as { value: string | number } | undefined;
// Older databases created this column with INTEGER affinity, while current
// databases use TEXT. SQLite returns the same persisted version with a
// different JS type depending on that legacy schema.
return row !== undefined && String(row.value) === LEXICAL_DAILY_ROLLUP_VERSION;
}
export function markLexicalDailyRollupsReady(db: DatabaseSync): void {
db.prepare(
`INSERT INTO imm_rollup_state(state_key, state_value)
VALUES (?, ?)
ON CONFLICT(state_key) DO UPDATE SET state_value = excluded.state_value`,
).run(LEXICAL_DAILY_ROLLUP_VERSION_KEY, LEXICAL_DAILY_ROLLUP_VERSION);
}
/** Rebuild from the first-seen source of truth; run off the UI/main DB thread. */
export function rebuildLexicalDailyRollups(db: DatabaseSync): void {
let transactionStarted = false;
try {
db.exec('BEGIN IMMEDIATE');
transactionStarted = true;
const scanVocabulary = db.prepare(
`SELECT id, word, headword, reading, part_of_speech AS partOfSpeech,
pos1, pos2, pos3, frequency_rank AS frequencyRank
FROM imm_words
WHERE id > ?
ORDER BY id
LIMIT ?`,
);
const updateVisibility = db.prepare(
`UPDATE imm_words SET vocabulary_visible = ? WHERE id = ? AND vocabulary_visible IS NOT ?`,
);
let lastId = Number.MIN_SAFE_INTEGER;
for (;;) {
const vocabularyRows = scanVocabulary.all(
lastId,
VOCABULARY_VISIBILITY_SCAN_BATCH_SIZE,
) as Array<VocabularyVisibilityRow & { id: number }>;
if (vocabularyRows.length === 0) break;
for (const row of vocabularyRows) {
const visible = isVocabularyStatsRowVisible(row) ? 1 : 0;
updateVisibility.run(visible, row.id, visible);
}
lastId = vocabularyRows[vocabularyRows.length - 1]!.id;
if (vocabularyRows.length < VOCABULARY_VISIBILITY_SCAN_BATCH_SIZE) break;
}
db.exec('DELETE FROM imm_lexical_daily_rollups');
db.exec(`
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${localEpochDaySql('first_seen')}, COUNT(*),
SUM(CASE WHEN pos2 = '固有名詞' THEN 0 ELSE 1 END), 0
FROM imm_words
WHERE first_seen IS NOT NULL AND vocabulary_visible = 1
GROUP BY ${localEpochDaySql('first_seen')};
INSERT INTO imm_lexical_daily_rollups(epoch_day, word_count, word_count_without_names, kanji_count)
SELECT ${localEpochDaySql('first_seen')}, 0, 0, COUNT(*)
FROM imm_kanji
WHERE first_seen IS NOT NULL
GROUP BY ${localEpochDaySql('first_seen')}
ON CONFLICT(epoch_day) DO UPDATE SET kanji_count = kanji_count + excluded.kanji_count;
`);
markLexicalDailyRollupsReady(db);
db.exec('COMMIT');
} catch (error) {
if (transactionStarted) {
try {
db.exec('ROLLBACK');
} catch {
// Preserve the rebuild failure; it is the actionable cause.
}
}
throw error;
}
}
export function getLexicalDailyRollups(db: DatabaseSync): LexicalDailyRollup[] {
return db
.prepare(
`
SELECT epoch_day AS epochDay, word_count AS wordCount,
word_count_without_names AS wordCountWithoutNames, kanji_count AS kanjiCount
FROM imm_lexical_daily_rollups
ORDER BY epoch_day ASC
`,
)
.all() as LexicalDailyRollup[];
}
@@ -1,6 +1,4 @@
import type { DatabaseSync } from './sqlite';
import { PartOfSpeech, type MergedToken } from '../../../types';
import { shouldExcludeTokenFromVocabularyPersistence } from '../tokenizer/annotation-stage';
import type {
KanjiAnimeAppearanceRow,
KanjiDetailRow,
@@ -13,19 +11,38 @@ import type {
SimilarWordRow,
StatsExcludedWordRow,
VocabularyStatsRow,
VocabularyStatsSummary,
WordAnimeAppearanceRow,
WordDetailRow,
WordOccurrenceRow,
} from './types';
import { fromDbTimestamp, toDbTimestamp } from './query-shared';
import { nowMs } from './time';
import {
areLexicalDailyRollupsReady,
getLexicalDailyRollups,
localEpochDaySql,
} from './lexical-rollups';
import { isVocabularyStatsRowVisible } from './vocabulary-visibility';
const VOCABULARY_STATS_FILTER_OVERSAMPLE_FACTOR = 4;
const VOCABULARY_STATS_FILTER_OVERSAMPLE_MIN = 100;
const VOCABULARY_CHART_LIMIT = 12;
const VOCABULARY_CHART_PAGE_SIZE = 100;
const EXCLUSION_ALIAS_BATCH_SIZE = 300;
const VOCABULARY_SUMMARY_SCAN_BATCH_SIZE = 5_000;
const SENTENCE_SEARCH_DEFAULT_LIMIT = 50;
const SENTENCE_SEARCH_MAX_LIMIT = 100;
const KANJI_PATTERN = /\p{Script=Han}/gu;
export interface VocabularyChartData {
ready: boolean;
topWords: Array<{ wordId: number; headword: string; frequency: number }>;
topWordsWithoutNames: Array<{ wordId: number; headword: string; frequency: number }>;
newWordsTimeline: Array<{ epochDay: number; wordCount: number }>;
newWordsTimelineWithoutNames: Array<{ epochDay: number; wordCount: number }>;
}
function resolveSentenceSearchLimit(limit: number): number {
if (!Number.isFinite(limit)) return SENTENCE_SEARCH_DEFAULT_LIMIT;
const normalized = Math.floor(limit);
@@ -73,33 +90,6 @@ function uniqueKanji(text: string): string[] {
return Array.from(new Set(text.match(KANJI_PATTERN) ?? []));
}
function toVocabularyToken(row: VocabularyStatsRow): MergedToken {
const partOfSpeech =
row.partOfSpeech && Object.values(PartOfSpeech).includes(row.partOfSpeech as PartOfSpeech)
? (row.partOfSpeech as PartOfSpeech)
: PartOfSpeech.other;
return {
surface: row.word,
reading: row.reading ?? '',
headword: row.headword,
startPos: 0,
endPos: row.word.length,
partOfSpeech,
pos1: row.pos1 ?? '',
pos2: row.pos2 ?? '',
pos3: row.pos3 ?? '',
frequencyRank: row.frequencyRank ?? undefined,
isMerged: false,
isKnown: false,
isNPlusOneTarget: false,
};
}
function isVocabularyStatsRowVisible(row: VocabularyStatsRow): boolean {
return !shouldExcludeTokenFromVocabularyPersistence(toVocabularyToken(row));
}
export function getVocabularyStats(
db: DatabaseSync,
limit = 100,
@@ -153,6 +143,198 @@ export function getVocabularyStats(
return visibleRows.slice(0, limit);
}
/**
* Chart data is intentionally independent of the paginated vocabulary tables.
* Top words use the frequency index; new-word history reads permanent daily
* lexical rollups rather than loading every vocabulary row into the dashboard.
*/
export function getVocabularyChartData(db: DatabaseSync): VocabularyChartData {
const ready = areLexicalDailyRollupsReady(db);
const excludedAliases = new Set(
getStatsExcludedWords(db).flatMap((word) => excludedVocabularyAliases(word)),
);
const isExcluded = (word: Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading'>): boolean =>
excludedVocabularyAliases(word).some((alias) => excludedAliases.has(alias));
const topWords = getTopVocabularyChartWords(db, isExcluded);
const rollups = ready ? getLexicalDailyRollups(db) : [];
const timeline = new Map(rollups.map((row) => [row.epochDay, { ...row }]));
if (excludedAliases.size > 0 && ready) {
const aliases = [...excludedAliases];
const excludedRows = new Map<
number,
Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading' | 'pos2'> & {
wordId: number;
epochDay: number;
}
>();
for (let offset = 0; offset < aliases.length; offset += EXCLUSION_ALIAS_BATCH_SIZE) {
const batch = aliases.slice(offset, offset + EXCLUSION_ALIAS_BATCH_SIZE);
const placeholders = batch.map(() => '?').join(', ');
const rows = db
.prepare(
`
SELECT id AS wordId, headword, word, reading, pos2,
${localEpochDaySql('first_seen')} AS epochDay
FROM imm_words
WHERE vocabulary_visible = 1
AND (headword IN (${placeholders}) OR word IN (${placeholders}) OR reading IN (${placeholders}))
`,
)
.all(...batch, ...batch, ...batch) as Array<
Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading' | 'pos2'> & {
wordId: number;
epochDay: number;
}
>;
for (const row of rows) excludedRows.set(row.wordId, row);
}
for (const word of excludedRows.values()) {
if (!isExcluded(word)) continue;
const rollup = timeline.get(word.epochDay);
if (!rollup) continue;
rollup.wordCount -= 1;
if (word.pos2 !== '固有名詞') rollup.wordCountWithoutNames -= 1;
}
}
return {
ready,
topWords: topWords.all.map((word) => ({
wordId: word.wordId,
headword: vocabularyDisplayHeadword(word),
frequency: word.frequency,
})),
topWordsWithoutNames: topWords.withoutNames.map((word) => ({
wordId: word.wordId,
headword: vocabularyDisplayHeadword(word),
frequency: word.frequency,
})),
newWordsTimeline: [...timeline.values()]
.filter((row) => row.wordCount > 0)
.map((row) => ({ epochDay: row.epochDay, wordCount: row.wordCount })),
newWordsTimelineWithoutNames: [...timeline.values()]
.filter((row) => row.wordCountWithoutNames > 0)
.map((row) => ({ epochDay: row.epochDay, wordCount: row.wordCountWithoutNames })),
};
}
function getTopVocabularyChartWords(
db: DatabaseSync,
isExcluded: (word: Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading'>) => boolean,
): { all: VocabularyStatsRow[]; withoutNames: VocabularyStatsRow[] } {
const stmt = db.prepare(`
SELECT id AS wordId, headword, word, reading,
part_of_speech AS partOfSpeech, pos1, pos2, pos3,
frequency, frequency_rank AS frequencyRank,
first_seen AS firstSeen, last_seen AS lastSeen,
0 AS animeCount
FROM imm_words
ORDER BY frequency DESC, id
LIMIT ? OFFSET ?
`);
const all: VocabularyStatsRow[] = [];
const withoutNames: VocabularyStatsRow[] = [];
let offset = 0;
while (all.length < VOCABULARY_CHART_LIMIT || withoutNames.length < VOCABULARY_CHART_LIMIT) {
const page = stmt.all(VOCABULARY_CHART_PAGE_SIZE, offset) as VocabularyStatsRow[];
if (page.length === 0) break;
for (const word of page) {
if (!isVocabularyStatsRowVisible(word) || isExcluded(word)) continue;
if (all.length < VOCABULARY_CHART_LIMIT) all.push(word);
if (word.pos2 !== '固有名詞' && withoutNames.length < VOCABULARY_CHART_LIMIT) {
withoutNames.push(word);
}
}
offset += page.length;
}
return { all, withoutNames };
}
function excludedVocabularyAliases(
word: Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading'>,
): string[] {
const aliases = [word.headword?.trim() ?? '', word.word?.trim() ?? ''].filter(Boolean);
if (aliases.length === 0) aliases.push(word.reading?.trim() ?? '');
return [...new Set(aliases)];
}
function vocabularyDisplayHeadword(
word: Pick<VocabularyStatsRow, 'headword' | 'word' | 'reading'>,
): string {
return word.headword?.trim() || word.word?.trim() || word.reading?.trim() || '';
}
function timestampSeconds(timestamp: number): number {
return timestamp < 10_000_000_000 ? timestamp : Math.floor(timestamp / 1000);
}
export function getVocabularySummary(
db: DatabaseSync,
knownWords: ReadonlySet<string> | null,
nowMs: number = Date.now(),
scanBatchSize: number = VOCABULARY_SUMMARY_SCAN_BATCH_SIZE,
): VocabularyStatsSummary {
// Visibility and exclusion rules live in JS, so rows are scanned in id-keyed
// batches to keep memory bounded on large vocabularies.
const scanStmt = db.prepare(`
SELECT id AS wordId, headword, word, reading,
part_of_speech AS partOfSpeech, pos1, pos2, pos3,
frequency, frequency_rank AS frequencyRank,
first_seen AS firstSeen, last_seen AS lastSeen,
0 AS animeCount
FROM imm_words
WHERE id > ?
ORDER BY id
LIMIT ?
`);
const excludedAliases = new Set(
getStatsExcludedWords(db).flatMap((word) => excludedVocabularyAliases(word)),
);
const weekAgoSec = nowMs / 1000 - 7 * 86_400;
const summary: VocabularyStatsSummary = {
uniqueWords: 0,
uniqueWordsWithoutNames: 0,
uniqueKanji: (db.prepare('SELECT COUNT(*) AS count FROM imm_kanji').get() as { count: number })
.count,
newThisWeek: 0,
newThisWeekWithoutNames: 0,
knownWordCount: knownWords ? 0 : null,
knownWordCountWithoutNames: knownWords ? 0 : null,
};
let lastId = Number.MIN_SAFE_INTEGER;
for (;;) {
const words = scanStmt.all(lastId, scanBatchSize) as VocabularyStatsRow[];
if (words.length === 0) break;
lastId = words[words.length - 1]!.wordId;
for (const word of words) {
if (
!isVocabularyStatsRowVisible(word) ||
excludedVocabularyAliases(word).some((alias) => excludedAliases.has(alias))
) {
continue;
}
const isName = word.pos2 === '固有名詞';
const isNewThisWeek = timestampSeconds(fromDbTimestamp(word.firstSeen) ?? 0) >= weekAgoSec;
const isKnown = knownWords?.has(vocabularyDisplayHeadword(word)) ?? false;
summary.uniqueWords += 1;
if (!isName) summary.uniqueWordsWithoutNames += 1;
if (isNewThisWeek) {
summary.newThisWeek += 1;
if (!isName) summary.newThisWeekWithoutNames += 1;
}
if (isKnown) {
summary.knownWordCount! += 1;
if (!isName) summary.knownWordCountWithoutNames! += 1;
}
}
if (words.length < scanBatchSize) break;
}
return summary;
}
export function getStatsExcludedWords(db: DatabaseSync): StatsExcludedWordRow[] {
return db
.prepare(
@@ -13,6 +13,7 @@ import {
toDbTimestamp,
} from './query-shared';
import { getDailyRollups, getMonthlyRollups } from './query-sessions';
import { areLexicalDailyRollupsReady, getLexicalDailyRollups } from './lexical-rollups';
type TrendRange = '7d' | '30d' | '90d' | '365d' | 'all';
type TrendGroupBy = 'day' | 'month';
@@ -660,6 +661,16 @@ function buildNewWordsPerDay(
cutoffMs: string | null,
axis: number[] | null,
): TrendChartPoint[] {
if (areLexicalDailyRollupsReady(db)) {
// A trend range is defined in calendar buckets, so the rollup includes the
// complete local cutoff day rather than applying a time-of-day boundary.
const cutoffDay = cutoffMs === null ? null : getLocalEpochDay(db, cutoffMs);
const rows = getLexicalDailyRollups(db).filter(
(row) => cutoffDay === null || row.epochDay >= cutoffDay,
);
return fillAxisPoints(axis, new Map(rows.map((row) => [row.epochDay, row.wordCount])));
}
const whereClause = cutoffMs === null ? '' : 'AND first_seen >= ?';
const prepared = db.prepare(`
SELECT
@@ -691,6 +702,18 @@ function buildNewWordsPerMonth(
cutoffMs: string | null,
axis: number[] | null,
): TrendChartPoint[] {
if (areLexicalDailyRollupsReady(db)) {
const cutoffDay = cutoffMs === null ? null : getLocalEpochDay(db, cutoffMs);
const byMonth = new Map<number, number>();
for (const row of getLexicalDailyRollups(db)) {
if (cutoffDay !== null && row.epochDay < cutoffDay) continue;
const { year, month } = dayPartsFromEpochDay(row.epochDay);
const monthKey = year * 100 + month;
byMonth.set(monthKey, (byMonth.get(monthKey) ?? 0) + row.wordCount);
}
return fillAxisPoints(axis, byMonth);
}
const whereClause = cutoffMs === null ? '' : 'AND first_seen >= ?';
const prepared = db.prepare(`
SELECT
@@ -184,6 +184,51 @@ test('ensureSchema adds manual assignment locks when upgrading the previous sche
}
});
test('ensureSchema preserves durable session rollups across unrelated schema upgrades', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
db.exec(`
INSERT INTO imm_videos (
video_id, video_key, canonical_title, source_type, duration_ms, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (1, 'local:/tmp/preserved.mkv', 'Preserved', 1, 0, '1', '1');
INSERT INTO imm_daily_rollups (
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards
) VALUES (20000, 1, 2, 30, 40, 50, 3);
INSERT INTO imm_monthly_rollups (
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
total_tokens_seen, total_cards
) VALUES (202410, 1, 2, 30, 40, 50, 3);
UPDATE imm_rollup_state
SET state_value = '123'
WHERE state_key = 'last_rollup_sample_ms';
UPDATE imm_schema_version SET schema_version = 21;
`);
ensureSchema(db);
const daily = db
.prepare('SELECT total_sessions AS totalSessions FROM imm_daily_rollups')
.get() as { totalSessions: number } | null;
const monthly = db
.prepare('SELECT total_sessions AS totalSessions FROM imm_monthly_rollups')
.get() as { totalSessions: number } | null;
const rollupState = db
.prepare(`SELECT state_value AS value FROM imm_rollup_state WHERE state_key = ?`)
.get('last_rollup_sample_ms') as { value: string } | null;
assert.equal(daily?.totalSessions, 2);
assert.equal(monthly?.totalSessions, 2);
assert.equal(rollupState?.value, '123');
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('stats excluded words are replaced and read from sqlite storage', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
+26 -10
View File
@@ -4,6 +4,7 @@ import { parseMediaInfo } from '../../../jimaku/utils';
import { normalizeTitleIdentity } from '../../utils/title-normalization';
import type { DatabaseSync } from './sqlite';
import { nowMs } from './time';
import { ensureLexicalDailyRollupTables, markLexicalDailyRollupsReady } from './lexical-rollups';
import { SCHEMA_VERSION } from './types';
import type { QueuedWrite, VideoMetadata, YoutubeVideoMetadata } from './types';
import { toDbMs, toDbTimestamp } from './query-shared';
@@ -890,11 +891,11 @@ export function ensureSchema(db: DatabaseSync): void {
VALUES ('last_rollup_sample_ms', 0)
ON CONFLICT(state_key) DO NOTHING
`);
const currentVersion = db
.prepare('SELECT schema_version FROM imm_schema_version ORDER BY schema_version DESC LIMIT 1')
.get() as { schema_version: number } | null;
if (currentVersion?.schema_version === SCHEMA_VERSION) {
ensureLexicalDailyRollupTables(db);
ensureLifetimeSummaryTables(db);
ensureStatsExcludedWordsTable(db);
ensureAnimeMergeTables(db);
@@ -1068,6 +1069,7 @@ export function ensureSchema(db: DatabaseSync): void {
last_seen REAL,
frequency INTEGER,
frequency_rank INTEGER,
vocabulary_visible INTEGER NOT NULL DEFAULT 1 CHECK(vocabulary_visible IN (0, 1)),
UNIQUE(headword, word, reading)
);
`);
@@ -1451,8 +1453,18 @@ export function ensureSchema(db: DatabaseSync): void {
addColumnIfMissing(db, 'imm_sessions', 'ended_media_ms', 'INTEGER');
}
if (currentVersion?.schema_version && currentVersion.schema_version < 23) {
addColumnIfMissing(
db,
'imm_words',
'vocabulary_visible',
'INTEGER NOT NULL DEFAULT 1 CHECK(vocabulary_visible IN (0, 1))',
);
}
migrateSessionEventTimestampsToText(db);
ensureLexicalDailyRollupTables(db);
ensureLifetimeSummaryTables(db);
ensureStatsExcludedWordsTable(db);
@@ -1572,19 +1584,21 @@ export function ensureSchema(db: DatabaseSync): void {
ON imm_youtube_videos(youtube_video_id)
`);
if (currentVersion?.schema_version && currentVersion.schema_version < SCHEMA_VERSION) {
db.exec('DELETE FROM imm_daily_rollups');
db.exec('DELETE FROM imm_monthly_rollups');
db.exec(
`UPDATE imm_rollup_state SET state_value = 0 WHERE state_key = 'last_rollup_sample_ms'`,
);
}
// Session rollups intentionally outlive raw session and telemetry retention.
// Preserve them across unrelated schema upgrades because deleted historical
// buckets cannot be rebuilt after their source rows have been pruned.
db.exec(`
INSERT INTO imm_schema_version(schema_version, applied_at_ms)
VALUES (${SCHEMA_VERSION}, ${toDbTimestamp(nowMs())})
ON CONFLICT DO NOTHING
`);
// A new database has no history to materialize. Upgrades are populated by the
// background worker so startup never scans the existing vocabulary table.
if (!currentVersion) {
markLexicalDailyRollupsReady(db);
}
}
export function createTrackerPreparedStatements(db: DatabaseSync): TrackerPreparedStatements {
@@ -1617,9 +1631,10 @@ export function createTrackerPreparedStatements(db: DatabaseSync): TrackerPrepar
`),
wordUpsertStmt: db.prepare(`
INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3, first_seen, last_seen, frequency, frequency_rank
headword, word, reading, part_of_speech, pos1, pos2, pos3, first_seen, last_seen,
frequency, frequency_rank, vocabulary_visible
) VALUES (
?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?
?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?, 1
)
ON CONFLICT(headword, word, reading) DO UPDATE SET
frequency = COALESCE(frequency, 0) + 1,
@@ -1632,6 +1647,7 @@ export function createTrackerPreparedStatements(db: DatabaseSync): TrackerPrepar
pos1 = COALESCE(NULLIF(imm_words.pos1, ''), excluded.pos1),
pos2 = COALESCE(NULLIF(imm_words.pos2, ''), excluded.pos2),
pos3 = COALESCE(NULLIF(imm_words.pos3, ''), excluded.pos3),
vocabulary_visible = 1,
first_seen = MIN(COALESCE(first_seen, excluded.first_seen), excluded.first_seen),
last_seen = MAX(COALESCE(last_seen, excluded.last_seen), excluded.last_seen),
frequency_rank = CASE
+11 -1
View File
@@ -1,4 +1,4 @@
export const SCHEMA_VERSION = 21;
export const SCHEMA_VERSION = 23;
export const DEFAULT_QUEUE_CAP = 1_000;
export const DEFAULT_BATCH_SIZE = 25;
export const DEFAULT_FLUSH_INTERVAL_MS = 500;
@@ -306,6 +306,16 @@ export interface VocabularyStatsRow {
lastSeen: number;
}
export interface VocabularyStatsSummary {
uniqueWords: number;
uniqueWordsWithoutNames: number;
uniqueKanji: number;
newThisWeek: number;
newThisWeekWithoutNames: number;
knownWordCount: number | null;
knownWordCountWithoutNames: number | null;
}
export interface StatsExcludedWordRow {
headword: string;
word: string;
@@ -0,0 +1,100 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
import {
resolveVocabularySummaryWorkerPath,
VocabularySummaryWorkerRuntime,
} from './vocabulary-summary-worker-runtime';
import { Database } from './sqlite';
import { applyPragmas, ensureSchema } from './storage';
test('vocabulary summary worker reads the database from a separate connection', async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-vocabulary-summary-worker-'));
const dbPath = path.join(tempDir, 'immersion.sqlite');
const runtime = new VocabularySummaryWorkerRuntime();
const db = new Database(dbPath);
try {
applyPragmas(db);
ensureSchema(db);
db.prepare(
`
INSERT INTO imm_words (
headword, word, reading, part_of_speech, pos1, pos2, pos3,
first_seen, last_seen, frequency
) VALUES ('猫', '猫', 'ねこ', 'noun', '名詞', '一般', '', 1, 1, 1)
`,
).run();
db.close();
const summary = await runtime.run(dbPath, new Set(['猫']));
assert.equal(summary.uniqueWords, 1);
assert.equal(summary.knownWordCount, 1);
} finally {
runtime.destroy();
try {
db.close();
} catch {
// The worker needs the setup connection closed before it starts.
}
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
test('vocabulary summary worker module resolves in the current layout', () => {
const workerPath = resolveVocabularySummaryWorkerPath();
assert.ok(workerPath, 'expected the vocabulary summary worker module to resolve');
assert.ok(workerPath.endsWith(__filename.endsWith('.ts') ? '.ts' : '.js'));
});
test('vocabulary summary worker never falls back to the caller thread', async () => {
const runtime = new VocabularySummaryWorkerRuntime({
resolveWorkerPath: () => null,
warn: () => {},
});
try {
await assert.rejects(
runtime.run('/tmp/subminer-summary-worker-not-used.sqlite', null),
/worker unavailable/i,
);
} finally {
runtime.destroy();
}
});
test('vocabulary summary worker times out when it never responds', async () => {
let terminated = false;
const runtime = new VocabularySummaryWorkerRuntime({
resolveWorkerPath: () => '/tmp/fake-worker.js',
createWorker: async () => ({
once() {
return this;
},
terminate: async () => {
terminated = true;
return 0;
},
}),
timeoutMs: 1,
warn: () => {},
} as never);
try {
const outcome = await Promise.race([
runtime.run('/tmp/not-used.sqlite', null).then(
() => 'resolved',
(error: unknown) => String(error),
),
new Promise<string>((resolve) => setTimeout(() => resolve('still pending'), 50)),
]);
assert.match(outcome, /timed out/);
assert.equal(terminated, true);
} finally {
runtime.destroy();
}
});
@@ -0,0 +1,133 @@
import fs from 'node:fs';
import path from 'node:path';
import { createLogger } from '../../../logger';
import type { VocabularyStatsSummary } from './types';
interface VocabularySummaryWorkerResponse {
summary?: VocabularyStatsSummary;
error?: unknown;
}
interface VocabularySummaryWorkerHandle {
once(event: 'message', listener: (message: VocabularySummaryWorkerResponse) => void): this;
once(event: 'error', listener: (error: Error) => void): this;
once(event: 'exit', listener: (code: number) => void): this;
terminate(): Promise<number>;
}
interface VocabularySummaryWorkerRuntimeOptions {
resolveWorkerPath?: () => string | null;
createWorker?: (
workerPath: string,
workerData: { dbPath: string; knownWords: string[] | null },
) => Promise<VocabularySummaryWorkerHandle>;
timeoutMs?: number;
warn?: (message: string, ...meta: unknown[]) => void;
}
export type RunVocabularySummaryTask = (
dbPath: string,
knownWords: ReadonlySet<string> | null,
) => Promise<VocabularyStatsSummary>;
export function resolveVocabularySummaryWorkerPath(): string | null {
const fileName = __filename.endsWith('.ts')
? 'vocabulary-summary-worker-thread.ts'
: 'vocabulary-summary-worker-thread.js';
const workerPath = path.join(__dirname, fileName);
return fs.existsSync(workerPath) ? workerPath : null;
}
const logger = createLogger('main:immersion-tracker:vocabulary-summary-worker');
const DEFAULT_WORKER_TIMEOUT_MS = 5 * 60 * 1_000;
export class VocabularySummaryWorkerRuntime {
private readonly activeWorkers = new Set<VocabularySummaryWorkerHandle>();
private destroyed = false;
constructor(private readonly options: VocabularySummaryWorkerRuntimeOptions = {}) {}
async run(
dbPath: string,
knownWords: ReadonlySet<string> | null,
): Promise<VocabularyStatsSummary> {
if (this.destroyed) throw new Error('Vocabulary summary worker is shut down');
const workerData = { dbPath, knownWords: knownWords ? [...knownWords] : null };
let worker: VocabularySummaryWorkerHandle;
try {
const workerPath = (this.options.resolveWorkerPath ?? resolveVocabularySummaryWorkerPath)();
if (!workerPath) throw new Error('Emitted vocabulary summary worker module was not found');
const createWorker =
this.options.createWorker ??
(async (resolvedPath, data) => {
const { Worker } = await import('node:worker_threads');
return new Worker(resolvedPath, { workerData: data });
});
worker = await createWorker(workerPath, workerData);
} catch (error) {
if (this.destroyed) throw new Error('Vocabulary summary worker is shut down');
(this.options.warn ?? logger.warn)(
'Vocabulary summary worker unavailable; refusing to scan vocabulary on the current thread',
error,
);
throw new Error('Vocabulary summary worker unavailable');
}
if (this.destroyed) {
await worker.terminate().catch(() => undefined);
throw new Error('Vocabulary summary worker is shut down');
}
return new Promise<VocabularyStatsSummary>((resolve, reject) => {
let settled = false;
let timeout: ReturnType<typeof setTimeout> | null = null;
this.activeWorkers.add(worker);
const settle = (result: VocabularyStatsSummary | Error) => {
if (settled) return;
settled = true;
if (timeout) clearTimeout(timeout);
this.activeWorkers.delete(worker);
void worker.terminate().catch(() => undefined);
if (result instanceof Error) reject(result);
else resolve(result);
};
timeout = setTimeout(
() => settle(new Error('Vocabulary summary worker timed out')),
this.options.timeoutMs ?? DEFAULT_WORKER_TIMEOUT_MS,
);
worker.once('message', (message) => {
if (message.summary) {
settle(message.summary);
return;
}
settle(
new Error(
`Vocabulary summary failed: ${String(message.error ?? 'unknown worker error')}`,
),
);
});
worker.once('error', (error) => settle(error));
worker.once('exit', (code) => {
if (!settled) {
settle(
new Error(
code === 0
? 'Vocabulary summary worker exited without a response'
: `Vocabulary summary worker exited with code ${code}`,
),
);
}
});
});
}
destroy(): void {
if (this.destroyed) return;
this.destroyed = true;
for (const worker of this.activeWorkers) {
void worker.terminate().catch(() => undefined);
}
this.activeWorkers.clear();
}
}
@@ -0,0 +1,19 @@
import { parentPort, workerData } from 'node:worker_threads';
import { executeVocabularySummaryTask } from './vocabulary-summary-worker';
interface VocabularySummaryWorkerData {
dbPath: string;
knownWords: string[] | null;
}
if (!parentPort) throw new Error('vocabulary summary worker missing parent port');
const request = workerData as VocabularySummaryWorkerData;
try {
parentPort.postMessage({
summary: executeVocabularySummaryTask(request.dbPath, request.knownWords),
});
} catch (error) {
parentPort.postMessage({ error: error instanceof Error ? error.message : String(error) });
}
@@ -0,0 +1,17 @@
import { getVocabularySummary } from './query-lexical';
import { Database } from './sqlite';
import { applyPragmas } from './storage';
import type { VocabularyStatsSummary } from './types';
export function executeVocabularySummaryTask(
dbPath: string,
knownWords: string[] | null,
): VocabularyStatsSummary {
const db = new Database(dbPath);
try {
applyPragmas(db);
return getVocabularySummary(db, knownWords ? new Set(knownWords) : null);
} finally {
db.close();
}
}
@@ -0,0 +1,43 @@
import { PartOfSpeech, type MergedToken } from '../../../types';
import { shouldExcludeTokenFromVocabularyPersistence } from '../tokenizer/annotation-stage';
export interface VocabularyVisibilityRow {
word: string | null;
headword: string | null;
reading?: string | null;
partOfSpeech?: string | null;
pos1?: string | null;
pos2?: string | null;
pos3?: string | null;
frequencyRank?: number | null;
}
function toVocabularyToken(row: VocabularyVisibilityRow): MergedToken {
const word = row.word ?? '';
const headword = row.headword ?? word;
const partOfSpeech =
row.partOfSpeech && Object.values(PartOfSpeech).includes(row.partOfSpeech as PartOfSpeech)
? (row.partOfSpeech as PartOfSpeech)
: PartOfSpeech.other;
return {
surface: word,
reading: row.reading ?? '',
headword,
startPos: 0,
endPos: word.length,
partOfSpeech,
pos1: row.pos1 ?? '',
pos2: row.pos2 ?? '',
pos3: row.pos3 ?? '',
frequencyRank: row.frequencyRank ?? undefined,
isMerged: false,
isKnown: false,
isNPlusOneTarget: false,
};
}
export function isVocabularyStatsRowVisible(row: VocabularyVisibilityRow): boolean {
if (!(row.word?.trim() || row.headword?.trim())) return false;
return !shouldExcludeTokenFromVocabularyPersistence(toVocabularyToken(row));
}

Some files were not shown because too many files have changed in this diff Show More