Compare commits

...

3 Commits

17 changed files with 793 additions and 60 deletions
@@ -0,0 +1,44 @@
---
id: TASK-287
title: Restore Lua parser compatibility for mpv plugin modules
status: Done
assignee: []
created_date: '2026-04-11 21:25'
updated_date: '2026-04-11 21:29'
labels:
- bug
- mpv-plugin
- lua
dependencies: []
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Users with Lua runtimes that do not accept the current `goto continue` pattern in the mpv plugin should be able to load the plugin without syntax errors. Remove the parser-incompatible control-flow usage from the affected plugin modules without changing plugin behavior.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 The mpv plugin source no longer relies on parser-incompatible `goto continue` labels in the affected Lua modules.
- [x] #2 Automated coverage fails on the old parser-incompatible source and passes once the compatibility fix is in place.
- [x] #3 Existing plugin start/gate verification still passes after the compatibility fix.
<!-- AC:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
Reused existing local cleanups in `plugin/subminer/hover.lua` and `plugin/subminer/environment.lua` to remove `goto continue` / `::continue::` control flow without behavior changes.
Added `scripts/test-plugin-lua-compat.lua` and wired it into `test:plugin:src`; the regression checks reject the legacy pattern structurally and verify parse success with `luajit` when available.
Verification run on 2026-04-11: `lua scripts/test-plugin-lua-compat.lua` ✅, `bun run test:plugin:src` ✅, `bun run changelog:lint` ✅, `bun run typecheck` ✅, `bun run test:env` ✅, `bun run build` ✅, `bun run test:smoke:dist` ✅.
`bun run test:fast` remains red for unrelated existing immersion-tracker assertions in `src/core/services/immersion-tracker/__tests__/query-split-modules.test.ts` and `src/core/services/immersion-tracker/__tests__/query.test.ts` (`tsMs`/`lastWatchedMs` observed as `-2147483648`).
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Removed parser-incompatible `goto continue` usage from the affected mpv Lua plugin modules, added a dedicated Lua compatibility regression script to the plugin test lane, and added a changelog fragment for the user-visible fix. Requested plugin verification is green; unrelated existing `test:fast` immersion-tracker failures remain outside this task.
<!-- SECTION:FINAL_SUMMARY:END -->
@@ -0,0 +1,42 @@
---
id: TASK-288
title: Stabilize immersion-tracker CI timestamp handling under libsql/Bun
status: Done
assignee: []
created_date: '2026-04-11 21:34'
updated_date: '2026-04-11 21:43'
labels:
- bug
- ci
- immersion-tracker
dependencies: []
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
`bun run test:fast` is currently failing because large millisecond timestamps are not handled safely through the libsql/Bun path. Fix timestamp parsing/storage so lifetime/library and session-event queries return correct wall-clock values in CI and runtime.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 Large wall-clock timestamps round-trip correctly through immersion-tracker lifetime/library queries under the repo's Bun/libsql runtime.
- [x] #2 Session-event timestamps round-trip correctly for real wall-clock values used by runtime event inserts.
- [x] #3 Targeted immersion-tracker regression coverage passes, and the previously failing `test:fast` lane no longer fails on these timestamp assertions.
<!-- AC:END -->
## Implementation Notes
<!-- SECTION:NOTES:BEGIN -->
Root cause split in two places: Bun/libsql corrupts large millisecond timestamp strings when coerced through `Number(...)`, and `imm_session_events.ts_ms` being `INTEGER` let runtime event inserts/readbacks return `-2147483648` on CI/runtime.
Fix shipped by parsing timestamp strings without the broken `Number(largeString)` path, migrating `imm_session_events.ts_ms` to `TEXT`, ordering/retention queries via `CAST(ts_ms AS REAL)`, and avoiding `Number(currentMs)` when reusing already-normalized timestamp strings.
Added regression coverage for both real runtime event inserts and schema migration/repair of previously truncated session-event rows.
<!-- SECTION:NOTES:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Fixed immersion-tracker timestamp handling under Bun/libsql so large wall-clock millisecond values survive runtime writes, query reads, and schema migration. `bun run test:fast`, `bun run typecheck`, `bun run test:env`, `bun run build`, `bun run test:smoke:dist`, and `bun run changelog:lint` all pass after the patch.
<!-- SECTION:FINAL_SUMMARY:END -->
@@ -0,0 +1,4 @@
type: fixed
area: stats
- Fixed immersion-tracker timestamp handling under Bun/libsql so library rows, session timelines, and lifetime summaries keep real wall-clock millisecond values instead of truncating to invalid negative timestamps.
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: mpv-plugin
- Fixed the mpv Lua plugin so hover and environment modules no longer use the `goto continue` pattern that can fail to parse on some user Lua runtimes.
+1 -1
View File
@@ -44,7 +44,7 @@
"test:config:src": "bun test src/config/config.test.ts src/config/path-resolution.test.ts src/config/resolve/anki-connect.test.ts src/config/resolve/integrations.test.ts src/config/resolve/subtitle-style.test.ts src/config/resolve/jellyfin.test.ts src/config/definitions/domain-registry.test.ts src/generate-config-example.test.ts src/verify-config-example.test.ts",
"test:config:dist": "bun test dist/config/config.test.js dist/config/path-resolution.test.js dist/config/resolve/anki-connect.test.js dist/config/resolve/integrations.test.js dist/config/resolve/subtitle-style.test.js dist/config/resolve/jellyfin.test.js dist/config/definitions/domain-registry.test.js dist/generate-config-example.test.js dist/verify-config-example.test.js",
"test:config:smoke:dist": "bun test dist/config/path-resolution.test.js",
"test:plugin:src": "lua scripts/test-plugin-start-gate.lua && lua scripts/test-plugin-binary-windows.lua",
"test:plugin:src": "lua scripts/test-plugin-lua-compat.lua && lua scripts/test-plugin-start-gate.lua && lua scripts/test-plugin-binary-windows.lua",
"test:launcher:smoke:src": "bun test launcher/smoke.e2e.test.ts",
"test:launcher:src": "bun test launcher/config.test.ts launcher/config-domain-parsers.test.ts launcher/config/cli-parser-builder.test.ts launcher/config/args-normalizer.test.ts launcher/mpv.test.ts launcher/picker.test.ts launcher/parse-args.test.ts launcher/main.test.ts launcher/commands/command-modules.test.ts launcher/smoke.e2e.test.ts && bun run test:plugin:src",
"test:core:src": "bun test src/cli/args.test.ts src/cli/help.test.ts src/shared/setup-state.test.ts src/core/services/cli-command.test.ts src/core/services/field-grouping-overlay.test.ts src/core/services/numeric-shortcut-session.test.ts src/core/services/secondary-subtitle.test.ts src/core/services/mpv-render-metrics.test.ts src/core/services/overlay-content-measurement.test.ts src/core/services/mpv-control.test.ts src/core/services/mpv.test.ts src/core/services/runtime-options-ipc.test.ts src/core/services/runtime-config.test.ts src/core/services/yomitan-extension-paths.test.ts src/core/services/config-hot-reload.test.ts src/core/services/discord-presence.test.ts src/core/services/tokenizer.test.ts src/core/services/tokenizer/annotation-stage.test.ts src/core/services/tokenizer/parser-selection-stage.test.ts src/core/services/tokenizer/parser-enrichment-stage.test.ts src/core/services/subsync.test.ts src/core/services/overlay-bridge.test.ts src/core/services/overlay-shortcut-handler.test.ts src/core/services/stats-window.test.ts src/core/services/mining.test.ts src/core/services/anki-jimaku.test.ts src/core/services/jimaku-download-path.test.ts src/core/services/jellyfin.test.ts src/core/services/jellyfin-remote.test.ts src/core/services/immersion-tracker-service.test.ts src/core/services/overlay-runtime-init.test.ts src/core/services/app-ready.test.ts src/core/services/startup-bootstrap.test.ts src/core/services/subtitle-processing-controller.test.ts src/core/services/anilist/anilist-update-queue.test.ts src/core/services/anilist/rate-limiter.test.ts src/core/services/jlpt-token-filter.test.ts src/core/services/subtitle-position.test.ts src/core/utils/shortcut-config.test.ts src/main/runtime/first-run-setup-plugin.test.ts src/main/runtime/first-run-setup-service.test.ts src/main/runtime/first-run-setup-window.test.ts src/main/runtime/tray-runtime.test.ts src/main/runtime/tray-main-actions.test.ts src/main/runtime/tray-main-deps.test.ts src/main/runtime/tray-runtime-handlers.test.ts src/main/runtime/cli-command-context-main-deps.test.ts src/main/runtime/app-ready-main-deps.test.ts src/renderer/error-recovery.test.ts src/renderer/subtitle-render.test.ts src/renderer/handlers/mouse.test.ts src/renderer/handlers/keyboard.test.ts src/renderer/modals/jimaku.test.ts src/subsync/utils.test.ts src/main/anilist-url-guard.test.ts src/window-trackers/hyprland-tracker.test.ts src/window-trackers/x11-tracker.test.ts src/window-trackers/windows-helper.test.ts src/window-trackers/windows-tracker.test.ts launcher/config.test.ts launcher/config-domain-parsers.test.ts launcher/config/cli-parser-builder.test.ts launcher/config/args-normalizer.test.ts launcher/parse-args.test.ts launcher/main.test.ts launcher/commands/command-modules.test.ts launcher/setup-gate.test.ts stats/src/lib/api-client.test.ts",
+15 -22
View File
@@ -55,33 +55,26 @@ function M.create(ctx)
if not image then
image = line:match('^"([^"]+)"')
end
if not image then
goto continue
end
if image == "subminer" or image == "subminer.exe" or image == "subminer.appimage" or image == "subminer.app" then
return true
end
if image:find("subminer", 1, true) and not image:find(".lua", 1, true) then
return true
if image then
if image == "subminer" or image == "subminer.exe" or image == "subminer.appimage" or image == "subminer.app" then
return true
end
if image:find("subminer", 1, true) and not image:find(".lua", 1, true) then
return true
end
end
else
local argv0 = line:match('^"([^"]+)"') or line:match("^%s*([^%s]+)")
if not argv0 then
goto continue
end
if argv0:find("subminer.lua", 1, true) or argv0:find("subminer.conf", 1, true) then
goto continue
end
local exe = argv0:match("([^/\\]+)$") or argv0
if exe == "SubMiner" or exe == "SubMiner.AppImage" or exe == "SubMiner.exe" or exe == "subminer" or exe == "subminer.appimage" or exe == "subminer.exe" then
return true
end
if exe:find("subminer", 1, true) and exe:find("%.lua", 1, true) == nil and exe:find("%.app", 1, true) == nil then
return true
if argv0 and not argv0:find("subminer.lua", 1, true) and not argv0:find("subminer.conf", 1, true) then
local exe = argv0:match("([^/\\]+)$") or argv0
if exe == "SubMiner" or exe == "SubMiner.AppImage" or exe == "SubMiner.exe" or exe == "subminer" or exe == "subminer.appimage" or exe == "subminer.exe" then
return true
end
if exe:find("subminer", 1, true) and exe:find("%.lua", 1, true) == nil and exe:find("%.app", 1, true) == nil then
return true
end
end
end
::continue::
end
return false
end
+23 -27
View File
@@ -189,41 +189,37 @@ function M.create(ctx)
local source_len = #plain
local cursor = 1
for _, token in ipairs(payload.tokens or {}) do
if type(token) ~= "table" or type(token.text) ~= "string" or token.text == "" then
goto continue
end
if type(token) == "table" and type(token.text) == "string" and token.text ~= "" then
local token_text = token.text
local start_pos = nil
local end_pos = nil
local token_text = token.text
local start_pos = nil
local end_pos = nil
if type(token.startPos) == "number" and type(token.endPos) == "number" then
if token.startPos >= 0 and token.endPos >= token.startPos then
local candidate_start = token.startPos + 1
local candidate_stop = token.endPos
if candidate_start >= 1 and candidate_stop <= source_len and candidate_stop >= candidate_start and plain:sub(candidate_start, candidate_stop) == token_text then
start_pos = candidate_start
end_pos = candidate_stop
if type(token.startPos) == "number" and type(token.endPos) == "number" then
if token.startPos >= 0 and token.endPos >= token.startPos then
local candidate_start = token.startPos + 1
local candidate_stop = token.endPos
if candidate_start >= 1 and candidate_stop <= source_len and candidate_stop >= candidate_start and plain:sub(candidate_start, candidate_stop) == token_text then
start_pos = candidate_start
end_pos = candidate_stop
end
end
end
end
if not start_pos or not end_pos then
local fallback_start, fallback_stop = plain:find(token_text, cursor, true)
if not fallback_start then
fallback_start, fallback_stop = plain:find(token_text, 1, true)
if not start_pos or not end_pos then
local fallback_start, fallback_stop = plain:find(token_text, cursor, true)
if not fallback_start then
fallback_start, fallback_stop = plain:find(token_text, 1, true)
end
start_pos, end_pos = fallback_start, fallback_stop
end
start_pos, end_pos = fallback_start, fallback_stop
end
if start_pos and end_pos then
if token.index == payload.hoveredTokenIndex then
return start_pos, end_pos
if start_pos and end_pos then
if token.index == payload.hoveredTokenIndex then
return start_pos, end_pos
end
cursor = end_pos + 1
end
cursor = end_pos + 1
end
::continue::
end
return nil
+141
View File
@@ -0,0 +1,141 @@
local MODULE_PATHS = {
"plugin/subminer/hover.lua",
"plugin/subminer/environment.lua",
}
local LEGACY_PARSER_CANDIDATES = {
"luajit",
"lua5.1",
"lua51",
}
local function assert_true(condition, message)
if condition then
return
end
error(message or "assert_true failed")
end
local function read_file(path)
local file = assert(io.open(path, "r"), "failed to open " .. path)
local content = file:read("*a")
file:close()
return content
end
local function find_legacy_incompatible_continue(source)
local goto_start, goto_end = source:find("%f[%a]goto%s+continue%f[%A]")
if goto_start then
return "goto continue", goto_start, goto_end
end
local label_start, label_end = source:find("::continue::", 1, true)
if label_start then
return "::continue::", label_start, label_end
end
return nil
end
local function assert_no_legacy_incompatible_continue(path)
local source = read_file(path)
local match = find_legacy_incompatible_continue(source)
assert_true(match == nil, path .. " still contains legacy-incompatible continue control flow: " .. tostring(match))
end
local function assert_loadfile_ok(path)
local chunk, err = loadfile(path)
assert_true(chunk ~= nil, "loadfile failed for " .. path .. ": " .. tostring(err))
end
local function normalize_execute_result(ok, why, code)
if type(ok) == "number" then
return ok == 0, ok
end
if type(ok) == "boolean" then
if ok then
return true, code or 0
end
return false, code or 1
end
return false, code or 1
end
local function command_succeeds(command)
local ok, why, code = os.execute(command)
return normalize_execute_result(ok, why, code)
end
local function command_exists(command)
local shell = package.config:sub(1, 1) == "\\" and "where " or "command -v "
local redirect = package.config:sub(1, 1) == "\\" and " >NUL 2>NUL" or " >/dev/null 2>&1"
local escaped = command
local success = command_succeeds(shell .. escaped .. redirect)
return success
end
local function find_legacy_parser()
for _, command in ipairs(LEGACY_PARSER_CANDIDATES) do
if command_exists(command) then
return command
end
end
return nil
end
local function shell_redirect()
if package.config:sub(1, 1) == "\\" then
return " >NUL 2>NUL"
end
return " >/dev/null 2>&1"
end
local function assert_parser_accepts_file(parser, path)
local command = string.format("%s -e %q%s", parser, "assert(loadfile(" .. string.format("%q", path) .. "))", shell_redirect())
local success = command_succeeds(command)
assert_true(success, parser .. " failed to parse " .. path)
end
local function assert_parser_rejects_legacy_fixture(parser)
local legacy_fixture = [[
local tokens = {}
for _, token in ipairs(tokens or {}) do
if type(token) ~= "table" then
goto continue
end
::continue::
end
]]
local command = string.format("%s -e %q%s", parser, legacy_fixture, shell_redirect())
local success = command_succeeds(command)
assert_true(not success, parser .. " unexpectedly accepted legacy goto/label continue fixture")
end
do
local legacy_fixture = [[
for _, token in ipairs(tokens or {}) do
if type(token) ~= "table" then
goto continue
end
::continue::
end
]]
local match = find_legacy_incompatible_continue(legacy_fixture)
assert_true(match ~= nil, "legacy fixture should trigger incompatible continue detector")
end
for _, path in ipairs(MODULE_PATHS) do
assert_no_legacy_incompatible_continue(path)
assert_loadfile_ok(path)
end
local parser = find_legacy_parser()
if parser then
assert_parser_rejects_legacy_fixture(parser)
for _, path in ipairs(MODULE_PATHS) do
assert_parser_accepts_file(parser, path)
end
print("plugin lua compatibility regression tests: OK (" .. parser .. ")")
else
print("plugin lua compatibility regression tests: OK (legacy parser unavailable; structural checks only)")
end
@@ -1938,6 +1938,50 @@ test('getSessionEvents returns events ordered by ts_ms ascending', () => {
}
});
test('getSessionEvents round-trips wall-clock timestamps written through event inserts', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
ensureSchema(db);
const stmts = createTrackerPreparedStatements(db);
const videoId = getOrCreateVideoRecord(db, 'local:/tmp/events-wall-clock.mkv', {
canonicalTitle: 'Events Wall Clock',
sourcePath: '/tmp/events-wall-clock.mkv',
sourceUrl: null,
sourceType: SOURCE_TYPE_LOCAL,
});
const startedAtMs = Date.now() - 10_000;
const eventTsMs = startedAtMs + 5_000;
const { sessionId } = startSessionRecord(db, videoId, startedAtMs);
stmts.eventInsertStmt.run(
sessionId,
toDbTimestamp(eventTsMs),
EVENT_SUBTITLE_LINE,
0,
0,
500,
1,
0,
'{"line":"wall-clock"}',
toDbTimestamp(eventTsMs),
toDbTimestamp(eventTsMs),
);
const events = getSessionEvents(db, sessionId, 10);
assert.equal(events.length, 1);
assert.equal(events[0]?.tsMs, eventTsMs);
assert.equal(events[0]?.payload, '{"line":"wall-clock"}');
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('getSessionEvents returns empty array for session with no events', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
@@ -66,7 +66,7 @@ export function pruneRawRetention(
const deletedSessionEvents = Number.isFinite(policy.eventsRetentionMs)
? (
db
.prepare(`DELETE FROM imm_session_events WHERE ts_ms < ?`)
.prepare(`DELETE FROM imm_session_events WHERE CAST(ts_ms AS REAL) < CAST(? AS REAL)`)
.run(resolveCutoff(policy.eventsRetentionMs, policy.eventsRetentionDays)) as {
changes: number;
}
@@ -133,7 +133,7 @@ export function getSessionEvents(
if (!eventTypes || eventTypes.length === 0) {
const stmt = db.prepare(`
SELECT event_type AS eventType, ts_ms AS tsMs, payload_json AS payload
FROM imm_session_events WHERE session_id = ? ORDER BY ts_ms ASC LIMIT ?
FROM imm_session_events WHERE session_id = ? ORDER BY CAST(ts_ms AS REAL) ASC LIMIT ?
`);
const rows = stmt.all(sessionId, limit) as Array<SessionEventRow & { tsMs: number | string }>;
return rows.map((row) => ({
@@ -147,7 +147,7 @@ export function getSessionEvents(
SELECT event_type AS eventType, ts_ms AS tsMs, payload_json AS payload
FROM imm_session_events
WHERE session_id = ? AND event_type IN (${placeholders})
ORDER BY ts_ms ASC
ORDER BY CAST(ts_ms AS REAL) ASC
LIMIT ?
`);
const rows = stmt.all(sessionId, ...eventTypes, limit) as Array<
@@ -602,7 +602,7 @@ export function getEpisodeCardEvents(db: DatabaseSync, videoId: number): Episode
FROM imm_session_events e
JOIN imm_sessions s ON s.session_id = e.session_id
WHERE s.video_id = ? AND e.event_type = 4
ORDER BY e.ts_ms DESC
ORDER BY CAST(e.ts_ms AS REAL) DESC
`,
)
.all(videoId) as Array<{
@@ -345,7 +345,11 @@ export function fromDbTimestamp(ms: number | bigint | string | null | undefined)
if (typeof ms === 'bigint') {
return Number(ms);
}
return Number(ms);
const normalized = normalizeTimestampString(ms);
if (/^-?\d+$/.test(normalized)) {
return Number(BigInt(normalized));
}
return Math.trunc(Number.parseFloat(normalized));
}
function getNumericCalendarValue(
@@ -263,6 +263,370 @@ test('ensureSchema adds youtube metadata table to existing schema version 15 dat
}
});
test('ensureSchema migrates session event timestamps to text and repairs libsql-truncated wall-clock values', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
try {
db.exec(`
CREATE TABLE imm_schema_version (
schema_version INTEGER PRIMARY KEY,
applied_at_ms INTEGER NOT NULL
);
INSERT INTO imm_schema_version(schema_version, applied_at_ms) VALUES (16, 1000);
CREATE TABLE imm_rollup_state(
state_key TEXT PRIMARY KEY,
state_value INTEGER NOT NULL
);
INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0);
CREATE TABLE imm_anime(
anime_id INTEGER PRIMARY KEY AUTOINCREMENT,
normalized_title_key TEXT NOT NULL UNIQUE,
canonical_title TEXT NOT NULL,
anilist_id INTEGER UNIQUE,
title_romaji TEXT,
title_english TEXT,
title_native TEXT,
episodes_total INTEGER,
description TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_videos(
video_id INTEGER PRIMARY KEY AUTOINCREMENT,
video_key TEXT NOT NULL UNIQUE,
anime_id INTEGER,
canonical_title TEXT NOT NULL,
source_type INTEGER NOT NULL,
source_path TEXT,
source_url TEXT,
parsed_basename TEXT,
parsed_title TEXT,
parsed_season INTEGER,
parsed_episode INTEGER,
parser_source TEXT,
parser_confidence REAL,
parse_metadata_json TEXT,
watched INTEGER NOT NULL DEFAULT 0,
duration_ms INTEGER NOT NULL CHECK(duration_ms>=0),
file_size_bytes INTEGER CHECK(file_size_bytes>=0),
codec_id INTEGER, container_id INTEGER,
width_px INTEGER, height_px INTEGER, fps_x100 INTEGER,
bitrate_kbps INTEGER, audio_codec_id INTEGER,
hash_sha256 TEXT, screenshot_path TEXT,
metadata_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_sessions(
session_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_uuid TEXT NOT NULL UNIQUE,
video_id INTEGER NOT NULL,
started_at_ms TEXT NOT NULL,
ended_at_ms TEXT,
status INTEGER NOT NULL,
locale_id INTEGER,
target_lang_id INTEGER,
difficulty_tier INTEGER,
subtitle_mode INTEGER,
ended_media_ms INTEGER,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id)
);
CREATE TABLE imm_session_telemetry(
telemetry_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
sample_ms TEXT NOT NULL,
total_watched_ms INTEGER NOT NULL DEFAULT 0,
active_watched_ms INTEGER NOT NULL DEFAULT 0,
lines_seen INTEGER NOT NULL DEFAULT 0,
tokens_seen INTEGER NOT NULL DEFAULT 0,
cards_mined INTEGER NOT NULL DEFAULT 0,
lookup_count INTEGER NOT NULL DEFAULT 0,
lookup_hits INTEGER NOT NULL DEFAULT 0,
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
pause_count INTEGER NOT NULL DEFAULT 0,
pause_ms INTEGER NOT NULL DEFAULT 0,
seek_forward_count INTEGER NOT NULL DEFAULT 0,
seek_backward_count INTEGER NOT NULL DEFAULT 0,
media_buffer_events INTEGER NOT NULL DEFAULT 0,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_session_events(
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
ts_ms INTEGER NOT NULL,
event_type INTEGER NOT NULL,
line_index INTEGER,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
tokens_delta INTEGER NOT NULL DEFAULT 0,
cards_delta INTEGER NOT NULL DEFAULT 0,
payload_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_daily_rollups(
rollup_day INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
cards_per_hour REAL,
tokens_per_min REAL,
lookup_hit_rate REAL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_day, video_id)
);
CREATE TABLE imm_monthly_rollups(
rollup_month INTEGER NOT NULL,
video_id INTEGER,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_min REAL NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
cards_per_hour REAL,
tokens_per_min REAL,
lookup_hit_rate REAL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
PRIMARY KEY (rollup_month, video_id)
);
CREATE TABLE imm_words(
id INTEGER PRIMARY KEY AUTOINCREMENT,
headword TEXT NOT NULL,
word TEXT NOT NULL,
reading TEXT NOT NULL,
part_of_speech TEXT,
pos1 TEXT,
pos2 TEXT,
pos3 TEXT,
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
frequency INTEGER NOT NULL DEFAULT 0,
frequency_rank INTEGER,
UNIQUE(headword, word, reading)
);
CREATE TABLE imm_kanji(
id INTEGER PRIMARY KEY AUTOINCREMENT,
kanji TEXT NOT NULL UNIQUE,
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
frequency INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE imm_subtitle_lines(
line_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
event_id INTEGER,
video_id INTEGER NOT NULL,
anime_id INTEGER,
line_index INTEGER NOT NULL,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
text TEXT NOT NULL,
secondary_text TEXT,
CREATED_DATE INTEGER,
LAST_UPDATE_DATE INTEGER,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE,
FOREIGN KEY(event_id) REFERENCES imm_session_events(event_id) ON DELETE SET NULL,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
);
CREATE TABLE imm_word_line_occurrences(
line_id INTEGER NOT NULL,
word_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, word_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(word_id) REFERENCES imm_words(id) ON DELETE CASCADE
);
CREATE TABLE imm_kanji_line_occurrences(
line_id INTEGER NOT NULL,
kanji_id INTEGER NOT NULL,
occurrence_count INTEGER NOT NULL,
PRIMARY KEY(line_id, kanji_id),
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
FOREIGN KEY(kanji_id) REFERENCES imm_kanji(id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_global(
global_id INTEGER PRIMARY KEY CHECK(global_id = 1),
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
active_days INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
anime_completed INTEGER NOT NULL DEFAULT 0,
last_rebuilt_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_lifetime_anime(
anime_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
episodes_started INTEGER NOT NULL DEFAULT 0,
episodes_completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_media(
video_id INTEGER PRIMARY KEY,
total_sessions INTEGER NOT NULL DEFAULT 0,
total_active_ms INTEGER NOT NULL DEFAULT 0,
total_cards INTEGER NOT NULL DEFAULT 0,
total_lines_seen INTEGER NOT NULL DEFAULT 0,
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
completed INTEGER NOT NULL DEFAULT 0,
first_watched_ms TEXT,
last_watched_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_lifetime_applied_sessions(
session_id INTEGER PRIMARY KEY,
applied_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
CREATE TABLE imm_media_art(
video_id INTEGER PRIMARY KEY,
anilist_id INTEGER,
cover_url TEXT,
cover_blob BLOB,
cover_blob_hash TEXT,
fetched_at_ms TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
CREATE TABLE imm_cover_art_blobs(
blob_hash TEXT PRIMARY KEY,
cover_blob BLOB NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT
);
CREATE TABLE imm_youtube_videos(
video_id INTEGER PRIMARY KEY,
youtube_video_id TEXT,
video_url TEXT,
video_title TEXT,
video_thumbnail_url TEXT,
channel_id TEXT,
channel_name TEXT,
channel_url TEXT,
channel_thumbnail_url TEXT,
uploader_id TEXT,
uploader_url TEXT,
description TEXT,
metadata_json TEXT,
fetched_at_ms TEXT NOT NULL,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
);
INSERT INTO imm_videos (
video_id, video_key, canonical_title, source_type, source_path, source_url, watched, duration_ms,
CREATED_DATE, LAST_UPDATE_DATE
) VALUES (
1, 'local:/tmp/repaired-event.mkv', 'Repaired Event', 1, '/tmp/repaired-event.mkv', NULL, 0, 0, '1000', '1000'
);
INSERT INTO imm_sessions (
session_id, session_uuid, video_id, started_at_ms, status, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (
1, 'session-1', 1, '1775940000000', 1, '1775940000000', '1775940000000'
);
INSERT INTO imm_session_events (
event_id, session_id, ts_ms, event_type, line_index, segment_start_ms, segment_end_ms,
tokens_delta, cards_delta, payload_json, CREATED_DATE, LAST_UPDATE_DATE
) VALUES (
1, 1, -2147483648, 4, NULL, NULL, NULL, 0, 1, '{\"noteIds\":[1]}', '1775943304128', '1775943304128'
);
`);
ensureSchema(db);
const column = db.prepare(`PRAGMA table_info(imm_session_events)`).all() as Array<{
name: string;
type: string;
}>;
assert.equal(column.find((entry) => entry.name === 'ts_ms')?.type, 'TEXT');
const row = db.prepare(
`
SELECT ts_ms AS tsMs, typeof(ts_ms) AS tsType, CREATED_DATE AS createdDate
FROM imm_session_events
WHERE event_id = 1
`,
).get() as {
tsMs: string;
tsType: string;
createdDate: string;
};
assert.equal(row.tsType, 'text');
assert.equal(row.tsMs, '1775943304128');
assert.equal(row.createdDate, '1775943304128');
} finally {
db.close();
cleanupDbPath(dbPath);
}
});
test('ensureSchema creates large-history performance indexes', () => {
const dbPath = makeDbPath();
const db = new Database(dbPath);
+100 -3
View File
@@ -170,6 +170,14 @@ function hasColumn(db: DatabaseSync, tableName: string, columnName: string): boo
.some((row: unknown) => (row as { name: string }).name === columnName);
}
function getColumnType(db: DatabaseSync, tableName: string, columnName: string): string | null {
const row = (db.prepare(`PRAGMA table_info(${tableName})`).all() as Array<{
name: string;
type: string;
}>).find((entry) => entry.name === columnName);
return row?.type ?? null;
}
function addColumnIfMissing(
db: DatabaseSync,
tableName: string,
@@ -187,6 +195,92 @@ function dropColumnIfExists(db: DatabaseSync, tableName: string, columnName: str
}
}
function migrateSessionEventTimestampsToText(db: DatabaseSync): void {
if (getColumnType(db, 'imm_session_events', 'ts_ms') === 'TEXT') {
return;
}
const lineIndexExpr = hasColumn(db, 'imm_session_events', 'line_index') ? 'line_index' : 'NULL';
const segmentStartExpr = hasColumn(db, 'imm_session_events', 'segment_start_ms')
? 'segment_start_ms'
: 'NULL';
const segmentEndExpr = hasColumn(db, 'imm_session_events', 'segment_end_ms')
? 'segment_end_ms'
: 'NULL';
const tokensDeltaExpr = hasColumn(db, 'imm_session_events', 'tokens_delta')
? 'tokens_delta'
: '0';
const cardsDeltaExpr = hasColumn(db, 'imm_session_events', 'cards_delta') ? 'cards_delta' : '0';
const payloadExpr = hasColumn(db, 'imm_session_events', 'payload_json') ? 'payload_json' : 'NULL';
const createdDateExpr = hasColumn(db, 'imm_session_events', 'CREATED_DATE')
? 'CAST(CREATED_DATE AS TEXT)'
: 'NULL';
const lastUpdateExpr = hasColumn(db, 'imm_session_events', 'LAST_UPDATE_DATE')
? 'CAST(LAST_UPDATE_DATE AS TEXT)'
: 'NULL';
const repairedTimestampExpr =
hasColumn(db, 'imm_session_events', 'CREATED_DATE') ||
hasColumn(db, 'imm_session_events', 'LAST_UPDATE_DATE')
? `CASE
WHEN ts_ms < 0 AND COALESCE(CREATED_DATE, LAST_UPDATE_DATE) IS NOT NULL
THEN CAST(COALESCE(CREATED_DATE, LAST_UPDATE_DATE) AS TEXT)
ELSE CAST(ts_ms AS TEXT)
END`
: 'CAST(ts_ms AS TEXT)';
db.exec('PRAGMA foreign_keys = OFF');
db.exec(`
CREATE TABLE imm_session_events_new(
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
ts_ms TEXT NOT NULL,
event_type INTEGER NOT NULL,
line_index INTEGER,
segment_start_ms INTEGER,
segment_end_ms INTEGER,
tokens_delta INTEGER NOT NULL DEFAULT 0,
cards_delta INTEGER NOT NULL DEFAULT 0,
payload_json TEXT,
CREATED_DATE TEXT,
LAST_UPDATE_DATE TEXT,
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
);
`);
db.exec(`
INSERT INTO imm_session_events_new(
event_id,
session_id,
ts_ms,
event_type,
line_index,
segment_start_ms,
segment_end_ms,
tokens_delta,
cards_delta,
payload_json,
CREATED_DATE,
LAST_UPDATE_DATE
)
SELECT
event_id,
session_id,
${repairedTimestampExpr},
event_type,
${lineIndexExpr},
${segmentStartExpr},
${segmentEndExpr},
${tokensDeltaExpr},
${cardsDeltaExpr},
${payloadExpr},
${createdDateExpr},
${lastUpdateExpr}
FROM imm_session_events
`);
db.exec('DROP TABLE imm_session_events');
db.exec('ALTER TABLE imm_session_events_new RENAME TO imm_session_events');
db.exec('PRAGMA foreign_keys = ON');
}
export function applyPragmas(db: DatabaseSync): void {
db.exec('PRAGMA journal_mode = WAL');
db.exec('PRAGMA synchronous = NORMAL');
@@ -685,7 +779,7 @@ export function ensureSchema(db: DatabaseSync): void {
CREATE TABLE IF NOT EXISTS imm_session_events(
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id INTEGER NOT NULL,
ts_ms INTEGER NOT NULL,
ts_ms TEXT NOT NULL,
event_type INTEGER NOT NULL,
line_index INTEGER,
segment_start_ms INTEGER,
@@ -1122,6 +1216,8 @@ export function ensureSchema(db: DatabaseSync): void {
addColumnIfMissing(db, 'imm_sessions', 'ended_media_ms', 'INTEGER');
}
migrateSessionEventTimestampsToText(db);
ensureLifetimeSummaryTables(db);
db.exec(`
@@ -1420,7 +1516,8 @@ export function executeQueuedWrite(write: QueuedWrite, stmts: TrackerPreparedSta
) {
throw new Error('Incomplete telemetry write');
}
const telemetrySampleMs = toDbTimestamp(write.sampleMs ?? Number(currentMs));
const telemetrySampleMs =
write.sampleMs === undefined ? currentMs : toDbTimestamp(write.sampleMs);
stmts.telemetryInsertStmt.run(
write.sessionId,
telemetrySampleMs,
@@ -1495,7 +1592,7 @@ export function executeQueuedWrite(write: QueuedWrite, stmts: TrackerPreparedSta
stmts.eventInsertStmt.run(
write.sessionId,
toDbTimestamp(write.sampleMs ?? Number(currentMs)),
write.sampleMs === undefined ? currentMs : toDbTimestamp(write.sampleMs),
write.eventType ?? 0,
write.lineIndex ?? null,
write.segmentStartMs ?? null,
+1 -1
View File
@@ -1,4 +1,4 @@
export const SCHEMA_VERSION = 16;
export const SCHEMA_VERSION = 17;
export const DEFAULT_QUEUE_CAP = 1_000;
export const DEFAULT_BATCH_SIZE = 25;
export const DEFAULT_FLUSH_INTERVAL_MS = 500;