mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-07-27 04:49:49 -07:00
feat(launcher): add subminer sync for cross-machine stats/history merge over SSH (#152)
This commit is contained in:
@@ -0,0 +1,4 @@
|
||||
type: added
|
||||
area: launcher
|
||||
|
||||
- Added `subminer sync <host>` to merge immersion stats and watch history between two machines over SSH, with `--push` and `--pull` for one-way insert-only transfers. Each side snapshots its database (`VACUUM INTO`), snapshots are exchanged with `scp`, and each machine merges the other's data as an insert-only union keyed on session UUIDs / video keys / series title keys, so re-syncing is idempotent and nothing is double-counted. Lifetime totals and daily/monthly rollups are updated incrementally (history older than the session retention window is preserved); remote-only historical rollups are copied only when they do not conflict with retained local session history. `subminer sync --snapshot <file>` and `subminer sync --merge <file>` expose the underlying steps for manual transfers. The command refuses to run while the stats daemon or a live mpv session is active (`--force` overrides), ignores stale mpv socket files, keeps the guard in place through local/remote merges, supplies standard SubMiner and Bun paths to non-interactive SSH commands, verifies the remote launcher starts, reports remote stderr on failures, and aborts on stats schema version mismatches.
|
||||
@@ -25,6 +25,7 @@ Episode completion for local `watched` state uses the shared `DEFAULT_MIN_WATCH_
|
||||
|
||||
- Leave `dbPath` empty to use the default location (`immersion.sqlite` in SubMiner's app-data directory).
|
||||
- Set an explicit path to move the database (useful for backups, cloud syncing, or external tools).
|
||||
- To share stats and watch history between two machines, use [`subminer sync <host>`](/launcher-script#sync-between-machines) instead of file-level cloud sync — it merges both databases without one side overwriting the other.
|
||||
|
||||
## Stats Dashboard
|
||||
|
||||
|
||||
@@ -78,6 +78,33 @@ The first menu lists every locally watched series, most recently watched first,
|
||||
|
||||
Series whose directories are not currently accessible (e.g. an unmounted network share) are hidden from the list. Watch history requires the immersion tracker database (`immersionTracking.dbPath`, default `<config dir>/immersion.sqlite`), which SubMiner populates during playback.
|
||||
|
||||
## Sync Between Machines
|
||||
|
||||
`subminer sync <host>` merges immersion stats and watch history between two machines over SSH, so both end up with the union of sessions, lifetime totals, vocabulary counts, daily/monthly charts, and `--history` entries. `<host>` is anything `ssh` accepts (`user@hostname` or an ssh config alias); SubMiner must be installed on both machines at the same version.
|
||||
|
||||
```bash
|
||||
subminer sync macbook # two-way sync with the host "macbook"
|
||||
subminer sync macbook --push # merge local data into macbook only
|
||||
subminer sync macbook --pull # merge macbook data into local only
|
||||
subminer sync user@192.168.1.20 # explicit user@host
|
||||
subminer sync macbook --remote-cmd ~/bin/subminer # custom remote launcher path
|
||||
```
|
||||
|
||||
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over `scp`, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time — syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
|
||||
|
||||
For a one-way transfer, `--push` snapshots the local database and merges it into the host without changing the local database. `--pull` snapshots the host and merges it into the local database without changing the host. These modes add missing data; they do not delete destination-only data or make the destination an exact mirror.
|
||||
|
||||
Close SubMiner (and stop the background stats daemon, `subminer stats -s`) on both machines before syncing; the command refuses to run while a SubMiner process may be writing the database (`--force` overrides). The mpv safety check requires a live socket connection, so a stale socket file left after mpv exits does not block sync. Both machines must be on the same SubMiner version — the sync aborts on a stats schema mismatch. Remote sync checks standard SubMiner and Bun locations (`~/.local/bin`, `~/.bun/bin`, Homebrew, `/usr/local/bin`, `/usr/bin`, and `/bin`) even when the non-interactive SSH shell omits them from `PATH`.
|
||||
|
||||
Two lower-level modes are used internally over SSH and also work standalone for manual transfers (e.g. via a USB drive):
|
||||
|
||||
```bash
|
||||
subminer sync --snapshot /tmp/stats.sqlite # write a consistent snapshot of the local database
|
||||
subminer sync --merge /tmp/stats.sqlite # merge a snapshot file into the local database
|
||||
```
|
||||
|
||||
Unfinished sessions (a crash mid-playback) are skipped until the app finalizes them; they sync on the next run. Word/kanji "known" state from Anki is not part of the database and does not sync — each machine derives it from its own Anki collection.
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
@@ -106,6 +133,9 @@ subminer stats -b # start background stats daemon
|
||||
| `subminer mpv status` | Check mpv socket readiness |
|
||||
| `subminer mpv socket` | Print active socket path |
|
||||
| `subminer mpv idle` | Launch detached idle mpv instance |
|
||||
| `subminer sync <host>` | Two-way stats/history sync with another machine over SSH |
|
||||
| `subminer sync <host> --push` | Merge local stats/history into another machine only |
|
||||
| `subminer sync <host> --pull` | Merge another machine's stats/history into the local database only |
|
||||
| `subminer dictionary <path>` | Generate character dictionary ZIP from file/dir target |
|
||||
| `subminer dictionary --candidates <path>` | List AniList candidate matches for character dictionary correction |
|
||||
| `subminer dictionary --select <id> <path>` | Pin an AniList media ID for that target series |
|
||||
|
||||
@@ -111,6 +111,11 @@ subminer config show # Print active config contents
|
||||
subminer mpv socket # Print active mpv socket path
|
||||
subminer mpv status # Exit 0 if socket is ready, else exit 1
|
||||
subminer mpv idle # Launch detached idle mpv with SubMiner defaults
|
||||
subminer sync media-box # Sync stats/watch history with an SSH host
|
||||
subminer sync media-box --push # Merge this machine's stats into the host only
|
||||
subminer sync media-box --pull # Merge the host's stats into this machine only
|
||||
subminer sync --snapshot ~/subminer-snapshot.sqlite # Write a local DB snapshot
|
||||
subminer sync --merge ~/subminer-snapshot.sqlite # Merge a snapshot into the local DB
|
||||
subminer dictionary /path/to/file-or-directory # Generate character dictionary ZIP from target (manual Yomitan import)
|
||||
subminer dictionary --candidates /path/to/file.mkv
|
||||
subminer dictionary --select 21355 /path/to/file.mkv
|
||||
@@ -194,6 +199,7 @@ This flow requires `mpv.exe` to be discoverable. Leave `mpv.executablePath` blan
|
||||
- `subminer logs -e`: export a sanitized ZIP of today's local-date logs, or the most recent logs when no current-day log exists. The exported copy masks common PII and secrets; on-disk logs are unchanged.
|
||||
- `subminer config`: config file helpers (`path`, `show`).
|
||||
- `subminer mpv`: mpv helpers (`status`, `socket`, `idle`).
|
||||
- `subminer sync <host>`: sync immersion stats and watch history with another machine over SSH. The host is the SSH destination (`user@host` or an SSH config alias). Use `--push` to merge only this machine's data into the host, or `--pull` to merge only the host's data into this machine; both remain insert-only and do not make either database an exact mirror. Remote launcher checks include standard SubMiner and Bun paths even when SSH omits them from `PATH`. Use `--snapshot <file>` to write a consistent local stats DB snapshot, `--merge <file>` to merge a snapshot into the local stats DB, and `--force` to skip the running stats/mpv safety check. Advanced options: `--db <file>` overrides the local stats DB path, and `--remote-cmd <cmd>` overrides the `subminer` command used on the remote host.
|
||||
- `subminer dictionary <path>`: generates a Yomitan-importable character dictionary ZIP from a file/directory target.
|
||||
- Use `subminer dictionary --candidates <path>` and `subminer dictionary --select <id> <path>` to correct AniList character-dictionary matches for a whole series.
|
||||
- `subminer texthooker`: texthooker-only shortcut (same behavior as `--texthooker`). A _texthooker_ is a web page that displays the current subtitle line as selectable text, so browser-based dictionary extensions and other tools can read along with playback.
|
||||
|
||||
@@ -36,6 +36,14 @@ function createContext(): LauncherCommandContext {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'info',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import type { Args } from '../types.js';
|
||||
import { createEmptyMergeSummary } from '../sync/sync-shared.js';
|
||||
import type { LauncherCommandContext } from './context.js';
|
||||
import { ensureTrackerQuiescent, runSyncCommand, type SyncCommandDeps } from './sync-command.js';
|
||||
|
||||
function makeContext(overrides: Partial<Args>): LauncherCommandContext {
|
||||
return {
|
||||
args: {
|
||||
sync: true,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'warn',
|
||||
...overrides,
|
||||
} as Args,
|
||||
scriptPath: '/tmp/subminer',
|
||||
scriptName: 'subminer',
|
||||
mpvSocketPath: '',
|
||||
pluginRuntimeConfig: {},
|
||||
appPath: null,
|
||||
launcherJellyfinConfig: {},
|
||||
processAdapter: process,
|
||||
} as unknown as LauncherCommandContext;
|
||||
}
|
||||
|
||||
function ok(stdout = ''): { status: number; stdout: string; stderr: string } {
|
||||
return { status: 0, stdout, stderr: '' };
|
||||
}
|
||||
|
||||
test('ensureTrackerQuiescent ignores stale sockets but rejects live sockets', async () => {
|
||||
const context = makeContext({ syncDbPath: '/tmp/local.sqlite' });
|
||||
context.mpvSocketPath = '/tmp/subminer-socket';
|
||||
let socketConnectable = false;
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
realpathSync: (() => '/tracker.sqlite') as unknown as typeof fs.realpathSync,
|
||||
findLiveStatsDaemonPid: () => null,
|
||||
canConnectUnixSocket: async () => socketConnectable,
|
||||
fail: (message: string): never => {
|
||||
throw new Error(message);
|
||||
},
|
||||
};
|
||||
|
||||
await ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps);
|
||||
|
||||
socketConnectable = true;
|
||||
await assert.rejects(
|
||||
async () => ensureTrackerQuiescent(context, '/tmp/local.sqlite', deps),
|
||||
/mpv\/SubMiner session appears to be running/,
|
||||
);
|
||||
});
|
||||
|
||||
test('runSyncCommand dispatches snapshot, merge, host, and missing-target modes', async () => {
|
||||
const calls: string[] = [];
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${dbPath}->${outPath}`);
|
||||
},
|
||||
mergeSnapshotIntoDb: (dbPath: string, snapshotPath: string) => {
|
||||
calls.push(`merge:${dbPath}<-${snapshotPath}`);
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: (host: string) => {
|
||||
calls.push(`host:${host}`);
|
||||
},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
return command.startsWith('mktemp ') ? ok('/tmp/subminer-sync.remote\n') : ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
},
|
||||
fail: (message: string): never => {
|
||||
throw new Error(message);
|
||||
},
|
||||
};
|
||||
|
||||
assert.equal(
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncSnapshotPath: '/tmp/out.sqlite' }),
|
||||
deps,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.ok(calls.includes('snapshot:/tmp/local.sqlite->/tmp/out.sqlite'));
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncMergePath: '/tmp/in.sqlite' }),
|
||||
deps,
|
||||
);
|
||||
assert.ok(calls.includes('quiescent'));
|
||||
assert.ok(calls.includes('merge:/tmp/local.sqlite<-/tmp/in.sqlite'));
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }),
|
||||
deps,
|
||||
);
|
||||
assert.ok(calls.includes('host:media-box'));
|
||||
|
||||
await assert.rejects(
|
||||
() => runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite' }), deps),
|
||||
/sync requires a host, --snapshot <file>, or --merge <file>/,
|
||||
);
|
||||
});
|
||||
|
||||
test('runHostSync keeps tracker quiescent through local and remote merge and cleans up after failure', async () => {
|
||||
const calls: string[] = [];
|
||||
let localTmpDir = '';
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${outPath}`);
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
mergeSnapshotIntoDb: () => {
|
||||
calls.push('local-merge');
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
mkdtempSync: ((prefix: string) => {
|
||||
localTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), path.basename(prefix)));
|
||||
return localTmpDir;
|
||||
}) as typeof fs.mkdtempSync,
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
if (command.includes(' sync --snapshot ')) return ok();
|
||||
if (command.includes(' sync --merge ')) {
|
||||
return { status: 9, stdout: 'remote output', stderr: 'remote merge exploded' };
|
||||
}
|
||||
return ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
|
||||
},
|
||||
};
|
||||
|
||||
await assert.rejects(
|
||||
() =>
|
||||
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
|
||||
/Remote merge failed on media-box[\s\S]*remote merge exploded/,
|
||||
);
|
||||
assert.equal(calls.filter((call) => call === 'quiescent').length, 3);
|
||||
assert.ok(calls.indexOf('quiescent') < calls.findIndex((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(calls.includes('local-merge'));
|
||||
assert.ok(calls.some((call) => call.startsWith('ssh:rm -rf ')));
|
||||
assert.equal(fs.existsSync(localTmpDir), false);
|
||||
});
|
||||
|
||||
test('runHostSync includes remote snapshot stderr in failures', async () => {
|
||||
const deps: Partial<SyncCommandDeps> = {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
ensureTrackerQuiescent: async () => {},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
if (command.includes(' sync --snapshot ')) {
|
||||
return { status: 5, stdout: '', stderr: 'snapshot permission denied' };
|
||||
}
|
||||
return ok();
|
||||
},
|
||||
runScp: () => {},
|
||||
};
|
||||
|
||||
await assert.rejects(
|
||||
() =>
|
||||
runSyncCommand(makeContext({ syncDbPath: '/tmp/local.sqlite', syncHost: 'media-box' }), deps),
|
||||
/Remote snapshot failed on media-box[\s\S]*snapshot permission denied/,
|
||||
);
|
||||
});
|
||||
|
||||
function makeDirectionDeps(calls: string[]): Partial<SyncCommandDeps> {
|
||||
return {
|
||||
createDbSnapshot: (_dbPath: string, outPath: string) => {
|
||||
calls.push(`snapshot:${outPath}`);
|
||||
fs.writeFileSync(outPath, 'snapshot');
|
||||
},
|
||||
mergeSnapshotIntoDb: () => {
|
||||
calls.push('local-merge');
|
||||
return createEmptyMergeSummary();
|
||||
},
|
||||
formatMergeSummary: () => 'summary',
|
||||
ensureTrackerQuiescent: async () => {
|
||||
calls.push('quiescent');
|
||||
},
|
||||
assertSafeSshHost: () => {},
|
||||
resolveRemoteSubminerCommand: () => 'subminer',
|
||||
runSsh: (_host: string, command: string) => {
|
||||
calls.push(`ssh:${command}`);
|
||||
if (command.startsWith('mktemp ')) return ok('/tmp/subminer-sync.remote\n');
|
||||
return ok();
|
||||
},
|
||||
runScp: (from: string, to: string) => {
|
||||
calls.push(`scp:${from}->${to}`);
|
||||
if (!to.includes(':')) fs.writeFileSync(to, 'pulled');
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test('runHostSync push only snapshots locally and merges remotely', async () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({
|
||||
syncDbPath: '/tmp/local.sqlite',
|
||||
syncHost: 'media-box',
|
||||
syncDirection: 'push',
|
||||
}),
|
||||
makeDirectionDeps(calls),
|
||||
);
|
||||
|
||||
assert.ok(calls.some((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(calls.some((call) => call.includes(' sync --merge ')));
|
||||
assert.ok(calls.some((call) => call.startsWith('scp:') && call.includes('->media-box:')));
|
||||
assert.ok(!calls.some((call) => call.includes(' sync --snapshot ')));
|
||||
assert.ok(!calls.includes('local-merge'));
|
||||
});
|
||||
|
||||
test('runHostSync pull only snapshots remotely and merges locally', async () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
await runSyncCommand(
|
||||
makeContext({
|
||||
syncDbPath: '/tmp/local.sqlite',
|
||||
syncHost: 'media-box',
|
||||
syncDirection: 'pull',
|
||||
}),
|
||||
makeDirectionDeps(calls),
|
||||
);
|
||||
|
||||
assert.ok(calls.some((call) => call.includes(' sync --snapshot ')));
|
||||
assert.ok(calls.some((call) => call.startsWith('scp:media-box:')));
|
||||
assert.ok(calls.includes('local-merge'));
|
||||
assert.ok(!calls.some((call) => call.startsWith('snapshot:')));
|
||||
assert.ok(!calls.some((call) => call.includes(' sync --merge ')));
|
||||
});
|
||||
@@ -0,0 +1,253 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fail, log } from '../log.js';
|
||||
import { resolveImmersionDbPath } from '../history-db.js';
|
||||
import {
|
||||
createDbSnapshot,
|
||||
findLiveStatsDaemonPid,
|
||||
formatMergeSummary,
|
||||
mergeSnapshotIntoDb,
|
||||
} from '../sync/sync-db.js';
|
||||
import {
|
||||
assertSafeSshHost,
|
||||
resolveRemoteSubminerCommand,
|
||||
runScp,
|
||||
runSsh,
|
||||
shellQuote,
|
||||
} from '../sync/ssh.js';
|
||||
import { resolvePathMaybe } from '../util.js';
|
||||
import { canConnectUnixSocket } from '../mpv.js';
|
||||
import type { LauncherCommandContext } from './context.js';
|
||||
import type { RemoteRunResult } from '../sync/ssh.js';
|
||||
|
||||
export interface SyncCommandDeps {
|
||||
createDbSnapshot: typeof createDbSnapshot;
|
||||
mergeSnapshotIntoDb: typeof mergeSnapshotIntoDb;
|
||||
formatMergeSummary: typeof formatMergeSummary;
|
||||
findLiveStatsDaemonPid: typeof findLiveStatsDaemonPid;
|
||||
assertSafeSshHost: typeof assertSafeSshHost;
|
||||
resolveRemoteSubminerCommand: typeof resolveRemoteSubminerCommand;
|
||||
runScp: typeof runScp;
|
||||
runSsh: typeof runSsh;
|
||||
fail: typeof fail;
|
||||
log: typeof log;
|
||||
canConnectUnixSocket: typeof canConnectUnixSocket;
|
||||
realpathSync: typeof fs.realpathSync;
|
||||
mkdtempSync: typeof fs.mkdtempSync;
|
||||
rmSync: typeof fs.rmSync;
|
||||
consoleLog: typeof console.log;
|
||||
writeStdout: typeof process.stdout.write;
|
||||
ensureTrackerQuiescent: (context: LauncherCommandContext, dbPath: string) => Promise<void>;
|
||||
}
|
||||
|
||||
function resolveDbPath(context: LauncherCommandContext): string {
|
||||
const override = context.args.syncDbPath.trim();
|
||||
return override ? resolvePathMaybe(override) : resolveImmersionDbPath();
|
||||
}
|
||||
|
||||
function isTrackerDb(dbPath: string, deps: SyncCommandDeps): boolean {
|
||||
const trackerDbPath = resolveImmersionDbPath();
|
||||
try {
|
||||
return deps.realpathSync(dbPath) === deps.realpathSync(trackerDbPath);
|
||||
} catch {
|
||||
return dbPath === trackerDbPath;
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensureTrackerQuiescent(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
if (context.args.syncForce) return;
|
||||
// A running SubMiner only holds the tracker's own database; --db pointed
|
||||
// elsewhere needs no guard.
|
||||
if (!isTrackerDb(dbPath, deps)) return;
|
||||
const daemonPid = deps.findLiveStatsDaemonPid(dbPath);
|
||||
if (daemonPid !== null) {
|
||||
deps.fail(
|
||||
`The SubMiner stats server is running (pid ${daemonPid}). Stop it with "subminer stats -s" (or close SubMiner) before syncing, or pass --force.`,
|
||||
);
|
||||
}
|
||||
if (context.mpvSocketPath && (await deps.canConnectUnixSocket(context.mpvSocketPath))) {
|
||||
deps.fail(
|
||||
`An mpv/SubMiner session appears to be running (socket ${context.mpvSocketPath}). Close it before syncing, or pass --force.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const defaultSyncCommandDeps: SyncCommandDeps = {
|
||||
createDbSnapshot,
|
||||
mergeSnapshotIntoDb,
|
||||
formatMergeSummary,
|
||||
findLiveStatsDaemonPid,
|
||||
assertSafeSshHost,
|
||||
resolveRemoteSubminerCommand,
|
||||
runScp,
|
||||
runSsh,
|
||||
fail,
|
||||
log,
|
||||
canConnectUnixSocket,
|
||||
realpathSync: fs.realpathSync,
|
||||
mkdtempSync: fs.mkdtempSync,
|
||||
rmSync: fs.rmSync,
|
||||
consoleLog: console.log,
|
||||
writeStdout: process.stdout.write.bind(process.stdout),
|
||||
ensureTrackerQuiescent: async (context, dbPath) => ensureTrackerQuiescent(context, dbPath),
|
||||
};
|
||||
|
||||
function resolveSyncCommandDeps(inputDeps: Partial<SyncCommandDeps> = {}): SyncCommandDeps {
|
||||
return { ...defaultSyncCommandDeps, ...inputDeps };
|
||||
}
|
||||
|
||||
export function runSnapshotMode(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): void {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const outPath = resolvePathMaybe(context.args.syncSnapshotPath);
|
||||
deps.createDbSnapshot(dbPath, outPath);
|
||||
deps.consoleLog(outPath);
|
||||
}
|
||||
|
||||
export async function runMergeMode(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const snapshotPath = resolvePathMaybe(context.args.syncMergePath);
|
||||
const summary = deps.mergeSnapshotIntoDb(dbPath, snapshotPath);
|
||||
deps.consoleLog(deps.formatMergeSummary(summary));
|
||||
}
|
||||
|
||||
function cleanupRemote(host: string, remoteTmpDir: string, deps: SyncCommandDeps): void {
|
||||
if (!remoteTmpDir.startsWith('/tmp/')) return;
|
||||
deps.runSsh(host, `rm -rf ${shellQuote(remoteTmpDir)}`);
|
||||
}
|
||||
|
||||
function formatRemoteRunError(message: string, run: RemoteRunResult): string {
|
||||
const stderr = run.stderr.trim();
|
||||
return stderr ? `${message}\n${stderr}` : message;
|
||||
}
|
||||
|
||||
export async function runHostSync(
|
||||
context: LauncherCommandContext,
|
||||
dbPath: string,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<void> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const { args } = context;
|
||||
const host = args.syncHost;
|
||||
const direction = args.syncDirection ?? 'both';
|
||||
const shouldPull = direction !== 'push';
|
||||
const shouldPush = direction !== 'pull';
|
||||
deps.assertSafeSshHost(host);
|
||||
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
|
||||
const remoteCmd = deps.resolveRemoteSubminerCommand(host, args.syncRemoteCmd || null);
|
||||
deps.log('debug', args.logLevel, `Remote subminer command: ${remoteCmd}`);
|
||||
|
||||
const localTmpDir = deps.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-'));
|
||||
let remoteTmpDir = '';
|
||||
try {
|
||||
// Signal failures by throwing (not fail(), which exits synchronously and
|
||||
// would skip the finally cleanup, leaking temp dirs holding snapshot data).
|
||||
// main().catch() reports the message the same way fail() would.
|
||||
const mktemp = deps.runSsh(host, 'mktemp -d /tmp/subminer-sync.XXXXXX');
|
||||
remoteTmpDir = mktemp.stdout.trim();
|
||||
if (mktemp.status !== 0 || !remoteTmpDir.startsWith('/tmp/')) {
|
||||
throw new Error(`Could not create a temporary directory on ${host}.`);
|
||||
}
|
||||
|
||||
const forceFlag = args.syncForce ? ' --force' : '';
|
||||
|
||||
const localSnapshot = path.join(localTmpDir, 'local.sqlite');
|
||||
if (shouldPush) {
|
||||
deps.consoleLog(`Snapshotting local database (${dbPath})...`);
|
||||
deps.createDbSnapshot(dbPath, localSnapshot);
|
||||
}
|
||||
|
||||
const remoteSnapshot = `${remoteTmpDir}/snapshot.sqlite`;
|
||||
if (shouldPull) {
|
||||
deps.consoleLog(`Snapshotting ${host}...`);
|
||||
const snapshotRun = deps.runSsh(
|
||||
host,
|
||||
`${remoteCmd} sync --snapshot ${shellQuote(remoteSnapshot)}${forceFlag}`,
|
||||
);
|
||||
if (snapshotRun.status !== 0) {
|
||||
throw new Error(formatRemoteRunError(`Remote snapshot failed on ${host}.`, snapshotRun));
|
||||
}
|
||||
}
|
||||
|
||||
const pulledSnapshot = path.join(localTmpDir, 'remote.sqlite');
|
||||
if (shouldPull) deps.runScp(`${host}:${remoteSnapshot}`, pulledSnapshot);
|
||||
const incomingSnapshot = `${remoteTmpDir}/incoming.sqlite`;
|
||||
if (shouldPush) deps.runScp(localSnapshot, `${host}:${incomingSnapshot}`);
|
||||
|
||||
if (shouldPull) {
|
||||
deps.consoleLog(`\nMerging ${host} -> local:`);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const summary = deps.mergeSnapshotIntoDb(dbPath, pulledSnapshot);
|
||||
deps.consoleLog(deps.formatMergeSummary(summary));
|
||||
}
|
||||
|
||||
if (shouldPush) {
|
||||
deps.consoleLog(`\nMerging local -> ${host}:`);
|
||||
await deps.ensureTrackerQuiescent(context, dbPath);
|
||||
const mergeRun = deps.runSsh(
|
||||
host,
|
||||
`${remoteCmd} sync --merge ${shellQuote(incomingSnapshot)}${forceFlag}`,
|
||||
);
|
||||
deps.writeStdout(mergeRun.stdout);
|
||||
if (mergeRun.status !== 0) {
|
||||
const retryCommand =
|
||||
direction === 'push' ? `subminer sync ${host} --push` : `subminer sync ${host}`;
|
||||
const localUpdate = shouldPull ? ' The local database was updated;' : '';
|
||||
throw new Error(
|
||||
formatRemoteRunError(
|
||||
`Remote merge failed on ${host}.${localUpdate} re-run "${retryCommand}" once the remote issue is fixed.`,
|
||||
mergeRun,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
deps.consoleLog('\nSync complete.');
|
||||
} finally {
|
||||
deps.rmSync(localTmpDir, { recursive: true, force: true });
|
||||
if (remoteTmpDir) {
|
||||
try {
|
||||
cleanupRemote(host, remoteTmpDir, deps);
|
||||
} catch {
|
||||
// best effort
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function runSyncCommand(
|
||||
context: LauncherCommandContext,
|
||||
inputDeps: Partial<SyncCommandDeps> = {},
|
||||
): Promise<boolean> {
|
||||
const deps = resolveSyncCommandDeps(inputDeps);
|
||||
const { args } = context;
|
||||
if (!args.sync) return false;
|
||||
|
||||
const dbPath = resolveDbPath(context);
|
||||
if (args.syncSnapshotPath) {
|
||||
runSnapshotMode(context, dbPath, deps);
|
||||
} else if (args.syncMergePath) {
|
||||
await runMergeMode(context, dbPath, deps);
|
||||
} else if (args.syncHost) {
|
||||
await runHostSync(context, dbPath, deps);
|
||||
} else {
|
||||
deps.fail('sync requires a host, --snapshot <file>, or --merge <file>.');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -135,6 +135,15 @@ test('applyInvocationsToArgs maps config and jellyfin invocation state', () => {
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -181,6 +190,15 @@ test('applyInvocationsToArgs maps settings invocation to settings window', () =>
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -220,6 +238,15 @@ test('applyInvocationsToArgs fails when config invocation has no action', () =>
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
@@ -257,6 +284,15 @@ test('applyInvocationsToArgs maps texthooker browser-open request', () => {
|
||||
statsCleanupVocab: false,
|
||||
statsCleanupLifetime: false,
|
||||
statsLogLevel: null,
|
||||
syncTriggered: false,
|
||||
syncHost: null,
|
||||
syncSnapshotPath: null,
|
||||
syncMergePath: null,
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: null,
|
||||
syncDbPath: null,
|
||||
syncForce: false,
|
||||
syncLogLevel: null,
|
||||
doctorTriggered: false,
|
||||
doctorLogLevel: null,
|
||||
doctorRefreshKnownWords: false,
|
||||
|
||||
@@ -199,6 +199,14 @@ export function createDefaultArgs(
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: loggingConfig.level ?? 'warn',
|
||||
logRotation: loggingConfig.rotation ?? 7,
|
||||
passwordStore: '',
|
||||
@@ -264,6 +272,17 @@ export function applyInvocationsToArgs(parsed: Args, invocations: CliInvocations
|
||||
) {
|
||||
fail('Dictionary target path is required.');
|
||||
}
|
||||
if (invocations.syncTriggered) {
|
||||
parsed.sync = true;
|
||||
parsed.syncHost = invocations.syncHost ?? '';
|
||||
parsed.syncSnapshotPath = invocations.syncSnapshotPath ?? '';
|
||||
parsed.syncMergePath = invocations.syncMergePath ?? '';
|
||||
parsed.syncDirection = invocations.syncDirection;
|
||||
parsed.syncRemoteCmd = invocations.syncRemoteCmd ?? '';
|
||||
parsed.syncDbPath = invocations.syncDbPath ?? '';
|
||||
parsed.syncForce = invocations.syncForce;
|
||||
if (invocations.syncLogLevel) parsed.logLevel = parseLogLevel(invocations.syncLogLevel);
|
||||
}
|
||||
if (invocations.doctorTriggered) parsed.doctor = true;
|
||||
if (invocations.doctorRefreshKnownWords) parsed.doctorRefreshKnownWords = true;
|
||||
if (invocations.logsTriggered && !invocations.logsExport) {
|
||||
|
||||
@@ -42,3 +42,22 @@ test('parseCliPrograms captures texthooker browser-open flag', () => {
|
||||
assert.equal(result.invocations.texthookerTriggered, true);
|
||||
assert.equal(result.invocations.texthookerOpenBrowser, true);
|
||||
});
|
||||
|
||||
test('parseCliPrograms captures one-way sync directions', () => {
|
||||
const push = parseCliPrograms(['sync', 'media-box', '--push'], 'subminer');
|
||||
const pull = parseCliPrograms(['sync', 'media-box', '--pull'], 'subminer');
|
||||
|
||||
assert.equal(push.invocations.syncDirection, 'push');
|
||||
assert.equal(pull.invocations.syncDirection, 'pull');
|
||||
});
|
||||
|
||||
test('parseCliPrograms rejects conflicting or hostless one-way sync directions', () => {
|
||||
assert.throws(
|
||||
() => parseCliPrograms(['sync', 'media-box', '--push', '--pull'], 'subminer'),
|
||||
/--push and --pull cannot be combined/,
|
||||
);
|
||||
assert.throws(
|
||||
() => parseCliPrograms(['sync', '--snapshot', '/tmp/stats.sqlite', '--push'], 'subminer'),
|
||||
/--push and --pull require a host/,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -38,6 +38,15 @@ export interface CliInvocations {
|
||||
statsCleanupVocab: boolean;
|
||||
statsCleanupLifetime: boolean;
|
||||
statsLogLevel: string | null;
|
||||
syncTriggered: boolean;
|
||||
syncHost: string | null;
|
||||
syncSnapshotPath: string | null;
|
||||
syncMergePath: string | null;
|
||||
syncDirection: 'both' | 'push' | 'pull';
|
||||
syncRemoteCmd: string | null;
|
||||
syncDbPath: string | null;
|
||||
syncForce: boolean;
|
||||
syncLogLevel: string | null;
|
||||
doctorTriggered: boolean;
|
||||
doctorLogLevel: string | null;
|
||||
doctorRefreshKnownWords: boolean;
|
||||
@@ -98,6 +107,7 @@ function getTopLevelCommand(argv: string[]): { name: string; index: number } | n
|
||||
'dictionary',
|
||||
'dict',
|
||||
'stats',
|
||||
'sync',
|
||||
'texthooker',
|
||||
'app',
|
||||
'bin',
|
||||
@@ -160,6 +170,15 @@ export function parseCliPrograms(
|
||||
let statsCleanupVocab = false;
|
||||
let statsCleanupLifetime = false;
|
||||
let statsLogLevel: string | null = null;
|
||||
let syncTriggered = false;
|
||||
let syncHost: string | null = null;
|
||||
let syncSnapshotPath: string | null = null;
|
||||
let syncMergePath: string | null = null;
|
||||
let syncDirection: 'both' | 'push' | 'pull' = 'both';
|
||||
let syncRemoteCmd: string | null = null;
|
||||
let syncDbPath: string | null = null;
|
||||
let syncForce = false;
|
||||
let syncLogLevel: string | null = null;
|
||||
let doctorLogLevel: string | null = null;
|
||||
let doctorRefreshKnownWords = false;
|
||||
let logsTriggered = false;
|
||||
@@ -289,6 +308,49 @@ export function parseCliPrograms(
|
||||
statsLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
|
||||
});
|
||||
|
||||
commandProgram
|
||||
.command('sync')
|
||||
.description('Sync stats and watch history with another machine over SSH')
|
||||
.argument('[host]', 'SSH destination (user@host or an ssh config alias)')
|
||||
.option('--snapshot <file>', 'Write a consistent snapshot of the local stats database')
|
||||
.option('--merge <file>', 'Merge a snapshot database file into the local stats database')
|
||||
.option('--push', 'Only merge local stats into the SSH host')
|
||||
.option('--pull', 'Only merge stats from the SSH host into the local database')
|
||||
.option('--db <file>', 'Override the local stats database path')
|
||||
.option('--remote-cmd <cmd>', 'subminer command to run on the remote host')
|
||||
.option('-f, --force', 'Skip the running-app safety check')
|
||||
.option('--log-level <level>', 'Log level')
|
||||
.action((rawHost: string | undefined, options: Record<string, unknown>) => {
|
||||
const host = typeof rawHost === 'string' ? rawHost.trim() : '';
|
||||
const snapshot = typeof options.snapshot === 'string' ? options.snapshot.trim() : '';
|
||||
const merge = typeof options.merge === 'string' ? options.merge.trim() : '';
|
||||
const push = options.push === true;
|
||||
const pull = options.pull === true;
|
||||
if (push && pull) {
|
||||
throw new Error('Sync --push and --pull cannot be combined.');
|
||||
}
|
||||
if ((push || pull) && !host) {
|
||||
throw new Error('Sync --push and --pull require a host.');
|
||||
}
|
||||
const modes = [Boolean(host), Boolean(snapshot), Boolean(merge)].filter(Boolean).length;
|
||||
if (modes === 0) {
|
||||
throw new Error('Sync requires a host, --snapshot <file>, or --merge <file>.');
|
||||
}
|
||||
if (modes > 1) {
|
||||
throw new Error('Sync host, --snapshot, and --merge cannot be combined.');
|
||||
}
|
||||
syncTriggered = true;
|
||||
syncHost = host || null;
|
||||
syncSnapshotPath = snapshot || null;
|
||||
syncMergePath = merge || null;
|
||||
syncDirection = push ? 'push' : pull ? 'pull' : 'both';
|
||||
syncRemoteCmd =
|
||||
typeof options.remoteCmd === 'string' ? options.remoteCmd.trim() || null : null;
|
||||
syncDbPath = typeof options.db === 'string' ? options.db.trim() || null : null;
|
||||
syncForce = options.force === true;
|
||||
syncLogLevel = typeof options.logLevel === 'string' ? options.logLevel : null;
|
||||
});
|
||||
|
||||
commandProgram
|
||||
.command('doctor')
|
||||
.description('Run dependency and environment checks')
|
||||
@@ -400,6 +462,15 @@ export function parseCliPrograms(
|
||||
statsCleanupVocab,
|
||||
statsCleanupLifetime,
|
||||
statsLogLevel,
|
||||
syncTriggered,
|
||||
syncHost,
|
||||
syncSnapshotPath,
|
||||
syncMergePath,
|
||||
syncDirection,
|
||||
syncRemoteCmd,
|
||||
syncDbPath,
|
||||
syncForce,
|
||||
syncLogLevel,
|
||||
doctorTriggered,
|
||||
doctorLogLevel,
|
||||
doctorRefreshKnownWords,
|
||||
|
||||
@@ -30,6 +30,14 @@ function createArgs(): Args {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'info',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
@@ -22,6 +22,7 @@ import { runLogsCommand } from './commands/logs-command.js';
|
||||
import { runStatsCommand } from './commands/stats-command.js';
|
||||
import { runJellyfinCommand } from './commands/jellyfin-command.js';
|
||||
import { runHistoryCommand } from './commands/history-command.js';
|
||||
import { runSyncCommand } from './commands/sync-command.js';
|
||||
import { runPlaybackCommand } from './commands/playback-command.js';
|
||||
import { runUpdateCommand } from './commands/update-command.js';
|
||||
|
||||
@@ -107,6 +108,10 @@ async function main(): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
if (await runSyncCommand(context)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const resolvedAppPath = ensureAppPath(context);
|
||||
state.appPath = resolvedAppPath;
|
||||
log('debug', args.logLevel, `Using SubMiner app binary: ${resolvedAppPath}`);
|
||||
|
||||
@@ -571,6 +571,14 @@ function makeArgs(overrides: Partial<Args> = {}): Args {
|
||||
texthookerOpenBrowser: false,
|
||||
useRofi: false,
|
||||
history: false,
|
||||
sync: false,
|
||||
syncHost: '',
|
||||
syncSnapshotPath: '',
|
||||
syncMergePath: '',
|
||||
syncDirection: 'both',
|
||||
syncRemoteCmd: '',
|
||||
syncDbPath: '',
|
||||
syncForce: false,
|
||||
logLevel: 'error',
|
||||
logRotation: 7,
|
||||
passwordStore: '',
|
||||
|
||||
+1
-1
@@ -1683,7 +1683,7 @@ async function sleepMs(ms: number): Promise<void> {
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
|
||||
export async function canConnectUnixSocket(socketPath: string): Promise<boolean> {
|
||||
return await new Promise<boolean>((resolve) => {
|
||||
const socket = net.createConnection(socketPath);
|
||||
let settled = false;
|
||||
|
||||
@@ -0,0 +1,423 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { insertRow, tableExists, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
const ANIME_COPY_COLUMNS = [
|
||||
'normalized_title_key',
|
||||
'canonical_title',
|
||||
'anilist_id',
|
||||
'title_romaji',
|
||||
'title_english',
|
||||
'title_native',
|
||||
'episodes_total',
|
||||
'description',
|
||||
'metadata_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const VIDEO_COPY_COLUMNS = [
|
||||
'video_key',
|
||||
'canonical_title',
|
||||
'source_type',
|
||||
'source_path',
|
||||
'source_url',
|
||||
'parsed_basename',
|
||||
'parsed_title',
|
||||
'parsed_season',
|
||||
'parsed_episode',
|
||||
'parser_source',
|
||||
'parser_confidence',
|
||||
'parse_metadata_json',
|
||||
'watched',
|
||||
'duration_ms',
|
||||
'file_size_bytes',
|
||||
'codec_id',
|
||||
'container_id',
|
||||
'width_px',
|
||||
'height_px',
|
||||
'fps_x100',
|
||||
'bitrate_kbps',
|
||||
'audio_codec_id',
|
||||
'hash_sha256',
|
||||
'screenshot_path',
|
||||
'metadata_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const MEDIA_ART_COPY_COLUMNS = [
|
||||
'anilist_id',
|
||||
'cover_url',
|
||||
'cover_blob',
|
||||
'cover_blob_hash',
|
||||
'title_romaji',
|
||||
'title_english',
|
||||
'episodes_total',
|
||||
'fetched_at_ms',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const YOUTUBE_COPY_COLUMNS = [
|
||||
'youtube_video_id',
|
||||
'video_url',
|
||||
'video_title',
|
||||
'video_thumbnail_url',
|
||||
'channel_id',
|
||||
'channel_name',
|
||||
'channel_url',
|
||||
'channel_thumbnail_url',
|
||||
'uploader_id',
|
||||
'uploader_url',
|
||||
'description',
|
||||
'metadata_json',
|
||||
'fetched_at_ms',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const WORD_COPY_COLUMNS = [
|
||||
'headword',
|
||||
'word',
|
||||
'reading',
|
||||
'part_of_speech',
|
||||
'pos1',
|
||||
'pos2',
|
||||
'pos3',
|
||||
'first_seen',
|
||||
'last_seen',
|
||||
'frequency',
|
||||
'frequency_rank',
|
||||
] as const;
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
function selectAll(db: Database, sql: string, params: unknown[] = []): SqlRow[] {
|
||||
return db.query<SqlRow>(sql).all(...params);
|
||||
}
|
||||
|
||||
export function mergeAnime(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
summary: SyncMergeSummary,
|
||||
): Map<number, number> {
|
||||
const map = new Map<number, number>();
|
||||
const byAnilist = local.prepare<SqlRow>('SELECT anime_id FROM imm_anime WHERE anilist_id = ?');
|
||||
const byTitleKey = local.prepare<SqlRow>(
|
||||
'SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?',
|
||||
);
|
||||
const fillMissing = local.prepare(
|
||||
`UPDATE imm_anime
|
||||
SET
|
||||
title_romaji = COALESCE(title_romaji, ?),
|
||||
title_english = COALESCE(title_english, ?),
|
||||
title_native = COALESCE(title_native, ?),
|
||||
episodes_total = COALESCE(episodes_total, ?),
|
||||
description = COALESCE(description, ?)
|
||||
WHERE anime_id = ?`,
|
||||
);
|
||||
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
`SELECT anime_id, ${ANIME_COPY_COLUMNS.join(', ')} FROM imm_anime`,
|
||||
)) {
|
||||
const remoteId = Number(row.anime_id);
|
||||
const existing =
|
||||
(row.anilist_id !== null ? byAnilist.get(row.anilist_id) : undefined) ??
|
||||
byTitleKey.get(row.normalized_title_key);
|
||||
if (existing) {
|
||||
const localId = Number(existing.anime_id);
|
||||
map.set(remoteId, localId);
|
||||
fillMissing.run(
|
||||
row.title_romaji,
|
||||
row.title_english,
|
||||
row.title_native,
|
||||
row.episodes_total,
|
||||
row.description,
|
||||
localId,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
// No local row matched by anilist_id (checked first in `existing` above)
|
||||
// or title key, so the remote anilist_id — if any — is free to insert as-is.
|
||||
const values = ANIME_COPY_COLUMNS.map((column) => row[column]);
|
||||
map.set(remoteId, insertRow(local, 'imm_anime', ANIME_COPY_COLUMNS, values));
|
||||
summary.animeAdded += 1;
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
export interface VideoMergeResult {
|
||||
videoIdMap: Map<number, number>;
|
||||
addedVideoIds: Set<number>;
|
||||
}
|
||||
|
||||
export function mergeVideos(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
animeIdMap: Map<number, number>,
|
||||
summary: SyncMergeSummary,
|
||||
): VideoMergeResult {
|
||||
const videoIdMap = new Map<number, number>();
|
||||
const addedVideoIds = new Set<number>();
|
||||
const byKey = local.prepare<SqlRow>(
|
||||
'SELECT video_id, watched FROM imm_videos WHERE video_key = ?',
|
||||
);
|
||||
const setWatched = local.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?');
|
||||
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
`SELECT video_id, anime_id, ${VIDEO_COPY_COLUMNS.join(', ')} FROM imm_videos`,
|
||||
)) {
|
||||
const remoteId = Number(row.video_id);
|
||||
const mappedAnimeId =
|
||||
row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
|
||||
const existing = byKey.get(row.video_key);
|
||||
if (existing) {
|
||||
const localId = Number(existing.video_id);
|
||||
videoIdMap.set(remoteId, localId);
|
||||
if (Number(row.watched) > 0 && Number(existing.watched) <= 0) {
|
||||
setWatched.run(localId);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const columns = ['anime_id', ...VIDEO_COPY_COLUMNS];
|
||||
const values = [mappedAnimeId, ...VIDEO_COPY_COLUMNS.map((column) => row[column])];
|
||||
const localId = insertRow(local, 'imm_videos', columns, values);
|
||||
videoIdMap.set(remoteId, localId);
|
||||
addedVideoIds.add(remoteId);
|
||||
summary.videosAdded += 1;
|
||||
}
|
||||
return { videoIdMap, addedVideoIds };
|
||||
}
|
||||
|
||||
export function mergeMediaMetadata(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
addedVideoIds: Set<number>,
|
||||
): void {
|
||||
if (videoIdMap.size === 0) return;
|
||||
const metadataVideoIds = new Set<number>([...addedVideoIds, ...videoIdMap.keys()]);
|
||||
|
||||
const hasBlobStore =
|
||||
tableExists(local, 'imm_cover_art_blobs') && tableExists(remote, 'imm_cover_art_blobs');
|
||||
const copyBlob = hasBlobStore
|
||||
? local.prepare(
|
||||
`INSERT INTO imm_cover_art_blobs (blob_hash, cover_blob, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(blob_hash) DO NOTHING`,
|
||||
)
|
||||
: null;
|
||||
const readBlob = hasBlobStore
|
||||
? remote.prepare<SqlRow>('SELECT * FROM imm_cover_art_blobs WHERE blob_hash = ?')
|
||||
: null;
|
||||
|
||||
if (tableExists(remote, 'imm_media_art') && tableExists(local, 'imm_media_art')) {
|
||||
const localArtExists = local.prepare<SqlRow>(
|
||||
'SELECT 1 FROM imm_media_art WHERE video_id = ? LIMIT 1',
|
||||
);
|
||||
for (const remoteVideoId of metadataVideoIds) {
|
||||
const localVideoId = videoIdMap.get(remoteVideoId)!;
|
||||
if (localArtExists.get(localVideoId)) continue;
|
||||
const row = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${MEDIA_ART_COPY_COLUMNS.join(', ')} FROM imm_media_art WHERE video_id = ?`,
|
||||
)
|
||||
.get(remoteVideoId);
|
||||
if (!row) continue;
|
||||
if (row.cover_blob_hash && copyBlob && readBlob) {
|
||||
const blob = readBlob.get(row.cover_blob_hash);
|
||||
if (blob) {
|
||||
copyBlob.run(blob.blob_hash, blob.cover_blob, blob.CREATED_DATE, blob.LAST_UPDATE_DATE);
|
||||
}
|
||||
}
|
||||
insertRow(
|
||||
local,
|
||||
'imm_media_art',
|
||||
['video_id', ...MEDIA_ART_COPY_COLUMNS],
|
||||
[localVideoId, ...MEDIA_ART_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (tableExists(remote, 'imm_youtube_videos') && tableExists(local, 'imm_youtube_videos')) {
|
||||
const localYoutubeExists = local.prepare<SqlRow>(
|
||||
'SELECT 1 FROM imm_youtube_videos WHERE video_id = ? LIMIT 1',
|
||||
);
|
||||
for (const remoteVideoId of metadataVideoIds) {
|
||||
const localVideoId = videoIdMap.get(remoteVideoId)!;
|
||||
if (localYoutubeExists.get(localVideoId)) continue;
|
||||
const row = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${YOUTUBE_COPY_COLUMNS.join(', ')} FROM imm_youtube_videos WHERE video_id = ?`,
|
||||
)
|
||||
.get(remoteVideoId);
|
||||
if (!row) continue;
|
||||
insertRow(
|
||||
local,
|
||||
'imm_youtube_videos',
|
||||
['video_id', ...YOUTUBE_COPY_COLUMNS],
|
||||
[localVideoId, ...YOUTUBE_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function mergeExcludedWords(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (
|
||||
!tableExists(remote, 'imm_stats_excluded_words') ||
|
||||
!tableExists(local, 'imm_stats_excluded_words')
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const insert = local.prepare(
|
||||
`INSERT INTO imm_stats_excluded_words (headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(headword, word, reading) DO NOTHING`,
|
||||
);
|
||||
for (const row of selectAll(
|
||||
remote,
|
||||
'SELECT headword, word, reading, CREATED_DATE, LAST_UPDATE_DATE FROM imm_stats_excluded_words',
|
||||
)) {
|
||||
const result = insert.run(
|
||||
row.headword,
|
||||
row.word,
|
||||
row.reading,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.excludedWordsAdded += result.changes;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Lazily maps remote imm_words / imm_kanji ids onto local rows by natural key
|
||||
* ((headword, word, reading) / kanji). New rows are copied with the remote's
|
||||
* accumulated frequency; rows that already exist locally get their frequency
|
||||
* incremented later with only the occurrence counts this merge adds (the
|
||||
* remote total would double-count lines merged in earlier syncs).
|
||||
*/
|
||||
export class LexiconResolver {
|
||||
private readonly wordMap = new Map<number, { localId: number; isNew: boolean }>();
|
||||
private readonly kanjiMap = new Map<number, { localId: number; isNew: boolean }>();
|
||||
readonly wordFrequencyDeltas = new Map<number, number>();
|
||||
readonly kanjiFrequencyDeltas = new Map<number, number>();
|
||||
|
||||
constructor(
|
||||
private readonly local: Database,
|
||||
private readonly remote: Database,
|
||||
private readonly summary: SyncMergeSummary,
|
||||
) {}
|
||||
|
||||
resolveWord(remoteWordId: number): number {
|
||||
const cached = this.wordMap.get(remoteWordId);
|
||||
if (cached) return cached.localId;
|
||||
|
||||
const row = this.remote
|
||||
.query<SqlRow>(`SELECT ${WORD_COPY_COLUMNS.join(', ')} FROM imm_words WHERE id = ?`)
|
||||
.get(remoteWordId);
|
||||
if (!row) throw new Error(`Snapshot references missing imm_words row ${remoteWordId}`);
|
||||
|
||||
const existing = this.local
|
||||
.query<SqlRow>('SELECT id FROM imm_words WHERE headword IS ? AND word IS ? AND reading IS ?')
|
||||
.get(row.headword, row.word, row.reading);
|
||||
let entry: { localId: number; isNew: boolean };
|
||||
if (existing) {
|
||||
entry = { localId: Number(existing.id), isNew: false };
|
||||
this.local
|
||||
.prepare(
|
||||
`UPDATE imm_words
|
||||
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
|
||||
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
|
||||
WHERE id = ?`,
|
||||
)
|
||||
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
|
||||
} else {
|
||||
const localId = insertRow(
|
||||
this.local,
|
||||
'imm_words',
|
||||
WORD_COPY_COLUMNS,
|
||||
WORD_COPY_COLUMNS.map((column) => row[column]),
|
||||
);
|
||||
entry = { localId, isNew: true };
|
||||
this.summary.wordsAdded += 1;
|
||||
}
|
||||
this.wordMap.set(remoteWordId, entry);
|
||||
return entry.localId;
|
||||
}
|
||||
|
||||
resolveKanji(remoteKanjiId: number): number {
|
||||
const cached = this.kanjiMap.get(remoteKanjiId);
|
||||
if (cached) return cached.localId;
|
||||
|
||||
const row = this.remote
|
||||
.query<SqlRow>('SELECT kanji, first_seen, last_seen, frequency FROM imm_kanji WHERE id = ?')
|
||||
.get(remoteKanjiId);
|
||||
if (!row) throw new Error(`Snapshot references missing imm_kanji row ${remoteKanjiId}`);
|
||||
|
||||
const existing = this.local
|
||||
.query<SqlRow>('SELECT id FROM imm_kanji WHERE kanji IS ?')
|
||||
.get(row.kanji);
|
||||
let entry: { localId: number; isNew: boolean };
|
||||
if (existing) {
|
||||
entry = { localId: Number(existing.id), isNew: false };
|
||||
this.local
|
||||
.prepare(
|
||||
`UPDATE imm_kanji
|
||||
SET first_seen = MIN(COALESCE(first_seen, ?), COALESCE(?, first_seen)),
|
||||
last_seen = MAX(COALESCE(last_seen, ?), COALESCE(?, last_seen))
|
||||
WHERE id = ?`,
|
||||
)
|
||||
.run(row.first_seen, row.first_seen, row.last_seen, row.last_seen, entry.localId);
|
||||
} else {
|
||||
const localId = insertRow(
|
||||
this.local,
|
||||
'imm_kanji',
|
||||
['kanji', 'first_seen', 'last_seen', 'frequency'],
|
||||
[row.kanji, row.first_seen, row.last_seen, row.frequency],
|
||||
);
|
||||
entry = { localId, isNew: true };
|
||||
this.summary.kanjiAdded += 1;
|
||||
}
|
||||
this.kanjiMap.set(remoteKanjiId, entry);
|
||||
return entry.localId;
|
||||
}
|
||||
|
||||
addWordOccurrences(remoteWordId: number, count: number): void {
|
||||
const entry = this.wordMap.get(remoteWordId);
|
||||
if (!entry || entry.isNew) return;
|
||||
this.wordFrequencyDeltas.set(
|
||||
entry.localId,
|
||||
(this.wordFrequencyDeltas.get(entry.localId) ?? 0) + count,
|
||||
);
|
||||
}
|
||||
|
||||
addKanjiOccurrences(remoteKanjiId: number, count: number): void {
|
||||
const entry = this.kanjiMap.get(remoteKanjiId);
|
||||
if (!entry || entry.isNew) return;
|
||||
this.kanjiFrequencyDeltas.set(
|
||||
entry.localId,
|
||||
(this.kanjiFrequencyDeltas.get(entry.localId) ?? 0) + count,
|
||||
);
|
||||
}
|
||||
|
||||
applyFrequencyDeltas(): void {
|
||||
const updateWord = this.local.prepare(
|
||||
'UPDATE imm_words SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
|
||||
);
|
||||
for (const [localId, delta] of this.wordFrequencyDeltas) {
|
||||
updateWord.run(delta, localId);
|
||||
}
|
||||
const updateKanji = this.local.prepare(
|
||||
'UPDATE imm_kanji SET frequency = COALESCE(frequency, 0) + ? WHERE id = ?',
|
||||
);
|
||||
for (const [localId, delta] of this.kanjiFrequencyDeltas) {
|
||||
updateKanji.run(delta, localId);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { nowDbTimestamp, tableExists, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
const LOCAL_DAY_EXPR = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
|
||||
const LOCAL_MONTH_EXPR = `CAST(strftime('%Y%m', CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') AS INTEGER)`;
|
||||
|
||||
// Ported from upsertDailyRollupsForGroups / upsertMonthlyRollupsForGroups in
|
||||
// src/core/services/immersion-tracker/maintenance.ts — must stay in sync.
|
||||
const DAILY_ROLLUP_UPSERT = `
|
||||
WITH matching_sessions AS (
|
||||
SELECT * FROM imm_sessions
|
||||
WHERE ${LOCAL_DAY_EXPR} = ? AND video_id = ?
|
||||
),
|
||||
session_metrics AS (
|
||||
SELECT
|
||||
t.session_id,
|
||||
MAX(t.active_watched_ms) AS max_active_ms,
|
||||
MAX(t.lines_seen) AS max_lines,
|
||||
MAX(t.tokens_seen) AS max_tokens,
|
||||
MAX(t.cards_mined) AS max_cards,
|
||||
MAX(t.lookup_count) AS max_lookups,
|
||||
MAX(t.lookup_hits) AS max_hits
|
||||
FROM imm_session_telemetry t
|
||||
JOIN matching_sessions s ON s.session_id = t.session_id
|
||||
GROUP BY t.session_id
|
||||
)
|
||||
INSERT INTO imm_daily_rollups (
|
||||
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
SELECT
|
||||
${LOCAL_DAY_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_day,
|
||||
s.video_id AS video_id,
|
||||
COUNT(DISTINCT s.session_id) AS total_sessions,
|
||||
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
|
||||
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
|
||||
THEN (COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) * 60.0)
|
||||
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
|
||||
ELSE NULL
|
||||
END AS cards_per_hour,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) > 0
|
||||
THEN COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0)
|
||||
/ (COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0)
|
||||
ELSE NULL
|
||||
END AS tokens_per_min,
|
||||
CASE
|
||||
WHEN COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) > 0
|
||||
THEN CAST(COALESCE(SUM(COALESCE(sm.max_hits, s.lookup_hits)), 0) AS REAL)
|
||||
/ CAST(COALESCE(SUM(COALESCE(sm.max_lookups, s.lookup_count)), 0) AS REAL)
|
||||
ELSE NULL
|
||||
END AS lookup_hit_rate,
|
||||
? AS CREATED_DATE,
|
||||
? AS LAST_UPDATE_DATE
|
||||
FROM matching_sessions s
|
||||
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
|
||||
GROUP BY rollup_day, s.video_id
|
||||
ON CONFLICT (rollup_day, video_id) DO UPDATE SET
|
||||
total_sessions = excluded.total_sessions,
|
||||
total_active_min = excluded.total_active_min,
|
||||
total_lines_seen = excluded.total_lines_seen,
|
||||
total_tokens_seen = excluded.total_tokens_seen,
|
||||
total_cards = excluded.total_cards,
|
||||
cards_per_hour = excluded.cards_per_hour,
|
||||
tokens_per_min = excluded.tokens_per_min,
|
||||
lookup_hit_rate = excluded.lookup_hit_rate,
|
||||
CREATED_DATE = COALESCE(imm_daily_rollups.CREATED_DATE, excluded.CREATED_DATE),
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
|
||||
`;
|
||||
|
||||
const MONTHLY_ROLLUP_UPSERT = `
|
||||
WITH matching_sessions AS (
|
||||
SELECT * FROM imm_sessions
|
||||
WHERE ${LOCAL_MONTH_EXPR} = ? AND video_id = ?
|
||||
),
|
||||
session_metrics AS (
|
||||
SELECT
|
||||
t.session_id,
|
||||
MAX(t.active_watched_ms) AS max_active_ms,
|
||||
MAX(t.lines_seen) AS max_lines,
|
||||
MAX(t.tokens_seen) AS max_tokens,
|
||||
MAX(t.cards_mined) AS max_cards
|
||||
FROM imm_session_telemetry t
|
||||
JOIN matching_sessions s ON s.session_id = t.session_id
|
||||
GROUP BY t.session_id
|
||||
)
|
||||
INSERT INTO imm_monthly_rollups (
|
||||
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
SELECT
|
||||
${LOCAL_MONTH_EXPR.replace('started_at_ms', 's.started_at_ms')} AS rollup_month,
|
||||
s.video_id AS video_id,
|
||||
COUNT(DISTINCT s.session_id) AS total_sessions,
|
||||
COALESCE(SUM(COALESCE(sm.max_active_ms, s.active_watched_ms)), 0) / 60000.0 AS total_active_min,
|
||||
COALESCE(SUM(COALESCE(sm.max_lines, s.lines_seen)), 0) AS total_lines_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_tokens, s.tokens_seen)), 0) AS total_tokens_seen,
|
||||
COALESCE(SUM(COALESCE(sm.max_cards, s.cards_mined)), 0) AS total_cards,
|
||||
? AS CREATED_DATE,
|
||||
? AS LAST_UPDATE_DATE
|
||||
FROM matching_sessions s
|
||||
LEFT JOIN session_metrics sm ON s.session_id = sm.session_id
|
||||
GROUP BY rollup_month, s.video_id
|
||||
ON CONFLICT (rollup_month, video_id) DO UPDATE SET
|
||||
total_sessions = excluded.total_sessions,
|
||||
total_active_min = excluded.total_active_min,
|
||||
total_lines_seen = excluded.total_lines_seen,
|
||||
total_tokens_seen = excluded.total_tokens_seen,
|
||||
total_cards = excluded.total_cards,
|
||||
CREATED_DATE = COALESCE(imm_monthly_rollups.CREATED_DATE, excluded.CREATED_DATE),
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE
|
||||
`;
|
||||
|
||||
/**
|
||||
* Recompute daily/monthly rollup groups touched by the newly merged sessions
|
||||
* from the (now merged) local session + telemetry data. The maintenance
|
||||
* watermark is left alone: telemetry newer than it gets recomputed again by
|
||||
* the app later, which is idempotent.
|
||||
*/
|
||||
export function refreshRollupsForNewSessions(
|
||||
local: Database,
|
||||
newSessionIds: number[],
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (newSessionIds.length === 0) return;
|
||||
|
||||
const groups = new Map<string, { day: number; month: number; videoId: number }>();
|
||||
for (let offset = 0; offset < newSessionIds.length; offset += 500) {
|
||||
const chunk = newSessionIds.slice(offset, offset + 500);
|
||||
const rows = local
|
||||
.query<SqlRow>(
|
||||
`SELECT DISTINCT ${LOCAL_DAY_EXPR} AS rollup_day, ${LOCAL_MONTH_EXPR} AS rollup_month, video_id
|
||||
FROM imm_sessions WHERE session_id IN (${chunk.map(() => '?').join(',')})`,
|
||||
)
|
||||
.all(...chunk);
|
||||
for (const row of rows) {
|
||||
const day = Number(row.rollup_day);
|
||||
const month = Number(row.rollup_month);
|
||||
const videoId = Number(row.video_id);
|
||||
groups.set(`${day}-${videoId}`, { day, month, videoId });
|
||||
}
|
||||
}
|
||||
|
||||
const stampMs = nowDbTimestamp();
|
||||
const deleteDaily = local.prepare(
|
||||
'DELETE FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ?',
|
||||
);
|
||||
const deleteMonthly = local.prepare(
|
||||
'DELETE FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ?',
|
||||
);
|
||||
const upsertDaily = local.prepare(DAILY_ROLLUP_UPSERT);
|
||||
const upsertMonthly = local.prepare(MONTHLY_ROLLUP_UPSERT);
|
||||
|
||||
const monthlyGroups = new Set<string>();
|
||||
for (const { day, month, videoId } of groups.values()) {
|
||||
deleteDaily.run(day, videoId);
|
||||
upsertDaily.run(day, videoId, stampMs, stampMs);
|
||||
summary.rollupGroupsRecomputed += 1;
|
||||
const monthKey = `${month}-${videoId}`;
|
||||
if (!monthlyGroups.has(monthKey)) {
|
||||
monthlyGroups.add(monthKey);
|
||||
deleteMonthly.run(month, videoId);
|
||||
upsertMonthly.run(month, videoId, stampMs, stampMs);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sessions are pruned after a retention window, but rollups are kept much
|
||||
* longer — the remote's older rollup history can't be reconstructed from
|
||||
* merged sessions. Copy remote rollup rows for groups where the local DB has
|
||||
* neither a rollup row nor any sessions (i.e. history only the remote knows).
|
||||
* Groups both machines have data for are never summed, to avoid
|
||||
* double-counting sessions that earlier syncs already shared.
|
||||
*/
|
||||
export function copyRemoteOnlyRollups(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
if (!tableExists(remote, 'imm_daily_rollups') || !tableExists(local, 'imm_daily_rollups')) return;
|
||||
|
||||
const localDailyExists = local.prepare(
|
||||
'SELECT 1 FROM imm_daily_rollups WHERE rollup_day = ? AND video_id = ? LIMIT 1',
|
||||
);
|
||||
const localDaySessions = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_DAY_EXPR} = ? LIMIT 1`,
|
||||
);
|
||||
const localMonthSessions = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions WHERE video_id = ? AND ${LOCAL_MONTH_EXPR} = ? LIMIT 1`,
|
||||
);
|
||||
const localMonthSessionsForDay = local.prepare(
|
||||
`SELECT 1 FROM imm_sessions
|
||||
WHERE video_id = ?
|
||||
AND ${LOCAL_MONTH_EXPR} = CAST(strftime('%Y%m', CAST(? AS INTEGER) * 86400, 'unixepoch', 'localtime') AS INTEGER)
|
||||
LIMIT 1`,
|
||||
);
|
||||
const insertDaily = local.prepare(
|
||||
`INSERT INTO imm_daily_rollups (
|
||||
rollup_day, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, cards_per_hour, tokens_per_min, lookup_hit_rate,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
);
|
||||
for (const row of remote.query<SqlRow>('SELECT * FROM imm_daily_rollups').all()) {
|
||||
if (row.video_id === null) continue;
|
||||
const localVideoId = videoIdMap.get(Number(row.video_id));
|
||||
if (localVideoId === undefined) continue;
|
||||
if (localDailyExists.get(row.rollup_day, localVideoId)) continue;
|
||||
if (localDaySessions.get(localVideoId, row.rollup_day)) continue;
|
||||
if (localMonthSessionsForDay.get(localVideoId, row.rollup_day)) continue;
|
||||
insertDaily.run(
|
||||
row.rollup_day,
|
||||
localVideoId,
|
||||
row.total_sessions,
|
||||
row.total_active_min,
|
||||
row.total_lines_seen,
|
||||
row.total_tokens_seen,
|
||||
row.total_cards,
|
||||
row.cards_per_hour,
|
||||
row.tokens_per_min,
|
||||
row.lookup_hit_rate,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.dailyRollupsCopied += 1;
|
||||
}
|
||||
|
||||
const localMonthlyExists = local.prepare(
|
||||
'SELECT 1 FROM imm_monthly_rollups WHERE rollup_month = ? AND video_id = ? LIMIT 1',
|
||||
);
|
||||
const insertMonthly = local.prepare(
|
||||
`INSERT INTO imm_monthly_rollups (
|
||||
rollup_month, video_id, total_sessions, total_active_min, total_lines_seen,
|
||||
total_tokens_seen, total_cards, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
);
|
||||
for (const row of remote.query<SqlRow>('SELECT * FROM imm_monthly_rollups').all()) {
|
||||
if (row.video_id === null) continue;
|
||||
const localVideoId = videoIdMap.get(Number(row.video_id));
|
||||
if (localVideoId === undefined) continue;
|
||||
if (localMonthlyExists.get(row.rollup_month, localVideoId)) continue;
|
||||
if (localMonthSessions.get(localVideoId, row.rollup_month)) continue;
|
||||
insertMonthly.run(
|
||||
row.rollup_month,
|
||||
localVideoId,
|
||||
row.total_sessions,
|
||||
row.total_active_min,
|
||||
row.total_lines_seen,
|
||||
row.total_tokens_seen,
|
||||
row.total_cards,
|
||||
row.CREATED_DATE,
|
||||
row.LAST_UPDATE_DATE,
|
||||
);
|
||||
summary.monthlyRollupsCopied += 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,465 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import type { LexiconResolver } from './merge-catalog.js';
|
||||
import { insertRow, nowDbTimestamp, type SyncMergeSummary } from './sync-shared.js';
|
||||
|
||||
const SESSION_COPY_COLUMNS = [
|
||||
'session_uuid',
|
||||
'started_at_ms',
|
||||
'ended_at_ms',
|
||||
'status',
|
||||
'locale_id',
|
||||
'target_lang_id',
|
||||
'difficulty_tier',
|
||||
'subtitle_mode',
|
||||
'ended_media_ms',
|
||||
'total_watched_ms',
|
||||
'active_watched_ms',
|
||||
'lines_seen',
|
||||
'tokens_seen',
|
||||
'cards_mined',
|
||||
'lookup_count',
|
||||
'lookup_hits',
|
||||
'yomitan_lookup_count',
|
||||
'pause_count',
|
||||
'pause_ms',
|
||||
'seek_forward_count',
|
||||
'seek_backward_count',
|
||||
'media_buffer_events',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const TELEMETRY_COPY_COLUMNS = [
|
||||
'sample_ms',
|
||||
'total_watched_ms',
|
||||
'active_watched_ms',
|
||||
'lines_seen',
|
||||
'tokens_seen',
|
||||
'cards_mined',
|
||||
'lookup_count',
|
||||
'lookup_hits',
|
||||
'yomitan_lookup_count',
|
||||
'pause_count',
|
||||
'pause_ms',
|
||||
'seek_forward_count',
|
||||
'seek_backward_count',
|
||||
'media_buffer_events',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const EVENT_COPY_COLUMNS = [
|
||||
'ts_ms',
|
||||
'event_type',
|
||||
'line_index',
|
||||
'segment_start_ms',
|
||||
'segment_end_ms',
|
||||
'tokens_delta',
|
||||
'cards_delta',
|
||||
'payload_json',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
const LINE_COPY_COLUMNS = [
|
||||
'line_index',
|
||||
'segment_start_ms',
|
||||
'segment_end_ms',
|
||||
'text',
|
||||
'secondary_text',
|
||||
'CREATED_DATE',
|
||||
'LAST_UPDATE_DATE',
|
||||
] as const;
|
||||
|
||||
type SqlRow = Record<string, unknown>;
|
||||
|
||||
export interface SessionMergeResult {
|
||||
newSessionIds: number[];
|
||||
}
|
||||
|
||||
export function mergeSessions(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
videoIdMap: Map<number, number>,
|
||||
animeIdMap: Map<number, number>,
|
||||
lexicon: LexiconResolver,
|
||||
summary: SyncMergeSummary,
|
||||
): SessionMergeResult {
|
||||
const newSessionIds: number[] = [];
|
||||
const uuidExists = local.prepare<SqlRow>(
|
||||
'SELECT session_id FROM imm_sessions WHERE session_uuid = ?',
|
||||
);
|
||||
|
||||
const remoteSessions = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT session_id, video_id, ${SESSION_COPY_COLUMNS.join(', ')}
|
||||
FROM imm_sessions
|
||||
ORDER BY CAST(started_at_ms AS REAL) ASC, session_id ASC`,
|
||||
)
|
||||
.all();
|
||||
|
||||
for (const session of remoteSessions) {
|
||||
if (session.ended_at_ms === null) {
|
||||
// Stale ACTIVE sessions are finalized by the app on its next startup;
|
||||
// they will sync once they carry final numbers.
|
||||
summary.activeSessionsSkipped += 1;
|
||||
continue;
|
||||
}
|
||||
if (uuidExists.get(session.session_uuid)) {
|
||||
summary.sessionsAlreadyPresent += 1;
|
||||
continue;
|
||||
}
|
||||
const localVideoId = videoIdMap.get(Number(session.video_id));
|
||||
if (localVideoId === undefined) {
|
||||
throw new Error(`Snapshot session ${String(session.session_uuid)} references missing video row`);
|
||||
}
|
||||
|
||||
const localSessionId = insertRow(
|
||||
local,
|
||||
'imm_sessions',
|
||||
['video_id', ...SESSION_COPY_COLUMNS],
|
||||
[localVideoId, ...SESSION_COPY_COLUMNS.map((column) => session[column])],
|
||||
);
|
||||
newSessionIds.push(localSessionId);
|
||||
summary.sessionsMerged += 1;
|
||||
|
||||
const remoteSessionId = Number(session.session_id);
|
||||
copyTelemetry(local, remote, remoteSessionId, localSessionId, summary);
|
||||
const eventIdMap = copyEvents(local, remote, remoteSessionId, localSessionId, summary);
|
||||
copySubtitleLines(
|
||||
local,
|
||||
remote,
|
||||
remoteSessionId,
|
||||
localSessionId,
|
||||
localVideoId,
|
||||
animeIdMap,
|
||||
eventIdMap,
|
||||
lexicon,
|
||||
summary,
|
||||
);
|
||||
applyMergedSessionLifetime(local, localSessionId, localVideoId, session);
|
||||
}
|
||||
|
||||
return { newSessionIds };
|
||||
}
|
||||
|
||||
function copyTelemetry(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT ${TELEMETRY_COPY_COLUMNS.join(', ')} FROM imm_session_telemetry
|
||||
WHERE session_id = ? ORDER BY telemetry_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
for (const row of rows) {
|
||||
insertRow(
|
||||
local,
|
||||
'imm_session_telemetry',
|
||||
['session_id', ...TELEMETRY_COPY_COLUMNS],
|
||||
[localSessionId, ...TELEMETRY_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
summary.telemetryRowsAdded += 1;
|
||||
}
|
||||
}
|
||||
|
||||
function copyEvents(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
summary: SyncMergeSummary,
|
||||
): Map<number, number> {
|
||||
const eventIdMap = new Map<number, number>();
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT event_id, ${EVENT_COPY_COLUMNS.join(', ')} FROM imm_session_events
|
||||
WHERE session_id = ? ORDER BY event_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
for (const row of rows) {
|
||||
const localEventId = insertRow(
|
||||
local,
|
||||
'imm_session_events',
|
||||
['session_id', ...EVENT_COPY_COLUMNS],
|
||||
[localSessionId, ...EVENT_COPY_COLUMNS.map((column) => row[column])],
|
||||
);
|
||||
eventIdMap.set(Number(row.event_id), localEventId);
|
||||
summary.eventsAdded += 1;
|
||||
}
|
||||
return eventIdMap;
|
||||
}
|
||||
|
||||
function copySubtitleLines(
|
||||
local: Database,
|
||||
remote: Database,
|
||||
remoteSessionId: number,
|
||||
localSessionId: number,
|
||||
localVideoId: number,
|
||||
animeIdMap: Map<number, number>,
|
||||
eventIdMap: Map<number, number>,
|
||||
lexicon: LexiconResolver,
|
||||
summary: SyncMergeSummary,
|
||||
): void {
|
||||
const rows = remote
|
||||
.query<SqlRow>(
|
||||
`SELECT line_id, event_id, anime_id, ${LINE_COPY_COLUMNS.join(', ')} FROM imm_subtitle_lines
|
||||
WHERE session_id = ? ORDER BY line_id ASC`,
|
||||
)
|
||||
.all(remoteSessionId);
|
||||
const wordOccurrences = remote.prepare<SqlRow>(
|
||||
'SELECT word_id, occurrence_count FROM imm_word_line_occurrences WHERE line_id = ?',
|
||||
);
|
||||
const kanjiOccurrences = remote.prepare<SqlRow>(
|
||||
'SELECT kanji_id, occurrence_count FROM imm_kanji_line_occurrences WHERE line_id = ?',
|
||||
);
|
||||
const insertWordOccurrence = local.prepare(
|
||||
`INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)
|
||||
ON CONFLICT(line_id, word_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
|
||||
);
|
||||
const insertKanjiOccurrence = local.prepare(
|
||||
`INSERT INTO imm_kanji_line_occurrences (line_id, kanji_id, occurrence_count) VALUES (?, ?, ?)
|
||||
ON CONFLICT(line_id, kanji_id) DO UPDATE SET occurrence_count = occurrence_count + excluded.occurrence_count`,
|
||||
);
|
||||
|
||||
for (const row of rows) {
|
||||
const localEventId = row.event_id === null ? null : (eventIdMap.get(Number(row.event_id)) ?? null);
|
||||
const localAnimeId = row.anime_id === null ? null : (animeIdMap.get(Number(row.anime_id)) ?? null);
|
||||
const localLineId = insertRow(
|
||||
local,
|
||||
'imm_subtitle_lines',
|
||||
['session_id', 'event_id', 'video_id', 'anime_id', ...LINE_COPY_COLUMNS],
|
||||
[
|
||||
localSessionId,
|
||||
localEventId,
|
||||
localVideoId,
|
||||
localAnimeId,
|
||||
...LINE_COPY_COLUMNS.map((column) => row[column]),
|
||||
],
|
||||
);
|
||||
summary.subtitleLinesAdded += 1;
|
||||
|
||||
for (const occurrence of wordOccurrences.all(row.line_id)) {
|
||||
const localWordId = lexicon.resolveWord(Number(occurrence.word_id));
|
||||
const count = Number(occurrence.occurrence_count);
|
||||
insertWordOccurrence.run(localLineId, localWordId, count);
|
||||
lexicon.addWordOccurrences(Number(occurrence.word_id), count);
|
||||
}
|
||||
for (const occurrence of kanjiOccurrences.all(row.line_id)) {
|
||||
const localKanjiId = lexicon.resolveKanji(Number(occurrence.kanji_id));
|
||||
const count = Number(occurrence.occurrence_count);
|
||||
insertKanjiOccurrence.run(localLineId, localKanjiId, count);
|
||||
lexicon.addKanjiOccurrences(Number(occurrence.kanji_id), count);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Port of applySessionLifetimeSummary (src/core/services/immersion-tracker/
|
||||
* lifetime.ts) for sessions arriving out of chronological order. The
|
||||
* "first session of the day / for this video" checks are order-independent
|
||||
* here (any other session counts, not just earlier ones): the local machine
|
||||
* already credited active_days/episodes_started when its own session was
|
||||
* applied, even if the merged session started earlier that day.
|
||||
*/
|
||||
function applyMergedSessionLifetime(
|
||||
local: Database,
|
||||
sessionId: number,
|
||||
videoId: number,
|
||||
session: SqlRow,
|
||||
): void {
|
||||
const updatedAtMs = nowDbTimestamp();
|
||||
const applied = local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(session_id) DO NOTHING`,
|
||||
)
|
||||
.run(sessionId, session.ended_at_ms, updatedAtMs, updatedAtMs);
|
||||
if (applied.changes <= 0) return;
|
||||
|
||||
const telemetry = local
|
||||
.query<SqlRow>(
|
||||
`SELECT active_watched_ms, cards_mined, lines_seen, tokens_seen
|
||||
FROM imm_session_telemetry
|
||||
WHERE session_id = ?
|
||||
ORDER BY sample_ms DESC, telemetry_id DESC
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get(sessionId);
|
||||
|
||||
const metric = (telemetryValue: unknown, sessionValue: unknown): number => {
|
||||
const fromTelemetry = telemetry ? Number(telemetryValue) : Number.NaN;
|
||||
const value = Number.isFinite(fromTelemetry) ? fromTelemetry : Number(sessionValue);
|
||||
return Math.max(0, Math.floor(Number.isFinite(value) ? value : 0));
|
||||
};
|
||||
const activeMs = metric(telemetry?.active_watched_ms, session.active_watched_ms);
|
||||
const cardsMined = metric(telemetry?.cards_mined, session.cards_mined);
|
||||
const linesSeen = metric(telemetry?.lines_seen, session.lines_seen);
|
||||
const tokensSeen = metric(telemetry?.tokens_seen, session.tokens_seen);
|
||||
|
||||
const video = local
|
||||
.query<SqlRow>('SELECT anime_id, watched FROM imm_videos WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const watched = Number(video?.watched ?? 0);
|
||||
const animeId = video?.anime_id === null || video?.anime_id === undefined ? null : Number(video.anime_id);
|
||||
|
||||
const mediaLifetime = local
|
||||
.query<SqlRow>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const hasOtherSessionForVideo = Boolean(
|
||||
local
|
||||
.query('SELECT 1 FROM imm_sessions WHERE video_id = ? AND session_id != ? LIMIT 1')
|
||||
.get(videoId, sessionId),
|
||||
);
|
||||
const isFirstSessionForVideoRun = !mediaLifetime && !hasOtherSessionForVideo;
|
||||
const isFirstCompletedSessionForVideoRun = watched > 0 && Number(mediaLifetime?.completed ?? 0) <= 0;
|
||||
|
||||
const hasOtherSessionOnDay = Boolean(
|
||||
local
|
||||
.query(
|
||||
`SELECT 1 FROM imm_sessions
|
||||
WHERE session_id != ?
|
||||
AND CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
|
||||
= CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get(sessionId, session.started_at_ms),
|
||||
);
|
||||
|
||||
let animeCompletedDelta = 0;
|
||||
if (animeId !== null && watched > 0 && isFirstCompletedSessionForVideoRun) {
|
||||
const animeLifetime = local
|
||||
.query<SqlRow>('SELECT episodes_completed FROM imm_lifetime_anime WHERE anime_id = ?')
|
||||
.get(animeId);
|
||||
const anime = local
|
||||
.query<SqlRow>('SELECT episodes_total FROM imm_anime WHERE anime_id = ?')
|
||||
.get(animeId);
|
||||
const episodesCompletedBefore = Number(animeLifetime?.episodes_completed ?? 0);
|
||||
const episodesTotal = anime?.episodes_total === null || anime?.episodes_total === undefined
|
||||
? null
|
||||
: Number(anime.episodes_total);
|
||||
if (
|
||||
episodesTotal !== null &&
|
||||
episodesTotal > 0 &&
|
||||
episodesCompletedBefore < episodesTotal &&
|
||||
episodesCompletedBefore + 1 >= episodesTotal
|
||||
) {
|
||||
animeCompletedDelta = 1;
|
||||
}
|
||||
}
|
||||
|
||||
local
|
||||
.prepare(
|
||||
`UPDATE imm_lifetime_global
|
||||
SET total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + ?,
|
||||
total_cards = total_cards + ?,
|
||||
active_days = active_days + ?,
|
||||
episodes_started = episodes_started + ?,
|
||||
episodes_completed = episodes_completed + ?,
|
||||
anime_completed = anime_completed + ?,
|
||||
LAST_UPDATE_DATE = ?
|
||||
WHERE global_id = 1`,
|
||||
)
|
||||
.run(
|
||||
activeMs,
|
||||
cardsMined,
|
||||
hasOtherSessionOnDay ? 0 : 1,
|
||||
isFirstSessionForVideoRun ? 1 : 0,
|
||||
isFirstCompletedSessionForVideoRun ? 1 : 0,
|
||||
animeCompletedDelta,
|
||||
updatedAtMs,
|
||||
);
|
||||
|
||||
local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_media(
|
||||
video_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
|
||||
total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(video_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
completed = MAX(completed, excluded.completed),
|
||||
first_watched_ms = CASE
|
||||
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
|
||||
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
|
||||
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
|
||||
ELSE first_watched_ms
|
||||
END,
|
||||
last_watched_ms = CASE
|
||||
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
|
||||
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
|
||||
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
|
||||
ELSE last_watched_ms
|
||||
END,
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
|
||||
)
|
||||
.run(
|
||||
videoId,
|
||||
activeMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
watched > 0 ? 1 : 0,
|
||||
session.started_at_ms,
|
||||
session.ended_at_ms,
|
||||
updatedAtMs,
|
||||
updatedAtMs,
|
||||
);
|
||||
|
||||
if (animeId !== null) {
|
||||
local
|
||||
.prepare(
|
||||
`INSERT INTO imm_lifetime_anime(
|
||||
anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen,
|
||||
total_tokens_seen, episodes_started, episodes_completed, first_watched_ms,
|
||||
last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE
|
||||
)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(anime_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
episodes_started = episodes_started + excluded.episodes_started,
|
||||
episodes_completed = episodes_completed + excluded.episodes_completed,
|
||||
first_watched_ms = CASE
|
||||
WHEN excluded.first_watched_ms IS NULL THEN first_watched_ms
|
||||
WHEN first_watched_ms IS NULL THEN excluded.first_watched_ms
|
||||
WHEN excluded.first_watched_ms < first_watched_ms THEN excluded.first_watched_ms
|
||||
ELSE first_watched_ms
|
||||
END,
|
||||
last_watched_ms = CASE
|
||||
WHEN excluded.last_watched_ms IS NULL THEN last_watched_ms
|
||||
WHEN last_watched_ms IS NULL THEN excluded.last_watched_ms
|
||||
WHEN excluded.last_watched_ms > last_watched_ms THEN excluded.last_watched_ms
|
||||
ELSE last_watched_ms
|
||||
END,
|
||||
LAST_UPDATE_DATE = excluded.LAST_UPDATE_DATE`,
|
||||
)
|
||||
.run(
|
||||
animeId,
|
||||
activeMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
isFirstSessionForVideoRun ? 1 : 0,
|
||||
isFirstCompletedSessionForVideoRun ? 1 : 0,
|
||||
session.started_at_ms,
|
||||
session.ended_at_ms,
|
||||
updatedAtMs,
|
||||
updatedAtMs,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { assertSafeSshHost, resolveRemoteSubminerCommand, runScp, shellQuote } from './ssh.js';
|
||||
|
||||
test('assertSafeSshHost rejects option-like hosts', () => {
|
||||
assert.throws(() => assertSafeSshHost('-oProxyCommand=touch pwned'), /looks like an option/);
|
||||
assert.throws(() => assertSafeSshHost('-lroot'), /looks like an option/);
|
||||
});
|
||||
|
||||
test('assertSafeSshHost accepts normal destinations', () => {
|
||||
assert.doesNotThrow(() => assertSafeSshHost('macbook'));
|
||||
assert.doesNotThrow(() => assertSafeSshHost('user@192.168.1.20'));
|
||||
assert.doesNotThrow(() => assertSafeSshHost('ssh-alias'));
|
||||
});
|
||||
|
||||
test('shellQuote escapes single quotes and wraps in quotes', () => {
|
||||
assert.equal(shellQuote('subminer'), `'subminer'`);
|
||||
assert.equal(shellQuote(`a'; rm -rf ~; '`), `'a'\\''; rm -rf ~; '\\'''`);
|
||||
});
|
||||
|
||||
test('runScp rejects option-like local endpoints before spawning scp', () => {
|
||||
assert.throws(() => runScp('-oProxyCommand=sh', '/tmp/out.sqlite'), /looks like an option/);
|
||||
assert.throws(() => runScp('/tmp/in.sqlite', '-bad-destination'), /looks like an option/);
|
||||
});
|
||||
|
||||
test('runScp rejects option-like remote host components', () => {
|
||||
assert.throws(
|
||||
() => runScp('-oProxyCommand=sh:/tmp/in.sqlite', '/tmp/out.sqlite'),
|
||||
/SSH host that looks like an option/,
|
||||
);
|
||||
});
|
||||
|
||||
test('resolveRemoteSubminerCommand verifies the launcher under the remote runtime PATH', () => {
|
||||
const calls: Array<{ host: string; remoteCommand: string }> = [];
|
||||
const command = resolveRemoteSubminerCommand('macbook', null, (host, remoteCommand) => {
|
||||
calls.push({ host, remoteCommand });
|
||||
return { status: 0, stdout: '', stderr: '' };
|
||||
});
|
||||
|
||||
assert.equal(
|
||||
command,
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer',
|
||||
);
|
||||
assert.deepEqual(calls, [
|
||||
{
|
||||
host: 'macbook',
|
||||
remoteCommand:
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" subminer --help >/dev/null 2>&1',
|
||||
},
|
||||
]);
|
||||
});
|
||||
@@ -0,0 +1,106 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
export interface RemoteRunResult {
|
||||
status: number;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* ssh/scp have no `--` terminator for the destination, so a host that starts
|
||||
* with `-` (e.g. `-oProxyCommand=...`) is parsed as an option. Reject those
|
||||
* before spawning.
|
||||
*/
|
||||
export function assertSafeSshHost(host: string): void {
|
||||
if (host.startsWith('-')) {
|
||||
throw new Error(`Refusing to use SSH host that looks like an option: ${host}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a command on the SSH host. stdin stays attached so interactive prompts
|
||||
* can still read from the terminal; stdout/stderr are captured for callers
|
||||
* that need actionable remote failure messages.
|
||||
*/
|
||||
export function runSsh(host: string, remoteCommand: string): RemoteRunResult {
|
||||
assertSafeSshHost(host);
|
||||
const result = spawnSync('ssh', [host, remoteCommand], {
|
||||
encoding: 'utf8',
|
||||
stdio: ['inherit', 'pipe', 'pipe'],
|
||||
});
|
||||
if (result.error) {
|
||||
throw new Error(`Failed to run ssh: ${(result.error as Error).message}`);
|
||||
}
|
||||
return { status: result.status ?? 1, stdout: result.stdout ?? '', stderr: result.stderr ?? '' };
|
||||
}
|
||||
|
||||
function assertSafeScpEndpoint(endpoint: string): void {
|
||||
const colon = endpoint.indexOf(':');
|
||||
const slash = endpoint.indexOf('/');
|
||||
if (colon <= 0 || (slash !== -1 && slash < colon)) {
|
||||
if (endpoint.startsWith('-')) {
|
||||
throw new Error(`Refusing to use scp endpoint that looks like an option: ${endpoint}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const host = endpoint.slice(0, colon);
|
||||
const remotePath = endpoint.slice(colon + 1);
|
||||
assertSafeSshHost(host);
|
||||
if (remotePath.startsWith('-')) {
|
||||
throw new Error(`Refusing to use scp remote path that looks like an option: ${remotePath}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function runScp(from: string, to: string): void {
|
||||
assertSafeScpEndpoint(from);
|
||||
assertSafeScpEndpoint(to);
|
||||
const result = spawnSync('scp', ['-q', from, to], {
|
||||
encoding: 'utf8',
|
||||
stdio: ['inherit', 'inherit', 'inherit'],
|
||||
});
|
||||
if (result.error) {
|
||||
throw new Error(`Failed to run scp: ${(result.error as Error).message}`);
|
||||
}
|
||||
if ((result.status ?? 1) !== 0) {
|
||||
throw new Error(`scp failed copying ${from} -> ${to}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function shellQuote(value: string): string {
|
||||
return `'${value.replaceAll("'", `'\\''`)}'`;
|
||||
}
|
||||
|
||||
const REMOTE_RUNTIME_PATH =
|
||||
'PATH="$HOME/.local/bin:$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"';
|
||||
|
||||
/**
|
||||
* Non-interactive SSH shells often miss user-installed launchers and Bun.
|
||||
* Probe the launcher under the same deterministic PATH used by sync itself.
|
||||
*/
|
||||
export function resolveRemoteSubminerCommand(
|
||||
host: string,
|
||||
preferred: string | null,
|
||||
runRemote: typeof runSsh = runSsh,
|
||||
): string {
|
||||
// Trusted defaults stay unquoted so the remote shell expands `~`; a
|
||||
// user-supplied override is shell-quoted to prevent command injection.
|
||||
const candidates: Array<{ value: string; invocation: string }> = preferred
|
||||
? [{ value: preferred, invocation: shellQuote(preferred) }]
|
||||
: [
|
||||
{ value: 'subminer', invocation: 'subminer' },
|
||||
{ value: '~/.local/bin/subminer', invocation: '~/.local/bin/subminer' },
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
const command = `${REMOTE_RUNTIME_PATH} ${candidate.invocation}`;
|
||||
const probe = runRemote(host, `${command} --help >/dev/null 2>&1`);
|
||||
if (probe.status === 0) {
|
||||
return command;
|
||||
}
|
||||
}
|
||||
throw new Error(
|
||||
preferred
|
||||
? `Remote command not found on ${host}: ${preferred}`
|
||||
: `subminer not found on ${host} (tried PATH and ~/.local/bin/subminer). Pass --remote-cmd <path>.`,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,598 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { createDbSnapshot, mergeSnapshotIntoDb } from './sync-db.js';
|
||||
import {
|
||||
createImmersionDbFixture,
|
||||
insertFixtureSession,
|
||||
} from '../test-support/immersion-db-fixture.js';
|
||||
|
||||
const DAY_MS = 86_400_000;
|
||||
const BASE_MS = Date.UTC(2026, 5, 1, 12, 0, 0);
|
||||
|
||||
function makeTmpDir(): string {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-sync-test-'));
|
||||
}
|
||||
|
||||
function makeDbPair(): { dir: string; localPath: string; remotePath: string } {
|
||||
const dir = makeTmpDir();
|
||||
const localPath = path.join(dir, 'local.sqlite');
|
||||
const remotePath = path.join(dir, 'remote.sqlite');
|
||||
createImmersionDbFixture(localPath);
|
||||
createImmersionDbFixture(remotePath);
|
||||
return { dir, localPath, remotePath };
|
||||
}
|
||||
|
||||
function queryOne<T extends Record<string, unknown>>(
|
||||
dbPath: string,
|
||||
sql: string,
|
||||
params: unknown[] = [],
|
||||
): T | undefined {
|
||||
const db = new Database(dbPath, { readonly: true });
|
||||
try {
|
||||
return db.query<T>(sql).get(...params) as T | undefined;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function count(dbPath: string, sql: string, params: unknown[] = []): number {
|
||||
return Number(queryOne<{ n: number }>(dbPath, sql, params)?.n ?? 0);
|
||||
}
|
||||
|
||||
function withWritableDb<T>(dbPath: string, fn: (db: Database) => T): T {
|
||||
const db = new Database(dbPath, { readwrite: true });
|
||||
try {
|
||||
return fn(db);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
test('merges remote-only sessions with catalog, lifetime, and rollups', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 2 }],
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
animeTitleKey: 'showb',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
activeWatchedMs: 900_000,
|
||||
cardsMined: 3,
|
||||
applyLifetime: true,
|
||||
words: [
|
||||
{ headword: '見る', word: '見た', reading: 'みた', count: 5 },
|
||||
{ headword: '食べる', word: '食べた', reading: 'たべた', count: 1 },
|
||||
],
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.sessionsMerged, 1);
|
||||
assert.equal(summary.animeAdded, 1);
|
||||
assert.equal(summary.videosAdded, 1);
|
||||
assert.equal(summary.wordsAdded, 1);
|
||||
assert.equal(summary.subtitleLinesAdded, 2);
|
||||
assert.equal(summary.telemetryRowsAdded, 1);
|
||||
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 2);
|
||||
const global = queryOne<{
|
||||
total_sessions: number;
|
||||
total_active_ms: number;
|
||||
total_cards: number;
|
||||
active_days: number;
|
||||
episodes_started: number;
|
||||
}>(
|
||||
localPath,
|
||||
'SELECT total_sessions, total_active_ms, total_cards, active_days, episodes_started FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.total_active_ms, 1_200_000 + 900_000);
|
||||
assert.equal(global?.total_cards, 2 + 3);
|
||||
assert.equal(global?.active_days, 2);
|
||||
assert.equal(global?.episodes_started, 2);
|
||||
|
||||
// Existing word: local 2 + merged 5; new word carries remote frequency.
|
||||
const sharedWord = queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '見た'`,
|
||||
);
|
||||
assert.equal(sharedWord?.frequency, 7);
|
||||
const newWord = queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '食べた'`,
|
||||
);
|
||||
assert.equal(newWord?.frequency, 1);
|
||||
|
||||
// The merged session's rollup group was recomputed.
|
||||
assert.equal(summary.rollupGroupsRecomputed, 1);
|
||||
const mergedVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'showb-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
assert.equal(
|
||||
count(localPath, 'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE video_id = ?', [
|
||||
mergedVideoId,
|
||||
]),
|
||||
1,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('is idempotent: re-merging the same snapshot changes nothing', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
words: [{ headword: '見る', word: '見た', reading: 'みた', count: 4 }],
|
||||
});
|
||||
|
||||
mergeSnapshotIntoDb(localPath, remotePath);
|
||||
const globalAfterFirst = queryOne<Record<string, unknown>>(
|
||||
localPath,
|
||||
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
|
||||
assert.equal(summary.sessionsMerged, 0);
|
||||
assert.equal(summary.sessionsAlreadyPresent, 1);
|
||||
assert.equal(summary.wordsAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 1);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_subtitle_lines'), 1);
|
||||
const globalAfterSecond = queryOne<Record<string, unknown>>(
|
||||
localPath,
|
||||
'SELECT * FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.deepEqual(
|
||||
{ ...globalAfterSecond, LAST_UPDATE_DATE: null },
|
||||
{ ...globalAfterFirst, LAST_UPDATE_DATE: null },
|
||||
);
|
||||
assert.equal(
|
||||
Number(
|
||||
queryOne<{ frequency: number }>(
|
||||
localPath,
|
||||
`SELECT frequency FROM imm_words WHERE word = '見た'`,
|
||||
)?.frequency,
|
||||
),
|
||||
4,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('preserves anilist_id when inserting a new anime from the snapshot', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
animeTitleKey: 'showb',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 12345 WHERE normalized_title_key = 'showb'`)
|
||||
.run();
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.animeAdded, 1);
|
||||
const anime = queryOne<{ anilist_id: number }>(
|
||||
localPath,
|
||||
`SELECT anilist_id FROM imm_anime WHERE normalized_title_key = 'showb'`,
|
||||
);
|
||||
assert.equal(anime?.anilist_id, 12345);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('matches an existing local anime by anilist_id even when the title key differs', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'show-romaji',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e2',
|
||||
animeTitleKey: 'show-native',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(localPath, (localDb) => {
|
||||
localDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-romaji'`)
|
||||
.run();
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb
|
||||
.prepare(`UPDATE imm_anime SET anilist_id = 999 WHERE normalized_title_key = 'show-native'`)
|
||||
.run();
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
// Same anilist_id → one anime, not two.
|
||||
assert.equal(summary.animeAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_anime'), 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('matches shared videos by video_key and merges the watched flag', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
watched: true,
|
||||
applyLifetime: true,
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.videosAdded, 0);
|
||||
assert.equal(summary.animeAdded, 0);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_videos'), 1);
|
||||
const video = queryOne<{ watched: number }>(
|
||||
localPath,
|
||||
`SELECT watched FROM imm_videos WHERE video_key = 'showa-e1'`,
|
||||
);
|
||||
assert.equal(video?.watched, 1);
|
||||
|
||||
// Both sessions now credit the same video; episode was started once and
|
||||
// completed once (by the remote session that watched it to the end).
|
||||
const global = queryOne<{
|
||||
episodes_started: number;
|
||||
episodes_completed: number;
|
||||
total_sessions: number;
|
||||
}>(
|
||||
localPath,
|
||||
'SELECT episodes_started, episodes_completed, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.episodes_started, 1);
|
||||
assert.equal(global?.episodes_completed, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('backfills media metadata for matched videos when local metadata is absent', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
animeTitleKey: 'showa',
|
||||
startedAtMs: BASE_MS + DAY_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const remoteDb = new Database(remotePath, { readwrite: true });
|
||||
try {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_media_art (video_id, anilist_id, cover_url, title_romaji, fetched_at_ms)
|
||||
VALUES (?, 123, 'https://example.test/cover.jpg', 'Show A', ?)`,
|
||||
)
|
||||
.run(remoteVideoId, String(BASE_MS));
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_youtube_videos (video_id, youtube_video_id, video_url, video_title, fetched_at_ms)
|
||||
VALUES (?, 'yt-1', 'https://youtube.test/watch?v=yt-1', 'Remote Video', ?)`,
|
||||
)
|
||||
.run(remoteVideoId, String(BASE_MS));
|
||||
} finally {
|
||||
remoteDb.close();
|
||||
}
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.videosAdded, 0);
|
||||
const localVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'showa-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
const art = queryOne<{ cover_url: string; title_romaji: string }>(
|
||||
localPath,
|
||||
'SELECT cover_url, title_romaji FROM imm_media_art WHERE video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(art?.cover_url, 'https://example.test/cover.jpg');
|
||||
assert.equal(art?.title_romaji, 'Show A');
|
||||
const youtube = queryOne<{ youtube_video_id: string; video_title: string }>(
|
||||
localPath,
|
||||
'SELECT youtube_video_id, video_title FROM imm_youtube_videos WHERE video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(youtube?.youtube_video_id, 'yt-1');
|
||||
assert.equal(youtube?.video_title, 'Remote Video');
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('does not double-count active_days when a merged session starts earlier on an already-credited day', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS + 3_600_000,
|
||||
applyLifetime: true,
|
||||
});
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showb-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
|
||||
mergeSnapshotIntoDb(localPath, remotePath);
|
||||
const global = queryOne<{ active_days: number; total_sessions: number }>(
|
||||
localPath,
|
||||
'SELECT active_days, total_sessions FROM imm_lifetime_global WHERE global_id = 1',
|
||||
);
|
||||
assert.equal(global?.total_sessions, 2);
|
||||
assert.equal(global?.active_days, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('skips unfinished sessions', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-active',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
endedAtMs: null,
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.sessionsMerged, 0);
|
||||
assert.equal(summary.activeSessionsSkipped, 1);
|
||||
assert.equal(count(localPath, 'SELECT COUNT(*) AS n FROM imm_sessions'), 0);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('copies remote-only historical rollups but never sums shared groups', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
// Remote has a video plus an old rollup row whose sessions were pruned.
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'old-show-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (10000, ?, 4, 120.5, 400, 3200, 9)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
});
|
||||
|
||||
// Local already has its own rollup row for a shared group.
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'old-show-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const localVideoId = withWritableDb(localPath, (localDb) => {
|
||||
const videoId = Number(
|
||||
localDb
|
||||
.query<{
|
||||
video_id: number;
|
||||
}>(`SELECT video_id FROM imm_videos WHERE video_key = 'old-show-e1'`)
|
||||
.get()?.video_id,
|
||||
);
|
||||
localDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (10000, ?, 2, 60.0, 200, 1600, 4)`,
|
||||
)
|
||||
.run(videoId);
|
||||
return videoId;
|
||||
});
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
// Shared group (day 10000) untouched; the remote-only session's own group
|
||||
// was recomputed, not copied.
|
||||
assert.equal(summary.dailyRollupsCopied, 0);
|
||||
const shared = queryOne<{ total_sessions: number }>(
|
||||
localPath,
|
||||
'SELECT total_sessions FROM imm_daily_rollups WHERE rollup_day = 10000 AND video_id = ?',
|
||||
[localVideoId],
|
||||
);
|
||||
assert.equal(shared?.total_sessions, 2);
|
||||
|
||||
// Now a rollup for a video with no local sessions at all gets copied.
|
||||
withWritableDb(remotePath, (remoteDb2) => {
|
||||
const orphanVideoId = Number(
|
||||
remoteDb2
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
|
||||
VALUES ('pruned-show-e1', 'pruned-show-e1', 1, 1, 1440000)`,
|
||||
)
|
||||
.run().lastInsertRowid,
|
||||
);
|
||||
remoteDb2
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (9000, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(orphanVideoId);
|
||||
});
|
||||
|
||||
const summary2 = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary2.dailyRollupsCopied, 1);
|
||||
const copiedVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'pruned-show-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
const copied = queryOne<{ total_sessions: number; total_active_min: number }>(
|
||||
localPath,
|
||||
'SELECT total_sessions, total_active_min FROM imm_daily_rollups WHERE rollup_day = 9000 AND video_id = ?',
|
||||
[copiedVideoId],
|
||||
);
|
||||
assert.equal(copied?.total_sessions, 3);
|
||||
assert.equal(copied?.total_active_min, 90.0);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('does not copy remote-only daily rollups into months with local sessions', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(localPath, {
|
||||
uuid: 'local-1',
|
||||
videoKey: 'mixed-month-e1',
|
||||
animeTitleKey: 'mixed-month',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const remoteDb = new Database(remotePath, { readwrite: true });
|
||||
try {
|
||||
const remoteVideoId = Number(
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, canonical_title, source_type, watched, duration_ms)
|
||||
VALUES ('mixed-month-e1', 'mixed-month-e1', 1, 1, 1440000)`,
|
||||
)
|
||||
.run().lastInsertRowid,
|
||||
);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_daily_rollups (rollup_day, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (20615, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
remoteDb
|
||||
.prepare(
|
||||
`INSERT INTO imm_monthly_rollups (rollup_month, video_id, total_sessions, total_active_min, total_lines_seen, total_tokens_seen, total_cards)
|
||||
VALUES (202606, ?, 3, 90.0, 300, 2400, 6)`,
|
||||
)
|
||||
.run(remoteVideoId);
|
||||
} finally {
|
||||
remoteDb.close();
|
||||
}
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, remotePath);
|
||||
assert.equal(summary.dailyRollupsCopied, 0);
|
||||
assert.equal(summary.monthlyRollupsCopied, 0);
|
||||
const localVideoId = Number(
|
||||
queryOne<{ video_id: number }>(
|
||||
localPath,
|
||||
`SELECT video_id FROM imm_videos WHERE video_key = 'mixed-month-e1'`,
|
||||
)?.video_id,
|
||||
);
|
||||
assert.equal(
|
||||
count(
|
||||
localPath,
|
||||
'SELECT COUNT(*) AS n FROM imm_daily_rollups WHERE rollup_day = 20615 AND video_id = ?',
|
||||
[localVideoId],
|
||||
),
|
||||
0,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects snapshots at a different schema version', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
withWritableDb(remotePath, (remoteDb) => {
|
||||
remoteDb.prepare('UPDATE imm_schema_version SET schema_version = 17').run();
|
||||
});
|
||||
|
||||
assert.throws(() => mergeSnapshotIntoDb(localPath, remotePath), /schema version 17/);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('createDbSnapshot produces a mergeable copy', () => {
|
||||
const { dir, localPath, remotePath } = makeDbPair();
|
||||
try {
|
||||
insertFixtureSession(remotePath, {
|
||||
uuid: 'remote-1',
|
||||
videoKey: 'showa-e1',
|
||||
startedAtMs: BASE_MS,
|
||||
applyLifetime: true,
|
||||
});
|
||||
const snapshotPath = path.join(dir, 'snapshot.sqlite');
|
||||
createDbSnapshot(remotePath, snapshotPath);
|
||||
assert.ok(fs.existsSync(snapshotPath));
|
||||
|
||||
const summary = mergeSnapshotIntoDb(localPath, snapshotPath);
|
||||
assert.equal(summary.sessionsMerged, 1);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,105 @@
|
||||
import fs from 'node:fs';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import {
|
||||
LexiconResolver,
|
||||
mergeAnime,
|
||||
mergeExcludedWords,
|
||||
mergeMediaMetadata,
|
||||
mergeVideos,
|
||||
} from './merge-catalog.js';
|
||||
import { mergeSessions } from './merge-sessions.js';
|
||||
import { copyRemoteOnlyRollups, refreshRollupsForNewSessions } from './merge-rollups.js';
|
||||
import {
|
||||
assertMergeableSchema,
|
||||
createEmptyMergeSummary,
|
||||
type SyncMergeSummary,
|
||||
} from './sync-shared.js';
|
||||
|
||||
export type { SyncMergeSummary } from './sync-shared.js';
|
||||
export { createDbSnapshot, findLiveStatsDaemonPid } from './sync-shared.js';
|
||||
|
||||
/**
|
||||
* Merge a snapshot of another machine's immersion database into the local
|
||||
* one. Insert-only union keyed on natural keys (session_uuid, video_key,
|
||||
* normalized_title_key, word/kanji identity); lifetime and rollup aggregates
|
||||
* are updated incrementally so history older than the session retention
|
||||
* window is preserved on both sides. Idempotent: re-merging the same
|
||||
* snapshot is a no-op.
|
||||
*/
|
||||
export function mergeSnapshotIntoDb(localDbPath: string, snapshotPath: string): SyncMergeSummary {
|
||||
if (!fs.existsSync(localDbPath)) {
|
||||
throw new Error(`Local stats database not found: ${localDbPath}`);
|
||||
}
|
||||
if (!fs.existsSync(snapshotPath)) {
|
||||
throw new Error(`Snapshot database not found: ${snapshotPath}`);
|
||||
}
|
||||
|
||||
const remote = new Database(snapshotPath, { readonly: true });
|
||||
let local: Database;
|
||||
try {
|
||||
local = new Database(localDbPath, { readwrite: true, create: false });
|
||||
} catch (error) {
|
||||
remote.close();
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
assertMergeableSchema(remote, 'Snapshot');
|
||||
assertMergeableSchema(local, 'Local');
|
||||
|
||||
const summary = createEmptyMergeSummary();
|
||||
local.run('PRAGMA foreign_keys = ON');
|
||||
local.run('PRAGMA busy_timeout = 5000');
|
||||
local.run('BEGIN IMMEDIATE');
|
||||
try {
|
||||
const animeIdMap = mergeAnime(local, remote, summary);
|
||||
const { videoIdMap, addedVideoIds } = mergeVideos(local, remote, animeIdMap, summary);
|
||||
mergeMediaMetadata(local, remote, videoIdMap, addedVideoIds);
|
||||
mergeExcludedWords(local, remote, summary);
|
||||
|
||||
const lexicon = new LexiconResolver(local, remote, summary);
|
||||
const { newSessionIds } = mergeSessions(
|
||||
local,
|
||||
remote,
|
||||
videoIdMap,
|
||||
animeIdMap,
|
||||
lexicon,
|
||||
summary,
|
||||
);
|
||||
lexicon.applyFrequencyDeltas();
|
||||
refreshRollupsForNewSessions(local, newSessionIds, summary);
|
||||
copyRemoteOnlyRollups(local, remote, videoIdMap, summary);
|
||||
|
||||
local.run('COMMIT');
|
||||
return summary;
|
||||
} catch (error) {
|
||||
local.run('ROLLBACK');
|
||||
throw error;
|
||||
}
|
||||
} finally {
|
||||
local.close();
|
||||
remote.close();
|
||||
}
|
||||
}
|
||||
|
||||
export function formatMergeSummary(summary: SyncMergeSummary): string {
|
||||
const lines = [
|
||||
`Sessions merged: ${summary.sessionsMerged} (${summary.sessionsAlreadyPresent} already present, ${summary.activeSessionsSkipped} unfinished skipped)`,
|
||||
];
|
||||
const detail: string[] = [];
|
||||
if (summary.animeAdded) detail.push(`${summary.animeAdded} series`);
|
||||
if (summary.videosAdded) detail.push(`${summary.videosAdded} videos`);
|
||||
if (summary.wordsAdded) detail.push(`${summary.wordsAdded} words`);
|
||||
if (summary.kanjiAdded) detail.push(`${summary.kanjiAdded} kanji`);
|
||||
if (summary.subtitleLinesAdded) detail.push(`${summary.subtitleLinesAdded} subtitle lines`);
|
||||
if (summary.excludedWordsAdded) detail.push(`${summary.excludedWordsAdded} excluded words`);
|
||||
if (detail.length > 0) lines.push(`Added: ${detail.join(', ')}`);
|
||||
if (summary.dailyRollupsCopied || summary.monthlyRollupsCopied) {
|
||||
lines.push(
|
||||
`Historical rollups copied: ${summary.dailyRollupsCopied} daily, ${summary.monthlyRollupsCopied} monthly`,
|
||||
);
|
||||
}
|
||||
if (summary.rollupGroupsRecomputed) {
|
||||
lines.push(`Rollup groups recomputed: ${summary.rollupGroupsRecomputed}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
|
||||
import { withReadonlyWalRetry } from '../history-db.js';
|
||||
import { resolveConfigDir } from '../../src/config/path-resolution.js';
|
||||
|
||||
export { SCHEMA_VERSION };
|
||||
|
||||
export interface SyncMergeSummary {
|
||||
sessionsMerged: number;
|
||||
sessionsAlreadyPresent: number;
|
||||
activeSessionsSkipped: number;
|
||||
animeAdded: number;
|
||||
videosAdded: number;
|
||||
wordsAdded: number;
|
||||
kanjiAdded: number;
|
||||
subtitleLinesAdded: number;
|
||||
telemetryRowsAdded: number;
|
||||
eventsAdded: number;
|
||||
excludedWordsAdded: number;
|
||||
dailyRollupsCopied: number;
|
||||
monthlyRollupsCopied: number;
|
||||
rollupGroupsRecomputed: number;
|
||||
}
|
||||
|
||||
export function createEmptyMergeSummary(): SyncMergeSummary {
|
||||
return {
|
||||
sessionsMerged: 0,
|
||||
sessionsAlreadyPresent: 0,
|
||||
activeSessionsSkipped: 0,
|
||||
animeAdded: 0,
|
||||
videosAdded: 0,
|
||||
wordsAdded: 0,
|
||||
kanjiAdded: 0,
|
||||
subtitleLinesAdded: 0,
|
||||
telemetryRowsAdded: 0,
|
||||
eventsAdded: 0,
|
||||
excludedWordsAdded: 0,
|
||||
dailyRollupsCopied: 0,
|
||||
monthlyRollupsCopied: 0,
|
||||
rollupGroupsRecomputed: 0,
|
||||
};
|
||||
}
|
||||
|
||||
export function nowDbTimestamp(): string {
|
||||
return String(Date.now());
|
||||
}
|
||||
|
||||
export function tableExists(db: Database, tableName: string): boolean {
|
||||
return Boolean(
|
||||
db.query(`SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?`).get(tableName),
|
||||
);
|
||||
}
|
||||
|
||||
export function readSchemaVersion(db: Database): number | null {
|
||||
if (!tableExists(db, 'imm_schema_version')) return null;
|
||||
const row = db
|
||||
.query<{ schema_version: number }>(
|
||||
'SELECT MAX(schema_version) AS schema_version FROM imm_schema_version',
|
||||
)
|
||||
.get();
|
||||
return typeof row?.schema_version === 'number' ? row.schema_version : null;
|
||||
}
|
||||
|
||||
export function assertMergeableSchema(db: Database, label: string): void {
|
||||
const version = readSchemaVersion(db);
|
||||
if (version === null) {
|
||||
throw new Error(
|
||||
`${label} database has no schema version. Run SubMiner once on that machine so the stats database is initialized.`,
|
||||
);
|
||||
}
|
||||
if (version !== SCHEMA_VERSION) {
|
||||
throw new Error(
|
||||
`${label} database is at schema version ${version} but this launcher expects ${SCHEMA_VERSION}. Update SubMiner on both machines to the same version and run each app once before syncing.`,
|
||||
);
|
||||
}
|
||||
for (const table of ['imm_sessions', 'imm_videos', 'imm_lifetime_global']) {
|
||||
if (!tableExists(db, table)) {
|
||||
throw new Error(`${label} database is missing table ${table}; cannot sync.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function insertRow(
|
||||
db: Database,
|
||||
table: string,
|
||||
columns: readonly string[],
|
||||
values: unknown[],
|
||||
): number {
|
||||
const sql = `INSERT INTO ${table} (${columns.join(', ')}) VALUES (${columns.map(() => '?').join(', ')})`;
|
||||
// db.query() caches the prepared statement per SQL string; this runs once
|
||||
// per copied row, so re-preparing via db.prepare() would dominate merge time.
|
||||
const result = db.query(sql).run(...values);
|
||||
return Number(result.lastInsertRowid);
|
||||
}
|
||||
|
||||
export function createDbSnapshot(dbPath: string, outPath: string): void {
|
||||
if (!fs.existsSync(dbPath)) {
|
||||
throw new Error(`Stats database not found: ${dbPath}`);
|
||||
}
|
||||
fs.rmSync(outPath, { force: true });
|
||||
fs.mkdirSync(path.dirname(outPath), { recursive: true });
|
||||
withReadonlyWalRetry(dbPath, (options) => {
|
||||
const db = new Database(dbPath, options);
|
||||
try {
|
||||
assertMergeableSchema(db, 'Local');
|
||||
db.prepare('VACUUM INTO ?').run(outPath);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
interface DaemonStateFile {
|
||||
pid?: unknown;
|
||||
}
|
||||
|
||||
function isProcessAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
// EPERM means the process exists but we can't signal it → still alive.
|
||||
// Only ESRCH (no such process) means it's actually gone.
|
||||
return (error as NodeJS.ErrnoException)?.code === 'EPERM';
|
||||
}
|
||||
}
|
||||
|
||||
function statsDaemonStateCandidates(dbPath: string): string[] {
|
||||
const homeDir = os.homedir();
|
||||
const candidates = new Set<string>([path.join(path.dirname(dbPath), 'stats-daemon.json')]);
|
||||
const configDir = resolveConfigDir({
|
||||
platform: process.platform,
|
||||
appDataDir: process.env.APPDATA,
|
||||
xdgConfigHome: process.env.XDG_CONFIG_HOME,
|
||||
homeDir,
|
||||
existsSync: fs.existsSync,
|
||||
});
|
||||
candidates.add(path.join(configDir, 'stats-daemon.json'));
|
||||
if (process.platform === 'darwin') {
|
||||
candidates.add(path.join(homeDir, 'Library', 'Application Support', 'SubMiner', 'stats-daemon.json'));
|
||||
}
|
||||
return [...candidates];
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort guard against merging while a SubMiner process holds the
|
||||
* tracker's write queue in memory. Detects the background stats daemon via
|
||||
* its pid state file; the interactive app is caught by the mpv-socket check
|
||||
* in the sync command.
|
||||
*/
|
||||
export function findLiveStatsDaemonPid(dbPath: string): number | null {
|
||||
for (const statePath of statsDaemonStateCandidates(dbPath)) {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = fs.readFileSync(statePath, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as DaemonStateFile;
|
||||
const pid = typeof parsed.pid === 'number' && Number.isInteger(parsed.pid) ? parsed.pid : 0;
|
||||
if (pid > 0 && isProcessAlive(pid)) {
|
||||
return pid;
|
||||
}
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,323 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { SCHEMA_VERSION } from '../../src/core/services/immersion-tracker/types.js';
|
||||
import { IMMERSION_DB_FIXTURE_DDL } from './immersion-db-schema.js';
|
||||
|
||||
export function createImmersionDbFixture(dbPath: string): void {
|
||||
const db = new Database(dbPath, { create: true });
|
||||
try {
|
||||
db.run('PRAGMA foreign_keys = ON');
|
||||
db.run('PRAGMA journal_mode = WAL');
|
||||
for (const statement of IMMERSION_DB_FIXTURE_DDL.split(';')) {
|
||||
const sql = statement.trim();
|
||||
if (sql) db.run(sql);
|
||||
}
|
||||
db.prepare('INSERT INTO imm_schema_version (schema_version, applied_at_ms) VALUES (?, ?)').run(
|
||||
SCHEMA_VERSION,
|
||||
String(Date.now()),
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO imm_rollup_state(state_key, state_value) VALUES ('last_rollup_sample_ms', 0)`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_global(global_id, CREATED_DATE, LAST_UPDATE_DATE) VALUES (1, ?, ?)`,
|
||||
).run(String(Date.now()), String(Date.now()));
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
let uniqueCounter = 0;
|
||||
|
||||
export interface FixtureSessionInput {
|
||||
uuid: string;
|
||||
videoKey: string;
|
||||
animeTitleKey?: string | null;
|
||||
animeEpisodesTotal?: number | null;
|
||||
startedAtMs: number;
|
||||
endedAtMs?: number | null;
|
||||
activeWatchedMs?: number;
|
||||
cardsMined?: number;
|
||||
linesSeen?: number;
|
||||
tokensSeen?: number;
|
||||
watched?: boolean;
|
||||
words?: Array<{ headword: string; word: string; reading: string; count: number }>;
|
||||
applyLifetime?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert a session (with video/anime/subtitle-line/occurrence rows) the way
|
||||
* the app would have recorded it, optionally crediting lifetime aggregates
|
||||
* the way applySessionLifetimeSummary does for locally-recorded sessions.
|
||||
*/
|
||||
export function insertFixtureSession(dbPath: string, input: FixtureSessionInput): void {
|
||||
const db = new Database(dbPath, { readwrite: true });
|
||||
const stamp = String(input.startedAtMs);
|
||||
try {
|
||||
db.run('PRAGMA foreign_keys = ON');
|
||||
let animeId: number | null = null;
|
||||
if (input.animeTitleKey) {
|
||||
const existing = db
|
||||
.query<{
|
||||
anime_id: number;
|
||||
}>('SELECT anime_id FROM imm_anime WHERE normalized_title_key = ?')
|
||||
.get(input.animeTitleKey);
|
||||
animeId = existing
|
||||
? existing.anime_id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_anime (normalized_title_key, canonical_title, episodes_total, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.animeTitleKey,
|
||||
input.animeTitleKey,
|
||||
input.animeEpisodesTotal ?? null,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
}
|
||||
|
||||
const existingVideo = db
|
||||
.query<{ video_id: number }>('SELECT video_id FROM imm_videos WHERE video_key = ?')
|
||||
.get(input.videoKey);
|
||||
const videoId = existingVideo
|
||||
? existingVideo.video_id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_videos (video_key, anime_id, canonical_title, source_type, source_path, watched, duration_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, 1, ?, ?, 1440000, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.videoKey,
|
||||
animeId,
|
||||
input.videoKey,
|
||||
`/videos/${input.videoKey}.mkv`,
|
||||
input.watched ? 1 : 0,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
if (input.watched) {
|
||||
db.prepare('UPDATE imm_videos SET watched = 1 WHERE video_id = ?').run(videoId);
|
||||
}
|
||||
|
||||
const endedAtMs =
|
||||
input.endedAtMs === undefined ? input.startedAtMs + 1_500_000 : input.endedAtMs;
|
||||
const activeWatchedMs = input.activeWatchedMs ?? 1_200_000;
|
||||
const cardsMined = input.cardsMined ?? 2;
|
||||
const linesSeen = input.linesSeen ?? 100;
|
||||
const tokensSeen = input.tokensSeen ?? 800;
|
||||
const sessionId = Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_sessions (
|
||||
session_uuid, video_id, started_at_ms, ended_at_ms, status,
|
||||
total_watched_ms, active_watched_ms, lines_seen, tokens_seen, cards_mined,
|
||||
CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
input.uuid,
|
||||
videoId,
|
||||
String(input.startedAtMs),
|
||||
endedAtMs === null ? null : String(endedAtMs),
|
||||
endedAtMs === null ? 1 : 2,
|
||||
activeWatchedMs,
|
||||
activeWatchedMs,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
cardsMined,
|
||||
stamp,
|
||||
stamp,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO imm_session_telemetry (
|
||||
session_id, sample_ms, total_watched_ms, active_watched_ms, lines_seen, tokens_seen,
|
||||
cards_mined, CREATED_DATE, LAST_UPDATE_DATE
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
sessionId,
|
||||
String(endedAtMs ?? input.startedAtMs),
|
||||
activeWatchedMs,
|
||||
activeWatchedMs,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
cardsMined,
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
|
||||
for (const word of input.words ?? []) {
|
||||
const existing = db
|
||||
.query<{
|
||||
id: number;
|
||||
}>('SELECT id FROM imm_words WHERE headword = ? AND word = ? AND reading = ?')
|
||||
.get(word.headword, word.word, word.reading);
|
||||
const wordId = existing
|
||||
? existing.id
|
||||
: Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_words (headword, word, reading, first_seen, last_seen, frequency)
|
||||
VALUES (?, ?, ?, ?, ?, 0)`,
|
||||
)
|
||||
.run(
|
||||
word.headword,
|
||||
word.word,
|
||||
word.reading,
|
||||
Math.floor(input.startedAtMs / 1000),
|
||||
Math.floor(input.startedAtMs / 1000),
|
||||
).lastInsertRowid,
|
||||
);
|
||||
uniqueCounter += 1;
|
||||
const lineId = Number(
|
||||
db
|
||||
.prepare(
|
||||
`INSERT INTO imm_subtitle_lines (session_id, video_id, anime_id, line_index, text, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
sessionId,
|
||||
videoId,
|
||||
animeId,
|
||||
uniqueCounter,
|
||||
`line ${word.word}`,
|
||||
input.startedAtMs,
|
||||
input.startedAtMs,
|
||||
).lastInsertRowid,
|
||||
);
|
||||
db.prepare(
|
||||
'INSERT INTO imm_word_line_occurrences (line_id, word_id, occurrence_count) VALUES (?, ?, ?)',
|
||||
).run(lineId, wordId, word.count);
|
||||
db.prepare('UPDATE imm_words SET frequency = frequency + ? WHERE id = ?').run(
|
||||
word.count,
|
||||
wordId,
|
||||
);
|
||||
}
|
||||
|
||||
if (input.applyLifetime && endedAtMs !== null) {
|
||||
applyFixtureLifetime(db, sessionId, videoId, animeId, {
|
||||
endedAtMs,
|
||||
startedAtMs: input.startedAtMs,
|
||||
activeWatchedMs,
|
||||
cardsMined,
|
||||
linesSeen,
|
||||
tokensSeen,
|
||||
watched: Boolean(input.watched),
|
||||
episodesTotal: input.animeEpisodesTotal ?? null,
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function applyFixtureLifetime(
|
||||
db: Database,
|
||||
sessionId: number,
|
||||
videoId: number,
|
||||
animeId: number | null,
|
||||
data: {
|
||||
endedAtMs: number;
|
||||
startedAtMs: number;
|
||||
activeWatchedMs: number;
|
||||
cardsMined: number;
|
||||
linesSeen: number;
|
||||
tokensSeen: number;
|
||||
watched: boolean;
|
||||
episodesTotal: number | null;
|
||||
},
|
||||
): void {
|
||||
const stamp = String(data.endedAtMs);
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_applied_sessions (session_id, applied_at_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, ?, ?, ?)`,
|
||||
).run(sessionId, String(data.endedAtMs), stamp, stamp);
|
||||
|
||||
const mediaLifetime = db
|
||||
.query<{ completed: number }>('SELECT completed FROM imm_lifetime_media WHERE video_id = ?')
|
||||
.get(videoId);
|
||||
const isFirstSessionForVideo = !mediaLifetime;
|
||||
const isFirstCompleted = data.watched && Number(mediaLifetime?.completed ?? 0) <= 0;
|
||||
const dayExpr = `CAST(julianday(CAST(started_at_ms AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER)`;
|
||||
const otherOnDay = db
|
||||
.query(
|
||||
`SELECT 1 FROM imm_sessions WHERE session_id != ? AND ${dayExpr} =
|
||||
CAST(julianday(CAST(? AS REAL) / 1000, 'unixepoch', 'localtime') - 2440587.5 AS INTEGER) LIMIT 1`,
|
||||
)
|
||||
.get(sessionId, String(data.startedAtMs));
|
||||
|
||||
db.prepare(
|
||||
`UPDATE imm_lifetime_global SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + ?,
|
||||
total_cards = total_cards + ?,
|
||||
active_days = active_days + ?,
|
||||
episodes_started = episodes_started + ?,
|
||||
episodes_completed = episodes_completed + ?
|
||||
WHERE global_id = 1`,
|
||||
).run(
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
otherOnDay ? 0 : 1,
|
||||
isFirstSessionForVideo ? 1 : 0,
|
||||
isFirstCompleted ? 1 : 0,
|
||||
);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_media (video_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(video_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
completed = MAX(completed, excluded.completed),
|
||||
last_watched_ms = excluded.last_watched_ms`,
|
||||
).run(
|
||||
videoId,
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
data.linesSeen,
|
||||
data.tokensSeen,
|
||||
data.watched ? 1 : 0,
|
||||
String(data.startedAtMs),
|
||||
String(data.endedAtMs),
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
|
||||
if (animeId !== null) {
|
||||
db.prepare(
|
||||
`INSERT INTO imm_lifetime_anime (anime_id, total_sessions, total_active_ms, total_cards, total_lines_seen, total_tokens_seen, episodes_started, episodes_completed, first_watched_ms, last_watched_ms, CREATED_DATE, LAST_UPDATE_DATE)
|
||||
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(anime_id) DO UPDATE SET
|
||||
total_sessions = total_sessions + 1,
|
||||
total_active_ms = total_active_ms + excluded.total_active_ms,
|
||||
total_cards = total_cards + excluded.total_cards,
|
||||
total_lines_seen = total_lines_seen + excluded.total_lines_seen,
|
||||
total_tokens_seen = total_tokens_seen + excluded.total_tokens_seen,
|
||||
episodes_started = episodes_started + excluded.episodes_started,
|
||||
episodes_completed = episodes_completed + excluded.episodes_completed,
|
||||
last_watched_ms = excluded.last_watched_ms`,
|
||||
).run(
|
||||
animeId,
|
||||
data.activeWatchedMs,
|
||||
data.cardsMined,
|
||||
data.linesSeen,
|
||||
data.tokensSeen,
|
||||
isFirstSessionForVideo ? 1 : 0,
|
||||
isFirstCompleted ? 1 : 0,
|
||||
String(data.startedAtMs),
|
||||
String(data.endedAtMs),
|
||||
stamp,
|
||||
stamp,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Database as BunDatabase } from 'bun:sqlite';
|
||||
import { ensureSchema } from '../../src/core/services/immersion-tracker/storage.js';
|
||||
import { createImmersionDbFixture, insertFixtureSession } from './immersion-db-fixture.js';
|
||||
|
||||
type SchemaRow = { type: string; name: string; tbl_name: string; sql: string | null };
|
||||
|
||||
const SYNC_SCHEMA_OBJECTS = [
|
||||
'imm_anime',
|
||||
'imm_videos',
|
||||
'imm_sessions',
|
||||
'imm_session_telemetry',
|
||||
'imm_session_events',
|
||||
'imm_daily_rollups',
|
||||
'imm_monthly_rollups',
|
||||
'imm_words',
|
||||
'imm_kanji',
|
||||
'imm_subtitle_lines',
|
||||
'imm_word_line_occurrences',
|
||||
'imm_kanji_line_occurrences',
|
||||
'imm_media_art',
|
||||
'imm_youtube_videos',
|
||||
'imm_cover_art_blobs',
|
||||
'imm_lifetime_global',
|
||||
'imm_lifetime_anime',
|
||||
'imm_lifetime_media',
|
||||
'imm_lifetime_applied_sessions',
|
||||
'imm_stats_excluded_words',
|
||||
'idx_anime_normalized_title',
|
||||
'idx_anime_anilist_id',
|
||||
'idx_videos_anime_id',
|
||||
'idx_sessions_video_started',
|
||||
'idx_sessions_status_started',
|
||||
'idx_sessions_started_at',
|
||||
'idx_sessions_ended_at',
|
||||
'idx_telemetry_session_sample',
|
||||
'idx_telemetry_sample_ms',
|
||||
'idx_events_session_ts',
|
||||
'idx_events_type_ts',
|
||||
'idx_rollups_day_video',
|
||||
'idx_rollups_month_video',
|
||||
'idx_words_headword_word_reading',
|
||||
'idx_words_frequency',
|
||||
'idx_kanji_kanji',
|
||||
'idx_kanji_frequency',
|
||||
'idx_subtitle_lines_session_line',
|
||||
'idx_subtitle_lines_video_line',
|
||||
'idx_subtitle_lines_anime_line',
|
||||
'idx_word_line_occurrences_word',
|
||||
'idx_kanji_line_occurrences_kanji',
|
||||
'idx_media_art_cover_blob_hash',
|
||||
'idx_media_art_anilist_id',
|
||||
'idx_media_art_cover_url',
|
||||
'idx_youtube_videos_channel_id',
|
||||
'idx_youtube_videos_youtube_video_id',
|
||||
] as const;
|
||||
|
||||
function normalizeSql(sql: string | null): string {
|
||||
return (sql ?? '')
|
||||
.replace(/\bIF NOT EXISTS\b/gi, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.replace(/\s+([(),])/g, '$1')
|
||||
.replace(/,\s+/g, ', ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function readSchema(dbPath: string): Map<string, string> {
|
||||
const db = new BunDatabase(dbPath, { readonly: true });
|
||||
try {
|
||||
const rows = db
|
||||
.query<SchemaRow>(
|
||||
`SELECT type, name, tbl_name, sql
|
||||
FROM sqlite_schema
|
||||
WHERE name IN (${SYNC_SCHEMA_OBJECTS.map(() => '?').join(',')})
|
||||
ORDER BY type, name`,
|
||||
)
|
||||
.all(...SYNC_SCHEMA_OBJECTS);
|
||||
return new Map(rows.map((row) => [row.name, normalizeSql(row.sql)]));
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
test('fixture schema stays aligned with production sync-touched tables and indexes', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-schema-'));
|
||||
const fixturePath = path.join(dir, 'fixture.sqlite');
|
||||
const productionPath = path.join(dir, 'production.sqlite');
|
||||
const productionDb = new BunDatabase(productionPath, { create: true });
|
||||
try {
|
||||
createImmersionDbFixture(fixturePath);
|
||||
ensureSchema(productionDb as never);
|
||||
productionDb.close();
|
||||
|
||||
const fixtureSchema = readSchema(fixturePath);
|
||||
const productionSchema = readSchema(productionPath);
|
||||
assert.deepEqual(fixtureSchema, productionSchema);
|
||||
} finally {
|
||||
try {
|
||||
productionDb.close();
|
||||
} catch {
|
||||
// already closed
|
||||
}
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('fixture session inserts enforce foreign keys', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'subminer-fixture-foreign-keys-'));
|
||||
const fixturePath = path.join(dir, 'fixture.sqlite');
|
||||
try {
|
||||
createImmersionDbFixture(fixturePath);
|
||||
const db = new BunDatabase(fixturePath, { readwrite: true });
|
||||
try {
|
||||
db.run(`
|
||||
CREATE TRIGGER remove_fixture_video
|
||||
BEFORE INSERT ON imm_sessions
|
||||
BEGIN
|
||||
DELETE FROM imm_videos WHERE video_id = NEW.video_id;
|
||||
END
|
||||
`);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
|
||||
assert.throws(
|
||||
() =>
|
||||
insertFixtureSession(fixturePath, {
|
||||
uuid: 'foreign-key-check',
|
||||
videoKey: 'foreign-key-check',
|
||||
startedAtMs: Date.UTC(2026, 6, 9),
|
||||
}),
|
||||
/FOREIGN KEY constraint failed/,
|
||||
);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,323 @@
|
||||
// Schema-version-18 shape of the tables the sync merge touches (plus the
|
||||
// app's indexes), mirroring ensureSchema / ensureLifetimeSummaryTables /
|
||||
// ensureStatsExcludedWordsTable in src/core/services/immersion-tracker/storage.ts.
|
||||
export const IMMERSION_DB_FIXTURE_DDL = `
|
||||
CREATE TABLE imm_schema_version (
|
||||
schema_version INTEGER PRIMARY KEY,
|
||||
applied_at_ms TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE imm_rollup_state(
|
||||
state_key TEXT PRIMARY KEY,
|
||||
state_value TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE imm_anime(
|
||||
anime_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
normalized_title_key TEXT NOT NULL UNIQUE,
|
||||
canonical_title TEXT NOT NULL,
|
||||
anilist_id INTEGER UNIQUE,
|
||||
title_romaji TEXT,
|
||||
title_english TEXT,
|
||||
title_native TEXT,
|
||||
episodes_total INTEGER,
|
||||
description TEXT,
|
||||
metadata_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_videos(
|
||||
video_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
video_key TEXT NOT NULL UNIQUE,
|
||||
anime_id INTEGER,
|
||||
canonical_title TEXT NOT NULL,
|
||||
source_type INTEGER NOT NULL,
|
||||
source_path TEXT,
|
||||
source_url TEXT,
|
||||
parsed_basename TEXT,
|
||||
parsed_title TEXT,
|
||||
parsed_season INTEGER,
|
||||
parsed_episode INTEGER,
|
||||
parser_source TEXT,
|
||||
parser_confidence REAL,
|
||||
parse_metadata_json TEXT,
|
||||
watched INTEGER NOT NULL DEFAULT 0,
|
||||
duration_ms INTEGER NOT NULL CHECK(duration_ms>=0),
|
||||
file_size_bytes INTEGER CHECK(file_size_bytes>=0),
|
||||
codec_id INTEGER, container_id INTEGER,
|
||||
width_px INTEGER, height_px INTEGER, fps_x100 INTEGER,
|
||||
bitrate_kbps INTEGER, audio_codec_id INTEGER,
|
||||
hash_sha256 TEXT, screenshot_path TEXT,
|
||||
metadata_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
|
||||
);
|
||||
CREATE TABLE imm_sessions(
|
||||
session_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_uuid TEXT NOT NULL UNIQUE,
|
||||
video_id INTEGER NOT NULL,
|
||||
started_at_ms TEXT NOT NULL, ended_at_ms TEXT,
|
||||
status INTEGER NOT NULL,
|
||||
locale_id INTEGER, target_lang_id INTEGER,
|
||||
difficulty_tier INTEGER, subtitle_mode INTEGER,
|
||||
ended_media_ms INTEGER,
|
||||
total_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
active_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
cards_mined INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_hits INTEGER NOT NULL DEFAULT 0,
|
||||
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_ms INTEGER NOT NULL DEFAULT 0,
|
||||
seek_forward_count INTEGER NOT NULL DEFAULT 0,
|
||||
seek_backward_count INTEGER NOT NULL DEFAULT 0,
|
||||
media_buffer_events INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id)
|
||||
);
|
||||
CREATE TABLE imm_session_telemetry(
|
||||
telemetry_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
sample_ms TEXT NOT NULL,
|
||||
total_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
active_watched_ms INTEGER NOT NULL DEFAULT 0,
|
||||
lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
cards_mined INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
lookup_hits INTEGER NOT NULL DEFAULT 0,
|
||||
yomitan_lookup_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_count INTEGER NOT NULL DEFAULT 0,
|
||||
pause_ms INTEGER NOT NULL DEFAULT 0,
|
||||
seek_forward_count INTEGER NOT NULL DEFAULT 0,
|
||||
seek_backward_count INTEGER NOT NULL DEFAULT 0,
|
||||
media_buffer_events INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_session_events(
|
||||
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
ts_ms TEXT NOT NULL,
|
||||
event_type INTEGER NOT NULL,
|
||||
line_index INTEGER,
|
||||
segment_start_ms INTEGER,
|
||||
segment_end_ms INTEGER,
|
||||
tokens_delta INTEGER NOT NULL DEFAULT 0,
|
||||
cards_delta INTEGER NOT NULL DEFAULT 0,
|
||||
payload_json TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_daily_rollups(
|
||||
rollup_day INTEGER NOT NULL,
|
||||
video_id INTEGER,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_min REAL NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
cards_per_hour REAL,
|
||||
tokens_per_min REAL,
|
||||
lookup_hit_rate REAL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY (rollup_day, video_id)
|
||||
);
|
||||
CREATE TABLE imm_monthly_rollups(
|
||||
rollup_month INTEGER NOT NULL,
|
||||
video_id INTEGER,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_min REAL NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY (rollup_month, video_id)
|
||||
);
|
||||
CREATE TABLE imm_words(
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
headword TEXT,
|
||||
word TEXT,
|
||||
reading TEXT,
|
||||
part_of_speech TEXT,
|
||||
pos1 TEXT,
|
||||
pos2 TEXT,
|
||||
pos3 TEXT,
|
||||
first_seen REAL,
|
||||
last_seen REAL,
|
||||
frequency INTEGER,
|
||||
frequency_rank INTEGER,
|
||||
UNIQUE(headword, word, reading)
|
||||
);
|
||||
CREATE TABLE imm_kanji(
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
kanji TEXT,
|
||||
first_seen REAL,
|
||||
last_seen REAL,
|
||||
frequency INTEGER,
|
||||
UNIQUE(kanji)
|
||||
);
|
||||
CREATE TABLE imm_subtitle_lines(
|
||||
line_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id INTEGER NOT NULL,
|
||||
event_id INTEGER,
|
||||
video_id INTEGER NOT NULL,
|
||||
anime_id INTEGER,
|
||||
line_index INTEGER NOT NULL,
|
||||
segment_start_ms INTEGER,
|
||||
segment_end_ms INTEGER,
|
||||
text TEXT NOT NULL,
|
||||
secondary_text TEXT,
|
||||
CREATED_DATE INTEGER,
|
||||
LAST_UPDATE_DATE INTEGER,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(event_id) REFERENCES imm_session_events(event_id) ON DELETE SET NULL,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE SET NULL
|
||||
);
|
||||
CREATE TABLE imm_word_line_occurrences(
|
||||
line_id INTEGER NOT NULL,
|
||||
word_id INTEGER NOT NULL,
|
||||
occurrence_count INTEGER NOT NULL,
|
||||
PRIMARY KEY(line_id, word_id),
|
||||
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(word_id) REFERENCES imm_words(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_kanji_line_occurrences(
|
||||
line_id INTEGER NOT NULL,
|
||||
kanji_id INTEGER NOT NULL,
|
||||
occurrence_count INTEGER NOT NULL,
|
||||
PRIMARY KEY(line_id, kanji_id),
|
||||
FOREIGN KEY(line_id) REFERENCES imm_subtitle_lines(line_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(kanji_id) REFERENCES imm_kanji(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_media_art(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
anilist_id INTEGER,
|
||||
cover_url TEXT,
|
||||
cover_blob BLOB,
|
||||
cover_blob_hash TEXT,
|
||||
title_romaji TEXT,
|
||||
title_english TEXT,
|
||||
episodes_total INTEGER,
|
||||
fetched_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_youtube_videos(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
youtube_video_id TEXT NOT NULL,
|
||||
video_url TEXT NOT NULL,
|
||||
video_title TEXT,
|
||||
video_thumbnail_url TEXT,
|
||||
channel_id TEXT,
|
||||
channel_name TEXT,
|
||||
channel_url TEXT,
|
||||
channel_thumbnail_url TEXT,
|
||||
uploader_id TEXT,
|
||||
uploader_url TEXT,
|
||||
description TEXT,
|
||||
metadata_json TEXT,
|
||||
fetched_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_cover_art_blobs(
|
||||
blob_hash TEXT PRIMARY KEY,
|
||||
cover_blob BLOB NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_lifetime_global(
|
||||
global_id INTEGER PRIMARY KEY CHECK(global_id = 1),
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
active_days INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_started INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_completed INTEGER NOT NULL DEFAULT 0,
|
||||
anime_completed INTEGER NOT NULL DEFAULT 0,
|
||||
last_rebuilt_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT
|
||||
);
|
||||
CREATE TABLE imm_lifetime_anime(
|
||||
anime_id INTEGER PRIMARY KEY,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_started INTEGER NOT NULL DEFAULT 0,
|
||||
episodes_completed INTEGER NOT NULL DEFAULT 0,
|
||||
first_watched_ms TEXT,
|
||||
last_watched_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(anime_id) REFERENCES imm_anime(anime_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_lifetime_media(
|
||||
video_id INTEGER PRIMARY KEY,
|
||||
total_sessions INTEGER NOT NULL DEFAULT 0,
|
||||
total_active_ms INTEGER NOT NULL DEFAULT 0,
|
||||
total_cards INTEGER NOT NULL DEFAULT 0,
|
||||
total_lines_seen INTEGER NOT NULL DEFAULT 0,
|
||||
total_tokens_seen INTEGER NOT NULL DEFAULT 0,
|
||||
completed INTEGER NOT NULL DEFAULT 0,
|
||||
first_watched_ms TEXT,
|
||||
last_watched_ms TEXT,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(video_id) REFERENCES imm_videos(video_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_lifetime_applied_sessions(
|
||||
session_id INTEGER PRIMARY KEY,
|
||||
applied_at_ms TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
FOREIGN KEY(session_id) REFERENCES imm_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE imm_stats_excluded_words(
|
||||
headword TEXT NOT NULL,
|
||||
word TEXT NOT NULL,
|
||||
reading TEXT NOT NULL,
|
||||
CREATED_DATE TEXT,
|
||||
LAST_UPDATE_DATE TEXT,
|
||||
PRIMARY KEY(headword, word, reading)
|
||||
);
|
||||
CREATE INDEX idx_anime_normalized_title ON imm_anime(normalized_title_key);
|
||||
CREATE INDEX idx_anime_anilist_id ON imm_anime(anilist_id);
|
||||
CREATE INDEX idx_videos_anime_id ON imm_videos(anime_id);
|
||||
CREATE INDEX idx_sessions_video_started ON imm_sessions(video_id, started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_status_started ON imm_sessions(status, started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_started_at ON imm_sessions(started_at_ms DESC);
|
||||
CREATE INDEX idx_sessions_ended_at ON imm_sessions(ended_at_ms DESC);
|
||||
CREATE INDEX idx_telemetry_session_sample ON imm_session_telemetry(session_id, sample_ms DESC);
|
||||
CREATE INDEX idx_telemetry_sample_ms ON imm_session_telemetry(sample_ms DESC);
|
||||
CREATE INDEX idx_events_session_ts ON imm_session_events(session_id, ts_ms DESC);
|
||||
CREATE INDEX idx_events_type_ts ON imm_session_events(event_type, ts_ms DESC);
|
||||
CREATE INDEX idx_rollups_day_video ON imm_daily_rollups(rollup_day, video_id);
|
||||
CREATE INDEX idx_rollups_month_video ON imm_monthly_rollups(rollup_month, video_id);
|
||||
CREATE INDEX idx_words_headword_word_reading ON imm_words(headword, word, reading);
|
||||
CREATE INDEX idx_words_frequency ON imm_words(frequency DESC);
|
||||
CREATE INDEX idx_kanji_kanji ON imm_kanji(kanji);
|
||||
CREATE INDEX idx_kanji_frequency ON imm_kanji(frequency DESC);
|
||||
CREATE INDEX idx_subtitle_lines_session_line ON imm_subtitle_lines(session_id, line_index);
|
||||
CREATE INDEX idx_subtitle_lines_video_line ON imm_subtitle_lines(video_id, line_index);
|
||||
CREATE INDEX idx_subtitle_lines_anime_line ON imm_subtitle_lines(anime_id, line_index);
|
||||
CREATE INDEX idx_word_line_occurrences_word ON imm_word_line_occurrences(word_id, line_id);
|
||||
CREATE INDEX idx_kanji_line_occurrences_kanji ON imm_kanji_line_occurrences(kanji_id, line_id);
|
||||
CREATE INDEX idx_media_art_cover_blob_hash ON imm_media_art(cover_blob_hash);
|
||||
CREATE INDEX idx_media_art_anilist_id ON imm_media_art(anilist_id);
|
||||
CREATE INDEX idx_media_art_cover_url ON imm_media_art(cover_url);
|
||||
CREATE INDEX idx_youtube_videos_channel_id ON imm_youtube_videos(channel_id);
|
||||
CREATE INDEX idx_youtube_videos_youtube_video_id ON imm_youtube_videos(youtube_video_id);
|
||||
`;
|
||||
@@ -113,6 +113,14 @@ export interface Args {
|
||||
texthookerOpenBrowser: boolean;
|
||||
useRofi: boolean;
|
||||
history: boolean;
|
||||
sync: boolean;
|
||||
syncHost: string;
|
||||
syncSnapshotPath: string;
|
||||
syncMergePath: string;
|
||||
syncDirection: 'both' | 'push' | 'pull';
|
||||
syncRemoteCmd: string;
|
||||
syncDbPath: string;
|
||||
syncForce: boolean;
|
||||
logLevel: LogLevel;
|
||||
logRotation: LogRotation;
|
||||
passwordStore: string;
|
||||
|
||||
Reference in New Issue
Block a user