mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-09-10 17:16:20 -07:00
perf(sync): cache received snapshots for incremental transfers
This commit is contained in:
@@ -101,10 +101,14 @@ subminer sync macbook --check # test SSH + remote SubMiner without sync
|
||||
subminer sync --ui # open the sync window (also in the tray menu)
|
||||
```
|
||||
|
||||
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over `scp`, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time. Syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
|
||||
How it works: each side takes a consistent snapshot of its database (`VACUUM INTO`), the snapshots are exchanged over SSH, and each machine merges the other's snapshot into its own database. The merge is an insert-only union keyed on stable identifiers (session UUIDs, video keys, series title keys, word/kanji identity), so it is safe to re-run at any time. Syncing twice changes nothing, and nothing is ever overwritten or summed twice. Lifetime totals and rollup charts are updated incrementally, so history older than the session retention window is preserved on both sides.
|
||||
|
||||
On macOS and Linux, sync automatically uses compressed `rsync` transfers when compatible `rsync` commands are available on both machines. The last successfully received snapshot supplies matching blocks for later transfers, so unchanged data can be reused without sending it again. Only unmatched data needs to cross the connection, with compression reducing it further. Without a cached snapshot, sync sends a full compressed snapshot. Windows endpoints and machines without compatible `rsync` use compressed `scp` automatically. No extra configuration is required, and both methods work across different networks, including Tailscale connections.
|
||||
|
||||
For a one-way transfer, `--push` snapshots the local database and merges it into the host without changing the local database. `--pull` snapshots the host and merges it into the local database without changing the host. These modes add missing data; they do not delete destination-only data or make the destination an exact mirror.
|
||||
|
||||
Transfers write separate temporary files and verify the reconstructed content before merging. Cached comparison snapshots are preserved throughout the transfer. After a successful rsync sync, each receiver keeps one snapshot per peer/database identity in `sync-transfer-cache/` under its SubMiner config directory. This uses roughly one database-sized file per identity; deleting that cache is safe and only makes the next sync transfer more data. Missing or unwritable caches do not prevent syncing. Older peers without the cache helper still support compressed transfers, but cannot retain the upload comparison copy.
|
||||
|
||||
Command-line sync defaults to a cold-start safety check: close SubMiner (and stop the background stats daemon with `subminer stats -s`) on both machines before running it, or pass `--force`. Syncs started from the Sync window use live mode automatically, including scheduled auto-syncs while SubMiner or playback is active. SQLite WAL provides a consistent snapshot, the transactional merge serializes with live writes, and each machine's unfinished session is excluded from the transfer; that session syncs normally after it finishes. The mpv safety check requires a live socket connection, so a stale socket file left after mpv exits does not block command-line sync. Both machines must be on the same SubMiner version; otherwise, the sync aborts on a stats schema mismatch.
|
||||
|
||||
On the remote, sync looks for the `subminer` launcher first (PATH and `~/.local/bin`), then the app binary in `--sync-cli` mode (`SubMiner` on PATH, then the standard macOS `/Applications` and `~/Applications` installs), checking standard SubMiner and Bun locations (`~/.local/bin`, `~/.bun/bin`, Homebrew, `/usr/local/bin`, `/usr/bin`, and `/bin`) even when the non-interactive SSH shell omits them from `PATH`. An AppImage in a custom location can be addressed with `--remote-cmd /path/to/SubMiner.AppImage` (or symlink it as `SubMiner` somewhere on the remote PATH).
|
||||
@@ -122,7 +126,7 @@ Unfinished sessions (a crash mid-playback) are skipped until the app finalizes t
|
||||
|
||||
`subminer sync <host> --check` verifies a host without touching any data: it probes the SSH connection, locates SubMiner on the remote (launcher or app binary), and reports its version. `--json` switches any sync mode to machine-readable NDJSON progress output (this is what the sync window consumes).
|
||||
|
||||
`sync --make-temp` creates a restricted temporary directory and prints its path; `sync --remove-temp <dir>` removes one created by that command. They are internal SSH transfer helpers, exposed for compatibility but normally invoked only by sync itself. `SubMiner --sync-cli sync ...` is the packaged app's headless compatibility entrypoint; use `SubMiner --sync-cli --help` for its sync-specific help. The `subminer sync` launcher command selects this entrypoint automatically and runs AppImages in Node-only mode, so remote sync does not require a graphical session.
|
||||
`sync --make-temp` creates a restricted temporary directory and prints its path; `sync --remove-temp <dir>` removes one created by that command. The internal `--transfer-cache <key>` option seeds the temporary directory from a previous received snapshot when creating it, or saves the received snapshot before removing it after a successful sync. Keys are 64-character lowercase hexadecimal identifiers. These are internal SSH transfer helpers, exposed for compatibility but normally invoked only by sync itself. `SubMiner --sync-cli sync ...` is the packaged app's headless compatibility entrypoint; use `SubMiner --sync-cli --help` for its sync-specific help. The `subminer sync` launcher command selects this entrypoint automatically and runs AppImages in Node-only mode, so remote sync does not require a graphical session.
|
||||
|
||||
### Sync window
|
||||
|
||||
|
||||
@@ -191,6 +191,8 @@ The launcher groups related work under subcommands: `jellyfin` (aliased `jf`), `
|
||||
|
||||
Every subcommand has its own help page, for example `subminer jellyfin -h`. See [Launcher Script - Subcommands](/launcher-script#subcommands) for the full table, and [Sync Between Machines](/launcher-script#sync-between-machines) for the SSH stats/history sync.
|
||||
|
||||
Sync selects compressed transfers automatically and reuses cached snapshots when rsync is available. Its `--transfer-cache <key>` option belongs to the internal `--make-temp` / `--remove-temp` helpers; normal `subminer sync <host>` commands manage it for you. See [Sync Between Machines](/launcher-script#sync-between-machines) for cache storage and compatibility details.
|
||||
|
||||
A _texthooker_ is a web page that displays the current subtitle line as selectable text, so browser-based dictionary extensions and other tools can read along with playback.
|
||||
|
||||
### First-Run Setup
|
||||
|
||||
Reference in New Issue
Block a user