fix(stats): stop counting duplicate typeset subtitle lines (#191)

2026-08-15 01:55:52 -07:00 · 2026-08-14 00:12:56 -07:00
parent 046e74ea91
commit b98d4d65c7
34 changed files with 2644 additions and 35 deletions
@@ -34,7 +34,7 @@ The same immersion data powers the stats dashboard.
 - In-app overlay: focus the visible overlay, then press the key from `stats.toggleKey` (default: `` ` `` / `Backquote`).
 - Launcher command: run `subminer stats` to start the local stats server on demand (it also opens the dashboard in your browser when `stats.autoOpenBrowser` is enabled; the default is `false`).
 - Background server: run `subminer stats -b` to start or reuse a dedicated background stats daemon without keeping the launcher attached, and `subminer stats -s` to stop that daemon.
- Maintenance commands: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand; `subminer stats cleanup -l` repairs lifetime summary tables. `subminer stats rebuild` and `subminer stats backfill` rebuild or backfill rollup data.
+- Maintenance commands: run `subminer stats cleanup` or `subminer stats cleanup -v` to backfill/repair vocabulary metadata (`headword`, `reading`, POS) and purge stale or excluded rows from `imm_words` on demand; `subminer stats cleanup -l` repairs lifetime summary tables; `subminer stats cleanup --duplicate-lines` collapses repeated lines left behind by typeset subtitles (see [Repeated Line Cleanup](#repeated-line-cleanup)). `subminer stats rebuild` and `subminer stats backfill` rebuild or backfill rollup data.
 - Browser page: open `http://127.0.0.1:6969` directly if the local stats server is already running.

 ### Dashboard Tabs
@@ -125,6 +125,34 @@ Secondary subtitle text (typically English translations) is stored alongside pri

 The Vocabulary tab toolbar includes an **Exclusions** button for hiding words from all vocabulary views. Excluded words are stored in the immersion database, with older browser localStorage exclusions imported on first load after upgrade. They can be managed (restored or cleared) from the exclusion modal. Exclusions affect stat cards, charts, the frequency rank table, and the word list.

+### Repeated Line Cleanup
+
+Karaoke openings and animated signs are authored as one subtitle event per animation frame, all carrying the same text. Playback reports every one of those frames, so a single OP lyric could be recorded hundreds of times and dominate "Top Repeated Words".
+
+Recording now collapses those runs as they happen, matching what the subtitle sidebar shows:
+
+- When the active subtitle source has been parsed, its cue list has already had duplicate events and animation bursts merged. A line landing inside a surviving cue but after that cue's start is a frame the sidebar merged away, and is not recorded.
+- When no parsed cue covers the live timing, including while a subtitle source is changing or shifted, the strict metadata-free rule applies: a run of identical, contiguous lines each shorter than 0.1s stops being recorded after a few frames. Runs are tracked per line of text, so dual-line karaoke (a kanji and a romaji line frame-flipped together) collapses both lines. Ordinary repeated dialogue, and lines held for a normal beat, always record.
+
+For stats recorded before this, the Vocabulary tab toolbar has a **Duplicates** button:
+
+- Pick how far back to look (7 days, 30 days, 90 days, 1 year, or all time). A narrower window does less work and keeps older history untouched.
+- **Scan** reports the bursts found, the lines they added, and the word and kanji counts they inflated, without writing anything.
+- **Clean Up** applies exactly what the scan reported: each run collapses to its first line (extended to cover the run), and the removed lines' word and kanji occurrences are subtracted from the vocabulary aggregates.
+
+The same thing runs from the terminal:
+
+```bash
+subminer stats cleanup --duplicate-lines --dry-run --lookback-days 30
+subminer stats cleanup --duplicate-lines --lookback-days 30
+```
+
+`--duplicate-lines` (short: `-d`) picks the cleanup mode, so it cannot be combined with `--vocab` or `--lifetime`, and `--dry-run` and `--lookback-days <days>` only apply to it. Omitting `--lookback-days` scans all history; the value must be at least one day.
+
+The cleanup chains runs per line of text, so interleaved dual-line karaoke collapses each of its lines. It also removes the short residue the live rule stores before a run is long enough to recognize: a run one frame short of the usual minimum qualifies when every event is under the strict 0.1s bound.
+
+Runs never cross a session boundary, so rewatching an episode keeps both watches. Session telemetry (watch time, lines seen, tokens seen) and the rollups derived from it are left as recorded: they are cumulative samples taken during playback, and cannot be recomputed for sessions whose raw rows have since been pruned.
+
 ## Retention Defaults

 By default, SubMiner keeps all retention tables and raw data (`0` means keep all) while continuing daily/monthly rollup maintenance:
@@ -151,6 +151,7 @@ subminer stats -b                       # start background stats daemon
 | `subminer stats`                           | Start the stats server (opens the dashboard when `stats.autoOpenBrowser` is on)                   |
 | `subminer stats -b` / `-s`                 | Start/reuse or stop the background stats daemon                                                   |
 | `subminer stats cleanup`                   | Backfill vocabulary metadata and prune stale rows (`-v` vocab, `-l` lifetime summaries)           |
+| `subminer stats cleanup -d`                | Collapse repeated lines from typeset subs (`--dry-run`, `--lookback-days <n>`)                    |
 | `subminer stats rebuild` / `backfill`      | Rebuild or backfill rollup data                                                                   |
 | `subminer doctor`                          | Dependency + config + socket diagnostics (`--refresh-known-words` refreshes the known-word cache) |
 | `subminer settings`                        | Open the SubMiner settings window                                                                 |
@@ -95,6 +95,8 @@ subminer texthooker               # Texthooker-only mode (-o also opens the brow
 subminer stats -b                 # Start/reuse the background stats daemon
 subminer stats -s                 # Stop the background stats daemon
 subminer stats cleanup            # Backfill vocabulary metadata, prune stale rows
+subminer stats cleanup -d --dry-run  # Preview cleanup of repeated typeset subtitle lines
+subminer stats cleanup -d --lookback-days 30  # Clean only lines recorded in the last 30 days
 subminer stats rebuild            # Rebuild rollup data
 subminer doctor --refresh-known-words  # Refresh the known-word cache
 subminer logs -e                  # Export a sanitized log ZIP and print its path
@@ -107,6 +109,8 @@ subminer app --stop               # Stop the background app
 subminer --version                # Print the launcher's version
 ```

+`stats cleanup` runs one mode per invocation: `-v`/`--vocab` (the default), `-l`/`--lifetime`, or `-d`/`--duplicate-lines`; explicitly selected modes cannot be combined. `--dry-run` and `--lookback-days <days>` apply to `--duplicate-lines` only and are rejected without it; `--lookback-days` must be at least one day, and leaving it off scans all history.
+
 Jellyfin, cross-machine sync, and character-dictionary commands have their own sections: [Jellyfin](/jellyfin-integration), [Sync Between Machines](/launcher-script#sync-between-machines), and [Character Dictionary](/character-dictionary).

 </details>
@@ -137,7 +141,7 @@ SubMiner.AppImage --start --log-level debug  # Verbose logging without dev mode
 SubMiner.AppImage --help                  # Show all options
 ```

-The remaining flags are internal or scripting-only surfaces: the `--jellyfin-*` family (login, library listing, item playback, cast announce), `--sync-cli` (the app's headless sync entrypoint that `subminer sync` proxies to), `--dictionary-candidates` / `--dictionary-select`, and `--playback-feedback <text>`. Run `SubMiner.AppImage --help` for the complete list. The previous `--open-animetosho` flag is still accepted as a deprecated alias for `--open-tsukihime`.
+The remaining flags are internal or scripting-only surfaces: the `--jellyfin-*` family (login, library listing, item playback, cast announce), `--sync-cli` (the app's headless sync entrypoint that `subminer sync` proxies to), the `--stats-cleanup-*` family that `subminer stats cleanup` forwards (`--stats-cleanup-vocab`, `--stats-cleanup-lifetime`, `--stats-cleanup-duplicate-lines`, and its `--stats-cleanup-dry-run` / `--stats-cleanup-lookback-days <days>` modifiers), `--dictionary-candidates` / `--dictionary-select`, and `--playback-feedback <text>`. Run `SubMiner.AppImage --help` for the complete list. The previous `--open-animetosho` flag is still accepted as a deprecated alias for `--open-tsukihime`.

 </details>