SubMiner/docs/architecture/stats-trends-data-flow.md

# Stats Trends Data Flow

read_when: touching stats trend charts, changing stats API payloads, or debugging dashboard performance

## Summary

Trend charts now consume one chart-oriented backend payload from `/api/stats/trends/dashboard`.

## Why

- remove repeated client-side dataset rebuilding in `TrendsTab`
- collapse multiple network round-trips into one request
- keep heavy chart shaping close to tracker/query logic

## Date Ranges And Buckets

- A range resolves to calendar days (`resolveTrendWindow`). Rollups are selected by day span, never by "last N rows", so sparse activity cannot pull older days into a range. Custom ranges are `{ startDay, endDay }` from `range=custom&fromDay&toDay`.
- `all` spans the recorded data, not today.
- Day and week buckets read daily rollups. Month buckets read monthly rollups for months wholly in range, because the `minimal` and `deep-history` retention presets prune daily rollups first, and daily rollups for a partly covered month. Weeks start on Monday.
- `buckets` is the one axis for `activity`, `progress`, and `ratios`, index for index. Charts use it to map a clicked bar back to its days.
- Each `progress` point is the total at the end of its period, and the chart adds the zero starting point.

## Data Sources

- rollup-backed:
  - activity charts and running totals
  - `titleTimeline`: sparse per-title minutes for active days only, because a dense title x day payload grew to several MB
- session-metric-backed:
  - lookups and lookup rate
  - `watchTimeByDayHour`, attributed to the hour each session started
- rollup word counts and session metrics share one session query per request (`getTrendSessionRows`), so the per-session display-word count runs once instead of once per rollup query
- `imm_videos.watched`-backed:
  - finished episodes count each watched video once, on the last day it has a rollup, since the flag has no timestamp
- vocabulary-backed:
  - `getNewWordFirstSeenRows` folds `imm_words` rows into dictionary words first, because the table is unique on (headword, word, reading) and stores each conjugation separately
  - the aggregate scans every word, so it is cached per connection until `total_changes()` or `PRAGMA data_version` moves
  - Trends new-word metrics exclude names; the Vocabulary tab offers both
  - `imm_lexical_daily_rollups` is no longer read for new-word counts (it counts surface forms). Its version state still gates the `vocabulary_visible` backfill.

## Vocabulary Word List

`/api/stats/vocabulary/words` (`query-vocabulary-words.ts`) folds `imm_words` into dictionary words on the server and filters, sorts, and pages there, because a client-side list can only hold a slice of the table. The fold is cached per connection until a write, the same way as the new-word aggregate. `titles` is computed for the returned rows; sorting or filtering by it needs every word's count (about 900ms), so that map tolerates five minutes of staleness. Names are detected with `isProperNounPos2`, since merged tokens store tags like `固有名詞|一般`.

## Rates

Rates are `null` when the underlying data is under 30 active minutes or 2,000 words, and charts draw that as a gap. Daily views rate a trailing 7 calendar days, volume-weighted, which is why the query loads days before the range. Week and month views rate each bucket.

## Title Filter

`title` narrows totals, charts, and patterns. `librarySummary` and `titleTimeline` stay complete so the page can switch titles. New words have no per-title record, so `learning` and `summary.current.newWords` are `null` while filtered.

## Metric Semantics

- subtitle-count stats now use Yomitan merged-token counts as the source of truth
- `tokensSeen` is the only active subtitle-count metric in tracker/session/rollup/query paths
- no whitespace/CJK-character fallback remains in the live stats path

## Contract

Stats data crosses the process boundary over HTTP only. Shared JSON models live in `src/types/stats-wire.ts`; endpoint request/response mappings and the client interface live in `src/types/stats-http-contract.ts`. The server and stats client both type-check against those files. `stats/src/types/stats.ts` is a compatibility re-export, not an independent contract copy.

The stats preload bridge remains limited to native window behavior such as confirmation-dialog layering. Do not add stats data request channels back to Electron IPC.

The stats UI should treat the trends payload as chart-ready data. Presentation-only work in the client is fine (including the title timeline layout), but rebuilding the main trend datasets from raw sessions should stay out of the render path.

For session detail timelines, omitting `limit` now means "return the full retained session telemetry/history". Explicit `limit` remains available for bounded callers, but the default stats UI path should not trim long sessions to the newest 200 samples.