feat(subtitles): improve mpv hovered-token highlighting flow

- add subtitleStyle.hoverTokenColor config default + validation

- normalize hover color payloads and propagate configured color to mpv runtime

- refresh invisible overlay tokenization with current subtitle text and tighten hover overlay cleanup hooks

- record TASK-98 and subagent coordination updates
This commit is contained in:
2026-02-21 22:20:56 -08:00
parent 430c4e7120
commit 01f01f18e3
17 changed files with 244 additions and 15 deletions

View File

@@ -0,0 +1,59 @@
---
id: TASK-98
title: Add mpv OSD hovered-token highlighting from Electron token state
status: Done
assignee: []
created_date: '2026-02-21 23:16'
updated_date: '2026-02-21 23:33'
labels:
- mpv
- subtitles
- electron
- ux
dependencies: []
priority: high
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Implement hovered-token highlighting in mpv subtitle OSD by reusing token/hover state that already exists in Electron overlays. When pointer hovers a token in overlay, propagate hovered token identity to mpv subtitle rendering path so the corresponding mpv subtitle token is color-highlighted consistently.
<!-- SECTION:DESCRIPTION:END -->
## Suggestions
<!-- SECTION:SUGGESTIONS:BEGIN -->
- Extend Electron->main IPC hover payload to include stable token index/id usable by mpv renderer.
- Keep mpv highlight style configurable through existing subtitle style settings where possible.
- Reset mpv highlight state on subtitle changes and hover leave events.
<!-- SECTION:SUGGESTIONS:END -->
## Action Steps
<!-- SECTION:PLAN:BEGIN -->
1. Identify current overlay hover event source and token identity payload shape.
2. Extend main runtime message path to carry hovered token to mpv OSD renderer service.
3. Update mpv subtitle ASS rendering to apply hover color override to matching token span.
4. Add regression tests for hover enter/move/leave and subtitle replacement edge cases.
5. Verify behavior in plugin flow (`--texthooker` -> `--start`) and normal overlay mode.
<!-- SECTION:PLAN:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [x] #1 Hovering a token in Electron overlay highlights corresponding token in mpv subtitle OSD.
- [x] #2 Hover leave (or subtitle switch) clears mpv hovered-token highlight reliably.
- [x] #3 No regressions in existing token color annotations (known/N+1/frequency/JLPT).
- [x] #4 Automated tests cover hover payload propagation + mpv rendering highlight behavior.
<!-- AC:END -->
## Definition of Done
<!-- DOD:BEGIN -->
- [x] #1 Relevant unit/integration tests pass
- [x] #2 Docs/config notes updated for any new setting or payload contract
<!-- DOD:END -->
## Final Summary
<!-- SECTION:FINAL_SUMMARY:BEGIN -->
Implemented invisible-overlay-only hovered token highlighting in mpv using a dedicated `osd-overlay` ASS layer. Renderer now tags token spans with stable token indices and reports hover index changes (`subtitle-token-hover:set`) through preload IPC. Main caches latest tokenized subtitle payload, tracks hovered token index, and pushes/clears ASS highlight overlay via new runtime helper `src/main/runtime/mpv-hover-highlight.ts`. Added regression tests for ASS/command generation and IPC dependency wiring. Validation: `bun run build`, `node --test dist/main/runtime/mpv-hover-highlight.test.js dist/core/services/ipc.test.js`, `node --test dist/renderer/subtitle-render.test.js`.
<!-- SECTION:FINAL_SUMMARY:END -->

View File

@@ -0,0 +1,45 @@
# Agent: `codex-mpv-hover-token-highlight-20260221T231509Z-n8x2`
- alias: `codex-mpv-hover-token-highlight`
- mission: `Design and implement mpv OSD token hover highlighting fed from electron token hover state`
- status: `completed`
- branch: `main`
- started_at: `2026-02-21T23:15:09Z`
- heartbeat_minutes: `5`
## Current Work (newest first)
- [2026-02-21T23:33:04Z] handoff: implemented invisible-overlay-only hovered-token mpv OSD highlighting path with renderer IPC -> main -> mpv `osd-overlay`; build + targeted tests green.
- [2026-02-21T23:28:10Z] test: `bun run build`; `node --test dist/main/runtime/mpv-hover-highlight.test.js dist/core/services/ipc.test.js`; `node --test dist/renderer/subtitle-render.test.js`.
- [2026-02-21T23:24:20Z] edit: added `mpv-hover-highlight` runtime helper + tests and wired app state + IPC hover reporting + renderer token index datasets/hover reporting.
- [2026-02-21T23:18:40Z] progress: reviewed `Yomipv` selector hover render; feasible model is ASS re-render with per-token highlight and hitbox mapping.
- [2026-02-21T23:17:50Z] progress: created backlog `TASK-98`; traced renderer token DOM + IPC handlers + mpv protocol events; no existing hover-token IPC path.
- [2026-02-21T23:15:09Z] intent: read coordination + skills, create backlog linkage, inspect mpv/electron token highlight pipeline.
## Files Touched
- `backlog/tasks/task-98 - Add-mpv-OSD-hovered-token-highlighting-from-electron-token-state.md`
- `src/main/runtime/mpv-hover-highlight.ts`
- `src/main/runtime/mpv-hover-highlight.test.ts`
- `src/main.ts`
- `src/main/state.ts`
- `src/main/dependencies.ts`
- `src/core/services/ipc.ts`
- `src/core/services/ipc.test.ts`
- `src/preload.ts`
- `src/types.ts`
- `src/renderer/renderer.ts`
- `src/renderer/handlers/mouse.ts`
- `src/renderer/subtitle-render.ts`
- `src/renderer/state.ts`
- `docs/subagents/agents/codex-mpv-hover-token-highlight-20260221T231509Z-n8x2.md`
## Assumptions
- `osd-overlay` command path is available in connected mpv builds used by SubMiner.
## Open Questions / Blockers
- None.
## Next Step
- Wait for user validation in real mpv session; adjust highlight color/style if requested.
- [2026-02-22T00:30:37Z] follow-up fix: invisible renderer path now renders token spans (when tokenized payload exists) instead of plain text only; hover token index reporting now has actual token targets.
- [2026-02-22T00:30:37Z] test: rebuilt and reran `dist/main/runtime/mpv-hover-highlight.test.js`, `dist/core/services/ipc.test.js`, `dist/renderer/subtitle-render.test.js` (pass).
- [2026-02-22T00:32:45Z] follow-up fix: overlay ASS now redraws full subtitle line and recolors hovered token instead of rendering only the hovered glyph; removes separate-character artifact.

View File

@@ -0,0 +1,14 @@
# Agent: `codex-mpv-plugin-hover-20260222T004010Z-b3k9`
- alias: `codex-mpv-plugin-hover`
- mission: `Shift mpv subtitle token highlighting from overlay hack to plugin-driven subtitle color mutation`
- status: `completed`
- branch: `main`
- started_at: `2026-02-22T00:40:10Z`
- heartbeat_minutes: `5`
## Current Work
- [2026-02-22T01:22:00Z] implemented plugin-side hover handler in `plugin/subminer.lua`: parses `subminer-hover-token` payload, aligns tokens via startPos/endPos, renders highlighted subtitle via `mp.set_osd_ass`, and clears on invalid/revision-stale messages.
- [2026-02-22T21:06:00Z] implemented plugin-directed payload path: Electron now sends `script-message-to yomipv yomipv-hover-token` JSON instead of osd-overlay ASS; added subtitle revision guard and hover payload metadata in runtime.
- [2026-02-22T21:06:00Z] added `subtitle/hover_highlight.lua` module in Yomipv and wired init from `main.lua`.
- [2026-02-22T21:06:00Z] added runtime tests for payload command generation and hover-clear behavior (`src/main/runtime/mpv-hover-highlight.test.ts`).

View File

@@ -92,7 +92,7 @@ local state = {
local HOVER_MESSAGE_NAME = "subminer-hover-token"
local HOVER_MESSAGE_NAME_LEGACY = "yomipv-hover-token"
local DEFAULT_HOVER_BASE_COLOR = "FFFFFF"
local DEFAULT_HOVER_COLOR = "E7C06A"
local DEFAULT_HOVER_COLOR = "C6A0F6"
local LOG_LEVEL_PRIORITY = {
debug = 10,
@@ -1232,6 +1232,7 @@ local function on_file_loaded()
end
local function on_shutdown()
clear_hover_overlay()
if (state.overlay_running or state.texthooker_running) and state.binary_available then
subminer_log("info", "lifecycle", "mpv shutting down, stopping SubMiner process")
show_osd("Shutting down...")
@@ -1280,6 +1281,9 @@ local function init()
mp.register_event("file-loaded", clear_hover_overlay)
mp.register_event("end-file", clear_hover_overlay)
mp.register_event("shutdown", clear_hover_overlay)
mp.add_hook("on_unload", 10, function()
clear_hover_overlay()
end)
mp.observe_property("sub-start", "native", function()
clear_hover_overlay()
end)

View File

@@ -25,6 +25,7 @@ test('loads defaults when config is missing', () => {
assert.equal(config.jellyfin.remoteControlDeviceName, 'SubMiner');
assert.equal(config.subtitleStyle.backgroundColor, 'rgb(30, 32, 48, 0.88)');
assert.equal(config.subtitleStyle.preserveLineBreaks, false);
assert.equal(config.subtitleStyle.hoverTokenColor, '#c6a0f6');
assert.equal(config.immersionTracking.enabled, true);
assert.equal(config.immersionTracking.dbPath, '');
assert.equal(config.immersionTracking.batchSize, 25);
@@ -95,6 +96,44 @@ test('parses subtitleStyle.preserveLineBreaks and warns on invalid values', () =
);
});
test('parses subtitleStyle.hoverTokenColor and warns on invalid values', () => {
const validDir = makeTempDir();
fs.writeFileSync(
path.join(validDir, 'config.jsonc'),
`{
"subtitleStyle": {
"hoverTokenColor": "#c6a0f6"
}
}`,
'utf-8',
);
const validService = new ConfigService(validDir);
assert.equal(validService.getConfig().subtitleStyle.hoverTokenColor, '#c6a0f6');
const invalidDir = makeTempDir();
fs.writeFileSync(
path.join(invalidDir, 'config.jsonc'),
`{
"subtitleStyle": {
"hoverTokenColor": "purple"
}
}`,
'utf-8',
);
const invalidService = new ConfigService(invalidDir);
assert.equal(
invalidService.getConfig().subtitleStyle.hoverTokenColor,
DEFAULT_CONFIG.subtitleStyle.hoverTokenColor,
);
assert.ok(
invalidService
.getWarnings()
.some((warning) => warning.path === 'subtitleStyle.hoverTokenColor'),
);
});
test('parses anilist.enabled and warns for invalid value', () => {
const dir = makeTempDir();
fs.writeFileSync(

View File

@@ -4,6 +4,7 @@ export const SUBTITLE_DEFAULT_CONFIG: Pick<ResolvedConfig, 'subtitleStyle'> = {
subtitleStyle: {
enableJlpt: false,
preserveLineBreaks: false,
hoverTokenColor: '#c6a0f6',
fontFamily:
'M PLUS 1, Noto Sans CJK JP Regular, Noto Sans CJK JP, Hiragino Sans, Hiragino Kaku Gothic ProN, Yu Gothic, Arial Unicode MS, Arial, sans-serif',
fontSize: 35,

View File

@@ -21,6 +21,12 @@ export function buildSubtitleConfigOptionRegistry(
'Preserve line breaks in visible overlay subtitle rendering. ' +
'When false, line breaks are flattened to spaces for a single-line flow.',
},
{
path: 'subtitleStyle.hoverTokenColor',
kind: 'string',
defaultValue: defaultConfig.subtitleStyle.hoverTokenColor,
description: 'Hex color used for hovered subtitle token highlight in mpv.',
},
{
path: 'subtitleStyle.frequencyDictionary.enabled',
kind: 'boolean',

View File

@@ -99,6 +99,7 @@ export function applySubtitleDomainConfig(context: ResolveContext): void {
if (isObject(src.subtitleStyle)) {
const fallbackSubtitleStyleEnableJlpt = resolved.subtitleStyle.enableJlpt;
const fallbackSubtitleStylePreserveLineBreaks = resolved.subtitleStyle.preserveLineBreaks;
const fallbackSubtitleStyleHoverTokenColor = resolved.subtitleStyle.hoverTokenColor;
resolved.subtitleStyle = {
...resolved.subtitleStyle,
...(src.subtitleStyle as ResolvedConfig['subtitleStyle']),
@@ -140,6 +141,19 @@ export function applySubtitleDomainConfig(context: ResolveContext): void {
);
}
const hoverTokenColor = asColor((src.subtitleStyle as { hoverTokenColor?: unknown }).hoverTokenColor);
if (hoverTokenColor !== undefined) {
resolved.subtitleStyle.hoverTokenColor = hoverTokenColor;
} else if ((src.subtitleStyle as { hoverTokenColor?: unknown }).hoverTokenColor !== undefined) {
resolved.subtitleStyle.hoverTokenColor = fallbackSubtitleStyleHoverTokenColor;
warn(
'subtitleStyle.hoverTokenColor',
(src.subtitleStyle as { hoverTokenColor?: unknown }).hoverTokenColor,
resolved.subtitleStyle.hoverTokenColor,
'Expected hex color.',
);
}
const frequencyDictionary = isObject(
(src.subtitleStyle as { frequencyDictionary?: unknown }).frequencyDictionary,
)

View File

@@ -99,3 +99,16 @@ test('subtitle processing can refresh current subtitle without text change', asy
{ text: 'same', tokens: [] },
]);
});
test('subtitle processing refresh can use explicit text override', async () => {
const emitted: SubtitleData[] = [];
const controller = createSubtitleProcessingController({
tokenizeSubtitle: async (text) => ({ text, tokens: [] }),
emitSubtitle: (payload) => emitted.push(payload),
});
controller.refreshCurrentSubtitle('initial');
await flushMicrotasks();
assert.deepEqual(emitted, [{ text: 'initial', tokens: [] }]);
});

View File

@@ -9,7 +9,7 @@ export interface SubtitleProcessingControllerDeps {
export interface SubtitleProcessingController {
onSubtitleChange: (text: string) => void;
refreshCurrentSubtitle: () => void;
refreshCurrentSubtitle: (textOverride?: string) => void;
}
export function createSubtitleProcessingController(
@@ -87,7 +87,10 @@ export function createSubtitleProcessingController(
latestText = text;
processLatest();
},
refreshCurrentSubtitle: () => {
refreshCurrentSubtitle: (textOverride?: string) => {
if (typeof textOverride === 'string') {
latestText = textOverride;
}
if (!latestText.trim()) {
return;
}

View File

@@ -654,6 +654,7 @@ const applyHoveredTokenOverlay = createApplyHoveredTokenOverlayHandler({
getCurrentSubtitleData: () => appState.currentSubtitleData,
getHoveredTokenIndex: () => appState.hoveredSubtitleTokenIndex,
getHoveredSubtitleRevision: () => appState.hoveredSubtitleRevision,
getHoverTokenColor: () => getResolvedConfig().subtitleStyle.hoverTokenColor ?? null,
});
const buildImmersionMediaRuntimeMainDepsHandler = createBuildImmersionMediaRuntimeMainDepsHandler({
getResolvedConfig: () => getResolvedConfig(),
@@ -2973,7 +2974,7 @@ function setVisibleOverlayVisible(visible: boolean): void {
function setInvisibleOverlayVisible(visible: boolean): void {
setInvisibleOverlayVisibleHandler(visible);
if (visible) {
subtitleProcessingController.refreshCurrentSubtitle();
subtitleProcessingController.refreshCurrentSubtitle(appState.currentSubText);
}
}

View File

@@ -74,7 +74,18 @@ test('buildHoveredTokenPayload normalizes metadata and strips empty tokens', ()
assert.equal(payload.tokens[0]?.text, '昨日');
assert.equal(payload.tokens[0]?.index, 0);
assert.equal(payload.tokens[1]?.index, 1);
assert.equal(payload.colors.hover, 'E7C06A');
assert.equal(payload.colors.hover, 'C6A0F6');
});
test('buildHoveredTokenPayload normalizes hover color override', () => {
const payload = buildHoveredTokenPayload({
subtitle: SUBTITLE,
hoveredTokenIndex: 1,
revision: 7,
hoverColor: '#c6a0f6',
});
assert.equal(payload.colors.hover, 'C6A0F6');
});
test('buildHoveredTokenMessageCommand sends script-message-to subminer payload', () => {
@@ -111,6 +122,7 @@ test('createApplyHoveredTokenOverlayHandler sends clear payload when hovered tok
getCurrentSubtitleData: () => SUBTITLE,
getHoveredTokenIndex: () => null,
getHoveredSubtitleRevision: () => 3,
getHoverTokenColor: () => null,
});
apply();
@@ -134,6 +146,7 @@ test('createApplyHoveredTokenOverlayHandler sends highlight payload when hover i
getCurrentSubtitleData: () => SUBTITLE,
getHoveredTokenIndex: () => 0,
getHoveredSubtitleRevision: () => 3,
getHoverTokenColor: () => '#c6a0f6',
});
apply();
@@ -142,6 +155,7 @@ test('createApplyHoveredTokenOverlayHandler sends highlight payload when hover i
assert.equal(parsed.hoveredTokenIndex, 0);
assert.equal(parsed.subtitle, '昨日は雨だった。');
assert.equal(parsed.tokens.length, 4);
assert.equal(parsed.colors.hover, 'C6A0F6');
assert.equal(commands[0]?.[0], 'script-message-to');
assert.equal(commands[0]?.[1], HOVER_SCRIPT_NAME);
});

View File

@@ -3,7 +3,7 @@ import type { SubtitleData } from '../../types';
export const HOVER_SCRIPT_NAME = 'subminer';
export const HOVER_TOKEN_MESSAGE = 'subminer-hover-token';
const DEFAULT_HOVER_TOKEN_COLOR = 'E7C06A';
const DEFAULT_HOVER_TOKEN_COLOR = 'C6A0F6';
const DEFAULT_TOKEN_COLOR = 'FFFFFF';
export type HoverPayloadToken = {
@@ -28,8 +28,17 @@ type HoverTokenInput = {
subtitle: SubtitleData | null;
hoveredTokenIndex: number | null;
revision: number;
hoverColor?: string | null;
};
function normalizeHexColor(color: string | null | undefined, fallback: string): string {
if (typeof color !== 'string') {
return fallback;
}
const normalized = color.trim().replace(/^#/, '').toUpperCase();
return /^[0-9A-F]{6}$/.test(normalized) ? normalized : fallback;
}
function sanitizeSubtitleText(text: string): string {
return text
.replace(/\\N/g, '\n')
@@ -51,7 +60,7 @@ function hasHoveredToken(subtitle: SubtitleData | null, hoveredTokenIndex: numbe
}
export function buildHoveredTokenPayload(input: HoverTokenInput): HoverTokenPayload {
const { subtitle, hoveredTokenIndex, revision } = input;
const { subtitle, hoveredTokenIndex, revision, hoverColor } = input;
const tokens: HoverPayloadToken[] = [];
@@ -83,7 +92,7 @@ export function buildHoveredTokenPayload(input: HoverTokenInput): HoverTokenPayl
tokens,
colors: {
base: DEFAULT_TOKEN_COLOR,
hover: DEFAULT_HOVER_TOKEN_COLOR,
hover: normalizeHexColor(hoverColor, DEFAULT_HOVER_TOKEN_COLOR),
},
};
}
@@ -105,6 +114,7 @@ export function createApplyHoveredTokenOverlayHandler(deps: {
getCurrentSubtitleData: () => SubtitleData | null;
getHoveredTokenIndex: () => number | null;
getHoveredSubtitleRevision: () => number;
getHoverTokenColor: () => string | null;
}) {
return (): void => {
const mpvClient = deps.getMpvClient();
@@ -115,10 +125,12 @@ export function createApplyHoveredTokenOverlayHandler(deps: {
const subtitle = deps.getCurrentSubtitleData();
const hoveredTokenIndex = deps.getHoveredTokenIndex();
const revision = deps.getHoveredSubtitleRevision();
const hoverColor = deps.getHoverTokenColor();
const payload = buildHoveredTokenPayload({
subtitle: subtitle && hasHoveredToken(subtitle, hoveredTokenIndex) ? subtitle : null,
hoveredTokenIndex: hoveredTokenIndex,
revision,
hoverColor,
});
mpvClient.send({ command: buildHoveredTokenMessageCommand(payload) });

View File

@@ -54,13 +54,13 @@ export function applyVerticalPosition(
bottomInset: number;
marginY: number;
effectiveFontSize: number;
borderPx: number;
shadowPx: number;
vAlign: 0 | 1 | 2;
},
): void {
const lineCount = Math.max(1, ctx.state.currentInvisibleSubtitleLineCount);
const multiline = lineCount > 1;
const baselineCompensationFactor = lineCount >= 3 ? 0.46 : multiline ? 0.58 : 0.7;
const baselineCompensationPx = Math.max(0, params.effectiveFontSize * baselineCompensationFactor);
const usableHeight = Math.max(1, params.renderAreaHeight - params.topInset - params.bottomInset);
const baselineCompensationPx = Math.max(0, (params.borderPx + params.shadowPx) * 5);
if (params.vAlign === 2) {
ctx.dom.subtitleContainer.style.top = `${Math.max(
@@ -78,9 +78,9 @@ export function applyVerticalPosition(
return;
}
const subPosMargin = ((100 - params.metrics.subPos) / 100) * params.renderAreaHeight;
const effectiveMargin = Math.max(params.marginY, subPosMargin);
const bottomPx = Math.max(0, params.bottomInset + effectiveMargin + baselineCompensationPx);
const anchorY =
params.topInset + (usableHeight * params.metrics.subPos) / 100 - params.marginY + baselineCompensationPx;
const bottomPx = Math.max(0, params.renderAreaHeight - anchorY);
ctx.dom.subtitleContainer.style.top = '';
ctx.dom.subtitleContainer.style.bottom = `${bottomPx}px`;

View File

@@ -33,6 +33,7 @@ export function createMpvSubtitleLayoutController(
applySubtitleFontSize(geometry.effectiveFontSize);
const effectiveBorderSize = metrics.subBorderSize * geometry.pxPerScaledPixel;
const effectiveShadowOffset = metrics.subShadowOffset * geometry.pxPerScaledPixel;
document.documentElement.style.setProperty('--sub-border-size', `${effectiveBorderSize}px`);
@@ -53,6 +54,8 @@ export function createMpvSubtitleLayoutController(
bottomInset: geometry.bottomInset,
marginY: geometry.marginY,
effectiveFontSize: geometry.effectiveFontSize,
borderPx: effectiveBorderSize,
shadowPx: effectiveShadowOffset,
vAlign: alignment.vAlign,
});

View File

@@ -271,6 +271,7 @@ export interface AnkiConnectConfig {
export interface SubtitleStyleConfig {
enableJlpt?: boolean;
preserveLineBreaks?: boolean;
hoverTokenColor?: string;
fontFamily?: string;
fontSize?: number;
fontColor?: string;