fix(subtitles): widen halfwidth katakana where subtitle text enters the app (#295)

* fix(subtitles): widen halfwidth katakana where subtitle text enters the app

Broadcast-derived Japanese subs write katakana halfwidth (アンタ, ダキニ), so
overlay lines mixed squashed and normal-width text. Normalize live mpv
sub-text and parsed cue text to fullwidth so display, lookups, mining,
and stats all see one form. The tokenizer reuses the shared helper.

* fix(subtitles): fold stray voiced marks and widen before cue dedup

- A lone halfwidth ゙ after fullwidth kana (ガ) now composes to ガ
  instead of leaving a combining mark that broke name matching.
- Parsed cue text is widened before duplicate merging so ガク and ガク
  on the same span collapse into one sidebar row.
- The YouTube track-load refresh widens the sub-text it reads directly.

* fix(stats): widen halfwidth katakana in stored lines and mined cards (#296)

Lines recorded before ingest widening still hold halfwidth katakana.
Occurrence and sentence-search queries widen line text on read, and the
mine-card route widens the sentence and word it receives, so the
dashboard and cards mined from history match newer rows. Word identity
columns are left as stored since the dashboard queries by them.

* fix(stats): match katakana in either width in sentence search

- Add narrowFullwidthKana, derived from widenHalfwidthKana so the two stay in sync
- Match subtitle text against the original, widened, and narrowed forms of each term so a search finds older lines stored as halfwidth
This commit is contained in:
sudacode authored and GitHub committed 2026-10-13 22:26:01 -07:00
1 parent bf4e5bb609
commit 01613f1be5
16 files changed
+285 -26

No files matched your search

+24
View File
@@ -0,0 +1,24 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { narrowFullwidthKana, widenHalfwidthKana } from './halfwidth-kana';
test('widenHalfwidthKana widens halfwidth katakana and composes voiced marks', () => {
assert.equal(
widenHalfwidthKana('(トーカ)ダキニお姉ちゃん アンタがアータルたん。'),
'(トーカ)ダキニお姉ちゃん アンタがアータルたん。',
);
});
test('widenHalfwidthKana leaves other compatibility characters alone', () => {
assert.equal(widenHalfwidthKana('見て見てっ!①㍑AB'), '見て見てっ!①㍑AB');
});
test('widenHalfwidthKana folds a lone halfwidth voiced mark into the kana before it', () => {
assert.equal(widenHalfwidthKana('山ガク パン'), '山ガク パン');
assert.equal(widenHalfwidthKana('ア゙'), 'ア\u3099');
});
test('narrowFullwidthKana inverts widening and leaves other text alone', () => {
assert.equal(narrowFullwidthKana('ダキニお姉ちゃん アンタ。ヴ'), 'ダキニお姉ちゃん アンタ。ヴ');
assert.equal(narrowFullwidthKana('漢字ひらがな!'), '漢字ひらがな!');
});
+49
View File
@@ -0,0 +1,49 @@
// No `u` flag: the range is entirely BMP so it changes nothing here, and
// Bun's unicode-mode matcher mis-handles this class next to certain ligatures.
// The optional leading character is captured so a run opening with a voiced
// mark (ガ) can fold the mark into the fullwidth kana before it.
const HALFWIDTH_KANA_RUN = /([^。-゚]?)([。-゚]+)/g;
const LEADING_COMBINING_VOICED_MARK = /^[゙゚]/;
/**
* Rewrites halfwidth katakana and its punctuation (アンタ, ダキニ, 。「」、・) to the fullwidth
* forms. Broadcast-derived subtitles write katakana halfwidth to save space; subtitle
* text is normalized with this where it enters the app (live mpv text, parsed cues) so
* display, lookups, mining, and stats all see one form.
*
* NFKC runs over the halfwidth runs only, never the whole string: it composes the voiced
* pairs (カ + ゙) into single characters, but over everything it would also rewrite
* unrelated text (① → 1, ! → !, ㍑ → リットル).
*/
export function widenHalfwidthKana(text: string): string {
return text.replace(HALFWIDTH_KANA_RUN, (_match, previous: string, run: string) => {
const widened = run.normalize('NFKC');
if (!previous || !LEADING_COMBINING_VOICED_MARK.test(widened)) {
return previous + widened;
}
// A lone ゙ widens to a combining mark; NFC composes it with the kana before it
// (カ + ゛ → ガ) and leaves pairs that have no composed form untouched.
return (previous + widened[0]).normalize('NFC') + widened.slice(1);
});
}
// Inverse of widenHalfwidthKana per character, derived from it so the two stay in sync.
// Voiced pairs (ガ) are included because they widen to a single character (ガ).
const HALFWIDTH_BY_FULLWIDTH = new Map<string, string>();
for (let code = 0xff61; code <= 0xff9f; code += 1) {
const halfwidth = String.fromCharCode(code);
for (const form of [halfwidth, `${halfwidth}゙`, `${halfwidth}゚`]) {
const widened = widenHalfwidthKana(form);
if (widened.length === 1 && !HALFWIDTH_BY_FULLWIDTH.has(widened)) {
HALFWIDTH_BY_FULLWIDTH.set(widened, form);
}
}
}
/**
* Rewrites fullwidth katakana and punctuation to halfwidth forms (アンタ → アンタ). Used to
* search text stored before subtitle ingest widened it; display paths should widen instead.
*/
export function narrowFullwidthKana(text: string): string {
return Array.from(text, (char) => HALFWIDTH_BY_FULLWIDTH.get(char) ?? char).join('');
}