feat(tokenizer): use Yomitan word classes for subtitle POS filtering (#57)

* feat(tokenizer): use Yomitan word classes for subtitle POS filtering

- Carry matched headword wordClasses from termsFind into YomitanScanToken
- Map recognized Yomitan wordClasses to SubMiner coarse POS before annotation
- MeCab enrichment now fills only missing POS fields, preserving existing coarse pos1
- Exclude standalone grammar particles, して helper fragments, and single-kana surfaces from annotations
- Respect source-text punctuation gaps when counting N+1 sentence words
- Preserve known-word highlight on excluded kanji-containing tokens
- Add backlog tasks 304 (N+1 boundary bug) and 305 (wordClasses POS, done)

* fix(tokenizer): preserve annotation and enrichment behavior

* fix: restore jlpt subtitle underlines

* fix: exclude kana-only n+1 targets

* fix: refresh overlay on Hyprland fullscreen

* fix: address fullscreen and n-plus-one review notes

* fix: address CodeRabbit review comments

* fix: accept modified digits for multi-line sentence mining

* Cancel pending Linux MPV fullscreen overlay refresh bursts

- return a cancel handle from the Linux refresh burst scheduler
- clear pending refresh bursts when overlays hide or windows close
- tighten the burst test polling to wait for the async refresh

* fix: suppress N+1 for kana-only candidates and fix minSentenceWords coun

- Treat kana-only tokens with surrounding subtitle punctuation (…, ―, etc.) as kana-only so they are not promoted to N+1 targets
- Exclude unknown tokens filtered from N+1 targeting from the minSentenceWords count so filtered kana-only unknowns cannot satisfy sentence length threshold
- Add regression tests for kana-only candidate suppression and filtered-unknown padding cases

* Suppress subtitle annotations for grammar fragments

- Hide annotation metadata for auxiliary inflection and ja-nai endings
- Preserve lexical `くれる` forms and add regression coverage

* Fix kana-only N+1 tokenizer regression test

- Use a pure-kana fixture for the subtitle token N+1 case
- Update task notes for the latest CodeRabbit follow-up

* Fix managed playback exit and tokenizer grammar splits

- Ignore background stats daemons during regular app startup
- Split standalone grammar endings before applying annotations
- Clear helper-span annotations for auxiliary-only tokens

* fix: refresh current subtitle after known-word mining

* fix: suppress sigh interjection annotations

* fix: preserve jlpt underline color after lookup

* Replace grammar-ending permutations with shared matcher; preserve word a

- Extract `grammar-ending.ts` with `isStandaloneGrammarEndingText` / `isSubtitleGrammarEndingText` pattern matchers
- Replace `STANDALONE_GRAMMAR_ENDINGS` set in parser-selection-stage with shared matcher
- Replace generated phrase sets in subtitle-annotation-filter with shared matcher
- Remove stale duplicate subtitle-exclusion constants and helpers from annotation-stage
- Manual clipboard card updates now write only to the sentence audio field, leaving word/expression audio untouched

* fix: CI changelog, annotation options threading, and Jellyfin quit

- Add `type: fixed` / `area:` frontmatter to `changes/319` to pass `changelog:lint`
- Thread `TokenizerAnnotationOptions` through `stripSubtitleAnnotationMetadata` so `sourceText` is honored
- Include `jellyfinPlay` in `shouldQuitOnDisconnectWhenOverlayRuntimeInitialized` predicate
- Make mouse test `elementFromPoint` stubs coordinate-sensitive
- Make Lua test `.tmp` mkdir portable on Windows

* Preserve overlay across macOS flaps and mpv playlist changes

- keep visible overlays alive during transient macOS tracker loss
- reuse the running mpv overlay path on playlist navigation
- update regression coverage and changelog fragments

* fix: restore stats daemon deferral

* fix: keep subtitle prefetch alive after cache hits

* Fix JLPT underline color drift and AniList skipped-threshold sync

- Replace JLPT `text-decoration` underlines with `border-bottom` so Chromium selection/hover cannot repaint them to another annotation's color
- Lock JLPT underline color for combined annotation selectors (known, n+1, frequency) and character hover/selection states
- Trigger AniList post-watch check on every mpv time-position update to catch skipped completion thresholds
- Fall back to filename-parser season/episode when guessit omits them

* fix: address coderabbit feedback

* fix: sync AniList after seeked completion

* fix: preserve ordinal frequency annotations

* fix: preserve known highlighting for filtered tokens

* fix: address PR #57 CodeRabbit feedback

- Acquire AniList post-watch in-flight lock before async gating to prevent duplicate writes
- Isolate manual watched mark result from AniList post-watch callback failures
- Report known-word cache clears as mutations during immediate append when state existed
- Add regression tests for each fix

* fix: stop AniList setup reopening on Linux when keyring token exists

- Gate setup success on token persistence: `saveToken` now returns `boolean`; on failure, keeps the setup window open instead of reporting success
- Config reload passes `allowSetupPrompt: false` so playback reloads don't re-open the setup window
- Add regression test for persistence-failure path

* fix: suppress known highlights for subtitle particles

* fix: retry transient AniList safeStorage failures

* fix: hide overlay focus ring

* fix: align Hyprland fullscreen overlays

* fix: restore subtitle playback keybindings

* fix: align Hyprland overlay windows to mpv and stop pinning them

- Force-apply exact Hyprland move/resize/setprop dispatches when bounds are provided
- Stop pinning overlay windows; toggle pin off when Hyprland reports pinned=true
- Compensate stats overlay outer placement for Electron/Wayland content insets
- Make stats overlay window and page opaque so mpv cannot show through transparent insets
- Constrain stats app to h-screen with internal scroll so content covers mpv from y=0
- Lock overlay/stats window titles against page-title-updated events
- Add regression coverage for placement dispatches, inset compensation, and CSS overlay mode

* fix: retain frequency rank for honorific prefix-noun tokens

- Add `shouldAllowHonorificPrefixNounFrequency` to exempt お/ご/御 + noun merged tokens from frequency exclusion
- Add regression test for `ご機嫌` asserting rank 5484 is preserved after MeCab enrichment and annotation
- Close TASK-341

* fix: map openCharacterDictionary session action to --open-character-dict

- Add missing Lua CLI dispatch entry for openCharacterDictionary
- Add regression test for Alt+Meta+A binding and CLI flag forwarding

* fix: keep macOS overlay interactive while mpv remains active

- Overlay no longer hides or becomes click-through during tracker refreshes when mpv is the focused window
- Preserve already-visible overlay when tracker is temporarily not ready but mpv target signal is active
- Add regression tests for active-mpv tracker refresh and transient tracker-not-ready paths

* fix: address coderabbit subtitle follow-ups

* fix: resolve media detail from sessions when lifetime summary is absent

- Change `getMediaDetail` JOIN to LEFT JOIN on `imm_lifetime_media` and fall back to aggregated session metrics when no lifetime row exists
- Add filter `AND (lm.video_id IS NOT NULL OR s.session_id IS NOT NULL)` to keep results valid
- Add regression test covering the session-visible / media-detail-missing mismatch

* fix: address PR-57 CodeRabbit findings and CI failures

- use filtered word counts in media detail session token aggregation
- cancel fullscreen refresh burst on exit via updateLinuxMpvFullscreenOverlayRefreshBurst
- guard Hyprland JSON.parse in try/catch; exclude windowtitle from geometry events
- narrow focus suppression from :focus to :focus-visible
- apply JLPT lock selectors to word-name-match tokens (N1–N5)

* fix: macOS overlay z-order and Yomitan compound token known highlighting

- Release always-on-top when tracked mpv loses foreground on macOS
- Skip visible overlay blur restacking on macOS to avoid covering unrelated windows
- Prefer Yomitan internal parse tokens over fragmented scanner output for known-word decisions
- Add regression tests for both behaviors

* fix: macOS visible-overlay blur no longer invokes Windows-only blur call

- Split win32/darwin branches in handleOverlayWindowBlurred so darwin visible blur returns early without calling onWindowsVisibleOverlayBlur
- Add regression test asserting Windows callback stays inactive on macOS visible overlay blur
- Close TASK-347
This commit is contained in:
2026-05-12 12:08:09 -07:00
committed by GitHub
parent b68d17614d
commit 430373f010
176 changed files with 8174 additions and 569 deletions
@@ -126,7 +126,7 @@ function createManualUpdateService(overrides: Partial<CardCreationDeps> = {}): {
};
}
test('manual clipboard subtitle update replaces expression and sentence audio even when overwriteAudio is disabled', async () => {
test('manual clipboard subtitle update replaces sentence audio without touching expression audio', async () => {
const { service, updatedFields, mergeCalls, storedMedia } = createManualUpdateService();
await service.updateLastAddedFromClipboard('字幕');
@@ -134,10 +134,44 @@ test('manual clipboard subtitle update replaces expression and sentence audio ev
assert.equal(updatedFields.length, 1);
assert.equal(storedMedia.length, 1);
const audioValue = `[sound:${storedMedia[0]}]`;
assert.equal(updatedFields[0]?.ExpressionAudio, audioValue);
assert.equal(updatedFields[0]?.SentenceAudio, audioValue);
assert.equal('ExpressionAudio' in updatedFields[0]!, false);
assert.deepEqual(
mergeCalls.map((call) => call.overwrite),
[true, true],
[true],
);
});
test('manual clipboard subtitle update skips audio when sentence audio field is missing', async () => {
const { service, updatedFields, mergeCalls, storedMedia } = createManualUpdateService({
client: {
addNote: async () => 0,
addTags: async () => undefined,
notesInfo: async () => [
{
noteId: 42,
fields: {
Expression: { value: '単語' },
Sentence: { value: '' },
ExpressionAudio: { value: '[sound:auto-expression.mp3]' },
},
},
],
updateNoteFields: async (_noteId, fields) => {
updatedFields.push(fields);
},
storeMediaFile: async (filename) => {
storedMedia.push(filename);
},
findNotes: async () => [42],
retrieveMediaFile: async () => '',
},
});
await service.updateLastAddedFromClipboard('字幕');
assert.equal(storedMedia.length, 1);
assert.equal(updatedFields.length, 1);
assert.deepEqual(updatedFields[0], { Sentence: '字幕' });
assert.equal(mergeCalls.length, 0);
});
+15 -19
View File
@@ -218,11 +218,7 @@ export class CardCreationService {
fields,
this.deps.getConfig(),
);
const sentenceAudioField = this.getResolvedSentenceAudioFieldName(noteInfo);
const expressionAudioField = this.deps.resolveConfiguredFieldName(
noteInfo,
this.deps.getConfig().fields?.audio || 'ExpressionAudio',
);
const sentenceAudioField = this.getResolvedSentenceOnlyAudioFieldName(noteInfo);
const sentenceField = this.deps.getEffectiveSentenceCardConfig().sentenceField;
const sentence = blocks.join(' ');
@@ -252,22 +248,15 @@ export class CardCreationService {
if (audioBuffer) {
await this.deps.client.storeMediaFile(audioFilename, audioBuffer);
if (sentenceAudioField || expressionAudioField) {
if (sentenceAudioField) {
const audioValue = `[sound:${audioFilename}]`;
const audioFields = new Set(
[sentenceAudioField, expressionAudioField].filter(
(fieldName): fieldName is string => Boolean(fieldName),
),
const existingAudio = noteInfo.fields[sentenceAudioField]?.value || '';
// Manual clipboard updates intentionally replace old captured sentence audio.
updatedFields[sentenceAudioField] = this.deps.mergeFieldValue(
existingAudio,
audioValue,
true,
);
for (const audioField of audioFields) {
const existingAudio = noteInfo.fields[audioField]?.value || '';
// Manual clipboard updates intentionally replace old captured audio.
updatedFields[audioField] = this.deps.mergeFieldValue(
existingAudio,
audioValue,
true,
);
}
}
miscInfoFilename = audioFilename;
updatePerformed = true;
@@ -732,6 +721,13 @@ export class CardCreationService {
);
}
private getResolvedSentenceOnlyAudioFieldName(noteInfo: CardCreationNoteInfo): string | null {
return this.deps.resolveNoteFieldName(
noteInfo,
this.deps.getEffectiveSentenceCardConfig().audioField || 'SentenceAudio',
);
}
private createPendingNoteInfo(fields: Record<string, string>): CardCreationNoteInfo {
return {
noteId: -1,
@@ -520,6 +520,51 @@ test('KnownWordCacheManager uses the current deck fields for immediate append',
}
});
test('KnownWordCacheManager reports immediate append cache clears as mutations', () => {
const config: AnkiConnectConfig = {
fields: {
word: 'Expression',
},
knownWords: {
highlightEnabled: true,
refreshMinutes: 60,
},
};
const { manager, statePath, cleanup } = createKnownWordCacheHarness(config);
try {
fs.writeFileSync(
statePath,
JSON.stringify({
version: 2,
refreshedAtMs: Date.now(),
scope: '{"refreshMinutes":60,"scope":"is:note","fieldsWord":"Expression"}',
words: ['猫'],
notes: {
'1': ['猫'],
},
}),
'utf-8',
);
manager.startLifecycle();
assert.equal(manager.isKnownWord('猫'), true);
config.fields = { word: 'Word' };
const changed = manager.appendFromNoteInfo({
noteId: 2,
fields: {
Word: { value: '' },
},
});
assert.equal(changed, true);
assert.equal(manager.isKnownWord('猫'), false);
} finally {
manager.stopLifecycle();
cleanup();
}
});
test('KnownWordCacheManager skips immediate append when addMinedWordsImmediately is disabled', () => {
const config: AnkiConnectConfig = {
knownWords: {
+7 -4
View File
@@ -165,13 +165,15 @@ export class KnownWordCacheManager {
}
}
appendFromNoteInfo(noteInfo: KnownWordCacheNoteInfo): void {
appendFromNoteInfo(noteInfo: KnownWordCacheNoteInfo): boolean {
if (!this.isKnownWordCacheEnabled() || !this.shouldAddMinedWordsImmediately()) {
return;
return false;
}
let didMutateCache = false;
const currentStateKey = this.getKnownWordCacheStateKey();
if (this.knownWordsStateKey && this.knownWordsStateKey !== currentStateKey) {
didMutateCache = this.knownWords.size > 0 || this.noteWordsById.size > 0;
this.clearKnownWordCacheState();
}
if (!this.knownWordsStateKey) {
@@ -180,13 +182,13 @@ export class KnownWordCacheManager {
const preferredFields = this.getImmediateAppendFields();
if (!preferredFields) {
return;
return didMutateCache;
}
const nextWords = this.extractNormalizedKnownWordsFromNoteInfo(noteInfo, preferredFields);
const changed = this.replaceNoteSnapshot(noteInfo.noteId, nextWords);
if (!changed) {
return;
return didMutateCache;
}
if (this.knownWordsLastRefreshedAtMs <= 0) {
@@ -199,6 +201,7 @@ export class KnownWordCacheManager {
`wordCount=${nextWords.length}`,
`scope=${getKnownWordCacheScopeForConfig(this.deps.getConfig())}`,
);
return true;
}
clearKnownWordCacheState(): void {