SubMiner/changes/name-match-span-merge.md at 84c75f50aa6b51d5940335ca2b9b6cbc06ab5576

sudacode/SubMiner

Fork 0

mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 04:49:49 -07:00

Files

T

sudacode a2e49b369b

fix(tokenizer): merge scanner metadata per token instead of all-or-nothi

- Replace hasSameTokenSpans + full-discard with mergeScannerTokensIntoParseTokens
- Grafts isNameMatch/frequencyRank/etc onto matching parse spans; filler chunks degrade only themselves
- Fixes name annotations dropping for entire subtitle lines containing unmatched interjections

2026-07-08 22:26:34 -07:00

341 B

Raw Blame History

type: fixed area: overlay

Fixed character-name annotations dropping for an entire subtitle line when it contained any chunk the dictionary scanner could not match (e.g. an interjection like やほっ before a name): scanner metadata is now merged per token into the parseText segmentation instead of being discarded on any span mismatch.

341 B Raw Blame History

341 B

Raw Blame History