sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-28 04:49:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): merge scanner metadata per token instead of all-or-nothi

Browse Source 
- Replace hasSameTokenSpans + full-discard with mergeScannerTokensIntoParseTokens
- Grafts isNameMatch/frequencyRank/etc onto matching parse spans; filler chunks degrade only themselves
- Fixes name annotations dropping for entire subtitle lines containing unmatched interjections
This commit is contained in:
Kyle Yasuda
2026-07-08 22:26:11 -07:00
parent 7f13aed50a
commit a2e49b369b
3 changed files with 151 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/name-match-span-merge.md
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Fixed character-name annotations dropping for an entire subtitle line when it contained any chunk the dictionary scanner could not match (e.g. an interjection like やほっ before a name): scanner metadata is now merged per token into the parseText segmentation instead of being discarded on any span mismatch.