 sudacode
|
a2e49b369b
|
fix(tokenizer): merge scanner metadata per token instead of all-or-nothi
- Replace hasSameTokenSpans + full-discard with mergeScannerTokensIntoParseTokens
- Grafts isNameMatch/frequencyRank/etc onto matching parse spans; filler chunks degrade only themselves
- Fixes name annotations dropping for entire subtitle lines containing unmatched interjections
|
2026-07-08 22:26:34 -07:00 |
|