sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 16:49:51 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): greedy name pre-pass to prevent generic matches swallowing character names (#151)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-08 23:53:45 -07:00
committed by GitHub
parent cdb1475a54
commit ae40934d3a
3 changed files with 419 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/greedy-name-parsing-priority.md
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Fixed character names being swallowed by longer generic dictionary matches during subtitle tokenization (e.g. ヨータ in 美姫とヨータ was lost because とヨー normalizes to とよう and matched 渡洋), so the name never tokenized and got no highlight, portrait, or hover lookup. Character-dictionary matches now claim their spans in a greedy pre-pass before the left-to-right scan, and re-segmented name spans override the parse tokenization on merge. The pre-pass runs only when a SubMiner character dictionary is enabled in the active Yomitan profile.