sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 16:49:51 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-09 00:28:38 -07:00
committed by GitHub
parent ae40934d3a
commit 6c251502b3
13 changed files with 138 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/unparsed-token-hover-lookup.md
+1 -1
View File
@@ -1,4 +1,4 @@
type: fixed
area: overlay
- Fixed subtitle text that Yomitan's parser cannot match (e.g. the truncated volitional in とこ戻ろ…) being rendered as plain, non-interactive text: it was invisible to hover/lookup and excluded from the n+1 word count, which could wrongly mark a sentence as n+1. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation; bracketed SFX/speaker captions and punctuation-only runs are still skipped.
- Fixed subtitle text that Yomitan's parser cannot match (truncated inflections like the volitional in とこ戻ろ…, elongation runs like ぅ～/ぉ〜) being rendered as plain, non-interactive text with no hover lookup. Unparsed runs are now kept as hoverable tokens matching Yomitan's own segmentation, while being fully ignored by frequency/JLPT highlighting, the N+1 candidate math, and vocabulary stats. Bracketed SFX/speaker captions and punctuation-only runs are still skipped.