sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-08 01:08:53 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings (#147)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-07 23:57:47 -07:00
committed by GitHub
parent 0e254cbbef
commit 187f68e5b6
5 changed files with 87 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/particle-known-word-highlight.md
+5
View File
@@ -0,0 +1,5 @@
type: fixed
area: overlay
- Fixed single-kana grammar tokens counting as known words by borrowing the reading of an unrelated Anki note (よ in 全然いいよ matched a card read よ such as 夜, standalone え matched 絵), which painted them with the known-word highlight. Reading-only known-word matching now requires at least two kana; single-kana cards still match by their word field, and tokens genuinely present in the known-words cache (e.g. です) keep their highlight.
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.