sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-29 01:01:34 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-09 00:28:38 -07:00
committed by GitHub
parent ae40934d3a
commit 6c251502b3
13 changed files with 138 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/particle-known-word-highlight.md
+1 -2
View File
@@ -1,5 +1,4 @@
type: fixed
type: changed
area: overlay
- Fixed single-kana grammar tokens counting as known words by borrowing the reading of an unrelated Anki note (よ in 全然いいよ matched a card read よ such as 夜, standalone え matched 絵), which painted them with the known-word highlight. Reading-only known-word matching now requires at least two kana; single-kana cards still match by their word field, and tokens genuinely present in the known-words cache (e.g. です) keep their highlight.
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.