SubMiner/changes/particle-known-word-highlight.md at 84c75f50aa6b51d5940335ca2b9b6cbc06ab5576 - SubMiner - The Box Gitea

sudacode/SubMiner

mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 04:49:49 -07:00

Files

T

sudacode 6c251502b3 fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153 )

2026-07-09 00:28:38 -07:00

335 B

Raw Blame History

type: changed area: overlay

Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.