fix(tokenizer): keep frequency rank for lexicalized kana expressions (#156)

2026-07-27 16:49:51 -07:00 · 2026-07-10 00:54:17 -07:00
parent 7b1a005a65
commit 8712780d08
4 changed files with 41 additions and 3 deletions
@@ -0,0 +1,4 @@
+type: fixed
+area: overlay
+
+- Fixed lexicalized kana expressions such as `かといって` losing frequency annotations when their merged MeCab parts included particles.