sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 16:49:51 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): keep frequency rank for lexicalized kana expressions (#156)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-10 00:54:17 -07:00
committed by GitHub
parent 7b1a005a65
commit 8712780d08
4 changed files with 41 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/lexicalized-expression-frequency.md
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: overlay
- Fixed lexicalized kana expressions such as `かといって` losing frequency annotations when their merged MeCab parts included particles.