sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 04:49:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): keep frequency rank for lexicalized kana expressions (#156)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-10 00:54:17 -07:00
committed by GitHub
parent 7b1a005a65
commit 8712780d08
4 changed files with 41 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs-site/subtitle-annotations.md
+1 -1
View File
@@ -90,7 +90,7 @@ SubMiner looks up each token's `frequencyRank` from `term_meta_bank_*.json` file
When `sourcePath` is omitted, SubMiner searches default install/runtime locations for `frequency-dictionary` directories automatically.
::: info
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist.
Frequency highlighting skips tokens that look like non-lexical noise (kana reduplication, short kana endings like `っ`), even when dictionary ranks exist. For merged kana tokens, SubMiner keeps a rank when the dictionary headword reading covers the full token (for example, `かと言って` / `かといって`), while grammar wrapped around a shorter lemma remains unannotated.
:::
::: info