sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-04-28 04:19:27 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): preserve annotation and enrichment behavior

Browse Source
This commit is contained in:
Kyle Yasuda
2026-04-26 17:57:39 -07:00
parent 96894ff85c
commit b10a7b3e98
6 changed files with 55 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
changes/305-tokenizer-word-class-pos-filtering.md Normal file
View File

@@ -0,0 +1,6 @@
type: fixed
area: tokenizer
- Use Yomitan `wordClasses` metadata for subtitle POS filtering.
- Backfill blank MeCab POS detail fields during parser enrichment.
- Keep subtitle annotation metadata stripped from token results.