refactor(tokenizer): extract subtitle annotation filter into rule table (#162)

2026-07-27 16:49:51 -07:00 · 2026-07-12 23:54:13 -07:00
parent 21a06c12fb
commit 0ab840b362
6 changed files with 706 additions and 415 deletions
@@ -0,0 +1,4 @@
+type: fixed
+area: overlay
+
+- Applied configured primary POS exclusions consistently to merged trailing quote-particle tokens, preserved annotations for supplementary-plane kanji, and stopped treating katakana punctuation as kana-only annotation noise.