sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-29 07:21:33 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): exclude unparsed-run tokens from annotations and N+1 (#153)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-09 00:28:38 -07:00
committed by GitHub
parent ae40934d3a
commit 6c251502b3
13 changed files with 138 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/known-word-reading-aware-matching.md
+4 -2
View File
@@ -1,4 +1,6 @@
type: fixed
type: changed
area: overlay
- Fixed words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" matching a known 床/ゆか "floor" card). The known-word cache now stores each card's word together with its reading and only matches when the token's reading agrees; cards without a reading field keep matching in any reading as before.
- New reading-aware subtitle parsing and known-word matching: the known-word cache now stores each Anki card's word together with its reading (cache format v3), and a token only gets the known-word highlight when its parsed reading agrees with the card. The cache and the stats server upgrade automatically.
- Fixes words being highlighted green as known when a same-spelled Anki card taught a different reading (e.g. とこ parsed as 床 "bed" no longer matches a known 床/ゆか "floor" card). Cards without a reading field keep matching in any reading as before.
- Single-kana grammar tokens (よ in 全然いいよ, standalone え) no longer borrow the reading of an unrelated card (such as 夜 or 絵) and get painted as known: reading-only matching requires at least two kana, while single-kana cards still match by their word field.