sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-07-27 16:49:51 -07:00
Code Issues Packages Projects Releases Wiki Activity

test(tokenizer): add golden-file regression corpus with 11 fixtures (#157)

Browse Source
This commit is contained in:
Kyle Yasuda
2026-07-11 00:50:03 -07:00
committed by GitHub
parent 321461c50f
commit 8acc78cc1c
23 changed files with 117535 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changes/golden-tokenizer-corpus.md
+7
View File
@@ -0,0 +1,7 @@
type: internal
area: tokenizer
- Added a golden-file regression corpus for the tokenizer/annotation pipeline: recorded Yomitan backend responses and MeCab tokens replay through the real tokenizeSubtitle pipeline in bun tests without Electron or dictionaries.
- Added `record-tokenizer-fixture:electron` script to capture new fixtures from a live Yomitan/MeCab session, with flags for known words, JLPT levels, and annotation toggles.
- Seeded eleven fixtures covering the #147–#156 regression classes (grammar-helper suppression, lexical くれる, kanji non-independent nouns, N+1 targeting, reading collisions, unparsed runs, ordinal/honorific prefixes).
- Added `compare-yomitan-api:electron` script that diffs SubMiner tokenization against a stock Yomitan instance via the yomitan-api bridge (segmentation, readings, headword forms).