sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-03-02 18:22:42 -08:00
Code Issues Packages Projects Releases Wiki Activity

fix: make tokenization warmup one-shot

Browse Source
This commit is contained in:
Kyle Yasuda
2026-03-02 01:33:09 -08:00
parent 9a91951656
commit 7161fc3513
3 changed files with 150 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
backlog/tasks/task-81 - Tokenization-performance-disable-yomitan-mecab-and-persistent-local-mecab.md
View File

@@ -4,7 +4,7 @@ title: 'Tokenization performance: disable Yomitan MeCab parser, gate local MeCab
status: Done
assignee: []
created_date: '2026-03-02 07:44'
updated_date: '2026-03-02 07:46'
updated_date: '2026-03-02 09:20'
labels: []
dependencies: []
priority: high
@@ -43,6 +43,8 @@ Implemented tokenizer latency optimizations:
- added annotation-aware MeCab initialization gating in runtime warmup flow;
- added persistent local MeCab process (default idle shutdown: 30s) with queued requests, retry-on-process-end, idle auto-shutdown, and automatic restart on new work;
- added regression tests for Yomitan parse flag, MeCab warmup gating, and persistent/idle lifecycle behavior;
- fixed tokenization warmup gate so first-use warmup completion is sticky (`tokenizationWarmupCompleted`) and sequential `tokenizeSubtitle` calls no longer re-run Yomitan/dictionary warmup path;
- added regression coverage in `src/main/runtime/composers/mpv-runtime-composer.test.ts` for sequential tokenize calls (`warmup` side effects run once);
- validated with targeted tests and `tsc --noEmit`.
<!-- SECTION:FINAL_SUMMARY:END -->