mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-04 07:21:32 -07:00
sudacode
e7cef039f3
- drop the duplicate parseText full parse per line; the termsFind scanner walk is now authoritative and emits its own unparsed filler runs (parseText kept only as error fallback) - install scan helpers once per parser window (__subminerYomitanScan) instead of re-shipping ~500 lines of script per subtitle line - persist termsFind results across lines in a window-scoped LRU keyed by substring, invalidated via a cache epoch on dictionary/settings changes - skip lookups at punctuation/whitespace positions and cap the shrinking-window retry ladder at 4 lookups per position - build tokenizer runtime deps once (JLPT lookup cache never hit before; mecab availability check ran per line) - stop restarting the prefetch run on every subtitle change; resume prefetch only after the tokenized payload lands, not on provisional raw emits - add per-stage debug timings (scanMs/mecabMs/frequencyMs/annotateMs)
1.8 KiB
1.8 KiB
type: changed area: subtitles
- Subtitle tokenization no longer runs a duplicate full
parseTextpass per line: the termsFind scanner walk is now the only tokenizer and emits its own hoverable filler runs for unmatched text (parseText is kept only as an error fallback). This roughly halves the dictionary work per line. - The Yomitan scanning helpers are now installed once per parser window (
__subminerYomitanScan) instead of re-shipping and re-parsing a ~500-line script for every subtitle line; each line only evaluates a tiny call. - termsFind lookups are cached across subtitle lines in a window-persistent LRU keyed by substring, so repeated particles and verb forms stop costing backend round trips. The cache invalidates on dictionary/settings changes and window reloads.
- The scanner walk now skips lookups at punctuation and whitespace positions (latin letters and digits still look up, e.g. Tシャツ) and caps the shrinking-window retry ladder at four extra lookups per position.
- Tokenizer runtime dependencies are built once instead of per line, fixing a JLPT lookup cache that never hit (it was keyed on a per-call closure identity and leaked a Map per line) and a
which mecabavailability check that re-ran synchronously on every line when MeCab is absent. - Subtitle changes no longer restart the prefetch run per line (which discarded in-flight tokenization work); prefetch now only pauses for the live line and restarts on real seeks, cache invalidation, or option changes. Prefetch also stays paused across a provisional raw-subtitle emit and resumes only after the tokenized payload lands, so it never competes with the on-screen line for the parser window.
- Added per-stage debug timings (
scanMs,mecabMs,frequencyMs,annotateMs) to the subtitle tokenization pipeline log.