mirror of
https://github.com/ksyasuda/SubMiner.git
synced 2026-08-18 00:21:41 -07:00
fix(dictionary): isolate concurrent snapshot writes and shorten zip build blocks
The streamed snapshot writer keyed its temp file on the pid alone, so two overlapping writes for the same media (a manual generate racing auto-sync) streamed into one file and tore it; add a per-write sequence suffix. Term banks were stringified 10k entries at a time, measured at ~38MB and ~135ms per bank on a real merged dictionary. Halving that block matters more than the write itself: at 2k entries the longest event-loop stall in a merged build drops from ~135ms to 28ms.
This commit is contained in:
@@ -78,7 +78,11 @@ export async function buildDictionaryZip(
|
||||
};
|
||||
}
|
||||
|
||||
const entriesPerBank = 10_000;
|
||||
// Each bank is stringified in one shot, so the bank size sets the longest single block in the
|
||||
// build. 10k entries measured ~38MB and ~135ms per bank on a real merged dictionary; 2k keeps
|
||||
// every bank under the archive writer's yield budget at ~27ms. Yomitan reads any number of
|
||||
// term_bank_N.json files, so this only changes how the terms are split across them.
|
||||
const entriesPerBank = 2_000;
|
||||
for (let i = 0; i < termEntries.length; i += entriesPerBank) {
|
||||
yield {
|
||||
name: `term_bank_${Math.floor(i / entriesPerBank) + 1}.json`,
|
||||
|
||||
Reference in New Issue
Block a user