Compare commits

..

6 Commits

17 changed files with 543 additions and 132 deletions
+1 -1
View File
@@ -1,4 +1,4 @@
type: fixed
type: added
area: mining
- Normalized generated card audio by default during media extraction, with `ankiConnect.media.normalizeAudio` available to keep raw source loudness when needed.
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: stats
- Cover art is now fetched eagerly when a new series starts playing, instead of waiting for the first visit to its series detail page, so the stats timeline shows the best-guess AniList image right away. The stats covers endpoint also backfills missing series art in the background, so existing series without an image pick one up on the next stats page load.
+4
View File
@@ -0,0 +1,4 @@
type: changed
area: launcher
- Moved fzf previews below launcher menus so long titles and metadata have more horizontal room.
+5
View File
@@ -0,0 +1,5 @@
type: fixed
area: overlay
- Fixed single-kana grammar tokens counting as known words by borrowing the reading of an unrelated Anki note (よ in 全然いいよ matched a card read よ such as 夜, standalone え matched 絵), which painted them with the known-word highlight. Reading-only known-word matching now requires at least two kana; single-kana cards still match by their word field, and tokens genuinely present in the known-words cache (e.g. です) keep their highlight.
- Standalone suffix tokens (MeCab pos2 接尾, e.g. さん, れる) are now excluded from JLPT/frequency/N+1 annotations by default, matching how particles and interjections are treated. Cache-backed known-word highlighting still applies; override via the pos2 exclusion config if you want them annotated.
+4
View File
@@ -0,0 +1,4 @@
type: fixed
area: stats
- Session known-word counts no longer show 0 everywhere. The stats server's known-word cache parser only understood the v1/v2 cache formats, so after the reading-aware v3 cache upgrade it silently treated the cache as missing; it now flattens v3 note entries into the headword set.
+2 -2
View File
@@ -56,7 +56,7 @@ export function showFzfFlatMenu(
`--prompt=${prompt}`,
'--delimiter=\t',
'--with-nth=2',
'--preview-window=right:50%:wrap',
'--preview-window=down:50%:wrap',
'--preview',
previewCommand,
];
@@ -468,7 +468,7 @@ thumb=$(get_thumb)
'--prompt=Select Video: ',
'--delimiter=\t',
'--with-nth=1',
'--preview-window=right:50%:wrap',
'--preview-window=down:50%:wrap',
'--preview',
previewCmd,
],
+1 -1
View File
@@ -2,7 +2,7 @@
"name": "subminer",
"productName": "SubMiner",
"desktopName": "SubMiner.desktop",
"version": "0.18.0-beta.1",
"version": "0.18.0-beta.2",
"description": "All-in-one sentence mining overlay with AnkiConnect and dictionary integration",
"packageManager": "bun@1.3.5",
"main": "dist/main-entry.js",
+16 -6
View File
@@ -5,21 +5,28 @@
## Highlights
### Added
- **Watch History Browser**
- New `subminer -H` / `--history` command to browse your local watch history, replay the last episode, jump to the next one, or pick an episode via fzf/rofi.
- New `subminer -H` / `--history` command lets you browse your local watch history, replay the last episode, jump to the next one, or pick an episode via fzf or rofi.
- The rofi picker now shows AniList cover art for each show, making it easier to spot the right title at a glance.
- **Card Audio Normalization**
- Audio extracted for Anki cards is now volume-normalized by default, giving more consistent playback loudness across cards.
- Prefer the original source volume? Disable it via the new `ankiConnect.media.normalizeAudio` setting.
### Changed
- **New App Icon**
- SubMiner now ships pixel-art submarine artwork, contributed by an anonymous community member.
- SubMiner now ships pixel-art submarine artwork contributed by an anonymous community member.
- Applied across the app icon, tray icon, notifications, README, docs site, and stats page.
- **Launcher Preview Layout**
- fzf previews in the launcher now sit below the menu instead of beside it, giving long titles and metadata more horizontal room.
### Fixed
- **Character Name Highlighting in Subtitles**
- Fixed unspaced Japanese names (e.g. 東紫乃, 渡辺真奈美) being split at the wrong point, which left surnames like 東 and 渡辺 without their character portrait or hover lookup.
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling or by conflicting grammar tagging.
- Fixed names getting cut off or losing their highlight when caught by the subtitle scanner's punctuation handling, or stripped entirely when grammar tagging misclassified the name token.
- No action needed — existing data upgrades automatically the next time a matching name is seen.
- **Known-Word Highlighting**
- Words are no longer marked "known" (green) just because they share spelling with a known Anki card that actually teaches a different reading (e.g. 床 read as とこ no longer falsely matches a known 床/ゆか card).
- Single-kana grammar tokens (particles like よ, え) no longer borrow an unrelated card's reading and get falsely painted as known.
- Stats sessions now correctly reflect known-word counts again after the reading-aware matching upgrade, instead of showing 0 everywhere.
- **Unparsed Subtitle Text**
- Subtitle text the dictionary can't recognize (like a truncated verb form) is now still hoverable for lookup and correctly counted toward a sentence's difficulty, instead of showing as dead, non-interactive text.
- **Kiku Manual Field Grouping**
@@ -27,15 +34,15 @@
- Fixed a duplicate "Field grouping cancelled" notification appearing when grouping was cancelled via the trigger shortcut.
- **Secondary Subtitles**
- Karaoke-style secondary subtitles (common in opening/ending songs) no longer spam dozens of lines down the screen; repeated lines are now collapsed and the subtitle area is capped to a strip at the top.
- **Card Audio Normalization**
- Audio extracted for Anki cards is now volume-normalized by default for more consistent playback loudness.
- If you prefer the original source volume, disable it via the new `ankiConnect.media.normalizeAudio` setting.
- **YouTube Extraction**
- Fixed direct YouTube stream extraction occasionally corrupting the stream URL and causing failed audio/video capture.
- **Background Stats Server**
- Launching SubMiner in the background now correctly auto-starts the stats server when enabled, and won't start a duplicate if one's already running.
- **Stats Trend Charts**
- All trend chart titles now show by default, with the ability to hide specific titles (remembered across sessions) and cap how many top titles a chart displays.
- **Stats Cover Art**
- Cover art now loads as soon as a series starts playing instead of waiting for your first visit to its detail page, so the stats timeline shows artwork right away.
- Existing series missing art are backfilled automatically the next time you open the stats page.
## What's Changed
@@ -50,6 +57,9 @@
- fix(stats): start stats server on background app launch by @ksyasuda in #144
- fix(tokenizer): keep unparsed Yomitan tokens hoverable by @ksyasuda in #145
- fix(overlay): resolve unspaced Japanese name splits and scan recovery by @ksyasuda in #146
- fix(tokenizer): prevent grammar tokens from borrowing known-word highlight via unrelated readings by @ksyasuda in #147
- fix(stats): fetch cover art eagerly at session start instead of on series page visit by @ksyasuda in #148
- fix(stats): parse v3 reading-aware known-word cache in stats server by @ksyasuda in #149
## Installation
@@ -734,6 +734,49 @@ test('KnownWordCacheManager disambiguates known words by note reading', async ()
}
});
test('KnownWordCacheManager does not match single-kana text by reading alone', async () => {
const config: AnkiConnectConfig = {
fields: {
word: 'Word',
},
knownWords: {
highlightEnabled: true,
},
};
const { manager, clientState, cleanup } = createKnownWordCacheHarness(config);
try {
clientState.findNotesResult = [1, 2];
clientState.notesInfoResult = [
{
noteId: 1,
fields: {
Word: { value: '夜' },
'Word Reading': { value: 'よ' },
},
},
{
noteId: 2,
fields: {
Word: { value: 'え' },
},
},
];
await manager.refresh(true);
// よ must not count as known just because 夜 is read よ.
assert.equal(manager.isKnownWord('よ'), false);
assert.equal(manager.isKnownWord('ヨ'), false);
assert.equal(manager.isKnownWord('夜'), true);
assert.equal(manager.isKnownWord('夜', 'よ'), true);
// A literal single-kana word entry still matches via the word map.
assert.equal(manager.isKnownWord('え'), true);
} finally {
cleanup();
}
});
test('KnownWordCacheManager probes reading fields even with per-deck word fields configured', async () => {
const config: AnkiConnectConfig = {
fields: {
+8 -1
View File
@@ -163,7 +163,14 @@ export class KnownWordCacheManager {
);
}
return this.readingCounts.has(convertKatakanaToHiragana(normalized));
// Reading-only fallback, except for single-kana text: particles and
// interjections (よ, ね, え…) would otherwise borrow the reading of an
// unrelated note (夜「よ」, 絵「え」) and count as known.
const hiragana = convertKatakanaToHiragana(normalized);
if ([...hiragana].length === 1) {
return false;
}
return this.readingCounts.has(hiragana);
}
refresh(force = false): Promise<void> {
@@ -301,6 +301,7 @@ function createMockTracker(
{ epochDay: Math.floor(Date.now() / 86_400_000) - 1, totalActiveMin: 30 },
{ epochDay: Math.floor(Date.now() / 86_400_000), totalActiveMin: 45 },
],
ensureAnimeCoverArt: async () => false,
getAnimeCoverArt: async (animeId: number) =>
animeId === 1
? {
@@ -520,6 +521,44 @@ describe('stats server API routes', () => {
});
});
it('GET /api/stats/sessions enriches known-word metrics from a v3 reading-aware cache', async () => {
await withTempDir(async (dir) => {
const cachePath = path.join(dir, 'known-words.json');
fs.writeFileSync(
cachePath,
JSON.stringify({
version: 3,
refreshedAtMs: 1,
scope: 'deck:test',
notes: {
'101': [{ word: 'する', reading: 'する' }],
'102': [{ word: '猫', reading: null }],
},
}),
);
const app = createStatsApp(
createMockTracker({
getSessionWordsByLine: async (sessionId: number) =>
sessionId === 1
? [
{ lineIndex: 1, headword: 'する', occurrenceCount: 2 },
{ lineIndex: 2, headword: '未知', occurrenceCount: 1 },
]
: [],
}),
{ knownWordCachePath: cachePath },
);
const res = await app.request('/api/stats/sessions?limit=5');
assert.equal(res.status, 200);
const body = await res.json();
const first = body[0];
assert.equal(first.knownWordsSeen, 2);
assert.equal(first.knownWordRate, 66.7);
});
});
it('GET /api/stats/sessions/:id/events forwards event type filters to the tracker', async () => {
let seenSessionId = 0;
let seenLimit = 0;
@@ -994,8 +1033,9 @@ describe('stats server API routes', () => {
assert.equal(res.status, 404);
});
it('POST /api/stats/covers batches stored cover art without fetching missing art', async () => {
it('POST /api/stats/covers batches stored cover art and backfills missing anime art in the background', async () => {
let ensureCoverArtCalls = 0;
const ensureAnimeCoverArtCalls: number[] = [];
const app = createStatsApp(
createMockTracker({
getCoverArt: async (videoId: number) =>
@@ -1015,6 +1055,10 @@ describe('stats server API routes', () => {
ensureCoverArtCalls += 1;
return true;
},
ensureAnimeCoverArt: async (animeId: number) => {
ensureAnimeCoverArtCalls.push(animeId);
return false;
},
}),
);
@@ -1042,6 +1086,68 @@ describe('stats server API routes', () => {
},
});
assert.equal(ensureCoverArtCalls, 0);
assert.deepEqual(ensureAnimeCoverArtCalls, [99999]);
});
it('POST /api/stats/covers limits concurrent missing anime cover backfills', async () => {
let activeBackfills = 0;
let maxActiveBackfills = 0;
const pendingBackfills: Array<() => void> = [];
const app = createStatsApp(
createMockTracker({
getAnimeCoverArt: async () => null,
ensureAnimeCoverArt: async () => {
activeBackfills += 1;
maxActiveBackfills = Math.max(maxActiveBackfills, activeBackfills);
await new Promise<void>((resolve) => {
pendingBackfills.push(resolve);
});
activeBackfills -= 1;
return false;
},
}),
);
const res = await app.request('/api/stats/covers', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ animeIds: [101, 102, 103, 104, 105] }),
});
assert.equal(res.status, 200);
assert.equal(maxActiveBackfills, 3);
for (const resolveBackfill of pendingBackfills) {
resolveBackfill();
}
});
it('GET /api/stats/anime/:animeId/cover fetches missing art before serving', async () => {
let fetched = false;
const app = createStatsApp(
createMockTracker({
getAnimeCoverArt: async () =>
fetched
? {
videoId: 1,
anilistId: 21858,
coverUrl: 'https://example.com/cover.jpg',
coverBlob: Buffer.from([0xff, 0xd8, 0xff, 0xd9]),
titleRomaji: 'Little Witch Academia',
titleEnglish: 'Little Witch Academia',
episodesTotal: 25,
fetchedAtMs: Date.now(),
}
: null,
ensureAnimeCoverArt: async () => {
fetched = true;
return true;
},
}),
);
const res = await app.request('/api/stats/anime/1/cover');
assert.equal(res.status, 200);
assert.equal(res.headers.get('content-type'), 'image/jpeg');
});
it('GET /api/stats/anime/:animeId/words returns top words for an anime', async () => {
@@ -4041,3 +4041,91 @@ test('markActiveVideoWatched returns false when no active session', async () =>
cleanupDbPath(dbPath);
}
});
test('handleMediaChange prefetches cover art at session start', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor({ dbPath });
const fetchedVideoIds: number[] = [];
tracker.setCoverArtFetcher({
fetchIfMissing: async (_db, videoId) => {
fetchedVideoIds.push(videoId);
return false;
},
});
tracker.handleMediaChange('/tmp/Little Witch Academia S02E05.mkv', 'Episode 5');
await waitForPendingAnimeMetadata(tracker);
await waitForCondition(() => fetchedVideoIds.length > 0);
const privateApi = tracker as unknown as {
sessionState: { videoId: number } | null;
};
assert.deepEqual(fetchedVideoIds, [privateApi.sessionState?.videoId]);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
test('ensureAnimeCoverArt fetches art via the latest video of the anime', async () => {
const dbPath = makeDbPath();
let tracker: ImmersionTrackerService | null = null;
try {
const Ctor = await loadTrackerCtor();
tracker = new Ctor({ dbPath });
const privateApi = tracker as unknown as { db: DatabaseSync };
privateApi.db.exec(`
INSERT INTO imm_anime (
anime_id,
normalized_title_key,
canonical_title,
CREATED_DATE,
LAST_UPDATE_DATE
) VALUES (
1,
'little witch academia',
'Little Witch Academia',
1000,
1000
);
INSERT INTO imm_videos (
video_id,
video_key,
canonical_title,
source_type,
duration_ms,
anime_id,
CREATED_DATE,
LAST_UPDATE_DATE
) VALUES
(1, 'local:/tmp/lwa-1.mkv', 'Little Witch Academia S01E01', 1, 0, 1, 1000, 1000),
(2, 'local:/tmp/lwa-2.mkv', 'Little Witch Academia S01E02', 1, 0, 1, 1000, 1000);
`);
const fetchedVideoIds: number[] = [];
tracker.setCoverArtFetcher({
fetchIfMissing: async (_db, videoId) => {
fetchedVideoIds.push(videoId);
return false;
},
});
const result = await tracker.ensureAnimeCoverArt(1);
assert.equal(result, false);
assert.deepEqual(fetchedVideoIds, [2]);
const missing = await tracker.ensureAnimeCoverArt(999);
assert.equal(missing, false);
assert.deepEqual(fetchedVideoIds, [2]);
} finally {
tracker?.destroy();
cleanupDbPath(dbPath);
}
});
+41 -2
View File
@@ -854,6 +854,22 @@ export class ImmersionTrackerService {
this.coverArtFetcher = fetcher;
}
async ensureAnimeCoverArt(animeId: number): Promise<boolean> {
const existing = await this.getAnimeCoverArt(animeId);
if (existing?.coverBlob) {
return true;
}
const row = this.db
.prepare(
'SELECT video_id AS videoId FROM imm_videos WHERE anime_id = ? ORDER BY video_id DESC LIMIT 1',
)
.get(animeId) as { videoId: number } | undefined;
if (!row?.videoId) {
return false;
}
return this.ensureCoverArt(row.videoId);
}
async ensureCoverArt(videoId: number): Promise<boolean> {
const existing = await this.getCoverArt(videoId);
if (existing?.coverBlob) {
@@ -879,8 +895,10 @@ export class ImmersionTrackerService {
}
const fetchPromise = (async () => {
const detail = getMediaDetail(this.db, videoId);
const canonicalTitle = detail?.canonicalTitle?.trim();
const titleRow = this.db
.prepare('SELECT canonical_title AS canonicalTitle FROM imm_videos WHERE video_id = ?')
.get(videoId) as { canonicalTitle: string | null } | undefined;
const canonicalTitle = titleRow?.canonicalTitle?.trim();
if (!canonicalTitle) {
return false;
}
@@ -1342,6 +1360,9 @@ export class ImmersionTrackerService {
} else if (!this.hasJellyfinMetadata(sessionInfo.videoId)) {
this.captureAnimeMetadataAsync(sessionInfo.videoId, normalizedPath, normalizedTitle || null);
}
if (!youtubeVideoId) {
this.prefetchCoverArtAsync(sessionInfo.videoId);
}
this.captureVideoMetadataAsync(sessionInfo.videoId, sourceType, normalizedPath);
}
@@ -1924,6 +1945,24 @@ export class ImmersionTrackerService {
});
}
// Fetch cover art eagerly at session start (after anime metadata parsing
// settles) so new series show art on the stats timeline without requiring a
// visit to the series detail page first.
private prefetchCoverArtAsync(videoId: number): void {
const pendingMetadata = this.pendingAnimeMetadataUpdates.get(videoId);
void (async () => {
try {
await pendingMetadata;
if (this.isDestroyed) {
return;
}
await this.ensureCoverArt(videoId);
} catch (error) {
this.logger.warn('Unable to prefetch cover art', (error as Error).message);
}
})();
}
private updateVideoTitleForActiveSession(canonicalTitle: string): void {
if (!this.sessionState) return;
updateVideoTitleRecord(this.db, this.sessionState.videoId, canonicalTitle);
+171
View File
@@ -0,0 +1,171 @@
import type { Hono } from 'hono';
import type { ImmersionTrackerService } from './immersion-tracker-service.js';
type StatsCoverImagePayload = {
contentType: string;
dataUrl: string;
} | null;
type StatsCoverBatchBody = {
animeIds?: unknown;
videoIds?: unknown;
};
const MAX_BACKGROUND_ANIME_COVER_FETCHES = 3;
function parseIntQuery(raw: string | undefined, fallback: number, maxLimit?: number): number {
if (raw === undefined) return fallback;
const n = Number(raw);
if (!Number.isFinite(n) || n < 0) {
return fallback;
}
const parsed = Math.floor(n);
return maxLimit === undefined ? parsed : Math.min(parsed, maxLimit);
}
function parsePositiveIdList(raw: unknown, maxItems = 100): number[] {
if (!Array.isArray(raw)) return [];
const ids = new Set<number>();
for (const rawId of raw) {
const id = typeof rawId === 'number' ? rawId : typeof rawId === 'string' ? Number(rawId) : NaN;
if (Number.isFinite(id) && id > 0) {
ids.add(Math.floor(id));
if (ids.size >= maxItems) break;
}
}
return Array.from(ids).sort((a, b) => a - b);
}
function coverImagePayload(
art: { coverBlob?: Uint8Array | null } | null | undefined,
): StatsCoverImagePayload {
if (!art?.coverBlob) return null;
const bytes = new Uint8Array(art.coverBlob);
const contentType = detectImageContentType(bytes);
return {
contentType,
dataUrl: `data:${contentType};base64,${Buffer.from(bytes).toString('base64')}`,
};
}
function detectImageContentType(bytes: Uint8Array): string {
if (
bytes.length >= 8 &&
bytes[0] === 0x89 &&
bytes[1] === 0x50 &&
bytes[2] === 0x4e &&
bytes[3] === 0x47
) {
return 'image/png';
}
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) {
return 'image/jpeg';
}
if (
bytes.length >= 12 &&
bytes[0] === 0x52 &&
bytes[1] === 0x49 &&
bytes[2] === 0x46 &&
bytes[3] === 0x46 &&
bytes[8] === 0x57 &&
bytes[9] === 0x45 &&
bytes[10] === 0x42 &&
bytes[11] === 0x50
) {
return 'image/webp';
}
return 'application/octet-stream';
}
function createLimitedTaskRunner(maxConcurrentTasks: number): (task: () => Promise<void>) => void {
const queue: Array<() => Promise<void>> = [];
let activeTasks = 0;
const drain = (): void => {
while (activeTasks < maxConcurrentTasks && queue.length > 0) {
const task = queue.shift();
if (!task) return;
activeTasks += 1;
void task()
.catch(() => {})
.finally(() => {
activeTasks -= 1;
drain();
});
}
};
return (task: () => Promise<void>): void => {
queue.push(task);
drain();
};
}
export function registerStatsCoverRoutes(app: Hono, tracker: ImmersionTrackerService): void {
const enqueueAnimeCoverBackfill = createLimitedTaskRunner(MAX_BACKGROUND_ANIME_COVER_FETCHES);
app.post('/api/stats/covers', async (c) => {
const body = (await c.req.json().catch(() => null)) as StatsCoverBatchBody | null;
const animeIds = parsePositiveIdList(body?.animeIds);
const videoIds = parsePositiveIdList(body?.videoIds);
const anime: Record<number, StatsCoverImagePayload> = {};
const media: Record<number, StatsCoverImagePayload> = {};
await Promise.all(
animeIds.map(async (animeId) => {
const art = await tracker.getAnimeCoverArt(animeId);
if (!art?.coverBlob) {
enqueueAnimeCoverBackfill(async () => {
await tracker.ensureAnimeCoverArt(animeId);
});
}
anime[animeId] = coverImagePayload(art);
}),
);
await Promise.all(
videoIds.map(async (videoId) => {
media[videoId] = coverImagePayload(await tracker.getCoverArt(videoId));
}),
);
return c.json({ anime, media });
});
app.get('/api/stats/anime/:animeId/cover', async (c) => {
const animeId = parseIntQuery(c.req.param('animeId'), 0);
if (animeId <= 0) return c.body(null, 404);
let art = await tracker.getAnimeCoverArt(animeId);
if (!art?.coverBlob) {
await tracker.ensureAnimeCoverArt(animeId);
art = await tracker.getAnimeCoverArt(animeId);
}
if (!art?.coverBlob) return c.body(null, 404);
const bytes = new Uint8Array(art.coverBlob);
return new Response(bytes, {
headers: {
'Content-Type': detectImageContentType(bytes),
'Cache-Control': 'public, max-age=86400',
},
});
});
app.get('/api/stats/media/:videoId/cover', async (c) => {
const videoId = parseIntQuery(c.req.param('videoId'), 0);
if (videoId <= 0) return c.body(null, 404);
let art = await tracker.getCoverArt(videoId);
if (!art?.coverBlob) {
await tracker.ensureCoverArt(videoId);
art = await tracker.getCoverArt(videoId);
}
if (!art?.coverBlob) return c.body(null, 404);
const bytes = new Uint8Array(art.coverBlob);
return new Response(bytes, {
headers: {
'Content-Type': detectImageContentType(bytes),
'Cache-Control': 'public, max-age=604800',
},
});
});
}
+17 -118
View File
@@ -17,6 +17,7 @@ import {
} from '../../anki-field-config.js';
import { resolveAnimatedImageLeadInSeconds } from '../../anki-integration/animated-image-sync.js';
import type { AnilistRateLimiter } from './anilist/rate-limiter.js';
import { registerStatsCoverRoutes } from './stats-cover-routes.js';
import {
resolveRetimedSecondarySubtitleTextFromSidecar,
resolveSecondarySubtitleTextFromSidecar,
@@ -51,16 +52,6 @@ type StatsExcludedWordPayload = {
reading: string;
};
type StatsCoverImagePayload = {
contentType: string;
dataUrl: string;
} | null;
type StatsCoverBatchBody = {
animeIds?: unknown;
videoIds?: unknown;
};
function parseIntQuery(raw: string | undefined, fallback: number, maxLimit?: number): number {
if (raw === undefined) return fallback;
const n = Number(raw);
@@ -113,62 +104,6 @@ function parseExcludedWordsBody(body: unknown): StatsExcludedWordPayload[] | nul
return words;
}
function parsePositiveIdList(raw: unknown, maxItems = 100): number[] {
if (!Array.isArray(raw)) return [];
const ids = new Set<number>();
for (const rawId of raw) {
const id = typeof rawId === 'number' ? rawId : typeof rawId === 'string' ? Number(rawId) : NaN;
if (Number.isFinite(id) && id > 0) {
ids.add(Math.floor(id));
if (ids.size >= maxItems) break;
}
}
return Array.from(ids).sort((a, b) => a - b);
}
function coverImagePayload(
art: { coverBlob?: Uint8Array | null } | null | undefined,
): StatsCoverImagePayload {
if (!art?.coverBlob) return null;
const bytes = new Uint8Array(art.coverBlob);
const contentType = detectImageContentType(bytes);
return {
contentType,
dataUrl: `data:${contentType};base64,${Buffer.from(bytes).toString('base64')}`,
};
}
function detectImageContentType(bytes: Uint8Array): string {
if (
bytes.length >= 8 &&
bytes[0] === 0x89 &&
bytes[1] === 0x50 &&
bytes[2] === 0x4e &&
bytes[3] === 0x47
) {
return 'image/png';
}
if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) {
return 'image/jpeg';
}
if (
bytes.length >= 12 &&
bytes[0] === 0x52 &&
bytes[1] === 0x49 &&
bytes[2] === 0x46 &&
bytes[3] === 0x46 &&
bytes[8] === 0x57 &&
bytes[9] === 0x45 &&
bytes[10] === 0x42 &&
bytes[11] === 0x50
) {
return 'image/webp';
}
return 'application/octet-stream';
}
function resolveStatsNoteFieldName(
noteInfo: StatsServerNoteInfo,
...preferredNames: (string | undefined)[]
@@ -326,10 +261,25 @@ function loadKnownWordsSet(cachePath: string | undefined): Set<string> | null {
const raw = JSON.parse(readFileSync(cachePath, 'utf-8')) as {
version?: number;
words?: string[];
notes?: Record<string, Array<{ word?: unknown; reading?: unknown }>>;
};
if ((raw.version === 1 || raw.version === 2) && Array.isArray(raw.words)) {
return new Set(raw.words);
}
// v3 stores reading-aware entries per note; stats rows only carry
// headwords, so flatten to a word set (reading-agnostic, fail-open).
if (raw.version === 3 && raw.notes && typeof raw.notes === 'object') {
const words = new Set<string>();
for (const entries of Object.values(raw.notes)) {
if (!Array.isArray(entries)) continue;
for (const entry of entries) {
if (entry && typeof entry.word === 'string' && entry.word) {
words.add(entry.word);
}
}
}
return words;
}
} catch {
/* ignore */
}
@@ -1017,58 +967,7 @@ export function createStatsApp(
return c.json({ ok: true });
});
app.post('/api/stats/covers', async (c) => {
const body = (await c.req.json().catch(() => null)) as StatsCoverBatchBody | null;
const animeIds = parsePositiveIdList(body?.animeIds);
const videoIds = parsePositiveIdList(body?.videoIds);
const anime: Record<number, StatsCoverImagePayload> = {};
const media: Record<number, StatsCoverImagePayload> = {};
await Promise.all(
animeIds.map(async (animeId) => {
anime[animeId] = coverImagePayload(await tracker.getAnimeCoverArt(animeId));
}),
);
await Promise.all(
videoIds.map(async (videoId) => {
media[videoId] = coverImagePayload(await tracker.getCoverArt(videoId));
}),
);
return c.json({ anime, media });
});
app.get('/api/stats/anime/:animeId/cover', async (c) => {
const animeId = parseIntQuery(c.req.param('animeId'), 0);
if (animeId <= 0) return c.body(null, 404);
const art = await tracker.getAnimeCoverArt(animeId);
if (!art?.coverBlob) return c.body(null, 404);
const bytes = new Uint8Array(art.coverBlob);
return new Response(bytes, {
headers: {
'Content-Type': detectImageContentType(bytes),
'Cache-Control': 'public, max-age=86400',
},
});
});
app.get('/api/stats/media/:videoId/cover', async (c) => {
const videoId = parseIntQuery(c.req.param('videoId'), 0);
if (videoId <= 0) return c.body(null, 404);
let art = await tracker.getCoverArt(videoId);
if (!art?.coverBlob) {
await tracker.ensureCoverArt(videoId);
art = await tracker.getCoverArt(videoId);
}
if (!art?.coverBlob) return c.body(null, 404);
const bytes = new Uint8Array(art.coverBlob);
return new Response(bytes, {
headers: {
'Content-Type': detectImageContentType(bytes),
'Cache-Control': 'public, max-age=604800',
},
});
});
registerStatsCoverRoutes(app, tracker);
app.get('/api/stats/episode/:videoId/detail', async (c) => {
const videoId = parseIntQuery(c.req.param('videoId'), 0);
@@ -1722,6 +1722,36 @@ test('annotateTokens keeps known status while clearing other annotations for sta
}
});
test('annotateTokens excludes standalone noun-suffix tokens from annotations while keeping cache-backed known status', () => {
const tokens = [
makeToken({
surface: 'さん',
headword: 'さん',
reading: 'サン',
partOfSpeech: PartOfSpeech.noun,
pos1: '名詞',
pos2: '接尾',
startPos: 0,
endPos: 2,
frequencyRank: 33,
}),
];
const result = annotateTokens(
tokens,
makeDeps({
isKnownWord: (text) => text === 'さん',
getJlptLevel: (text) => (text === 'さん' ? 'N5' : null),
}),
{ minSentenceWordsForNPlusOne: 1 },
);
assert.equal(result[0]?.isKnown, true);
assert.equal(result[0]?.isNPlusOneTarget, false);
assert.equal(result[0]?.frequencyRank, undefined);
assert.equal(result[0]?.jlptLevel, undefined);
});
test('annotateTokens keeps known status while clearing other annotations for auxiliary-only te-kureru helper spans', () => {
const tokens = [
makeToken({
+1
View File
@@ -3,6 +3,7 @@ import { normalizePos1ExclusionList } from './token-pos1-exclusions';
export const DEFAULT_ANNOTATION_POS2_EXCLUSION_DEFAULTS = Object.freeze([
'非自立',
'接尾',
]) as readonly string[];
export const DEFAULT_ANNOTATION_POS2_EXCLUSION_CONFIG: ResolvedTokenPos2ExclusionConfig = {