sudacode/SubMiner
1
0
Fork 0
mirror of https://github.com/ksyasuda/SubMiner.git synced 2026-04-26 04:19:27 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix: exclude standalone interjection annotations

Browse Source
This commit is contained in:
Kyle Yasuda
2026-04-25 15:52:31 -07:00
parent 5b326978e9
commit 60435fee10
4 changed files with 66 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
backlog/tasks/task-293 - Fix-interjection-tokens-receiving-subtitle-annotations.md Normal file
View File

@@ -0,0 +1,25 @@
---
id: TASK-293
title: Fix interjection tokens receiving subtitle annotations
status: In Progress
assignee: []
created_date: '2026-04-25 22:50'
labels:
- tokenizer
- bug
dependencies: []
priority: medium
---
## Description
<!-- SECTION:DESCRIPTION:BEGIN -->
Standalone interjections such as あ should remain hoverable dictionary tokens but must not receive N+1, frequency, JLPT, or known-word subtitle annotation metadata.
<!-- SECTION:DESCRIPTION:END -->
## Acceptance Criteria
<!-- AC:BEGIN -->
- [ ] #1 A MeCab 感動詞 token like あ is excluded by the shared subtitle annotation gate.
- [ ] #2 annotateTokens strips N+1/frequency/JLPT/known metadata from the interjection while preserving token lookup fields.
- [ ] #3 Focused tokenizer regression passes.
<!-- AC:END -->