SubMiner/docs/plans/2026-05-02-ai-changelog-polish-design.md

AI-Polished Changelog Design

Date: 2026-05-02 Status: Approved, awaiting implementation plan

Problem

Today every user-visible PR drops a fragment under changes/. The fragments are written by whoever shipped the PR, so the prose tone, granularity, and noise level vary wildly. By the time a stable release is cut, the assembled CHANGELOG section can run 40+ bullets and lean heavily on internal area labels (Overlay:, Launcher:, etc.). End users skim it, miss the actual story of the release, and the GitHub release body inherits the same clutter.

Goal

Replace the mechanical renderGroupedChanges step in scripts/build-changelog.ts with a single non-interactive claude -p call that:

• Merges related fragments into coherent feature-level bullets.
• Drops PR-housekeeping noise that no user benefits from reading.
• Reorganizes by user-visible feature, not by area: prefix.
• Produces a tight ## v<version> body suitable for both CHANGELOG.md and the GitHub release notes.

The fragment file format and authoring workflow stay the same. Only the rendering step changes.

Decisions

1. Pipeline: Replace the current bullet renderer. changelog:build and changelog:prerelease-notes always invoke claude -p. There is no legacy fallback path.
2. Review: Write straight to CHANGELOG.md / release/release-notes.md / release/prerelease-notes.md. The release engineer reviews the diff before tagging; no extra confirmation prompt.
3. CI: Local-only. The release workflow no longer auto-runs changelog:build when fragments are pending; instead it fails the tagged release with a clear message asking the user to run the build locally and commit the result. CI does not need claude on PATH or any Anthropic credentials.
4. Polish depth: Heavy rewrite plus dedupe. Claude is allowed to merge bullets, drop trivial ones, and reorder by feature.
5. Internal section: Dropped from release notes entirely. Kept in CHANGELOG.md inside a <details><summary>Internal changes</summary> collapse so the historical record survives without taking over the page.
6. Prereleases: Same polish path. Beta and RC notes look as clean as stable.
7. Claude invocation: claude -p --bare --model sonnet --permission-mode bypassPermissions --output-format text over stdin. --bare skips hooks, MCP, auto-memory, and CLAUDE.md discovery so the call is self-contained and reproducible.
8. Failure mode: Hard fail. Missing claude binary, non-zero exit, empty/short output, or output missing the required section headers all abort the build.

Architecture

Affected files

• scripts/build-changelog.ts — primary edit. Replace renderGroupedChanges with polishFragmentsWithClaude. Wire it into writeChangelogArtifacts, writePrereleaseNotesForVersion, and any other path that currently calls renderGroupedChanges.
• scripts/build-changelog.test.ts — add runClaude to the dep injection surface, stub it in unit tests, add new coverage for failure modes and mode='changelog' vs mode='release-notes'.
• .github/workflows/release.yml (or wherever the auto-build lives) — remove the auto-changelog:build fallback; add a guard that fails the run if changes/*.md exist on a tag-based release.
• src/release-workflow.test.ts, src/prerelease-workflow.test.ts — update if they exercise the CI auto-run path.
• docs/RELEASING.md — document the new local-only build step and the claude PATH requirement.
• changes/README.md — note that fragments will be merged and rewritten, so authors should write raw notes rather than polished prose.
• changes/<n>-ai-changelog-polish.md — fragment for this change itself (type: internal, area: release).

New function: polishFragmentsWithClaude

polishFragmentsWithClaude(
  fragments: ChangeFragment[],
  options: {
    mode: 'changelog' | 'release-notes',
    version: string,
    date?: string,        // changelog mode only
    deps?: { runClaude?: (prompt: string) => string },
  },
): string

• Filters out internal fragments when mode === 'release-notes'.
• Serializes the surviving fragments into the prompt format below.
• Invokes claude (via deps.runClaude or the default execFileSync wrapper).
• Validates the output contains the expected section headers; throws on failure.
• Returns the Markdown body that will be inserted under the ## v<version> heading.

The existing prependReleaseSection, extractReleaseSectionBody, writeReleaseNotesFile, and generateDocsChangelog functions stay exactly as they are — they consume the polished body the same way they consumed the rendered body.

Prompt input format

MODE: changelog | release-notes
VERSION: 0.13.0
DATE: 2026-05-02            (changelog mode only)

FRAGMENT changes/291-character-dictionary-selection.md
type: added
area: dictionary
breaking: false
- Added CLI and in-app AniList selection for character dictionary mismatches...
- Added launcher support through `subminer dictionary --candidates`...

FRAGMENT changes/293-interjection-annotation-filter.md
type: fixed
area: tokenizer
breaking: false
- Stopped standalone `あ` interjections from receiving subtitle annotation metadata...

...

Prompt output contract

Claude is instructed to emit Markdown only, no preamble, conforming to:

• Sections in this order, omitting empty ones: ### Breaking Changes, ### Added, ### Changed, ### Fixed, ### Docs.

• For mode === 'changelog', an additional terminal section:

  <details>
  <summary>Internal changes</summary>
  
  ### Internal
  - …
  - …
  
  </details>
  

• Bullets lead with a feature name (e.g. Playlist browser:, Windows overlay:) — not with the raw area: slug.

• Bullets may merge multiple fragments, but breaking changes must retain their substance and stay in ### Breaking Changes.

• Bullets that document only PR-housekeeping or CodeRabbit follow-ups are dropped unless they describe a user-visible behavior change.

After parsing, the body is passed unchanged to prependReleaseSection, which wraps it under the standard ## v<version> (<date>) heading.

Determinism and review

Claude is non-deterministic. The mitigation is the existing release workflow: the polished CHANGELOG/release-notes are committed before tagging, so what's in the repo is what ships. The release engineer diffs the commit before pushing.

Testing

scripts/build-changelog.test.ts already exposes a deps injection seam for fs operations. Extend it with runClaude?: (prompt: string) => string. Tests stub it to return canned Markdown; the default implementation (used in production) wraps execFileSync('claude', …).

New cases to cover:

• Mode 'release-notes' filters internal fragments before sending to Claude.
• Mode 'changelog' includes internal fragments and the <details> wrapper expectation is propagated through to the prompt.
• Missing claude binary throws a clear error.
• Non-zero exit code from claude throws.
• Empty / whitespace-only output throws.
• Output missing the expected section headers throws.
• Prerelease path uses release-notes mode and writes release/prerelease-notes.md with the existing disclaimer.
• The polished body still flows correctly through prependReleaseSection, extractReleaseSectionBody, and generateDocsChangelog.

release-workflow.test.ts and prerelease-workflow.test.ts are updated to cover the new "fragments pending on a tag" failure mode.

Documentation

• docs/RELEASING.md updates the stable and prerelease sections to state that changelog:build and changelog:prerelease-notes invoke claude locally, list the PATH requirement, and remove the note about CI auto-running changelog:build.
• changes/README.md adds a short note: fragments will be merged and rewritten by Claude during release, so write raw notes — don't polish them.

Out of Scope (YAGNI)

• No caching of polished output between runs.
• No diff-and-confirm interactive prompt.
• No SDK fallback if the CLI is missing.
• No --no-polish escape hatch. Revert the commit if you genuinely need the legacy renderer back.
• No change to the fragment authoring schema or the pr-check enforcement.

Open Risks

• Claude regresses on a release. Mitigated by the commit-before-tag review step. If the polish is bad, the release engineer edits CHANGELOG.md directly and re-commits before tagging.
• --bare mode behavior changes upstream. Pinning to --model sonnet and --bare reduces drift, but Claude Code is still an external tool. If invocation flags change, the release engineer will see a hard failure and can patch the script.
• CI surface area increases. Removing the auto-changelog:build fallback means a tag pushed before running the local build will fail CI. The error message must clearly point at the fix.