calesthio/openmontage

hyperframes-media

Audio and media assets for HyperFrames compositions, produced by one shared audio engine (scripts/audio.mjs) — multi-provider TTS (HeyGen / ElevenLabs / Kokoro local), background music + sound effects (HeyGen audio-library retrieval by default, with local L…

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/calesthio/openmontage --skill hyperframes-media

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

HyperFrames Media

Create the audio and media assets a composition needs — voiceover (TTS), background music + sound effects, transcription, captions, background removal — then consume and animate that data in HTML. For placing assets into compositions, see hyperframes-core.

The audio engine — one source for TTS · BGM · SFX

Workflows do NOT hand-roll audio or vendor a copy. There is one engine — `scripts/audio.mjs` — that takes a neutral audio_request.json and writes audio_meta.json (plus assets under assets/voice|bgm|sfx):

bash
# <MEDIA_DIR> = this skill's directory
node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json

All three capabilities degrade on ONE switch — whether a HeyGen credential is present (resolved from $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen, not the CLI):

CapabilityHeyGen credential presentabsent
TTSHeyGen Starfish REST (native word timestamps)→ ElevenLabs → Kokoro (chain transcribe for words)
BGMHeyGen music retrievalLyria → MusicGen local generation (detached)
SFXHeyGen sound-effects retrieval (min_score 0.4)bundled 21-file library (assets/sfx/)
  • Request (audio_request.json): { provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }. id joins each line back to the caller's model (a frame number, a scene id, …). bgm.mode = retrieve | generate | none; omit for auto (retrieve when credentialed, else generate). An explicit retrieve is strict — it skips rather than starting a detached generate (for callers with no wait-bgm step).
  • Output (audio_meta.json, id-keyed): { tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }.
  • --only tts,bgm,sfx runs a subset and merges into an existing --out (e.g. TTS+BGM early, SFX once cues exist).
  • BGM generate is spawned detached (bgm_pending: true) — run scripts/wait-bgm.mjs before assembling.
  • scripts/heygen-tts.mjs is a single-shot CLI over the same code (one text → wav + words) for when you just need HeyGen TTS without a request file.

Full flag list + the audio_meta.json schema live in the header of scripts/audio.mjs. The references below cover the provider details and edge cases behind each capability.

Preflight — show sign-in status before any audio

Always run this before generating voice or BGM — inside a full workflow _or_ a one-off "generate me a BGM/voiceover" request. No HeyGen credential is not a reason to silently fall back to local engines: first recommend signing in and let the user decide. Run the shared preflight and relay its output verbatim — don't improvise your own "missing key" prompt, and don't offer to write keys into a per-repo .env:

bash
npx hyperframes auth status
  • Signed in → it prints the account; proceed.
  • Not signed in (exit 1 is expected here — "not signed in" is a normal state, not a failure) → it prints registration-first guidance. Recommend signing in: npx hyperframes auth login is browser OAuth — it signs in and creates an account (always available through this repo's CLI). To use an existing HeyGen API key (from app.heygen.com/settings/api), run npx hyperframes auth login --api-key — it saves to the shared ~/.heygen (no per-repo .env). The output also lists the local engines voice/BGM will fall back to and a pip hint when deps are missing. Relay this output as-is — don't paraphrase it into your own wording. Then STOP and wait for the user to choose — sign in, or say "go" / "local" to continue offline — before generating anything. This is a real decision point, not a passing note: don't fold it into another question, and don't proceed past it on your own. (Exception: in autonomous / non-interactive mode, note the status and continue offline.)
  • npx hyperframes auth status --json returns { configured, recommended_action, offline_engines } for deterministic branching.
  • If the CLI can't run (not on PATH and npx can't fetch it) → still recommend signing in (npx hyperframes auth login) and STOP for the user's choice — don't treat "no credential" as a silent green light for local generation.

Credential resolution, full key priority, and the local-dependency list are in references/requirements.md.

Provider chains (the detail behind the engine)

TTS — first available provider wins (the engine, or npx hyperframes tts "..."):

OrderProviderDetected whenWord timestamps
1HeyGen (Starfish)$HEYGEN_API_KEY / hyperframes auth loginYes, native — pass --words narration.words.json to capture
2ElevenLabs$ELEVENLABS_API_KEY setNo — chain transcribe after
3Kokoro-82M (local, 54 voices)always (no key required)No — chain transcribe after
The published hyperframes tts CLI is often the local-only build (its --help says "Kokoro-82M", no --provider/--words) and silently falls back to Kokoro even with $HEYGEN_API_KEY set. That is why the engine's HeyGen path is the self-contained scripts/heygen-tts.mjs (REST), NOT the CLI; the CLI is used only for the Kokoro path. See references/tts.md.

BGM & SFX — by default retrieved from the HeyGen audio library (/v3/audio/sounds), same credential as HeyGen TTS, with the no-credential fallback from the switch above:

AssetHeyGen typeLands inFallback (no credential)
BGMmusicassets/bgm/track.mp3 (retrieve) · track.wav (generate)Lyria / MusicGen generation
SFXsound_effects (min_score 0.4)assets/sfx/<slug>.mp3bundled 21-file library (assets/sfx/* + manifest.json)

See references/bgm.md and references/sfx.md.

Routing

TaskRead
The audio engine — request/meta schema, --only, the switchscripts/audio.mjs (header comment)
npx hyperframes tts / heygen-tts.mjs — providers, voices, wordsreferences/tts.md
BGM — HeyGen retrieval + local Lyria / MusicGen generationreferences/bgm.md
SFX — HeyGen retrieval (min_score 0.4) + bundled local libraryreferences/sfx.md
npx hyperframes transcribe — Whisper, model rules, output shapereferences/transcribe.md
npx hyperframes remove-background — transparent cutoutsreferences/remove-background.md
TTS → transcription → captions (no recorded voiceover)references/tts-to-captions.md
Caption authoring — style detection, layout, word grouping, exitreferences/captions/authoring.md
Transcript handling — input formats, quality gates, cleanup, APIsreferences/captions/transcript-handling.md
Caption motion — karaoke, marker effects, audio-reactivereferences/captions/motion.md
Model caches, system dependencies, troubleshootingreferences/requirements.md

Non-negotiable rules

  • One engine, no vendored copies. Produce audio via scripts/audio.mjs (or heygen-tts.mjs for one-shot HeyGen TTS). Don't re-implement TTS/BGM/SFX inside a workflow — write an audio_request.json adapter and call the engine.
  • "HeyGen available" = a resolvable credential, not the CLI. The whole switch keys off heygenCredential(); the published hyperframes tts may be Kokoro-only, and there is no hyperframes bgm / hyperframes sfx command at all.
  • Voice IDs are provider-specific. am_michael is Kokoro-only; HeyGen UUIDs don't work on Kokoro. If you pass --voice, also pin --provider to avoid silent provider drift when the user's env changes.
  • Always pass `--model` to `transcribe`. The CLI default small.en silently translates non-English audio. See references/transcribe.md → "Language Rule".
  • HeyGen returns word timestamps; ElevenLabs / Kokoro do not. The engine chains transcribe automatically for the latter two; standalone, pass --words to HeyGen or run transcribe against the audio file.
  • Captions consume the flat word-array format with { id, text, start, end }. See references/transcribe.md → "Output Shape".
  • `remove-background --background-output` is hole-cut, not inpainted. For "scene without the person", a different tool is needed. See references/remove-background.md → "When NOT the right tool".
  • BGM/SFX default to HeyGen retrieval; the no-credential fallback is generation (BGM) or the bundled library (SFX). /audio/sounds ranks by a text query — name effects concretely (glass shatter, not dramatic sound); a no-match skips, never blocks the render. SFX sit at volume ~0.35 under voice + BGM. See references/sfx.md / references/bgm.md.
  • Treat workflow caption HTML as generated output. For preset-backed videos, the reusable skin source lives at .hyperframes/caption-skin.html and the workflow script writes compositions/captions.html; do not edit generated compositions/captions.html to fix the skin. Rebuild via the workflow's captions.mjs, or use that workflow's explicit overrides mechanism when present.

タスクに関連する項目

同じリポジトリから

関連する Skills

すべての Skills
calesthio
コミュニティ

hyperframes-cli

HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, preview, play, render, publish, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (hyperframes lambda deploy / render / progress / destroy / policies).

1時間の導入数
2
GitHub Stars
7.8万
更新日
11月3日
calesthio
コミュニティ

pose-library-design

Design reusable 2D character pose libraries, action cycles, and expression states for data-driven animation.

1時間の導入数
2
GitHub Stars
7.8万
更新日
11月3日
calesthio
コミュニティ

create-video

Create videos from a text prompt using HeyGen's Video Agent. Use when: (1) Creating a video from a description or idea, (2) Generating explainer, demo, or marketing videos from a prompt, (3) Making a video without specifying exact avatars, voices, or scenes, (4) Quick video prototyping or drafts, (5) One-shot prompt-to-video generation, (6) User says "make me a video" or "create a video about X".

1時間の導入数
2
GitHub Stars
7.6万
更新日
11月3日
calesthio
コミュニティ

ffmpeg

Video and audio processing with FFmpeg. Use for format conversion, resizing, compression, audio extraction, and preparing assets for Remotion. Triggers include converting GIF to MP4, resizing video, extracting audio, compressing files, or any media transformation task.

1時間の導入数
2
GitHub Stars
7.6万
更新日
11月3日