digitalsamba/claude-code-video-toolkit

elevenlabs

Generate AI voiceovers, sound effects, and music using ElevenLabs APIs.

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/digitalsamba/claude-code-video-toolkit --skill elevenlabs

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

ElevenLabs Audio Generation

Requires ELEVENLABS_API_KEY in .env.

Text-to-Speech

python
from elevenlabs.client import ElevenLabs
from elevenlabs import save, VoiceSettings
import os

client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))

audio = client.text_to_speech.convert(
    text="Welcome to my video!",
    voice_id="JBFqnCBsd6RMkjVDRZzb",
    model_id="eleven_multilingual_v2",
    voice_settings=VoiceSettings(
        stability=0.5,
        similarity_boost=0.75,
        style=0.5,
        speed=1.0
    )
)
save(audio, "voiceover.mp3")

Models

ModelQualitySSML SupportNotes
eleven_multilingual_v2Highest consistencyNoneStable, production-ready, 29 languages. Toolkit default; supports style + speed
eleven_v4Most expressiveNone (audio tags, IPA)Newest (released 2026-09-28), 90+ languages, best clone accuracy. Stability + similarity only, no style/speed
eleven_flash_v2_5Good<break>, <phoneme>Fastest, cheapest; supports pause/pronunciation tags
eleven_v3ExpressiveNonePrevious generation — prefer v4

Choose: multilingual_v2 for reliability and speed/style control, v4 for maximum expressiveness and cloned voices (test a few takes), flash for SSML control.

Deprecated: eleven_turbo_v2_5 → use eleven_flash_v2_5 (same quality, lower latency; see deprecated models). The toolkit still accepts --model eleven_turbo_v2_5: it prints a one-line warning and sends eleven_flash_v2_5.

Eleven v4

Released 2026-09-28 (changelog, model page). The toolkit calls v4 through the Text to Dialogue API with one input (the narrator voice), as ElevenLabs recommends for content creation and long-form audio:

python
from elevenlabs import DialogueInput

audio = client.text_to_dialogue.convert(
    inputs=[DialogueInput(text="[warm] Welcome to my video!", voice_id="JBFqnCBsd6RMkjVDRZzb")],
    model_id="eleven_v4",
    settings={"stability": 0.5, "similarity": 0.75},  # the only two settings v4 takes
)
save(audio, "voiceover.mp3")
  • uv run tools/voiceover.py --model eleven_v4 ... (and redub.py --model eleven_v4) do this for you, warn once that --style/--speed are ignored, and split text over ~2,000 chars per request (the reliable dialogue limit) at sentence boundaries. Pace with --max-wpm instead of --speed.
  • client.text_to_dialogue.convert_with_timestamps(...) returns the same character alignment as TTS convert_with_timestamps, so redub.py --sync works with v4 and sends the same stability + similarity.
  • Pass settings as a dict: SDK versions before 2.69 type it with a stability-only model. The API reads similarity there since 2026-09-28 (it rejects an out-of-range value with a 422). Text to Dialogue needs elevenlabs>=2.21.
  • Direct delivery with audio tags ([whispering], [warm], [sigh]) and punctuation. Write tags that clearly describe the voice, because v4 can also read a tag as a sound effect.
  • A voice can change accent on one word. A British library voice read "thirty dollars" and "once a card is

on the account" with an American r, in every plain retake of those lines. A direction at the start of the line held the accent in three takes of three: [crisp British English accent] So what does thirty dollars buy?. v4 doesn't read the tag aloud. Put it in the text sent for voicing only, and keep the .txt beside the MP3 as the spoken words, since captions are spelled from it. It can be measured before anyone listens: with the word times from captions.py, an American r pulls the third formant under about 2,200 Hz at the end of "dollars" or "card" in a woman's voice, and a British ending leaves it near 2,700 (Praat, uv sync --extra prosody). templates/howto/scripts/accent.py does this for a folder of narration (--scenes DIR) and lists the lowest first. A word before a vowel reads low in British speech too ("your account" has a linking r), so listen to what it flags before re-voicing.

  • A pace direction slows a line a little. v4 ignores speed, but `[crisp British English accent, measured and

unhurried] at the start gave takes about 8% longer than the same line without "measured and unhurried" (four takes of three lines). For an opening line, make three or four takes and let pick_takes.py choose. Its wpm counts the words of the tag, so put the spoken words back in each take's .txt before picking, not only before captions.py`: with the tag still there it read 33 lines as 180 to 300 wpm and slowed 31 of them, where the true paces were 150 to 185 and 6 needed a touch.

  • eleven_v4_turbo is the real-time variant, available only over the Text to Dialogue WebSocket. The toolkit doesn't use it, since offline renders don't benefit from low latency.

Voice Settings by Style

Stylestabilitysimilaritystylespeed
Natural/professional0.75-0.850.90.0-0.11.0
Conversational0.5-0.60.850.3-0.40.9-1.0
Energetic/YouTuber0.3-0.50.750.5-0.71.0-1.1

voiceover.py's own defaults (stability 0.85, style 0.0) sit at the top of the first row: steady but flat. A library voice picked for its preview usually sounds slower and duller at those defaults than in the preview. Lower the stability first (0.45 on v4 brought Marilyn back to her preview's pace and pitch range), then measure the takes with voice_prosody.py measure and listen.

Pauses Between Sections

With flash models: Use SSML break tags inline:

...end of section. <break time="1.5s" /> Start of next...

Max 3 seconds per break. Excessive breaks can cause speed artifacts.

With multilingual_v2 / v3 / v4: No SSML support. Options:

  • Paragraph breaks (blank lines) — creates ~0.3-0.5s natural pause
  • v3/v4: audio tags such as [long pause]
  • Post-process with ffmpeg: split audio and insert silence

WARNING: ... (ellipsis) is NOT a reliable pause — it can be vocalized as a word/sound. Do not use ellipsis as a pause mechanism.

Pronunciation Control

Phonetic spelling (any model): Write words as you want them pronounced:

  • Janus → Jan-us
  • nginx → engine-x
  • Use dashes, capitals, apostrophes to guide pronunciation

SSML phoneme tags (flash only):

<phoneme alphabet="ipa" ph="ˈdʒeɪnəs">Janus</phoneme>

IPA in v4: no tags needed. Wrap the IPA in slashes and double quotes in place of the word:

The term "/ˌbaɪoʊˈkemɪstri/" refers to the study of chemical processes.

Iterative Workflow

  1. Generate → listen → identify pronunciation/pacing issues
  2. Adjust: phonetic spellings, break tags, voice settings
  3. Regenerate. If pauses aren't precise enough, add silence in post with ffmpeg rather than fighting the TTS engine.

Speech to Text (Scribe)

python
with open("voiceover.mp3", "rb") as f:
    result = client.speech_to_text.convert(file=f, model_id="scribe_v2", tag_audio_events=False)
words = [w for w in result.words if w.type == "word"]  # .text, .start, .end (seconds)

scribe_v2 is current and the toolkit default. scribe_v1 is deprecated, with removal announced for 2026-07-09 (changelog), but still answers, so --stt-model scribe_v1 (redub) and --model scribe_v1 (aligncaptions) still run it, with a warning. `scribev1experimental`, a 2025 preview of scribev1, is accepted with a warning and sent as scribe_v2. The toolkit uses Scribe in redub.py (transcribe + --sync word timing) and align_captions.py.

Voice Cloning

Instant Voice Clone

python
with open("sample.mp3", "rb") as f:
    voice = client.voices.ivc.create(
        name="My Voice",
        files=[f],
        remove_background_noise=True
    )
print(f"Voice ID: {voice.voice_id}")
  • Use client.voices.ivc.create() (not client.voices.clone())
  • Pass file handles in binary mode ("rb"), not paths
  • Convert m4a first: ffmpeg -i input.m4a -codec:a libmp3lame -qscale:a 2 output.mp3
  • Multiple samples (2-3 clips) improve accuracy
  • Save voice ID for reuse

Professional Voice Clone: Requires Creator plan+, 30+ min audio. See reference.md.

Sound Effects

Max 22 seconds per generation.

python
result = client.text_to_sound_effects.convert(
    text="Thunder rumbling followed by heavy rain",
    duration_seconds=10,
    prompt_influence=0.3
)
with open("thunder.mp3", "wb") as f:
    for chunk in result:
        f.write(chunk)

Prompt tips: Be specific — "Heavy footsteps on wooden floorboards, slow and deliberate, with creaking"

Music Generation

10 seconds to 5 minutes. Use client.music.compose() (not .generate()).

python
result = client.music.compose(
    prompt="Upbeat indie rock, catchy guitar riff, energetic drums, travel vlog",
    music_length_ms=60000,
    force_instrumental=True
)
with open("music.mp3", "wb") as f:
    for chunk in result:
        f.write(chunk)

Prompt structure: Genre, mood, instruments, tempo, use case. Add "no vocals" or use force_instrumental=True for background music.

Remotion Integration

Complete Workflow: Script to Synchronized Scene

VOICEOVER-SCRIPT.md → voiceover.py → public/audio/ → Remotion composition
        ↓                  ↓               ↓                 ↓
  Scene narration    Generate MP3    Audio files     <Audio> component
  with durations     per scene       with timing     synced to scenes

Step 1: Generate Per-Scene Audio

Use the toolkit's voiceover tool to generate audio for each scene:

bash
# Generate voiceover files for each scene
uv run tools/voiceover.py --scene-dir public/audio/scenes --json

# Output:
# public/audio/scenes/
#   ├── scene-01-title.mp3
#   ├── scene-02-problem.mp3
#   ├── scene-03-solution.mp3
#   └── manifest.json  (durations for each file)

The manifest.json contains timing info:

json
{
  "scenes": [
    { "file": "scene-01-title.mp3", "duration": 4.2 },
    { "file": "scene-02-problem.mp3", "duration": 12.8 },
    { "file": "scene-03-solution.mp3", "duration": 15.3 }
  ],
  "totalDuration": 32.3
}

Step 2: Use Audio in Remotion Composition

tsx
// src/Composition.tsx
import { Audio, staticFile, Series, useVideoConfig } from 'remotion';

// Import scene components
import { TitleSlide } from './scenes/TitleSlide';
import { ProblemSlide } from './scenes/ProblemSlide';
import { SolutionSlide } from './scenes/SolutionSlide';

// Scene durations (from manifest.json, converted to frames at 30fps)
const SCENE_DURATIONS = {
  title: Math.ceil(4.2 * 30),      // 126 frames
  problem: Math.ceil(12.8 * 30),   // 384 frames
  solution: Math.ceil(15.3 * 30),  // 459 frames
};

export const MainComposition: React.FC = () => {
  return (
    <>
      {/* Scene sequence */}
      <Series>
        <Series.Sequence durationInFrames={SCENE_DURATIONS.title}>
          <TitleSlide />
        </Series.Sequence>
        <Series.Sequence durationInFrames={SCENE_DURATIONS.problem}>
          <ProblemSlide />
        </Series.Sequence>
        <Series.Sequence durationInFrames={SCENE_DURATIONS.solution}>
          <SolutionSlide />
        </Series.Sequence>
      </Series>

      {/* Audio track - plays continuously across all scenes */}
      <Audio src={staticFile('audio/voiceover.mp3')} volume={1} />

      {/* Optional: Background music at lower volume */}
      <Audio src={staticFile('audio/music.mp3')} volume={0.15} />
    </>
  );
};

Step 3: Per-Scene Audio (Alternative)

For more control, add audio to each scene individually:

tsx
// src/scenes/ProblemSlide.tsx
import { Audio, staticFile, useCurrentFrame } from 'remotion';

export const ProblemSlide: React.FC = () => {
  const frame = useCurrentFrame();

  return (
    <div style={{ /* slide styles */ }}>
      <h1>The Problem</h1>
      {/* Scene content */}

      {/* Audio starts when this scene starts (frame 0 of this sequence) */}
      <Audio src={staticFile('audio/scenes/scene-02-problem.mp3')} />
    </div>
  );
};

Syncing Visuals to Voiceover

Calculate scene duration from audio, not the other way around:

tsx
// src/config/timing.ts
import manifest from '../../public/audio/scenes/manifest.json';

const FPS = 30;

// Convert audio durations to frame counts
export const sceneDurations = manifest.scenes.reduce((acc, scene) => {
  const name = scene.file.replace(/^scene-\d+-/, '').replace('.mp3', '');
  acc[name] = Math.ceil(scene.duration * FPS);
  return acc;
}, {} as Record<string, number>);

// Usage in composition:
// <Series.Sequence durationInFrames={sceneDurations.title}>

Audio Timing Patterns

tsx
import { Audio, Sequence, interpolate, useCurrentFrame } from 'remotion';

// Fade in audio
export const FadeInAudio: React.FC<{ src: string; fadeFrames?: number }> = ({
  src,
  fadeFrames = 30
}) => {
  const frame = useCurrentFrame();
  const volume = interpolate(frame, [0, fadeFrames], [0, 1], {
    extrapolateRight: 'clamp',
  });
  return <Audio src={src} volume={volume} />;
};

// Delayed audio start
export const DelayedAudio: React.FC<{ src: string; delayFrames: number }> = ({
  src,
  delayFrames
}) => (
  <Sequence from={delayFrames}>
    <Audio src={src} />
  </Sequence>
);

// Usage:
// <FadeInAudio src={staticFile('audio/music.mp3')} fadeFrames={60} />
// <DelayedAudio src={staticFile('audio/sfx/whoosh.mp3')} delayFrames={45} />

Voiceover + Demo Video Sync

When a scene has both voiceover and demo video:

tsx
import { Audio, OffthreadVideo, staticFile, useVideoConfig } from 'remotion';

export const DemoScene: React.FC = () => {
  const { durationInFrames, fps } = useVideoConfig();

  // Calculate playback rate to fit demo into voiceover duration
  const demoDuration = 45; // seconds (original demo length)
  const sceneDuration = durationInFrames / fps; // seconds (from voiceover)
  const playbackRate = demoDuration / sceneDuration;

  return (
    <>
      <OffthreadVideo
        src={staticFile('demos/feature-demo.mp4')}
        playbackRate={playbackRate}
      />
      <Audio src={staticFile('audio/scenes/scene-04-demo.mp3')} />
    </>
  );
};

Error Handling

tsx
import { Audio, staticFile, delayRender, continueRender } from 'remotion';
import { useEffect, useState } from 'react';

export const SafeAudio: React.FC<{ src: string }> = ({ src }) => {
  const [handle] = useState(() => delayRender());
  const [audioReady, setAudioReady] = useState(false);

  useEffect(() => {
    const audio = new window.Audio(src);
    audio.oncanplaythrough = () => {
      setAudioReady(true);
      continueRender(handle);
    };
    audio.onerror = () => {
      console.error(`Failed to load audio: ${src}`);
      continueRender(handle); // Continue without audio rather than hang
    };
  }, [src, handle]);

  if (!audioReady) return null;
  return <Audio src={src} />;
};

Toolkit Command: /generate-voiceover

The /generate-voiceover command handles the full workflow:

/generate-voiceover

1. Reads VOICEOVER-SCRIPT.md
2. Extracts narration for each scene
3. Generates audio via ElevenLabs API
4. Saves to public/audio/scenes/
5. Creates manifest.json with durations
6. Updates project.json with timing info

Popular Voices

  • George: JBFqnCBsd6RMkjVDRZzb (warm narrator)
  • Rachel: 21m00Tcm4TlvDq8ikWAM (clear female)
  • Adam: pNInz6obpgDQGcFmaJgB (professional male)

List all: client.voices.get_all()

Voice library (a particular accent or character)

The premade voices are mostly American. For another accent, search the shared voice library: free to search and to hear, and every voice has a preview MP3.

bash
curl -s -G https://api.elevenlabs.io/v1/shared-voices -H "xi-api-key: $ELEVENLABS_API_KEY" \
  --data-urlencode "search=irish" -d gender=female -d page_size=30   # each voice has voice_id, accent, preview_url
  • Search with `search=`. The accent= filter alone came back empty for irish, scottish and british.
  • Using a library voice through the API needs a paid plan. A free key gets 402 paid_plan_required.

The cheapest tier (Starter) is enough; the voice ID then works directly in --voice-id or a brand.

  • Don't clone a preview into Qwen3 or another model. The owner licensed the voice through ElevenLabs

(and is paid per use there), not for cloning elsewhere.

  • Qwen3 VoiceDesign is not a substitute for an accent. Irish, Scottish and Manchester briefs all came out

American. An accent has to come from a real recording.

  • Voices already picked by ear are in assets/voices/library-shortlist.json. Start there.

Brand voices

A brand's voice.json can hold several narrators (voices map plus default), one per video: voiceover.py --brand video-toolkit --voice bren. The brand's model, settings (stability, similarityBoost, style, speed) and maxWpm apply wherever the command line left its default. See docs/creating-brands.md.

For full API docs, see reference.md.

aus demselben Repository

Weitere Skills

Alle Skills
digitalsamba
Community

video-design

Visual design for videos - colour by act, type scale, chart and layout rules for 16:9 and 9:16 scenes, a design.md handoff that any design skill can fill, and a contact-sheet critique of the render. Use when designing or refining how a video looks (title cards, data scenes, slides, a whole video's palette), when a scene looks generic or busy, when choosing colours for a video, or when /design or /scene-review asks for visual work. Works on its own with a brand, or takes direction from a design s

Installationen / 1h
2
GitHub Stars
2651
Aktualisiert
17. Nov.
digitalsamba
Community

acestep

AI music generation with ACE-Step 1.5 — background music, vocal tracks, covers, stem extraction, audio repainting, and continuation for video production. Use when generating music, soundtracks, jingles, or working with audio stems. Triggers include background music, soundtrack, jingle, music generation, stem extraction, cover, style transfer, repaint, continuation, or musical composition tasks.

Installationen / 1h
4
GitHub Stars
2627
Aktualisiert
17. Nov.
digitalsamba
Community

blender

3D scenes, character acting and product shots from a shot spec, rendered in Blender, as a finished film, as clips for any other build (moviepy, a video job, an editor), or as elements of a Remotion video (transparent characters over a scene, a Remotion render on a 3D laptop, phone or floating card that the camera pulls back from, an imported 3D model on a studio stage), or a brand's opening and closing ident built from its SVG logo. A rigged cartoon cast described in JSON (build, face, hair, clo

Installationen / 1h
4
GitHub Stars
2627
Aktualisiert
17. Nov.
digitalsamba
Community

ideogram4

Prompting patterns for Ideogram 4 text-to-image — best-in-class in-image text rendering and exact color/layout control via structured JSON captions. Use when generating images that need legible on-image text (title cards, thumbnails, logos, signage, CTAs), precise brand colors, or controlled spatial layout. Triggers include title slide image, thumbnail with text, on-image text, legible text in image, brand color palette image, bounding-box layout, Ideogram.

Installationen / 1h
4
GitHub Stars
2627
Aktualisiert
17. Nov.