sound-design

Sound design

Most AI-made motion graphics are silent. Sound is what makes a piece feel produced.

The sfx/ engine synthesizes every sound in pure Python stdlib: no samples, no licenses, no installs, same output on every run. It reads the same timeline JSON as the picture, so sound can't drift out of sync.

How it fits together

python3 -m sfx --list                   # generators with what each is for
python3 -m sfx sfx/cues/<video>.py      # render → mastered WAV

Writing a cue sheet for a new video

Create sfx/cues/<video>.py:

from sfx.synth import click, pop, whoosh, chime, riser, impact

TIMELINE = "timeline.json"      # the same file the composition reads
OUT = "public/sfx.wav"
TARGET_LUFS = -14.0             # optional, this is the default

def score(mix, tl):
    s1 = tl["s1"]
    mix.put(s1["title"], whoosh(0.4, 500, 2400, peak=0.45), 0.12)
    for i, t in enumerate(s1["items"]):
        mix.put(t, pop(900 + i * 150, 400), 0.3, pan=-0.4 + i * 0.3)

• mix.put(t, samples, gain, pan): t always comes from tl, never a typed number. Offsets from a beat (s1["exit"] + 0.05) are fine.
• mix.typing(start, end, chars): one humanized keystroke per character.
• mix.put_stereo(t, left, right, gain): for stereo sources like pad().
• Mastering is automatic: the engine soft-clips and sets gain to hit TARGET_LUFS, raising drive only as far as needed to keep the true peak ≤ −2 dBFS (room for AAC overshoot, so the MP4 stays under −1 dBTP). A mix that's mostly short hits with silence between can't reach −14 without crushing; the engine then comes out quieter and says so. Fix it with sustained sound (a pad, longer tails), not by forcing it.
• Add an "sfx" script in package.json pointing at the new cue sheet, or run it directly.

sfx/cues/effort.py is a complete, real example: 128 sounds plus a pad bed.

Visual event → sound

Accent, don't narrate: not every motion needs a sound. Continuous motion (drift, grain, rotation) stays silent.

Levels and space

• Hero hits: gain 0.25–0.35. Supporting sounds: 0.08–0.2. Pad: ~0.03, ducked ~65% under the hero transition.
• Pan follows on-screen position (−1 left … +1 right). Sequences sweep across the stereo field.
• Pick one key and keep every tonal sound in it; pitch rises with progress and resolves on the payoff. (In the effort example the pad and slider notes are D major, but the scene 3 pops and chime are still in C major. Don't copy that.)

Adding a generator

Write a function in sfx/synth.py that returns a list of floats at SR. Put what it's for in the first docstring line; --list shows it. Use the module-level random for noise so renders stay reproducible. Build on bandnoise() for anything airy.

Verify

npm run sfx                              # or python3 -m sfx sfx/cues/<video>.py
npm run wave                             # out/wave.png + loudness of the WAV
npm run build
node scripts/wave.mjs out/effort.mp4     # loudness of what you'll actually post

In out/wave.png every coral line is a timeline beat. Each transient should sit on a line or a hair after it. A hit between lines means a hardcoded time or a stale WAV. Pass --timeline for a video with its own timeline file.

Optional: generated audio

For voiceover or sounds you can't synthesize, the ElevenLabs MCP (see .mcp.json.example) can generate files into public/. Place them with <Audio src={staticFile(...)} /> inside a <Sequence from={beat * FPS}>.