Original skill document
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
FFmpeg + OpenCV Integration Guide
Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in references/opencv-pipelines-and-libraries.md.
When to Use
- Decode with FFmpeg and process frames with OpenCV
- Encode OpenCV-generated/processed frames with FFmpeg
- Compare
cv2.VideoCapture, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord - Fix RGB/BGR color bugs or
(height, width)dimension-order bugs - Preserve audio while replacing processed video frames
- Build GPU-assisted video I/O around CPU OpenCV processing
Library Selection
| Need | Best option | Why |
|---|---|---|
| Simple local file | cv2.VideoCapture | Built-in and simple |
| Full FFmpeg format/protocol support | subprocess pipe | Exact CLI behavior |
| GPU video I/O | ffmpegcv | NVDEC/NVENC with OpenCV-like API |
| Network/RTSP streaming | VidGear | Threaded capture and stream helpers |
| ML batch loading | Decord | Fast random/batch frame access |
| Frame-level libav control | PyAV | Direct FFmpeg library access |
Critical Gotchas
- OpenCV is BGR. FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
- NumPy dimensions are `(height, width, channels)`. Pixel access is
img[y, x], notimg[x, y]. - Video filters can drop audio. Preserve or remux original audio intentionally.
- Release resources. Always close
VideoCapture, pipes, writers, and PyAV containers. - Rawvideo pipes require exact frame size.
width * height * channelsmust match-pix_fmt.
Minimal Pipe Patterns
FFmpeg to OpenCV using BGR frames:
python
cmd = [
"ffmpeg", "-i", input_path,
"-f", "rawvideo", "-pix_fmt", "bgr24", "-"
]OpenCV to FFmpeg using BGR frames:
python
cmd = [
"ffmpeg", "-y",
"-f", "rawvideo", "-vcodec", "rawvideo",
"-s", f"{width}x{height}", "-pix_fmt", "bgr24",
"-r", str(fps), "-i", "-",
"-c:v", "libx264", "-preset", "fast", "-crf", "23",
"-pix_fmt", "yuv420p", output_path
]Core Workflow
- Probe input dimensions, fps, duration, and audio streams.
- Pick the I/O library based on the selection table.
- Lock pixel format at boundaries (
bgr24for OpenCV pipes;yuv420pfor final H.264 output). - Process frames in generators/batches; avoid loading full videos unless they are small.
- Preserve or re-encode audio explicitly.
- Verify output duration, fps, resolution, codec, and A/V sync.
Reference Map
references/opencv-pipelines-and-libraries.md- Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.
Related Skills
ffmpeg-python-integration-reference- Type-safe parameters, colors, time unitsffmpeg-pyav-integration- PyAV API detailsffmpeg-hardware-acceleration- GPU decode/encode and filter pipelines
