Pushing your own frames into an FFmpeg pipeline in Rust (headless frame source)

ez-ffmpeg is a Rust crate that runs FFmpeg pipelines inside your process — linked libav libraries behind a high-level API, no subprocess (and no relation to the JavaScript "Ez FFmpeg" project). The narrow claim of this post: its VideoWriter (added in 0.14) takes frames you render in Rust — one tightly packed byte buffer per frame — and pushes them through a real filter → encode → mux pipeline into an MP4, shown below as a complete listing that compiles, runs, and is ffprobe-verified verbatim. It's on crates.io and GitHub.

Here's how old this itch is. In 2015, the original rust-ffmpeg binding received its issue number one (meh/rust-ffmpeg#1). The author, jaredly, wanted the most ordinary thing: he had frames drawn in code and wanted them encoded into a video file. He "couldn't find a way to: open a file as writer; encode video frames to it". His workaround: "rendering to a bunch of images and then using ffmpeg on the cli". A decade later, Rust code that renders, simulates, or visualizes still mostly ships that 2015 workaround.

The workaround, costed

Here is the image-sequence detour in its modern form (PPM keeps it dependency-free — a three-line text header over raw RGB is a legal image):

// Step 1: render every frame to its own image file on disk.
let mut rgb = vec![0u8; (WIDTH * HEIGHT * 3) as usize];
for i in 0..120 {
    render_gradient_rgb(&mut rgb, WIDTH, HEIGHT, i as f32 / 30.0);
    let mut f = File::create(format!("frame_{i:04}.ppm"))?;
    write!(f, "P6\n{WIDTH} {HEIGHT}\n255\n")?; // PPM needs no image crate
    f.write_all(&rgb)?;
}

// Step 2: hope the ffmpeg on PATH exists and agrees with your flags.
let status = Command::new("ffmpeg")
    .args(["-y", "-framerate", "30", "-i", "frame_%04d.ppm",
           "-c:v", "mpeg4", "-q:v", "5", "images.mp4"])
    .status()?;

Enter fullscreen mode Exit fullscreen mode

It works — I ran it, 120 frames in, 120 frames out. It also taxes you three ways. Disk: a 640×360, 30 fps, 4-second animation writes 120 PPM files totalling 80 MB of intermediates for a final MP4 of 213 KiB, and the intermediate size scales with resolution and duration. Naming: frame_{i:04} and frame_%04d are the same implicit contract written in two languages; change the zero-padding on one side and ffmpeg silently stops at the first filename that doesn't match. Failure surface: step one fails as a Rust Result, step two fails as an exit code plus stderr text — "encoder not found" and "frame 37 is corrupt" both arrive as log lines for you to fish out.

Fairness demands the third old way on the table too: skip the disk entirely and pipe raw frames into ffmpeg's stdin — ffmpeg -f rawvideo -pix_fmt rgba -s 640x360 -framerate 30 -i - out.mp4. Zero temp files, zero naming contract, and a full pipe blocks the writer, so backpressure comes free (ffmpeg-sidecar wraps exactly this pattern). That kills the disk tax — but the other two stand: failures are still an exit code plus stderr text, the right ffmpeg binary still has to exist at runtime, and a structurally invalid filter graph only blows up mid-run instead of being rejected up front. The API below earns its keep on that second half — typed errors, open()-time validation, no subprocess lifecycle — not on saving temp files.

The in-process version

your render loop              VideoWriter pipeline (same process)
┌────────────┐ write()/      ┌─────────────┐   ┌────────┐   ┌─────────┐
│ render(buf)│─write_owned──▶│bounded queue│──▶│ filters│──▶│ encoder │─▶ gradient.mp4
└────────────┘ blocks if full└─────────────┘   └────────┘   └─────────┘
   a frame-source worker sits where a decoder would; end = explicit in-band EOF

Enter fullscreen mode Exit fullscreen mode

Same animation, pushed straight into the pipeline. One dependency:

[dependencies]
ez-ffmpeg = "0.15"   # needs FFmpeg 7.1-8.x installed (links libav)

Enter fullscreen mode Exit fullscreen mode

The complete listing — copy it whole:

use ez_ffmpeg::{Output, VideoWriter};

const WIDTH: u32 = 640;
const HEIGHT: u32 = 360;
const FPS: i32 = 30;
const SECONDS: i32 = 4;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Name the encoder explicitly. With a bare "gradient.mp4" the linked
    // FFmpeg build picks the container default (H.264 when libx264 is
    // compiled in, otherwise mpeg4 at a low default bitrate).
    let output = Output::from("gradient.mp4")
        .set_video_codec("mpeg4")
        .set_video_qscale(5);

    let mut writer = VideoWriter::builder(WIDTH, HEIGHT)
        .pixel_format("rgba") // the default, spelled out
        .fps(FPS, 1)
        .open(output)?;

    // The writer tells you the exact byte count of one tightly packed frame.
    let mut frame = vec![0u8; writer.frame_size()];

    let total = FPS * SECONDS;
    for i in 0..total {
        render_gradient(&mut frame, WIDTH, HEIGHT, i as f32 / FPS as f32);
        writer.write(&frame)?; // copies the slice; `frame` is reusable
    }

    writer.finish()?; // drains the encoder, writes the trailer, reports errors
    println!("wrote {total} frames to gradient.mp4");
    Ok(())
}

/// Fills `buf` (tightly packed RGBA) with a gradient that drifts right over time.
fn render_gradient(buf: &mut [u8], width: u32, height: u32, t: f32) {
    for y in 0..height {
        let fy = y as f32 / height as f32;
        for x in 0..width {
            let fx = x as f32 / width as f32;
            let idx = ((y * width + x) * 4) as usize;
            buf[idx] = (((fx + t * 0.25) % 1.0) * 255.0) as u8; // red wave, drifting
            buf[idx + 1] = (fy * 255.0) as u8; // green: vertical ramp
            buf[idx + 2] = ((1.0 - fx) * 255.0) as u8; // blue: horizontal ramp
            buf[idx + 3] = 255; // opaque
        }
    }
}

Enter fullscreen mode Exit fullscreen mode

Two honest caveats first. Installing and linking FFmpeg is not solved here — the crate sits on the libav libraries, FFmpeg 7.1–8.x must be present, and that pain is unchanged; what disappears is the subprocess, the 80 MB of intermediates, and the stderr fishing, not the FFmpeg dependency itself. And this API is marked experimental in the docs: new in 0.14, its surface may still be refined in minor releases.

Call by call:

• Output::from("gradient.mp4").set_video_codec("mpeg4").set_video_qscale(5) — the destination is the crate's ordinary Output, so encoder, container, bitrate, codec options, and format options (movflags…) are all configured at that layer. mpeg4 keeps this listing runnable on FFmpeg builds without libx264; swap in set_video_codec("libx264") for H.264. Do name an encoder: with a bare path the linked build picks the container default, which on a libx264-less build means mpeg4 at a low default bitrate.
• VideoWriter::builder(WIDTH, HEIGHT) — width and height are the only required parameters, positional so they cannot be forgotten.
• .pixel_format("rgba") — the default, spelled out. Any non-hardware AVPixelFormat name works (rgb24, gray8, yuv420p, nv12, …); frames are tightly packed, planes concatenated in descriptor order (yuv420p = all of Y, then U, then V). Hardware formats like cuda are rejected with a typed error — a CPU byte buffer cannot fill them.
• .fps(FPS, 1) — constant frame rate as a rational; NTSC rates like fps(30000, 1001) work. Every written frame advances exactly den/num seconds.
• .open(output)? — validates the configuration (dimensions, fps, pixel format, filter shape, and that the output actually consumes video), builds the pipeline, and starts it without waiting for a first frame. An unknown encoder name fails right here.
• writer.frame_size() — the exact byte count one frame must have: av_image_get_buffer_size(pix_fmt, w, h, 1). A wrong-sized buffer gets a typed InvalidSize { expected, got }, not a garbled picture.
• writer.write(&frame)? — pushes one frame by copying the borrowed slice, so the same allocation serves the whole loop. Blocks when the internal queue is full; more on that below.
• writer.finish()? — closes ingress, drains the encoder, finalizes the container, and returns the pipeline's first error. This is the result path.

How it works: the four contracts

Ownership: write copies, write_owned moves. From the verification build, the contrast in twelve lines:

    // write(): you keep the buffer. The slice is copied into the pipeline,
    // so one allocation serves the whole run.
    let mut frame = vec![0u8; writer.frame_size()];
    for i in 0..2u8 {
        frame.fill(i * 100);
        writer.write(&frame)?;
    }

    // write_owned(): the Vec moves into the pipeline — no borrow-copy on the
    // caller side. Use it when each frame is born as its own Vec anyway.
    for i in 2..4u8 {
        let owned = vec![i * 60; writer.frame_size()];
        writer.write_owned(owned)?;
    }

Enter fullscreen mode Exit fullscreen mode

Both paths still pay one copy at the far end: the frame-source worker fills the bytes plane-by-plane into a pooled, aligned AVFrame — that copy is not skippable. Two corners worth knowing: a Vec handed to write_owned is queued as-is, so spare capacity beyond its length stays allocated while it waits; and write takes &mut self, so the total frame order is fixed at compile time — the writer is Send (move it to a dedicated producer thread) but deliberately not Sync.

There is no demuxer and no decoder behind this. The builder assembles a pipeline whose filtergraph (buffersrc → [your filter chain or null] → buffersink) is fed by a frame-source worker sitting exactly where a decoder would sit. End of stream is an explicit in-band marker, which is why filters that buffer everything (reverse, tpad) still flush correctly at the end.

Backpressure: the queue is bounded, write waits. The ingress queue is bounded by frame count, defaulting to max(1, min(4, 64 MiB / frame_size)) — four frames of 1080p RGBA, two of 4K RGBA. When rendering outruns encoding, write blocks instead of buffering your animation in RAM. A pipeline that dies mid-run does not leave you parked: a blocked write re-checks pipeline status every 100 ms and returns PushError::PipelineClosed. That error is a signpost, not a verdict:

    let mut pushed = 0;
    loop {
        let frame = vec![0u8; writer.frame_size()];
        match writer.write_owned(frame) {
            Ok(()) => pushed += 1,
            // Not an error by itself: the pipeline stopped taking frames.
            // finish() holds the verdict — the real failure, or Ok when the
            // job simply completed (as the frame limit does here).
            Err(PushError::PipelineClosed) => break,
            Err(e) => return Err(e.into()),
        }
    }

Enter fullscreen mode Exit fullscreen mode

Concretely: put set_max_video_frames(3) on the Output and the pipeline completes itself after three frames — subsequent writes report PipelineClosed, finish() returns Ok, and the file holds exactly 3 frames (verified). If instead an encoder genuinely fails mid-stream, the same finish() hands you the real error. The test suite pins this: worker_failure_unblocks_write_and_finish_reports opens an encoder that rejects the frame size only at first-frame time, and asserts that write unblocks and finish reports — the writer never hangs against a dead consumer.

Timing: constant frame rate, no per-frame PTS. That is the explicit v1 scope: no variable frame rate, no audio. What you do get is a correct tail: 120 frames at 30 fps probe as duration 4.000000, not a clipped 119/30 — the explicit EOF marker closes the stream at the accumulated frame-end time, so the last frame keeps its full duration (test: eof_preserves_final_frame_duration, covered for integral and NTSC rates).

Teardown: finish is the verdict, Drop is the safety net, abort is the discard. finish() is the only path that returns the Result. Forgetting it loses no frames: Drop performs the same close-drain-finalize (test: drop_without_finish_keeps_all_frames — push 10, drop the writer, the file probes 10), but an error is only logged, and the drop can block while the encoder drains. When you don't want the file at all, abort() discards the export; the partial output is not guaranteed playable (test: abort_returns_cleanly).

One more gate worth knowing about: filter_desc accepts any single-input single-output video chain ("hue=s=0" verifiably turns solid red frames gray; "pad=ceil(iw/2)*2:ceil(ih/2)*2" is the odd-dimensions remedy), and graphs that cannot work are rejected at open() with typed errors instead of surfacing as a mid-run hang — a stranded pad, a disconnected graph, or an output unreachable from the input each get their own error. The program output for filter_desc("split"), verbatim:

rejected at open(): Video writer error: filter_desc must have exactly one video input pad and one video output pad; found 1 input pad(s) (1 video) and 2 output pad(s) (2 video)

Enter fullscreen mode Exit fullscreen mode

Compositing over generated sources is still allowed when the pushed frames reach the output — "color=c=red:s=64x48[bg];[in][bg]overlay=shortest=1" runs and ends on input EOF, the same rule the CLI applies.

Run it

The listing above, then ffprobe, output verbatim:

$ ffprobe -v error -select_streams v:0 -show_entries \
    stream=codec_name,width,height,r_frame_rate,nb_frames,duration \
    -of default=noprint_wrappers=1 gradient.mp4
codec_name=mpeg4
width=640
height=360
r_frame_rate=30/1
duration=4.000000
nb_frames=120

Enter fullscreen mode Exit fullscreen mode

120 frames in, 120 frames out, duration exact. No intermediate files, no subprocess, typed errors end to end.

What the ecosystem already had

'Rust can't encode your own frames into an MP4' is false — video-rs's README literally encodes rainbow.mp4 from ndarray frames (and video-rs is itself FFmpeg-based, via ffmpeg-next), opencv-rust has VideoWriter, and openh264 + the mp4 crate do it in pure Rust. So the defensible claim is narrower and concrete: what this API adds over those is the contract work — filter chains validated at open() instead of failing mid-run, explicit backpressure and teardown semantics, typed errors end to end — plus an output surface beyond files. Not "nobody else can encode frames."

The filter-graph validation you saw above. On live targets, this post's evidence stops at the API surface: the module documentation lists RTMP targets among supported destinations, and an Output::from("rtmp://…").set_format("flv") writer session compile-checks — but I did not run one against a live server here, so treat that half of the claim as documented-and-type-checked, not exercised.

When you should not use this

The split is by scenario, and each path has its win. The images already exist on disk and you convert once: ffmpeg -framerate 30 -i frame_%04d.png out.mp4 in a terminal wins, and writing a Rust program for it is a detour. You need GStreamer-grade live pipeline orchestration — dynamic topology, multi-branch fan-out, fine-grained clocking: the gstreamer crate's mature pipeline model is the right tool, not this. You need variable frame rate, per-frame PTS, or an audio track muxed in: v1 does not cover any of those (video only, constant rate; stream maps are rejected; and since pushed frames are raw, video stream-copy does not apply — frames are always encoded). And if your frames come out of another video file rather than your own code, that's a transcode — the crate's ordinary Input/Output job covers it without VideoWriter.

What's left — render loops, simulation output, procedural video, anywhere the frames are born in your code and the destination is a file — is exactly what this API is for.

What the first build costs

The prerequisite is the same one the whole crate carries: FFmpeg 7.1–8.x installed where the linker can find it, because ez-ffmpeg links the libav libraries rather than shelling out. That cost predates this crate and survives it; budget your first cargo build accordingly. The writer itself opens no input format context, so there is no version-specific probing behavior across supported FFmpeg releases.

Where this leaves you

Your render loop now emits an MP4 (or MKV, or anything the linked build muxes) directly, with a filter chain if you want one, backpressure managing memory, and failures arriving as typed Results. In the repository, examples/frames_to_video (a plasma animation) and examples/bouncing_balls (a gravity toy with real per-frame state) run as-is: github.com/YeautyYE/ez-ffmpeg.