Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop

Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, ort — and...

Dev.to · Jul 25, 2026

Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop

Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, ort — and you want to feed a video to a model. Step one is always the same: get the frames. And in Rust today, getting frames means one of two things. Either you shell out to Command::new("ffmpeg"), write PNGs to disk or parse -f rawvideo off a pipe, and read them back; or you reach for the low-level bindings and hand-write the send_packet / receive_frame decode loop — forty lines before the first pixel, and you own the EAGAIN handling and the YUV→RGB conversion yourself. The second path is hard enough that an entire cottage industry of "frame grabber" gists grew up around it: search "rust extract frame from video" and the authoritative answer is frequently someone's gist. A decade on, something this basic still has no comfortable standard answer.

This post is the third path: ez-ffmpeg 0.15's FrameExtractor pulls frames in the same process, in one call, and hands you tightly packed RGB bytes ready for an ndarray view or a tensor. You will get runnable code, a sampling-strategy cheat sheet, and two things that actually earn their keep: why your thumbnails may have been subtly wrong on color this whole time, and how fast this path really is — with numbers you can reproduce.

The two old ways: cross a process, or hand-write the decode loop

Show the ugly first. The shell-out path is cheapest when the output is a file, but the instant frames have to come back into memory they cross a process boundary — PNGs to disk and read back, or a raw pipe you slice into frames yourself. The hand-written decode loop keeps everything in-process, but the boilerplate runs like this: open the demuxer, find the video stream, build the decoder, pair up send_packet / receive_frame, service EAGAIN, then run each frame through swscale from YUV to RGB — every step its own opportunity to be wrong. And that last step hides a trap: many hand-rolled paths (and some libraries) apply BT.601 conversion coefficients to HD video too, which skews saturated reds and greens visibly — with no error to tell you. More on that trap below.

Neither path is wrong. When the output is a file and the recipe exists, the CLI is still the right answer; when you need packet-level control, the raw bindings are irreplaceable. They just never made the specific thing — get a batch of frames, in-process, in one line, with the color right — comfortable.

FrameExtractor: a batch of frames in one line

One dependency line (it still links libav through ffmpeg-next, so FFmpeg 7.1–8.x has to be installed):

[dependencies]
ez-ffmpeg = "0.15"   # needs FFmpeg 7.1-8.x installed (links libav)

Enter fullscreen mode Exit fullscreen mode

Pull 32 frames to feed a VLM. Note UniformN — it takes N frames spread uniformly across the whole clip by presentation time, the primitive fixed-budget CLIP/VLM pipelines actually want:

use ez_ffmpeg::frame_export::{FrameExtractor, Sampling};

// 32 frames evenly spread over the full duration, scaled to 224 wide
// (height derived from the aspect ratio).
let frames = FrameExtractor::new("input.mp4")
    .sampling(Sampling::UniformN(32))
    .width(224)
    .collect_frames()?; // Vec<VideoFrame>

for f in &frames {
    // f.as_bytes() is tightly packed, top-down RGB24 with no row padding —
    // width*height*3 bytes, ready for an ndarray view or a tensor.
    let (w, h) = (f.width(), f.height());
    let rgb: &[u8] = f.as_bytes();
    // your_preprocess(rgb, w, h) ...
    println!("frame #{:>2} pts={:?}us {}x{}", f.index(), f.pts_us(), w, h);
}

Enter fullscreen mode Exit fullscreen mode

Three things worth pointing at. new(input) takes a path, a URL, or anything convertible into an Input; the packed layout defaults to Rgb24 and also offers Rgba32 / Gray8. width(224) sets only the width and derives the height from the source aspect ratio (internally scale=224:-2); pin the other side with height, or give both for an exact size. collect_frames() gathers a Vec; swap in frames() for a streaming iterator (Iterator<Item = Result<VideoFrame>>) that decodes and yields as you consume, without holding every frame in memory — reach for it when you export dense frames from a long file.

The shape of this API is not incidental — it deliberately tracks the FFmpeg CLI. The builder chain reads like a command; start_time_us / duration_us map to -ss / -t (seconds become microseconds); the filter graph it assembles internally is the same scale=…,format=rgb24 shape you would type on the command line, plus explicit color-management parameters. That "move the CLI into your Rust process unchanged" throughline is an argument this series' ecosystem survey makes at length; here it is enough to say your command-line muscle memory stays valid.

Sampling: how many frames, and which ones

Half of frame extraction is deciding which frames. Sampling lays the common strategies out:

Strategy	Meaning	Typical use
`All`	every decoded frame (default)	dense export, per-frame analysis
`EveryNth(n)`	one frame in every n	decimate by frame count
`EverySec(k)`	one frame per k seconds (float)	timeline scrubbing, preview strips
`KeyframesOnly`	keyframes only	shot/scene proxies, fast at decode time
`UniformN(n)`	exactly n frames spread by time	VLM / CLIP fixed-budget input

Two deserve a note. KeyframesOnly pins the decoder option skip_frame=nokey, so non-keyframes are skipped during decode — not decoded and then discarded, not decoded at all — which is where its speed comes from. UniformN(n) guarantees exactly n frames: an input too short to hold n distinct frames pads by repeating nearby frames (each repeat keeps its source frame's pts_us), so the "give me exactly 32" a model pipeline was promised is always honored, with no padding logic on the caller's side.

Color: HD converts as BT.709 by default, and that is not a footnote

Back to the color trap. YUV→RGB needs a coefficient set: SD video uses BT.601, HD video uses BT.709, and picking the wrong one skews saturated colors. Many "decode to RGB" shortcuts apply BT.601 to everything — fine for SD, wrong for 1080p, and silent about it; you only see it if you put the result on screen.

FrameExtractor converts per the frame's own color tags by default (ColorPolicy::Tagged): an HD frame tagged BT.709 converts as BT.709 — no guessing, no one-size-fits-all. This is the behavior in the module I am most willing to stand behind. For contrast: to_ndarray('rgb24') in older PyAV releases applied BT.601 across the board (that qualifier matters — newer PyAV changed it; do not quote it as current). For untagged frames there is TaggedOrResolutionGuess (guess by resolution: BT.709 at height ≥ 720), or Force { matrix, range } to pin one interpretation for the whole run. Most people never touch the default tier — but knowing it gets this right for you is worth the paragraph.

Who should not use it

The usual rule: point outward first, not just at my own crate.

The thumbnail is a file on disk, the recipe exists, and you convert once. ffmpeg -i in.mp4 -vf "select=..." -vsync vfr out_%03d.png is a one-liner; pulling in a crate for it is a detour.
You want frames but will not touch FFI or linking, and a subprocess is fine. ffmpeg-sidecar (~131K downloads/month) wraps any video as an RGB frame iterator, links no libav, and is genuinely pleasant — at the cost of an ffmpeg binary at runtime and frames crossing a process boundary.
Video only, WIP acceptable. video-rs iterates RGB ndarrays in ten lines (~32K downloads/month); it has no audio API and self-describes as work-in-progress, but it is strong at its single subject. To be clear: Rust has never been unable to hand you decoded frames — several paths can. FrameExtractor's difference is doing it in-process, with sampling policies and color management, in one line.
You want decord-style GPU batch decoding. That is a different capability tier; no parity is claimed.

And three plain facts to close on. One: the compile-and-link pain, ez-ffmpeg removes none of it — it still links libav, and the linking gauntlet is unchanged. Two: the project is ~340 stars, still early-stage, and a "safe Rust API" is not the same as "no C CVEs" — libav is underneath. Three: the whole frame_export module — FrameExtractor along with its siblings SampleExtractor (audio → 16 kHz PCM) and VideoWriter (frame-push encoding) — is still marked experimental in the docs: 0.15 settled the behavioral semantics (default precision, sampling), but the API shape may still shift.

Closing

Frames never leave the process, one call gets them, the color follows the tags, and the conversion stays on swscale's fast path — the decade-old "render to images, stitch with the CLI" cottage industry can retire. If you are doing ML in Rust, extracting frames should not send you back out to a subprocess.

Runnable examples are all in the repo: examples/uniform_thumbnails (UniformN into a 4×3 contact sheet), examples/extract_rgb_frames (RGB to PPM), examples/keyframe_thumbnails (keyframe proxies), examples/frame_sampling (the sampling strategies side by side). The crate lives at github.com/YeautyYE/ez-ffmpeg.

An open question I would genuinely like answered: in your Rust ML pipeline today, is frame extraction a shell-out, a hand-written decode loop, or something else? And if I've gotten anything above wrong, tell me and I'll fix it.

Read the original source

#Tooling