Our FFmpeg Alternative Stopped Reading the Whole File to Open a Recording

Our FFmpeg alternative stopped reading the whole file before it would tell you what a recording is. It is the house media tool in remade_ffmpeg_rs: the demuxers and decoders a home computer uses when a camera or a microphone has already written a file. Until 16 October 2026, opening that file meant copying it into RAM, and for Matroska, AVI, and Ogg it meant copying it twice. Probing a long porch recording could ask for as much memory as the file, or double. The rewrite streams. The bytes that come out of a well-formed file stay the same.

This page is the record of that change, measured on Windows in a release build with rusty_alloc. It is not a crates.io release. The commits sit on the working branch that closes those issues. Remade With Rust is the family the crates belong to. The machine that should open the file, when the wire exists, is the MATA home computer.

What reading the whole file was doing

A demuxer has two jobs. read_header learns the streams. read_packet hands the next chunk to a decoder. The old header path did both at once: it called read_to_end, then walked every sample while that buffer was still alive.

Two copies, then a queue of packets

Matroska, AVI, and Ogg were the expensive shape. The header read the input, then the packet walk allocated an owned copy of every block before the first packet was returned. Peak memory was about twice the file, so a 20 GB MKV wanted on the order of 40 GB just to open. MP4 was one copy: the whole file, then slices. MP4 cannot be fixed by streaming alone. The usual layout puts moov, the index, after mdat, the media, so the index is at the end. WAV, Y4M, IVF, MP3, and FLAC each held the payload and, for FLAC, emitted the entire file as one packet.

That is a host problem, not a chip problem. The ESP32 camera page in rusty_esp_video still drops frames instead of buffering them on the board. The laptop that checks that page is where reading the whole file used to hurt, whether the check used FFmpeg or our FFmpeg alternative. A pure Rust alternative to CameraWebServer is the page those tools read.

The numbers from the probes

Same machine, release, rusty_alloc, peak working set of the probe:

• A 239 MiB 1080p MKV: 516 MiB down to 6.2 MiB, and 283 ms down to 12 ms.
• A 239 MiB AVI: 522 MiB down to 6.2 MiB, and 372 ms down to 96 ms.
• A 239 MiB MP4 with H.264 and Opus, moov at the end: 246 MiB down to 7.3 MiB, and 119 ms down to 17 ms.
• A 220 MiB WAV: 226 MiB down to 6.2 MiB, and 250 ms down to 49 ms.
• A 15 MiB FLAC probe: 21.6 MiB down to 6.2 MiB.

A stream copy of the MP4, the MKV, the AVI, and the WAV was byte-identical before and after. Reading the whole file was not buying a different picture. It was buying a bigger process.

How the containers stopped reading the whole file

Each library kept the old reader as a test-only oracle, then compared streams and packets. Header reads of a few-megabyte input stay under 64 KiB. Truncated and garbage inputs are required to finish without a panic.

MP4 seeks; Matroska, AVI, and Ogg stream

On a seekable file, the MP4 demuxer walks top-level box headers, seeks past mdat, reads moov, and fetches each sample by offset. Contiguous samples do not seek again. A declared sample size is not an allocation. A pipe, a socket, or HLS cannot seek, so those inputs still buffer the MP4: a trailing index is unreachable until the media has passed. Local files now open as a buffered file and take the seekable factory. URLs, RTP, and HLS stay streams.

Matroska and WebM read from a plain Read through a 64 KiB buffer. The header takes Info and Tracks, capped, and stops at the first Cluster. Packets walk cluster children one at a time. nb_frames is now empty, which is what ffprobe reports for Matroska too, because a count of every packet is the whole file again. A second unknown-size Cluster no longer drops the packets after it. Info that appears after the first Cluster is not seen: a one-pass reader cannot go back.

AVI walks top-level chunks until LIST movi, reads the first header list, and then yields one sub-chunk per packet. A file cut off inside movi now returns every complete chunk before the cut. It used to return none. Ogg reads a page at a time. Only the first decodable Opus or Vorbis stream is exposed. The old demuxer interleaved every stream into the one stream it declared, so a decoder was fed another stream's bytes. Page CRCs are still not checked. That limit is written down, not patched over in this note.

FLAC frames, and the boxes that were one buffer

FLAC used to emit one packet for the file. It now reads metadata in the header and one frame per packet, with a CRC-16 so a sync pattern inside frame data cannot split a frame. The CRC runs slice-by-8. On a 68 MiB file that cut demux CPU from about 250 ms to about 115 ms. Decoding that file to WAV stayed byte-identical, and the CPU of the decode sat inside the noise of the old path. A 34 MB guitar file still reports 3,952 frames, matching ffprobe. Bytes after the last frame, such as an ID3v1 tag, are ignored instead of failing the decode.

WAV walks the header without holding LIST, JUNK, and fact chunks, and stops at data. Packets are blocks of whole frames. A data size of zero means read to EOF. RF64 is still unsupported. Y4M and IVF read a short header, then one frame at a time. MP3 scans a small window, skips a leading ID3v2 tag by its declared size without holding it, and skips trailing ID3v1 and APEv2 tags instead of turning a sync-like pattern in a tag into a fake frame. Duration for MP3 stays empty. It was never computed.

What the debug suites were rejecting

Two other fixes landed in the same tree, and neither is a new picture format. They stop debug builds from treating valid input as a crash.

HEVC asserts that were stricter than the spec

Release builds of rusty_h265 were already bit-exact on 147 JCT-VC version-1 streams. Debug builds panicked. Four debug_assert! checks stated preconditions the code is built not to need. A horizontal deblock edge does not reach four pixels to the left, but the SIMD check required that for both directions. The QP wrap bound was documented as 0..2m and conformance hits a value past that, still inside the loop that actually runs. Intra and SAO paths fall back to a scalar twin on purpose past a lookup-table's headroom. The asserts called that fallback a bug. Debug conformance now completes. The picture was already right in release.

AV2, AVIF, and what is still open

An empty AV2 packet used to reach a debug check that aborts, outside catch_unwind. The adapter now returns an error before the engine sees an empty or oversized packet. A still-picture overflow in debug is a known upstream subtract that wraps to the right address in release, so overflow checks are off for that one dependency until the fix is published. The still-picture test stays byte-identical. On ARM, the published AV1 asm feature does not build, so ARM uses the pure-Rust kernels. x86-64 still uses asm.

Ogg still does not check page CRCs. RF64 WAV is unsupported. A non-seekable MP4 still reads the whole file. The home computer's live view is not wired yet: the MATA Home integration says so. The LAN listen is rusty_esp_audio, and a pure Rust alternative to ESP-ADF is that comparison. Smart home access on your home computer is where those files land.

Our FFmpeg alternative stopped reading the whole file so a probe can name the streams without becoming the archive. The See vision is the camera half. The Hear vision is the microphone half. The Learn index lists what is timed on a board. The language is Rust. The oracle for the same bytes remains FFmpeg.