ESP32 Camera Capture in Rust: Frame Pools, JPEG, Sensor Tables

rusty_esp_image is the part of Janus that turns an ESP32 camera sensor into a clean source of frames: it brings the sensor up from register tables, captures into a fixed frame pool, hands out borrowed frames, and probes or encodes JPEG, all in memory-safe Rust. On a Seeed XIAO ESP32-S3 Sense, two frame buffers produced 27.764 fps at 320×240, and 581 captures across six runs recorded zero failures, including runs where the pool was starved on purpose.

This field note covers the capture seam, the frame pool, the JPEG tools, the pixel kernels and the sensor tables, with an honest line on each about what ran on silicon. It sits under the See pillar, next to the streaming side in rusty_esp_video.

The ESP32 camera capture seam

Every ESP32 camera in the family, real or simulated, answers the same small trait. That is the seam: code above it does not care whether pixels came from an OV2640, an OV5640 or a test pattern.

One trait, borrowed frames

The core idea is that grab writes into memory the caller owns and returns a view that borrows it. No frame is allocated per capture and nothing is copied twice. On ESP-IDF (Track A), IdfCamera wraps Espressif's esp32-camera driver exactly once, behind that trait, and copies one frame per grab out of the driver's PSRAM buffer.

use rusty_esp_image::prelude::*;

let mut camera = IdfCamera::new(XIAO_ESP32S3_SENSE, Mode::new(FrameSize::Qvga, 12))?;
if let Some(frame) = camera.grab()? {
    let info = jpeg::probe(frame.bytes())?;      // width, height, components
    let clean = jpeg::find_eoi(frame.bytes());   // trim trailing DMA padding
}

A deterministic TestPattern implements the same trait, so a whole ESP32 camera pipeline can be exercised on a laptop with no sensor attached. The pure esp-hal backend for the S3 (Track B) is planned, not built.

The frame pool, measured on the chip

The ESP32 camera capture pool is a fixed number of slots over a caller slice, with no heap. The measurement that matters is how many slots you need:

• 1: fps, fast consumer: 13.882
• 2: fps, fast consumer: 27.764
• 3: fps, fast consumer: 27.764

With one buffer the driver cannot fill the next frame while you hold the current one, so it misses every second frame: a factor of 2.000003 from the raw counts. The third buffer bought nothing, the two elapsed times agreeing to under one part per million. Ask for two and keep the PSRAM.

The surprising result was the error counters. Every arm read zero no_frame, zero empty_frames and zero too_small, even when a slow consumer starved the pool. The driver waits silently rather than failing, so a starved pool only ever shows up as a lower frame rate. A firmware watching the error counter would watch forever.

JPEG and pixels without the heap

Most small ESP32 camera boards ask the sensor for JPEG directly, so the first job is to handle JPEG safely, and the second is to handle raw pixels when you want them.

Probing, trimming and encoding JPEG

jpeg::probe reads width, height, components and the progressive flag from the marker headers without decoding the image. find_eoi trims the trailing DMA padding a capture carries after the end marker. Both are covered by a no-panic gate: 40,000 random and mutated inputs, and each parser must return an error rather than crash.

For sensors without a JPEG mode, jpeg::encode goes through rusty_jpeg 0.4, which is no_std, writes into a caller-owned buffer and accepts packed YUYV as the sensor delivers it. It compiles for the S3's own bare-metal target and is tested on the host. An on-chip byte-identity gate against the host encoder is planned.

Frames off the board were judged by an outside decoder: three consecutive frames reassembled from the serial dump all passed ffprobe and a full ffmpeg decode, with three distinct checksums. The picture was the room's ceiling fan, which is where the board was pointing.

Pixel kernels: YUYV and RGB565

The raw ESP32 camera capture paths need conversions: YUYV to RGB565, RGB888 or grayscale, RGB565 to RGB888 and back, half-size downscales, crop and rotate. These started here with test vectors and have since moved to rusty_esp_dsp, the shared kernel home, byte-identical to the copies they replaced. The image package re-exports them.

One small change made them matter on a real device. IdfCamera used to accept only JPEG. It now lets the sensor deliver RGB565, YUYV422 or grayscale, which is what put twelve image kernels on a shipping firmware path instead of only in tests.

Sensor tables as data, and what is next

In many ESP32 camera stacks, sensor bring-up is driver code. Here, a sensor is a table and a tiny trait, which makes it easy to read, diff and test.

Register sequences you can read

The OV2640 and OV5640 register sequences were converted from esp32-camera's own headers: 302 steps across seven OV2640 tables and 216 steps across nine OV5640 tables, gamma and white balance included, with delay steps where the datasheet asks for them. They replay over the camera control bus (SCCB) through embedded-hal 1.0, and a test applies the full OV2640 init table over a fake bus. The source is Apache-2.0 and attributed.

On silicon so far: an AI-Thinker ESP32-CAM detected its OV2640 and served frames at 10 fps, and the XIAO ESP32-S3 Sense answered as an OV3660, one of the two sensors that board ships with.

Honest status for your home camera

Host-verified and on a board: the ESP32 camera capture seam, the frame pool, JPEG probing, the OV2640 on the ESP32-CAM. Planned: the esp-hal camera backend, on-chip JPEG byte identity, and ESP32-P4 MIPI-CSI capture with hardware JPEG, which has no board yet.

For a window ESP32 camera or a porch camera, pick hardware with choosing an ESP32 board, flash with espino, and give the device its own identity with rusty_esp_mid. The source is at github.com/Remade-With-Rust/rusty_esp_image, and the wider family is described at Remade With Rust.