Video input
With AUDIOFLOW_MEDIA=1, AudioFlow.Video.frames/2 reads H.264 video from a
local MP4 file. It also accepts a Membrane raw-video source, such as a camera.
It converts selected frames to packed RGB:
for frame <- AudioFlow.Video.frames("input.mp4", interval_ms: 1_000, width: 384, height: 384) do {:ok, turn} = AudioFlow.submit(session, frame) # Wait for this turn to finish before you read the next frame.endFrame selection
Section titled “Frame selection”The interval is the minimum time between selected frames. The reader uses
presentation timestamps, not frame counts. By default, time starts at the first
decoded frame. Set normalize_timestamps?: false to keep decoder timestamps, including MP4 composition offsets. These times do not account
for MP4 edit lists or provide audio and video synchronization.
The default track is the H.264 track with the lowest ID. Use :track_id to
select another track. Other tracks, including audio, are discarded. Files with
metadata at either end are supported. Unsupported tracks and decode errors
produce errors; they do not end the calling process when you use the manual API.
Limits
Section titled “Limits”The reader allows one pending read and uses small RGB queues. The default frame
limit is 6,291,456 bytes. Set :max_frame_bytes to change it. Optional :width
and :height must be positive even integers. Scaling keeps the aspect ratio and
adds padding. The frame limit applies after scaling. Decoder working memory and
MP4 indexes are separate from this limit. Decoding follows demand; the reader
does not pace frames against a clock.
Manual control
Section titled “Manual control”frames/2 closes the reader at the end of the file, on an early halt, or after
a consumer error. It raises on a read error. Its :timeout_ms option sets the
limit for each read (30 seconds by default). For manual control, use
Video.start_link(location: path) or Video.start_link(source: source), then
Video.next_frame(reader, timeout_ms) and Video.close(reader). A reader also
stops when its owner exits.
Example
Section titled “Example”AUDIOFLOW_ML=1 AUDIOFLOW_MEDIA=1 mix run examples/video.exs input.mp4 outputThe example uses shared BLIP, Qwen3, and Kokoro services. It waits for one frame
turn before it reads another. History supplies the previous description for
comparison. It saves descriptions, replies, timings, and speech PCM. AUDIOFLOW_VIDEO_INTERVAL_MS sets the
interval, and AUDIOFLOW_VIDEO_FRAMES sets the frame limit. This is sampled
frame analysis; it does not track objects or motion between frames. A two-scene
model check produced correct individual descriptions, but Qwen3-0.6B added an
unsupported action during comparison. Model output needs assessment for the
intended use.