Skip to content

Membrane input

Terminal window
AUDIOFLOW_MEDIA=1 mix deps.get
AUDIOFLOW_MEDIA=1 mix compile --warnings-as-errors

AudioFlow.PCMSink accepts mono :s16le raw audio. With its vad option, it submits utterances after endpoint detection:

%AudioFlow.PCMSink{
session: session,
vad: {AudioFlow.Silero, serving: AudioFlow.VAD},
endpoint_options: [min_speech_ms: 96, min_silence_ms: 320]
}

Start AudioFlow.VAD once with AudioFlow.Silero.serving/2 and AudioFlow.Models.child_spec/3. The Silero source can be a local ONNX file or {:hf, "onnx-community/silero-vad", "onnx/model.onnx"}. Each sink keeps its own recurrent state; only the weights are shared. Input must be 16 kHz. Pin the model with load_options: [revision: "e71cae966052b992a7eca6b17738916ce0eca4ec"]. Put a resampler before the sink for other rates.

After speech confirmation, the sink cancels the active reply and sends {:audio_flow_speech_started, session} to its parent. A configured output transport is flushed by the session. Otherwise, the parent must flush playback. After endpoint detection, the sink sends {:audio_flow_input, result}. The parent must handle busy input. At end of stream, the sink flushes the final utterance. Without vad, each input buffer must be one complete utterance.

AudioFlow.VoiceInput supplies the same frame processing without Membrane. It keeps pre-roll, confirms speech, uses separate start and stop thresholds, and limits utterance length. It pads a partial final frame with silence. Endpoint options are documented in the module.

Terminal window
AUDIOFLOW_ML=1 AUDIOFLOW_PCM=recording.s16le mix run examples/vad.exs

AudioFlow.Input.attach/3 adds FFmpeg resampling and the session sink to a Membrane source chain. It converts mono or stereo input to mono s16le at the selected rate (16 kHz by default). AudioFlow.InputPipeline supplies this chain as a complete pipeline. The source can be AudioFlow.PCMFileSource or Membrane.PortAudio.Source. The file source reads bounded chunks and preserves sample timestamps. It rejects incomplete PCM frames. Set the source format to match the actual file or device.

AudioFlow.Input.attach/3 requires a voice detector tuple.