Introduction
AudioFlow is an Elixir runtime for voice, image, and video agents. Each conversation runs in its own OTP session process. Sessions share model services through an adapter interface that does not depend on the inference engine.
A voice turn moves through four stages:
- Speech recognition turns the user’s audio into a transcript.
- The language model streams a reply as text.
- Speech synthesis turns each reply sentence into PCM audio.
- The output transport plays the audio, and stops it when the user interrupts.
Each stage is an adapter. An adapter can call a native Nx model, a local HTTP server, a cloud API, or a worker on another Erlang node. The session does not know which one it calls.
What is included
Section titled “What is included”- Sessions with one active turn, cancellation, deadlines, and bounded output.
- Conversation history with limits on turns and bytes, trimmed to the part of a reply the user heard after an interruption.
- Native models through Bumblebee, EXLA, and Transmuter: Silero VAD, Whisper, Qwen3 and other text models, Kokoro and VITS speech, BERT and CLIP embeddings, and BLIP captions, with language choices.
- HTTP language models through ReqLLM, for llama.cpp and cloud providers, with tool calls.
- Remote speech providers: Deepgram, Cartesia, and ElevenLabs, and speech-to-speech sessions with OpenAI Realtime and Gemini Live (providers).
- Transports: a raw PCM WebSocket, Twilio and Telnyx telephony, the Pipecat RTVI client, and optional WebRTC and LiveAvatar output (transports, WebRTC).
- Input processing: SpeexDSP denoising and echo cancellation, and Smart Turn endpoints (input processing).
- Conversation states with Jido handoffs, pgvector retrieval, and scripted evaluations.
- Recordings that are off by default, with transcript redaction (recording).
- Image and video input as packed RGB frames.
- Model routing across local and remote workers, with route changes while sessions stay open.
- Membrane input and output, with voice detection, resampling, and playback that can be flushed.
Build profiles
Section titled “Build profiles”The core needs no model downloads. Environment variables add the optional
parts. Each combination uses its own _build directory. Changing a feature
flag does not remove modules used by another build profile.
| Variable | Adds | Source directory |
|---|---|---|
| none | Sessions, history, routing, HTTP inference, remote providers | lib |
AUDIOFLOW_ML=1 |
Bumblebee, EXLA, Transmuter models | lib_ml |
AUDIOFLOW_MEDIA=1 |
Membrane audio and video | lib_media |
AUDIOFLOW_WEBRTC=1 |
ExWebRTC media output and the PCM bridge | lib_webrtc |
Status
Section titled “Status”This implements the session runtime, model routing, distributed inference workers, native Silero voice detection, Whisper and text-generation adapters, Kokoro and VITS speech synthesis, and Membrane audio and video input. The native media path has a recorded-speech integration check and a controlled Pipecat comparison. Output interruption is measured at the software transport. Acoustic interruption needs a device loopback check. HTTP language inference can use a compatible local server or cloud endpoint. Text embeddings, CLIP image embeddings, and BLIP image captions use shared native model services. Their model checks use independent PyTorch references.
Next steps
Section titled “Next steps”- Quickstart: run the demo and your first session.
- Chat example: a browser voice chat on the Apple GPU.
- Sessions: the event format and the turn rules.