Model checks
The regular tests need no model downloads. With the model flags, tests also check shared EXLA calls, PCM conversion, and Membrane interruption:
mix testAUDIOFLOW_ML=1 AUDIOFLOW_MEDIA=1 mix testModel tests are opt-in. They download pinned models unless the files are already cached:
AUDIOFLOW_ML=1 AUDIOFLOW_MEDIA=1 AUDIOFLOW_MODEL_TESTS=1 mix test| Model | Reference | Generator |
|---|---|---|
| Silero VAD | ONNX Runtime probabilities, two concurrent streams | scripts/silero_reference.py |
| Kokoro | ONNX Runtime waveform lengths, two voices and speeds | scripts/kokoro_reference.py |
| VITS | PyTorch duration from the Transmuter speech guide | None |
| BERT, CLIP | PyTorch vectors | scripts/multimodal_reference.py |
| BLIP | Complete reference caption | scripts/multimodal_reference.py |
| Qwen3 prompt | Original pinned Jinja template | scripts/qwen3_prompt_reference.py |
Silero probabilities are compared with independent ONNX Runtime values in
test/fixtures/silero_reference.json, for two concurrent streams. Set
AUDIOFLOW_SILERO_MODEL to use a local Silero file. The VITS test also checks PCM
size and non-silent output. The Kokoro checks also cover finite samples, PCM
output, and use through a session; they require espeak-ng.
The multimodal checks compare BERT and projected CLIP vectors against independent PyTorch outputs. The text model has two concurrent callers on one shared serving. The pinned cat image comes from the Transmuter test fixtures. FFmpeg RGB output must have the same SHA-256 as the reference image bytes. CLIP comparison allows small differences between image resize implementations; BLIP checks the complete reference text. These checks need FFmpeg, but no microphone or sound device.
AUDIOFLOW_ML=1 AUDIOFLOW_MODEL_TESTS=1 mix test test/multimodal_model_test.exsThe image session check uses real BLIP with controlled LLM and TTS adapters. It checks delivery through the session, rather than spoken audio quality.
These checks verify the inference path. They do not measure speech quality.
Full native pipeline
Section titled “Full native pipeline”The full media test uses the public LibriSpeech sample in the pinned Transmuter repository, with real Silero, Whisper, Qwen3, and Kokoro models and a Membrane test sink. It checks transcript content, reply text, non-silent PCM, and complete delivery through the output pipeline. It needs FFmpeg and espeak-ng. It does not open a sound device.
AUDIOFLOW_ML=1 AUDIOFLOW_MEDIA=1 AUDIOFLOW_PIPELINE_TESTS=1 \ mix test test/native_pipeline_test.exsSet AUDIOFLOW_PIPELINE_ARTIFACT_DIR to retain the transcript, reply, PCM, and
timings in a selected directory. Model loading and compilation can take time.
Conversation checks
Section titled “Conversation checks”The saved speech conversation checks
use two utterances with eSpeak NG and Kokoro input voices. With Kokoro inputs,
Whisper transcribed both utterances correctly, and the second spoken reply
recalled the codeword banana. Whisper tiny misread the eSpeak inputs. Both
results are retained with model revisions and PCM hashes. These are small checks
of transcription and context use; model quality needs a broader workload. The
native conversation model check uses one Qwen3 serving for two turns and checks
recall of a codeword from the first turn. Its STT and TTS adapters are controlled
test adapters. Qwen3 message prompts are also compared with the original pinned
Jinja template.
HTTP inference check
Section titled “HTTP inference check”AUDIOFLOW_HTTP_MODEL_TESTS=1 \ AUDIOFLOW_CHAT_ENDPOINT=http://127.0.0.1:18082/v1/chat/completions \ AUDIOFLOW_LLM=audioflow-qwen mix test test/req_llm_model_test.exsThis compares streamed session output with an independent request to the same
deterministic llama.cpp server. This check uses controlled STT and TTS adapters.
It validates transport output. Model answer accuracy requires a separate
evaluation. The checked local server was llama.cpp 0.8.0, build 11960
(8642541b2), with zero GPU layers. It used Qwen/Qwen3-0.6B-GGUF, revision
23749fefcc72300e3a2ad315e1317431b06b590a, file Qwen3-0.6B-Q8_0.gguf. The file
SHA-256 was
9465e63a22add5354d9bb4b99e90117043c7124007664907259bd16d043bb031. The server
used four threads, two slots, a 4,096-token context, and reasoning disabled.
Hybrid voice inference also passed with real Whisper and Kokoro. No hosted cloud
account was used for these checks.