Skip to content

Pipecat comparison

scripts/compare_pipecat.py measures AudioFlow and Pipecat over the same shared Nx model services. One Elixir process loads Silero, Whisper tiny, Qwen3-0.6B, and Kokoro. Pipecat calls these services through private local pipes, so its result includes that transfer cost; AudioFlow calls them in the same process. This does not compare Nx against Python inference engines. The run includes Membrane file input, resampling, endpoint detection, model queues, and software output delivery.

Two recordings, 128 measured turns, 64 warmup turns, and 32 serial interruption checks. All measured turns completed without errors. Detected input and output PCM hashes agreed, and transcripts and replies matched.

Time to first output, in seconds:

Simultaneous turns AudioFlow p50 AudioFlow p95 Pipecat p50 Pipecat p95
1 1.50 2.62 1.70 2.93
2 1.35 3.44 1.78 4.56
4 2.63 6.71 2.59 6.69
8 5.34 13.79 5.19 13.36

Both frameworks passed the 10-second p95 rule at four simultaneous turns and exceeded it at eight. These results apply to the listed models, CPU, software sinks, and closed workload. They do not establish production capacity.

See the data and the report.

Terminal window
AUDIOFLOW_ML=1 AUDIOFLOW_MEDIA=1 mix deps.get
uv run scripts/compare_pipecat.py FIRST.s16le SECOND.s16le \
--output benchmarks/comparison.json --concurrency 1 2 4 8 \
--warmup 8 --runs 16 --interrupt-runs 4
python3 scripts/comparison_report.py \
benchmarks/comparison.json benchmarks/comparison.html

The runner retains failed attempts, input hashes, detected-utterance hashes, output PCM hashes, model revisions, and source fingerprints. It separates warmup, measured turns, and serial interruption checks. Choose --max-first-output-ms before the run. The default rule allows no measured errors and requires p95 first output within 10 seconds. The report checks input agreement and all lower tested levels before it assigns a highest passing tested concurrency. This threshold is a test rule; it does not establish conversational quality.

Each input must be mono 16 kHz s16le PCM with one detected utterance. See scripts/COMPARISON.md for the full method.