HTTP language models
AudioFlow.Adapters.ReqLLM runs HTTP language models through
ReqLLM. It takes ReqLLM’s model, context, and
generation options directly. It groups the generated text into sentences, with
whitespace preserved, and runs tools in the turn. It uses the same adapter
interface as the native models, and needs no native model dependencies. Local Nx
adapters do not depend on ReqLLM.
llm = {AudioFlow.Adapters.ReqLLM, model: %{provider: :openai, id: "audioflow-qwen"}, base_url: "http://127.0.0.1:18082/v1", api_key: "local", system_prompt: AudioFlow.Prompt.default_system(), temperature: 0.0, max_tokens: 64}{:ok, session} = AudioFlow.start_session( Keyword.put(session_options, :llm, llm))Options
Section titled “Options”Use a ReqLLM catalog model such as "anthropic:claude-sonnet-4-5", or an inline
model such as %{provider: :openai, id: "..."} with :base_url for a local
OpenAI-compatible server. :base_url, :api_key, :system_prompt,
:temperature, :max_tokens, :provider_options, :req_http_options, and
native ReqLLM.Tool definitions are ReqLLM options. Resolve credentials when
you build the options, or use ReqLLM’s provider credential configuration. Pass
custom request headers with req_http_options: [headers: ...].
The adapter adds no system prompt. Pass :system_prompt;
AudioFlow.Prompt.default_system/0 requests one short spoken sentence.
Input can be a ReqLLM.Context, a list of messages, or one string. Session
history passes a list of messages.
Settings that belong to AudioFlow go under the :audio_flow keyword: tools,
tool limits, and the byte and time limits below.
Limits
Section titled “Limits”The adapter reads the HTTP stream in the turn process. It uses ReqLLM’s request builders, SSE parser, provider decoders, and chunk accumulator, but not ReqLLM’s stream server. The shared Finch pool uses HTTP/1, so the output callback applies back pressure to socket reads. Cancellation closes the active connection. A failed output callback also releases the connection.
Each request runs once. The adapter does not follow redirects, retry, use ReqLLM’s cache, or replay fixtures. Only HTTP SSE transports are supported, not ReqLLM’s WebSocket or in-process transports.
These limits go under :audio_flow:
| Option | Default | Bound |
|---|---|---|
max_request_bytes |
65,536 | Encoded request body |
max_response_bytes |
262,144 | Complete response body |
max_event_bytes |
16,384 | One SSE event |
max_text_bytes |
8,192 | Generated text, across tool rounds |
timeout_ms |
30,000 | Budget for response reads |
receive_timeout_ms |
10,000 | Wait for a response chunk |
pool_timeout_ms |
5,000 | Wait for a connection |
ReqLLM’s request_timeout, receive_timeout, and pool_timeout options are
also accepted; the :audio_flow values take precedence. Connections have a
five-second connection timeout. The response-read budget excludes pool checkout,
connection setup, and time blocked in the output callback. Use the session
deadline to bound the complete turn, including tools and a blocked callback.
Refusals, truncation, incomplete streams, invalid events, and empty replies
return errors. Earlier complete sentences can already have reached speech
synthesis. OpenAI-compatible streams need a finish reason and [DONE] before
the last text is emitted.
Tool calls
Section titled “Tool calls”Put Jido actions and AudioFlow.Tool maps under audio_flow: [tools: ...]. A
tool map has :name, :description, :parameters (JSON Schema), and a
one-argument :run function. Native ReqLLM.Tool structs go in the top-level
:tools option. Tool names must be unique. Both kinds run in linked tasks with
bounded results and deadlines, and cancellation stops them.
llm = {AudioFlow.Adapters.ReqLLM, model: "anthropic:claude-sonnet-4-5", system_prompt: AudioFlow.Prompt.default_system(), audio_flow: [ tools: [MyApp.Actions.Weather], tool_context: %{user_id: user_id}, tool_filler: "One moment."]}Tools run in tasks linked to the turn, so cancellation stops a running tool.
The results go back to the model in a new request. If the model calls a tool
without first saying something, :tool_filler is spoken while the tool runs.
Tool failures, timeouts (:tool_timeout_ms, 10,000), and oversized results
(:max_tool_result_bytes, 8,192) go back to the model as errors.
:max_tool_rounds (4) limits the requests in one turn, and :max_tool_calls
(8) limits the calls in one request.
Graph node
Section titled “Graph node”In graphs, use the llm.req_llm node. Its options are model (a
provider:model string), base_url, system_prompt, api_key_env (the name
of the environment variable, never the key), tools_profile, and timeout_ms.
The graph runner passes the session and turn to the tools as tool_context.
AUDIOFLOW_CHAT_ENDPOINT=http://127.0.0.1:18082/v1/chat/completions \ AUDIOFLOW_LLM=audioflow-qwen mix run examples/http.exs "Say hello."Set AUDIOFLOW_CHAT_API_KEY when the server needs authentication. To use HTTP
language inference with native Whisper and Kokoro, set
AUDIOFLOW_LLM_ENGINE=http and AUDIOFLOW_CHAT_ENDPOINT for
examples/local.exs or examples/media.exs. AUDIOFLOW_LLM becomes the server’s
model ID. This mode does not load a native language model. The benchmark
metadata uses mode=hybrid_inference and records the language engine as HTTP.
Optional AUDIOFLOW_LLM_SOURCE and AUDIOFLOW_LLM_REVISION identify the deployed
weights. The metadata does not contain the API key or endpoint query string.
The real HTTP check, and the llama.cpp server it was last run against, are described in Model checks.