Integration guide

How to put Tenms into a voice agent. Every API name below was checked against the code (python/tenms/**, crates/tenms-core/include/tenms.h, crates/tenms-proto/src/protocol.rs). For the wire-level reference see API reference; for internals see How Tenms works.

The three things every integration must get right

1. Echo reference. Focus can only remove the agent’s echo if it gets the audio the agent is playing, as it is played. The engine estimates the remaining delay itself (0-1000 ms, echo_delay_ms()), so the reference does not need to be sample-aligned, but it must be real-time: queued-but-unplayed audio that is later cancelled must not be fed.
2. Agent-speaking state. Interruptions are classified only while the engine knows the agent is speaking (set_agent_speaking(True) on each new agent utterance; each false → true edge re-arms one barge-in).
3. Interruption policy. Act only on barge_in (stop TTS, flush playback). Ignore backchannel, noise and other_speaker. Provisional backchannel/noise events may later be superseded by a barge_in of the same utterance.

Optional but valuable:

• Transcripts (push_transcript(text, confidence, is_final)): finals feed Pulse attribution; interim and final transcripts heard while the agent speaks confirm a barge-in much faster (~355 ms from onset vs ~1.0-1.3 s acoustically; core README) or downgrade acknowledgements to backchannels.
• Agent text (push_agent_text(text)): the agent’s TTS text, so transcripts of its own echo never confirm a barge-in.

Pipecat

pip install "tenms[pipecat]" (tested with pipecat-ai 1.15, python README).

from tenms.pipecat import TenmsCall

tenms = TenmsCall()   # one per call; kwargs go to TenmsSession: modules, license, public_key,
                      # focus_profile ("stt" default here), focus, turn, call_id, reporter, ...

transport = DailyTransport(room_url, token, "bot", DailyParams(
    audio_in_enabled=True, audio_out_enabled=True,
    audio_in_filter=tenms.focus_filter(),        # TenmsFocusFilter: Focus + drives the engine
    audio_out_filter=tenms.reference_filter(),   # TenmsReferenceFilter: playback -> echo reference
))
user_agg, assistant_agg = LLMContextAggregatorPair(context, user_params=LLMUserAggregatorParams(
    vad_analyzer=tenms.vad_analyzer(),                  # TenmsVADAnalyzer: gated VAD
    user_turn_strategies=tenms.user_turn_strategies(),  # Tenms start + TurnAnalyzer stop strategy
))
proc = tenms.processor()                                # TenmsProcessor: after STT, before user_agg
pipeline = Pipeline([transport.input(), stt, proc, user_agg, llm, tts, transport.output(), assistant_agg])
task = PipelineTask(pipeline, observers=[tenms.observer()])   # TenmsObserver: bot text -> push_agent_text

@proc.event_handler("on_call_summary")
async def on_summary(processor, summary): ...

What each piece does (python README):

Live evidence (python/tests/live_reports/pipecat_daily_latest.json): 0 interruptions within 2 s of a backchannel, barge-in → interruption 1.07 s, echo suppression 33.0 dB with reference active, compute p99 ~100 µs/frame, EOT delay p50/p95 360/610 ms. In that run the caller’s “Wait.” was classified as a backchannel and the barge-in fired on “Stop, please.” (report file dated 2026-11-05 08:33; it is the newest live report in the repo).

LiveKit Agents

pip install "tenms[livekit]" or pip install livekit-plugins-tenms (tested against livekit-agents 1.8 classes offline).

from livekit.agents import AgentSession, room_io
from livekit.plugins import tenms   # or: from tenms.livekit import TenmsLiveKit, TenmsAgent

async def entrypoint(ctx):
    call = tenms.TenmsLiveKit()        # kwargs go to TenmsSession, e.g. sample_rate=16000
    session = AgentSession(
        stt=deepgram.STT(), llm=openai.LLM(model="gpt-4o-mini"), tts=elevenlabs.TTS(),
        vad=call.vad(),
        turn_handling={"turn_detection": call.turn_detector(),
                       "interruption": {"mode": "vad", "min_duration": 0.3}},
    )
    await session.start(
        agent=tenms.TenmsAgent(call, instructions="..."),
        room=ctx.room,
        room_options=room_io.RoomOptions(audio_input=room_io.AudioInputOptions(
            noise_cancellation=call.noise_cancellation())),     # TenmsFocus (rtc.FrameProcessor)
    )
    call.attach(session)   # agent speaking state, output tap (echo reference), transcripts, summary

    @call.on_summary
    def _(summary): ...

If Focus is disabled (or the processor never installed), VAD/TurnDetector take over feeding the engine so VAD, Turn and Pulse stay live (test_disabling_focus_keeps_vad_and_turn_alive).

Not yet verified live. The live test needs a local server started as livekit-server --dev --bind 0.0.0.0 --node-ip <LAN IP>.

Focus profiles: stt, listening, listening_mini, listening_pro

Focus has two DSP tunings and two opt-in listening tiers (FocusProfile in Rust, focus_profile= in Python, start.focus_profile on Edge, ?focus_profile= on the ElevenLabs bridge, tenms_config_apply_focus_profile in C, --focus-profile in tenms-cli):

Measured on the Arena’s frozen sets with Deepgram nova-3 (Models and benchmarks): corpus WER over 110 noisy and echo-heavy calls is 0.233 with stt, 0.250 without Focus and 0.255 with listening; on the noise-only set full suppression costs +14 % WER, while on echo-heavy calls with the reference stt is -17 % vs no Focus. focus={...} options apply on top of the profile (e.g. focus_profile="stt", focus={"speaker_lock": False}).

Listening quality on the Arena’s enhance_listen set (48 calls with music, babble or stationary backgrounds, 8 / 16 kHz, half with agent echo; Models and benchmarks):

listening_mini (opt-in, Tenms model, human listening only)

The Arena’s enhance_listen winner within the latency budget (<= 10 ms preferred, <= 1 ms CPU): the Tenms engine with the listening settings and our own trained model’s Focus head. It runs in tenms-core’s native TNW1 runtime (no ONNX Runtime), so it works in the default Edge image and the Python SDK. The file is pinned by sha256 (models/manifest.json, tenms_core::listening_mini::SHA256) and every loader refuses a file that does not match. The model applies only to engines built for this profile: on an Edge replica that has it loaded, stt and listening sessions are bit-identical to a replica without it (tested).

• Edge: {"type":"start","focus_profile":"listening_mini"} on /v1/stream. Available on every Tenms Edge region; ready.focus_profile confirms it. If a replica has not loaded the model, start fails with profile_unavailable and the other profiles keep working.
• CLI: tenms-cli process --focus-profile listening_mini uses the shipped model ($TENMS_LISTENING_MINI_MODEL, /opt/tenms/models/... or the repository copy; sha256-verified); --model <file> replaces it (your file, not sha-pinned, must have a Focus head). Its output equals --focus-profile listening --model models/tenms-mini-focus-v1.tnw bit for bit. The JSON report names the model.
• Python SDK: tenms.Engine(16000, focus_profile="listening_mini") loads the model from listening_mini_model= (path), else $TENMS_LISTENING_MINI_MODEL, else the copy packaged in tenms/models/, else the repository’s models/ (development installs); engine.focus_profile is "listening_mini". EdgeClient(..., focus_profile="listening_mini") passes it to Edge.
• Rust: EngineConfig::new(m, rate).with_focus_profile(FocusProfile::ListeningMini) plus cfg.model = Some(tenms_core::listening_mini::load(path)?).
• C: cfg.model = tenms_model_load("tenms-mini-focus-v1.tnw"), then tenms_config_apply_focus_profile(&cfg, TENMS_FOCUS_PROFILE_LISTENING_MINI) (returns the server limit without a model that has a Focus head).

listening_pro (opt-in neural enhancer, human listening only)

For audio people listen to - call recordings, QA review, supervisor listen-in - when the background is music or other voices and 27-30 ms of latency is acceptable. It runs the Tenms engine with the listening settings minus the DSP noise suppressor (echo cancellation, residual-echo suppression and the speaker gate stay) and then FastEnhancer-M (MIT, DNS-trained, 2 MB ONNX), chosen by the Arena’s enhance_listen task among the neural enhancers. It costs ~15x the CPU of a DSP session: Edge caps concurrent listening_pro sessions per replica (the server limit (default 64); beyond it start fails with the retryable profile_capacity).

Opt-in only, and only where the neural enhancer is loaded:

• Edge: {"type":"start","focus_profile":"listening_pro"} on /v1/stream. ready.focus_profile confirms it. Replicas cap concurrent listening_pro sessions: a full replica returns the retryable profile_capacity (retry, or use listening_mini); a replica without the enhancer returns profile_unavailable.
• CLI: tenms-cli process --focus-profile listening_pro --models models/ [--assist none] (--output is delay-compensated as usual; the JSON report has a listening_pro block).
• Rust: tenms_models::AssistOptions { listening_pro: true, .. } with LoadOptions { enhancer: true, .. } and EngineConfig::with_focus_profile(FocusProfile::ListeningPro).
• Python SDK / C library: not available (they embed the DSP engine and the TNW1 runtime only): Python Engine(focus_profile="listening_pro") raises ValueError; C the server limit returns the server limit. EdgeClient(..., focus_profile="listening_pro") passes it to Edge.

Python engine directly

import tenms

engine = tenms.Engine(16000, modules=["focus", "turn", "pulse"], focus_profile="stt",  # output feeds an STT
                      turn={"confirm_threshold": 0.85}, focus={"echo_tail_ms": 320})
out, res = engine.process(mic_chunk, reference=agent_playback_chunk)  # any length; bytes or numpy
for ev in res.events:          # tenms.Event: .type, .at_ms, .probability, .silence_ms, .kind
    if ev.type == "turn_prediction": ...                       # start the LLM speculatively
    elif ev.type == "turn_end": ...                            # commit the user turn
    elif ev.type == "interruption" and ev.kind == "barge_in": ...   # stop TTS

engine.set_agent_speaking(True)
engine.push_agent_text("Your order ships tomorrow.")
kind = engine.push_transcript("wait", confidence=0.9, is_final=False)  # "barge_in" / "backchannel" / None
summary = engine.finish("call-123", source="sdk")             # CallSummary dict

Licensing: tenms.Engine(16000, license=token) (or TENMS_LICENSE in the env) verifies against the vendor key built into the wheel; pass public_key= only for self-hosted signing keys. tenms.Engine.from_vendor_license(token, sample_rate) is the explicit form. A customer wheel built with --no-default-features raises LicenseError without a license.

Telemetry: TenmsSession (used by both integrations) posts the summary to $TENMS_CONTROL_URL/v1/calls:batch with the license token when TENMS_CONTROL_URL is set; reporter=False disables it. tenms.telemetry.ControlPlaneReporter().submit(summary) does it manually (non-blocking, batched, retried).

Raw Edge WebSocket

Use this from any language. Full message reference: API reference.

import asyncio, json, websockets

async def call(url, token, mic_frames, ref_frames):
    async with websockets.connect(f"{url}/v1/stream", additional_headers={"Authorization": f"Bearer {token}"}) as ws:
        await ws.send(json.dumps({"type": "start", "sample_rate": 16000, "call_id": "call-123"}))
        # optional: "focus_profile": "listening" (default "stt")
        ready = json.loads(await ws.recv())                     # {"type": "ready", ..., "focus_profile": "stt"}
        await ws.send(json.dumps({"type": "agent_speaking", "speaking": True}))
        for mic, ref in zip(mic_frames, ref_frames):            # PCM16 LE bytes, real time
            if ref:
                await ws.send(b"\x02" + ref)                     # reference first, as played
            await ws.send(b"\x01" + mic)
        await ws.send(json.dumps({"type": "transcript", "text": "hold on", "is_final": False}))
        await ws.send(json.dumps({"type": "end"}))
        async for msg in ws:                                    # processed audio (0x01) and events
            if isinstance(msg, str) and json.loads(msg)["type"] == "summary":
                return json.loads(msg)["summary"]

Or the SDK client (tenms.edge.EdgeClient; /v1/stream is appended to a bare host):

from tenms.edge import EdgeClient, EdgeError

async with EdgeClient("wss://edge.tenms.cloud", token, sample_rate=16000,
                      on_event=print, connect_retries=3) as edge:
    await edge.send_reference(agent_chunk)
    await edge.send_audio(mic_chunk)
    await edge.set_agent_speaking(True)
    await edge.push_transcript("hello", confidence=0.9, is_final=True)
    processed = edge.audio_nowait()
    summary = await edge.end()

Rules that matter:

• Send audio in real time. Edge allows 1.5 × real time after a 5 s burst and stops reading the socket beyond that; summaries record processed audio clamped to wall time.
• Reference before or with the matching mic audio, at most 1 s ahead; overflow drops the newest reference samples.
• transcript.is_final defaults to false on the wire (protocol.rs); Pulse uses finals only, so set it for final transcripts. (The Python EdgeClient.push_transcript also defaults to is_final=False and always sends the field.)
• Send agent_text ({"type":"agent_text","text":"..."}) with the agent’s TTS text, the Edge equivalent of push_agent_text: the agent’s own words, heard back as echo and transcribed, then never confirm a barge-in. The ElevenLabs bridge does this automatically from agent_response events.
• Read continuously. A client that does not read for 10 s is disconnected.
• Reconnect policy: retry rate_limited, at_capacity, draining, server_shutdown, not_ready with backoff; mint a new token on license_expired; never retry license_revoked, invalid_license, agent_not_allowed.

ElevenLabs Agents

Full reference: ElevenLabs bridge. Summary:

1. Your license lists the ElevenLabs agents it may reach (the elevenlabs_agents claim, for example ["agent_abc"]). Tell us the agent ids when you request access.
2. Your backend mints a session token per conversation (below).
3. The browser uses the ElevenLabs SDK with the WebSocket transport and the Edge URL as the signed URL (WebRTC bypasses Edge):

const { token } = await (await fetch("/tenms-session-token", { method: "POST" })).json();
const conversation = await Conversation.startSession({
  signedUrl: `wss://edge.tenms.cloud/v1/elevenlabs/stream?agent_id=${agentId}&token=${token}`,
  connectionType: "websocket",
});

Edge does the get-signed-url call with its own ELEVENLABS_API_KEY, cleans caller audio, uses the agent’s audio as the paced echo reference, gates noise/other speakers while the agent talks, and posts a summary with source: "elevenlabs". Optional query parameters: modules, gate=off, tenms_events=1, playout_ms, branch_id, environment, version_id. Working clients: examples/elevenlabs/index.html (browser) and examples/elevenlabs/client.py (streams a WAV in real time).

Session tokens for browsers and devices

Never put a license in a web page or app. Mint a short-lived token on your backend:

# backend; TENMS_LICENSE and TENMS_CONTROL_URL in its environment
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)

@app.post("/tenms-session-token")
def tenms_session_token():
    # authenticate *your* user first
    r = requests.post(f"{os.environ['TENMS_CONTROL_URL']}/v1/session-tokens",
                      headers={"Authorization": f"Bearer {os.environ['TENMS_LICENSE']}"}, timeout=5)
    r.raise_for_status()
    return jsonify(token=r.json()["token"])

• Response: {"token", "expires_at", "session_id", "parent", "customer_id"}.
• The token must be used to connect within 5 minutes (the server limit); a conversation that started in time may run up to the server limit (4 h).
• It shares the parent’s per-license limits; revoking the parent ends it within one revocation poll.
• Optional body: {"ttl_secs": N} (shorter than the server TTL). The endpoint rejects unknown fields.

tenms.control.mint_session_token / amint_session_token send only ttl_secs (optional) and parse both the control plane’s nested {"error": {"code", "message"}} and Edge’s flat error bodies, so codes like license_revoked surface as themselves.

C ABI

Header: crates/tenms-core/include/tenms.h. Library: build tenms-core (cdylib / staticlib); customer builds with --no-default-features.

#include "tenms.h"

TenmsConfig cfg;
tenms_config_default(&cfg);                 /* all modules, 16 kHz */
cfg.input_sample_rate = 16000;
TenmsEngine *eng = tenms_engine_from_vendor_license(&cfg, license_token);
if (!eng) { fprintf(stderr, "%s\n", tenms_last_error()); return 1; }

size_t n = tenms_frame_len(eng);            /* 160 at 16 kHz */
TenmsFrameResult res;
tenms_set_agent_speaking(eng, 1);
tenms_push_agent_text(eng, "Your order ships tomorrow.");
/* every 10 ms; out may equal mic; reference may be NULL */
int32_t rc = tenms_process_frame_i16(eng, mic, n, ref, ref ? n : 0, out, n, &res);
for (uint32_t i = 0; i < res.n_events; i++) {
    if (res.events[i].kind == TENMS_EVENT_INTERRUPTION &&
        res.events[i].interruption == TENMS_INTERRUPTION_BARGE_IN) { /* stop TTS */ }
}
tenms_push_transcript(eng, "wait", 0.9f, 0);       /* interim */
char *json = tenms_finish_json(eng, "call-123", started_at_ms, "sdk", NULL);
/* ... upload json to POST /v1/calls:batch ... */
tenms_string_free(json);
tenms_engine_free(eng);

• Wrong mic/out/reference lengths return the server limit / the server limit and pass audio through untouched; nothing allocates, even on errors. A caught panic poisons the engine (the server limit until freed).
• tenms_model_load(path) loads a TNW1 file shareable between engines (cfg.model).
• tenms_engine_from_license(cfg, token, public_key_b64url, now_unix) exists for self-hosted keys and tests; it is honour-system (the host supplies key and clock).
• tenms_config_apply_focus_profile(&cfg, TENMS_FOCUS_PROFILE_STT) (after tenms_config_default) sets the Focus fields to the stt tuning; ..._LISTENING restores the defaults.
• The C config exposes a subset of options (noise/echo/speaker switches, echo_tail_ms, gain floors, prepare/confirm thresholds, max_silence_ms, barge_in_ms); the remaining Turn options keep their defaults. The header’s comments list stale defaults for echo_tail_ms (says 240; actual 320) and barge_in_ms (says 250; actual 150) — tenms_config_default() fills the real values from FocusConfig/TurnConfig.

Tuning knobs worth knowing

Defaults are FocusConfig::default() / TurnConfig::default() in crates/tenms-core/src/types.rs.