ElevenLabs bridge

ElevenLabs Agents (Conversational AI) run a hosted pipeline, so Tenms cannot run in-process there. Tenms Edge proxies the conversation instead:

caller client ──(ElevenLabs Agents WebSocket protocol)──► Tenms Edge ──(signed URL)──► ElevenLabs
              ◄──────────── relayed verbatim ─────────────            ◄──────────────

On the way through, Edge:

• runs Focus on the caller’s audio before forwarding it (noise, echo and other-speaker suppression);
• uses the agent’s audio coming back from ElevenLabs as the echo reference, paced to the client’s playback, so nothing extra is needed from the client;
• uses Turn’s interruption classification to keep noise and other speakers from interrupting the agent (the gating policy is below);
• feeds ElevenLabs’ user_transcript into Pulse and posts a CallSummary (source: "elevenlabs", call_id = ElevenLabs conversation_id) to the control plane.

Implementation: crates/tenms-edge/src/elevenlabs.rs. Examples: examples/elevenlabs/.

Endpoint

GET wss://edge.tenms.cloud/v1/elevenlabs/stream?agent_id=<agent_id>&token=<tenms session token>

• Auth is the same as /v1/stream: Authorization: Bearer <token> or ?token= (browsers cannot set WebSocket headers, so use ?token=). The token is a Tenms license or, for anything that runs in a browser or on a device, a short-lived session token (below). Tokens are verified offline; the revocation list, draining and capacity checks apply.
• Agent allow-list. Every conversation runs on the Edge operator’s ElevenLabs key, so a license reaches only the agents its elevenlabs_agents claim lists (["*"] = any). Licenses without the claim cannot use the bridge (HTTP 403 agent_not_allowed), and neither can a license that grants no Tenms module (403 no_modules). Refused connections never reach ElevenLabs.
• Per-license limits (per Edge replica, shared with /v1/stream, keyed by the license id, so session tokens count against their parent): at most max_sessions concurrent sessions (claim, or the server limit (default 100)) and a connection rate of the server limit per second with bursts of the server limit (default 10/s, burst 20). Over either: HTTP 429 license_session_limit / rate_limited.
• ElevenLabs auth happens on Edge. For each conversation, Edge calls GET {ELEVENLABS_API_BASE}/v1/convai/conversation/get-signed-url?agent_id=... with xi-api-key: $ELEVENLABS_API_KEY, then opens the returned signed wss:// URL. Private agents (enable_auth) therefore work, and the ElevenLabs key never reaches the client.
• If ELEVENLABS_API_KEY is not set on the Edge service, the endpoint answers HTTP 503 {"error":"elevenlabs_not_configured"}.

Tenms parameters are never forwarded to ElevenLabs.

Session tokens (browsers and devices)

Never put a Tenms license in a web page or app. A license is long-lived; anyone who reads it from the page can use it until it is revoked. Instead, the customer’s backend (which holds the license) asks the Tenms control plane for a session token right before each conversation and hands only that to the browser:

POST https://app.tenms.cloud/v1/session-tokens
Authorization: Bearer <tenms license>

200 {"token": "tenms1....", "expires_at": <unix seconds>, "session_id": "...", "parent": "<license id>", ...}

The session token is a regular tenms1 token signed by the control plane with parent = <license id>, the parent’s modules, elevenlabs_agents and max_sessions, a fresh lid, and exp = min(now + 5 minutes, the license’s exp). Session tokens cannot mint session tokens. Edge accepts it exactly like a license:

• it must be used to connect within its 5 minutes; a conversation that started in time may run past it (up to the server limit (default 4) h);
• revoking the parent license refuses new connections and ends running conversations (tenms_error license_revoked, close 1008) within one revocation poll;
• it shares the parent’s per-license limits.

Minimal backend endpoint (Python, any framework works the same way):

import os, requests
from flask import Flask, jsonify

app = Flask(__name__)

@app.post("/tenms-session-token")
def tenms_session_token():
    # Authenticate *your* user here first.
    r = requests.post(f"{os.environ['TENMS_CONTROL_URL']}/v1/session-tokens",
                      headers={"Authorization": f"Bearer {os.environ['TENMS_LICENSE']}"}, timeout=5)
    r.raise_for_status()
    return jsonify(token=r.json()["token"])

The browser example (examples/elevenlabs/index.html) fetches from such an endpoint; examples/elevenlabs/client.py can do either.

Using an existing ElevenLabs client

Any client that speaks the ElevenLabs Agents WebSocket protocol works unchanged: point its WebSocket URL at Edge. With the official JS SDK, pass the Edge URL as the signed URL and use the WebSocket transport, for example Conversation.startSession({ signedUrl: "wss://edge.tenms.cloud/v1/elevenlabs/stream?agent_id=...&token=...", connectionType: "websocket" }). The WebRTC transport bypasses Edge and is not supported. Clients that call get-signed-url themselves must skip that step: Edge does it.

Protocol mapping

Every message is relayed verbatim and in order: the same text, not re-serialised. Unknown and future message types, non-JSON text and binary frames are relayed too. The table lists what Edge additionally does with each message.

Audio formats. pcm_8000, pcm_16000, pcm_24000 and pcm_48000 (PCM16 LE mono) and ulaw_8000 (G.711 μ-law) are processed. pcm_22050 and pcm_44100 are not engine rates, so caller audio in those formats is relayed untouched. The agent’s output format can be any pcm_* or ulaw_8000: it is converted to the caller rate for the reference (polyphase FIR for 8↔16, 16↔24 and 16↔48 kHz, linear interpolation otherwise). Total sample count and format are preserved exactly. Samples that do not fill a 10 ms frame are carried into the next chunk, so one forwarded chunk may be up to 9.9 ms shorter or longer than its input. An odd PCM byte is carried as well. The processed audio has Focus’ algorithmic delay (5 ms at 16 kHz, 8 ms at 8 kHz).

Edge-originated messages

Tenms messages are namespaced so that ElevenLabs clients ignore them:

{"type":"tenms_error","tenms_error_event":{"code":"elevenlabs_agent_not_found","message":"..."}}
{"type":"tenms_event","tenms_event":{"type":"interruption","at_ms":4620,"kind":"noise"}}
{"type":"tenms_event","tenms_event":{"type":"summary","summary":{...CallSummary...}}}

tenms_event is sent only with ?tenms_events=1. It wraps the /v1/stream ServerEvents (speech_start, turn_prediction, turn_end, interruption, quality) and, just before the close, the call summary.

Refusals before the WebSocket upgrade are HTTP errors with a JSON body {"error": "<code>", "message": "..."}: 401 missing_license / invalid_license / license_expired, 403 license_revoked / agent_not_allowed / no_modules, 429 rate_limited / license_session_limit, 503 draining / not_ready (revocation list not loaded yet) / at_capacity / elevenlabs_not_configured.

Error messages never contain the API key or the signed URL. A normal ElevenLabs close (1000, for example when the agent ends the call) is relayed as is, with no error.

A client that stops reading (full TCP window) is disconnected after the server limit (default 10 s) so it cannot hold a session slot.

Caller audio must be sent in real time (as captured). Edge allows up to the server limit (1.5) × real time after an initial the server limit (5 s) of audio; faster input is back-pressured (Edge stops reading the socket until the client is back within the allowance). Pacing also assumes real-time input (below).

Echo reference pacing

ElevenLabs sends the agent’s audio in bursts, much faster than real time; the client plays it in real time. Edge models that playback:

1. Agent audio is queued at the caller’s sample rate (cap 60 s; past it the newest audio is dropped, since the queued audio is what plays next).

2. Every caller frame Edge processes consumes exactly one frame of the queue. The caller’s audio is the clock, so no wall-clock drift is involved.

3. When playback starts from idle, the queue is preceded by a delay equal to since_last_chunk + client_rtt + playout_ms − 30 ms. That is when the audio reaches the caller’s speaker, relative to the next caller sample Edge will process:

   • since_last_chunk: time since the last user_audio_chunk arrived (the next chunk started being captured then);
   • client_rtt: smoothed ping→pong time;
   • playout_ms: client playout and capture latency (query or env, default 40).

   The 30 ms bias makes the reference slightly early on purpose. The echo canceller models echo paths 0–240 ms after the reference, so an early reference still cancels but a late one does not.

4. An interruption clears the queue. The agent counts as speaking while the queue is non-empty, then for a 200 ms hangover (room tail).

Measured with a real-time simulated client in the tests (4 s agent response delivered in one burst, played through a 4-tap echo path, mic streamed in 20 ms chunks): ≈ 23 dB echo reduction with the paced reference (≈ 22 dB with pcm_24000 agent audio), 0 dB without a reference.

Interruption gating policy

Applied per 10 ms frame to the Focus output, after the engine, and only while the agent is speaking (paced playback, hangover, or within 300 ms of agent_response). It requires Turn in the license; ?gate=off disables it.

• “The sound ends” means 200 ms of consecutive frames with VAD < 0.5 and a level at least 15 dB below the blocked sound’s peak. VAD alone would release aperiodic noise immediately.
• Every gain change ramps linearly over one frame, so there are no clicks. Muting replaces samples with zeros (μ-law 0xFF), so chunk sizes and timing are unchanged and ElevenLabs’ VAD hears silence.
• The 2 s cap means that a long sound classified as noise (a fan switching on) cannot hide a real barge-in for the rest of the agent’s turn.
• Turn’s decision latencies (crates/tenms-core/README.md): other speaker ≈ +90 ms, noise ≈ +110–270 ms, barge-in: a single continuous burst needs 800 ms of voicing, two voiced segments decide on the second (≈ 1.0–1.3 s from onset without transcripts; ElevenLabs transcripts arrive too late to confirm sooner). Before the decision the sound passes at −6 dB. This is a compromise: holding audio back would delay real barge-ins, and passing it at full level lets ElevenLabs react to a cough sooner.
• Tested (real-time simulation, agent echo present): after the decision, other speakers and noise bursts are forwarded as exact zeros. Barge-ins pass at full level. With gate=off the same burst is forwarded.

Telemetry

When the conversation ends (either side), the engine produces the CallSummary: source = "elevenlabs", call_id = ElevenLabs conversation_id (or a UUID if no metadata arrived), region = the Edge region. It is queued to the control plane like /v1/stream summaries: batched and asynchronous, never blocking audio. A conversation that failed before the upstream connected produces no summary.

Latency added by Edge

The live test (tests/elevenlabs_live.rs, release build, Apple M-series, 16 kHz, all modules, 20 ms chunks, agent echo present) measured the time from receiving a caller chunk to writing the processed chunk upstream. This covers JSON parse, base64 decode, two engine frames, gating, base64 encode and the socket write:

(p50/p99 are histogram bucket upper bounds.) Per-frame engine compute p99 was ≤ 50 µs. Audio-path delay is Focus’ 5 ms at 16 kHz, plus up to 10 ms of frame buffering for chunks that are not a multiple of 10 ms. Network: Edge adds one extra hop, so deploy it in the region closest to callers (Edge runs in 4 Railway regions); the hop to ElevenLabs is server-to-server.

Limitations

• Only the WebSocket transport is bridged. ElevenLabs WebRTC sessions go directly to ElevenLabs.
• Pacing assumes the client plays agent audio as soon as it arrives (as the SDKs do) and sends caller audio in real time. A client that buffers playback much longer should pass its latency as playout_ms.
• Caller audio in pcm_22050 / pcm_44100 is not processed (no engine rate).
• Gating mutes only after Turn’s decision. The first 90–160 ms of a noise or other-speaker burst reach ElevenLabs at −6 dB.