Models & benchmarks
Models and benchmarks
How Tenms Arena decides what ships, the Focus profile measurements, the open models and the training pipeline.
Sources: training/README.md, bench/README.md,
bench/tenms_arena/config.py, bench/tenms_arena/decision.py, and the live leaderboard
bench/results/LEADERBOARD.md. The Arena is still
running; this page deliberately does not copy its numbers — read the leaderboard.
Strategy
| Stage | What runs | State |
|---|---|---|
| DSP baseline | Focus/Turn/Pulse as described in the core README | production today; thresholds tuned on synthetic signals and checked in a handful of live calls |
| Arena-selected open models | A candidate replaces a DSP slot only if it wins the Arena’s decision rule (below) | crates/tenms-models (ONNX Runtime, outside the core): Silero VAD v6 on by default; Smart Turn v3.2 and CAM++ opt-in. Decisions and tables: crates/tenms-models/EVALUATION.md |
tenms-gru-v1 | One causal GRU with Focus mask, Turn, Pulse and speaker heads; any missing head falls back to DSP | first model trained 2026-11-06 on an M4 Max (Training report); its Focus head ships as listening_mini; the full GPU recipe (~250-350 GPU-hours) is the next step (Improving listening_mini) |
TEN models are benchmark-only and never shipped. TEN VAD and TEN Turn Detection carry
license conditions (non-compete) that rule out shipping; the decision rule marks them
benchmark-only and they can never win. The same applies to the LiveKit turn detector
(LiveKit Model License: only inside LiveKit Agents) and NISQA (CC BY-NC-SA weights)
(bench README).
Tenms Arena
“Every model or technique that ships in Tenms must win a measured experiment here: same frozen audio, same CPU, same thread count, same metric code, one documented decision rule” (bench README).
Method
- Same audio. Eval sets are built once by the training simulator
(
tenms_train.sim.CallSimulator) with fixed per-clip seeds, stored as FLAC + labels with a sha256 manifest verified on every load. Halftel8k(G.711 / AMR-NB-style codecs, packet loss, 8 kHz), halfwb16k. - Same CPU. Apple M4 Max, every thread pool pinned to 1 thread; wall-clock per call via
perf_counter_ns, warm-up excluded. - Streaming and causal. Candidates are fed chunk by chunk at their native hop behind 10 ms engine framing; algorithmic latency is measured (cross-correlation against a clean probe), not taken from papers.
- ASR is Deepgram nova-3 with a content-addressed cache and a billing ledger
(
bench/data/cache/deepgram_ledger.jsonl), capped at 175 min per run. - Uncertainty: bootstrap 95 % CIs over clips.
Tasks and budgets
From bench/tenms_arena/config.py (the source of truth; the README table is older):
| Task | Primary metric | Budget (1 thread) | Tie tolerance |
|---|---|---|---|
vad | frame F1 of the main speaker on the 10 ms grid | p99 per 10 ms frame ≤ 1 ms; algorithmic latency ≤ 40 ms | 0.005 |
eot | eot_score = 0.4 real-speech (AMI) streaming EOT accuracy + 0.3 Smart Turn v3.2 test accuracy + 0.3 simulated-call EOT accuracy | p99 per inference ≤ 50 ms; mean decision delay ≤ 600 ms | 0.005 |
enhance | Deepgram nova-3 corpus WER (lower) | p99 per frame ≤ 1 ms; measured algorithmic latency ≤ 10 ms | 0.002 |
speaker | EER on 0.5-1.5 s windows vs 3 s enrollment (lower) | p99 per embedding ≤ 25 ms | 0.002 |
speaker_ondemand | EER of a 1.0 s embedding computed only when an interruption candidate fires | p99 ≤ 100 ms; decision latency ≤ 1.1 s | 0.002 |
quality | Spearman(score, −WER degradation) | real-time factor ≤ 0.05 | 0.01 |
enhance_listen | UTMOS22 of the output (listening quality; opt-in listening_pro profile, never STT input) | enhancer + engine p99 per 10 ms frame ≤ 1 ms; end-to-end added latency ≤ 30 ms (Tenms stage + model + the engine’s resamplers; founder, 2026-11-06; ideal < 10 ms) | 0.02 (ties: background reduction, then latency; < 10 ms preferred within 0.05) |
Budgets derive from SPEC: 10 ms frames, p99 < 1 ms per frame per stream with all modules, ≤ 10 ms algorithmic delay.
Decision rule (decision.py)
- Only candidates that ran are ranked;
unavailable/failedrows are listed, never dropped. - Not-shippable candidates (license) are ranked for reference but can never win:
benchmark-only. - A shippable candidate is within budget iff every budget metric was measured and is ≤ the limit (unmeasured →
unverified). - Winner = best primary metric among shippable, within-budget candidates; ties within tolerance go to the lowest-CPU one.
- Shippable, over-budget candidates that beat the winner are
opt-in: they may ship only as an explicitly enabled, documented mode. - Everything else within budget:
runner-up.
Reading the leaderboard
bench/results/LEADERBOARD.md has one section per task:
winner, opt-in candidates, every candidate’s decision with the reason, candidates that
could not run and why, and Deepgram usage, plus a FOCUS_PROFILE section that combines
enhance and enhance_echo (below). Known caveats (bench README): read-speech eval sets
penalise acoustic end-of-turn models on “mid-turn” pauses between sentences; DeepFilterNet3
and RNNoise get offline zero-delay resampling; LaCo-SENet’s public weights are
VoiceBank+DEMAND (out of domain); Mamba-FVAD has no public code or weights.
Focus for speech recognition: the stt profile (2026-11-05)
Focus was tuned for people listening. Measured by what a speech recogniser makes of it
(Deepgram nova-3 corpus WER, frozen Arena sets, real engine knobs via
tenms-cli process --ns / --min-gain-db / --speaker-gate / --focus-profile), full noise
suppression hurts recognition while echo removal helps:
| Focus settings | enhance WER (77 clips: noise, babble, reverb, residual echo; no reference) | enhance_echo WER (33 echo-heavy calls, reference given) | combined (110 clips) |
|---|---|---|---|
| passthrough (no Focus) | 0.2071 | 0.3305 | 0.2503 |
full suppression (listening, -20 dB floor) | 0.2360 (+14 %) | 0.2903 (-12 %) | 0.2550 |
| noise suppression off (AEC + residual echo + speaker gate) | 0.2116 | 0.2785 | 0.2350 |
| -6 dB noise floor | 0.2143 | 0.2735 | 0.2350 |
-10 dB noise floor (stt) | 0.2107 (+1.7 %) | 0.2735 (-17.3 %) | 0.2327 (-7.0 %) |
Per condition (enhance): stt noise 0.125 / babble 0.422 / residual echo 0.299 / clean
0.047 / 8 kHz 0.227, vs full suppression 0.169 / 0.447 / 0.312 / 0.047 / 0.260 and
passthrough 0.105 / 0.444 / 0.280 / 0.047 / 0.216. enhance_echo 8 kHz / 16 kHz: stt
0.342 / 0.214, full 0.342 / 0.245, passthrough 0.389 / 0.280.
Decision: FocusProfile::Stt = AEC + residual-echo suppression (-40 dB echo floor) +
main-speaker gate + noise suppression with a -10 dB floor (lowest combined WER), and it
is the default wherever Focus output feeds an STT: Pipecat TenmsFocusFilter / TenmsCall,
LiveKit TenmsFocus / TenmsLiveKit, Edge /v1/stream (start.focus_profile, default
"stt") and the ElevenLabs bridge (?focus_profile=). listening (full suppression)
remains the engine / SDK default for audio people hear (it scored best by ear). Caveats:
bootstrap 95 % CIs of each set’s WER are about ±0.05, so -6 dB, -10 dB and noise
suppression off are statistically tied; the decision is “gentle or no noise suppression,
keep echo removal”. -15 dB and speaker-gate-off variants are registered in the Arena but
were not run (Deepgram budget: 38 new minutes for this study). Every external neural
enhancer measured (GTCRN, DeepFilterNet3, RNNoise, LaCo-SENet) is worse on WER and over
the 1 ms / 10 ms budget.
Listening quality: the opt-in listening_mini and listening_pro profiles (2026-11-06)
Question. The founder heard background music survive both Focus profiles on enhance_0001
(the DSP suppressor lowered the level in speech pauses by 0.3-0.5 dB: it handles stationary noise,
not music or babble). Can a neural enhancer after Tenms echo cancellation fix that for audio people
listen to (recordings, review, listen-in), within a latency the product can carry? Speech-recognition
input is out of scope (neural enhancers raised WER in enhance), WER is recorded only.
Set (enhance_listen, frozen, 48 clips): a balanced 3 x 2 x 2 design - background MUSAN music
(fma tracks, 0-15 dB SNR) / babble (3-6 LibriSpeech talkers, SIR 0-12 dB) / stationary SLR28 noise
(-5..15 dB) x 8 kHz telephony (G.711 / AMR-NB, packet loss) / 16 kHz wideband x no echo / raw agent
echo with double talk and the far-end reference. Each call is processed at its own rate.
Rows. <model>:pipeline = the product: tenms-cli --focus-profile listening --ns off (AEC +
residual-echo suppression with the reference, speaker gate, no DSP noise suppression), then the model;
<model>:alone = the model on the raw mic (reference only). Metrics: UTMOS22 strong, DNSMOS P.835,
SI-SDR / PESQ / ESTOI vs the clean target, background reduction in the target’s speech pauses
(10 log10 E_mic / E_out), speech loss in target-dominated bins, Deepgram nova-3 WER on a fixed
12-clip subset; latency by cross-correlation of the whole row at 8 and 16 kHz plus the engine
resamplers’ group delay that the offline resampling omits; CPU of the model calls, 1 thread.
| row (product pipeline unless noted) | UTMOS (95 % CI) | music / babble / stationary | DNSMOS | PESQ | ESTOI | bg reduction dB (music / babble) | speech loss dB | WER (12 clips) | latency 8 / 16 kHz ms | licence |
|---|---|---|---|---|---|---|---|---|---|---|
| passthrough | 1.37 (1.30-1.45) | 1.42 / 1.25 / 1.45 | 1.84 | 1.36 | 0.550 | 0 | 0.10 | 0.353 | 0 / 0 | - |
Tenms listening (DSP, today’s default) | 1.50 (1.40-1.61) | 1.57 / 1.30 / 1.62 | 1.98 | 1.35 | 0.527 | 5.9 (7.4 / 3.2) | 1.79 | 0.364 | 8 / 5 | proprietary |
Tenms stt | 1.48 | 1.57 / 1.28 / 1.58 | 1.92 | 1.35 | 0.534 | 4.4 | 1.68 | 0.326 | 8 / 5 | proprietary |
| FastEnhancer-M (DNS, 16 kHz wav2wav) | 2.12 (1.90-2.34) | 2.36 / 1.60 / 2.39 | 2.57 | 1.52 | 0.578 | 37.6 (43.8 / 13.2) | 3.52 | 0.370 | 30 / 27 | MIT |
| DeepFilterNet3 low-latency (DFN3_ll) | 2.07 (1.85-2.27) | 2.35 / 1.49 / 2.37 | 2.60 | 1.49 | 0.564 | 26.6 (29.9 / 11.6) | 4.70 | 0.467 | 20 / 17 | MIT / Apache-2.0 |
| faster-enhancer.c (int8 FastEnhancer-M, 48 kHz, VoiceBank) | 2.02 | 2.16 / 1.61 / 2.29 | 2.40 | 1.48 | 0.565 | 18.7 | 3.37 | - | 28 / 25 | MIT |
| RNNoise | 1.79 | 1.96 / 1.30 / 2.10 | 2.37 | 1.37 | 0.439 | 22.3 | 8.21 | 0.625 | 30 / 27 | BSD-3-Clause |
| DPDFNet-4 (16 kHz) | 2.25 (2.01-2.48) | 2.53 / 1.64 / 2.58 | 2.68 | 1.65 | 0.607 | 42.1 (51.7 / 11.4) | 3.66 | 0.375 | 58 / 55 | Apache-2.0 |
| DPDFNet-8 rate-matched (8 kHz export on 8 kHz calls) | 2.24 | 2.58 / 1.63 / 2.50 | 2.69 | 1.72 | 0.637 | 32.4 | 3.19 | 0.332 | 58 / 55 | Apache-2.0 |
| DPDFNet-2 rate-matched | 2.20 | 2.52 / 1.58 / 2.50 | 2.71 | 1.74 | 0.633 | 35.3 | 2.84 | 0.370 | 58 / 55 | Apache-2.0 |
| FastEnhancer-L | 2.18 | 2.41 / 1.66 / 2.47 | 2.55 | 1.57 | 0.567 | 39.7 | 4.57 | - | 39 / 36 | MIT |
| DeepFilterNet3 (default, 2-frame look-ahead) | 2.06 | 2.27 / 1.56 / 2.34 | 2.56 | 1.50 | 0.553 | 31.1 | 5.06 | 0.500 | 40 / 37 | MIT / Apache-2.0 |
| FastEnhancer-S / -B / -T | 1.97 / 1.93 / 1.76 | 31.0 / 31.1 / 27.0 | 0.38 / 0.39 / 0.47 | 38 / 35 | MIT | |||||
| UL-UNAS / GTCRN / FSPEN (VoiceBank re-impl.) | 1.78 / 1.73 / 1.58 | 20.0 / 20.9 / 10.6 | 0.45 / 0.41 / - | 38 / 35 | MIT | |||||
| clean target (reference) | 3.54 | 0.054 |
Full table (every row, :alone rows, per-rate CPU, per-clip values):
bench/results/LEADERBOARD.md / bench/results/enhance_listen/2026-11-06.json.
Not run: CoFi-Lite (no licence file: all rights reserved), “One Model, Many Latencies” RE-USE
(NVIDIA non-commercial licence, CUDA-only Mamba step), LiSenNet redesign (HF claroche1/LiSenNet:
VoiceBank-only mask network with host-side features not ported; the original LiSenNet export failed
the latency probe), FSPEN official (no weights; the FastEnhancer authors’ re-implementation is
measured), LaCo-SENet (VoiceBank-only weights, ~240 ms per frame in enhance; its enhance_listen
run did not finish on the shared machine). TEN models are never candidates.
Latency tiers (end-to-end, worst call rate, real resamplers included):
- < 10 ms: no neural candidate. The shortest is DFN3_ll alone (10 ms measured at 16 kHz + 2 ms of 16/48 kHz resampling = 12 ms; 15 / 12 ms at 8 / 16 kHz); STFT windows of 20-32 ms set the floor for every public model. Only the DSP profiles (5 ms at 16 kHz, 8 ms at 8 kHz) are under 10 ms.
- <= 30 ms: FastEnhancer-M (30 / 27 ms), DFN3_ll (20 / 17), faster-enhancer.c (28 / 25), RNNoise (30 / 27). DPDFNet (55-58 ms: 20 ms window + 4 hops of look-ahead), FastEnhancer-S/B/T, GTCRN and UL-UNAS (16 ms hop: 35-38 ms behind 10 ms framing) and DFN3 default (37-40) are out.
CPU (shared machine). The trainer and a VM kept the load average at 20-70 for the whole session
(pausing the trainer was not permitted); wall-clock p99 then measures the scheduler (FastEnhancer-M
passes 1.0-12.8 ms), so arena retime uses the calling thread’s CPU time (preemption excluded, audio
QoS) - still ~2.2x inflated at load ~50: FastEnhancer-M 1.73, -B 0.91, -T 0.57, GTCRN 1.85, UL-UNAS
2.25, DPDFNet-2 2.50, RNNoise 1.59 ms per frame; subprocess binaries by child CPU, amortized: DFN3 0.87,
DFN3_ll 1.84, faster-enhancer.c 1.47. The least-contended measurement of the integrated Rust runner
(tests/enhancer.rs perf, load ~7): FastEnhancer-M p99 0.80 ms per 10 ms frame at 16 kHz, 0.83 ms at
8 kHz (p50 0.66), FastEnhancer-B 0.40, -T 0.25, GTCRN 0.81, UL-UNAS 1.04, DPDFNet-2 1.18-1.33; the
engine adds ~0.02-0.03 ms. Re-run uv run arena retime --task enhance_listen on an idle machine before
relying on the Arena’s CPU column.
Decision. By the founder’s rule (<= 30 ms end-to-end, <= 1 ms p99, best UTMOS; tie-break
background reduction then latency; < 10 ms preferred within 0.05 UTMOS) the in-limit neural rows are
FastEnhancer-M (UTMOS 2.12), DFN3_ll (2.07), faster-enhancer.c (2.02) and RNNoise (1.79). FastEnhancer-M
is best by more than the 0.02 tie tolerance (the bootstrap CIs overlap: ±0.2), so it wins as long as it
fits the CPU budget - which the integrated Rust runner shows at low load (0.80 / 0.83 ms p99) and the
contended Arena re-time could not (1.73 ms thread CPU at load ~50; the leaderboard therefore records
FastEnhancer-M as opt-in and DSP listening as the measured in-budget winner until an idle re-time).
No candidate is under 10 ms, so the preference rule does not apply. It gains +0.62 UTMOS over DSP
listening and removes 37.6 dB of background in speech pauses (43.8 dB on music, 13.2 dB on babble) vs
5.9 dB, at 3.5 dB speech loss (DSP 1.8) and an unchanged recorded WER (0.370 vs 0.364, 12 clips).
DPDFNet is the best-sounding model measured (UTMOS 2.20-2.25) and is rejected on latency (55-58 ms).
listening_pro ships FastEnhancer-M, opt-in, never for STT input; at 8 / 24 / 48 kHz it reads the
engine’s internal 16 kHz Focus output (one resampler: 30.0 ms end to end at 8 kHz; with the two extra
resamplers of a naive chain it would be 33 ms).
Within the 10 ms / 1 ms budget the Arena’s winner is our own model (tenms-gru-mac:listening,
leaderboard row 37: UTMOS 1.545 vs DSP listening 1.497, background reduction 12.1 vs 5.9 dB - music
13.3, babble 5.8 - speech loss 2.4 dB, 8 / 5 ms, 0.061 ms p99 per frame). It ships as the opt-in
listening_mini profile: the listening settings plus the model’s Focus head (gain = min(DSP
gain, mask)), from the focus-only file models/tenms-mini-focus-v1.tnw (= tenms-gru-v1-mac-focus.tnw,
int8 TNW1, 0.22 MB, sha256 ed97516c...dc18 pinned in models/manifest.json), so Turn / Pulse / the
speaker lock stay DSP (their heads did not win). The Arena row used the focus + turn + pulse file
(-fpt); the Focus mask is the same trunk and head, the Turn head there only moved the VAD that
gates speaker enrolment. Never for STT input (the Focus head raised WER: 0.2396 vs 0.2107). Product
measurements (Rust, crates/tenms-core/tests/listening_mini.rs, crates/tenms-edge/tests/listening_mini.rs,
M4 Max): measured = reported latency at 8 / 16 / 24 / 48 kHz (8 / 5 / 7 / 7 ms, unchanged from
listening); 16 kHz frame p50 40 us / p99 48 us with all modules and no reference (DSP listening
6 / 7 us); Edge session ~4.3 ms CPU per call-second vs ~1.0 for listening (same audio); zero
allocation per frame; stt / listening sessions on a replica that has it loaded are bit-identical.
| tier | latency 8 / 16 kHz | CPU p99 per frame | background removed | UTMOS | runtime |
|---|---|---|---|---|---|
listening (DSP) | 8 / 5 ms | 0.02 ms | 5.9 dB | 1.50 | tenms-core |
listening_mini | 8 / 5 ms | 0.06 ms | 12.1 dB | 1.55 | tenms-core (TNW1 int8) |
listening_pro | 30 / 27 ms | 0.8 ms | 37.6 dB | 2.12 | tenms-models (ONNX Runtime) |
Shipped open models (tenms-models)
| Model | License | Slot | Default | Why (EVALUATION.md) |
|---|---|---|---|---|
Silero VAD v6 (silero_vad_v6.onnx, 2.3 MB) | MIT | VAD, replaces DSP VAD when no echo is possible, with a DSP rescue for voices Silero rejects | on | frame F1 0.790 (plain replace 0.797) vs 0.742; streaming EOT accuracy sim 0.873 vs 0.737, real 0.790 vs 0.781; missed onsets 3.4 % (replace 5.6 %); interruption fixtures unchanged; frame p99 within 1 ms (0.6-0.9 ms on an idle CPU) |
| Smart Turn v3.2 CPU int8 (8.7 MB) | BSD-2-Clause | turn-end confirm / veto at pauses | opt-in | composite eot_score 0.839 vs 0.738 (Silero only) at 598 ms mean delay, but real-speech streaming accuracy 0.759 vs 0.799 and inference p99 over the 50 ms CPU budget under load |
FastEnhancer-M DNS (fastenhancer_m_dns.onnx, 2.0 MB) | MIT | neural enhancer of the opt-in listening_pro Focus profile, after the engine’s echo cancellation | off (session opt-in) | enhance_listen: UTMOS 2.12 vs 1.50 (DSP listening), background reduction in pauses 37.6 vs 5.9 dB; 27 / 30 ms end to end at 16 / 8 kHz; Rust p99 0.80 ms per frame (low load); parity with the official ONNX loop 1.0e-7 |
| CAM++ WeSpeaker (29 MB) | Apache-2.0 | on-demand speaker check of interruption candidates | opt-in | EER 0.133 (1 s) vs DSP 0.41, but at a threshold that keeps true barge-ins (FRR 2 %) it blocks 3/120 other-voice barge-ins, the same as DSP |
scripts/fetch_models.sh downloads them (four files: Silero, Smart Turn, CAM++, FastEnhancer-M) into models/ and verifies the sha256 pinned in
models/manifest.json (files are gitignored). Findings worth knowing: ONNX Runtime <= 1.28
(what ort 2.0.0-rc.13 links) mis-optimises the CAM++ graph (embeddings lose all speaker
information), so CAM++ loads with graph optimisation disabled (as fast); the int8 Smart Turn
output differs between ORT versions (0.736 on 1.22 vs 0.616 on 1.28/1.31 for one clip), so the
reference fixtures pin ORT 1.31 (the Arena’s). Our Rust front-ends reproduce the Arena exactly:
Silero F1 0.7973, Smart Turn test accuracy 0.960, CAM++ EER 0.1620 / 0.1327 / 0.2315.
Training pipeline
First trained model: tenms-gru-v1-mac (2026-11-06, local Apple M4 Max)
Config training/configs/mac.yaml:
- Data: LibriSpeech train-clean-100, 16 AMI IHM meetings (never EN2002), MUSAN music+noise, SLR28 noises/RIRs minus the Arena’s files.
- Labels: foreground VAD labels; ASR-risk from the simulator proxy. No teacher or distillation.
- Training: 10k steps at batch 192, ~1.9M simulated 8 s calls, 9 h 57 min on MPS. QAT int8 drop 0.013 dB.
- Files:
models/tenms-gru-v1-mac{,-fpt,-focus,-turn,-pulse}.tnw(all heads / focus+turn+pulse / one head; local experiment exports, gitignored). The Focus file ships, renamed, asmodels/tenms-mini-focus-v1.tnw(committed) for thelistening_miniprofile. - Arena candidates:
tenms-gru-mac*(bench/tenms_arena/candidates/tenms_model.py).
| head | Arena (2026-11-06) | decision |
|---|---|---|
| Focus, STT | WER 0.2396 vs passthrough 0.2071 / DSP :stt 0.2107 (babble 0.509 vs 0.444); with AEC 0.2819 vs 0.2735 | no |
| Focus, listening | winner enhance_listen: UTMOS 1.545 vs DSP 1.497, bg reduction 12.1 vs 5.9 dB (music 13.3, babble 5.8), 0.061 ms p99, 8 ms | ships as the opt-in listening_mini profile |
| Turn | VAD winner (F1 0.803 vs Silero 0.797); EOT 0.763 > Smart Turn 0.745 but 1158 ms mean delay (opt-in); barge-in fixtures 78/96 vs 96/96 | not as default |
| Pulse | quality SRCC 0.447 vs UTMOS 0.479 (DSP 0.128) | runner-up, no |
| speaker | EER 0.468 vs DSP 0.426 | no |
Full numbers: training/runs/mac/ (see REPORT.md there).
training/ trains tenms-gru-v1 and exports int8 TNW1. Binding contracts are in SPEC; the
training README’s “Contract notes” pin every detail the SPEC leaves open (STFT framing,
quantisation rounding, file layout, golden fixture semantics).
Components
| Part | What it does |
|---|---|
Call simulator (tenms_train/sim) | Phone calls with turns and mid-turn pauses, agent playback and echo paths, 1-2 interferers (SIR 0-20 dB), noise (−5 to 30 dB SNR), RIRs, cough/laugh events, backchannels, G.711/Opus/AMR-NB channels, packet loss with PLC, clipping, AGC. Deterministic per seed. Label modes main, foreground (Mamba-FVAD-style), any. |
Datasets (tenms_train/data) | Registry with licenses; resumable, checksummed downloader; --commercial-only drops non-commercial rows (WHAM!, NC FSD50K clips, CANDOR unless cleared, AudioSet audio) |
Labels (label.py) | Offline ASR-risk from Deepgram WER and DNSMOS MOS, cached and resumable |
| Teacher | Wide, bidirectional, non-causal model for distillation |
| Student | Causal GRU H = 128 (~250k parameters) |
| Losses | Focus SI-SDR + IRM MSE + over-suppression penalty; Turn BCE + latency-aware detection-delay and early-cut-off terms; auxiliary next-onset and endpoint-anticipation heads (training only); Pulse Huber/BCE; speaker AAM-softmax or contrastive; distillation |
QAT (qat.py) | int8 fake-quant; gate: int8 SI-SDR drop < 0.2 dB (exit code 2 otherwise); optional pruning + angle regulariser |
Export (export_tnw.py) | int8 TNW1 with metrics and git sha in meta; optional ONNX step (informational) |
Evaluation (evaluate.py) | Agent-centric metrics, baselines (Silero, Smart Turn, DeepFilterNet, FireRedVAD), benchmark hooks |
Compute budget (estimates, not measured; training README)
| Stage | Hardware | Wall-clock |
|---|---|---|
| Labels (200k × 8 s) | CPU + Deepgram | 6-10 h (~900 h of audio sent to the API) |
| Teacher (200k steps) | 8× A100 80GB | 10-16 h (100-130 GPU-h) |
| Student (300k steps) | 8× A100 + 96 vCPU | 8-12 h (60-100 GPU-h); CPU-bound by the simulator |
| Distill (100k steps) | 8× A100 | 6-8 h |
| QAT (20k steps) | 1× A100 | 1-2 h |
| Evaluate | CPU | 1 h |
| Total | ~250-350 GPU-hours (a single L4/A10 works at ~5× wall-clock) |
Release targets (training README; not yet measured)
| Module | Metric | Target |
|---|---|---|
| Focus | SI-SDR improvement | ≥ 8 dB |
| Focus | downstream WER vs unprocessed | ≤ −10 % relative, never worse on clean |
| Focus | int8 vs fp32 SI-SDR | drop < 0.2 dB |
| Turn | EOT accuracy at 0.5 | ≥ 0.90 |
| Turn | false cut-off in mid-turn pauses | ≤ 5 % |
| Turn | mean detection delay | ≤ 250 ms |
| Turn | VAD F1 / foreground selectivity | ≥ 0.93 / ≥ 0.90 |
| Turn | barge-in vs backchannel F1 | ≥ 0.85 |
| Pulse | SNR MAE / MOS MAE | ≤ 3 dB / ≤ 0.35 |
| Pulse | ASR-risk AUROC | ≥ 0.85 |
| Speaker | same/different AUROC (3 s enrollment) | ≥ 0.95 |
2026 research incorporated
From the training README (“2026 research folded in”) and the bench README:
| Idea | Where in the code | Reference |
|---|---|---|
| Foreground (dominant-speaker) VAD labels without enrollment; competing-speaker augmentation; foreground-selectivity metric | sim.label_mode: foreground, evaluation.py; Arena VAD “FG sel.” column | Mamba-FVAD, arXiv 2609.19856 (no public code/weights, so not benchmarked) |
| Time-to-next-speech-onset regression | aux_turn[0], losses.aux_turn_loss | Next-Turn, arXiv 2606.18094 |
| Endpoint anticipation up to 2.5 s ahead | aux_turn[1:6]; Arena candidate endpoint_anticipation | Endpoint Anticipation, arXiv 2606.13450 |
| Per-layer structured pruning + self-distillation, angle-based QAT | qat.py --prune-hidden K --angle-reg L, prune.py | kiloVAD, arXiv 2607.25870 |
| Latency-configurable causal enhancement (asymmetric temporal padding) — recommended next Focus architecture | configs/focus_laco.yaml (stub; needs a SPEC revision, new arch, look-ahead buffer) | LaCo-SENet, Interspeech 2026 |
| Turn benchmarks | evaluate.py --bench turnbench=… ihbench=… codett=… | TURNBENCH arXiv 2608.25218; IHBench arXiv 2606.19595; CoDeTT |
| MOS benchmarks for Pulse | --bench voicemos26=… urgent26=… | VoiceMOS Challenge 2026; URGENT 2026 |
| VAD baseline | --baselines fireredvad; Arena candidate fireredvad_stream | FireRedVAD (Apache-2.0) |