Training report

Run: 2026-11-05 23:51 → 2026-11-06 ~11:45 IST, on one Apple M4 Max. Status: evaluated, not deployed.

This report covers how we trained the first real Tenms model, what it learned, and what the benchmark decided to ship. Every number comes from the run logs (training/runs/mac/), the Tenms Arena results (bench/results/*/2026-11-06.json, bench/results/LEADERBOARD.md) or the exported model metadata. The figures are generated from those logs by training/scripts/plot_training_report.py.

TL;DR

• One small model, four jobs. A causal 2-layer GRU (H = 128, int8, ~0.22 MB) with four output heads: Focus (noise/voice mask), Turn (voice activity, end of turn, interruptions), Pulse (call quality) and speaker (embedding). It runs every 5 ms inside the Tenms engine.
• Trained in about 10 hours on a laptop GPU (Apple Metal), on ~1.9 M simulated 8-second phone calls (≈ 4,200 h of audio). int8 quantisation lost only 0.013 dB of output quality.
• Fast: p99 48 µs per 10 ms frame (77 µs with echo cancellation). That is 5% of the 1 ms budget, with 8 ms added latency.
• The benchmark shipped only one head. The Focus head wins for audio people listen to: it removes 2× more background than our DSP (12.1 vs 5.9 dB) at 8 ms, against FastEnhancer’s 27–30 ms. It loses on audio sent to speech recognition. The Turn head has the best voice detection we have measured (F1 0.803), but it is too slow to decide end of turn. Pulse came second. The speaker head never learned.

1. Why train our own model

Tenms runs in the audio path of voice agents (Pipecat, LiveKit, ElevenLabs) under hard constraints: CPU only, a 30 ms hard / < 10 ms ideal latency budget, and < 1 ms of CPU per 10 ms frame. Our signal-processing (DSP) engine meets those numbers. But a listening test showed its weakness: on a call with background music it removed only 0.3–0.5 dB. Classic noise suppression tracks steady noise, and music and babble are not steady.

The open neural enhancers we benchmarked do remove music, but with 27–58 ms of delay, and every one of them made speech recognition worse. A model of our own can run inside the existing 5 ms STFT frame, with no added window. That is the only route to removing music under 10 ms.

2. The model

mic (16 kHz) → STFT (10 ms window, 5 ms hop, 81 bins) → log-power features (81)
  → Linear(81→128) + ReLU → GRU(128) → GRU(128)        causal, one step per 5 ms hop
  → heads:  focus   81  → sigmoid gain mask per frequency bin
            turn     4  → speech, end-of-turn-within-200 ms, backchannel, non-speech event
            pulse    6  → SNR, MOS, ASR-risk, overlap, echo, dropout
            speaker 32  → L2-normalised speaker embedding

The engine uses a head only if it is present in the file. Each head can ship or be left out on its own, and anything left out falls back to DSP.

3. Data

All training data is commercially licensed: CC BY 4.0 or Apache 2.0, built with prepare --commercial-only. Nothing the benchmark tests on is in the training set (training/runs/mac/heldout_check.json, passed: true).

The call simulator builds every training example on the fly:

• A main speaker, plus 1–3 interfering talkers at 0–20 dB SIR half the time.
• Noise or music at −5 to 30 dB SNR (90% of calls).
• Room reverberation, and agent echo through a random echo path.
• Telephony codecs (G.711, AMR-NB style band-limiting, real Opus), packet loss and clipping.

Labels are computed from the clean sources:

• Focus: an ideal mask for the main speaker only.
• Turn: foreground voice activity (only the dominant speaker counts), end-of-turn targets, pauses and backchannels.
• Pulse: quality targets.

ASR-risk labels for this run came from a simulator proxy (SNR, codec, loss, reverb), not from real Deepgram transcripts.

4. Training setup

Throughput: the GPU waited on data.

• A GPU step took about 2.7 s at any batch size. The bottleneck was the CPU simulator generating calls: 0.14 s per 8 s call per core when idle, 0.23–0.38 s on the shared machine.
• Fixes made during the run:
  • An in-process Opus codec; spawning ffmpeg had been half of the simulator’s time.
  • Random crops from long music files instead of caching each file whole.
  • Lower prefetch.
  • A cache fix for macOS worker processes.
• Raising the batch to 192 brought throughput to 41–58 examples/s (3.3–4.7 s per step).
• The 15,000 steps originally planned would have taken 15–20 h, so the run was cut to 10,000.

5. Training progress

The training loss fell from 2.30 (first-100-step average) to 0.96 with no divergence or NaNs. Most of the drop came in the first 500 steps, followed by a slow, steady decline.

Validation ran on held-out simulated calls (dev-clean speakers, dev-split noises and rooms) every 1,000 steps:

• Focus improved steadily (+1.15 dB), and was still improving slightly at the end.
• Pulse improved steadily (−24% loss).
• Turn plateaued after about 4,000 steps.
• The speaker head never learned. Its loss stayed at chance (ln 192 ≈ 5.26), because about 250 training speakers at batch 192 is far too few for this kind of embedding loss.
• One side effect of the speaker head: it dominated the total validation loss, so the “best” checkpoint froze at step 3,000. The int8 fine-tune therefore started from step 10,000, the best checkpoint on every other head.

The learning-rate schedule ran as configured. The gradient norm stayed between about 0.8 and 1.0 throughout, with no spikes.

6. int8 quantisation

A 1,000-step quantisation-aware fine-tune at lr 1e-4 used int8 fake-quantisation per tensor.

7. Results: the Arena decides

The Tenms Arena tests every candidate on frozen evaluation sets it never trains on. It runs each one on the same CPU with one thread and measures downstream outcomes such as Deepgram word error rate, not only signal metrics. A head ships only if it beats the current default within the latency and CPU budget.

7.1 Focus, audio sent to speech-to-text (the voice-agent path)

The model produces much cleaner audio (2.97 vs 1.10 dB SI-SDR) yet more recognition errors. On clean speech it is identical (0.047). The losses come from babble (0.509 vs 0.444) and echo (0.340 vs 0.280). It is suppressing cues the recogniser uses. This is the “enhancement hurts ASR” effect reported in recent research (see §9). Decision: don’t ship for STT.

7.2 Focus, audio people listen to

Decision: ship for the listening profile - as the opt-in listening_mini Focus profile (models/tenms-mini-focus-v1.tnw, the Focus-only file, sha256-pinned; Edge, CLI, Python SDK).

• Within the budget, the trained model is the winner: it doubles background removal against DSP at the same 8 ms.
• FastEnhancer-M removes far more, but needs 27–30 ms and about 15× the CPU. That keeps it an opt-in mode (listening_pro) for recordings and listen-in.

7.3 Turn: voice activity and end of turn

Decision: don’t make it the default Turn head.

• It has the best voice detection and end-of-turn accuracy we have measured.
• But it waits too long to decide (1.16 s), and it misses 18 of 96 barge-ins.
• The engine can’t yet take only the head’s speech probability, which is the part that wins. Adding that option is the next step.

7.4 Pulse: call quality

Decision: don’t ship by the rule (runner-up).

• It is 3.5× better than DSP Pulse, and it fixes the SNR estimate that was badly wrong in the listening test.
• It’s a strong candidate for a product decision outside the strict rule.

7.5 Speaker

The speaker head scored EER 0.468, worse than the DSP baseline’s 0.426, so it doesn’t ship. CAM++ (EER 0.133 on demand) already wins this task.

8. Ship decision

Shipped file: models/tenms-mini-focus-v1.tnw (= tenms-gru-v1-mac-focus.tnw, Focus head only) for the listening_mini profile; Turn, Pulse and the speaker lock stay DSP there. Experiments can use models/tenms-gru-v1-mac-fpt.tnw (Focus + Turn + Pulse; local, not committed).

9. Listen to it

The listening page plays six simulated noisy calls through every version, with Deepgram transcripts and word error rates: unprocessed, Tenms DSP, FastEnhancer listening_pro, the trained model and the clean original. On the music clip the trained model removes 28 dB, against 0.4 dB for DSP.

10. Lessons

1. Cleaner is not the same as more recognisable. Our model nearly triples SI-SDR on STT audio and still raises WER. Recent work explains why:

   • over-suppression removes low-energy cues the recogniser relies on;
   • remixing a little of the original signal back in, or training against recogniser- or SSL-representation losses, closes the gap.

   Sources: arXiv 2607.11157, 2512.17562, 2602.20967, 2507.07631, 2609.11092.

2. On a laptop the simulator is the bottleneck, not the GPU. Moving codec work in-process and cropping long music files mattered more than batch size.

3. One model, four heads, per-head shipping works. We get the Focus win now without being held back by the heads that lost.

4. Watch what dominates your validation loss. A head stuck at chance froze the “best” checkpoint. Pick checkpoints per head, or exclude heads that aren’t learning.

5. Measure the outcome you sell. Signal metrics alone would have shipped the STT Focus head. The Arena’s downstream-WER rule stopped that.

11. Next steps

1. Focus for STT:
   • an ASR-aware loss (Deepgram-labelled shards via label.py, or an SSL-representation loss);
   • an adaptive mask floor and remix in the engine.
2. Turn:
   • an engine option to use only the model’s speech probability, which keeps the DSP interruption logic;
   • tune the end-of-turn threshold and minimum silence to bring the delay under 600 ms;
   • more barge-in-shaped training examples.
3. Speaker: many more speakers (LibriSpeech 360/500, VoxCeleb-style data), or keep CAM++.
4. Scale:
   • more data (train-clean-360, DNS noise, Fisher and Switchboard for telephone turn-taking);
   • the full GPU recipe: teacher, distillation, 300 k steps (training/GPU_RUNBOOK.md).
5. Real calls: design-partner recordings, the single biggest expected improvement.