Improving listening_mini · Docs

Written 2026-11-06, after the first training run (docs/TRAINING_REPORT.md).

Today the two listening tiers sit here on the Arena’s enhance_listen set (48 music, babble and noise clips; bench/results/LEADERBOARD.md):

	`listening` (DSP)	`listening_mini` (our model)	`listening_pro` (FastEnhancer-M)
Background removed in pauses	5.9 dB	12.1 dB	37.6 dB
Music / babble	7.4 / 3.2 dB	13.3 / 5.8 dB	43.8 / 13.2 dB
Quality (UTMOS)	1.497	1.545	2.12
Speech loss	1.8 dB	2.4 dB	3.5 dB
Added latency	5–8 ms	8 ms	27–30 ms
CPU p99 per 10 ms frame	0.02 ms	0.06 ms	~0.8 ms

The goal: close most of the gap to pro without giving up what makes mini worth having. That means at most 10 ms of latency (the ideal; 30 ms is the hard limit) and well under 1 ms of CPU per frame. A realistic first target is ≥ 25 dB background removed and UTMOS ≥ 1.9 at ≤ 10 ms. Every candidate goes through the Arena gate before it ships (bench/README.md).

Why `pro` is better today

Cause	Evidence	What it costs us
Frequency resolution. Our STFT is 10 ms / 81 bins (100 Hz per bin). FastEnhancer and DPDFNet use 20–32 ms windows.	That window is exactly where `pro`’s 27–30 ms comes from (`enhance_listen` latency probe)	Harmonics of music and of a second voice smear together in 100 Hz bins, so a per-bin mask can’t separate them
Training scale. 10 k steps, ~1.9 M simulated calls, one laptop, no teacher.	The GPU recipe plans 300 k steps plus teacher and distillation (`training/README.md`)	Focus SI-SDR was still rising at the end (3.03 → 3.05 dB, `fig2_validation.png`)
Shared capacity. Four heads share 128 GRU units, and the speaker head sat at chance all run.	Speaker AAM loss 5.18 ≈ ln 192 (`TRAINING_REPORT.md` §5)	Focus competes for capacity and gradient with heads that don’t ship in `mini`
Mask type. Real-valued gain per bin, no phase.	SPEC `tenms-gru-v1`	Can’t repair phase or fill in masked harmonics; caps quality at low SNR
Data. 42.6 h of music, 100 h of LibriSpeech, no real calls.	`training/configs/mac.yaml`	Narrow coverage of music genres, TVs and real phones
Objective. SI-SDR + mask + over-suppression, with no perceptual term.	`tenms_train/losses.py`	Optimises signal error, not what people hear (UTMOS tracks perception)

We have plenty of CPU headroom: mini uses 0.06 ms of a 1 ms frame budget (6%). We can afford a much larger model before CPU becomes a concern.

The levers, in the order to try them

Status: ready = works with today’s code, just change the config. code = needs new code. spec = changes the model/STFT contract (docs/SPEC.md) shared by training and the Rust engine.

#	Lever	Status	Expected effect	Latency	CPU
1	Focus-only model: drop the speaker head, lower Turn/Pulse weights or train Focus alone	ready	All capacity and gradient go to separation	none	lower
2	Bigger model: H = 192/256, 3 GRU layers	ready (H, layers are config keys)	More capacity for music and babble	none	~2–5× (estimate; still well under 1 ms)
3	Train longer, on more data: 50 k–300 k steps; LibriSpeech 360/500, DNS noise, FSD50K, more music	ready (registry + downloader)	Focus was still improving at 10 k steps	none	none
4	Distil from `listening_pro`: use FastEnhancer/DPDFNet outputs as soft targets on music and babble	code (teacher adapter for ONNX enhancers in `distill.py`)	The student learns what the strongest causal models remove, including the 58 ms DPDFNet that can’t ship directly	none	none
5	Perceptual + multi-resolution STFT losses, optionally an SSL-representation loss	code (`losses.py`)	Trains for what UTMOS measures; recent work shows SSL losses help both listening and recognition (2507.07631)	none	none
6	Asymmetric analysis window: long analysis window (e.g. 32 ms) with a short synthesis window so output delay stays ≈ 5–10 ms	spec + code (engine STFT + training STFT)	Directly attacks the 100 Hz resolution limit, the main cause of the gap	stays ≤ 10 ms	small
7	Causal deep filtering / complex mask: filter over the last few frames per bin (DeepFilterNet-style, past frames only)	spec + code (TNW v2 head)	Recovers harmonics a real mask can’t; large gains at low SNR	none if past-only	~2×
8	Speaker-conditioned Focus (v2): feed the caller’s embedding (CAM++, already on-demand in `tenms-models`) into the mask	spec + code	Removes the other voice in babble and TV, where `mini` is weakest (5.8 dB)	none	small
9	Real-call fine-tune: design-partner recordings, pseudo-labelled by `listening_pro` as teacher	data + code	Closes the simulation-to-reality gap; the biggest expected long-term gain	none	none

Quantisation isn’t the bottleneck: int8 lost only 0.013 dB. Keep the QAT gate as it is.

Experiment plan: Mac and GPU tracks

Both tracks use the same configs; GPU runs just go bigger and longer. Each experiment ends in the same Arena gate.

Track A: Mac (Apple Metal, what we ran first)

The simulator, on the CPU, is the bottleneck (about 41–58 examples/s; TRAINING_REPORT.md §4). Keep runs to one night.

Run	What to change in a copy of `configs/mac.yaml`	Wall clock (estimate)
A1. Focus-only, H = 128	`model.heads: [focus]`, drop `aux`; `loss.weights: {focus: 1.0}`; `train.steps: 10000`	~10 h
A2. Focus-only, H = 192	as A1 + `model.hidden: 192`	~10–12 h (GPU step is not the limit)
A3. A2 + more music	add MUSAN music twice more or FSD50K music to `data.manifests`; `sim.noise.p: 0.95`	~10–12 h
A4. Fine-tune from the shipped model	`--init runs/mac/student/ckpt_step10000.pt`, Focus-only, `train.lr: 3e-4`, `train.steps: 5000`	~5 h

cd training
cp configs/mac.yaml configs/mac_focus192.yaml        # edit as in the table
caffeinate -is uv run python train.py --config configs/mac_focus192.yaml --out runs/mac_focus192
uv run python qat.py --config configs/mac_qat.yaml --init runs/mac_focus192/ckpt_last.pt --out runs/mac_focus192/qat
uv run python scripts/export_mac.py   # adjust paths inside, or use export_tnw.py --ckpt ... --out ...

On the Mac: quit Docker Desktop and other heavy apps (the VM alone cost ~1 CPU core in the first run), and keep train.num_workers around 10.

Track B: GPU box (`training/GPU_RUNBOOK.md`)

The GPU removes the step-time limit. Give the simulator enough CPU, at least 12–32 vCPU per GPU, or the GPU will wait on data just as the Mac did.

Run	Change	Scale (estimate)
B1. Focus-only H = 256, 3 layers, full data	`model: {hidden: 256, layers: 3, heads: [focus]}`; datasets: LibriSpeech 100+360, MUSAN, DNS noise, FSD50K (non-NC), SLR26/28 + DNS RIRs	100 k steps, 8× A100 ≈ 8–12 h
B2. Teacher + distillation	`configs/teacher.yaml` (wide, non-causal) → `distill.py` into the B1 student	+10–16 h teacher, +6–8 h distill
B3. Distil from `listening_pro` (lever 4)	after the teacher adapter is coded: FastEnhancer-M / DPDFNet outputs as targets on music and babble	as B2
B4. Perceptual loss (lever 5)	after `losses.py` gains MR-STFT/SSL terms	as B1
B5. Asymmetric window / deep filter (levers 6–7)	after the SPEC v2 change on both the Rust and the training side	as B1, plus a Rust release

# on the GPU box (see GPU_RUNBOOK.md for setup)
bash training/scripts/setup_gpu.sh
cp training/configs/base.yaml training/configs/focus_v2.yaml      # edit as in B1
STAGES="train_student qat evaluate export final" bash training/scripts/run_pipeline.sh
# or directly:
uv run --project training torchrun --standalone --nproc_per_node 8 training/train.py \
  --config training/configs/focus_v2.yaml --out training/runs/focus_v2

Bring each result back (scp …/tenms-gru-*.tnw models/) and run the gate below on the Mac.

The gate (same for every run)

cd bench
uv run arena run --task enhance_listen --candidates tenms-dsp:listening,tenms-gru-mac:listening,<your model>
uv run arena run --task enhance --candidates passthrough,tenms-dsp:stt,<your model>:stt   # STT safety check
uv run arena retime --task enhance_listen      # on an idle machine
uv run arena report

A new model replaces the shipped models/tenms-mini-focus-v1.tnw only if all of these hold:

More background removed (in pauses, on music and on babble) and higher UTMOS than the current listening_mini, outside the tie tolerance.
Speech loss ≤ 3 dB. Removing the noise must not eat the voice.
Latency ≤ 10 ms (the ideal; 30 ms is the hard limit for any tier) and CPU p99 ≤ 1 ms per frame, re-timed on an idle machine.
int8 QAT gate passes (≤ 0.2 dB drop) and Rust parity ≤ 10⁻³.
Never used on the STT path unless it also wins enhance WER. That’s a separate, harder gate.

Then listen to it on the listening page before shipping.

What not to spend time on

Quantisation tricks: int8 already costs only 0.013 dB.
Pushing the 10 ms window harder with a real-valued mask alone: resolution is the ceiling (lever 6 is the fix).
The speaker head inside this model: CAM++ already wins on demand (EER 0.133). Leave speaker identity to it, and use its embedding as an input (lever 8).

Why pro is better today