reuse-rs: a plain-Rust port of NVIDIA's live speech denoiser runs faster on the CPU than on the GPU

NVIDIA’s Real-time RE-USE is a small streaming Mamba model that cleans noisy speech one 20 ms frame at a time. A developer who writes as altunenes wanted it running live from a microphone in Rust, and wrote up what happened when he stopped forcing it onto the GPU.

The GPU route

He exported the model to ONNX and ran it through ort. Mamba’s selective scan has no ONNX operator, so he mapped it onto ONNX Runtime’s com.microsoft.LinearAttention contrib op and filed two issues along the way. On his M3, per 20 ms frame, he measured:

Backend	4 layers	8 layers
ONNX Runtime, WebGPU	16.9 ms	20.6 ms
ONNX Runtime, CPU	about 70 ms
reuse-rs, plain Rust CPU	8.4 ms	14.7 ms

CoreML split the graph and fell back to the CPU at 168 to 198 ms. Anything over 20 ms falls behind the microphone.

Why the CPU won

Doubling the layers on WebGPU added only 3.7 ms, which told him most of the time was fixed overhead per frame, launching kernels and syncing back, not the model itself. A live stream cannot batch future frames, so that overhead never amortizes. His Rust version uses im2col plus one matrix multiply (Accelerate on macOS, the matrixmultiply crate elsewhere) and a fixed 16-float state that the compiler can vectorize. Memory fell from about 420 MB to about 150 MB, and the output stayed within 1.5e-6 of a float64 PyTorch reference, against 2.4e-3 for the ONNX export.

The caveats he gives

• All numbers are from one M3 with 16 GB. Without Accelerate, 8 layers takes 21.4 ms, so he suggests 4 to 6 layers on other platforms.
• His quality check is one VoiceBank-DEMAND clip: 16.4 dB SI-SDR at 4 layers, 16.7 dB at 8.
• It struggles when the background noise is other people talking.
• The crate is MIT or Apache 2.0, but NVIDIA’s weights use a noncommercial licence. A GStreamer element is in progress, not shipped.