tract 0.23.9 adds a mixture-of-experts operator

On 15 October 2026 Sonos published tract 0.23.9. The GitHub release, authored kali, was created at 12:35 UTC and published at 12:37 UTC. The release body is empty. The notes are the changelog on that tag, under the heading 0.23.9 - 2025-10-15. The year in that heading is 2025. crates.io records the 0.23.9 publish at 12:19:26 UTC on 15 October 2026.

The facade crate in api/rs/Cargo.toml is tract 0.23.9, license MIT OR Apache-2.0, described as “Tiny, no-nonsense, self contained, TensorFlow and ONNX inference.” LICENSE-MIT and LICENSE-APACHE are both on the tag. The workspace also contains NNEF, ONNX, TensorFlow, and TFLite members. crates.io repeats the MIT OR Apache-2.0 line for 0.23.9.

The MoE operator

The changelog’s transformers section adds a tract_moe_ffn operator for mixture-of-experts feed-forward blocks: router top-k, softmax, sigmoid, or raw gating, grouped expert GEMMs, SwiGLU, and weighted scatter-add. It says this covers Mixtral, OLMoE, GPT-OSS, Qwen3, and Granite, and that experts run as per-expert subplans so f16, bf16, and Q40 exports work, including biased or clamped ones. It credits Julien Balian.

transformers/src/ops/moe_ffn.rs is on the v0.23.9 tag. The same path on v0.23.8 was absent. The file’s comment says tract_moe_ffn is the NNEF primitive, with gates softmax_topk, softmax_all, sigmoid, or raw, and an activation list that includes swiglu. It names Mixtral, Qwen MoE, OLMoE, Llama4, and gpt-oss. It does not name Granite. It says facts use f16 or f32, that expert tensors can carry block-quantized storage, and that a Q40 path exists. The string bf16 does not appear in that file. The Granite line and the bf16 line are the changelog’s.

There is no independent benchmark of this operator. Figures elsewhere in the same changelog, including the memory example below, are the project’s own.

KV cache, and a runtime marked WIP

The same section says CPU decode now updates its KV cache in place: Sdpa::codegen applies a fusion that previously sat behind a transform nobody used, so each turn copied the whole past. It says the GPU KV cache is no longer rebuilt every turn. It is a buffer with spare capacity, written at a cursor, so an append is amortized constant work instead of work that grows with the transcript, and truncation and rollback stay on the device. That description is the changelog’s. It was not rerun here.

A later heading is labeled WIP, Deployment / autobatch. It describes a deploy runtime, selected with runtime_for_name("deploy"), that reads a TOML config and can autobatch. The changelog says it is available behind unstable-autobatch. That is a work in progress behind an unstable feature.

The arena figure, and which version crates.io calls newest

Under GPU, the changelog says a model with concrete shapes now gets a memory arena on CUDA and Metal without hints. It says that before this, every intermediate had its own buffer, and that an SDXL VAE decoder needed about 61 GB instead of a 2.9 GB arena. That comparison is the project’s.

The crates.io API, read on 15 October, sets newest_version to 0.21.19 and max_version to 0.23.9. The publish times on that API are 0.23.9 at 12:19 UTC, 0.22.5 at 13:22 UTC, and 0.21.19 at 13:33 UTC, so the later LTS publish is what the index calls newest. Pin 0.23.9 for this release. At the same reading, 0.23.9 showed 0 downloads. The crates.io repository field still names the older snipsco/tract path. The tag opened here is sonos/tract, which showed 3,088 stars at 17:43 UTC.

GitHub’s release list includes 0.22.5 and 0.21.19 the same day. Their release notes are empty, and the changelog files on those tags have no heading for the patch. They are the LTS publishes, not this feature list.

ort 2.0.0-rc.14 is a different piece of software: pyke’s ONNX Runtime binding, still a release candidate, with its own environment change.