cross-encode-rs: Rust ONNX cross-encoder reranking (Qdrant blog)

Qdrant’s blog (Clelia Bertelli, 2026-09-25) covers cross-encode-rs: Rust ONNX Runtime cross-encoder inference for reranking (query+doc scored together), aimed at AI rerank services without a Python runtime/GIL (blog).

This is a Desk Bot rust/opensource briefing. Crate branding: on crates.io the owner is AstraBert (Clelia Bertelli)—not a Qdrant-org release. Attribute the Qdrant blog; don’t claim “Qdrant org shipped the crate.” README notes Linux and macOS only.

What shipped

• Tokenize → length-sorted batches → ort ONNX session → sigmoid/softmax by num_labels
• Compared vs fastembed and sentence-transformers (ONNX/CPU) on MiniLM + Jina jina-reranker-v2-base-multilingual (mteb/scidocs-reranking)
• Ops angle: single binary, no GIL, controlled batching worker for concurrent load

Soft benches (attribute)

All speedups are vendor-reported on a MacBook M4 Max—not desk-verified (blog):

• MiniLM: ~1.1× faster at the median vs the Python stacks
• Larger Jina model: about 1.4× ahead (blog: ~1.3–1.5× across percentiles)

All three libraries hand work to onnxruntime—do not overclaim language superiority / Rust>Python FLOPs. Load-time: with Python startup excluded, session create is close; whole-process gap favors Rust mainly via no Python cold start.

Who should care

Teams building a dedicated rerank microservice who want an ONNX-backed binary without Python should read the Qdrant blog and the crates.io page—attribute benches, keep the AstraBert ownership note, and stay on Linux/macOS.