RustingStrata runs Qwen3.6-35B-A3B on a 12 GB RTX 3060 in pure Rust, splitting experts between GPU and CPU

RustingStrata is a new inference engine, written in Rust by Vasyl Trefilov, for running Qwen models in GGUF on a consumer GPU that is too small to hold them. Its reference machine is a 12 GB RTX 3060 running Qwen3.6-35B-A3B, an 11.7 GiB file that, as the README puts it, “does not fit in VRAM.”

How it works

The README describes the approach in a few lines:

• Experts split across GPU and CPU. “Experts that do not fit in VRAM run on the CPU at the same time as the GPU work, and the expert placement adapts to the routing it sees.”
• No C++ build step, no ggml. CUDA kernels are compiled at run time with NVRTC through the cudarc crate and cached. The same kernels also run on Vulkan, with the GLSL shaders compiled to SPIR-V and committed to the repo.
• Models: the Qwen3.5 and 3.6 family (qwen35moe, qwen35), plus Qwen3-Next, Qwen3.8-Flash-Next (qwen4exp) and Qwen3.
• Tools: strata calibrate measures the best settings for your machine once, and there are generate, chat, bench and serve commands.

strata serve speaks the OpenAI chat completions API, including tool calls, and handles one request at a time with one API key. The README warns that the server “treats all clients as one user”: cached-token counts and response times can reveal whether the previous request started with the same messages, “so do not share one server between people who must not learn each other’s prompts.”

The numbers, and what they do not show

All figures are the author’s, measured on one machine (RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM) against llama.cpp build 11876 tuned for the same file:

Qwen3.6-35B-A3B, UD-Q2_K_XL	RustingStrata (CUDA)	llama.cpp
Prefill, 4K prompt	2216 tok/s	~2236 tok/s
Decode after a 4K prompt	113.4 tok/s	66.5 tok/s
Decode, short prompt	118.2 tok/s	68.4 tok/s

So the gain is in decode, about 1.7 times llama.cpp’s speed in this setup. Prefill is level. The repo’s one-line GitHub description says “2x llama.cpp prefill,” but the README’s own table against tuned llama.cpp does not show that. The README also reports 98 to 99% greedy top-1 token agreement with llama.cpp over 20 prompts, and up to 160 tok/s with the model’s own multi-token-prediction drafts, which it says depends on the text.

A few limits from the README itself:

• Smaller cards are simulated. Figures for about 8 GB and 6 GB were “simulated on the 3060 with --vram-margin (not measured on real 8 GB or 6 GB cards).”
• Vulkan is tested on one GPU. The Vulkan path is “tested only on an RTX 3060.” Intel Arc lacks a required feature (shaderFloat64) and pre-RDNA AMD cards lack subgroup size 32.
• Release status. The README says “Status: v1.1.0,” and the repo has a v1.1.0 git tag, but there is no published GitHub release with notes.
• One author. It is a young project (the repo was created on 10 October) and we found no third-party reproduction of the numbers.

Source-available, not open source

RustingStrata ships under its own RustingStrata License 1.0. You may use, modify and distribute it, including commercially, and the license says you may “run a paid chat or API service that answers requests with RustingStrata.” What you may not do is sell, license for a fee, rent or charge for access to RustingStrata itself, or anything “marketed as a substitute” for it. Copies must keep the license and credit Vasyl Trefilov, and products built on it must carry the line “Uses RustingStrata by Vasyl Trefilov.” That resale restriction means it is not an OSI open-source license.

Why this matters

For mixture-of-experts models on a 12 GB card, the practical question is which experts sit in VRAM and which run on the CPU. RustingStrata answers it by adapting placement to observed routing, and on the author’s RTX 3060 that decodes about 1.7 times faster than llama.cpp with a hand-tuned expert split. It joins other Rust engines we have covered with a narrower hardware focus: Vade targets DGX Spark and hipfire targets AMD RDNA, while this one aims at the common NVIDIA card. If you own a 3060-class GPU it is a cheap test with strata bench, but treat the table as one person’s measurements and check the license before building a paid product on it.