Hugging Face tokenizers v1.0.0-rc.2: release candidate, not stable 1.0

On 2026-09-21, Hugging Face tagged Rust tokenizers v1.0.0-rc.2, a release candidate, not stable 1.0.0. Install path: cargo add tokenizers --pre. Companion post: tokenizers v1 on the HF blog.

This post draws on the GitHub pre-release and its blog. License: Apache-2.0 (repo LICENSE). Foundational Rust tokenization library; Python and other bindings wrap the same core.

Compatibility and architecture (HF’s framing)

HF says v1 aims to keep the same API, vocabulary / merge ranks, and token IDs as v0.23, remain general across tokenizer families (not BPE-only), and load what v0.23 loaded. Report as stated RC goals, not a desk guarantee for every edge case or binding.

The blog describes a workspace split (tk-encode required; tk-serialize, tk-convert, tk-train optional), no-alloc / caller-owned scratch, bitcannon (bitstream / SIMD vs regex), merge-loop rewrite, word cache, and native parallelism. Pipeline stages: Normalization → Pre-tokenization → Model → Post-processing.

Performance (attribute to HF)

From the HF blog / tokbench (Apple M4 Max, one thread for the encode range; blog may update):

• Encodes text 3 to 30 times faster than v0.23 (low end t5-base, high end gpt2)
• Scales at 76% of linear across eight workers
• Release notes also claim a ~6× smaller crate and reduced peak memory

These are Hugging Face’s numbers, not independent desk benchmarks. Python per-call overhead is not included in the published measurements.

Still before final 1.0.0

The blog lists remaining work (training-validation encoding path, optional offsets/masks, normalizer rework, simpler Python bindings, inference-only C/C++ for ExecuTorch / llama.cpp, and more). Exploratory GPU encode/decode ideas are not shipped.

Who should care

Teams on HF tokenizers who want the RC performance refactor while keeping v0.23 IDs should try --pre and read the tag plus blog. Treat this as a candidate, not a stable 1.0 cutover.

Update, 20 October 2026: rc.3 brings tokenizers to the browser

Hugging Face tagged v1.0.0-rc.3 on 19 October. The release notes call it “mainly a bindings release”: WebAssembly bindings are now on npm as tokenizers-wasm, installed with npm install tokenizers-wasm@next (at our check, both the latest and next tags point to 1.0.0-rc.3). Python gains pair encoding, and several loading bugs are fixed, including byte-level BPE vocabularies such as ModernBERT’s and tokenizer.json files with regex Split pre-tokenizers or Replace normalizers, which the notes say covers NLLB and every model transformers converted from SentencePiece. It is still a release candidate; HF says feature parity with 0.23 is the main milestone left before 1.0.0.

Update, 9 November 2026: rc.4 restores Python APIs, with one breaking change

Hugging Face tagged v1.0.0-rc.4 on 9 November, still a pre-release. It brings back the Python APIs that libraries such as transformers and vLLM need, including vocabulary lookups, added tokens, serialization and a DecodeStream compatible with vLLM. Install it with pip install --pre tokenizers==1.0.0rc4.

One change breaks code written against rc.3: a Python Tokenizer no longer encodes. It now holds an editable tokenizer.json config, and you call tokenizer.compile() to get a frozen CompiledTokenizer that encodes and decodes. Hugging Face says encoding then takes no lock and releases the GIL, and padding and truncation move into the config as encode defaults. HF still calls feature parity with 0.23 the main milestone before a stable 1.0.0.