Topic · 6 stories
6 stories tagged tokenization on AI Tamer, newest first.
Ai2's method for turning a subword model into one that reads raw bytes is a Nature article dated 7 October 2026. Hugging Face cards list Olmo, Qwen, and Llama checkpoints, and the licences are not all the same.
Padding to the longest input lets each batch set its length. Padding to a fixed length uses a chosen target, while truncation handles inputs that exceed a limit.
A chat template turns role-labeled messages into the token sequence a model expects. The markers and the start of a reply depend on the model.
Perplexity averages loss per token. When tokenizers split the same text differently, their raw scores need careful interpretation.
Padding lets prompts of different lengths share a batch. An attention mask marks which positions the model should attend to.
HF tagged tokenizers v1.0.0-rc.2 (2026-09-21): same API and token IDs as v0.23, performance refactor. Install with cargo add --pre. Encode 3–30× and related figures are HF/tokbench (M4 Max), not desk-measured.
