Topic · 17 stories
17 stories tagged pytorch on AI Tamer, newest first.
Barron, Gnanapragasam and Tschannen show max-shifted softmax zeroes the target gradient past a precision cliff, and give log-space code that fixes it.
OPAL wraps a PyTorch optimizer so Claude Code can change learning rates, clipping and update rules mid-run. Single-run gains, and it can make runs worse.
TorchTPU, built with Meta's PyTorch team, is open source. Google adds TPU backends for vLLM, SGLang, Ray and VeRL, and device_map="tpu" in Transformers.
AWS's TorchNeuron makes Trainium2 and Trainium3 a native PyTorch device, so .to('neuron') and torch.compile work. Apache-2.0 public beta opens 20 October.
A 15 October PyTorch post adds TorchTitan SFT, LoRA, and TitanRL for Muse Spark Open, plus Helion megakernels. Speed figures are the blog's.
PyTorch mmap loading delays access to checkpoint storage. Budget for resident pages, parameter copies, activations and device memory during model startup and serving.
Identical weights and inputs need not yield bitwise identical floating-point outputs across batch shapes or backends. Diagnose the size, location and consequence of differences.
Dropout changes a model’s forward pass according to module mode. Learn how masking, scaling, evaluation mode and gradient recording affect a reliable inference path.
AdamW shrinks selected parameters separately from Adam’s adaptive gradient step. That distinction changes the effect of regularization and makes parameter groups a deliberate training choice.
Checkpointing keeps fewer forward activations and reruns a segment during backpropagation. The memory saving comes with compute, RNG, and implementation tradeoffs.
Loss masking removes a position from cross entropy. It does not remove its token from a transformer input or prevent other positions from attending to it.
Label smoothing mixes a one-hot target with a uniform distribution. The loss then rewards a finite probability spread and changes how confidence should be interpreted.
A shared shift of logits preserves softmax probabilities and avoids a common overflow path. The same shift gives a stable log-sum-exp calculation.
A model checkpoint's file format affects what happens when you load it. Learn why pickle deserves caution and how to choose safer weights.
Several small training batches can contribute to one optimizer update. The loss must represent the whole update, especially when batches contain different numbers of tokens.
Both PyTorch modes skip gradient recording. Inference mode also changes the tensors it creates, which matters when later code needs autograd.
Matching seeds do not guarantee matching results. Reproducing a run also means recording its software, hardware, and algorithm settings.
