Dropout Must Know Whether the Model Is Training

A speech classifier gives different scores for the same audio window on two consecutive requests. Its weights have not changed. Before blaming the input pipeline, check whether a dropout layer is still in training mode.

PyTorch’s Dropout module independently decides, on each training forward pass, which input elements to set to zero. If the dropout probability is p = 0.2, each element has a 20% chance of being removed. The survivors are multiplied by 1 / (1 - p), or 1.25 in this example. That scaling keeps each element’s expected value aligned with its unmasked value across many draws. It does not promise that one pass preserves an exact activation sum, or that two passes use the same mask.

The layer changes behavior when its parent module changes mode. In evaluation mode, ordinary dropout is the identity operation: it neither draws a mask nor rescales activations. PyTorch’s module documentation says eval() sets evaluation mode and is equivalent to train(False). This setting matters for modules with mode dependent behavior, including dropout and batch normalization.

A common serving mistake is to wrap a forward pass in torch.no_grad() and assume dropout has been disabled. no_grad() changes gradient recording; it does not change the module’s training flag. A model left in training mode can therefore produce newly masked activations while recording no gradients. Conversely, model.eval() does not by itself disable gradient recording. The two switches solve different problems.

A minimal inference path makes both decisions explicit:

model.eval()
with torch.no_grad():
    scores = model(features)

This example assumes model owns the dropout layer and features has already passed the application’s normal preprocessing. For a voice command system, compare scores only after matching the same input bytes, preprocessing and model checkpoint. If outputs still vary, examine other stochastic operations and the numerical behavior of the chosen hardware; evaluation mode only controls modules that consult that mode.

When returning to training, call model.train() before the next training forward pass. Treat mode as part of the model’s operational state: set it deliberately at the training and serving boundaries, and check it when repeatability matters.