Beam Search Keeps Several Answers Alive

A strong opening can lose

A text generator chooses its next token from the options available after the prompt. Greedy search takes the most likely token at each step and continues from that choice. Once it commits to an opening, it follows that path. A promising continuation behind a weaker opening is out of reach. The Hugging Face generation guide describes this rule and the alternative called beam search.

Several paths stay open

Beam search keeps a limited set of partial sequences, called beams. At each step, it extends the surviving sequences and keeps the strongest candidates. It then selects a completed sequence by its overall probability. A less likely first token can survive long enough for its later tokens to make the whole sequence stronger. The first token alone cannot decide the result. The guide’s beam search section explains this behavior.

The Hugging Face decoding walkthrough gives a small word-level example. Greedy search follows “The nice woman.” Beam search also retains an opening through “dog,” which later leads to “The dog has.” In that teaching example, the second sequence has the higher overall probability.

The search has limits

The beam is a shortlist. At every step, lower-scoring partial sequences are pruned. A branch that drops out cannot return when a later word would have helped it. The walkthrough says beam search is not guaranteed to find the most likely possible output. It also shows repetition in an open-ended generation example. The generation guide points to input-grounded tasks, such as describing an image or recognizing speech, as good uses.

What to do

For a short answer where creativity is less important, start with greedy search. For an input-grounded task where several plausible openings matter, try beam search with num_beams greater than one, as the guide specifies. Read the completed outputs and check for repetition before deciding whether the change helped. Keep the beam’s limit in mind: it protects a few alternatives while pruning the rest.