A safety classifier can check both the prompt and the reply

A filter that checks only user input misses the case where a harmless-looking prompt produces a harmful reply. A filter that checks only output spends a model call on requests you would have refused anyway. Llama Guard is built to run at both points with one model.

What Llama Guard is

The paper Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (Inan et al.) describes a safeguard model built on Llama2-7b and instruction-tuned on a small, high-quality dataset. It uses a safety risk taxonomy to classify both user prompts and model responses. The authors report that it matches or exceeds existing content moderation tools on the OpenAI Moderation Evaluation dataset and ToxicChat, and they released the weights.

The full paper says the two tasks are separated “simply by change of wording in the instruction tasks for the same model”. The instruction states whether to classify the user messages, which the paper calls prompts, or the agent messages, which it calls responses.

How to read the output

The model answers in text. Per the paper, the output starts with safe or unsafe. If unsafe, “the output should contain a new line, listing the taxonomy categories that are violated.” The original model writes each category as a letter followed by its 1-based index in the policy, such as O3. A flagged result looks like this:

unsafe
O3

The original model card names six categories: Violence & Hate, Sexual Content, Guns & Illegal Weapons, Regulated or Controlled Substances, Suicide & Self Harm, and Criminal Planning.

Later releases changed the taxonomy. The Llama Guard 3 8B model card lists fourteen hazard codes, from S1 “Violent Crimes” to S14 “Code Interpreter Abuse”, including S7 “Privacy” and S13 “Elections”. It lists English, French, German, Hindi, Italian, Portuguese, Spanish and Thai as supported languages. That card says the model lists the violated categories when a result is unsafe. It does not spell out the line layout, so write a parser that accepts either code style, and map codes to names from the card of the exact model you deploy.

Where it sits in the request path

Run two checks around the main model call:

1. Classify the user turn. If unsafe, refuse and skip the main model.
2. Call the main model.
3. Classify the conversation including the reply. If unsafe, withhold the reply.

The Hugging Face model cards, including the one for Llama Guard 3 8B, show a moderate(chat) helper that applies the tokenizer’s chat template, generates up to 100 new tokens and decodes only the new text. A request handler around it can look like this, where refusal and generate_reply are your own functions:

import re

def parse_verdict(text):
    lines = text.strip().splitlines()
    head = lines[0].strip() if lines else ""
    if head not in ("safe", "unsafe"):
        raise ValueError(f"unexpected guard output: {text!r}")
    if head == "safe":
        return "safe", []
    return "unsafe", re.findall(r"[A-Z]\d+", " ".join(lines[1:]))

def handle(user_msg, generate_reply):
    chat = [{"role": "user", "content": user_msg}]
    verdict, codes = parse_verdict(moderate(chat))
    if verdict == "unsafe":
        return refusal(codes)
    reply = generate_reply(chat)
    chat.append({"role": "assistant", "content": reply})
    verdict, codes = parse_verdict(moderate(chat))
    if verdict == "unsafe":
        return refusal(codes)
    return reply

Three choices in that sketch are deliberate. An output the parser does not recognise raises an error, so the caller can fail closed instead of treating noise as safe. The category codes travel to refusal and into your logs, which tells you which policy fired. And the second check sends the whole conversation, as the model card’s own example does with a user message and an assistant reply. The paper says the instruction wording decides which role is assessed, so confirm in your chat template that the reply is the turn under assessment.

Using a score instead of the label

Both model cards say they take the probability of the first generated token as the “unsafe” class probability. The paper notes that this score can be read off directly and that the area under the precision-recall curve helps select a threshold that balances precision and recall. Neither source gives a recommended threshold for production. If you use the score, pick the threshold on labelled traffic from your own product.

Where this stops

The paper lists its own limits. Its common sense knowledge is limited by its training data. Its fine-tuning data and most of its pretraining data are in English. The authors do not claim perfect coverage of their policy. The model may be susceptible to prompt injection, and the Llama Guard 3 card repeats that risk. That card adds that some hazard categories may need up-to-date factual knowledge to judge.

The original model card warns that comparisons with other tools are not exactly apples-to-apples, because each tool uses a different taxonomy, and that the taxonomy does not necessarily reflect Meta’s own internal policies. Check that the categories match your own policy. The paper says instruction tuning lets you customize the taxonomy through zero-shot or few-shot prompting. The sources do not cover streaming. A full-reply check implies you hold the reply until the check passes. Treat the classifier as one layer next to your other controls.