Stop prompt injection with a classifier

A prompt-injection classifier is a small model trained on one question: is this text trying to override the assistant? It answers in a few milliseconds on a CPU, its scores are probabilities, and it has no instructions an attacker can argue with. The classify detector runs one, Prompt Guard, ProtectAI’s DeBERTa, any AutoModelForSequenceClassification export. It uses the same tokenizer, graph and windowing as the nlp detector, and lands its verdict over the segment for policy to act on.

Pin the checkpoint

Run this command:

$ pistra models pin protectai/deberta-v3-base-prompt-injection-v2
models:
  models:
    - name: protectai--deberta-v3-base-prompt-injection-v2
      hf:
        repo: protectai/deberta-v3-base-prompt-injection-v2
        revision: 7a2b…

The repository ships an ONNX export under onnx/. A checkpoint that does not is exported once with optimum-cli export onnx --task text-classification and served from your own store or an inference server, as for nlp. Gated repositories, mirrors and air-gapped hosts work the same way, see detecting names with a Hub model.

Reference it

Add this configuration:

guardrails:
  detectors:
    - type: classify
      name: injection
      classify:
        ref: protectai--deberta-v3-base-prompt-injection-v2
  rules:
    - name: no-injection
      when: types.exists(t, t == "injection/INJECTION")
      action: deny
      message: that looks like an attempt to override the assistant

The labels come from the checkpoint’s config.json, SAFE and INJECTION here, and so does whether they compete (problem_type). SAFE is the clean class, never a finding, and never in the vocabulary. A checkpoint that calls its clean class something else names it in negative. A checkpoint whose labels are LABEL_0 / LABEL_1 gets readable ones through label_map:

      classify:
        ref: meta-llama--Llama-Prompt-Guard-2-86M
        negative: [LABEL_0]
        label_map: {LABEL_1: INJECTION}

A finding is a verdict over the whole segment, pistra.span: segment, in the injection category unless category says otherwise (a moderation checkpoint’s labels belong in topic). deny and annotate are the actions written for a verdict.

What it reads, and how long text is scored

Left unset, apply_to is [user, tool_result]: the two kinds an outsider writes, and tool results are where indirect injection arrives. Name the kinds to change it.

A segment longer than the model’s window is scored window by window, overlapping by an eighth, and the best score any window gave a label is kept. That is the published guidance for these checkpoints, and the only reading under which cutting text into windows cannot hide anything.

The threshold is 0.5 unless threshold says otherwise. A classifier’s score is a probability, and one half is the boundary it was trained to. Raise it for a margin. Read the false-positive rate off the audit trail first, in mode: shadow.

Measure it

Run this command:

$ uv run eval/fetch.py deepset jailbreak safeguard
$ pistra guardrails eval -config eval/injection-protectai.yaml -sweep eval/data/jailbreak-test.jsonl
262 cases, 41.711s (159.202ms per case)
label              gold  found      P      R     F1
INJECTION           139    121  0.983  0.856  0.915

Precision / recall at the default threshold on the three public sets:

	deepset	jailbreak-classification	safe-guard
ProtectAI DeBERTa v2 (fp32)	1.00 / 0.37	0.98 / 0.86	1.00 / 0.85
Prompt Guard 2 86M (int8)	1.00 / 0.05	1.00 / 0.84	1.00 / 0.37

Neither produced a false positive, and a deny needs that. The recall tells you what each was trained to call an injection: Prompt Guard 2 is built for explicit jailbreak techniques and finds those. ProtectAI’s also catches the instruction-style attacks the other two sets are full of. A set’s definition matters too, deepset counts role-play prompts. A softmax classifier’s score is the winning class’s, so a sweep is flat below 0.5 and threshold is a margin above it. Measure a detector says how to mark your own traffic. That is the only set about your deployment.

Multi-label checkpoints

A moderation model emits several independent labels, toxic, threat, insult, each its own yes/no. That is sigmoid, read from problem_type: multi_label_classification in config.json or set with activation: sigmoid. Under it nothing is negative by default, and every label at or above the threshold is a finding.

    - type: classify
      name: moderation
      classify:
        ref: unitary--toxic-bert
        category: topic

Classifier or judge?

	`classify`	`llm` judge
the question	the one the checkpoint was trained on	anything you can write down
can be argued with	no, nothing follows instructions	yes, it reads what the attacker wrote
cost	milliseconds, CPU, no bill	a model call and its tokens
scores	calibrated probabilities	the model’s opinion
languages	the checkpoint’s (Prompt Guard 2: eight, no Arabic)	the judge model’s

Reach for the classifier for injection and moderation, where trained checkpoints exist. Reach for the judge for policy in prose and questions nobody trained a model on. For scope and for jailbreak phrasings you have already seen, embed matches your own examples with no training. Run more than one when you can afford to, because they fail differently.

Limits, stated plainly

A classifier catches the phrasings it was trained on. A novel attack, or one in a language it was not trained on, passes.
It false-positives on prose about security, “ignore previous instructions” quoted in a blog post being summarised.
Where the graph runs is a deployment choice, not a correctness one. classify.remote sends token ids to Triton, KServe or OpenVINO Model Server and reads scores back, as nlp.remote does.