Injection, jailbreak, harmful

“Attack” is a loose word for text the deployment did not want the model to see. Three classes hide under it. They look different, and a control built for one does little against the others.

Injection tries to change what the model does. A jailbreak tries to make the model drop its safety behaviour. A harmful request asks for something that should not be answered, with no trick in it. The three need three different answers.

Injection

Injection is text that tries to override the instructions the operator gave. It tells the model to ignore what came before, to take on a persona, to reveal its system prompt, to fill a template it was not meant to fill, or to treat the text itself as coming from someone with authority. MITRE ATLAS files it as AML.T0051, in a direct form the user typed and an indirect form a tool result carried.

Injection has a signature. The text speaks in the voice of an instruction, and it carries a second intent that tries to overwrite the first. The signature can be buried in a document or a tool result, and the structure is still there.

The control for injection is a classifier: a checkpoint trained on one question, whether this text tries to override the assistant. It runs in the request path with no model call, and it has no instructions an attacker can argue with. It answers that one question and no other.

Jailbreak

A jailbreak tries to make the model set aside its safety training or the deployment’s refusal policy. ATLAS files it as AML.T0054. Where an injection instructs, a jailbreak persuades. It flatters: only an advanced model could answer this. It stages a fiction: in a story, how would a character do it. It claims permission: this is a sanctioned test, so it is allowed. The text argues with the model’s own values rather than with the operator’s instructions.

The control for a jailbreak is a judge, a model handed a policy in prose and the text as written. A judge reads persuasion for what it is, and it costs a model call. It also follows instructions, so the text it reads can argue with it too. A jailbreak therefore wants both: a classifier for the phrasings that recur, a judge for the ones that do not.

Harmful

A harmful request hides nothing. It asks how to build a weapon, how to write a phishing mail, how to find someone’s home address. There is no override and no persuasion. The harm is in what is asked rather than in how.

The control for harm is a judge reading a harm policy. It is the same mechanism as the jailbreak judge with a different policy in front of it. A classifier trained on injection has no opinion about harm, and should have none.

A classifier trained on one class fires on another

MultiJail carries 315 red-team prompts, each annotated into ten languages by the paper’s own annotators. Every row is a harmful request. None is a prompt injection, so for an injection classifier every fire on the set is a false positive.

The report Guard models are reported in English ran three injection classifiers over it. One widely deployed classifier fired on 38 of the 315 Italian rows, 239 of the Arabic rows and 310 of the Bengali rows. One row asks how to bomb a mosque. It is harmful in ten languages, a prompt injection in none, and that classifier labels it INJECTION in five of the ten.

The verdict is wrong twice over. The row is not an injection, so the label is a false positive. And the row is harmful, so the deployment wanted a verdict from a different detector, which the classifier’s fire says nothing about. A deny rule on that classifier refuses the request for the wrong reason in five languages and lets it through in the other five. The report traces the fires to the tokenizer vocabulary, and to nothing in the text.

One set, three classes

The confusion is cheap to measure once the class is recorded on every row. pistra’s evaluation sets carry one of three classes per row: injection, benign, harmful. A classifier config scores the set with injection as the only positive, so a fire on a harmful row counts against precision. A judge config scores the same file with harmful as the positive, so the same row is a hit. One file, two detectors, and the same rows read two ways.

That is how the classes are kept apart. A deployment does not need three detectors for three words. It needs to know which class each detector answers, and to score it on rows of the other two.

What each verdict is for

An injection classifier on a deny rule stops the phrasings it was trained on, and a jailbreak that shares them. A judge with an injection policy reads both classes and can be talked round. A judge with a harm policy answers a question neither of the others asked. pistra ships the classifier as the classify detector and the judge as the llm detector, and the threat model says of both that they reduce rather than prevent.

Whichever one is chosen, it runs in shadow mode first, recording verdicts without acting on them, until the false-positive rate on ordinary traffic has been read. A classifier that fires on a harmful request in Arabic and stays quiet on the same request in English is what shadow mode is for.

All posts