Meta’s Llama Prompt Guard 2 86M scored perfectly on our invariance table. Zero fires across 3,150 cases, ten languages, not one row whose verdict changed with the language.
It also caught one of eight Arabic jailbreak turns.
Metrics that a detector can win by doing nothing
Invariance asks whether a verdict depends on the language. Give a detector the same meaning in ten columns of a row, and whatever the right answer is, it is the same answer ten times, so a verdict that changes between them is a defect. The appeal is that this needs no labelled data in any of the ten languages. It is a measurement you can take where you cannot afford ground truth, which is most languages, for most teams, most of the time.
The arithmetic has a floor it cannot see. A detector that never fires gives every row the same verdict in every language. It is perfectly invariant. It is also useless, and the metric has no way to say so.
That shape is not unique to invariance. A false-positive rate falls toward zero as a detector fires less. An agreement score against a second detector rises as both of them go quiet. A pass rate on a benchmark carrying no positive cases is highest for a model that never flags anything. In each, the best possible value is reached by a system that has stopped answering the question, and the metric reports that as success.
The tell is structural rather than empirical. Ask what a detector that outputs nothing would score. When the answer is “the maximum”, the metric measures consistency or restraint, and neither is the same as being right.
The pairing
Every metric of that kind has a companion that breaks its silence, and neither half is readable alone.
Invariance pairs with recall on a labelled set. A false-positive rate pairs with recall on the same traffic. An agreement score pairs with a set somebody marked, since two detectors agreeing tells you nothing about whether either is correct. Without its companion the first number is unreadable, and publishing it alone is how a measurement becomes a claim.
Our own report is the worked example, and it goes both ways. ProtectAI v2 flags 1.2% of English customer turns and 80.2% of Arabic ones, and 75.9% of the Arabic MultiJail rows against none of the English. Invariance catches that with no Arabic labels needed, and the labelled set says what it costs: on a deny rule, four in five Arabic customers refused. Prompt Guard 2 is the mirror. Invariance says it is flawless and recall says it answers almost nothing, with an F1 of 0.54 on safe-guard, one of eight Arabic jailbreak turns and five of eight English ones. Two models, opposite failures, and the invariance column alone would rank the worse one first.
Why the harness prints the fire count
pistra guardrails eval -invariance puts a fired count on every row of the table, beside the rate.
It is there because the rate on its own is unreadable. A row saying a language agrees with the reference on every case means one thing when 239 of 315 fired, as Arabic did under ProtectAI v2, and another when none did, and the second is not a result. The count does not fix the blind spot, which no arrangement of that table can. It puts the evidence of the blind spot on the same line as the score, so a reader cannot take the rate without seeing what produced it.
The rows the verdict split on are listed under the table for the same reason, by row number rather than text: a defect you can go and look at, rather than a summary statistic you have to trust.
What it costs us to say this
The rule points inward. We recommend pinning the multilingual model for traffic that includes Arabic, and that recommendation rests on Arabic recall measured over eight turns, because no public Arabic prompt-injection set exists to measure it on. Eight turns is not a recall number. It is the number we have.
So the guard we pin is chosen on a false-positive measurement that is strong, taken across ten languages against 315 parallel rows, and a recall measurement that is thin. Naming which half is which is the whole of the discipline here. A report that gave one number for the model and let a reader assume both halves were equally supported would be doing the thing this post is about.
The set that fixes it is the Arabic one we are building, and it has a post of its own in What reaches the model.
As of 7 September 2026
Every figure here is from Guard models are reported in English, measured 31 August 2026 on build pistra devel 82ccc3176ba3. Llama Prompt Guard 2 86M is gravitee-io/Llama-Prompt-Guard-2-86M-onnx@45a05fbd, int8; ProtectAI v2 is protectai/deberta-v3-base-prompt-injection-v2 at revision 90c9989b, fp32. The parallel set is 315 red-team prompts in ten languages from DAMO-NLP-SG/MultiJail (MIT), none of which is a prompt injection, so every fire is a false positive. The customer turns are ours: 89 per language, 81 ordinary across billing, tech support, account and medical, and 8 tagged as jailbreaks.
The invariance table and its columns are documented in Measure a detector. Language is what the finding means for a deployment, and choosing a detector is the method behind it.