Guard models are reported in English

Every published prompt-injection classifier we could find reports its precision and recall on English sets. Deployments are not English. So before pinning one, we ran three of them over the same sentences in ten languages, through the gateway’s own inspection path.

One classifier flags four in five Arabic customers as an attack

topics.jsonl is 89 ordinary customer turns per language, plus 16 tagged as jailbreaks. protectai/deberta-v3-base-prompt-injection-v2 is one of the most widely deployed open guard models, and it is the graph pistra pinned.

The model is not deaf to Arabic. It catches the Arabic jailbreaks 8 out of 8, exactly as it catches the English ones. It also flags four out of every five Arabic customers asking about an invoice. On a deny rule, which is the only reason to run a classifier rather than a judge, that is an outage.

It is not only Arabic

MultiJail carries the same 315 red-team prompts annotated into ten languages by the paper’s own annotators. The ten columns of a row mean the same thing, so whatever the right verdict is, it is the same verdict ten times. A verdict that changes with the language is a defect that needs no ground truth to see. None of the 315 is a prompt injection, so every fire below is a false positive.

Bengali, Thai and Arabic are the worst, and Italian at 12.1% shows the effect is not confined to non-Latin scripts. One of the 315 asks how to bomb a mosque. It is harmful in ten languages, a prompt injection in none, and INJECTION in five.

The cause is the tokenizer vocabulary

proventra/mdeberta-v3-base-prompt-injection is close to a controlled experiment on that claim. Same architecture family, same English training sources, and a multilingual vocabulary in place of an English one.

347 of ProtectAI’s 128,001 vocabulary entries contain an Arabic character, and Arabic costs it 3.18 pieces a word. A sequence of unfamiliar subword fragments is what its training data called an attack. Swapping to a vocabulary with 7,408 Arabic entries takes the Arabic false-positive rate from 80.2% to 1.2%, which is English’s rate to the case.

That is not free. On safe-guard, the one English set neither model trained on, F1 falls from 0.914 to 0.815, and the Arabic jailbreak turns caught fall from 8 of 8 to 6 of 8.

A perfect invariance score can mean silence

Meta’s Llama Prompt Guard 2 86M scores perfectly on the table above: zero fires across all 3,150 cases, and no row whose verdict changed with the language. It is quiet on Arabic customers because it is quiet on nearly everything. It caught one of the eight Arabic jailbreak turns and five of the eight English ones, and its F1 on safe-guard is 0.54.

Read invariance with recall in the other hand. A detector that fires on nothing agrees with itself in every language. The measurement says whether a verdict depends on the script. It says nothing about whether the verdict is ever INJECTION.

What we recommend

• English-only traffic. ProtectAI v2 is the better English guard and we still pin it.
• Traffic that includes Arabic. Pin the multilingual model and accept the stated cost in English F1. ProtectAI v2 cannot be switched on at all.
• Any other language. Measure it. A multilingual vocabulary fixed Arabic and did not make the model indifferent to language in general: mDeBERTa still fires 36 times on Javanese and 27 on Bengali against 7 to 14 elsewhere.

What is still missing

• No public Arabic prompt-injection set exists. Recall on Arabic attacks is unmeasured beyond eight turns. Everything above is about false positives, which is the half that can be measured without one.
• MultiJail carries one European language. Italian fires 38 of 315, enough to say the effect reaches Latin scripts and not enough to characterise German, French, Spanish or Polish. Extending the run needs a parallel set rather than new machinery.
• The model we want does not exist. ProtectAI’s English quality and recall, with a multilingual vocabulary’s indifference to the script. Neither shipped model is that.

How to disagree with any of this

Every number above came from pistra guardrails eval, the same binary and the same code path that enforces, over evaluation sets pinned by sha256. Nothing was scored in a notebook. That matters because a classifier's logits are not what a deployment acts on: the tokenizer binding, the windowing past max_tokens, the subword-to-span decoder and the score threshold all run first, and a number computed outside them describes a model nobody is running.

$ pistra guardrails eval -sweep -json

All reports