- what is pinned
- two injection checkpoints that fail in opposite directions
- what picks one
profiles[].guardrails.detectors, and the rule's ownrequires- the trade
- 80.2% to 1.2% of Arabic customers, at 0.914 to 0.815 English F1
- what is not claimed
- six of the ten languages, and Arabic recall past eight turns
Eleven languages have been measured through the gateway's own inspection path so far. Ten of them carry the same 315 red-team prompts, so the verdicts can be compared row for row: English, Chinese, Italian, Vietnamese, Korean, Arabic, Thai, Javanese, Swahili and Bengali. German was measured separately, on the German half of a public injection set. Arabic is the one with a gold set of its own, because it is the language the team can author and review.
Eleven is not your deployment. The command that produced every number below takes a file of your own labelled turns, so the answer for a language nobody here speaks is a measurement you run, not a claim you take. How a detector is measured.
Two checkpoints, failing in opposite directions
pistra pins both. protectai/deberta-v3-base-prompt-injection-v2 is the better English guard and one of the most widely deployed open ones. proventra/mdeberta-v3-base-prompt-injection is the same architecture family trained on the same English sources, with a multilingual vocabulary in place of an English one. The four rows below are what decides between them.
Picked correctly, your Arabic traffic is flagged at the rate your English traffic is: 1 turn in 81, against 65 in 81 on the English-vocabulary guard.
| ProtectAI v2 | mDeBERTa | |
|---|---|---|
| what it costs to serve Arabic | 65 of 81 ordinary turns flagged | 1 of 81 |
| what it costs to serve English | 1 of 81 ordinary turns flagged | 1 of 81 |
| Arabic jailbreak turns caught | 8 of 8 | 6 of 8 |
| English quality, safe-guard F1 | 0.914 | 0.815 |
Neither column is the one to ship everywhere. Serving Arabic on the left column refuses four in five ordinary customers, which on a deny rule is an outage. Serving English on the right gives up a tenth of the F1 on the one English set neither model trained on, and two of the eight Arabic jailbreak turns with it. The choice is a property of the traffic a team carries.
Which one inspects which traffic
Both are declared as detectors in one ensemble, and profiles[].guardrails.detectors narrows which of them inspect a profile's traffic. A profile selects rather than defines: there is one set of detector instances behind the deployment, so two profiles that both select the multilingual guard are inspected by the same one and share its delta-scan cache. The Arabic-serving team gets one checkpoint, the English-only team keeps the other, on one deployment, and neither pays for the other's trade.
A rule then says which findings it acts on. Every annotation carries the name of the detector that produced it, so a CEL condition can deny on one checkpoint's verdict and annotate on the other's while both run. A rule also names the detectors its decision depends on, in requires, which is what stops a profile's selection from silently leaving a deny rule that can never match. Give a team its own policy walks it, and the policy reference is the annotation a condition reads.
What the choice rests on
A guard's blind spot is a property of its vocabulary, and the vocabulary is a property of the checkpoint you pin. So it is measured per checkpoint rather than inferred. The two tables below are why a deployment holds both.
That measurement is the reason the trade above can be stated in numbers at all. Without it the choice is a preference between two model cards, both of them reporting English.
Why no single checkpoint wins
MultiJail carries the same 315 red-team prompts in ten languages. The paper sorts them into three tiers by each language's share of the CommonCrawl corpus: above one percent is high-resource, below a tenth of a percent is low. None of the 315 is a prompt injection, so every fire below is a false positive. The fires are protectai/deberta-v3-base-prompt-injection-v2 through the gateway's own inspection path, measured August 2026 on build 82ccc3176ba3.
| language | MultiJail tier | fires of 315 | rate |
|---|---|---|---|
| English en | high | 0 | 0.0% |
| Chinese zh | high | 19 | 6.0% |
| Italian it | high | 38 | 12.1% |
| Vietnamese vi | high | 138 | 43.8% |
| Korean ko | medium | 63 | 20.0% |
| Arabic ar | medium | 239 | 75.9% |
| Thai th | medium | 294 | 93.3% |
| Javanese jv | low | 129 | 41.0% |
| Swahili sw | low | 141 | 44.8% |
| Bengali bn | low | 310 | 98.4% |
The fires do not follow the tiers. Vietnamese is high-resource and fires on 138. Arabic is medium-resource and fires on 239. Swahili is low-resource and fires on 141. Italian at 38 says the effect reaches Latin scripts. A language's share of a training corpus does not predict what this checkpoint does to it.
Nor does the region, and German is the case that settles it. On the 27 ordinary German prompts in deepset/prompt-injections, both pinned checkpoints fire on none, exactly as they fire on none of the 29 English ones beside them. German and Italian are both European, high-resource and Latin-script, and one of them is fine while the other flags one prompt in eight. The set publishes no language column, so eval/fetch.py tags each row with lingua over the two languages it contains. The tag is ours and the command reports the rate per tag.
So the answer for a language you serve is a measurement, not an expectation. Point pistra guardrails eval at your own labelled turns and read the rate.
The vocabulary does. Arabic costs ProtectAI's tokenizer 3.18 pieces a word against English's 1.52, because 347 of its 128,001 entries contain an Arabic character:
| base | vocabulary | Arabic entries | pieces per Arabic word | per English word |
|---|---|---|---|---|
xlm-roberta-base | 250,002 | 14,639 | 1.76 | 1.41 |
mdeberta-v3-base | 250,102 | 7,408 | 2.03 | 1.60 |
deberta-v3-base-prompt-injection-v2 | 128,001 | 347 | 3.18 | 1.52 |
A sequence of unfamiliar fragments is what the training data called an attack, which is how ordinary Arabic becomes one. Swapping to the multilingual vocabulary takes the Arabic false-positive rate on ordinary customer turns from 80.2% to 1.2%, English's rate to the case, and that is the left column of the table at the top of this page turning into the right one. The report carries the full tables, the digests and the build.
Read it with recall in the other hand
A parallel set exposes this with no gold in any of the ten languages. The same meaning in every column of a row has the same right verdict in each, so a verdict that changes between them is a defect on its own, and the rows a language lost from the reference are the bypass an operator would see. The harness reads that off with -invariance, and choosing a detector is what else it answers.
It cannot tell a quiet detector from a consistent one. Meta's Prompt Guard 2 86M scores perfectly on the table, zero fires across all 3,150 cases, because it is quiet on nearly everything: one of eight Arabic jailbreak turns, five of eight English ones. Invariance says whether a verdict depends on the script. Recall says whether the verdict is ever INJECTION, and that half needs gold in the language.
Arabic first
Recall in a language needs gold a native speaker reviewed, and Arabic is the language this team can author and review. That is why the evaluation set pistra is building is Arabic, and why the case study on this page is Arabic while the table above is ten languages. The set keeps injection, benign and harmful apart, pairs every row with its English original, and records how each row came to exist. Gulf identifiers and Arabic names follow the same rule: measured on the traffic a deployment carries, with the clitic problem published rather than hidden. Find Gulf identifiers, Arabic names and the clitic.
Threat model
The asset is the deny rule's false-positive rate, and behind it the deployment's ability to serve customers who do not write English. Nobody attacks it. An ordinary customer asking about an invoice in Arabic is enough to trip it, and on a deny rule that is an outage for four in five of them. The attacker's version of the same defect is the language a guard stays quiet in, which is a bypass, and that half needs gold in the language to measure.
Two mechanisms produce the gap, and they are different. The MultiJail paper's thesis is about the model: alignment data is English, so a harmful request translated into a language with less of it is more likely to be answered. That is a resource-level effect. The guard's defect is about vocabulary: a tokenizer with 347 Arabic entries turns ordinary Arabic into unfamiliar fragments, and a sequence of unfamiliar fragments is what its training data called an attack.
The entry this section expands, and the ten beside it: threats and limits.
What this does not do
- No claim about a language it has not run. Ten languages are measured for false positives. German, French, Spanish and Polish are not among them, and the one European language in the set is enough to say the effect reaches Latin scripts and not enough to say by how much.
- Arabic recall is unmeasured beyond eight turns. Everything above is about false positives, the half that can be measured without a labelled set in the language.
- A multilingual vocabulary fixed Arabic and did not fix language. The multilingual checkpoint still fires 36 times on Javanese and 27 on Bengali against 7 to 14 elsewhere.
- The model we want does not exist. ProtectAI's English quality with a multilingual vocabulary's indifference to script. Neither shipped checkpoint is that, and the page says so rather than picking one and calling it solved.
The argument
- Safety is measured in English. Harmful requests in low-resource languages get unsafe answers more often because alignment data is English, and the guards built to stop them share the same defect from the same root.
- Read invariance with recall in the other hand. Some metrics reach their best value as a detector fires less, so a detector that answers nothing scores perfectly on them. Invariance is one. So is a false-positive rate, an agreement score, and a benchmark holding no positive cases. Each is read beside the number that breaks the silence.
- An evaluation set in a language with none. No public Arabic prompt-injection set exists, so recall on Arabic attacks is unmeasured. pistra is building one with parallel pairs so the verdict can be compared across languages, hard negatives that outnumber attacks, and an origin tag on every row.