Arabic NER corpora put the clitic inside the name

A names detector is the part of a redactor that decides which bytes get rewritten. We measured two multilingual NER models to pin one for Arabic traffic. Both scored around 0.23 F1 on the slice of turns where the name carries a prefixed و, ل or ب, against 0.75 to 0.80 on the rest. The cause is not the models. Both are reproducing the boundary their training corpus taught them, and that boundary is wrong for redaction.

The prediction is the name plus one letter

gulf-names.jsonl marks the name alone. A clitic is a preposition or a conjunction glued to the front of the following word, and it carries no identity, so the set leaves it outside the span. Both models put it inside.

اجتمعت بسلمان البلوشي ونورة الرشيدي في مكتب الشركة
  marked   PERSON("سلمان البلوشي")   PERSON("نورة الرشيدي")
  returned PERSON("بسلمان البلوشي")  PERSON("ونورة الرشيدي")

ارسل نسخة لتركي ونسخة لمحمد الدوسري
  marked   PERSON("تركي")   PERSON("محمد الدوسري")
  returned PERSON("لتركي")  PERSON("لمحمد الدوسري")

The byte offsets are right for what the models predicted. The subword-to-span decoder produced one contiguous span of the correct type at 0.88 to 1.00 confidence. Nothing is misaligned. The models are asserting that the name starts one letter earlier than we marked it.

Overlap-allowed precision on the clitic slice is exactly 1.000 for both models. Every span they returned contains the name it was looking for. None of them loses a byte of it.

Both models make the same error at the same rate

The 35 clitic turns carry 70 marked names.

Half the names, in both models, differ from the mark by one letter. Two models with different architectures, different tokenizers and different vocabularies do not agree to within 1.4 points by accident. They agree because they learned the same convention.

ANERcorp is where they learned it

Both models take their Arabic from ANERcorp. It is annotated a token per line with one tag per token, and Arabic clitics attach inside the token, so a clitic-prefixed name is a single token whose tag covers the clitic.

وأسامة   B-PERS
وسيد     B-PERS
بصدام    B-PERS
لجعفر    B-PERS

Counting strictly, a B-PERS token whose first letter is a proclitic and whose remainder the corpus also tags as a person somewhere else: 41 of 858 B-PERS tokens in the test split, 4.8%. Every one of them puts the clitic inside the entity. There is no annotation in the corpus that separates them, so there is nothing for a model to learn from.

The convention is correct for the corpus. A token is the unit, the tag applies to the token, and splitting one would need a segmentation layer the format has no column for.

It is the wrong boundary for a redactor

A redactor acts on the span. Given ارسل نسخة لتركي, a span that includes the ل rewrites the preposition along with the name:

marked span   ارسل نسخة ل[NAME]
returned span ارسل نسخة [NAME]

The sentence loses the word that said what to do with the copy. With format-preserving replacement the synthetic name absorbs the ل, so what reaches the provider is grammatical Arabic that means something else. A CEL rule matching on the entity text sees وسلمان rather than سلمان, and a rule written against one spelling misses the other.

Nothing leaks. Overlap precision of 1.000 means the name is always fully covered, so this costs correctness downstream rather than confidentiality. It is over-redaction, and over-redaction of the one letter that carried the sentence’s grammar.

The rest of the numbers

PERSON only, exact F1 and overlap-allowed F1, every model on the same build.

XLM-R wins on three of five, is level on aqmar, and is faster on all five: 3.68 ms a case against 4.40 on gulf-names, 7.87 against 9.68 on anercorp-test. A better model that also costs less is unusual and worth stating plainly.

The exception is wikiann-ar-test, where mBERT scores 0.556 against XLM-R’s 0.440. WikiANN’s Arabic labels are silver-standard, generated rather than annotated, and we keep the row as a sanity check rather than a result. A disagreement on a set neither model should be graded against is not evidence about either one.

anercorp-test is the ceiling and not generalisation. It is the test split of the corpus both models trained their Arabic on, which is why it carries the highest numbers on the page.

What the tag rows say

gulf-names.jsonl stacks one hazard per tag, so the totals repeat per slice.

Two rows besides the clitic are worth reading. variant spells a name with the other hamza, taa marbuta or alef maqsura, and mBERT drops to 0.244 there while XLM-R holds 0.698. The vocabularies are the same distance apart as they were in the guard-model report: mBERT cased carries 4,873 entries containing an Arabic character out of 119,547, and XLM-R base carries 14,639 out of 250,002. A respelling that stays one piece in the larger vocabulary fragments in the smaller one, which is the same mechanism that report found and a second place it shows up. title and latin reaching 1.000 on XLM-R says the harder-looking hazards are not the ones that fail.

The negative slice is 20 turns containing no person: common words used as words, places, organisations, and bare numbers. Both models fired once. One false name on 20 turns built to draw one is the number we wanted and did not expect.

What we recommend

• Pin XLM-R for Arabic traffic. Better on three of five sets and level on a fourth, better on nine of eleven hazard tags, and faster on every set. The one set it loses is the one with silver labels.
• Trim the clitic in the span layer, not the model. The models will keep returning the corpus’s boundary because that is the only boundary annotated anywhere. A trim belongs after the decoder, and there is no clitic handling in the detector today.
• Do not trim blindly. The proclitic is one letter from a closed set, but بندر and بشرى are names that start with one, and ال is inside every family name in the set. A trim that fires on the letter alone breaks more names than it fixes. The remainder has to still be a name, which needs a second pass rather than a regular expression, and that pass needs measuring before it goes in.

What is still missing

• The set is ours. gulf-names.jsonl is generated by a script we wrote, so it proves what our own convention says and not what a Gulf deployment contains. No public corpus carries a kunya, a بن chain, a name inside a clitic or a Saudi identity number beside one, and none of them is a chat turn. The convention finding does not depend on it: ANERcorp is public and shows the same thing.
• Three clitics, not five. The set generates و, ل and ب. Arabic also proclitics ف and ك, and the definite article ال behaves differently again. Those are unmeasured.
• The other errors are not the convention. XLM-R swallowed two names into one span across a و in 12.9% of cases and missed 8.6% outright. Trimming the clitic fixes neither. They are the ordinary recall problem and they need their own report.
• No dialect beyond Levantine. Wojood’s public sample carries Levantine. The Gulf dialects the set is named for have no annotated corpus we could score against, which is the reason the set exists and also the reason it cannot be validated.

How to disagree with any of this

Every number above came from pistra guardrails eval, the same binary and the same code path that enforces, over evaluation sets pinned by sha256. Nothing was scored in a notebook. That matters because a classifier's logits are not what a deployment acts on: the tokenizer binding, the windowing past max_tokens, the subword-to-span decoder and the score threshold all run first, and a number computed outside them describes a model nobody is running.

$ pistra guardrails eval -sweep -json

All reports