Skip to content
pistra

Add your own recognizer

The pii detector ships Presidio’s recognizers: cards, national ids, bank numbers, network identifiers. It does not ship your employee-id format, your customer-number scheme or the codenames of the projects your staff must not mention to a model. Nobody but you knows them. Add a custom recognizer to declare them.

A custom recognizer is Presidio’s PatternRecognizer: patterns with scores, a deny list of whole words, and context words. Once compiled it is an ordinary recognizer, scored, boosted by context, deduplicated and held back at the streaming horizon like a built-in. An entry from a presidio-analyzer recognizers file pastes across.

Write it

Section titled “Write it”

Write it on the pii detector, beside the recognizers: selection:

guardrails:
  detectors:
    - type: pii
      custom_recognizers:
        - name: EmployeeIDRecognizer
          entity: EMPLOYEE_ID
          patterns:
            - name: employee id
              regex: '\bEMP-\d{6}\b'
              score: 0.6
          context: [employee, badge, staff]
        - name: CodenameRecognizer
          entity: CODENAME
          deny_list: [Bluebird, Kestrel, Osprey]
  rules:
    - name: no codenames leave
      action: deny
      when: 'annotations.exists(a, a.type == "pii/CODENAME")'
      message: internal project names cannot be sent to a model
    - name: hide employee ids
      action: redact
      operator: placeholder
      select: 'a.entity_type == "EMPLOYEE_ID"'

Five things to know about the regex:

  • It is RE2 syntax, not PCRE and not Python re: no lookahead or lookbehind, no backreferences, no atomic groups, possessive quantifiers or conditionals. Presidio’s own patterns are Python re and went through a translation pass when they were extracted. Yours does not, so anything outside RE2 is refused at load with the position and the reason. Say the same thing without it. Most lookarounds around identifiers are \b in disguise, and a backreference is usually a repeated group written out.
  • RE2 is unicode-aware for the text. \pL walks héllo and (?i)é matches É. Its Perl escapes are ASCII by specification: \w is [0-9A-Za-z_], \s is ASCII space, and \b is a boundary between those and anything else. Python’s are unicode, so a Presidio pattern that leans on \w to cross an accented letter will not do so here. Write [\pL\pN_] where you mean it.
  • Digits are the exception, and you write nothing for it. The engine folds every script’s decimal digits onto ASCII before any pattern runs, so \d, [0-9] and a literal 7 all match ٧, ७ and ７. The span reported is on the text as sent.
  • A leading or trailing \b is the one place the engine corrects for that. It is held to unicode word boundaries at the match edges, the way the built-ins are, so \bEMP-\d{6}\b does not match inside éEMP-123456 even though RE2’s own \b would.
  • It is case-insensitive unless the recognizer says case_sensitive: true. That is Presidio’s default for every pattern recognizer, and keeping it makes a pasted entry behave the same here.

A pattern’s score sets what a match is worth before context words raise it. A deny-list match scores deny_list_score, 1.0 unless set. A recognizer needs at least one of the two.

Pick it the way the built-ins do. A shape that is unmistakable on its own, a literal prefix, a long fixed format, scores high. A shape that is really “a run of digits of about the right length” scores low on purpose, near 0.05, and earns its keep from context: below. A context word raises any match to at least 0.4, so a weak pattern with good context words is a better recognizer than a strong pattern with none. PII coverage lists what every built-in scores, and the arithmetic that moves it.

Nothing is dropped for scoring low. score_threshold is 0 by default, so a weak pattern does reach policy. The rule that wants only corroborated findings asks for the corroboration rather than a threshold:

annotations.exists(a, a.type == "pii/CODENAME" &&
                      (a.validated || a.context_supported))

Give it a checksum

Section titled “Give it a checksum”

A pattern for a national identifier is, on its own, a pattern for any ten-digit number. The identifiers worth a recognizer nearly all carry a check digit, and the checksums they use are few, so you name one rather than write it:

        - name: sa_national_id
          entity: SA_NATIONAL_ID
          patterns:
            - regex: '\b[12]\d{9}\b'
              score: 0.3
          validator:
            kind: luhn
          context: [iqama, هوية, id]

kind is luhn, verhoeff, iso7064 (MOD 97-10, with letters read A=10 to Z=35), iban (the same check as ISO 13616 applies it, with the country code and check digits moved behind the account number first, for an IBAN country the built-in recognizer does not know yet) or weighted. The semantics are the built-in recognizers’: a value the checksum accepts scores 1.0, a value it rejects is dropped, and the verdict rides on the annotation. strip lists separators to remove first, strip: "- " for a value written with dashes and spaces. weighted states the scheme directly, with the check digit’s own weight in the list: the UK NHS number is

          validator:
            kind: weighted
            weights: [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
            modulus: 11

Digits in any script are digits. ١٠٥٥٥٦٦٦٧١ is found by the pattern above and checked by the same Luhn, and the fpe operator writes its replacement back in the same script.

Hold it to examples

Section titled “Hold it to examples”

A recognizer can carry the evidence that it works:

          examples:
            - text: "iqama 1055566671 expires in March"
              finds: ["1055566671"]
            - text: "رقم الهوية ١٠٥٥٥٦٦٦٧١"
              finds: ["١٠٥٥٥٦٦٦٧١"]
            - text: "not 1055566678, the checksum is wrong"

Each text is scanned with this recognizer alone and must yield exactly finds, in order. An example with no finds must yield nothing. That is how a near miss is pinned as one. A recognizer whose examples fail is refused at load, with the example and what was found instead. With a validator declared, every found value is also altered one digit at a time and must then not be found. The checksum is proven to decide, not just declared.

The examples run wherever the config is loaded: at startup, on a config write through the admin API, and under pistra check -config …, which compiles the config the way the gateway would and exits.

Select it

Section titled “Select it”

recognizers: and entities: on the detector select custom recognizers the way they select built-in classes, by name, or by the entity they emit. A custom recognizer that the selection leaves out is refused rather than left inert:

detectors[0] (pii): custom recognizer "EmployeeIDRecognizer" is not in
recognizers:, so it would never run. Add it there, or drop it.

With no selection at all, every built-in and every custom recognizer runs.

Address the type

Section titled “Address the type”

The entity you named is now a type in the vocabulary, pii/EMPLOYEE_ID to a rule and EMPLOYEE_ID everywhere else. Two places check that vocabulary at load, so a misspelling is refused where you are reading it rather than found by the first request that carries the value:

  • fpe_alphabets may only name a type some configured detector emits. A custom type has no built-in alphabet, so a rule that format-preserves it needs one declared: fpe_alphabets: {EMPLOYEE_ID: alnum_upper}.
  • An MCP server’s restore: list may only name a type some detector emits and the fpe operator has an alphabet for.

The check runs only when every detector could say what it emits. A remote analyzer with no entities: filter leaves the vocabulary open, and an open vocabulary refuses nothing. A name the gateway cannot see is still one the analyzer may produce.

To see the vocabulary this node compiled, custom types included, with which detector produces each:

Terminal window
curl -s localhost:8484/admin/v1/guardrails/entity-types | jq

The console offers the same list wherever a document names an entity type, and the configuration schema carries the built-in vocabulary as examples an editor completes from. Neither is a constraint. The set is not the document’s to close.

Beside a remote analyzer

Section titled “Beside a remote analyzer”

Custom recognizers are not forwarded to a remote analyzer as Presidio’s ad_hoc_recognizers, on purpose. The server would compile the same regex with Python re while this side compiles it with RE2. One string would mean two things on two machines, with nothing to say which you got. The server would also recompile them on every call. Ad-hoc recognizers are built for that and a registry is not. A Presidio server takes its own recognizers file at start, and that is where its custom recognizers belong.

To apply your recognizers here, beside a remote analyzer, give them a pii detector of their own:

guardrails:
  detectors:
    - type: remote
      remote: {endpoint: http://presidio-analyzer:3000}
      entities: [PERSON, LOCATION, EMAIL_ADDRESS]
    - type: pii
      recognizers: [EmployeeIDRecognizer]
      custom_recognizers:
        - name: EmployeeIDRecognizer
          entity: EMPLOYEE_ID
          patterns: [{regex: '\bEMP-\d{6}\b', score: 0.6}]

A pii detector running only your patterns costs about what those patterns cost and nothing else. The vocabulary stays exact, because the remote says what it emits through entities: and the pii detector says so through its recognizers.

Start from a built-in

Section titled “Start from a built-in”

The recognizers pistra adds beside Presidio’s, the Gulf identifiers, are written in this shape, in guardrails/pii/native/*.yaml, with their provenance in comments and their evidence in examples:. One is a worked example of every key on this page. Copy it onto a pii detector under a name of your own to try a variation. pistra check holds the copy to its examples as it holds yours.

Test it

Section titled “Test it”

examples: prove the recognizer finds what it should. The rule suite proves policy does something about it. A case that hands the rules an annotation of the new type is the cheapest proof that policy addresses it the way you meant. See test your rules before they ship.

What is not here

Section titled “What is not here”

There is no prefilter. No generated region finder knows your pattern, so it scans every segment whole, about a millisecond per pattern per 64 KB of text. Two or three are lost in the noise of a scan. Ten cut clean-prose throughput four times. A country’s worth of identifiers is built into the tables for that reason, rather than shipped as a pack of these. If you find yourself writing that many, that is the place for them.

Section titled “Related”