PII detection · adversarial-robustness benchmark
AdvPIIBenchThe Adversarial Collapse of PII Detection in LLM Interactions
Rewrite a card number so a person still reads it: the pattern matchers go blind, but the language models still catch it. Add one plausible sentence around it and they fail too. Different layers, different victims — and no family survives both.
| Detector | Family | clean | PII sculpting | PII sculpting + content attack |
|---|---|---|---|---|
| Presidio | rule-based | ✓ | ✕ | – |
| GLiNER | transformer NER | ✓ | ✕ | – |
| OpenAI Privacy Filter | transformer NER | ✓ | ✕ | – |
| Llama Guard 8B | SLM safety guard | ✓ | ✓ | ✕ |
| GPT-4o-mini | LLM | ✓ | ✓ | ✕ |
Detection under each detector’s worst attack of that kind. The two layers defeat different families — and every family loses to one of them.
- ✓ recall above 80%
- ✕ below 25%
- – already defeated at the value layer
Framing cannot matter to a regex. Excludes detector-specific taxonomy attacks, which do not transfer.
Abstract
Personally Identifiable Information (PII) remains a persistent privacy and security concern for both individuals and organizations. The widespread adoption of Large Language Models and AI Agents introduces new threat vectors: sensitive data is not only pasted by users into chat interfaces, but also accessed by agents, increasing the risk of unintended leakage. Traditional PII detection systems, based on static rules and patterns, lack contextual understanding and remain inherently limited. More advanced pre-trained models, including Small Language Models, add contextual reasoning but have complementary failure modes to pattern detectors, leaving both families vulnerable in isolation.
We develop a taxonomy of inference-time attacks and release AdvPIIBench, a dataset of synthetic but realistic user prompts containing high-risk PII types such as IBANs, SSNs, and credit-card numbers, generated via an LLM-guided pipeline. We systematically evaluate rule-based, transformer-based, SLM-based, and LLM-based detectors on this dataset, showing that detection recall can collapse to zero under adversarial conditions.
The transformations
Red-Teaming · attacks the value
PII Sculpting
The value itself is rewritten. Every form below still reads as a card number to a person, and every one is produced by the benchmark’s own attack code.
Red-Teaming · attacks the cue
Contextual PII Sculpting
Implemented as supportive_context. The value is left completely untouched — only the PII indicator that announces it is transformed, so detectors leaning on that cue lose their anchor.
Template-Based · attacks the framing
Affix Injection and Prompt Injection
Neither the value nor its cue is touched. These are wrappers placed around the whole prompt, aimed at the detector rather than the user’s request.
Neural Prompt-to-Prompt · attacks by rewriting
An agent that rewrites until it gets through
The attacker is an LLM holding the prompt, a tactic list and a target detector. It rewrites, tests, and rewrites again — composing techniques across rounds instead of applying one. We release this as an adaptive red-teaming procedure; unlike the static techniques above, no attack-success rate is claimed for it.
Tactic list and rewrite taken verbatim from neural_prompt_to_prompt/prompts.yaml.
What the benchmark shows
Orthogonal collapse
Every detector family has an attack that drives its recall to near-zero — and the attacks that break one family leave another untouched.
No family is safe alone
Across 13 detectors, every one has at least one configuration that drives it to near-zero. No single system holds up alone; robustness needs complementary families combined.
Recall alone is not enough
The ranking inverts on hard, PII-lookalike negatives: the OpenAI Privacy Filter leads on clean recall at 97.4% and also leads on false positives at 58.5%. Recall and false-positive rate have to be read together.
One sentence undoes the best detector
GPT-4o-mini is the strongest detector tested — 99.8% recall on clean prompts, and still 96.0% once the card number is sculpted. Prepend a single line of plausible framing and the same model reports no PII at all.
Resources
Cite
@inproceedings{arpaly2026advpiibench,
title = {{AdvPIIBench}: The Adversarial Collapse of
{PII} Detection in {LLM} Interactions},
author = {Arpaly, Roei and Birman, Yoni},
booktitle = {Computer Security. ESORICS 2026 International
Workshops (SECAI)},
series = {Lecture Notes in Computer Science},
publisher = {Springer},
address = {Cham},
year = {2026},
url = {https://roeiarpaly.github.io/llms-pii-leakage/},
note = {To appear; accepted manuscript available at the project page}
}