OpenAI Privacy Filter vs Layrin on RedactionBench: A 200-Document Privacy Benchmark

RedactionBench was introduced in June 2026 as a benchmark for contextual PII redaction. Instead of assuming that every recognizable entity should always be removed, it separates annotated text into three broad classes.

Mandatory entities are information units the benchmark considers unsafe to share across all contexts. Contextual entities may or may not require protection depending on the circumstances. Everything else forms the unlabelled gaps between those annotations.

This distinction matters.

A privacy system that simply removes every person, organization, date, URL, identifier or technical resource may protect a great deal of information, but it may also destroy useful context. A system that preserves more contextual information may produce more useful text, but it also has less margin for missed sensitive information.

RedactionBench captures that trade-off through R-Score.

At a high level, R-Score rewards coverage of mandatory entities while adding denominator penalties for incomplete contextual redactions and false-positive redactions in benchmark-defined gaps. Mandatory entities receive character-level partial credit, and grouping rules prevent composite entities from being disproportionately weighted simply because they contain multiple annotated spans.

That makes R-Score a measure of selective redaction quality rather than simply a measure of how much mandatory information a system managed to protect.

For this study, we wanted to isolate another question:

When RedactionBench itself says an entity must always be protected, how much of that entity was actually protected?

We call this diagnostic Mandatory Entity Coverage.

It uses RedactionBench's mandatory annotations and entity grouping rules, but deliberately removes contextual-residual and false-positive penalties from the calculation.

For each mandatory entity, coverage ranges from 0 to 1 according to the proportion of its mandatory content protected by the system.

From that entity-level coverage we report several measures:

Micro mandatory coverage pools coverage across all 8,273 mandatory entities. Exact mandatory recall counts an entity only when its required content is completely covered. A completely missed entity has zero mandatory coverage. P20 and P50 describe the lower-performing tail and median of document-level mandatory coverage.

Mandatory Entity Coverage is therefore not an alternative implementation of R-Score, and it should not be presented as a RedactionBench leaderboard score.

It deliberately asks a narrower question.

R-Score measures protection and selectivity. Mandatory Entity Coverage measures failure to protect information that the benchmark has already classified as always sensitive.

On full R-Score, OpenAI Privacy Filter wins decisively in our reproduction:

0.6003 vs 0.3705.

On micro mandatory coverage, the direction reverses:

81.32% for Layrin vs 61.85% for OpenAI Privacy Filter.

The exact-recall difference is similarly large:

79.55% vs 60.98%.

And the raw miss count makes the distinction especially concrete.

Across 8,273 mandatory entities, OpenAI Privacy Filter completely missed 3,132. Layrin completely missed 1,447.

Layrin therefore fully protected 1,536 more mandatory entities, while OpenAI Privacy Filter left 1,685 more mandatory entities completely uncovered.

Yet OpenAI Privacy Filter still achieved the much higher combined R-Score.

The explanation is not hidden in mandatory coverage. It lies primarily in what happens outside mandatory entities.

Before drawing conclusions from that divergence, we needed confidence that our implementation of R-Score was reasonably faithful to the paper.

RedactionBench specifies the metric, entity-combination behavior, prediction-dependent contextual selection and false-positive rules in considerable detail. At the time of this evaluation, however, the dataset documentation still stated that the exact reference implementation would be released separately.

We therefore implemented a paper-faithful scorer and built conformance tests around the published rules and worked examples.

The scorer passed 29/29 conformance tests, including tests for entity grouping, partial mandatory coverage, contextual selection, combinators and benchmark-defined false-positive penalties.

We then applied it to frozen OpenAI Privacy Filter predictions.

The result was:

• reproduced mean R-Score: 0.6003
• published RedactionBench mean: approximately 0.58
• reproduced P20: 0.3350
• published P20: approximately 0.31
• reproduced P50: 0.6146
• published P50: approximately 0.59

RedactionBench reports OpenAI Privacy Filter at about 0.58 overall, with the corresponding P20 and P50 values shown in its published results.

The category profile was similarly close.

This close reproduction gives us confidence that the implementation is capturing the scoring behavior described by RedactionBench. We then applied the exact same scoring implementation unchanged to Layrin.

Under full R-Score, OpenAI Privacy Filter scores substantially higher. When mandatory protection is isolated, Layrin protects substantially more.

Both observations are true at the same time.

OpenAI reports excellent results for Privacy Filter on PII-Masking-300k: 96% F1, including 94.04% precision and 98.04% recall. On a corrected version addressing annotation issues identified during evaluation, OpenAI reports 97.43% F1, 96.79% precision and 98.08% recall.

Those results should not be directly compared numerically with our RedactionBench results.

They measure different things.

Traditional PII detection benchmarks generally evaluate whether a model recognizes entities belonging to a predefined taxonomy. F1 combines precision and recall against those target labels.

RedactionBench was created specifically because the authors argue that entity recognition and privacy semantics are not equivalent. Its documents are substantially more realistic than short synthetic entity examples, and the privacy decision may depend on the role an entity plays in the document.

This distinction is important when moving from a detection model to a production privacy workflow.

A high PII-detection F1 tells us a great deal about a model's ability to recognize its target taxonomy.

It does not by itself tell us:

• how many always-sensitive entities will be completely missed across full documents;
• how performance varies across document types;
• how aggressively a system protects information outside the benchmark's target policy;
• or how a broader protection policy affects the privacy-versus-utility trade-off.

That is why document-level evaluation adds useful information rather than contradicting model-level F1.

OpenAI describes Privacy Filter as a token-classification model for PII detection and masking intended for high-throughput data-sanitization workflows. It can operate locally and supports a defined privacy taxonomy. OpenAI also explicitly cautions that Privacy Filter is a redaction and data-minimization aid, not an anonymization, compliance or safety guarantee, and notes that appropriate privacy policies may differ between organizations.

Layrin has a different product-level objective.

Layrin is a local privacy layer for protecting sensitive text before AI use. Its workflow is built around local protection, human review and reversible typed tokenization.

Instead of simply deleting a value, a protected passage can retain its semantic role:

Sarah Chen from Northbridge Capital approved the transfer on September 18, 2026 for $4.2 million.

can become:

[PERSON_1] from [COMPANY_1] approved the transfer on [DATE_1] for [AMOUNT_1].

The exact values are withheld from the AI while relationships and grammatical structure remain available to the model.

After the AI produces its answer, the protected values can be restored locally.

That changes the utility calculation.

With irreversible redaction, protecting more information directly removes more usable content. With typed reversible tokenization, a system can withhold the exact value while preserving information such as:

• this entity is a person;
• this entity is a company;
• this value is a date;
• these references point consistently to the same protected entity.

This leads to a practical data-minimization question:

If an AI can perform the task without receiving the exact value, how much utility is really lost by withholding it?

That question is relevant well beyond traditional PII detection. It arises in legal documents, HR records, finance, research material and technical infrastructure.

The difference in mandatory protection was not driven by one unusual document class.

Layrin achieved higher micro mandatory coverage in all 11 RedactionBench categories.

The largest difference appears in Logs, where mandatory coverage was 92.04% for Layrin vs 45.78% for OpenAI Privacy Filter.

There is one useful nuance in Files.

OpenAI Privacy Filter had slightly higher mean document mandatory coverage in that category. But when mandatory entities are pooled, Layrin still had higher micro mandatory coverage: 70.68% vs 64.44%, as well as fewer completely missed mandatory entities: 778 vs 978.

This is one reason we report both document-level and entity-level metrics. Averages can answer different questions depending on how mandatory entities are distributed across documents.

The R-Score result makes clear that higher mandatory coverage did not come for free.

Across the benchmark, 23,476 Layrin-protected spans fell entirely inside RedactionBench-defined unlabeled gaps, with no overlap with mandatory or contextual annotations.

Some of these are genuine over-redaction.

But manual inspection also found repeated examples where Layrin protected information that the benchmark's annotation policy left visible, including:

AWS Secrets Manager references identifying production database credentials, production RDS hostnames, internal package-registry URLs, S3 paths pointing to production user exports, private IP addresses, and application .env paths.

These are reasonable examples of the difference between benchmark policy and deployment policy.

RedactionBench is not making a mistake by leaving them unlabelled. Its privacy framework has a defined objective.

But an organization preparing internal material for transmission to an external AI system may reasonably choose a broader boundary.

The protected gap spans also extended beyond infrastructure:

• 4,834 technical identifiers
• 2,698 date/time spans
• 2,508 hostnames or domains
• 2,490 organization/company spans
• 2,466 IP addresses
• 757 URLs
• 256 path/resource spans
• 7,467 other spans

This is why the phrase false positive needs context.

A benchmark-defined false positive is a false positive relative to the benchmark's annotation and privacy policy.

It is not automatically a real-world privacy or security false positive.

The concentration of gap protections was not evenly distributed.

Logs alone accounts for 11,986 of 23,476 gap-protection spans — 51.06% of the total.

At the same time, Logs is the category with the largest mandatory-coverage difference:

92.04% for Layrin vs 45.78% for OpenAI Privacy Filter.

That pairing makes the privacy-utility trade-off unusually visible.

Logs routinely contain dense technical context: infrastructure references, identifiers, hostnames, IP addresses, paths, environment information and operational metadata. A conservative protection policy can therefore redact substantially more than RedactionBench requires.

R-Score penalizes that loss of selectivity.

But in the same category, Layrin also protects far more of what RedactionBench itself classifies as mandatory.

Neither result should be hidden.

Together they explain the product-policy difference much more clearly than either score alone.

The 23,476 figure above is an analysis of Layrin prediction spans: each counted span falls entirely in an unlabelled benchmark gap and overlaps neither mandatory nor contextual annotations.

It should not be interpreted as “23,476 R-Score penalty points.”

RedactionBench constructs false-positive regions according to its own contiguous-region and full-gap rules. In our scorer, the Layrin predictions produced 23,768 R-Score false-positive regions. Of those, 153 covered a complete eligible gap and therefore received the paper's doubled denominator weight, producing a total false-positive denominator contribution of 23,921.

We report the 23,476 prediction-span figure separately because it is useful for inspecting what Layrin chose to protect, whereas the R-Score regions describe how the benchmark scores those predictions.

The distinction matters for reproducibility.

RedactionBench's own user study reinforces the idea that privacy is not purely an entity-recognition problem.

The study included 85 participants. Participants redacted mandatory units at a rate of 89.4% and preserved benchmark-defined gaps 94.1% of the time.

Contextual entities were very different: participants agreed with contextual redaction decisions only 47.7% of the time.

That is not noise around the edges of the benchmark. It is one of the reasons R-Score exists.

People broadly agree that certain information should clearly be protected and that clearly benign text should remain. The disagreement lies in the middle.

Consider something as ordinary as a date.

In one document, a date may be essential public context.

In another, it may reveal an appointment, employment action, confidential transaction, filing deadline or internal event.

The characters themselves do not tell us the full privacy story.

RedactionBench explicitly acknowledges a further limitation: documents are currently evaluated in isolation, without the user's query, conversation history or system prompt. The authors note that information benign in one context can become sensitive in another and identify query-aware redaction as future work.

That matters when evaluating systems designed for actual AI use.

The intended recipient, user request and surrounding workflow can change what an organization is comfortable disclosing.

Traditional redaction creates a direct trade-off:

protect more characters, lose more information.

Reversible tokenization changes that relationship.

Suppose an HR document contains:

Sarah Chen will join Northbridge Capital on September 18, 2026 with a $180,000 base salary.

A conventional redaction might produce:

██████████ will join ███████████████████ on ██████████████████ with a ███████ base salary.

A typed-token representation can instead preserve:

[PERSON_1] will join [COMPANY_1] on [DATE_1] with a [AMOUNT_1] base salary.

The AI no longer receives the exact identity, company, date or amount.

But it still understands that:

one person joins one organization, on a specific protected date, with a specific protected compensation amount.

For many drafting, summarization, translation, review and analytical tasks, those relationships carry much of the useful semantic information.

This is why data minimization for LLM prompts should not be reduced to simply detecting PII.

The operational question is not only:

What can we recognize as sensitive?

It is also:

What exact information does the AI genuinely need in order to perform the task?

A privacy workflow can then protect the difference.

R-Score appropriately penalizes unnecessary redaction because utility matters.

A privacy tool that protects everything indiscriminately would be safe in a narrow sense but unusable.

At the same time, the consequences of the two error directions are not necessarily symmetric.

Over-redaction costs utility. Leakage can cost confidentiality.

A hidden date that the AI could have used may reduce answer quality.

A missed credential, account identifier, medical detail, private address or confidential party name may expose information that cannot simply be recalled after transmission.

Different organizations will rationally place the decision boundary in different locations.

OpenAI itself acknowledges this policy issue in the Privacy Filter model card, noting that its default label policy may not satisfy every organization's governance requirements and that some users may choose different decision boundaries.

The objective, therefore, should not be to claim one universal privacy boundary.

It should be to make the trade-off measurable.

The most important result from this experiment is not that one system has a larger number in one column.

It is that R-Score and mandatory protection answer materially different questions.

R-Score asks: how selectively can a system protect sensitive information while preserving useful content?

Mandatory Entity Coverage asks: when the benchmark itself says information must always be protected, how much of it actually escaped protection?

Both matter.

A system that protects mandatory content while redacting enormous amounts of benign text is not ideal.

A system that carefully preserves benign text but completely misses large numbers of always-sensitive entities is not ideal either.

The surprising part of this experiment is how far the two objectives separated.

The system with the substantially higher combined R-Score protected substantially less of RedactionBench's mandatory information, while the system with substantially higher mandatory coverage accumulated enough over-redaction penalty to finish far behind on the combined score.

That does not invalidate R-Score.

It suggests that privacy evaluation may benefit from reporting the two dimensions separately.

A single combined metric is useful for ranking systems according to a chosen privacy-utility objective.

But practitioners making deployment decisions may also want to know explicitly:

• How much mandatory information escaped?
• How much additional information was protected?
• Where did those two behaviors occur?

For privacy systems intended to sit between confidential data and external AI, those questions can have very different operational consequences.

The broader question is therefore:

Should privacy benchmarks report protection failure and over-redaction separately, rather than allowing one to compensate for the other inside a single score?

Dataset

The complete RedactionBench test set was used:

• 200 documents
• 11 categories
• 8,273 mandatory entities after applying the benchmark's entity/combinator grouping rules used for this analysis
• 101 real and 99 synthetic source documents
• English-language benchmark data

RedactionBench itself contains 53,286 total span annotations across mandatory, contextual and structural/grouping annotations.

Layrin

Layrin was evaluated with Layrin Desktop 0.1.4.0 using its frozen production local-protection and reversible-tokenization pipeline.

The production configuration remained fixed throughout the evaluation.

Ground-truth RedactionBench annotations were not available to the inference pipeline.

Some structured inputs used deterministic inference segmentation. Segmentation altered inference boundaries only:

• source text was not normalized, shortened or rewritten;
• predictions were mapped back to exact original-document character offsets;
• each source file remained one RedactionBench document for scoring.

This segmented-recovery procedure produced predictions for all 200/200 documents.

No benchmark result was used to modify the frozen production configuration during the evaluation.

OpenAI Privacy Filter

OpenAI Privacy Filter was executed locally using OpenAI's public implementation, frozen for the experiment.

Its predictions were converted to character spans before scoring against the same original RedactionBench documents.

OpenAI describes the public model as a bidirectional token-classification model for PII detection and masking and provides local usage through standard model runtimes.

Mandatory Entity Coverage

Mandatory entities were constructed using RedactionBench's mandatory spans and applicable grouping/combinator rules.

For each entity:

coverage = mean character coverage of its constituent mandatory spans

Micro mandatory coverage is the arithmetic mean of entity coverage across all mandatory entities.

Exact mandatory recall requires entity coverage of 1.

A completely missed entity has coverage of 0.

Contextual spans and gap false positives do not influence Mandatory Entity Coverage.

R-Score

Our paper-faithful implementation follows the published RedactionBench formulation for:

mandatory entity coverage, contextual residuals, entity grouping, combinators, prediction-dependent contextual selection, false-positive regions and doubled penalties for eligible full-gap coverage.

The scorer passed 29/29 internal conformance tests derived from the paper's definitions and examples.

Because the RedactionBench reference implementation was not yet publicly available at the time of evaluation, the reproduction against the published OpenAI Privacy Filter result served as an external validation check. The reproduced overall distribution closely matched the paper's reported mean, P20, P50 and category profile.

Interpretation

This study compares two systems against the same benchmark annotations, but the systems do not represent identical product architectures or privacy policies.

The results should therefore be interpreted as an analysis of protection behavior under a common benchmark, not as evidence that the systems solve identical deployment problems.

RedactionBench itself advises against using benchmark performance alone as sufficient justification for deployment in sensitive legal, medical or financial settings.

1. Brynjólfsson, S., Jayakrishnan, S., Sali, E., Purwar, D., & Aggarwal, M. (2026). RedactionBench. arXiv:2606.18782. The paper introduces the 200-document benchmark, contextual-integrity labeling framework, R-Score and 85-participant human study. https://arxiv.org/abs/2606.18782 (opens in a new tab)
2. OpenAI. (2026). Introducing OpenAI Privacy Filter. OpenAI's April 22, 2026 release announcement describes local PII detection and masking and reports the PII-Masking-300k evaluation results. https://openai.com/index/introducing-openai-privacy-filter/ (opens in a new tab)
3. OpenAI. OpenAI Privacy Filter model card. Public model documentation covering intended use, token-classification behavior, local execution, risk of over-reliance and policy limitations. https://huggingface.co/openai/privacy-filter (opens in a new tab)
4. OpenAI. OpenAI Privacy Filter repository. Public implementation used for local execution in this evaluation. https://github.com/openai/privacy-filter (opens in a new tab)
5. A10 Networks. RedactionBench dataset. Public 200-document dataset and leaderboard, including the current note concerning release of the reference R-Score implementation. https://huggingface.co/datasets/RedactionBench/RedactionBench/ (opens in a new tab)