Data and provenance
We used NVIDIA Nemotron-PII, revisionb70ffaf5ff39e079776134c5bf4381f00a9fd1ed.
Attribution: NVIDIA; Amy Steier, Andre Manoel, Alexa Haushalter, and Maarten
Van Segbroeck, Nemotron-PII (2025), CC BY 4.0.
We selected seven entity types, validated source offsets, and derived grouped
partitions. We are reporting aggregate results rather than redistributing the dataset.
The source contained 100,000 training and 100,000 test documents. Structural
exclusions removed 19,101 documents for source-slice mismatches and 20 for
duplicate or overlapping annotations. Seven training-side documents were
removed for test-family overlap. Families were grouped using source identifiers,
exact text, and template signatures; this does not rule out every near-duplicate.
The pretrained encoder was
distilbert/distilbert-base-cased, revision
6ea81172465e8b0ad3fddeed32b986cdcdcffcf0. Fine-tuning ran for two epochs
with FP32 MPS, batch size 16, learning rate 0.00002, and weight decay 0.01.
The selected checkpoint had the lowest validation token loss.
The runtime uses manually constructed windows of up to 512 tokens with
42-token overlap and global UTF-8 byte offsets. This is finite boundary-test
evidence, not a guarantee for every possible entity length. Sixteen annotated
boundaries in the training source could not be represented by token boundaries.
Scoring and freeze
An exact typed-span match requires the same entity category, start offset, and end offset as the gold annotation. Overall scores are micro-averaged over the seven target categories. Target-clean documents remain in evaluation. Inference and evaluation inputs were hashed before final test scoring. The final audit verified those inputs were unchanged and independently recounted the 90,018 documents and the totals below. Two implementation repairs—tokenizer overflow handling and export evaluation mode—occurred before final test scoring. Neither used final test outcomes for tuning.
Totals: 242,853 true positives, 2,424 false positives, 5,886 false negatives.
At least one annotated entity was missed in 3,947 documents; at least one
spurious entity was predicted in 1,730 documents. Among 16,233 target-clean
documents, 81 had an accepted prediction.
UTF-8 byte-union redaction precision was 99.66%, recall 97.89%, and F1 98.76%.
This measure ignores entity categories and weights multibyte characters by
byte length: 63,160 gold bytes were missed and 10,061 unnecessary bytes redacted.
Regex comparison
The baseline was the published legacy DataFog Python 4.8.1 regex engine, not current DataFog Core. Only email and phone number were compared because those are the baseline’s supported categories within this experiment’s scope. Both systems saw all test documents, with the same exact-span scoring.
The combined row covers 71,279 gold entities. It must not be compared directly
with the model’s seven-category overall F1. No hybrid system was evaluated.
Confidence and operating point
Pooled isotonic calibration was fitted on 1,220 predicted spans from 501 documents. Only 48 predictions were incorrect. The acceptance threshold,0.64864865, maximized exact typed-span F1 on the separate calibration-select
partition, with ties resolved by recall and then higher threshold.
These measures cover all 250,771 predicted spans before thresholding. They do
not account for undetected entities. ECE depends on binning, sparse confidence
regions remain unreliable, and calibration may change on another population.
The selected threshold is experimental, not a recommended production default.
Native runtime observations
The bundle uses Rust with ONNX Runtime 1.30.0 on Apple arm64. It contains 300,076,892 bytes, approximately 300 MB. All 16 bundle checksums passed the final audit; the packaged executable processed Unicode and clean-input smoke cases with no Python on PATH.
These are individual end-to-end observations on the development Mac, not
latency percentiles or service guarantees. Cold process launch plus the first
empty-input inference took 170 ms. Maximum RSS sampled after responses was
496 MB; it is not an allocator high-water mark.
Python and Rust matched token IDs, masks, global offsets, findings, and
redaction on the parity fixtures. Maximum observed absolute logit difference
was zero on those cases, not a universal numerical guarantee.