DataFog Core is distributed as
datafog-core. It is separate from the
established datafog Python package and does not reproduce its legacy API.
Existing users should follow Migrate from DataFog
Python.What it does
Detect PII
Scan text for structured findings with explicit ranges and detector provenance.
Transform text
Redact, mask, remove, pseudonymize, or tokenize selected findings.
Control selection
Select entity types, apply per-entity overrides, and exempt approved values.
Restore tokens
Restore provider-issued tokens atomically under an exact authorization scope.
Supported entities
Text scans recognize these entities without a locale:EMAILPHONESSNCREDIT_CARDIP_ADDRESSDATEZIP_CODEBEARER_TOKEN(0.4.1; explicit Authorization headers)JWT(recognition rules)API_KEY(0.4.1; supported providers)PRIVATE_KEY(complete PEM blocks — format and boundaries)
PERSON in supported name fields. See
structured person discovery for field mappings.
Credit cards and decimal numbers: contiguous card candidates immediately
after a digit followed by
. or , are suppressed to reduce decimal false
positives. Formatted cards containing spaces or hyphens still undergo normal
card checks. Plain text such as 3,4111111111111111 is ambiguous: it could be
a comma-decimal number or two CSV fields, so the card candidate is suppressed.
Parse CSV input and scan each field separately to preserve field boundaries.Cards followed by more digits (0.4.5-beta.3+): a card number followed by
a CVV or expiry in the same run, such as
4111 1111 1111 1111 123, is
reported without the trailing digits when the full run fails the existing
card checks. A full-run card keeps its original span. The leading card part
must end at a space or hyphen, be written contiguously or as 4-4-4-1,
4-4-4-4, 4-4-4-4 plus one to
three digits, 4-6-5 or 4-6-4, and pass the usual issuer and Luhn checks. A
full-run card that starts at a later group of the same run is preferred.
Trailing digits excluded by the card fallback are not reported as part of
that card; other enabled detectors still scan them independently.German coverage (0.4.0+): an explicit German locale adds
DE_IBAN,
DE_VAT_ID, DE_TAX_ID, DE_SOCIAL_SECURITY_NUMBER, DE_POSTAL_CODE,
DE_PASSPORT_NUMBER, and DE_RESIDENCE_PERMIT_NUMBER. See the
German entity reference for examples, required
contexts, and format limitations. This coverage is available in Core 0.4.0 or newer.detect_uuid: true
option in Core 0.4.0 or newer. See UUID identifiers for the supported
syntax and why this detector is opt-in.
Built-in entity names are uppercase. The finding contract remains extensible so
custom detectors can introduce additional entity names in the future.
Operation model
transform never scans implicitly. Use scan_and_transform when you want the
explicit scan-then-transform convenience operation.
Start here
Install a package
Choose the distribution for your runtime.
Run the quickstart
Detect and redact an email address in a few lines.
US_ROUTING_NUMBER with explicit text context.
National Provider Identifiers detection adds NPI with explicit text context.
DataFog Core 0.4.1 is published across all four runtimes and includes Rust/Python capability discovery and a compatibility policy. The higher-level Python adapter is integrated in merged PR #179; its package release is separate.
Core 0.4.1 adds default API_KEY, BEARER_TOKEN, and CREDENTIAL_URI detection. The runtime registry reports 23 supported entities and 14 defaults. The local Python 4.9 adapter integration gate passed against the exact candidate wheel; the release notes record its evidence and limits.