Migrate from DataFog Python

This guide covers migration of the synchronous text detection and transformation API from datafog 4.8.x to datafog-core 0.2.x. DataFog Core is a new canonical API, not a drop-in replacement for the established DataFog Python package.

DataFog Core makes detection results, text ranges, transformation policy, and provider-backed operations consistent across Rust, Python, Node.js, and browser/WASM. That consistency requires a few deliberate API changes in Python. DataFog Core 0.3 does not replace the legacy package’s optional spaCy, GLiNER, OCR, distributed-processing, CLI, or application-guardrail features. Keep the established package for those workloads while adopting Core where its smaller, cross-runtime contract fits.

Change the package and import

python -m pip install datafog

python -m pip install datafog-core

from datafog.engine import Entity, redact, scan, scan_and_redact

from datafog_core import Finding, scan, scan_and_transform, transform

The distribution name uses a hyphen (datafog-core), while the Python import uses an underscore (datafog_core). The two distributions can be installed at the same time while an application is migrated incrementally.

Update scan results

DataFog Python returns a ScanResult wrapper. DataFog Core returns a list of Finding objects directly.

from datafog.engine import scan

result = scan(
    "Email jane@example.com",
    engine="regex",
    entity_types=["EMAIL"],
)

for entity in result.entities:
    print(entity.type, entity.text, entity.start, entity.end)

from datafog_core import scan

findings = scan("Email jane@example.com")

for finding in findings:
    print(
        finding.entity_type,
        finding.matched_text,
        finding.codepoint_range.start,
        finding.codepoint_range.end,
    )

Use codepoint_range when slicing a Python string. Use byte_range when addressing the UTF-8 encoded input. Both ranges are zero-based and end-exclusive.

DataFog Python 4.8.x	DataFog Core 0.3.x
`ScanResult.entities`	The return value from `scan`
`Entity.type`	`Finding.entity_type`
`Entity.text`	`Finding.matched_text`
`Entity.start`, `Entity.end`	`Finding.codepoint_range.start`, `.end` for Python string indexing
No distinct UTF-8 range	`Finding.byte_range`
`Entity.engine`	`Finding.detector_name` and `.detector_version`
`ScanResult.engine_used`	No aggregate equivalent; provenance is attached to each finding

Rule-based Core findings can have confidence=None. Do not assume every finding has a numeric confidence score.

Replace scan-and-redact calls

DataFog Core separates detection configuration from transformation policy. transform requires explicit findings; scan_and_transform is the convenience operation that performs both steps.

from datafog.engine import scan_and_redact

result = scan_and_redact(
    "Email jane@example.com",
    engine="regex",
    entity_types=["EMAIL"],
    strategy="mask",
)

print(result.redacted_text)

from datafog_core import scan_and_transform

result = scan_and_transform(
    "Email jane@example.com",
    {
        "transform": {
            "default": {"strategy": "mask"},
            "entities": ["EMAIL"],
        }
    },
)

print(result.text)

DataFog Python 4.8.x	DataFog Core 0.3.x
`scan_and_redact(...)`	`scan_and_transform(text, {"scan": ..., "transform": ...})`
`redact(text, entities, ...)`	`transform(text, findings, config)`
`RedactResult.redacted_text`	`TransformResult.text`
`RedactResult.entities`	`TransformResult.transformations`
`RedactResult.mapping`	No plaintext mapping is returned

Transformation records describe what was applied, including source and output ranges, but intentionally omit the original matched PII.

Choose the intended transformation

Do not migrate strategy names mechanically. In particular, legacy token and Core tokenize have different security and reversibility semantics.

DataFog Python 4.8.x strategy	DataFog Core migration
`mask`	Use `mask`. Core also supports explicit leading or trailing reveal rules.
`token`	For an irreversible type placeholder, use `redact`. Use `tokenize` only when provider-backed restoration is required.
`hash`	There is no unkeyed hash strategy. Use keyed `pseudonymize` only when stable, non-reversible linkage is required.
`pseudonymize`	Review the behavior rather than renaming it. Core pseudonymization is deterministic, keyed, and provider-backed.

Core also adds remove, which deletes only the exact finding span.

For a strictly non-reversible first pass, choose redact, mask, or remove. Core pseudonymize is keyed and one-way but intentionally linkable; Core tokenize is reversible through the configured token provider.

See Privacy transformations for the full behavior and threat-model distinctions.

Remove legacy engine selection

Do not translate engine="regex", "smart", "spacy", or "gliner" into Core configuration. DataFog Core 0.3 owns detector composition and exposes locale as its scan setting. Detector provenance appears on each finding. Entity names are exact and case-sensitive. The built-in Core entities are: EMAIL, PHONE, SSN, CREDIT_CARD, IP_ADDRESS, DATE, and ZIP_CODE. Use canonical names such as DATE and ZIP_CODE rather than legacy aliases such as DOB or ZIP. Entity selection belongs in the transformation config, not the scan config.

Migration checklist

Replace the datafog distribution and datafog.engine imports.
Consume the list returned by scan instead of ScanResult.entities.
Rename entity fields and select codepoint_range or byte_range explicitly.
Replace scan_and_redact with scan_and_transform and a transformation envelope.
Remove legacy engine selectors and normalize entity names.
Select a Core strategy by behavior, especially for token and pseudonymize.
Stop depending on plaintext mappings or original PII in transformation records.
Update error handling for the DataFog Core exceptions.

Continue with the Python reference, then review findings and ranges and configuration.

German detection migration

The seven German labels are implemented in Core, with shared Rust, installed Python/Node and browser/WASM conformance fixtures. This is format/context compatibility, not exact regex parity with Python 4.8.1. The comparison baseline is the published 4.8.1 contract and its tests/test_de_pii_regex.py; those legacy fixtures remain unchanged. fixtures/german.jsonl records the new contract.

Behavior	Python 4.8.1 comparison	Core policy
Activation	German locale or explicit German entity selection	Singular `locale`; transformation selection alone does not activate a detector
IBAN validation	Format detection, no checksum requirement	Same; checksum-invalid candidates are detected
Whitespace	Broad regex `\s` acceptance	Only space, tab, NBSP, narrow NBSP at prescribed positions; newline and other Unicode whitespace rejected
Digits	Unicode regex `\d`	ASCII `[0-9]` only
Context	Legacy context-substring matching	Immediate context + horizontal gap + value; ASCII boundary on context and value; no intervening prose/newline
Grouping	Legacy regex forms	Independent optional single separators at documented boundaries; no arbitrary or repeated value separators
Postal spans	Prefix included	Prefix included; `PLZ: 10115` explicitly rejected
Passport/residence	One-letter/eight-digit and AT/seven-digit heuristics	Same limited heuristics, not exhaustive official document coverage
Generic overlaps	Adapter selection behavior is legacy-specific	Scans retain both candidates; transformations use existing Core overlap rules

All matching preserves source casing and separators. Allowlist entries must use that source representation. Required context stays in the output, except postal prefixes, which are part of the finding and are replaced too. The Python adapter may translate explicit German entity requests into locale de and then filter results, but must separately test legacy overlap and selection semantics. Core adds no plural locales, engine switch or scan-time entity selector. The capability-aware higher-level adapter and its datafog-core>=0.4.0,<0.5 dependency were merged in PR #179. The higher-level Python package has its own release process; see the reviewed legacy NPI selection limitation.

Credential detector migration (0.4.1)

Core 0.4.1 adds API keys, explicit Bearer header tokens, and scoped PostgreSQL credential URIs. Python 4.9 retains its existing default backend and legacy overlap-before-selection behavior. A same-span API key can suppress a Bearer token, a Bearer token can suppress a JWT, and a containing credential URI can suppress an inner API key or JWT before legacy selection runs. Explicit selection can therefore return empty. Native Core and datafog.v5 retain scan candidates and apply selection before overlap during transformation. See the compatibility matrix and exact-wheel integration results.

Get started

Core concepts

Guides

Reference

Project

Migrate from DataFog Python

Change the package and import

Update scan results

Replace scan-and-redact calls

Choose the intended transformation

Remove legacy engine selection

Migration checklist

German detection migration

Credential detector migration (0.4.1)

​Change the package and import

​Update scan results

​Replace scan-and-redact calls

​Choose the intended transformation

​Remove legacy engine selection

​Migration checklist

​German detection migration

​Credential detector migration (0.4.1)

Change the package and import

Update scan results

Replace scan-and-redact calls

Choose the intended transformation

Remove legacy engine selection

Migration checklist

German detection migration

Credential detector migration (0.4.1)