Send a FHIR® resource to /fhir/$de-identify get it back anonymized and/or pseudonymized.

Based on the brilliant Tools for Health Data Anonymization.

docker run --rm -i -p 8080:8080 \
  -e PseudonymizationService="None" \
  -e UseSystemTextJsonFhirSerializer="true" \
  ghcr.io/miracum/fhir-pseudonymizer:v2.29.0

curl -X POST -H "Content-Type:application/fhir+json" "http://localhost:8080/fhir/\$de-identify" -d @benchmark/observation.json

This uses the default anonymization config which only changes the sample Observation's id.

An example for deploying using (Docker) Compose can be found in the compose folder.

The recommended deployment is on Kubernetes. See https://github.com/miracum/charts/tree/master/charts/fhir-pseudonymizer for a Helm Chart.

An OpenAPI definition for the FHIR operation endpoints is available at /swagger/:

The server provides a /fhir/$de-identify operation to de-identfiy received FHIR resources according to the configuration in the anonymization.yaml rules. See Tools for Health Data Anonymization for more details on the anonymization rule configuration.

The service comes with a sample configuration file to help meet the requirements of HIPAA Safe Harbor Method (2)(i): hipaa-anonymization.yaml.This configuration can be used by setting AnonymizationEngineConfigPath=/etc/hipaa-anonymization.yaml.

A new pseudonymize method was added to the default list of anonymization methods linked above. It uses either gPAS, Vfps, or entici to create pseudonyms and replace the values in the resource with them. For example, the following rule replaces all identifiers of type http://terminology.hl7.org/CodeSystem/v2-0203|MR with a pseudonym generated in the PATIENT domain.

fhirPathRules:
  - path: nodesByType('Identifier').where(type.coding.system='http://terminology.hl7.org/CodeSystem/v2-0203' and type.coding.code='MR').value
    method: pseudonymize
    domain: PATIENT

Note that if the domain setting is omitted, and an ID or reference is pseudonymized, then the resource name is used as the pseudonym domain. For example, pseudonymizing "reference": "Patient/123" will try to create a pseudonym for 123 in the Patient domain.

When using Vfps, the domain setting can instead also be set as namespace.

Note that all methods defined in Tools for Health Data Anonymization are supported. For example, to clamp a patient's birthdate if they were born before January 1st 1931 to 01/01/1930, use:

fhirPathRules:
  - path: Patient.birthDate
    method: generalize
    cases:
      "$this < @1931-01-01": "@1930-01-01"
    otherValues: keep

The /fhir/$de-pseudonymize operation is used to revert the pseudonymize and encrypt methods applied to any resource. Accessing this endpoint requires authentication. So make sure to set the APIKEY env var.

⚠ if decryption or de-pseudonymization of a value fails, then the original value is returned. This behavior may change or be made configurable in the future.

While not part of the "user" API, the application exposes metrics in the Prometheus format at the /metrics endpoint on port 8081.

In addition to the HTTP API, the application can consume FHIR resources/bundles directly from one or more Kafka topics, pseudonymize them, and publish the result to an output topic. The original message's key is preserved on the output message unchanged. The output topic for a given input topic is derived by applying a regular expression match-and-replace (Kafka__OutputTopicPattern / Kafka__OutputTopicReplacement) to the input topic's name. By default, this just prepends pseudonymized. to every topic, e.g. fhir.test becomes pseudonymized.fhir.test. To instead insert it after a common prefix, e.g. turning fhir.test into fhir.pseudonymized.test, set Kafka__OutputTopicPattern=^fhir\. and Kafka__OutputTopicReplacement=fhir.pseudonymized..

This is enabled by setting Kafka__Client__BootstrapServers and at least one topic in Kafka__Topics. See the Kafka configuration section below for all available options.

Messages are anonymized concurrently by a fixed pool of workers (Kafka__WorkerCount, defaulting to the number of CPU cores). Each Kafka partition is consistently routed to the same worker, so per-partition message order is preserved while different partitions are processed in parallel. Combined with Kafka's own partition-based consumer group scaling, this means throughput can be increased both by raising Kafka__WorkerCount within a replica and by running multiple replicas with the same Kafka__Consumer__GroupId.

Transient failures calling the configured pseudonymization service (gPAS, Vfps, entici, Mii) are already retried with backoff at the HTTP/gRPC client level. If processing a message still fails after that (e.g. malformed input, an unrecoverable error, or those retries being exhausted), the original, unmodified message is published unchanged to a dead letter topic named error.<input-topic>.<group-id> (mirroring Spring Kafka's default dead letter topic naming), with headers describing the failure (x-error-type, x-error-message, x-source-topic, x-source-partition, x-source-offset), and its offset is committed so the consumer can move past it. If producing to the dead letter topic itself fails, the message's offset is left uncommitted so it gets reprocessed after a restart.

If Kafka__ProvenanceTopic is set, every successfully pseudonymized resource - whether pseudonymized via the Kafka consumer above or via the REST $de-identify endpoint - also produces a Provenance audit trail: a single Provenance resource wrapped in a transaction Bundle and published to that topic. Its target references what pseudonymization produced - every pseudonymized resource (for a Bundle input, one target per contained resource; otherwise just the one resource), each by its post-pseudonymization <type>/<id> (since e.g. cryptoHash-ing Resource.id changes it) - while its entity (role source) references what each was derived from: the same resources' pre-pseudonymization identity, by <type>/<Resource.id> if they had one, otherwise by their first identifier (a Reference.identifier, for resources with no resolvable id). Its activity is always http://terminology.hl7.org/CodeSystem/iso-21089-lifecycle|deidentify ("De-Identify (Anononymize) Record Lifecycle Event").

Its agent identifies the FHIR Pseudonymizer itself: type is performer (http://terminology.hl7.org/CodeSystem/provenance-participant-type) and role is AUT/"author (originator)" (http://terminology.hl7.org/CodeSystem/v3-ParticipationType), the closest standard fit for an automated system that transformed the content. who references a Device resource (rather than just a display string) that is included in the same Bundle, naming and versioning the FHIR Pseudonymizer (its version truncated to major.minor.build, matching semver); that Device's id is the SHA-256 hash of its name and version, so it - and the Provenance's reference to it - stays the same across restarts and only changes on a version bump. It also carries a human-readable business identifier, e.g. fhir-pseudonymizer-v2.29.0, under the https://miracum.github.io/fhir-pseudonymizer/identifiers/device-id system.

The Provenance is a PUT to Provenance/<id>, where <id> is the SHA-256 hash (lowercase hex) of its targets' identity, joined together: for each target this is its pre-pseudonymization <type>/<Resource.id> if it had one, otherwise its identifier.system|identifier.value (using its first identifier). This makes re-pseudonymizing the same input idempotent: the resulting Provenance always gets the same id, so applying the bundle to a FHIR server upserts the existing record instead of accumulating duplicates. Targets with neither an id nor an identifier are skipped when deriving the id; if none of them have either, the Provenance falls back to a random id (so it will no longer be idempotent for that input). Since a Bundle only ever contains this one Provenance (plus the Device), the wrapping Bundle is given the Provenance's id, which also becomes the Kafka message key it's produced with - not the source message's key - so that repeated provenance for the same input lands on the same partition (e.g. for log compaction).

The provenance bundle carries the same headers as the source message (e.g. tracing context) when read from Kafka; REST calls produce it without any. A failure to produce the provenance bundle is logged but does not affect the pseudonymized resource being returned/delivered. Producing provenance bundles from the REST API requires Kafka__Client__BootstrapServers to be set even if Kafka__Topics (Kafka consuming) is left unset.

You can configure the anonymization and pseudonymization rules in the anonymization.yaml config file. It's mounted at /etc/anonymization.yaml within the container by default. See Tools for Health Data Anonymization for details on the syntax and options.

Additionally, there are some optional configuration values that can be set as environment variables:

See appsettings.json for additional options.

The application supports pseudonymization using either gPAS or Vfps which can be configured via the PseudonymizationService setting. Service-specific configuration settings are listed below.

When using entici as a pseudonymization backend, you need to set additional settings for each rule that uses the pseudonymize method. These can be set under a entici section inside the anonymization config:

fhirPathRules:
  - path: nodesByType('Identifier').where(type.coding.where(system='http://terminology.hl7.org/CodeSystem/v2-0203' and code='MR').exists()).value
    method: pseudonymize
    # the domain will be used as the system of the identifier when invoking the entici pseudonymize operation
    domain: https://fhir.example.com/identifiers/patient-id
    entici:
      # the type of FHIR resource this pseudonym should be associated with
      resourceType: Patient
      # (optional) the project within entici the pseudonym is a part of
      project: some-internal-entici-project-name

When using the cryptoHash method on a value, the result is a hex-encoded string of 64 characters length. You can truncate this to a specific maximum length using the truncateToMaxLength setting. For example:

fhirPathRules:
  - path: Resource.id
    method: cryptoHash
    truncateToMaxLength: 32

Will truncate the usually 64-character-long hash to the following:

{
  "resourceType": "Patient",
  "id": "b43a73c44e6d5b57644b63d89ee90cbf"
}

Anonymization and pseudonymization rules in the anonymization.yaml config file can be overridden and/or extended on a per request basis.

Pseudonymization supports a domain-prefix rule setting which can be used to dynamically configure the target domain by providing its value as part of the request body.

The following example shows how to use this feature to use a single service configuration in order to support multiple projects which have the same basic domain names, prefixed by a project name.

gPAS domains for patient IDs:

anonymization.yml:

---
fhirVersion: R4
fhirPathRules:
  - path: nodesByType('Identifier').where(type.coding.system='http://terminology.hl7.org/CodeSystem/v2-0203' and type.coding.code='MR').value
    method: pseudonymize
    domain: patient

Providing the prefix (i.e. miracum- or test-) via the request, pseudonymization can be done with the same rules for different projects.

Rule settings can be provided by using the Parameters resource with a settings parameter and parts consisting of the settings key and value. The resource parameter must contain the actual target resource.

The following request body and the (fixed) configuration settings above will result in the target domain miracum-patient.

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "settings",
      "part": [
        {
          "name": "domain-prefix",
          "valueString": "miracum-"
        }
      ]
    },
    {
      "name": "resource",
      "resource": {
        "resourceType": "Bundle",
        "type": "transaction",
        "entry": [
          {
            "fullUrl": "urn:uuid:3bc44de3-069d-442d-829b-f3ef68cae371",
            "resource": {
              "resourceType": "Patient",
              "identifier": [
                {
                  "type": {
                    "coding": [
                      {
                        "system": "http://terminology.hl7.org/CodeSystem/v2-0203",
                        "code": "MR"
                      }
                    ]
                  },
                  "system": "http://acme.org/mrns",
                  "value": "12345"
                }
              ],
              "name": [
                {
                  "family": "Jameson",
                  "given": ["J", "Jonah"]
                }
              ],
              "gender": "male"
            },
            "request": {
              "method": "POST",
              "url": "Patient/12345"
            }
          }
        ]
      }
    }
  ]
}

Note: The domain name could also have been replaced completely by overriding the domain setting with the desired value. This works for all rule settings regardless of the method value.

To test Vfps and tracing via Jaeger, run

docker compose -f compose.dev.yaml up

to also start a Keycloak instance with pre-configured fhir-pseudonymizer client, set --profile=keycloak:

docker compose -f compose.dev.yaml --profile=keycloak up

To also start a local Kafka broker for testing the Kafka consumer, set --profile=kafka:

docker compose -f compose.dev.yaml --profile=kafka up

dotnet restore
dotnet build

Or using Docker:

docker build -t ghcr.io/miracum/fhir-pseudonymizer:local-build .

dotnet run --project src/FhirPseudonymizer

dotnet test src/FhirPseudonymizer.Tests/

pre-commit install
pre-commit install --hook-type commit-msg

ITER8_CLI_URL="https://github.com/iter8-tools/iter8/releases/download/v0.13.18/iter8-linux-amd64.tar.gz"
curl -LSs "${ITER8_CLI_URL}" | tar xz
mv linux-amd64/iter8 /usr/local/bin/iter8
chmod +x /usr/local/bin/iter8
iter8 version

kind create cluster

export IMAGE_TAG="iter8-test"

docker build -t ghcr.io/miracum/fhir-pseudonymizer:${IMAGE_TAG} .

kind load docker-image ghcr.io/miracum/fhir-pseudonymizer:${IMAGE_TAG}

helm upgrade --install \
  --set="image.tag=${IMAGE_TAG}" \
  -f tests/iter8/values.yaml \
  --wait \
  --timeout=10m \
  fhir-pseudonymizer oci://ghcr.io/miracum/charts/fhir-pseudonymizer

kubectl apply -f tests/iter8/experiment.yaml

iter8 k assert -c completed --timeout 15m
iter8 k assert -c nofailure,slos
iter8 k report

# to restart:
kubectl delete job default-1-job
kubectl apply -f tests/iter8/experiment.yaml

Note Example runs were conducted on the following hardware:

OS=Windows 11 (10.0.22000.978/21H2)
12th Gen Intel Core i9-12900K, 1 CPU, 24 logical and 16 physical cores
32GiB of DDR5 4800MHz RAM
Samsung SSD 980 Pro 1TiB
.NET SDK=7.0.101

Prerequisites: https://github.com/codesenberg/bombardier

dotnet run -c Release --project=src/FhirPseudonymizer

In a different terminal

cd benchmark/
$ ./bombardier.sh

Bombarding http://localhost:5000/fhir/$de-identify for 1m0s using 125 connection(s)
[====================================================================================================================] 1m0s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     13107.78    1552.49   18917.77
  Latency        9.53ms   559.41us    53.88ms
  Latency Distribution
     50%     9.00ms
     75%    11.00ms
     90%    12.00ms
     95%    13.00ms
     99%    16.73ms
  HTTP codes:
    1xx - 0, 2xx - 786655, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    96.37MB/s

You can improve throughput and P99 latencies by opting-in to using the System.Text.Json based FHIR resource serializer. It can be enabled via appsettings.json or using the UseSystemTextJsonFhirSerializer environment variable:

UseSystemTextJsonFhirSerializer=true dotnet run -c Release --project=src/FhirPseudonymizer

Bombarding http://localhost:5000/fhir/$de-identify for 1m0s using 125 connection(s)
[====================================================================================================================] 1m0s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     21508.39    3751.90   38993.50
  Latency        5.80ms     1.63ms   442.82ms
  Latency Distribution
     50%     5.00ms
     75%     6.00ms
     90%     7.75ms
     95%     9.00ms
     99%    12.00ms
  HTTP codes:
    1xx - 0, 2xx - 1291159, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   158.17MB/s

Prerequisites:

• cosign
• slsa-verifier
• crane

All released container images are signed using cosign and SLSA Level 3 provenance is available for verification.

IMAGE=ghcr.io/miracum/fhir-pseudonymizer:v2.29.0
DIGEST=$(crane digest "${IMAGE}")
IMAGE_DIGEST_PINNED="ghcr.io/miracum/fhir-pseudonymizer@${DIGEST}"
IMAGE_TAG="${IMAGE#*:}"

cosign verify \
   --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
   --certificate-identity-regexp="https://github.com/miracum/.github/.github/workflows/standard-build.yaml@.*" \
   --certificate-github-workflow-name="ci" \
   --certificate-github-workflow-repository="miracum/fhir-pseudonymizer" \
   --certificate-github-workflow-trigger="release" \
   --certificate-github-workflow-ref="refs/tags/${IMAGE_TAG}" \
   "${IMAGE_DIGEST_PINNED}"

slsa-verifier verify-image \
    --source-uri github.com/miracum/fhir-pseudonymizer \
    --source-tag ${IMAGE_TAG} \
    "${IMAGE_DIGEST_PINNED}"

The project's versioning follows the SemVer convention. However, we exclude metrics (ie. anything under the :8081/metrics endpoint), traces, and the contents of the container image from this. Always be prepared to double-check the release notes before updating.

Icons made by Freepik from Flaticon.