GDPR Field-Level Encryption with Schema Validation

Combining client-side field-level encryption with a server-side $jsonSchema validator lets a MongoDB deployment satisfy two GDPR obligations at once: data protection by design, because personal data is encrypted before it ever leaves the application, and structural integrity, because the database still refuses to store a document whose sensitive fields are missing or — critically — written in the clear. Within the broader Schema Validation for Compliance & Audit Readiness framework, this page is the implementation reference for pairing MongoDB’s Client-Side Field Level Encryption (CSFLE) or Queryable Encryption with a collection validator that enforces the shape of encrypted records. The essential subtlety, which governs every design decision below, is that the validator runs on the server and sees only ciphertext: it can prove a field is present, prove it is stored as BSON binData, and prove it is not accidental plaintext, but it can never read or constrain the decrypted value. The precise keyword surface for that narrow-but-vital job is worked through in the companion Validating Encrypted Fields with $jsonSchema guide, and the threat-model reasoning for why a structural gate belongs on encrypted collections is developed in Security Boundaries in Schema Design. The deliverable here is a customer collection whose PII is encrypted by the driver and whose validator guarantees no record is ever persisted with a personal field left readable.

Architectural Context & Enforcement Boundaries

Field-level encryption in MongoDB is a client-side transformation. When an application configured with automatic encryption issues an insert, the driver (through the mongocryptd process or the crypt_shared library) rewrites the marked fields into ciphertext before the payload crosses the wire. The server receives, indexes, and stores those fields as BSON binary values — specifically binData with subtype 6, the reserved encrypted subtype. The server holds no data-encryption keys and performs no decryption; it treats an encrypted email field as an opaque blob. Decryption happens only on the return path, inside a client that possesses the key. This is what makes the pattern compliant: a database dump, an oplog capture, or a compromised backup exposes ciphertext, not personal data.

That architecture draws a hard boundary around what a collection validator can assert. A $jsonSchema validator is evaluated by the query engine on the write path, at the storage layer, with no access to keys. It sees the ciphertext exactly as the server stored it. Consequently a validator on an encrypted collection can enforce three things and only three things about a protected field: that it is present (via required), that it is stored as binData (via bsonType), and — the same assertion viewed from the other side — that it is not an unencrypted native type such as string. It cannot check that an email matches a pattern, that a status is one of an enum, or that a national ID has a minimum length, because those constraints would have to read the plaintext the server does not hold. Value-level rules must run client-side, before encryption; the server validator is a structural backstop.

Two schemas are in play, and conflating them is the most common design error. The driver’s automatic-encryption schema — an encryption schema map keyed by namespace — uses the JSON Schema keywords encryptMetadata and encrypt to tell the driver which fields to encrypt, with which algorithm, and under which data key. It never reaches the collection as a validator; it is a client-side instruction set consumed by libmongocrypt. The server-side collection validator is an ordinary $jsonSchema document set through createCollection or collMod that enforces structure on whatever the server ultimately stores. Keep them physically separate: the encryption schema lives in your application’s AutoEncryptionOpts, the structural validator lives in the collection metadata. The encryption schema decides how PII becomes ciphertext; the validator decides that ciphertext, and nothing plainer, is allowed to land.

Prerequisites & Operational Requirements

Automatic client-side encryption is not available in the Community server’s default configuration path; confirm the following before wiring the two schemas together.

• Server edition for automatic encryption: automatic CSFLE and Queryable Encryption require MongoDB Enterprise 4.2+ (CSFLE) or 7.0+ (Queryable Encryption GA), or MongoDB Atlas. The automatic-encryption logic is supplied by the crypt_shared library (preferred) or the mongocryptd binary, both distributed with Enterprise. The Community server can store binData and enforce the structural validator, but it cannot perform automatic encryption — a Community deployment must use explicit (manual) encryption in the client.
• Driver: PyMongo 4.x built with the encryption extra, pip install "pymongo[encryption]>=4.6,<5", which pulls in pymongocrypt and libmongocrypt. Pin it so AutoEncryptionOpts and the binData codec behavior stay stable across builds.
• A key vault and a KMS provider. Data-encryption keys are stored in a key-vault collection (conventionally encryption.__keyVault) and are themselves wrapped by a customer master key held in a KMS: AWS KMS, Azure Key Vault, GCP KMS, KMIP, or a local master key for development only. Provision the master key and create at least one data key before any encrypted write.
• Permissions: applying or altering the collection validator requires the collMod action, granted by dbAdmin on the database. The application principal that writes encrypted documents needs read access to the key vault; keep it distinct from the schema-administration principal so the encryption and enforcement responsibilities are separable for audit.
• A written field inventory. Enumerate every field carrying personal data under GDPR (direct identifiers, contact fields, national IDs, financial data) and decide per field whether it needs deterministic encryption (queryable for equality) or randomized encryption (not queryable, strongest). This inventory is the single source of truth that both the encryption schema and the validator are generated from.

Idempotent Deployment / Implementation Workflow

Deploying the pattern is a deterministic sequence: create keys, define the encryption schema the driver uses, then attach the structural validator the server enforces. Each step is independently verifiable and safe to re-run.

1. Create a data-encryption key in the key vault, wrapped by your KMS master key. The returned key id is referenced by every field the encryption schema protects.

   from pymongo import MongoClient
   from pymongo.encryption import ClientEncryption
   from bson.codec_options import CodecOptions
   
   key_vault_namespace = "encryption.__keyVault"
   kms_providers = {"local": {"key": LOCAL_MASTER_KEY}}  # 96-byte key; use a real KMS in prod
   
   client = MongoClient("mongodb://localhost:27017")
   client_encryption = ClientEncryption(
       kms_providers, key_vault_namespace, client, CodecOptions()
   )
   data_key_id = client_encryption.create_data_key("local", key_alt_names=["pii-key"])
   # data_key_id is a bson.binary.Binary (UUID) referenced by the encryption schema below.

2. Define the driver’s automatic-encryption schema for the target namespace using encryptMetadata and encrypt. This is what turns email and ssn into ciphertext; it is not the collection validator.

   from bson.binary import STANDARD  # UUID representation for keyId
   
   ENCRYPTION_SCHEMA = {
       "app.customers": {
           "bsonType": "object",
           "encryptMetadata": {"keyId": [data_key_id]},
           "properties": {
               "email": {
                   "encrypt": {
                       "bsonType": "string",
                       "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic",
                   }
               },
               "ssn": {
                   "encrypt": {
                       "bsonType": "string",
                       "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Random",
                   }
               },
           },
       }
   }

3. Attach the structural validator with mongosh (or the driver). It requires the encrypted fields, asserts each is binData, and — because it names no encrypt keyword — is a pure server-side structural gate:

   db.runCommand({
     collMod: "customers",
     validator: { $jsonSchema: {
       bsonType: "object",
       required: ["_id", "email", "ssn", "created_at"],
       properties: {
         _id: { bsonType: "objectId" },
         email: { bsonType: "binData" },
         ssn: { bsonType: "binData" },
         created_at: { bsonType: "date" }
       }
     }},
     validationLevel: "strict",
     validationAction: "error"
   })
   // Expected output: { ok: 1 }

4. Verify the round trip. Insert through the encrypting client and confirm the stored fields are binData, then read them back decrypted through the same client. The verification section below gives the exact assertions.

Production-Ready Automation Implementation

The following module wires both halves together and enforces the invariant that matters: a document written through the encrypting client is accepted, and a document written through a non-encrypting client — the accidental-plaintext case — is rejected by the validator rather than silently stored as readable PII. It configures AutoEncryptionOpts from the encryption schema, reconciles the structural validator idempotently, and surfaces the code 121 failure with its structured reason.

import logging
from typing import Any, Dict

from pymongo import MongoClient
from pymongo.encryption_options import AutoEncryptionOpts
from pymongo.errors import WriteError, OperationFailure

logger = logging.getLogger("gdpr-fle")

KEY_VAULT_NS = "encryption.__keyVault"

# Structural validator — deliberately distinct from the encryption schema.
# It can only assert presence, binData type, and "not plaintext".
STRUCTURAL_VALIDATOR = {
    "$jsonSchema": {
        "bsonType": "object",
        "required": ["_id", "email", "ssn", "created_at"],
        "properties": {
            "_id": {"bsonType": "objectId"},
            "email": {"bsonType": "binData"},
            "ssn": {"bsonType": "binData"},
            "created_at": {"bsonType": "date"},
        },
    }
}


def ensure_validator(db, coll: str, level: str = "strict", action: str = "error") -> bool:
    """Idempotently reconcile the structural validator; issue collMod only on drift."""
    info = db.command("listCollections", filter={"name": coll})
    batch = info["cursor"]["firstBatch"]
    current = batch[0].get("options", {}) if batch else {}
    if (
        current.get("validator") == STRUCTURAL_VALIDATOR
        and current.get("validationLevel") == level
        and current.get("validationAction") == action
    ):
        logger.info("validator already at target state; no collMod issued")
        return False
    if not batch:
        db.create_collection(coll, validator=STRUCTURAL_VALIDATOR,
                             validationLevel=level, validationAction=action)
    else:
        db.command("collMod", coll, validator=STRUCTURAL_VALIDATOR,
                   validationLevel=level, validationAction=action)
    logger.info("structural validator reconciled for %s", coll)
    return True


def encrypting_client(kms_providers: Dict[str, Any],
                      encryption_schema: Dict[str, Any]) -> MongoClient:
    """A client that auto-encrypts the fields named in the encryption schema map."""
    opts = AutoEncryptionOpts(
        kms_providers,
        KEY_VAULT_NS,
        schema_map=encryption_schema,
        # crypt_shared_lib_path="/opt/mongo/lib/mongo_crypt_v1.so",  # or run mongocryptd
    )
    return MongoClient("mongodb://localhost:27017", auto_encryption_opts=opts)


def insert_customer(client: MongoClient, doc: dict) -> str:
    """Insert PII through the encrypting client; the driver ciphers email and ssn."""
    try:
        result = client.app.customers.insert_one(doc)
        return str(result.inserted_id)
    except WriteError as exc:
        if exc.code == 121:
            rules = (exc.details or {}).get("errInfo", {}) \
                .get("details", {}).get("schemaRulesNotSatisfied")
            # A 121 here means a field reached the server as a native type,
            # i.e. it was NOT encrypted — a potential plaintext PII leak.
            logger.error("structural validation failed (possible plaintext PII): %s", rules)
        raise
    except OperationFailure as exc:
        logger.error("write failed (code %s): %s", exc.code, exc)
        raise

The load-bearing detail is the branch on code 121 inside insert_customer. Because the encrypting client always turns email and ssn into binData, that branch can only fire when a document reaches the server with one of those fields as a string — which happens when a write bypasses automatic encryption (a client built without AutoEncryptionOpts, or a field the encryption schema forgot to list). The validator converts that silent compliance failure into a hard, logged rejection at the exact moment the leak would otherwise be persisted.

Diagnostic Fingerprints & Fast Resolution

Encryption and validation failures produce a small set of exact signatures. Match on these to route the incident immediately.

Signature	Where it appears	Root cause	Resolution
WriteError code 121, schemaRulesNotSatisfied names email/ssn with failing bsonType	Driver, on insert	A field arrived as string — the write bypassed automatic encryption	Confirm the client was built with AutoEncryptionOpts and the field is in the encryption schema map
EncryptionError / mongocryptd spawn failure	Client at connect time	crypt_shared path wrong or mongocryptd not on PATH	Set crypt_shared_lib_path, or install the Enterprise mongocryptd binary
OperationFailure code 13 (Unauthorized) reading the key vault	Client at first encrypted write	App principal lacks read on encryption.__keyVault	Grant read on the key-vault namespace to the app role
Equality query on an encrypted field returns nothing	Application read path	Field encrypted with the -Random algorithm, which is not queryable	Re-key the field with the -Deterministic algorithm if it must be queried
collMod fails with FailedToParse on the validator	Schema deploy	Validator used a keyword such as pattern on a binData field	Remove value-level keywords; only bsonType, required, and not are meaningful on ciphertext

Read the live validator and confirm a field is genuinely stored encrypted rather than as text. The binData subtype 6 in the raw document is the fingerprint of a properly encrypted field:

// Ground truth: the stored type of an encrypted field, read WITHOUT an encrypting client.
db.getCollectionInfos({ name: "customers" })[0].options.validator
db.customers.findOne({}, { email: 1 })
// => { _id: ..., email: BinData(6, "AeI0...") }   // subtype 6 == encrypted; a string here is a leak

Edge Cases, Gotchas & Known Limitations

• The validator cannot assert the binData subtype. bsonType: "binData" matches any binary subtype, including generic subtype 0. It proves the field is not a native type such as string, but a determined caller could store a non-encrypted binary blob and pass. The validator is a strong guard against accidental plaintext, not a cryptographic proof of encryption; pair it with client discipline.
• Deterministic vs randomized is a compliance trade-off. -Deterministic encryption produces identical ciphertext for identical plaintext, which enables equality queries but leaks value-frequency information. Reserve it for fields that must be queried (e.g. email for lookup) and use -Random for everything else (e.g. ssn). The choice is invisible to the structural validator — both land as binData.
• Queryable Encryption manages its own metadata. With Queryable Encryption you supply an encrypted_fields map rather than an encrypt schema, and the server maintains internal metadata collections. A structural validator can still require the encrypted fields and assert binData, but do not attempt to add $jsonSchema rules to the QE state collections.
• $out/$merge and non-encrypting clients bypass encryption, not the validator. An aggregation that writes plaintext into the collection, or an analytics job connecting without AutoEncryptionOpts, will be caught by the structural validator if it targets a required encrypted field — this is precisely the safety the validator adds. But writes that never touch the encrypted fields slip past both.
• You cannot validate cleartext invariants server-side. Format checks (a valid email, a national-ID checksum, an allowlisted country) must run in the application before encryption. Once a value is ciphertext, pattern, enum, minLength, and minimum are meaningless to the server.
• Rotating a data key does not re-validate stored documents. Key rotation re-wraps data keys; it does not rewrite existing binData values and never triggers the validator, which only runs on the write path.

Verification & Rollback Procedures

Prove both halves are working before relying on them. The positive test is that an encrypting write is accepted and reads back as plaintext; the negative test is that a non-encrypting write of the same document is rejected with code 121, which is the only evidence the plaintext guard is live.

from pymongo import MongoClient
from pymongo.errors import WriteError

# 1. Encrypting client: insert succeeds and round-trips to plaintext.
enc = encrypting_client(kms_providers, ENCRYPTION_SCHEMA)
import datetime
_id = enc.app.customers.insert_one(
    {"email": "ana@example.com", "ssn": "123-45-6789",
     "created_at": datetime.datetime.utcnow()}
).inserted_id
assert enc.app.customers.find_one({"_id": _id})["email"] == "ana@example.com"

# 2. Plain client (no AutoEncryptionOpts): the SAME document must be rejected,
#    because email/ssn would land as plaintext strings.
plain = MongoClient("mongodb://localhost:27017")
try:
    plain.app.customers.insert_one(
        {"email": "leak@example.com", "ssn": "999-99-9999",
         "created_at": datetime.datetime.utcnow()})
    raise AssertionError("plaintext PII was NOT blocked — validator is not enforcing")
except WriteError as exc:
    assert exc.code == 121
    print("confirmed: plaintext PII write rejected with 121")

Rollback of the structural validator is a single metadata-only collMod that restores write availability without touching data or keys. A soft rollback keeps the schema attached but stops rejecting; a hard rollback detaches it entirely. Note that rolling back the validator never decrypts anything — encryption is independent of the validator and remains fully in force.

// Soft rollback — keep the contract, stop blocking writes.
db.runCommand({ collMod: "customers", validationLevel: "moderate", validationAction: "warn" })

// Hard rollback — remove the structural gate entirely (encryption still applies).
db.runCommand({ collMod: "customers", validator: {}, validationLevel: "off" })

Both take effect on the primary immediately and propagate through the oplog to secondaries, so time-to-recover is the replication lag of your slowest secondary — typically seconds. Because the validator and the encryption schema are decoupled, you can revert the structural gate in an incident without any risk to the confidentiality of already-stored data.

Frequently Asked Questions

Related

• Schema Validation for Compliance & Audit Readiness — the parent framework for validation controls that produce audit and regulatory evidence.
• Validating Encrypted Fields with $jsonSchema — the exact keyword surface usable on a binData field and a test that a plaintext write is rejected.
• Security Boundaries in Schema Design — the threat-model reasoning for hardening collections with structural constraints.
• Audit Log Schema Contracts — enforcing structure on the immutable records that evidence compliance.
• Strict vs Moderate Validation Levels — choosing the enforcement level for the structural validator on an existing encrypted collection.