Implementing Collection-Level Validators

Within the broader Automated Schema Enforcement & Monitoring framework, collection-level validators are the innermost enforcement boundary — the last synchronous gate between application logic and persistent storage in MongoDB. This guide is a complete implementation workflow for platform engineers: it shows how to deploy a $jsonSchema validator idempotently from Python, how to gate that deployment on real compliance data, how to read the exact failure signatures MongoDB emits, and how to verify and roll back safely. When a validator is deployed correctly it eliminates silent data corruption, enforces contractual guarantees across microservices, and gives upstream consumers deterministic failure modes rather than half-written documents. Treated carelessly, the same feature introduces write-path latency, exclusive metadata locks, and schema divergence that only surfaces during an incident.

The deliverable by the end of this page is a repeatable deployment function you can drop into a CI/CD job or a Kubernetes operator, plus the diagnostic and rollback commands to operate it in production.

Architectural Context & Enforcement Boundaries

MongoDB evaluates a $jsonSchema validator synchronously during insert, update, and replace operations. Because validation runs inside the write path, an overly complex schema or unoptimized pattern matching adds measurable latency to every write and increases lock contention on high-throughput collections. The validator engine supports the standard JSON Schema keywords (type, required, enum, pattern, minimum, maximum) plus MongoDB’s bsonType extension for nested-object validation, array constraints, and conditional branching via $or / $and. For the exact keyword semantics and draft-level differences, see the guidance on MongoDB $jsonSchema syntax.

A validator does not stand alone. It is the synchronous tier of a layered enforcement model: the collection validator guarantees integrity at rest, application middleware enforces business rules the schema cannot express, and asynchronous checks run deeper, version-aware validation off the critical path. When a write is rejected, the failure should not vanish into an application log — it should be routed. Cleanly categorizing schema validation errors lets you distinguish a transient application bug from a legacy-data migration gap from an intentional schema-evolution request, and documents that fail the contract can be diverted into fallback validation chains rather than dropped.

Prerequisites & Operational Requirements

The workflow below targets a supported production topology. Confirm the following before applying any validator to a live collection.

• MongoDB version: 5.0 or later. The rich details object (schemaRulesNotSatisfied) in validation errors was introduced in MongoDB 5.0; on 4.x you receive only code: 121 with a generic message and cannot pinpoint the failing path.
• Driver: PyMongo 4.x (pip install "pymongo>=4.6,<5"). Pin the driver in your automation image so collMod argument handling and error classes stay stable across builds.
• Permissions: the deploying principal needs the collMod action on the target collection, which the built-in dbAdmin role grants. Running the dry-run compliance count additionally needs find on the collection (read).
• Environment assumptions: a replica set (not a standalone), because collMod propagates through the oplog and you will want to schedule it against secondaries’ replication lag. Have a maintenance window or a rolling-deploy plan ready — the operation takes an exclusive lock (see Edge Cases).
• Schema source of truth: the target schema should come from a version-controlled registry, not be hand-edited on the server. Aligning this with your schema versioning strategies is what makes the deployment auditable.

The two enforcement dials you will set are validationLevel and validationAction. Their combined behavior governs which documents are checked and what happens on failure:

validationAction	validationLevel	Checks applied to	On failure
warn	moderate	Inserts + updates to already-valid docs	Logs a warning; write succeeds
warn	strict	All inserts and updates	Logs a warning; write succeeds
error	moderate	Inserts + updates to already-valid docs	Rejects with WriteError code 121
error	strict	All inserts and updates	Rejects with WriteError code 121

moderate is the migration-safe choice: it enforces the contract on new and already-conforming documents while leaving pre-existing non-compliant documents writable, which is exactly the behavior you want when hardening a collection that predates the schema. The trade-offs between these levels are covered in depth under strict vs moderate validation levels.

Idempotent Deployment Workflow

A production deployment must be deterministic and repeatable — running it twice must not mutate collection metadata twice, and it must never take an exclusive lock it does not need. Follow this sequence.

1. Extract the current validator. Retrieve the live configuration with listCollections; the validator lives in the collection’s options object, not in collStats (which returns storage metrics only):

   db.runCommand({ listCollections: 1, filter: { name: "orders" } })
     .cursor.firstBatch[0].options

2. Diff by structural hash. Serialize both the target and the active validator, normalize key ordering, and compute a SHA-256 hash of each. A hash match means the deployment is a no-op.

3. Apply conditionally. Invoke collMod only when the hashes diverge (or when validationLevel / validationAction differ). This guarantees idempotency and eliminates redundant exclusive-lock acquisition during repeated CI/CD runs.

4. Dry-run compliance count. Before enforcing, count how many existing documents would fail the proposed schema. $jsonSchema is a valid query operator, so $nor finds non-compliant documents without a validator being active:

   db.orders.countDocuments({ $nor: [ { $jsonSchema: <schema> } ] })

   Note that the validate command checks BSON storage and index integrity — it does not test $jsonSchema compliance, so it cannot substitute for this count.

5. Phase the rollout. Deploy first with validationAction: "warn", watch the compliance metrics land in your async validation monitoring dashboards, and promote to validationAction: "error" only once the rejection rate falls below your threshold. The exact promotion thresholds and rollback triggers are laid out in setting up validationAction warn vs error in production.

Production-Ready Automation Implementation

The following PyMongo implementation performs the full workflow: it reads the live validator, hashes both sides for idempotency, runs the dry-run compliance gate, and applies the change with bounded exponential backoff. It is safe to run repeatedly from a deployment pipeline, an operator reconcile loop, or a one-shot runner. Wiring this function into your broader Python integration for schema checks gives you a single audited path for every validator change.

import hashlib
import json
import logging
import time
from typing import Dict, Any

from pymongo import MongoClient
from pymongo.errors import OperationFailure, PyMongoError, ServerSelectionTimeoutError

logger = logging.getLogger(__name__)


def _compute_schema_hash(schema: Dict[str, Any]) -> str:
    """Generate a deterministic SHA-256 hash from a normalized JSON schema."""
    normalized = json.dumps(schema, sort_keys=True, default=str)
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()


def apply_collection_validator(
    client: MongoClient,
    db_name: str,
    collection_name: str,
    target_schema: Dict[str, Any],
    validation_level: str = "strict",
    validation_action: str = "warn",
    max_retries: int = 3,
    dry_run: bool = False,
) -> Dict[str, Any]:
    """
    Idempotently apply a $jsonSchema validator to a MongoDB collection.
    Returns deployment metadata including applied status, rejection rate, and hash.
    """
    db = client[db_name]
    coll = db[collection_name]
    target_hash = _compute_schema_hash(target_schema)

    # 1. Extract current validator via listCollections
    # (collStats returns storage metrics only; the validator lives in options).
    try:
        coll_info = db.command("listCollections", filter={"name": collection_name})
        batch = coll_info["cursor"]["firstBatch"]
        if not batch:
            raise RuntimeError(f"Collection {collection_name} not found in {db_name}")
        current_opts = batch[0].get("options", {})
        current_validator = current_opts.get("validator", {})
        current_action = current_opts.get("validationAction", "error")
        current_level = current_opts.get("validationLevel", "strict")
    except (IndexError, KeyError) as exc:
        raise RuntimeError(f"Failed to read metadata for {collection_name}: {exc}")

    current_hash = _compute_schema_hash(current_validator)

    # 2. Idempotency check — skip if schema and both dials already match.
    if (
        current_hash == target_hash
        and current_action == validation_action
        and current_level == validation_level
    ):
        logger.info("Validator already up-to-date; skipping deployment.")
        return {"applied": False, "hash": target_hash, "status": "no-op"}

    # 3. Dry-run compliance gate. $jsonSchema is a query operator, so $nor + $jsonSchema
    # counts documents that would fail the proposed schema. validate() checks BSON/index
    # integrity, not schema compliance, so it is not a substitute here.
    if dry_run:
        try:
            total = coll.estimated_document_count()
            invalid = coll.count_documents({"$nor": [{"$jsonSchema": target_schema}]})
            rejection_rate = (invalid / total) * 100 if total > 0 else 0.0
            logger.info("Dry-run rejection rate %.2f%% (%d/%d)", rejection_rate, invalid, total)
            if rejection_rate > 5.0:
                logger.warning("Rejection rate above threshold; aborting strict deployment.")
                return {
                    "applied": False,
                    "hash": target_hash,
                    "status": "dry-run-aborted",
                    "rejection_rate": rejection_rate,
                }
        except OperationFailure as exc:
            logger.error("Dry-run compliance check failed: %s", exc)
            raise

    # 4. Apply the validator with bounded exponential backoff.
    validator_cmd = {
        "validator": {"$jsonSchema": target_schema},
        "validationLevel": validation_level,
        "validationAction": validation_action,
    }

    for attempt in range(1, max_retries + 1):
        try:
            db.command("collMod", collection_name, **validator_cmd)
            logger.info("Validator applied on attempt %d.", attempt)
            return {"applied": True, "hash": target_hash, "status": "success"}
        except OperationFailure as exc:
            if attempt == max_retries:
                logger.error("collMod failed after %d attempts: %s", max_retries, exc)
                raise
            backoff = 2 ** attempt
            logger.warning("OperationFailure on attempt %d; retrying in %ds.", attempt, backoff)
            time.sleep(backoff)
        except (PyMongoError, ServerSelectionTimeoutError) as exc:
            logger.error("Connection or server error during deployment: %s", exc)
            raise

    return {"applied": False, "hash": target_hash, "status": "failed"}

Key Operational Safeguards

• Structural hashing prevents redundant collMod calls that would otherwise trigger an exclusive metadata lock even when the schema payload is byte-for-byte identical.
• Explicit retry boundaries limit retries to transient OperationFailure states while bubbling connection and authentication errors up immediately, so a bad credential fails fast instead of after three backoffs.
• Dry-run gating counts schema-noncompliant documents before enforcing, preventing the write storm that legacy data would cause the moment validationAction flips to error.
• Decoupled action and level let you ship the schema definition in warn mode and change enforcement behavior later without editing the schema itself.

Diagnostic Fingerprints & Fast Resolution

When a validator rejects a write, MongoDB returns a WriteError with code: 121 and errmsg: "Document failed validation". On MongoDB 5.0+, the driver payload carries a details object whose schemaRulesNotSatisfied array isolates the exact failing rule and JSON path:

{
  "code": 121,
  "codeName": "DocumentValidationFailure",
  "errmsg": "Document failed validation",
  "details": {
    "operatorName": "$jsonSchema",
    "schemaRulesNotSatisfied": [
      {
        "operatorName": "required",
        "specifiedAs": { "required": ["total"] },
        "missingProperties": ["total"]
      },
      {
        "operatorName": "bsonType",
        "specifiedAs": { "bsonType": "double" },
        "reason": "type did not match",
        "consideredType": "string",
        "consideredValue": "19.99"
      }
    ]
  }
}

In PyMongo this surfaces as pymongo.errors.WriteError (or BulkWriteError for batch operations); inspect exc.details["errInfo"]["details"] to reach the same structure. A copy-paste diagnostic that reproduces the failing set from mongosh:

// List the exact rules the current documents violate, one sample per failure.
db.orders.aggregate([
  { $match: { $nor: [ { $jsonSchema: db.getCollectionInfos(
      { name: "orders" })[0].options.validator.$jsonSchema } ] } },
  { $limit: 5 },
  { $project: { _id: 1, total: 1, status: 1 } }
])

Common fingerprints and their fix:

• OperationFailure: Collection 'db.orders' already has a validator is not raised by collMod — collMod replaces the validator wholesale, so if you see stale rules you applied to the wrong namespace, re-check db_name.
• code: 121 on a document you believe is valid almost always means a bsonType mismatch: JSON number maps to bsonType: "double" or "int", so a schema demanding "double" rejects an integer-typed field. Use bsonType: ["double", "int", "long", "decimal"] when numeric width is not part of the contract.
• NotWritablePrimary / ServerSelectionTimeoutError during collMod means you targeted a secondary; DDL must run against the primary.

Edge Cases, Gotchas & Known Limitations

• collMod takes an exclusive (MODE_X) collection lock. It is fast for a metadata-only change, but on an extremely hot collection even a brief exclusive lock stalls concurrent writes. Schedule it in a maintenance window or accept a sub-second write pause.
• Validators are not indexed. The engine evaluates the schema against the document in memory; it never uses an index to satisfy validation. A pattern regex or deep $and / $or tree runs on every write, so keep the schema shallow and prefer enum over pattern where possible.
• moderate does not retroactively validate. Switching an existing collection to strict will start rejecting updates to legacy documents that were previously writable — this is the most common cause of a “sudden” 121 spike after a routine deploy. Run the dry-run count first.
• additionalProperties: false is a breaking contract. It rejects any field not named in properties, which silently breaks a downstream service that adds a new field before the schema is updated. Reserve it for collections with a truly frozen shape.
• bsonType vs type divergence. Use bsonType for MongoDB-native types (objectId, date, decimal, long); plain JSON Schema type cannot express them and will misclassify a Date as an object.

Verification & Rollback Procedures

Confirm the change landed exactly as intended before you walk away:

// 1. Verify the active validator, action, and level.
db.getCollectionInfos({ name: "orders" })[0].options
// 2. Confirm enforcement is live by attempting a known-bad insert.
db.orders.insertOne({ intentionallyMissing: true })   // expect WriteError 121

If the deployment misbehaves — an unexpected rejection spike, or a downstream service that cannot yet meet the contract — rolling back is a single, reversible collMod. Dropping to warn restores write availability instantly while you keep collecting compliance telemetry:

// Soft rollback: keep the schema, stop rejecting (time-to-recover: seconds).
db.runCommand({ collMod: "orders", validationAction: "warn" })

// Hard rollback: remove the validator entirely.
db.runCommand({ collMod: "orders", validator: {}, validationLevel: "off" })

Both commands are metadata-only and take effect immediately on the primary, propagating to secondaries through the oplog; time-to-recover for the soft rollback is effectively the replication lag of your slowest secondary.

Frequently Asked Questions

Related

• Automated Schema Enforcement & Monitoring — the parent architecture this synchronous enforcement layer plugs into, spanning validators, middleware, and pre-flight checks.
• Setting up validationAction warn vs error in production — the exact promotion thresholds and rollback triggers for moving a validator from warn to error.
• Categorizing schema validation errors — how to classify a code-121 rejection into a bug, a migration gap, or an intentional schema change.
• Building fallback validation chains — where rejected documents go so a strict validator never means data loss.
• Async validation monitoring dashboards — surfacing rejection rates and drift so you know when it is safe to promote enforcement.
• Python integration for schema checks — the wider PyMongo automation surface this deployment function belongs to.