Schema Validation

Graceful Degradation for Legacy Document Formats in MongoDB

When platform teams enforce strict $jsonSchema validators on mature collections, legacy documents trigger WriteError code 121 failures that cascade into pipeline stalls and migration rollbacks. This page — part of the building fallback validation chains workflow inside the broader Automated Schema Enforcement & Monitoring framework — gives MongoDB developers and data engineers a runnable dual-path ingestion playbook that keeps writes available while progressively aligning historical payloads with the modern validation contract. Graceful degradation here is not a tolerance for schema drift; it is a deterministic routing strategy that diverts non-compliant documents to a staging path instead of dropping them, so a strict validator never means data loss.

Dual-path ingestion that keeps writes available under a strict validator An incoming payload hits the strict $jsonSchema validator on the write path. Conforming documents persist in the primary collection. A document that fails with WriteError code 121 is diverted, under the same _id, to a _legacy_staging collection. Off the write path, an asynchronous reconciliation worker transforms and pre-flight-validates each staged document, then upserts compliant results back into the primary collection; documents that are still non-compliant remain staged for the next pass. asynchronous · off the write path Incoming payload insert · update · replace Strict $jsonSchema validator synchronous write-path gate Primary collection conforming · durable _legacy_staging same _id · lineage kept Reconciliation worker transform · pre-flight validate pass fail · code 121 upsert still invalid → stay staged

Operational Mechanics and Write-Path Impact

MongoDB evaluates a $jsonSchema validator synchronously on every insert, update, and replace. A legacy document that violates the contract is rejected inside the write path before it reaches storage, so the degradation strategy you choose is really a decision about what happens at the moment of rejection. The two server-side dials that shape that moment are validationLevel (which documents are checked) and validationAction (what happens on failure); the deeper trade-offs between them are covered under strict vs moderate validation levels.

The table below maps each degradation strategy to its write-path effect on a legacy payload that fails the schema:

Strategy Configuration Legacy write outcome Data lineage Use when
Hard reject validationAction: "error", validationLevel: "strict" Rejected with WriteError 121 None — caller must handle New collections with no historical debt
Update-tolerant validationAction: "error", validationLevel: "moderate" Existing non-compliant docs stay writable; only new inserts checked Preserved in place Hardening a collection that predates the schema
Observe-only validationAction: "warn" Write succeeds; violation logged Preserved in place Measuring drift before enforcing
Dual-path route error + application-layer interception Diverted to _legacy_staging, reconciled async Full — same _id in staging Migrations that must not stall the pipeline

moderate is the migration-safe server dial: it enforces the contract on new and already-conforming documents while leaving pre-existing non-compliant documents writable, which is exactly what routine background jobs (TTL cleanup, aggregation materializations, audit updates) need. The dual-path route goes further — it is the pattern that keeps a strict error validator in force for correctness while guaranteeing availability, and it is the one this playbook implements. Three misconfigurations account for the vast majority of legacy rejections:

  1. Overconstrained additionalProperties: false. Legacy documents often carry deprecated telemetry fields or untyped nested objects. Setting additionalProperties: false at the root or a nested level immediately rejects any historical document containing unregistered keys.
  2. Type-coercion gaps. A legacy price: "19.99" (string) fails a validator expecting bsonType: "double", even if the application layer previously cast it implicitly. The validator engine does not coerce types — see the $jsonSchema syntax reference for exact bsonType semantics.
  3. validationLevel: "strict" on active collections. Strict evaluates every write against the full schema, so any background touch of a legacy record fails immediately.

Exact Diagnostic Fingerprints and Fast Resolution

Legacy validation failures surface predictably at the storage-engine boundary. The definitive signature is a WriteError with code: 121 and errmsg: "Document failed validation"; the server response carries a details object whose schemaRulesNotSatisfied array isolates the failing JSON path and constraint:

{
  "code": 121,
  "codeName": "DocumentValidationFailure",
  "errmsg": "Document failed validation",
  "details": {
    "operatorName": "$jsonSchema",
    "schemaRulesNotSatisfied": [
      {
        "operatorName": "required",
        "specifiedAs": {"required": ["metadata.version"]},
        "missingProperties": ["metadata.version"]
      },
      {
        "operatorName": "bsonType",
        "specifiedAs": {"bsonType": "double"},
        "reason": "type did not match",
        "consideredType": "string",
        "consideredValue": "19.99"
      }
    ]
  }
}

Platform teams frequently misdiagnose these as network timeouts or write-concern issues. The unambiguous tell is the presence of schemaRulesNotSatisfied in the response — that field only appears on strict-validator rejection. In bulk operations (insertMany, updateMany) the error surfaces at the first non-compliant index when ordered: true (the default), halting the whole batch; with ordered: false the batch continues and each failure is captured in BulkWriteError.details["writeErrors"].

Before enforcing anything, quantify how many existing documents would fail. Because $jsonSchema is a valid query operator, a single mongosh command counts non-compliant documents with no validator active:

// Copy-paste triage: how much legacy debt does this schema imply?
db.orders.countDocuments({ $nor: [ { $jsonSchema: db.getCollectionInfos({ name: "orders" })[0].options.validator.$jsonSchema } ] })

Classifying those rejections cleanly — bug vs. migration gap vs. intentional evolution — is the job of categorizing schema validation errors; the count above tells you which bucket dominates before you flip enforcement on.

Step-by-Step Playbook

This sequence restores write availability within minutes and drains the legacy backlog without ever disabling validation for new data. Deploy the routing logic at the application or middleware layer, following the fallback routing for invalid documents pattern for deterministic, idempotent retries.

  1. Unblock legacy updates immediately. If background jobs touching historical documents are failing, switch the server dial so only new inserts are checked:

    db.runCommand({ collMod: "orders", validationLevel: "moderate" })
// expect: { ok: 1 }  — existing non-compliant docs are now writable

  2. Intercept code: 121 and route to staging. Wrap ingestion so a validation failure diverts the payload to _legacy_staging under the same _id, and let every other error propagate:

    from datetime import datetime, timezone

from pymongo import MongoClient, errors

def ingest_with_degradation(db, collection_name, doc):
    """Insert into the primary collection; on a $jsonSchema violation
    (code 121) route to a legacy-staging collection for async
    reconciliation. All other write errors propagate immediately."""
    try:
        db[collection_name].insert_one(doc)
        return "primary"
    except errors.WriteError as e:
        if e.code == 121:
            db[f"{collection_name}_legacy_staging"].replace_one(
                {"_id": doc["_id"]},
                {**doc,
                 "_validation_error": e.details.get("errmsg"),
                 "_rules": e.details.get("details", {}).get("schemaRulesNotSatisfied"),
                 "_routed_at": datetime.now(timezone.utc)},
                upsert=True,
            )
            return "staged"
        raise

  3. Reconcile and upsert. A background worker transforms each staged document — coercing types, backfilling required fields — validates it locally, and upserts it into the primary collection. Pre-flighting with the jsonschema library keeps a doomed write off the wire:

    import jsonschema

def reconcile(db, collection_name, target_schema):
    staging = db[f"{collection_name}_legacy_staging"]
    for staged in staging.find():
        doc = transform_legacy(staged)          # your field-level migration
        try:
            jsonschema.validate(doc, target_schema)   # pre-flight, no round trip
        except jsonschema.ValidationError:
            continue                              # still non-compliant; leave staged
        db[collection_name].replace_one({"_id": doc["_id"]}, doc, upsert=True)
        staging.delete_one({"_id": staged["_id"]})

  4. Re-enforce strictness. Once the staging backlog drops below your threshold (commonly under 1% of volume), restore full enforcement and archive the staging collection:

    db.runCommand({ collMod: "orders", validationLevel: "strict" })
// expect: { ok: 1 }  — every write is checked again; staging is now empty

Wire the staging depth and rejection rate into your async validation monitoring dashboards so step 4 is triggered by data, not guesswork. For bulk backfills, always run with ordered=False to capture per-document errors instead of aborting on the first failure — the wider PyMongo surface for this lives under Python integration for schema checks.

Failure Modes & Rollback

Each step has a distinct way to go wrong and a bounded recovery:

  • _id collisions in staging. Using insert_one instead of replace_one on retry raises DuplicateKeyError when a payload is reprocessed. The playbook uses an upsert keyed on _id, so retries are idempotent. Recovery: switch the staging write to the upsert form above; time-to-recover is a single redeploy.

  • Reconciliation lag under load. If staged volume grows faster than the worker drains it, the primary collection falls behind. Recovery: parallelize reconcile across _id ranges and watch write-lock contention and oplog size; time-to-recover scales with worker count.

  • moderate masking a real regression. While the collection is on moderate, a genuinely buggy new insert to an already-non-compliant document can slip through. Recovery: keep validationAction: "error" throughout (never drop to warn on the primary) so new writes are still rejected — only the level changes.

  • Premature re-enforcement. Flipping back to strict while staging is non-empty re-triggers the alert storm. Rollback is a single reversible collMod:

    // Roll back to migration-safe enforcement (time-to-recover: seconds).
db.runCommand({ collMod: "orders", validationLevel: "moderate" })

    The command is metadata-only and takes effect on the primary immediately, propagating to secondaries through the oplog; time-to-recover equals the replication lag of your slowest secondary.

Frequently Asked Questions

Does switching to validationLevel: "moderate" stop rejecting bad new documents?

No. moderate only narrows which documents are checked — it skips validation on updates to documents that are already non-compliant, but every fresh insert and every update to an already-valid document is still fully checked. As long as validationAction stays error, a malformed new payload is still rejected with code 121. That is precisely why the playbook changes the level, never the action, on the primary collection.

Why route to a separate _legacy_staging collection instead of just logging the failure?

Logging discards the payload; staging preserves it under the same _id, which keeps full data lineage and makes reconciliation idempotent. You can transform, re-validate, and upsert the document later without asking the upstream producer to resend it. A log line cannot be replayed into the primary collection; a staged document can.

Can I distinguish a legacy-format failure from an application bug at the point of rejection?

Yes — parse the schemaRulesNotSatisfied array in e.details. A missing required field or a string-where-double bsonType mismatch on a historical shape signals legacy debt to be reconciled; a violation on a field your current code writes signals a bug to fix upstream. Routing on that distinction is the subject of categorizing schema validation errors.

Related in this section

Async Validation Monitoring DashboardsWithin the broader Automated Schema Enforcement & Monitoring framework, an asynchronous validation monitoring dashboard is the… Building Fallback Validation Chains for MongoDB IngestionWithin the broader Automated Schema Enforcement & Monitoring framework, a fallback validation chain is the pattern that keeps ingestion… Categorizing Schema Validation ErrorsWithin the broader Automated Schema Enforcement & Monitoring framework, error categorization is the translation layer that turns a raw… Implementing Collection-Level ValidatorsWithin the broader Automated Schema Enforcement & Monitoring framework, collection-level validators are the innermost enforcement boundary…