Schema Validation

Backfilling New Required Fields in Batches

Adding a field to a MongoDB required array is a schema tightening that will reject writes to every historical document still missing that field — this page is the operational playbook for backfilling the value across a live collection before the validator can bite, sitting under document transformation pipelines within the broader Zero-Downtime Schema Migration Automation in MongoDB framework. The reader outcome is a collection where every document already carries the new field, so promoting the validator to strict + error adds zero rejections. The trap is timing: if you attach a required constraint and flip enforcement while un-backfilled documents remain, the next application update to any of them fails with code 121 — not because the update is wrong, but because the resulting document still lacks the mandatory field. The fix is a batched, idempotent backfill keyed on {field: {$exists: false}}, run under a permissive validation level, that stamps a computed default onto every document that lacks the field before enforcement is tightened.

Operational Mechanics and Write-Path Impact

MongoDB validates a document as a whole, not as a diff. When required: ["region"] is active under validationLevel: "strict", any write that produces a document without region is rejected — and crucially, a partial $set that does not touch region still produces such a document if the field was never there. This is why a new required field is uniquely dangerous: unlike a bsonType tightening, which only fails documents actively carrying a bad value, a missing-required failure fires on every write to every un-backfilled document, including unrelated field updates. The validationLevel dial decides who is exposed, and the interaction is the whole reason the backfill must precede the flip:

validationLevel validationAction Update to a doc missing the new required field Backfill safe to run?
moderate warn Grandfathered — succeeds, logged only Yes — the ideal backfill window
moderate error Grandfathered — succeeds (doc was already invalid) Yes — updates to invalid docs bypass the check
strict warn Succeeds, but every write is logged as a violation Yes, though the log floods
strict error Rejected with code 121 No — the backfill fights the validator

The safe posture is moderate throughout the backfill: under moderate, an updateMany that adds the field to a currently-invalid document is grandfathered through even under validationAction: "error", because MongoDB exempts updates to already-non-compliant documents from the check. The moment the backfill completes and the count reaches zero, the collection is uniformly compliant and the promotion to strict is a metadata-only flip with no blast radius. The deeper grandfathering semantics are developed in strict vs moderate validation levels; what matters here is that the backfill is a write, and its safety depends entirely on the level being permissive while it runs.

Ordering a required-field backfill before enforcement A timeline with four stops. First attach the validator as moderate plus warn so the new required field is defined but not enforced. Second run the batched backfill that stamps a computed default onto every document matching field exists false. Third verify the count of documents missing the field has reached zero. Fourth promote to strict plus error, which is now a metadata-only flip with no rejections because every document already carries the field. A dashed danger arrow shows that flipping to strict plus error before the backfill completes causes code 121 rejections on ordinary updates. flip before backfill completes → code 121 on ordinary updates validationLevel: moderate — updates to un-backfilled docs are grandfathered 1 Attach validator moderate + warn 2 Batched backfill $exists:false → $set 3 Verify count = 0 go / no-go 4 Promote strict + error The required field is enforced only at stop 4, after every document already carries it

Exact Diagnostic Fingerprints and Fast Resolution

Before backfilling, size the population that lacks the field. The $exists: false filter is exact, index-friendly if the field is indexed, and cheaper than a full $jsonSchema scan when the only change is one added field:

// How many documents are missing the new required field?
db.accounts.countDocuments({ region: { $exists: false } })
// Expected output: an integer, e.g. 20794

Once the field is added to the required array and enforcement is live, a document that slipped the backfill announces itself with a specific signature. These are the fingerprints to match:

Signature Where it appears Root cause Resolution
code: 121, failing keyword required, missing property region Driver / client after promotion to error An un-backfilled document was updated Read errInfo.details.schemaRulesNotSatisfied; re-run the backfill for that _id
Log id 51803, "Document would fail validation", keyword required mongod log while in strict + warn Backfill has not yet reached this document Let the batched job continue; the count is your burn-down
pymongo.errors.BulkWriteError with one code 121 per un-backfilled doc PyMongo backfill run under strict + error The level was tightened too early Roll back to moderate, finish the backfill, then re-promote

The schemaRulesNotSatisfied array (MongoDB 5.0+) names the exact keyword and property path, so a required/region pair is unambiguous — it is always a missing field, never a wrong value, which distinguishes this failure class from a bsonType mismatch handled by coercing bsonType mismatches in place.

Step-by-Step Playbook

The sequence stamps a computed default onto every document lacking the field, in throttled idempotent batches, and only tightens enforcement once the backlog is zero.

  1. Attach the field as required under moderate + warn. The constraint is now defined and logged but never blocks a write:

    db.runCommand({
  collMod: "accounts",
  validator: { $jsonSchema: {
    bsonType: "object",
    required: ["account_id", "region"],
    properties: {
      account_id: { bsonType: "string" },
      region: { bsonType: "string", enum: ["us", "eu", "apac"] }
    }
  }},
  validationLevel: "moderate",
  validationAction: "warn"
})
// Expected output: { ok: 1 }

  2. Run the batched, idempotent backfill. Iterate only documents where the field is absent, compute the default deterministically, and apply $set in unordered batches. The $exists: false filter makes the job resumable — a backfilled document no longer matches, so a restart never re-touches it. The loop throttles on replication lag so the oplog stays healthy:

    import logging
import time

from pymongo import MongoClient, UpdateOne
from pymongo.errors import BulkWriteError

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("backfill_region")

client = MongoClient("mongodb://primary:27017/?replicaSet=rs0")
accounts = client.production_db.accounts

MISSING = {"region": {"$exists": False}}

# Deterministic default so a retried batch computes the identical value.
REGION_BY_PREFIX = {"US": "us", "EU": "eu", "AP": "apac"}

def default_region(doc: dict) -> str:
    code = str(doc.get("account_id", ""))[:2].upper()
    return REGION_BY_PREFIX.get(code, "us")  # safe fallback

def max_secondary_lag() -> float:
    status = client.admin.command("replSetGetStatus")
    members = status["members"]
    primary = next(m for m in members if m["stateStr"] == "PRIMARY")
    secs = [m for m in members if m["stateStr"] == "SECONDARY"]
    return max(((primary["optimeDate"] - s["optimeDate"]).total_seconds()
                for s in secs), default=0.0)

def backfill(batch_size: int = 1000, max_lag: float = 10.0) -> int:
    written = 0
    ops: list = []
    cursor = accounts.find(MISSING, {"account_id": 1}).batch_size(batch_size)
    for doc in cursor:
        ops.append(UpdateOne(
            {"_id": doc["_id"], "region": {"$exists": False}},  # idempotent guard
            {"$set": {"region": default_region(doc)}},
        ))
        if len(ops) >= batch_size:
            written += _flush(ops)
            ops = []
            while max_secondary_lag() > max_lag:
                logger.warning("replication lag high; pausing")
                time.sleep(2)
    written += _flush(ops)
    logger.info("backfilled %d documents", written)
    return written

def _flush(ops: list) -> int:
    if not ops:
        return 0
    try:
        return accounts.bulk_write(ops, ordered=False).modified_count
    except BulkWriteError as bwe:
        logger.warning("partial batch: %s", bwe.details["writeErrors"][:3])
        return bwe.details.get("nModified", 0)

if __name__ == "__main__":
    backfill()

    The {"region": {"$exists": False}} clause inside each UpdateOne filter is a second idempotency guard: even if the same _id is somehow queued twice, the update is a no-op the second time because the field now exists. For a hardened version with retry and structured logging, adapt the PyMongo validation wrapper scripts pattern.

  3. Verify the count is zero. Re-run the $exists: false count. It must return 0 — this is the go/no-go gate:

    db.accounts.countDocuments({ region: { $exists: false } })
// Expected output: 0

  4. Promote to strict + error. With every document carrying the field, tighten enforcement. The flip is atomic and metadata-only:

    db.runCommand({ collMod: "accounts", validationLevel: "strict", validationAction: "error" })
// Expected output: { ok: 1 }

Failure Modes & Rollback

Each stage fails in a distinct, bounded way:

  • Code 121 spike right after step 4. A document escaped the backfill — usually because it was inserted mid-run without the field by a producer that predates the schema. If the 121 rate exceeds roughly 0.1% of writes, roll back immediately and re-run step 2; do not patch under load.
  • A non-deterministic default (step 2). If default_region depended on wall-clock time or a random value, a retried batch would write a different value than the first attempt, breaking idempotency. Keep every computed default a pure function of the document.
  • Oplog window pressure (step 2). One $set per document floods the oplog; a slow secondary can fall off the window and require a resync. The lag throttle guards this — lower batch_size or raise the pause if lag still climbs.
  • A field that has no sensible default. If you cannot compute a correct value (for example a genuinely unknown region), do not invent one — route the document to a dead-letter collection via fallback routing for invalid documents and hold enforcement until it is resolved out of band.

Rollback is a single metadata-only collMod that restores write availability without touching data; time-to-recover is the replication lag of the slowest secondary, typically seconds:

// Soft rollback — stop rejecting, keep the schema attached for telemetry.
db.runCommand({ collMod: "accounts", validationLevel: "moderate", validationAction: "warn" })

// Hard rollback — detach the validator entirely for emergency recovery.
db.runCommand({ collMod: "accounts", validator: {}, validationLevel: "off" })

The backfilled values themselves are safe to keep after a rollback — a document with a correct region is strictly more compliant than one without. Only revert the enforcement posture, then resume the backfill and re-attempt step 4.

Frequently Asked Questions

Why must the backfill run before flipping to strict + error, not after?

Because a missing-required failure fires on every write to an un-backfilled document, including updates that never touch the new field. If you enforce first, the next ordinary application update to any historical document produces a document that still lacks the field and is rejected with code 121. Backfilling first makes the collection uniformly compliant, so the promotion adds zero rejections. Under moderate level the backfill's own updates are grandfathered even while it runs.

Is a single updateMany with {$exists: false} enough instead of batches?

Functionally it would set the field on every matching document, but on a large collection a single unbounded updateMany generates a flood of oplog entries with no throttle, which can push a secondary off the oplog window and force a resync. The batched loop exists to pace the write rate against replication and to stay resumable if interrupted. For a small collection a single updateMany is fine; for a production-scale one, batch and throttle.

What if two backfill workers run concurrently on the same collection?

It is safe because every update is idempotent and guarded. Each UpdateOne filter includes region exists false, so if two workers both queue the same document, the first write sets the field and the second becomes a no-op that modifies nothing. You may double the read work, but you can never double-write a value or corrupt a document.

Related in this section

Document Transformation Pipelines for Schema MigrationWithin the broader Zero-Downtime Schema Migration Automation in MongoDB framework, a document transformation pipeline is the machinery that… Index Rebuild Ordering During Schema MigrationWhen a schema change adds a query pattern or tightens a validator, the indexes those writes and reads depend on must already exist and be… Online Migration with Dual-Write PatternsWithin the broader Zero-Downtime Schema Migration Automation in MongoDB framework, a dual-write migration is how you change the shape of a… Rollback Automation for Schema ChangesEvery collMod that tightens a $jsonSchema contract on a live collection is a one-way door unless you make it a two-way door by…