Rollback Automation for Schema Changes

Every collMod that tightens a $jsonSchema contract on a live collection is a one-way door unless you make it a two-way door by construction, and that is the discipline this workflow builds inside the broader Zero-Downtime Schema Migration Automation in MongoDB framework. The organizing idea is simple and non-negotiable: before any command mutates a validator, capture the collection’s exact pre-change enforcement state as a versioned artifact, so that reverting is the replay of a stored fact rather than a reconstruction from memory during an incident. Because a validator lives entirely in collection metadata and collMod never rewrites documents, the reversal of a schema change is itself a metadata operation whose time-to-recover equals your replication lag — seconds, not the hours a data rewrite would demand. This page delivers the full reversible-change subsystem: a capture-then-apply-then-restore wrapper, a one-command restore, the distinction between a soft rollback that merely stops rejecting and a hard rollback that detaches the rule entirely, and the diagnostics that tell you which to reach for. Two focused procedures sit beneath it — the capture mechanics in snapshotting validator state before collMod, and the executable revert generation in automating collMod rollback scripts.

Architectural Context & Enforcement Boundaries

A validator change has an asymmetry that dictates the entire rollback design: the data side of a migration is expensive and forward-biased, while the enforcement side is cheap and instantly reversible. Reshaping millions of documents to fit a tighter contract takes hours and is only undone by another equally expensive pass; attaching, tightening, or detaching the $jsonSchema that governs them is a single oplog entry the primary applies immediately. Rollback automation lives entirely on the cheap side of that line. It never tries to un-write data — it restores the contract the data is judged against, which is exactly the lever that re-opens a stalled write path fastest.

That boundary is why reversibility is captured as an artifact rather than assumed. The three fields that fully define enforcement — the validator document, the validationLevel, and the validationAction — are read straight from the collection catalog and serialized before the change command runs. The captured triple is the restore target: applying it back through collMod returns the collection to its pre-change posture byte-for-byte. Everything downstream, from the soft/hard rollback choice to the CI/CD one-click revert, is a consumer of that stored triple.

The strict vs moderate validation levels guide explains the two dials the snapshot preserves; rollback automation is what makes changing them safe, because every change now carries its own antidote. It pairs naturally with treating the schema as a versioned object under schema versioning strategies for NoSQL: the snapshot artifact is the previous version, and the change you are applying is the next one, so a rollback is nothing more exotic than pinning the collection back to the prior version integer.

Prerequisites & Operational Requirements

Confirm each of the following before you wire reversible changes into a deployment pipeline.

• MongoDB version: 5.0 or later. The structured errInfo.details.schemaRulesNotSatisfied object that the monitor step reads is a 5.0+ feature; on 4.x a rejection reason is an opaque string that cannot be routed automatically.
• Topology: a replica set. Rollback’s time-to-recover guarantee is the oplog replication lag, so a standalone deployment has no lag to measure and none of the failover safety the restore path assumes.
• Driver: PyMongo 4.x, pinned (pip install "pymongo>=4.6,<5"). Pin it in the automation image so OperationFailure.code, errInfo, and command-keyword handling stay stable across builds.
• Permissions: applying or restoring a validator needs the collMod action from the dbAdmin role on the target database; capturing the snapshot needs the listCollections privilege. Keep the automation principal least-privilege — a reversible change must never require clusterAdmin.
• Snapshot store: a durable location for the captured triples, versioned and timestamped. A dedicated _schema_snapshots collection in the same database is the simplest choice and keeps the artifact co-located with the data it governs; a Git-tracked file works equally well for pipeline-driven changes.
• Baseline audit: before promoting to strict + error, know the non-compliant document count via $jsonSchema used as a query operator: db.coll.countDocuments({ $nor: [{ $jsonSchema: <schema> }] }). A rollback restores the contract instantly, but it cannot un-reject a write that already failed, so drift must be zero before enforcement flips.

Idempotent Deployment / Implementation Workflow

Making a change reversible is a fixed five-step sequence. Each step is independently verifiable and safe to re-run.

1. Capture the pre-change triple. Read the live validator, level, and action from the catalog and persist them with a timestamp and the deploying commit hash. This is the artifact every later step depends on:

   // mongosh — capture BEFORE mutating anything.
   const before = db.getCollectionInfos({ name: "subscriptions" })[0].options
   db._schema_snapshots.insertOne({
     coll: "subscriptions",
     captured_at: new Date(),
     validator: before.validator || {},
     validationLevel: before.validationLevel || "off",
     validationAction: before.validationAction || "error"
   })
   // Expected output: an acknowledged insert with the snapshot _id

2. Apply the change in observe mode. Attach the tightened schema as strict + warn, so every write is checked but nothing is rejected while you watch:

   db.runCommand({
     collMod: "subscriptions",
     validator: { $jsonSchema: {
       bsonType: "object",
       required: ["account_id", "plan", "renews_at"],
       properties: {
         account_id: { bsonType: "string" },
         plan: { bsonType: "string", enum: ["free", "pro", "enterprise"] },
         renews_at: { bsonType: "date" }
       }
     }},
     validationLevel: "strict",
     validationAction: "warn"
   })
   // Expected output: { ok: 1 }

3. Monitor the observation window. Tail warn-mode diagnostics and confirm the flagged-write rate trends to zero as the backfill converges. This window is where a bad rule is caught before it can reject a single production write.

4. Promote or restore. If the signal is clean, promote with db.runCommand({ collMod: "subscriptions", validationAction: "error" }). If it is not, restore the captured triple — a single collMod returns the collection to exactly its pre-change posture.

5. Record the outcome. Stamp the snapshot artifact with the decision (promoted or rolled back) and the commit, so the change is an auditable event rather than a silent metadata mutation.

Production-Ready Automation Implementation

The module below is the reference “safe collMod” wrapper. It snapshots the current enforcement state to a durable store, applies the requested change idempotently (skipping the command entirely when the live state already matches, so repeated pipeline runs never take a redundant lock), and exposes a restore that reapplies the captured triple. Transient replica-set state conflicts retry with bounded backoff; authorization and parse failures never retry.

import logging
import random
import time
from datetime import datetime, timezone
from typing import Any, Dict, Optional

from pymongo.database import Database
from pymongo.errors import OperationFailure, PyMongoError

logger = logging.getLogger("reversible-collmod")

# collMod / auth failures that are deterministic and must never be retried.
FATAL_CODES = {13, 9, 26}  # Unauthorized, FailedToParse, NamespaceNotFound


class ReversibleCollMod:
    """Apply a validator change with a captured, restorable pre-change snapshot."""

    def __init__(self, db: Database, snapshot_coll: str = "_schema_snapshots",
                 max_retries: int = 3, base_delay: float = 0.5):
        self.db = db
        self.snapshots = db[snapshot_coll]
        self.max_retries = max_retries
        self.base_delay = base_delay

    def _live_options(self, coll: str) -> Dict[str, Any]:
        """Read the authoritative validator/level/action from the collection catalog."""
        info = self.db.command("listCollections", filter={"name": coll})
        batch = info["cursor"]["firstBatch"]
        return batch[0].get("options", {}) if batch else {}

    def snapshot(self, coll: str, commit: str = "unknown") -> Dict[str, Any]:
        """Persist the current enforcement triple; returns the stored artifact."""
        opts = self._live_options(coll)
        artifact = {
            "coll": coll,
            "captured_at": datetime.now(timezone.utc),
            "commit": commit,
            "validator": opts.get("validator", {}),
            "validationLevel": opts.get("validationLevel", "off"),
            "validationAction": opts.get("validationAction", "error"),
        }
        artifact["_id"] = self.snapshots.insert_one(artifact).inserted_id
        logger.info("snapshotted %s -> %s", coll, artifact["_id"])
        return artifact

    def _collmod(self, coll: str, **fields: Any) -> None:
        """Run collMod with bounded exponential backoff on transient failures."""
        for attempt in range(1, self.max_retries + 1):
            try:
                self.db.command("collMod", coll, **fields)
                logger.info("collMod on %s applied (attempt %d)", coll, attempt)
                return
            except OperationFailure as exc:
                if exc.code in FATAL_CODES or attempt == self.max_retries:
                    logger.error("collMod on %s failed (code %s): %s", coll, exc.code, exc)
                    raise
                delay = self.base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.2)
                logger.warning("transient collMod failure %d/%d; retrying in %.2fs",
                               attempt, self.max_retries, delay)
                time.sleep(delay)
            except PyMongoError as exc:
                logger.error("driver error during collMod on %s: %s", coll, exc)
                raise

    def apply_change(self, coll: str, validator: Dict[str, Any],
                     level: str = "strict", action: str = "warn",
                     commit: str = "unknown") -> Optional[Dict[str, Any]]:
        """Snapshot, then apply the target validator idempotently. Returns the snapshot, or None if unchanged."""
        live = self._live_options(coll)
        if (live.get("validator") == validator
                and live.get("validationLevel") == level
                and live.get("validationAction") == action):
            logger.info("%s already at target state; no snapshot, no collMod", coll)
            return None
        artifact = self.snapshot(coll, commit)          # capture BEFORE mutating
        self._collmod(coll, validator=validator,
                      validationLevel=level, validationAction=action)
        return artifact

    def restore(self, artifact: Dict[str, Any]) -> None:
        """Reapply a captured triple in one metadata-only command — restores the CONTRACT, not the data."""
        self._collmod(
            artifact["coll"],
            validator=artifact["validator"],
            validationLevel=artifact["validationLevel"],
            validationAction=artifact["validationAction"],
        )
        logger.info("restored %s to snapshot %s", artifact["coll"], artifact.get("_id"))

The contract of this class is what makes it safe: apply_change never mutates a validator without first writing the artifact that undoes it, so there is no window in which a change exists without its own rollback. The restore path is the mechanical heart of automating collMod rollback scripts, which turns a stored artifact into an executable one-click revert that any on-call engineer can run without reconstructing prior state.

Diagnostic Fingerprints & Fast Resolution

A misfiring change surfaces through a compact set of exact signatures. Match on these to choose between a soft and a hard rollback without guessing.

Signature	Root cause	Resolution
WriteError code 121 rate spikes at the moment validationAction flips to error	Drift was non-zero at promotion — the observation window was cut short.	Soft rollback to moderate + warn; re-run the drift count, remediate, retry.
pymongo.errors.OperationFailure code 9 (FailedToParse) on the apply collMod	Target schema uses a keyword the Draft 4 engine rejects (if/then, $defs, format).	Fix the schema on a throwaway collection; no rollback needed — nothing was applied.
OperationFailure code 13 (Unauthorized) on apply or restore	Automation principal lacks the collMod action.	Grant dbAdmin on the database; the snapshot is unaffected and the retry is safe.
Restore leaves the collection stricter than expected	The snapshot captured a state that was already mid-migration.	Confirm the artifact’s captured_at predates the migration; roll forward to a known-good version instead.
db.getCollectionInfos shows a validator you never applied	A concurrent collMod from another actor raced yours.	Re-snapshot to capture the true current state before any further change.

Read the live enforcement state directly from the catalog — this, not what a deploy script intended, is ground truth:

db.getCollectionInfos({ name: "subscriptions" })[0].options
// => { validator: {...}, validationLevel: "strict", validationAction: "warn" }

To pull the structured reason a specific write was rejected under enforcement, catch the driver exception and read errInfo:

from pymongo.errors import WriteError

try:
    db.subscriptions.update_one({"_id": doc_id}, {"$set": {"note": "x"}})
except WriteError as exc:
    rules = exc.details["errInfo"]["details"]["schemaRulesNotSatisfied"]
    print("failed keywords:", rules)   # names the exact operator and path that failed

Edge Cases, Gotchas & Known Limitations

• Rollback restores the contract, never the data. This is the single most important limitation. Once validationAction: "error" was active, any document that violated the schema was already rejected and never persisted; conversely, documents written under a new, looser contract are real writes that a rollback to the old rule cannot un-write. A validator rollback re-opens the write path — it does not reverse the durable effects of writes that already committed. Data reversal, if needed, is a separate forward migration.
• A stale snapshot restores the wrong world. If two changes were applied but only the first was snapshotted, restoring reverts to a state two hops back. Every change must carry its own fresh capture; never reuse an older artifact as a shortcut.
• collMod takes a brief exclusive lock. Both the apply and the restore acquire a short collection lock and must wait for in-flight writes to drain. On a very hot collection this is a momentary write stall, not an outage — but schedule enforcement flips accordingly.
• An empty validator: {} is the hard-rollback sentinel. Restoring an artifact whose captured validator was {} correctly detaches the rule. Guard against a snapshot that recorded {} by accident (for example, taken against a collection that never had a validator) being mistaken for an intentional hard rollback.
• validationLevel: "off" and validationAction: "warn" differ. A soft rollback to warn still checks every write and logs violations; off stops checking entirely. Restoring the wrong one silently changes how much telemetry you keep during recovery.
• Aggregation $merge/$out bypass the validator on the destination at any level, so a rollback window is exactly when non-compliant data can slip back in through a pipeline that rewrites documents. Freeze such jobs during a migration.

Verification & Rollback Procedures

Confirm a change is enforcing before you rely on it. The only proof that strict + error is genuinely active is a positive test — a known-bad write must be rejected:

from pymongo.errors import WriteError

# After promotion, a schema-violating write MUST raise code 121.
try:
    db.subscriptions.insert_one({"account_id": 99, "plan": "gold"})  # wrong bsonType + bad enum
    raise AssertionError("enforcement is NOT active")
except WriteError as exc:
    assert exc.code == 121
    print("strict/error confirmed: malformed insert rejected with 121.")

Rollback is a single atomic, metadata-only command that restores write availability within milliseconds of reaching the primary and loses no data. Choose the depth of the revert to the incident:

// Soft rollback — keep the schema attached, stop rejecting. Preserves telemetry.
db.runCommand({ collMod: "subscriptions", validationLevel: "moderate", validationAction: "warn" })

// Hard rollback — detach the validator entirely for emergency recovery.
db.runCommand({ collMod: "subscriptions", validator: {}, validationLevel: "off" })

Both take effect on the primary immediately and propagate through the oplog to secondaries, so the time-to-recover is the replication lag of your slowest secondary — typically seconds. Treat the soft rollback as your default incident response: it re-opens the write path while keeping the failure signal you need to finish remediation. The programmatic equivalent is ReversibleCollMod.restore(artifact), which reapplies the captured triple exactly and is the primitive the automating collMod rollback scripts workflow packages for one-click use.

Frequently Asked Questions

Related

• Zero-Downtime Schema Migration Automation in MongoDB — the parent section defining the coupled data-and-enforcement migration this rollback subsystem reverses.
• Snapshotting Validator State Before collMod — the exact capture mechanics that produce the restorable artifact every rollback depends on.
• Automating collMod Rollback Scripts — turning a stored snapshot into an idempotent, one-click executable revert wired into CI/CD.
• Document Transformation Pipelines for Schema Migration — the forward data-reshaping half whose expense is exactly why the enforcement side must stay cheaply reversible.
• Strict vs Moderate Validation Levels — the two dials a snapshot preserves and a soft rollback tunes back down.
• Schema Versioning Strategies for NoSQL — treating each captured snapshot as the previous schema version so a rollback is a version pin.