Online Migration with Dual-Write Patterns

Within the broader Zero-Downtime Schema Migration Automation in MongoDB framework, a dual-write migration is how you change the shape of a field on a live collection without a single moment where reads see the wrong thing. The technique is the parallel-change (expand/contract) pattern applied to document structure: for a transition window every write emits both the old representation and the new one, historical documents are backfilled into the new shape in the background, and only once the two representations are provably identical does the read path switch atomically to the new field and the old field get decommissioned. This guide is a complete implementation workflow for platform engineers reshaping a field with zero read downtime. It pins down how the $jsonSchema validator must be loosened to accept both shapes during the window and then tightened afterward, gives you an idempotent deployment sequence, a production-ready pymongo dual-write helper, the exact diagnostic signatures that a diverging pair of fields produces, and the verification and rollback commands to run the whole change safely. Two companion workflows do the heavy lifting on either side of the cutover: the batched backfill that populates the new field for old documents is a document transformation pipeline for schema migration, and detecting and repairing any divergence between the two fields before you flip reads is covered in depth in reconciling dual-write drift during cutover.

Architectural Context & Enforcement Boundaries

The running example throughout this guide is an events collection whose timestamp is stored as an epoch-milliseconds long in a field named ts, and which is being migrated to a BSON date in a field named occurred_at. This is a common and painful migration: a long epoch cannot drive a TTL index, cannot be bucketed by $dateTrunc, and forces every consumer to convert on read. The transform between the two is deterministic in both directions — occurred_at = new Date(ts) and ts = occurred_at.getTime() — which is exactly the property a dual-write migration needs, because the reconciliation and backfill logic must be able to derive one representation from the other with no ambiguity.

A dual-write migration lives on the write path, and the collection validator is the write-path gate it has to negotiate. Because MongoDB evaluates $jsonSchema synchronously on every insert, update, replace, and findAndModify, the validator cannot be left as-is while you add a second field — a rule that still says required: ["ts"] and pins nothing about occurred_at will either reject the new field or, worse, let it be written with the wrong bsonType. The migration therefore proceeds through three phases, and the validator has a distinct contract in each:

• Expand. Deploy a dual-write layer so every new write sets both ts (a long) and occurred_at (a date). The validator is loosened to accept a document that carries either field, using an anyOf that requires at least one of the two, while still type-pinning both. Reads continue to use ts; nothing downstream has changed yet.
• Migrate. Backfill occurred_at onto every historical document from its existing ts, in throttled batches, until every document carries both fields. The permissive anyOf validator stays in place so backfill updates to not-yet-migrated documents never trip a WriteError code 121.
• Contract. Once the two fields are reconciled and reads have been switched to occurred_at, stop writing ts, drop it from every document, and tighten the validator to require occurred_at as a date and forbid the resurrected old field.

The read-path switch is the pivotal, irreversible-feeling moment, and it must be gated on evidence rather than a calendar. That evidence is a zero-divergence proof between ts and occurred_at across the whole collection; producing that proof, and repairing the documents that break it, is the entire subject of the child workflow. Everything before the switch is reversible by simply continuing to read the old field.

Prerequisites & Operational Requirements

Confirm each of the following before you deploy the expand phase against a live collection.

• MongoDB version: 5.0 or later. The anyOf combinator the transition validator relies on is Draft 4 and works everywhere, but the structured errInfo.details.schemaRulesNotSatisfied object you will lean on when a dual write is rejected requires 5.0+. On 4.x you get an opaque failure string and lose most of the diagnostic value.
• Topology: a replica set. Dual writing roughly doubles the write amplification during the window, and the backfill adds more; a standalone gives you no failover margin and no oplog window to watch. Confirm your oplog holds several hours so a paused backfill can resume without a secondary resync.
• Driver: pymongo 4.x, pinned (pip install "pymongo>=4.6,<5") so bulk_write, UpdateOne, and OperationFailure.code semantics stay stable across CI images.
• Permissions: the validator changes at expand and contract require the collMod action from the dbAdmin role; the dual-write helper and backfill need ordinary readWrite. Keep them as separate principals so a runaway backfill can never alter a validator.
• A deterministic, total transform. Dual writing only works when the new representation is a pure function of the old one (and ideally vice versa). Confirm occurred_at = new Date(ts) holds for every value in range, including any sentinel or zero timestamps, before you trust the backfill. If the transform is lossy or ambiguous, the pattern does not apply and you need an application-mediated migration instead.
• A backward-compatible target contract. The shape you are converging on should be a considered evolution, not an ad-hoc rename. Design it against backward-compatible schema evolution patterns so the transition validator is a deliberate superset of the old and new rules rather than a temporary hole punched in your enforcement.

Idempotent Deployment / Implementation Workflow

Each step below is independently verifiable and safe to re-run; re-running never double-writes or corrupts a document because every operation is expressed as a deterministic upsert of a derived value.

1. Loosen the validator to accept both shapes (expand). Replace the old single-field rule with an anyOf that requires at least one of the two time fields while type-pinning each. A document with only ts, only occurred_at, or both now passes; a document with neither, or with a mistyped field, is still rejected.

   db.runCommand({
     collMod: "events",
     validator: { $jsonSchema: {
       bsonType: "object",
       required: ["event_id"],
       properties: {
         event_id:    { bsonType: "string" },
         ts:          { bsonType: "long" },   // legacy epoch millis
         occurred_at: { bsonType: "date" }    // new representation
       },
       anyOf: [
         { required: ["ts"] },
         { required: ["occurred_at"] }
       ]
     }},
     validationLevel: "moderate",
     validationAction: "error"
   })
   // Expected output: { ok: 1 }

2. Deploy the dual-write layer. Ship the application change (or the wrapper in the next section) so every write path that used to set only ts now also sets occurred_at = new Date(ts). From this instant forward, all new documents carry both fields in agreement. Reads are untouched and still consume ts.

3. Backfill historical documents. Run a throttled batch job that computes occurred_at from ts for every document that lacks it. Because the transform is a server-side expression, an update pipeline does it without shipping documents to the client:

   db.events.updateMany(
     { occurred_at: { $exists: false }, ts: { $exists: true } },
     [ { $set: { occurred_at: { $toDate: "$ts" } } } ]
   )
   // Run in batches in production; see the transformation-pipeline workflow.

   In production drive this in bounded batches with oplog-window awareness rather than one unbounded updateMany; the batching, throttling, and resume-token discipline are the subject of document transformation pipelines for schema migration.

4. Reconcile and gate. Before touching the read path, prove that ts and occurred_at agree on every document and that no document is missing the new field. This consistency gate — the divergence scan, the repair pass, and the go/no-go decision — is reconciling dual-write drift during cutover. Do not proceed until the divergence count is 0.

5. Switch the read path. Deploy the application change that reads occurred_at instead of ts. This is a code deploy, not a database change; it is atomic per instance and rolls forward across your fleet. Dual writing continues, so a rollback to reading ts is still safe at this point.

6. Stop writing the old field and drop it (contract). Once every reader is on occurred_at, remove ts from the write path, then unset it from stored documents:

   db.events.updateMany(
     { ts: { $exists: true } },
     { $unset: { ts: "" } }
   )

7. Tighten the validator. Replace the permissive anyOf with the final contract that requires occurred_at and no longer admits ts:

   db.runCommand({
     collMod: "events",
     validator: { $jsonSchema: {
       bsonType: "object",
       required: ["event_id", "occurred_at"],
       properties: {
         event_id:    { bsonType: "string" },
         occurred_at: { bsonType: "date" }
       },
       additionalProperties: false
     }},
     validationLevel: "strict",
     validationAction: "error"
   })
   // additionalProperties:false now rejects any lingering ts field.
   // Expected output: { ok: 1 }

Production-Ready Automation Implementation

The dual-write layer is the component most worth hardening, because it is the one that runs on the hot path for the entire window. The helper below wraps writes to events so callers supply only the new-shape value (occurred_at as a datetime) and the wrapper derives and co-writes the legacy ts. It is deliberately symmetric: a single _project function is the one place the two representations are related, so there is exactly one definition of “in agreement” shared by writes, backfill, and reconciliation. It classifies a code 121 as a caller error (never retried — the document is deterministically invalid) and retries only transient replica-set state failures with bounded backoff.

import logging
import random
import time
from datetime import datetime, timezone
from typing import Any, Dict

from pymongo.collection import Collection
from pymongo.errors import OperationFailure, WriteError

logger = logging.getLogger(__name__)

# Transient states worth retrying; 121 (validation) and 13 (auth) are not here.
RETRYABLE_CODES = {11602, 91, 189, 10107}  # interruptedDueToReplStateChange, etc.


def _project(occurred_at: datetime) -> Dict[str, Any]:
    """The single source of truth relating the two representations.

    Every write, the backfill, and the reconciliation scan derive the
    legacy `ts` from `occurred_at` through this one function, so the
    definition of 'the two fields agree' cannot drift between them.
    """
    if occurred_at.tzinfo is None:
        occurred_at = occurred_at.replace(tzinfo=timezone.utc)
    ts_millis = int(occurred_at.timestamp() * 1000)
    return {"occurred_at": occurred_at, "ts": ts_millis}


class DualWriteEvents:
    """Write `events` documents in both the new and legacy time shapes."""

    def __init__(self, collection: Collection, max_retries: int = 3, base_delay: float = 0.4):
        self.collection = collection
        self.max_retries = max_retries
        self.base_delay = base_delay

    def insert_event(self, event_id: str, occurred_at: datetime, **fields: Any) -> str:
        """Insert one event, co-writing occurred_at (date) and ts (long)."""
        doc = {"event_id": event_id, **fields, **_project(occurred_at)}
        return str(self._with_retry(lambda: self.collection.insert_one(doc)).inserted_id)

    def set_time(self, event_id: str, occurred_at: datetime) -> int:
        """Update an event's time, keeping both representations in lockstep."""
        res = self._with_retry(
            lambda: self.collection.update_one(
                {"event_id": event_id}, {"$set": _project(occurred_at)}
            )
        )
        return res.modified_count

    def _with_retry(self, op):
        for attempt in range(1, self.max_retries + 1):
            try:
                return op()
            except WriteError as exc:
                # 121 means the document is deterministically invalid — never retry.
                if exc.code == 121:
                    rules = (exc.details or {}).get("errInfo", {}).get("details", {})
                    logger.error("Validation rejected dual write: %s", rules)
                raise
            except OperationFailure as exc:
                if exc.code not in RETRYABLE_CODES or attempt == self.max_retries:
                    logger.error("Non-retryable/exhausted dual-write failure: %s", exc)
                    raise
                delay = self.base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.2)
                logger.warning("Transient write failure %d/%d; retry in %.2fs",
                               attempt, self.max_retries, delay)
                time.sleep(delay)

Routing every write through one _project function is what makes the later reconciliation tractable: drift can only arise from writers that bypass the helper or from a race, never from two independent formulas quietly disagreeing. Wire this class in as the only sanctioned write path to events for the duration of the window, and instrument set_time and insert_event with a counter so you can prove, from application metrics, that no unmanaged writer is still touching the collection.

Diagnostic Fingerprints & Fast Resolution

A dual-write window has one failure mode above all others — drift, where a document’s ts and occurred_at no longer encode the same instant — plus a handful of validator-shape mistakes. Match on these signatures to route the incident immediately.

Signature	Root cause	Resolution
Reconciliation scan reports occurred_at != toDate(ts) on a subset of documents	A writer updated only one field — bypassed the dual-write helper, or a partial $set	Repair from the chosen source of truth; see the reconciliation workflow before switching reads
occurred_at: { $exists: false } count stays above zero after backfill	Backfill filter missed docs (e.g. ts stored as int, not long), or new writes raced ahead of the batch cursor	Widen the backfill bsonType match; re-run — the update pipeline is idempotent
pymongo.errors.WriteError code 121 during the expand phase	Transition validator rejected the new field — usually occurred_at written as a string, not a date	Inspect errInfo.details.schemaRulesNotSatisfied; fix the driver-side type (pass a datetime, not an ISO string)
OperationFailure code 9 (FailedToParse) on the expand collMod	anyOf placed inside a property instead of at the schema root, or a Draft 2019 keyword used	Move anyOf to the object root next to required; use only Draft 4 combinators
121 spike the instant you tighten the validator (step 7)	A document still carried ts, or occurred_at was missing, when additionalProperties:false went live	Roll the validator back to the anyOf, re-run the $unset and backfill, verify, retry

The single most useful ground-truth query during the window counts documents whose two representations disagree, using $toDate to derive the new shape from the old and comparing inside $expr:

db.events.countDocuments({
  ts: { $exists: true },
  $expr: { $ne: ["$occurred_at", { $toDate: "$ts" }] }
})
// Expected output during a clean window: 0
// A non-zero result is your exact drift backlog before the read switch.

To pull the structured reason a specific dual write was rejected under the transition validator, catch the driver exception and read errInfo:

from pymongo.errors import WriteError

try:
    events.insert_one({"event_id": "e-1", "occurred_at": "2026-07-17T00:00:00Z"})  # string, not date
except WriteError as exc:
    print(exc.details["errInfo"]["details"]["schemaRulesNotSatisfied"])
    # => reports bsonType 'date' expected for occurred_at

Edge Cases, Gotchas & Known Limitations

• An unmanaged writer silently defeats the whole pattern. The guarantee that ts and occurred_at agree holds only if every writer goes through the dual-write path. A forgotten cron job, an admin mongosh $set, or a second service still on the old code will write one field and not the other, producing drift that surfaces only at reconciliation. Enumerate every writer before you trust the window.
• Read-modify-write races can desync the pair. If two concurrent updates each set the time — one via the helper, one via a stray $set on ts alone — the document can end with a fresh ts and a stale occurred_at. Prefer updating both fields in a single atomic $set (as the helper does); never let application code touch one representation in isolation once the window is open.
• anyOf weakens enforcement for the whole window. During transition the validator legitimately accepts a document with only the old field, so a bug that stops writing occurred_at will not be caught by the validator — it passes the anyOf. The reconciliation scan, not the validator, is your safety net until you tighten. Time-box the window accordingly.
• $toDate on a non-long ts throws. If some documents stored the epoch as an int or a double, the backfill and the comparison query error out on those. Normalise the bsonType of ts first, or guard the expression with $convert and an onError.
• moderate during expand is deliberate, strict at contract is deliberate. The expand validator uses moderate so an in-place backfill update of a document that is momentarily non-compliant is grandfathered rather than blocked; the final contract uses strict because by then there is no legacy shape left to grandfather. Mixing these up either blocks the backfill or leaves the contract soft.
• Dropping ts and tightening are two commands, and order matters. Unset the field from documents before you deploy additionalProperties: false. Tighten first and every not-yet-cleaned document becomes a latent 121 on its next update.
• $merge/$out bypass the validator. If any pipeline rewrites events via $merge, it can reintroduce a bare ts or an occurred_at-less document after you have tightened, with no 121 raised. Audit aggregation writers as part of the contract phase.

Verification & Rollback Procedures

Verification is a positive proof at each boundary, not a single check at the end. Before switching reads, the divergence count above must be 0 and the missing-occurred_at count must be 0. After tightening, a known-bad write must be rejected:

from pymongo.errors import WriteError

# After contract: a document carrying the decommissioned ts field MUST be rejected.
try:
    events.insert_one({"event_id": "e-legacy", "occurred_at": datetime.now(timezone.utc),
                       "ts": 1_752_710_400_000})
    raise AssertionError("contract validator is NOT enforcing additionalProperties")
except WriteError as exc:
    assert exc.code == 121
    print("contract confirmed: resurrected ts field rejected with 121.")

Rollback is defined by which phase you are in, and the pattern’s whole appeal is that the reversible region is large. Before the read switch, recovery is trivial — reads still consume ts, so you simply stop the migration: halt the backfill, and optionally revert the validator to the original single-field rule. No data is lost because the old field was never touched.

// Pre-cutover rollback — restore the original contract; ts is still authoritative.
db.runCommand({
  collMod: "events",
  validator: { $jsonSchema: {
    bsonType: "object",
    required: ["event_id", "ts"],
    properties: { event_id: { bsonType: "string" }, ts: { bsonType: "long" } }
  }},
  validationLevel: "moderate",
  validationAction: "error"
})

If the read switch itself misbehaves, roll the application back to reading ts — the field is still present and still being dual-written, so the fallback is intact and time-to-recover is one deploy. The only genuinely hard-to-reverse step is dropping ts at contract; that is why the drop is the last action, taken only after reads have soaked on occurred_at. Keep a snapshot or a short oplog-based point-in-time window covering the $unset so a late-discovered reader can be recovered. Treat “stop new writes to the old field, keep the old field readable” as the default incident stance throughout the window — it is always available until the very last command.

Frequently Asked Questions

Related

• Zero-Downtime Schema Migration Automation in MongoDB — the parent reference defining the transformation, indexing, cutover, and rollback machinery this dual-write workflow plugs into.
• Reconciling Dual-Write Drift During Cutover — the divergence scan, repair pass, and consistency gate that must pass before you switch the read path.
• Document Transformation Pipelines for Schema Migration — the batched, throttled backfill that populates the new field for historical documents.
• Backward-Compatible Schema Evolution Patterns — designing the target contract so the transition validator is a deliberate superset rather than a temporary hole.