Reconciling Dual-Write Drift During Cutover

Detecting and repairing the divergence between an old and a new field before you switch the read path is the precise operational question this page answers, sitting under online migration with dual-write patterns within the broader Zero-Downtime Schema Migration Automation in MongoDB section. A dual-write window promises that every document carries the old and new representation in agreement, but reality leaks: a writer that bypasses the shared write path updates only one field, a concurrent update races, or the backfill misses a document written a millisecond after its cursor passed. The reader outcome here is a hard consistency gate — a reconciliation scan that categorises every mismatch, a targeted repair pass that converges divergence to zero, and an explicit source-of-truth decision — so the read-path switch happens on evidence, not hope. Using the section’s running example, an events collection whose ts field (epoch-milliseconds long) is migrating to an occurred_at field (BSON date), the invariant you must prove is that occurred_at equals toDate(ts) on every single document.

Operational Mechanics and Write-Path Impact

Drift is not a validator problem, so the validator will never warn you about it. During the transition the collection runs a permissive anyOf rule that accepts a document carrying either field, which means a document with a stale occurred_at and a fresh ts passes validation cleanly — it is structurally valid and semantically wrong. Reconciliation therefore runs entirely as a read-side aggregation over stored data; it touches the write path only when the repair pass issues updates, and those updates go through the same validator as any other write.

Every mismatch falls into exactly one of three categories, and the repair differs by category, so the scan must classify rather than merely count. The $expr comparison occurred_at == toDate(ts) combined with existence checks partitions the collection completely:

Category	Detection predicate	Root cause	Repair direction
Missing new field	occurred_at absent, ts present	Backfill gap, or a new write that raced the backfill cursor	Compute occurred_at from ts
Stale new field	both present, occurred_at != toDate(ts), ts is newer	A legacy writer set only ts	Recompute occurred_at from ts
Stale old field	both present, occurred_at != toDate(ts), occurred_at is newer	A new-path writer set only occurred_at, skipping the dual write	Recompute ts from occurred_at, or accept occurred_at
Consistent	occurred_at == toDate(ts)	Written through the dual-write helper	None

The source-of-truth decision is what turns those categories into a deterministic repair, and it must be explicit rather than assumed. The defensible default before cutover is that the field the current read path consumes is authoritative — production still reads ts, so ts is truth and occurred_at is recomputed from it. That single rule resolves both the “missing new field” and “stale new field” categories. The “stale old field” category is the one exception worth a conscious call: it means a newer value was written only to occurred_at, and blindly recomputing from ts would revert it. In practice you eliminate that category by finding and fixing the offending writer before repairing, so the reconciliation converges rather than fighting a live source of new drift.

Exact Diagnostic Fingerprints and Fast Resolution

The one query that matters counts divergence directly, deriving the new shape from the old inside $expr so no application code is involved:

db.events.countDocuments({
  ts: { $exists: true },
  $expr: { $ne: ["$occurred_at", { $toDate: "$ts" }] }
})
// Expected output at the gate: 0. Any positive number is your drift backlog.

For triage you want the breakdown, not just the total. A $facet runs all three category counts in a single pass and gives you a dashboard-ready object:

db.events.aggregate([
  { $facet: {
    missing_new: [
      { $match: { ts: { $exists: true }, occurred_at: { $exists: false } } },
      { $count: "n" }
    ],
    diverged: [
      { $match: { ts: { $exists: true }, occurred_at: { $exists: true },
                  $expr: { $ne: ["$occurred_at", { $toDate: "$ts" }] } } },
      { $count: "n" }
    ]
  } }
])
// Expected clean output: { missing_new: [], diverged: [] }

These are the signatures to match while the scan is running:

Signature	Where it appears	Root cause	Resolution
diverged count is non-zero and stable	Reconciliation $facet	A batch of documents written by a one-field writer	Identify the writer, then run the repair pass over the diverged set
diverged count rises between scans	Successive reconciliation runs	A live writer is still touching one field only	Stop the rogue writer before repairing — repair cannot win against active drift
$toDate / $convert error, code 16006	mongod log during the scan	Some ts values are int or double, not long	Normalise the ts bsonType first, or wrap $toDate in $convert with onError
pymongo.errors.WriteError code 121 during repair	Repair pass client	A repaired document violates the transition anyOf	Inspect errInfo.details.schemaRulesNotSatisfied; usually occurred_at built as a string, not a date

Step-by-Step Playbook

The sequence converges divergence to zero and holds it there as a gate. It is safe to re-run end to end; the repair is an idempotent recompute, so a document already consistent is left untouched.

1. Scan and classify. Run the $facet above to size and partition the drift. Record the numbers; a stable non-zero count is repairable, a rising count means a live writer is still causing drift and must be stopped first.

2. Fix the source of new drift. If the “stale old field” bucket is non-empty, a new-path writer is skipping the dual write. Find it (a changeStream filtered to updates that set occurred_at without ts will name it) and route it through the sanctioned write path before repairing, so the reconciliation converges.

3. Run the repair pass. Recompute occurred_at from the authoritative ts for every diverged or missing document. A single update pipeline does it server-side and is idempotent:

db.events.updateMany(
  { ts: { $exists: true },
    $or: [ { occurred_at: { $exists: false } },
           { $expr: { $ne: ["$occurred_at", { $toDate: "$ts" }] } } ] },
  [ { $set: { occurred_at: { $toDate: "$ts" } } } ]
)
// Expected output: matched/modified counts equal to the drift backlog.

4. Re-scan to zero. Re-run the step-1 count. It must return 0. If it does not, a writer is still active — return to step 2. This re-scan is the consistency gate.

5. Hand off to cutover. Only with a zero result do you proceed to switch the read path to occurred_at, as described in the parent workflow. The gate result is the go/no-go signal for that deploy.

For production runs, drive the scan and repair from pymongo so the gate is a callable check your pipeline can assert on:

from pymongo.collection import Collection


def divergence_report(events: Collection) -> dict:
    """Return per-category drift counts; all zero means the gate is green."""
    pipeline = [
        {"$facet": {
            "missing_new": [
                {"$match": {"ts": {"$exists": True}, "occurred_at": {"$exists": False}}},
                {"$count": "n"},
            ],
            "diverged": [
                {"$match": {"ts": {"$exists": True}, "occurred_at": {"$exists": True},
                            "$expr": {"$ne": ["$occurred_at", {"$toDate": "$ts"}]}}},
                {"$count": "n"},
            ],
        }}
    ]
    facet = next(iter(events.aggregate(pipeline)))
    return {k: (v[0]["n"] if v else 0) for k, v in facet.items()}


def repair(events: Collection) -> int:
    """Recompute occurred_at from the authoritative ts. Idempotent."""
    result = events.update_many(
        {"ts": {"$exists": True},
         "$or": [{"occurred_at": {"$exists": False}},
                 {"$expr": {"$ne": ["$occurred_at", {"$toDate": "$ts"}]}}]},
        [{"$set": {"occurred_at": {"$toDate": "$ts"}}}],
    )
    return result.modified_count


def gate(events: Collection) -> bool:
    """The go/no-go check before switching reads."""
    report = divergence_report(events)
    return report["missing_new"] == 0 and report["diverged"] == 0


if __name__ == "__main__":
    # In production, batch the repair by _id range and watch the oplog window.
    from pymongo import MongoClient
    coll = MongoClient("mongodb://primary:27017").production_db.events
    print("before:", divergence_report(coll))
    print("repaired:", repair(coll))
    assert gate(coll), "drift remains — do not switch reads"
    print("gate green: safe to switch reads")

Failure Modes & Rollback

Reconciliation is itself reversible because the repair pass only ever rewrites occurred_at, the field production is not yet reading. Each failure has a bounded recovery:

• Divergence rises instead of falling. A live writer is out-running the repair. Do not keep repairing — you will burn oplog and never converge. Stop the writer (step 2), then repair once. Time-to-recover is however long it takes to redeploy the offending service.
• $toDate errors mid-scan (code 16006). Heterogeneous ts types. Repair is halted, but nothing is corrupted. Normalise the field first with a guarded $convert ({ $convert: { input: "$ts", to: "date", onError: null } }) so bad values surface as null rather than aborting the pass, then handle them explicitly.
• Repair pass floods the oplog. An unbounded update_many over millions of diverged documents can push a secondary off its window. Batch by _id range and throttle between batches; the idempotent recompute means an interrupted run resumes safely from where a re-scan says drift remains.
• A repaired value is wrong because ts was not authoritative. This only happens if you skipped step 2 and repaired over a genuine new-only write, reverting it. Because production still reads ts, no user saw the reverted value; recover by re-deriving from whichever field your source-of-truth decision actually names, then re-run the gate.

There is no destructive rollback to fear here: reconciliation never touches ts, never drops a field, and never tightens the validator. If in doubt, halt — the dual-write window stays open, production keeps reading ts, and you resume the scan later with no data lost.

Frequently Asked Questions

Related

• Online Migration with Dual-Write Patterns — the parent workflow that opens the dual-write window this reconciliation gates before its read switch.
• Zero-Downtime Schema Migration Automation in MongoDB — the section reference for the transformation, cutover, and rollback machinery around this gate.
• Document Transformation Pipelines for Schema Migration — the batched backfill whose gaps the “missing new field” category catches.
• Backward-Compatible Schema Evolution Patterns — designing the target contract so the transition validator tolerates both shapes while you reconcile.