Replaying Quarantined Documents After Schema Fixes

Once a validator has been widened or a producing service’s bug is fixed, the payloads sitting in quarantine are no longer permanently invalid — they can be re-driven back into the primary collection, and doing that safely is the operational question this page answers under fallback routing for invalid documents, within the broader MongoDB JSON Schema Validation Architecture. You will finish with an idempotent replay job that selects pending dead-letter envelopes, re-validates each one client-side against the current schema, re-inserts the ones that now pass, and marks each record replayed or leaves it pending with fresh failure detail. The reason replay needs its own discipline is that the original rejections were deterministic code 121 failures — nothing about re-submitting an unchanged payload against an unchanged schema will ever succeed, so a replay is only meaningful after the contract or the data has genuinely changed, and it must never double-insert a document that a partial earlier run already recovered.

Operational Mechanics and Write-Path Impact

A replay reverses the capture flow: it reads envelopes out of orders_deadletter, reconstructs the candidate document, and attempts to land it in the strictly-validated orders collection. Two properties make this safe to run, re-run, and interrupt. The first is client-side pre-validation: before spending a round-trip on a write the server will only reject, the job checks the reconstructed document against the current schema with the Python jsonschema library, so a payload that still fails is filtered out locally and its envelope updated in place rather than generating another server rejection. The second is idempotent keying on the original _id: the recovered document keeps the identifier it was born with, and the re-insert is expressed as an upsert on that _id, so a document that a crashed earlier pass already recovered is never written twice.

The write itself is an update_one with upsert=True and $setOnInsert, not a plain insert_one. That distinction is the whole idempotency guarantee: if the _id already exists in orders — because a prior replay, or a corrected live producer, already wrote it — the upsert matches the existing document and $setOnInsert applies nothing, so the operation is a harmless no-op. If the _id is absent, the upsert inserts the full recovered payload. Either way the envelope is then marked replayed. This closes the double-insert hole that a naive insert-and-catch loop leaves open whenever a batch is retried after a partial failure.

Replay property	Mechanism	What it prevents
Pre-validation	jsonschema.Draft4Validator on the reconstructed doc	Wasted round-trips and fresh code 121 on still-invalid payloads
Idempotent insert	update_one({_id}, {$setOnInsert: doc}, upsert=True)	Double-inserting a document a prior pass recovered
Lifecycle marking	status moves pending → replayed	Re-processing already-recovered envelopes on the next run
Ordered vs unordered	ordered=False on any bulk stage	One still-failing payload halting the whole batch

Ordering matters when a replay is expressed as a bulk operation. An ordered=True bulk stops at the first error, so a single lingering non-compliant payload would strand every later document in the batch; ordered=False lets MongoDB attempt every operation and report the failures individually, which is the only sane choice when the batch is a mixed bag of now-valid and still-invalid records. Because the write path enforces the same $jsonSchema described in strict vs moderate validation levels, a replay never weakens the contract — it simply re-tests old data against the current one.

Exact Diagnostic Fingerprints and Fast Resolution

A replay run produces three outcomes per envelope, and each has a clear signature you match to decide the next action. A payload that passes pre-validation and upserts cleanly is a success. A payload that still fails pre-validation means the schema fix did not cover its failure mode — its envelope stays pending with a refreshed failing_keywords. A payload that passes pre-validation but is nonetheless rejected by the server points to drift between the client schema and the registered validator, the one situation a replay must surface loudly rather than swallow.

Signature	Meaning	Resolution
Pre-validation passes, upsert matched_count == 1	_id already present — a prior pass or live producer recovered it	Mark replayed; this is a benign no-op, not an error
Pre-validation passes, upsert upserted_id set	Document newly recovered into orders	Mark replayed
Pre-validation fails locally	Schema fix does not yet cover this payload	Refresh failing_keywords, leave pending, count it
Pre-validation passes but server raises code 121	Client jsonschema and server validator disagree	Halt; reconcile the two schemas before continuing
BulkWriteError with mixed writeErrors	Some batch members still non-compliant under ordered=False	Mark the successes replayed, leave failures pending

The last-but-one row is the dangerous one: if a document your client believes is valid is still rejected server-side, your pre-validation schema has drifted from the deployed validator and every replay decision is now suspect. Keep the two definitions sourced from one artifact. The taxonomy for classifying the remaining still-failing envelopes is the subject of categorizing schema validation errors. This snippet reconstructs a candidate and reports which of the three outcomes it lands in:

from jsonschema import Draft4Validator

def classify(envelope: dict, validator: Draft4Validator) -> str:
    """Return 'valid', 'still_invalid' for a reconstructed dead-letter payload."""
    doc = envelope["original"]
    errors = sorted(validator.iter_errors(doc), key=lambda e: list(e.path))
    return "valid" if not errors else "still_invalid"

Step-by-Step Playbook

The replay drains pending envelopes in bounded pages, pre-validates each, upserts the survivors on their original _id, and transitions every envelope’s status. Documents that still fail are left in place with refreshed diagnostics for the next round after a further schema fix.

1. Select a bounded page of pending envelopes. Never load the whole backlog into memory; iterate in pages so the job is restartable and its memory footprint is flat:

from pymongo import MongoClient
from jsonschema import Draft4Validator

client = MongoClient("mongodb://localhost:27017/?replicaSet=rs0")
db = client.shop

# One artifact, used both here and as the server validator, so they never diverge.
ORDER_SCHEMA = {
    "type": "object",
    "required": ["_id", "customer_id", "status", "total"],
    "properties": {
        "customer_id": {"type": "string", "minLength": 1},
        "status": {"enum": ["pending", "paid", "shipped", "cancelled"]},
        "total": {"type": "number", "minimum": 0},
    },
    "additionalProperties": False,
}
checker = Draft4Validator(ORDER_SCHEMA)

def pending_page(limit: int = 500):
    return list(db.orders_deadletter.find({"status": "pending"}).limit(limit))

2. Pre-validate and transform if needed. Reconstruct the candidate, apply any deterministic fix-up the schema change now permits, and re-check locally. Only survivors advance to a write:

def reconstruct(envelope: dict) -> dict:
    """Rebuild the candidate document, applying any known-safe coercion."""
    doc = dict(envelope["original"])
    # Example: a producer bug shipped total as a string; the widened schema
    # still wants a number, so coerce where it is now unambiguous.
    if isinstance(doc.get("total"), str) and doc["total"].replace(".", "", 1).isdigit():
        doc["total"] = float(doc["total"])
    return doc

3. Upsert survivors and mark status. Express the re-insert as an idempotent upsert on the original _id, then transition the envelope. A still-failing payload gets refreshed diagnostics and stays pending:

import datetime as dt

def replay_page(limit: int = 500) -> dict:
    stats = {"replayed": 0, "still_pending": 0}
    for env in pending_page(limit):
        doc = reconstruct(env)
        errors = sorted(checker.iter_errors(doc), key=lambda e: list(e.path))
        if errors:
            db.orders_deadletter.update_one(
                {"_id": env["_id"]},
                {"$set": {"failing_keywords": [e.validator for e in errors]},
                 "$inc": {"attempts": 1}},
            )
            stats["still_pending"] += 1
            continue
        # Idempotent: an _id already in orders is a no-op, never a double insert.
        db.orders.update_one({"_id": doc["_id"]}, {"$setOnInsert": doc}, upsert=True)
        db.orders_deadletter.update_one(
            {"_id": env["_id"]},
            {"$set": {"status": "replayed",
                      "replayed_at": dt.datetime.now(dt.timezone.utc)}},
        )
        stats["replayed"] += 1
    return stats

# Expected output: e.g. {'replayed': 487, 'still_pending': 13}

Failure Modes & Rollback

Replay concentrates risk at the write and status-transition boundary; these are the failures worth designing against:

• Partial-batch interruption. If the job crashes between the orders upsert and the envelope status update, the document is in the primary collection but its envelope is still pending. The next run reconstructs it, upserts again — a no-op because the _id already matches — and then marks it replayed. The upsert-on-_id design makes this crash-safe with no manual cleanup.
• Double-insert from a naive insert. Replacing the upsert with a plain insert_one reintroduces the double-write hole: a retried batch inserts a second copy under a fresh _id if the reconstruction lost the original identifier. Always preserve and key on the original _id; never let the driver mint a new one on replay.
• Client/server schema drift. If pre-validation passes but the server still returns code 121, the two schema definitions have diverged. Halt the run — do not mark anything replayed — and reconcile the client jsonschema against the registered validator before resuming.
• Replaying too early. Re-driving before the schema is actually widened or the producer fixed just reproduces the original deterministic rejections and burns write capacity. Gate the job on a confirmed schema version bump.

Rollback is straightforward because replay only ever adds to orders and flips status forward. To undo a bad run, reset the affected envelopes with db.orders_deadletter.updateMany({status: "replayed", replayed_at: {$gte: <run_start>}}, {$set: {status: "pending"}}) and, if a faulty transform wrote bad data, delete the upserted _ids from orders — they are exactly the set you can enumerate from the replayed envelopes. Time-to-recover is a single pair of bulk updates. Documents that never pass replay should be escalated through the fallback validation chains for deeper transformation rather than looped indefinitely.

Frequently Asked Questions

Related

• Fallback Routing for Invalid Documents — the parent workflow that positions replay as the recovery half of the quarantine-and-replay pattern.
• MongoDB JSON Schema Validation Architecture — the section reference defining the write-path contract a replay re-tests old data against.
• Building a Dead-Letter Collection for Rejected Writes — how the envelopes this job drains are captured and shaped in the first place.
• Routing WriteError 121 to a Retry Queue — why deterministic 121s are quarantined for replay rather than retried in line.
• Categorizing Schema Validation Errors — classifying the still-failing envelopes that a replay leaves pending.
• Building Fallback Validation Chains for MongoDB Ingestion — the deeper transformation path for documents that never pass replay.