Building a Dead-Letter Collection for Rejected Writes

A dead-letter collection is the durable sink that catches every payload a $jsonSchema validator rejects, so a WriteError with code 121 becomes a recoverable record instead of a lost message — the exact design problem this page solves under fallback routing for invalid documents, inside the broader MongoDB JSON Schema Validation Architecture. You will finish with a concrete orders_deadletter schema, a pymongo write wrapper that intercepts both single-document and bulk rejections, extracts the structured failure reason, and lands it in quarantine with a TTL retention window. The governing fact that shapes every decision here is that a code 121 is deterministic: the document failed against a fixed rule set, so re-issuing the identical insert is guaranteed to fail identically. Invalid documents must therefore be captured for later inspection and replay, never blindly retried against the same validator.

Operational Mechanics and Write-Path Impact

The dead-letter collection is a second, deliberately permissive namespace that sits beside each strictly-validated collection. Where the primary orders collection enforces a tight contract, orders_deadletter accepts anything the capture layer hands it — because the whole point is to store payloads that already failed a strict rule. The record you store is not the raw rejected document alone; it is an envelope that preserves the original payload verbatim alongside the forensic context needed to remediate it later.

A well-formed dead-letter envelope carries six load-bearing fields. The original sub-document is the exact BSON the caller tried to write, untouched, so a later replay has a faithful source. The err_info field holds the server’s errInfo object, the authoritative reason the write was rejected. failing_keywords is a flattened, queryable projection of the operators that failed — pulled from errInfo.details.schemaRulesNotSatisfied — so a dashboard can group the backlog by root cause without walking nested BSON. first_seen timestamps the initial capture and anchors the TTL retention window. attempts counts how many times this payload has been submitted, and status tracks its lifecycle through the remediation queue.

Field	BSON type	Purpose
original	object	Verbatim copy of the rejected document, preserved for replay
err_info	object	The server errInfo payload — the authoritative rejection reason
failing_keywords	array of string	Flattened operator names from schemaRulesNotSatisfied, for grouping
first_seen	date	Capture timestamp; the anchor field for the TTL retention index
attempts	int	Submission counter, incremented on each recapture of the same payload
status	string	Lifecycle state: pending, replayed, discarded

The dead-letter collection must carry its own validator, but a permissive one — it should guarantee the envelope shape without ever rejecting the quarantined content. Enforce only that the six fields exist with the right container types, and never constrain the interior of original, which by definition holds malformed data. A TTL index on first_seen gives the collection bounded growth: MongoDB’s background TTL monitor deletes envelopes older than the retention window automatically, so an unattended quarantine namespace cannot fill a disk. Set the window to cover your maximum realistic remediation latency plus an audit margin — a fortnight is a common default.

db.createCollection("orders_deadletter", {
  validator: { $jsonSchema: {
    bsonType: "object",
    required: ["original", "err_info", "failing_keywords", "first_seen", "attempts", "status"],
    properties: {
      original:         { bsonType: "object" },
      err_info:         { bsonType: "object" },
      failing_keywords: { bsonType: "array", items: { bsonType: "string" } },
      first_seen:       { bsonType: "date" },
      attempts:         { bsonType: "int", minimum: 1 },
      status:           { enum: ["pending", "replayed", "discarded"] }
    }
  }},
  validationLevel: "moderate",
  validationAction: "warn"
})
// A 14-day retention window on the capture timestamp.
db.orders_deadletter.createIndex({ first_seen: 1 }, { expireAfterSeconds: 1209600 })
// Expected output: orders_deadletter_first_seen_1

Exact Diagnostic Fingerprints and Fast Resolution

Capture logic keys entirely off the driver exception surface. A single-document insert_one that violates the validator raises pymongo.errors.WriteError with code 121; the same failure inside an insert_many or bulk_write surfaces as a BulkWriteError whose details["writeErrors"] list holds one entry per rejected document, each carrying its own code, errmsg, and errInfo. Distinguishing these two shapes is the first branch in any capture wrapper.

Signature	Where it appears	Meaning	Capture action
WriteError, code 121	Single insert_one / update_one	One document failed the validator	Envelope the one payload into dead-letter
BulkWriteError, writeErrors[].code == 121	insert_many / bulk_write	Some batch members failed	Envelope only the failed members by index
errInfo.details.schemaRulesNotSatisfied present	Any 121 on MongoDB 5.0+	Structured per-keyword failure detail	Flatten into failing_keywords
code 121 with empty errInfo	Any 121 on MongoDB 4.x	Server predates structured reasons	Store the opaque errmsg; group is coarse
Non-121 WriteError (e.g. 11000)	Duplicate key, etc.	Not a validation failure	Do not dead-letter; re-raise

The schemaRulesNotSatisfied array is the richest artifact: each element names an operatorName (such as required, bsonType, or enum) and the property path it failed on. Flattening these operator names into failing_keywords is what later lets you answer “how much of my backlog is a missing required field versus a bad enum value” with a one-line aggregation. That same categorization discipline is developed further in categorizing schema validation errors. This helper extracts the keyword list from either exception shape:

def failing_keywords(err_info: dict) -> list[str]:
    """Flatten schemaRulesNotSatisfied into a list of failed operator names."""
    rules = (err_info or {}).get("details", {}).get("schemaRulesNotSatisfied", [])
    names: list[str] = []
    for rule in rules:
        if "operatorName" in rule:
            names.append(rule["operatorName"])
        # propertiesNotSatisfied nests per-field operator failures one level down.
        for prop in rule.get("propertiesNotSatisfied", []):
            for detail in prop.get("details", []):
                if "operatorName" in detail:
                    names.append(detail["operatorName"])
    return names or ["unknown"]

Step-by-Step Playbook

The capture wrapper wraps every write to the protected collection, lets successful writes through untouched, and diverts only code 121 rejections into the dead-letter namespace. Non-validation errors are re-raised so genuine infrastructure faults are never misfiled as schema violations.

1. Guard a single insert. Wrap insert_one so a code 121 is enveloped rather than propagated. The wrapper stamps first_seen, initialises attempts to 1, and sets status to pending:

import datetime as dt
from pymongo import MongoClient
from pymongo.errors import WriteError

client = MongoClient("mongodb://localhost:27017")
db = client.shop

def guarded_insert(doc: dict) -> str | None:
    """Insert into orders; divert a code-121 rejection to orders_deadletter."""
    try:
        return str(db.orders.insert_one(doc).inserted_id)
    except WriteError as exc:
        if exc.code != 121:
            raise  # not a validation failure — let it surface
        info = exc.details.get("errInfo", {}) if exc.details else {}
        db.orders_deadletter.insert_one({
            "original": doc,
            "err_info": info,
            "failing_keywords": failing_keywords(info),
            "first_seen": dt.datetime.now(dt.timezone.utc),
            "attempts": 1,
            "status": "pending",
        })
        return None  # caller sees a soft failure, not a 500

# Expected: valid docs return an id string; invalid docs return None and appear in quarantine.

2. Guard a bulk insert. For insert_many, catch BulkWriteError and envelope only the members whose writeErrors entry reports code 121, addressed by their batch index:

from pymongo.errors import BulkWriteError

def guarded_insert_many(docs: list[dict]) -> int:
    """Insert a batch unordered; dead-letter every code-121 member by index."""
    try:
        return len(db.orders.insert_many(docs, ordered=False).inserted_ids)
    except BulkWriteError as bwe:
        envelopes = []
        for we in bwe.details.get("writeErrors", []):
            if we.get("code") != 121:
                raise  # a non-validation write error in the batch — abort
            info = we.get("errInfo", {})
            envelopes.append({
                "original": docs[we["index"]],
                "err_info": info,
                "failing_keywords": failing_keywords(info),
                "first_seen": dt.datetime.now(dt.timezone.utc),
                "attempts": 1,
                "status": "pending",
            })
        if envelopes:
            db.orders_deadletter.insert_many(envelopes, ordered=False)
        return bwe.details.get("nInserted", 0)

3. Confirm the capture. Query the quarantine namespace grouped by failing keyword to prove nothing was silently dropped and to size the backlog by root cause:

db.orders_deadletter.aggregate([
  { $match: { status: "pending" } },
  { $unwind: "$failing_keywords" },
  { $group: { _id: "$failing_keywords", n: { $sum: 1 } } },
  { $sort: { n: -1 } }
])
// Expected output: e.g. [ { _id: "required", n: 41 }, { _id: "enum", n: 12 } ]

Failure Modes & Rollback

Each moving part fails in a characteristic way, and each has a bounded recovery:

• The dead-letter validator is too strict. If orders_deadletter inherits a tight schema, capturing a malformed payload itself raises code 121 and the wrapper throws inside its own except block — losing the document it was trying to save. Keep the envelope validator permissive, constraining only the six top-level fields, never the interior of original. Recovery: collMod the dead-letter validator back to the loose shape shown above.
• Missing TTL index. Without the expireAfterSeconds index, quarantine grows without bound and eventually pressures disk. Recovery: create the index — MongoDB’s TTL monitor begins reaping expired envelopes within 60 seconds of the next sweep.
• Recapture inflation. If an upstream producer resubmits the same broken payload every minute, you accumulate near-identical envelopes. Add a deterministic dedupe key (a hash of original) with a unique index and $inc the attempts counter on conflict instead of inserting a new row, which keeps the backlog proportional to distinct failures.
• Swallowing real errors. A wrapper that dead-letters any exception hides duplicate-key and network faults as if they were schema violations. The exc.code != 121 guard is mandatory; without it, an incident becomes invisible.

The rollback for the entire capture layer is to stop diverting and let the raw code 121 surface again — a one-line change removing the except branch — while the accumulated envelopes remain queryable for replay after the schema is fixed. Time-to-recover is immediate on the next deploy; no data is lost because the quarantine collection is append-only until a replay job consumes it.

Frequently Asked Questions

Related

• Fallback Routing for Invalid Documents — the parent workflow that positions the dead-letter collection within the broader quarantine-and-replay pattern.
• MongoDB JSON Schema Validation Architecture — the section reference defining the write-path enforcement contract that produces the rejections you capture.
• Replaying Quarantined Documents After Schema Fixes — how to drain the dead-letter backlog idempotently once the schema widens or a producer bug is fixed.
• Routing WriteError 121 to a Retry Queue — deciding which failures to dead-letter versus retry, so transient faults never land in quarantine.
• Categorizing Schema Validation Errors — turning the failing_keywords projection into a root-cause taxonomy for the backlog.
• Building Fallback Validation Chains for MongoDB Ingestion — the ingestion-side sibling that feeds this same capture layer on the enforcement-and-monitoring platform.