Routing WriteError 121 to a Retry Queue

The central mistake this page exists to prevent is treating a schema-validation WriteError like any other write failure and feeding it into a retry queue — because a code 121 is deterministic and every retry fails identically, whereas a transient fault such as a network drop or a primary stepdown clears on its own. Sitting under fallback routing for invalid documents within the broader MongoDB JSON Schema Validation Architecture, this guide gives you a dispatcher that inspects the driver exception’s code, retries only the transient class with exponential backoff, and diverts a 121 straight to quarantine where it belongs. The reader outcome is a single routing function that can never turn a validation failure into a retry storm, and a classification table you can encode directly into your write path.

Operational Mechanics and Write-Path Impact

Every failed write in MongoDB carries a numeric error code, and that code is the entire basis for the retry-versus-divert decision. Failures split into two disjoint classes. Transient failures reflect a temporary state of the deployment — a stepping-down primary, a WriteConflict under contention, a socket timeout — and the correct response is to wait and re-issue the identical operation, because the next attempt runs against a healthier replica set state. Deterministic failures reflect a property of the document itself; a code 121 means the payload violated a fixed $jsonSchema, and no amount of waiting changes the rule or the data, so re-issuing the same write is guaranteed to reproduce the same rejection.

Conflating the two is what produces a retry storm. If a dispatcher retries a 121 with exponential backoff, it burns the full retry budget on a write that was never going to succeed, multiplies write-path load during the backoff loop, and delays the moment the bad document reaches quarantine where it could actually be remediated. The dispatcher must therefore branch on exc.code before deciding anything: a transient code enters the backoff loop, and a 121 skips it entirely and is handed to the dead-letter path.

Error code	Symbol	Class	Disposition
121	DocumentValidationFailure	Deterministic	Divert to dead-letter — never retry
112	WriteConflict	Transient	Retry with backoff
11600	InterruptedAtShutdown	Transient	Retry with backoff
11602	InterruptedDueToReplStateChange	Transient	Retry with backoff
10107	NotWritablePrimary	Transient	Retry with backoff
13435	NotPrimaryNoSecondaryOk	Transient	Retry with backoff
189	PrimarySteppedDown	Transient	Retry with backoff
91	ShutdownInProgress	Transient	Retry with backoff
11000	DuplicateKey	Deterministic	Do not retry; route to conflict handling

MongoDB’s own drivers already label most of the transient codes with the RetryableWriteError error label, and retryable writes will transparently re-attempt many of them once. That built-in mechanism does not cover 121 and never will, precisely because the server knows a validation failure is not retryable. Your dispatcher is the layer that makes the same distinction explicit for the failures the driver hands back to application code, and it draws the retryable set from the transient codes above rather than blindly retrying anything that raised.

Exact Diagnostic Fingerprints and Fast Resolution

The signal you branch on is exc.code on a pymongo.errors.OperationFailure or WriteError (single writes), or each writeErrors[].code inside a BulkWriteError (batches). A 121 additionally carries the errInfo.details.schemaRulesNotSatisfied structure that names the failing keyword, which is what makes it safe to route to a dead-letter for later inspection. The signatures below tell a misclassification apart from correct behaviour at a glance:

Signature	Interpretation	Correct routing
exc.code == 121, errInfo present	Deterministic validation failure	Dead-letter immediately; zero retries
exc.code in TRANSIENT, RetryableWriteError label	Temporary replica set state	Backoff loop, bounded attempts
Same code 121 seen repeatedly for one _id in retry logs	A 121 was wrongly enqueued for retry	Remove from retry queue; divert to quarantine
Retry-queue depth climbing while error rate flat	Retry storm from misclassified deterministic errors	Audit the classifier; 121 or 11000 leaked into the retry path
exc.code == 112 clearing after one backoff	Healthy transient recovery	No action — working as intended

A retry storm has a recognisable fingerprint: a growing retry-queue depth or CPU climb that does not correspond to any increase in new failures, because the same deterministic documents are being re-enqueued in a loop. Sorting a transient code from a deterministic one is the same discipline formalised in categorizing schema validation errors. This helper encodes the classification as pure data so the routing decision is testable in isolation:

# Transient codes worth retrying with backoff — everything else is deterministic.
TRANSIENT_CODES = {112, 189, 91, 10107, 11600, 11602, 13435}
VALIDATION_CODE = 121

def is_retryable(code: int) -> bool:
    """A 121 is deterministic and must never be retried; only transient codes may be."""
    return code in TRANSIENT_CODES

# is_retryable(121) -> False   (divert to dead-letter)
# is_retryable(112) -> True    (retry with backoff)

Step-by-Step Playbook

The dispatcher wraps a single write, retries only the transient class with bounded exponential backoff, and hands a code 121 to the dead-letter path immediately. It never blends the two decisions.

1. Classify before acting. The first line of the failure handler branches on the code. A 121 is diverted with zero retries; only a transient code is allowed near the backoff loop:

from pymongo.errors import OperationFailure

def route_failure(exc: OperationFailure, doc: dict) -> str:
    """Decide disposition purely from the error code."""
    if exc.code == VALIDATION_CODE:
        dead_letter(doc, exc.details.get("errInfo", {}) if exc.details else {})
        return "dead_lettered"     # deterministic — remediation, not retry
    if is_retryable(exc.code):
        return "retry"             # transient — safe to re-issue
    raise exc                      # unknown class — surface it, never silently loop

2. Retry only the transient class with backoff. The retry loop is reserved for codes that a later attempt can actually clear. Delay grows exponentially with jitter, and the attempt budget is bounded so a persistent transient fault escalates instead of looping forever:

import random
import time
from pymongo.collection import Collection

def dispatch_write(coll: Collection, doc: dict, max_attempts: int = 4,
                   base_delay: float = 0.25) -> str:
    """Insert doc, retrying only transient failures; divert a 121 immediately."""
    for attempt in range(1, max_attempts + 1):
        try:
            coll.insert_one(doc)
            return "written"
        except OperationFailure as exc:
            decision = route_failure(exc, doc)
            if decision != "retry":
                return decision            # dead_lettered — loop never runs again
            if attempt == max_attempts:
                raise                       # transient, but budget exhausted
            delay = base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.1)
            time.sleep(delay)
    return "escalated"

# Expected: a valid doc -> "written"; an invalid doc -> "dead_lettered" on attempt 1;
# a WriteConflict -> retried, then "written" once contention clears.

3. Prove a 121 never entered the retry loop. Assert the invariant directly so a regression in the classifier is caught in tests, not in a production storm:

class FakeValidationError(OperationFailure):
    def __init__(self):
        super().__init__("Document failed validation", 121, {"errInfo": {}})

seen = {"attempts": 0}
def counting_insert(_doc):
    seen["attempts"] += 1
    raise FakeValidationError()

# The dispatcher must dead-letter on the FIRST attempt and never loop.
# assert dispatch_write(...) == "dead_lettered" and seen["attempts"] == 1

Failure Modes & Rollback

The failure modes here are almost all forms of misclassification, and each has a precise remedy:

• A 121 leaks into the retry path. If the classifier omits the exc.code == 121 guard, a validation failure is retried through the full backoff budget and re-enqueued, producing a retry storm whose fingerprint is climbing queue depth with a flat new-failure rate. Rollback: restore the deterministic guard so a 121 is dead-lettered on attempt one; drain the queue of any 121s already stuck in it.
• 11000 treated as transient. A duplicate-key error is deterministic like a 121 — the key already exists, and retrying cannot change that. Keep it out of TRANSIENT_CODES; route it to conflict handling, not backoff.
• Unbounded retries. A retry loop without a budget turns a persistent primary stepdown into an infinite spin. Always cap max_attempts and escalate when it is spent.
• No jitter. Synchronised backoff across many workers produces a thundering herd that re-hits a recovering primary in lockstep. The random.uniform jitter term de-synchronises them.

Rollback for a live retry storm is immediate: ship the corrected classifier so 121 and 11000 divert instead of retry, then purge the retry queue of deterministic entries — they can be moved straight to the dead-letter collection for eventual replay after the schema is fixed. Time-to-recover is one deploy plus a queue drain; no data is lost because diverted documents are preserved, not dropped. The broader multi-tier arrangement of these handlers is described in building fallback validation chains.

Frequently Asked Questions

Related

• Fallback Routing for Invalid Documents — the parent workflow this dispatcher plugs into as the retry-versus-divert decision point.
• MongoDB JSON Schema Validation Architecture — the section reference defining the code 121 contract the classifier keys off.
• Building a Dead-Letter Collection for Rejected Writes — where a diverted 121 lands once the dispatcher rules out retry.
• Replaying Quarantined Documents After Schema Fixes — how the deterministic failures this page diverts are eventually recovered.
• Categorizing Schema Validation Errors — the taxonomy that separates a transient fault from a deterministic validation failure.
• Building Fallback Validation Chains for MongoDB Ingestion — the multi-tier arrangement these retry and divert handlers compose into.