SOC 2 Evidence from Validation Telemetry

A SOC 2 processing-integrity examination asks a blunt question about every automated control: did it operate effectively across the entire audit period, and can you prove it? A MongoDB $jsonSchema validator is such a control — it decides, on the write path, whether a document is allowed to persist — but an auditor never accepts the mere existence of a validator as proof. What satisfies the criterion is telemetry: the count of writes the validator rejected with code 121, the stream of warn-mode diagnostics it logged under a fixed message id, the periodic scans confirming stored documents still satisfy the contract, and the change-stream record showing nobody quietly loosened the validator between control tests. Within the broader Schema Validation for Compliance & Audit Readiness framework, this page is the complete workflow for turning those raw signals into a defensible evidence file. You will stand up a collector that samples validation metrics into a durable evidence collection, decide what to retain and for how long, and package a period report an examiner can read without touching your cluster. The concrete export mechanics — rolling the sampled metrics into a single signed, timestamped artifact — are handled by its companion, Exporting Validation Metrics as Audit Evidence.

Architectural Context & Enforcement Boundaries

The validator itself is described elsewhere; here it is a source of evidence, and the distinction changes what you build. A control test in SOC 2 is a claim of the form “over the period 2026-01-01 to 2026-12-31, the orders collection rejected every non-conforming write, and no rejection was silently suppressed.” That claim decomposes into four measurable signals, each emitted at a different point on the write path. Rejection counts come from writes that failed with WriteError/OperationFailure code 121 — captured either application-side in the driver or by parsing the server log. Permissive-observation signals come from collections running validationAction: "warn", where each non-conforming write is logged to mongod under message id 20294 ("Document would fail validation") but still persists; that log line is evidence the control saw the write even though it did not block it. Compliance-over-time comes from a periodic drift scan — a $nor + $jsonSchema count that proves the stored population still satisfies the active contract. Control-integrity comes from watching the collection’s own configuration for change, so a collMod that weakens the validator is itself an audit event rather than a blind spot.

The collector’s job is to sample these signals on a schedule and land them in an append-only evidence collection, from which a period report is generated. The runtime detection and alerting side of this — the dashboards and thresholds that fire in real time — belongs to Automated Schema Enforcement & Monitoring; what this workflow adds is durability and provenance, the properties an auditor cares about that a live dashboard lacks.

Prerequisites & Operational Requirements

Confirm each item before the collector is allowed to write into an evidence store an auditor will later rely on.

• MongoDB version: 5.0 or later for the structured errInfo.details.schemaRulesNotSatisfied object that gives a rejection its keyword-level reason, and 6.0 or later if you intend to capture validator-configuration changes from a change stream — DDL events such as the modify operation type from a collMod are only delivered when the stream is opened with showExpandedEvents: true.
• Driver: PyMongo 4.6+ (pip install "pymongo>=4.6,<5"), pinned in the collector image so OperationFailure.code, errInfo, and change-stream resume-token semantics are reproducible across every scheduled run for the whole period.
• Permissions: the collector principal needs read on the audited databases, changeStream and find for the watch, and write access to the evidence database only. It must not hold collMod — the process that reads the control must be unable to alter it, a separation of duties an examiner will look for directly.
• A source of rejection counts: either your application already catches code 121 and increments a counter you can query, or mongod is configured to log rejected writes. Decide this up front; a period with no rejection source is a period with an evidence gap.
• Clock discipline: every sample is timestamped, and the report asserts a period boundary. Run the collector against an NTP-synced host and store timestamps in UTC, because an auditor reconciles your evidence against events in other systems by wall-clock time.
• Evidence retention: the evidence collection must outlive the audit — SOC 2 Type II reports commonly cover a twelve-month window examined months later. Place it on a deployment with a retention and backup policy that guarantees the samples survive well past the period they document.

Idempotent Deployment / Implementation Workflow

Each step is independently verifiable and safe to re-run; a scheduled collector that crashes mid-period must be able to resume without double-counting or leaving a hole.

1. Create the evidence collection with its own contract. The evidence store is itself governed by a validator, so a malformed sample can never dilute the record. Give it an index on the sample key so the rollup is a covered range scan:

   db.createCollection("evidence_metrics", {
     validator: { $jsonSchema: {
       bsonType: "object",
       required: ["collection", "signal", "window_start", "value", "sampled_at"],
       properties: {
         collection:   { bsonType: "string" },
         signal:       { bsonType: "string", enum: ["rejections", "warnings", "drift", "config_change"] },
         window_start: { bsonType: "date" },
         value:        { bsonType: ["int", "long"] },
         sampled_at:   { bsonType: "date" }
       },
       additionalProperties: true
     }},
     validationLevel: "strict",
     validationAction: "error"
   })
   db.evidence_metrics.createIndex({ collection: 1, signal: 1, window_start: 1 }, { unique: true })
   // Expected output: { ok: 1 } then the index name "collection_1_signal_1_window_start_1"

2. Establish the drift baseline. For each audited collection, record the count of documents that do not satisfy the active schema. A healthy control starts at zero; a non-zero baseline is itself evidence you disclose, not hide:

   db.orders.countDocuments({ $nor: [{ $jsonSchema: db.getCollectionInfos({ name: "orders" })[0].options.validator.$jsonSchema }] })
   // Expected output: 0

3. Schedule the sampler. Run the collector on a fixed cadence (hourly for rejection and warn counters, daily for the drift scan) writing one document per signal per window. The unique index makes a re-run of the same window an idempotent upsert, so a retried job overwrites rather than duplicates.

4. Attach the configuration watch. Open a change stream on each audited collection with showExpandedEvents: true and record every modify event as a config_change sample. This is the signal that the validator was not quietly loosened between control tests.

5. Generate the period report. At the close of each reporting window, roll the samples up into the signed artifact described in Exporting Validation Metrics as Audit Evidence, which becomes the document you hand the examiner.

Production-Ready Automation Implementation

The module below is the sampler that steps 3 and 4 schedule. It reads live validator configuration from listCollections, runs the drift scan against the current schema so the evidence tracks the contract as it actually stood, ingests pre-parsed rejection and warn counters supplied by your log pipeline or application counters, and upserts one durable sample per signal per window. Every write into the evidence store is keyed on (collection, signal, window_start), so the operation is idempotent under retry.

import logging
from datetime import datetime, timezone
from typing import Any, Dict, Optional

from pymongo import MongoClient, UpdateOne
from pymongo.collection import Collection
from pymongo.errors import OperationFailure, PyMongoError

logger = logging.getLogger(__name__)


class ValidationEvidenceCollector:
    """Sample $jsonSchema control telemetry into an append-only evidence collection."""

    def __init__(self, client: MongoClient, audited_db: str, evidence_db: str = "compliance"):
        self.client = client
        self.audited = client[audited_db]
        self.evidence: Collection = client[evidence_db]["evidence_metrics"]

    def _active_schema(self, coll_name: str) -> Optional[Dict[str, Any]]:
        """Read the live validator so the drift scan tests the contract as it stands now."""
        info = self.audited.command("listCollections", filter={"name": coll_name})
        batch = info["cursor"]["firstBatch"]
        if not batch:
            return None
        validator = (batch[0].get("options") or {}).get("validator") or {}
        return validator.get("$jsonSchema")

    def scan_drift(self, coll_name: str) -> int:
        """Count stored documents that would fail the active schema (read-only, no writes)."""
        schema = self._active_schema(coll_name)
        if schema is None:
            raise RuntimeError(f"No active validator on {coll_name}; nothing to attest.")
        return self.audited[coll_name].count_documents({"$nor": [{"$jsonSchema": schema}]})

    def _sample(self, coll_name: str, signal: str, window_start: datetime, value: int) -> UpdateOne:
        now = datetime.now(timezone.utc)
        return UpdateOne(
            {"collection": coll_name, "signal": signal, "window_start": window_start},
            {"$set": {"value": int(value), "sampled_at": now},
             "$setOnInsert": {"collection": coll_name, "signal": signal,
                              "window_start": window_start}},
            upsert=True,
        )

    def record_window(
        self,
        coll_name: str,
        window_start: datetime,
        rejections: int,
        warnings: int,
    ) -> None:
        """Persist rejection, warn and drift samples for one collection and window."""
        if window_start.tzinfo is None:
            raise ValueError("window_start must be timezone-aware (UTC).")
        ops = [
            self._sample(coll_name, "rejections", window_start, rejections),
            self._sample(coll_name, "warnings", window_start, warnings),
            self._sample(coll_name, "drift", window_start, self.scan_drift(coll_name)),
        ]
        try:
            result = self.evidence.bulk_write(ops, ordered=False)
            logger.info("Evidence for %s @ %s: upserted=%d modified=%d",
                        coll_name, window_start.isoformat(),
                        result.upserted_count, result.modified_count)
        except OperationFailure as exc:
            # A 121 here means a malformed sample hit the evidence validator itself.
            logger.error("Evidence write rejected (code %s): %s", exc.code, exc.details)
            raise
        except PyMongoError as exc:
            logger.error("Driver error writing evidence for %s: %s", coll_name, exc)
            raise

    def watch_config(self, coll_name: str) -> None:
        """Record validator-configuration changes (collMod) as config_change evidence."""
        coll = self.audited[coll_name]
        with coll.watch(
            [{"$match": {"operationType": {"$in": ["modify", "invalidate"]}}}],
            show_expanded_events=True,
            full_document="updateLookup",
        ) as stream:
            for change in stream:
                self.evidence.insert_one({
                    "collection": coll_name,
                    "signal": "config_change",
                    "window_start": datetime.now(timezone.utc),
                    "value": 1,
                    "sampled_at": datetime.now(timezone.utc),
                    "operation_type": change["operationType"],
                    "resume_token": change["_id"],
                })
                logger.warning("Validator configuration changed on %s: %s",
                               coll_name, change["operationType"])

The design decision that makes this evidence rather than metrics is scan_drift re-reading the live schema on every window instead of a hard-coded copy: if someone tightens the validator mid-period, the drift figure automatically measures the population against the stricter contract, and if someone loosens it, the watch_config stream records the fact. The two signals are cross-checkable, which is exactly what an examiner reconciles.

Diagnostic Fingerprints & Fast Resolution

When the evidence itself looks wrong, match the signature before you trust or discard the sample.

Signature	Root cause	Resolution
OperationFailure code 121 writing to evidence_metrics	A sample is missing a required field or has a wrong bsonType — the evidence validator is doing its job.	Inspect exc.details["errInfo"]["details"]["schemaRulesNotSatisfied"]; fix the sampler payload, never loosen the evidence validator.
drift sample suddenly non-zero for a stable collection	An aggregation $out/$merge or a moderate-level update reintroduced non-conforming documents that bypass the write-path validator.	Disclose the count, then remediate; investigate the ingest path that bypassed enforcement.
A window has zero rejections and zero warnings and zero traffic	Either genuinely no writes, or the log/counter feed stalled — an evidence gap, not a clean period.	Cross-check against write volume from serverStatus; if writes occurred, treat the window as a gap and backfill from retained logs.
config_change sample with no corresponding change ticket	A collMod altered the validator outside change management.	Escalate as a control exception; the change-stream record is the primary evidence of when it happened.
Change stream raises OperationFailure code 286 (ChangeStreamHistoryLost)	The resume token aged out of the oplog while the watcher was down.	Restart from the current time and record the outage window as a known gap in coverage.

To confirm the collector is the only process reading — and that no long-running operation is skewing a drift scan — sample the live operation list:

db.getSiblingDB("admin").aggregate([
  { $currentOp: { allUsers: true, idleConnections: false } },
  { $match: { ns: /^production_db\./, "command.collMod": { $exists: true } } }
])
// Expected output: empty array when no validator change is in flight

Edge Cases, Gotchas & Known Limitations

• Log rotation silently truncates the rejection feed. If rejection and warn counts come from parsing mongod logs, a rotation (logRotate, or an external logrotate on the file) that fires between samples drops every line written since the last read. Track the byte offset or inode you last consumed and detect rotation explicitly; a rotated-away segment is an evidence gap that must be declared, not assumed empty.
• warn mode is a migration instrument, not a control posture. A collection left permanently on validationAction: "warn" produces 20294 signals but persists the non-conforming write anyway — to an auditor that reads as a control that detects but does not prevent. Time-box warn and let the config_change signal prove when you returned to error.
• The drift scan is a point-in-time snapshot. A $nor + $jsonSchema count taken once a day says nothing about the 23 hours between scans. Pair it with the continuous rejection counter so the evidence covers both the stored population and the live write path; neither alone attests the full period.
• Sampling gaps from a crashed collector are invisible unless you look for them. Because samples are keyed by window, a missing window is an absence, and absence is easy to overlook. Have the rollup assert an expected sample count per period and flag any window with no document, rather than trusting that what is present is complete.
• Change streams do not deliver events that predate the watcher. If the collector starts after a collMod, that change is not in the stream. Snapshot the validator configuration at collector start as a baseline config_change, so the very first control test has a known starting state.
• $out/$merge bypass the write-path validator entirely. A pipeline that rewrites an audited collection can raise the drift count without ever emitting a code 121 — which is why the periodic scan exists alongside the rejection counter, and why an audited collection should never be a $merge target without its own re-validation pass.

Verification & Rollback Procedures

Prove the evidence store is trustworthy before you present it. The evidence collection’s own validator is the first check — a known-bad sample must be rejected — and completeness is the second:

from pymongo.errors import WriteError

# 1. The evidence validator must reject a malformed sample.
try:
    db.compliance.evidence_metrics.insert_one({"collection": "orders", "signal": "bogus"})
    raise AssertionError("evidence validator is NOT enforcing")
except WriteError as exc:
    assert exc.code == 121
    print("evidence validator confirmed: malformed sample rejected with 121.")

# 2. Every expected window in the period must be present (no silent gaps).
missing = db.compliance.evidence_metrics.aggregate([
    {"$match": {"collection": "orders", "signal": "drift",
                "window_start": {"$gte": period_start, "$lt": period_end}}},
    {"$count": "windows_present"}
])
print("drift windows present:", list(missing))  # compare against expected count

Rollback here never means deleting evidence — a SOC 2 record is append-only by design, and destroying samples is itself a finding. Instead, “rollback” means safely standing the collector down and reverting a validator you were sampling. Stop the collector process first, then if a validator change must be reverted, do it through the change-managed path so the reversion is itself captured as a config_change:

// Stand the control back to enforcement (captured as a config_change event).
db.runCommand({ collMod: "orders", validationLevel: "strict", validationAction: "error" })
// Expected output: { ok: 1 }

If the evidence collection itself is misconfigured, correct its validator forward with collMod rather than dropping and recreating it, so the samples already gathered are preserved. Time-to-recover for a stalled collector is one scheduler cycle; the only unrecoverable loss is a window whose source logs have already rotated away, which is precisely why retention is a prerequisite and not an afterthought. Treat the immutability of the audited fields themselves as a hard contract by pairing this workflow with Audit Log Schema Contracts.

Frequently Asked Questions

Related

• Schema Validation for Compliance & Audit Readiness — the parent section defining the compliance contract these validation controls satisfy.
• Exporting Validation Metrics as Audit Evidence — the export job that rolls the sampled metrics into a single signed, timestamped artifact for the examiner.
• Audit Log Schema Contracts — enforcing the immutable, well-formed audit records this evidence workflow depends on.
• Automated Schema Enforcement & Monitoring — the runtime detection and enforcement framework that produces the signals sampled here.
• Tracking Validation Failures with MongoDB Atlas Alerts — the real-time alerting counterpart to this durable evidence store.