Exporting Validation Metrics as Audit Evidence

The precise question this page answers is how to turn a collection of sampled validation metrics into a single file an examiner can open, read, and trust without ever logging into your database — the export step that sits under SOC 2 Evidence from Validation Telemetry within the broader Schema Validation for Compliance & Audit Readiness section. The reader outcome is a repeatable export job: it aggregates the rejection, warn, and drift samples for one reporting period into an artifact that states the control description, the exact period boundaries, the number of samples that back each figure, and a pass or fail outcome, then seals that artifact with a timestamp and a cryptographic signature so any later tampering is detectable. The trap this avoids is handing an auditor a raw metric dump with no provenance — a number with no attestation of what it measured, over what window, or whether the series behind it was complete.

Rollup Mechanics and Artifact Structure

An evidence artifact is a claim plus the arithmetic that supports it, so the export does two things: it aggregates the underlying samples, and it records enough context to make the aggregate self-describing. The aggregation is a straightforward $group over the evidence_metrics collection populated by the sampler, bucketed by collection and signal for the target period. The context is what distinguishes evidence from a metric: each rolled-up figure carries the count of sample windows that produced it, so a reader can tell a genuine zero (many samples, all zero) from an absent series (no samples at all). The outcome field applies the control’s own pass criterion — typically zero unexplained rejections and zero residual drift — and records pass or exception accordingly.

Artifact field	Source	Why the auditor needs it
control	Static description of the validator control	Names what the evidence attests, in plain language
collection	evidence_metrics.collection	Scopes each figure to one enforced namespace
period_start / period_end	Report parameters (UTC)	Fixes the exact window the claim covers
total_rejections	$sum of rejections samples	Volume the control blocked on the write path
total_warnings	$sum of warnings samples	Writes the control observed but did not block
max_drift	$max of drift samples	Worst-case stored non-compliance in the window
sample_windows	$sum: 1 over samples	Proves the series is complete, not sparse
outcome	Derived from the pass criterion	The examiner’s bottom line: pass or exception
generated_at / signature	Export runtime	Makes the artifact tamper-evident

Exact Diagnostic Fingerprints and Fast Resolution

The failure mode unique to an export is a gap in the series: an aggregate that reads as clean because the samples behind it are missing, not because the control was quiet. The rollup must therefore compute the number of windows it actually found and compare it against the number the period should contain. This snippet surfaces every collection whose series is short:

from datetime import timezone

EXPECTED_WINDOWS = 24  # e.g. hourly rejection samples over one day

gap_pipeline = [
    {"$match": {"signal": "rejections",
                "window_start": {"$gte": period_start, "$lt": period_end}}},
    {"$group": {"_id": "$collection", "windows": {"$sum": 1}}},
    {"$match": {"windows": {"$lt": EXPECTED_WINDOWS}}},
]
gaps = list(db.compliance.evidence_metrics.aggregate(gap_pipeline))
for g in gaps:
    print(f"GAP: {g['_id']} has {g['windows']}/{EXPECTED_WINDOWS} windows")
# Expected output: no lines printed when every series is complete

Signature	Root cause	Resolution
windows less than EXPECTED_WINDOWS for a collection	The sampler missed windows — a crash, a scheduler miss, or rotated-away source logs	Backfill from retained logs if possible; otherwise declare the gap in the artifact rather than reporting a false zero
Every figure present but max_drift unexpectedly non-zero	A $merge/$out or moderate-level write reintroduced non-compliant documents that bypass the validator	Note the exception in outcome; remediate the offending ingest path
signature fails to verify on re-check	The artifact bytes changed after signing, or the wrong key was used	Re-derive the signature from the stored key; a genuine mismatch is a tamper finding, not a bug to paper over
Aggregation returns an empty result set	Wrong period boundaries or a timezone-naive parameter	Confirm period_start/period_end are timezone-aware UTC datetimes matching the stored window_start type

Step-by-Step Playbook

Run the export at the close of each reporting window. The sequence reads the samples, decides the outcome, seals the bytes, and writes both the artifact and a record that it was produced.

1. Aggregate the period. Roll every signal up per collection with its sample-window count:

def rollup(evidence, period_start, period_end):
    pipeline = [
        {"$match": {"window_start": {"$gte": period_start, "$lt": period_end}}},
        {"$group": {
            "_id": "$collection",
            "total_rejections": {"$sum": {"$cond": [{"$eq": ["$signal", "rejections"]}, "$value", 0]}},
            "total_warnings":   {"$sum": {"$cond": [{"$eq": ["$signal", "warnings"]},   "$value", 0]}},
            "max_drift":        {"$max": {"$cond": [{"$eq": ["$signal", "drift"]},      "$value", 0]}},
            "sample_windows":   {"$sum": 1},
        }},
        {"$sort": {"_id": 1}},
    ]
    return list(evidence.aggregate(pipeline))
# Expected output: one dict per audited collection with summed figures

2. Assemble the self-describing report. Wrap the rollup with the control description, period, and a derived outcome:

from datetime import datetime, timezone

def assemble(rows, period_start, period_end):
    collections = []
    outcome = "pass"
    for r in rows:
        residual = r["max_drift"] > 0
        if residual:
            outcome = "exception"
        collections.append({**r, "collection": r.pop("_id"), "residual_drift": residual})
    return {
        "control": "MongoDB $jsonSchema write-path validation (processing integrity)",
        "period_start": period_start.isoformat(),
        "period_end": period_end.isoformat(),
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "collections": collections,
        "outcome": outcome,
    }

3. Sign and serialize. Canonicalize the JSON (sorted keys, no incidental whitespace) and sign it with a key held outside the database, so the signature covers the exact bytes an auditor will read:

import hashlib, hmac, json

def sign_report(report: dict, secret: bytes) -> dict:
    canonical = json.dumps(report, sort_keys=True, separators=(",", ":")).encode("utf-8")
    report["signature"] = hmac.new(secret, canonical, hashlib.sha256).hexdigest()
    report["signature_alg"] = "HMAC-SHA256"
    return report
# The signature is computed BEFORE the signature field is added, so it is reproducible.

4. Write the artifact and record its production. Persist the JSON (a CSV flattening of collections is trivial to add) and insert an append-only note that the export ran, so the export itself is auditable:

def export(report: dict, path: str, evidence):
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(report, fh, indent=2, sort_keys=True)
    evidence.database["evidence_exports"].insert_one({
        "period_start": report["period_start"], "period_end": report["period_end"],
        "outcome": report["outcome"], "signature": report["signature"],
        "exported_at": datetime.now(timezone.utc), "artifact_path": path,
    })
# Expected output: a JSON file at `path` plus one evidence_exports record

Failure Modes & Rollback

Each stage fails in a way you can bound and recover from:

• Incomplete series at aggregation (step 1). The rollup sums whatever samples exist, so a gap produces a confidently wrong figure. Run the gap check from the diagnostics section before trusting the outcome; if a series is short, either backfill or annotate the artifact with the gap. Time-to-recover: one backfill pass, or immediate if you declare the gap.
• Wrong period boundaries (step 2). A timezone-naive or off-by-one boundary silently excludes samples. Assert both boundaries are UTC-aware and that period_end is exclusive before assembling. Recovery is re-running the export — it is a pure read, so it is safe to repeat.
• Key mismatch at signing (step 3). Signing with a rotated key makes later verification fail. Store the key id alongside the signature so verification always selects the right key. Recovery: re-sign from the stored canonical form with the correct key.
• Partial write of the artifact (step 4). A crash mid-write leaves a truncated file. Write to a temporary path and atomically rename on success, so a half-written artifact never masquerades as evidence.

Rollback never deletes an issued artifact — a superseded report is retained and a corrected one issued alongside it, with the evidence_exports record showing both, because destroying a produced artifact is itself an audit finding. If an export was generated from an incomplete series, mark it superseded in evidence_exports and re-export once the series is backfilled:

db.compliance.evidence_exports.update_one(
    {"period_start": ps, "period_end": pe, "signature": bad_sig},
    {"$set": {"superseded_at": datetime.now(timezone.utc), "superseded_reason": "series gap"}},
)
# Expected output: matched_count 1, modified_count 1 — the original record is preserved, not removed

Frequently Asked Questions

Related

• SOC 2 Evidence from Validation Telemetry — the parent workflow that samples the metrics this job exports.
• Schema Validation for Compliance & Audit Readiness — the overarching compliance section this export serves.
• Audit Log Schema Contracts — enforcing the immutable audit records that give the exported evidence its integrity guarantees.
• Tracking Validation Failures with MongoDB Atlas Alerts — the real-time alerting view that complements this point-in-time artifact.