FedRAMP Moderate tracing and token accounting architecture for enterprise LLM deployment — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · FedRAMP Moderate · layout diagram_first · implementation executed in CI before publish

The control that resolves this is a Complete-Trace Reconciliation Ledger, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is FedRAMP Moderate, and the question it actually asks is which NIST 800-53 control does each component satisfy. The shape below holds p95 ≤ 600ms over REST over HTTPS, and ships with a standard-library implementation that runs as written.

System architecture

Complete-Trace Reconciliation Ledger — the request path, with the decision points that must exist before the first model call:

flowchart LR
    REQ[Request] --> OPEN[Open trace<br/>model version pinned]
    OPEN --> CTX[Bind prompt digest + sources]
    CTX --> CALL[Model call] --> OUT[Bind output digest + tokens]
    OUT --> LED[(Ledger)]
    BILL[Provider invoice] --> REC{Reconcile}
    LED --> REC
    REC -->|gap| ALERT[Untraced traffic — find the path]
    REC -->|matches| OK[Completeness holds]

Modular implementation framework

1. Per-request trace — Model version, prompt digest, output digest, sources, tokens.
2. Completeness check — A trace missing any required field is reported, not accepted.
3. Source citation list — Which retrieved documents were in context for this answer.
4. Token accounting — Input and output tokens recorded at the call site.
5. Spend reconciliation — Traced tokens against billed tokens; a gap means untraced traffic.
6. Reconstruction API — Given a request id, return everything needed to re-derive the answer.

Enterprise implementation parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Why this blocks production

An answer is disputed weeks after it was given and nobody can say which model version, which documents and which prompt produced it. Traces exist, but they are sampled, so the disputed request is the one that was not kept.

What FedRAMP Moderate specifically requires

FedRAMP is control-by-control. Every component must map to a specific NIST 800-53 Rev 5 control identifier with an implementation statement, and anything crossing the authorisation boundary needs its own justification in the SSP.

The artifact that satisfies it is a control implementation summary mapped to control identifiers. The way teams most often fail it: a component outside the authorisation boundary that nobody documented, found during assessment.

Implementation artifact — Trace completeness with spend reconciliation

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind REST over HTTPS; the failure mode it guards against is a retried POST creating a duplicate record. The output you hand the assessor is the SSP control implementation summary.

"""Trace completeness: every response tied to the inputs that produced it.

The failure: an answer is disputed weeks later and nobody can reconstruct which
model version, which retrieved documents and which prompt produced it. Sampling
traces at 1% guarantees the disputed one is missing, so completeness is the
property to enforce — and untraced spend is the signal that it has broken.
"""
import hashlib


class Trace:
    def __init__(self, request_id: str, model_version: str):
        self.request_id = request_id
        self.model_version = model_version
        self.sources: list[str] = []
        self.prompt_digest = None
        self.output_digest = None
        self.tokens = {"in": 0, "out": 0}

    def bind_prompt(self, prompt: str) -> None:
        self.prompt_digest = hashlib.sha256(prompt.encode()).hexdigest()[:16]

    def bind_output(self, output: str) -> None:
        self.output_digest = hashlib.sha256(output.encode()).hexdigest()[:16]

    def cite(self, source_id: str) -> None:
        self.sources.append(source_id)

    def complete(self) -> bool:
        return all([self.prompt_digest, self.output_digest,
                    self.model_version, self.tokens["in"] > 0])

    def missing(self) -> list[str]:
        gaps = []
        if not self.prompt_digest: gaps.append("prompt_digest")
        if not self.output_digest: gaps.append("output_digest")
        if not self.model_version: gaps.append("model_version")
        if self.tokens["in"] <= 0: gaps.append("token_accounting")
        return gaps


class Ledger:
    """Reconciles traced spend against billed spend. A gap means requests are
    escaping the traced path, which is the thing to find — not a rounding
    error to tolerate."""

    def __init__(self, tolerance: float = 0.01):
        self.traces: dict[str, Trace] = {}
        self.tolerance = tolerance

    def open(self, request_id: str, model_version: str) -> Trace:
        t = Trace(request_id, model_version)
        self.traces[request_id] = t
        return t

    def reconstruct(self, request_id: str) -> dict | None:
        t = self.traces.get(request_id)
        if not t:
            return None
        return {"request_id": t.request_id, "model_version": t.model_version,
                "prompt_digest": t.prompt_digest, "output_digest": t.output_digest,
                "sources": list(t.sources), "tokens": dict(t.tokens),
                "complete": t.complete(), "missing": t.missing()}

    def traced_tokens(self) -> int:
        return sum(t.tokens["in"] + t.tokens["out"] for t in self.traces.values())

    def reconcile(self, billed_tokens: int) -> dict:
        traced = self.traced_tokens()
        gap = billed_tokens - traced
        ratio = gap / billed_tokens if billed_tokens else 0.0
        return {"traced": traced, "billed": billed_tokens, "gap": gap,
                "untraced_ratio": round(ratio, 4),
                "within_tolerance": abs(ratio) <= self.tolerance,
                "incomplete_traces": [r for r, t in self.traces.items()
                                      if not t.complete()]}


if __name__ == "__main__":
    led = Ledger()
    t = led.open("req-1", "gpt-x@2026-09")
    t.bind_prompt("What is our refund policy?")
    t.cite("doc://policies/refunds#v7")
    t.tokens = {"in": 820, "out": 240}
    t.bind_output("Refunds are processed within 14 days.")

    rec = led.reconstruct("req-1")
    assert rec["complete"], rec
    assert rec["sources"] == ["doc://policies/refunds#v7"], rec
    assert rec["model_version"] == "gpt-x@2026-09"

    # An incomplete trace must be visible, not silently acceptable.
    t2 = led.open("req-2", "gpt-x@2026-09")
    t2.tokens = {"in": 100, "out": 50}
    assert not t2.complete() and "prompt_digest" in t2.missing(), t2.missing()

    clean = led.reconcile(billed_tokens=1210)
    assert clean["within_tolerance"], clean
    assert clean["incomplete_traces"] == ["req-2"], clean

    # Requests escaping the traced path show up as an untraced gap.
    leaky = led.reconcile(billed_tokens=4000)
    assert not leaky["within_tolerance"], leaky
    assert leaky["untraced_ratio"] > 0.6, leaky

    assert led.reconstruct("req-missing") is None

    print("Trace completeness ledger: OK")
    print(f"  reconstructed req-1 complete={rec['complete']}, "
          f"untraced ratio clean={clean['untraced_ratio']} "
          f"leaky={leaky['untraced_ratio']}")

The FedRAMP Moderate control, as code

NIST 800-53 control map with a boundary-crossing check. This is the artifact an assessor asks for under FedRAMP Moderate specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""NIST 800-53 control satisfaction map with an authorisation-boundary check.

FedRAMP reviewers do not ask whether the system is secure. They ask which
control, by identifier, each component satisfies, and what crosses the
authorisation boundary. An external model endpoint is a boundary crossing, and
an undeclared crossing is the finding that stalls an ATO.
"""

MODERATE_BASELINE = {
    "AC-4": "Information flow enforcement",
    "AU-2": "Event logging",
    "AU-9": "Protection of audit information",
    "CA-3": "Information exchange / external connections",
    "SC-7": "Boundary protection",
    "SC-8": "Transmission confidentiality and integrity",
    "SC-28": "Protection of information at rest",
    "SI-4": "System monitoring",
}


class AuthorisationBoundary:
    def __init__(self, name: str):
        self.name = name
        self.components: dict[str, dict] = {}
        self.crossings: list[dict] = []

    def declare(self, component: str, inside: bool, controls: list[str]) -> None:
        unknown = [c for c in controls if c not in MODERATE_BASELINE]
        if unknown:
            raise KeyError(f"not in the Moderate baseline: {unknown}")
        self.components[component] = {"inside": inside, "controls": set(controls)}

    def connect(self, src: str, dst: str, declared: bool) -> None:
        for end in (src, dst):
            if end not in self.components:
                raise KeyError(f"undeclared component: {end}")
        crossing = self.components[src]["inside"] != self.components[dst]["inside"]
        self.crossings.append({"src": src, "dst": dst,
                               "boundary_crossing": crossing, "declared": declared})

    def undeclared_crossings(self) -> list[dict]:
        return [c for c in self.crossings if c["boundary_crossing"] and not c["declared"]]

    def coverage(self) -> dict:
        covered = set()
        for meta in self.components.values():
            covered |= meta["controls"]
        gaps = sorted(set(MODERATE_BASELINE) - covered)
        return {"covered": sorted(covered), "gaps": gaps,
                "ato_ready": not gaps and not self.undeclared_crossings()}


if __name__ == "__main__":
    b = AuthorisationBoundary("agency-llm-enclave")
    b.declare("gateway", True, ["AC-4", "SC-7", "SC-8"])
    b.declare("audit_store", True, ["AU-2", "AU-9", "SC-28"])
    b.declare("monitor", True, ["SI-4"])
    b.declare("external_model_api", False, ["CA-3"])

    b.connect("gateway", "audit_store", declared=True)
    b.connect("gateway", "external_model_api", declared=True)
    cov = b.coverage()
    assert cov["ato_ready"], cov
    assert cov["gaps"] == [], cov

    # The finding this exists to catch: a crossing nobody declared.
    b2 = AuthorisationBoundary("agency-llm-enclave")
    b2.declare("gateway", True, list(MODERATE_BASELINE))
    b2.declare("telemetry_sidecar", False, ["CA-3"])
    b2.connect("gateway", "telemetry_sidecar", declared=False)
    assert not b2.coverage()["ato_ready"], b2.coverage()
    assert len(b2.undeclared_crossings()) == 1, b2.undeclared_crossings()

    try:
        b2.declare("rogue", True, ["XX-1"])
        raise AssertionError("accepted a control outside the baseline")
    except KeyError:
        pass

    print("FedRAMP Moderate boundary + control map: OK")
    print(f"  controls covered {len(cov['covered'])}/{len(MODERATE_BASELINE)}, "
          f"undeclared crossings {len(b2.undeclared_crossings())}")

Risk and control matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Review questions and answers

• Which NIST 800-53 control does each component satisfy, and which components sit outside the authorisation boundary?
  Each component carries its control identifiers in configuration, and the boundary diagram is generated from that configuration so it cannot drift from reality.
• A customer disputes an answer from six weeks ago. What can you reconstruct?
  The model version, prompt digest, output digest, the retrieved sources in context, and token counts — enough to re-derive the answer rather than argue about it.
• How would you know if some requests never reach the traced path?
  Reconciliation. Traced token spend is compared against the provider invoice; a gap beyond tolerance means traffic is escaping the traced path, and that gap is the thing to chase.
• Which model version produced a specific output, and how is that known rather than inferred?
  It is pinned on the trace when the trace opens, before the call. Inferring it afterwards from deployment history is exactly how the wrong version gets attributed.
• Does the trace store the raw prompt, and what does that mean for personal data?
  Digests by default, with raw retained only where there is a lawful basis. The digest is enough to prove which prompt produced which output without storing the text.
• What does tracing add to the latency of the request it traces?
  Nothing on the critical path: trace writes are buffered and asynchronous, and the added p95 is measured against the agreed budget rather than assumed to be negligible.

Related reference architectures

• Part of enterprise ai compliance architecture.
• fedRAMP Moderate provider failover and degradation architecture applies the same mechanism under a different obligation.
• A different failure mode of the same component: fedRAMP Moderate cost control architecture for enterprise LLM deployment.
• For the related constraint, see fedRAMP Moderate legacy integration patterns for enterprise AI agents.
• For the related constraint, see fedRAMP Moderate agent permission and delegated access architecture.
• If this is the blocker in front of you: enterprise AI infrastructure review.