P95 cost control architecture for enterprise LLM deployment — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · SOC 2 Type II · layout spec_first · implementation executed in CI before publish

Technical parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The control that resolves this is a Pre-Flight Budget Governor, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The specific problem here is latency and percentile SLAs. The shape below holds p95 ≤ 600ms over durable queue (SQS / Kafka) with an outbox, and ships with a standard-library implementation that runs as written.

Why this blocks production

LLM spend is discovered in a monthly invoice and defended with a dashboard, so it is reported rather than controlled. Finance asks for a per-team cap and engineering cannot implement one, because nothing in the request path knows a budget exists.

What makes latency and percentile SLAs specifically hard

The contract is written on a percentile and the dashboard shows a mean, so the breach is invisible until a customer reports it. Worse, the slowest requests are usually the most valuable ones — long context, more retrieval, more tool calls.

What resolves it is a streaming per-tenant percentile window with alerting on the percentile. The failure to design against: a p95 breach hidden under a healthy-looking average.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

Working implementation — Deadline propagation and percentile shedding

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind durable queue (SQS / Kafka) with an outbox; the failure mode it guards against is a consumer crash between side effect and acknowledgement. The output you hand the assessor is the observation-window evidence export.

"""Deadline propagation and percentile-aware shedding.

Why this shape: an SLA is written on a percentile, so the system has to make
per-request decisions against a deadline rather than optimise an average. Two
controls, and the second is the one teams skip:

  * every request carries a DEADLINE, and work stops when it passes -- serving
    a response nobody is waiting for costs capacity and helps no one
  * the queue is bounded by the deadline, not by a fixed length, so load sheds
    at the door instead of accumulating into a latency avalanche
"""
import time
from dataclasses import dataclass, field


class DeadlineExceeded(Exception):
    pass


@dataclass
class Deadline:
    """Monotonic, because wall-clock jumps during NTP correction and that is
    exactly when you least want the deadline logic to misbehave."""
    at: float

    @classmethod
    def in_ms(cls, ms: float) -> "Deadline":
        return cls(at=time.monotonic() + ms / 1000.0)

    def remaining_ms(self) -> float:
        return max(0.0, (self.at - time.monotonic()) * 1000.0)

    def expired(self) -> bool:
        return time.monotonic() >= self.at

    def budget_for(self, fraction: float) -> float:
        """Sub-budget for one stage, so retrieval cannot eat the whole SLA."""
        return self.remaining_ms() * fraction


class PercentileWindow:
    """Bounded streaming window. Nearest-rank, because interpolation invents
    a value that never occurred -- unhelpful when arguing about a breach."""

    def __init__(self, size: int = 4096):
        self.size, self.samples = size, []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        o = sorted(self.samples)
        idx = min(len(o) - 1, max(0, int(round(q * len(o) + 0.5)) - 1))
        return o[idx]


@dataclass
class DeadlineQueue:
    """Admission bounded by whether the work can still finish in time."""
    service_time_ms: float
    concurrency: int
    inflight: int = 0
    shed: int = 0
    admitted: int = 0
    window: PercentileWindow = field(default_factory=PercentileWindow)

    def offer(self, deadline: Deadline) -> bool:
        queue_ahead = max(0, self.inflight - self.concurrency + 1)
        projected = (queue_ahead * self.service_time_ms
                     / max(self.concurrency, 1)) + self.service_time_ms
        if projected > deadline.remaining_ms():
            # Shed NOW. Accepting work that cannot meet its deadline converts
            # one late response into a queue of late responses.
            self.shed += 1
            return False
        self.inflight += 1
        self.admitted += 1
        return True

    def complete(self, elapsed_ms: float):
        self.inflight = max(0, self.inflight - 1)
        self.window.observe(elapsed_ms)

    def breaching(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.window.quantile(q) > slo_ms


if __name__ == "__main__":
    q = DeadlineQueue(service_time_ms=120, concurrency=8)
    for i in range(60):
        d = Deadline.in_ms(400 if i % 3 else 150)   # mixed deadline classes
        if q.offer(d):
            q.complete(110 + (i % 7) * 9)
    print(f"admitted {q.admitted}  shed {q.shed}")
    print(f"p95 {q.window.quantile(0.95):.0f}ms  p99 {q.window.quantile(0.99):.0f}ms")
    print("breaching 400ms SLO:", q.breaching(400))

    tight = Deadline.in_ms(20)
    q.inflight = 40
    print("tight deadline under load admitted:", q.offer(tight), "(must be False)")
    print("stage budget of a 400ms deadline:",
          f"{Deadline.in_ms(400).budget_for(0.3):.0f}ms for retrieval")

The SOC 2 Type II control, as code

Type II evidence sampling over the observation window. This is the artifact an assessor asks for under SOC 2 Type II specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""SOC 2 Type II evidence sampling over an observation window.

Type II is not "the control exists" — it is "the control operated throughout
the period". An auditor samples days from the window and asks for proof the
control ran on each sampled day. A gap of one day with no evidence is an
exception, so the population must be built from operational records rather
than asserted.
"""
import hashlib
from datetime import date, timedelta


class ObservationWindow:
    """The population an auditor samples from, plus deterministic selection."""

    def __init__(self, start: date, end: date, control_id: str):
        if end < start:
            raise ValueError("window end precedes start")
        self.start, self.end, self.control_id = start, end, control_id
        self._evidence: dict[date, list[str]] = {}

    def record(self, day: date, artifact_ref: str) -> None:
        if not (self.start <= day <= self.end):
            raise ValueError(f"{day} outside observation window")
        self._evidence.setdefault(day, []).append(artifact_ref)

    def population(self) -> list[date]:
        n = (self.end - self.start).days + 1
        return [self.start + timedelta(days=i) for i in range(n)]

    def gaps(self) -> list[date]:
        """Days with no evidence. Each one is a Type II exception."""
        return [d for d in self.population() if not self._evidence.get(d)]

    def sample(self, size: int) -> list[date]:
        """Deterministic, reproducible selection — an auditor must be able to
        re-derive the same sample from the same seed rather than trust ours."""
        pop = self.population()
        size = min(size, len(pop))
        seed = f"{self.control_id}:{self.start}:{self.end}"
        scored = sorted(
            pop, key=lambda d: hashlib.sha256(f"{seed}:{d}".encode()).hexdigest())
        return sorted(scored[:size])

    def evidence_for(self, day: date) -> list[str]:
        return list(self._evidence.get(day, []))

    def attestation(self, size: int = 25) -> dict:
        sampled = self.sample(size)
        missing = [d for d in sampled if not self._evidence.get(d)]
        return {
            "control_id": self.control_id,
            "window": f"{self.start}..{self.end}",
            "population_days": len(self.population()),
            "sampled_days": len(sampled),
            "exceptions": [str(d) for d in missing],
            "opinion": "unqualified" if not missing else "qualified",
        }


if __name__ == "__main__":
    start, end = date(2026, 1, 1), date(2026, 3, 31)
    w = ObservationWindow(start, end, "CC6.1-tenant-isolation")
    for d in w.population():
        w.record(d, f"s3://evidence/{d}/isolation-check.json")
    clean = w.attestation()
    assert clean["opinion"] == "unqualified", clean
    assert clean["population_days"] == 90, clean

    # One missing day is one exception — and it must survive sampling to matter.
    w2 = ObservationWindow(start, end, "CC6.1-tenant-isolation")
    sampled = set(w2.sample(25))
    for d in w2.population():
        if d != sorted(sampled)[0]:
            w2.record(d, "ok")
    assert w2.attestation()["opinion"] == "qualified", w2.attestation()
    assert len(w2.gaps()) == 1, w2.gaps()

    # The sample must be reproducible from the same seed.
    assert w.sample(25) == ObservationWindow(start, end,
                                             "CC6.1-tenant-isolation").sample(25)
    print("SOC 2 Type II evidence sampling: OK")
    print(f"  population {clean['population_days']} days, "
          f"sample {clean['sampled_days']}, opinion {clean['opinion']}")

The blueprint

Pre-Flight Budget Governor — the request path, with the decision points that must exist before the first model call:

flowchart LR
    REQ[Request] --> EST[Conservative token estimate]
    EST --> PR[Price at tier rate]
    PR --> RES{Reserve against budget<br/>spent + outstanding + projected}
    RES -->|over ceiling| REJ[429 BudgetExceeded]
    RES -->|no budget| FC[Refuse - fail closed]
    RES -->|ok| TR[Tier router<br/>cheapest tier meeting quality bar]
    TR --> M[Model call]
    M --> ST[Settle actual usage]
    ST --> LED[(Chargeback ledger)]
    LED --> FIN[Finance reconciliation]

Implementation components

1. Conservative estimator — Deliberately over-estimates; under-estimating makes the ceiling decorative.
2. Two-phase budget — Reserve the estimate, settle the actual — closes the concurrency race.
3. Rate table in config — Prices change; a stale rate in code silently misprices every chargeback.
4. Tier router — Routes to the cheapest tier that meets the quality bar for the task.
5. Chargeback ledger — Per-tenant, per-request actuals — the artifact finance reconciles against.
6. Fail-closed default — A tenant with no budget is refused, never admitted.

Risk mitigation matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Questions to answer before approval

• Show me the control-execution population for the full window, and the days where it is empty.
  Every request emits an execution record, so the population is continuous by construction. A gap is an alertable incident, not something discovered during the audit.
• Is your alert on a percentile or a mean, and which percentile is in the contract?
  The alert is on the contracted percentile, computed per tenant over a bounded streaming window. The mean is displayed but never alertable, because it hides exactly the breach we are exposed to.
• What refuses a request that would exceed this team's monthly cap, and where in the path does it run?
  The budget governor, at admission — before the model call. It prices a conservative token estimate against the tenant's remaining ceiling and refuses the request if the projection would exceed it.
• If 50 requests arrive at once against a nearly-exhausted budget, how many are admitted?
  Only those that fit. Admission counts outstanding reservations as well as settled spend, which closes the race where concurrent requests all read the same low figure and are all admitted.
• How does your chargeback figure reconcile to the provider invoice, and what is the variance?
  The chargeback ledger records per-request actuals priced from a versioned rate table with an effective date; reconcile it monthly and track the variance against the provider invoice.
• What happens to a tenant with no budget configured?
  It is refused. A missing budget fails closed — admitting an unbudgeted tenant is how the cap becomes decorative.
• Which rate table priced last month's chargeback, and when was it last updated?
  The rate table in versioned config, with its effective date recorded. A rate older than 90 days raises an alert, because a stale rate silently misprices every chargeback downstream.

Where this friction was observed

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• Show HN: FaultWall – Which tenant is killing your Postgres? Now you can find out hackernews · hn/story · 2026-03-27
• AX – Google’s Open Agentic Orchestrator hackernews · hn/comment · 2026-09-21

Adjacent problems

• Part of enterprise llm cost control.
• The same control appears in EU AI Act cost control architecture for enterprise LLM deployment.
• The same control appears in fedRAMP Moderate cost control architecture for enterprise LLM deployment.
• SOX cost control architecture for enterprise LLM deployment applies the same mechanism under a different obligation.
• For the related constraint, see SOC 2 Type II cost control architecture for enterprise LLM deployment.
• If this is the blocker in front of you: enterprise AI infrastructure review.