Latency architecture for enterprise LLM deployment at scale — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · SOC 2 Type II · layout diagram_first · implementation executed in CI before publish

The control that resolves this is a Tenant-Fair Inference Tier, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The specific problem here is latency and percentile SLAs. The shape below holds p95 ≤ 450ms over REST over HTTPS, and ships with a standard-library implementation that runs as written.

The request path

Tenant-Fair Inference Tier — the request path, with the decision points that must exist before the first model call:

flowchart TB
    subgraph Tenants
      TA[Tenant A - batch]
      TB[Tenant B - interactive p95 400ms]
    end
    TA --> AD{Admission}
    TB --> AD
    AD -->|bucket empty| S429[429 + Retry-After]
    AD -->|ceiling full,<br/>outside reservation| S429
    AD -->|admit| Q[Deadline queue]
    Q -->|deadline passed| SH[Shed - do not serve late]
    Q --> W[Worker pool]
    W --> P[Model / vLLM]
    P --> OBS[Per-tenant p95 / p99 window]
    OBS -->|breach| ALERT[Page + autoscale signal]

Component breakdown

1. Per-tenant token bucket — Sustained rate plus burst, so one tenant cannot spend another's budget.
2. Reservation ledger — Guaranteed in-flight slots per tenant; reservations must sum below the ceiling.
3. Global concurrency ceiling — Bounds the shared tier so it degrades predictably instead of collapsing.
4. Percentile window — Streaming p95/p99 per tenant — the number the contract is written on.
5. Queue with deadline — Requests carry a deadline and are shed rather than served late.
6. Shed path — 429 with an accurate Retry-After so clients back off instead of hot-looping.

Deployment parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The architecture friction point

A shared inference tier holds under average load and breaches its p95 under a neighbour's batch job. The SLA is written on a percentile, the dashboards show a mean, and admission control is a single global rate limit that protects the provider rather than any individual tenant.

What makes latency and percentile SLAs specifically hard

The contract is written on a percentile and the dashboard shows a mean, so the breach is invisible until a customer reports it. Worse, the slowest requests are usually the most valuable ones — long context, more retrieval, more tool calls.

What resolves it is a streaming per-tenant percentile window with alerting on the percentile. The failure to design against: a p95 breach hidden under a healthy-looking average.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

Implementation artifact — Deadline propagation and percentile shedding

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind REST over HTTPS; the failure mode it guards against is a retried POST creating a duplicate record. The output you hand the assessor is the observation-window evidence export.

"""Deadline propagation and percentile-aware shedding.

Why this shape: an SLA is written on a percentile, so the system has to make
per-request decisions against a deadline rather than optimise an average. Two
controls, and the second is the one teams skip:

  * every request carries a DEADLINE, and work stops when it passes -- serving
    a response nobody is waiting for costs capacity and helps no one
  * the queue is bounded by the deadline, not by a fixed length, so load sheds
    at the door instead of accumulating into a latency avalanche
"""
import time
from dataclasses import dataclass, field


class DeadlineExceeded(Exception):
    pass


@dataclass
class Deadline:
    """Monotonic, because wall-clock jumps during NTP correction and that is
    exactly when you least want the deadline logic to misbehave."""
    at: float

    @classmethod
    def in_ms(cls, ms: float) -> "Deadline":
        return cls(at=time.monotonic() + ms / 1000.0)

    def remaining_ms(self) -> float:
        return max(0.0, (self.at - time.monotonic()) * 1000.0)

    def expired(self) -> bool:
        return time.monotonic() >= self.at

    def budget_for(self, fraction: float) -> float:
        """Sub-budget for one stage, so retrieval cannot eat the whole SLA."""
        return self.remaining_ms() * fraction


class PercentileWindow:
    """Bounded streaming window. Nearest-rank, because interpolation invents
    a value that never occurred -- unhelpful when arguing about a breach."""

    def __init__(self, size: int = 4096):
        self.size, self.samples = size, []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        o = sorted(self.samples)
        idx = min(len(o) - 1, max(0, int(round(q * len(o) + 0.5)) - 1))
        return o[idx]


@dataclass
class DeadlineQueue:
    """Admission bounded by whether the work can still finish in time."""
    service_time_ms: float
    concurrency: int
    inflight: int = 0
    shed: int = 0
    admitted: int = 0
    window: PercentileWindow = field(default_factory=PercentileWindow)

    def offer(self, deadline: Deadline) -> bool:
        queue_ahead = max(0, self.inflight - self.concurrency + 1)
        projected = (queue_ahead * self.service_time_ms
                     / max(self.concurrency, 1)) + self.service_time_ms
        if projected > deadline.remaining_ms():
            # Shed NOW. Accepting work that cannot meet its deadline converts
            # one late response into a queue of late responses.
            self.shed += 1
            return False
        self.inflight += 1
        self.admitted += 1
        return True

    def complete(self, elapsed_ms: float):
        self.inflight = max(0, self.inflight - 1)
        self.window.observe(elapsed_ms)

    def breaching(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.window.quantile(q) > slo_ms


if __name__ == "__main__":
    q = DeadlineQueue(service_time_ms=120, concurrency=8)
    for i in range(60):
        d = Deadline.in_ms(400 if i % 3 else 150)   # mixed deadline classes
        if q.offer(d):
            q.complete(110 + (i % 7) * 9)
    print(f"admitted {q.admitted}  shed {q.shed}")
    print(f"p95 {q.window.quantile(0.95):.0f}ms  p99 {q.window.quantile(0.99):.0f}ms")
    print("breaching 400ms SLO:", q.breaching(400))

    tight = Deadline.in_ms(20)
    q.inflight = 40
    print("tight deadline under load admitted:", q.offer(tight), "(must be False)")
    print("stage budget of a 400ms deadline:",
          f"{Deadline.in_ms(400).budget_for(0.3):.0f}ms for retrieval")

Controls and acceptance parameters

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Questions to answer before approval

• Show me the control-execution population for the full window, and the days where it is empty.
  Every request emits an execution record, so the population is continuous by construction. A gap is an alertable incident, not something discovered during the audit.
• Is your alert on a percentile or a mean, and which percentile is in the contract?
  The alert is on the contracted percentile, computed per tenant over a bounded streaming window. The mean is displayed but never alertable, because it hides exactly the breach we are exposed to.
• Which tenants have contractual percentile targets, and what reserves capacity for them?
  Any tenant with a contractual percentile gets an entry in the reservation ledger — guaranteed in-flight slots inside the global ceiling. Reservations are validated at configuration time and must sum below the ceiling.
• What happens to tenant B's p95 when tenant A sends 10x its normal volume — measured, not assumed?
  Tenant B stays inside its SLO, because tenant A is bounded by its own token bucket and cannot consume B's reserved slots. This is testable: saturate A and watch B's p95 window.
• Is the alert on a percentile or a mean?
  A percentile. The SLA is written on p95, so the alert is on p95. The mean is dashboard-only, because it hides exactly the breach you are contractually exposed to.
• What is shed under saturation, and how does a client learn when to retry?
  Requests past their deadline are shed rather than served late, and the shed path returns 429 with an accurate Retry-After computed from the bucket's refill rate so clients back off instead of hot-looping.
• Can a cache entry created for one tenant ever be served to another?
  No. Every cache key is namespaced by tenant id; keying on prompt text alone is the mechanism by which one tenant's completion is served to another.

Where this friction was observed

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• BasaltCRM – The Open Source AI-Native CRM Built on Next.js 16 hackernews · hn/comment · 2026-02-23
• [dead] hackernews · hn/comment · 2026-03-04
• [RFC]: Support Dynamic Model Switching and Flexible Collective Communication in External Launcher Mode github · vllm-project/vllm · 2026-03-26
• [Bug]: bench serve latency metrics exclude client-side queueing under --max-concurrency github · vllm-project/vllm · 2026-08-27
• [Bug]: Gemma4 Fast Prefill Optimization degrades p95 inter-token latency significantly github · vllm-project/vllm · 2026-04-30
• docs: Weekly docs gap audit — 2026-08-19 github · Arize-ai/phoenix · 2026-08-19
• [Bug]: Non-monotonic latency at ctx=8192: concurrency 1 slower than concurrency 2 (local vLLM) github · vllm-project/vllm · 2026-03-16
• perf: 4-7x latency regression from v1.101.0 to v1.105.0 under load (new middleware + auth prefetch + double Redis calls) github · BerriAI/litellm · 2026-10-13

See also

• Part of enterprise llm architecture at scale.
• latency integration patterns for enterprise AI agents covers the adjacent case.
• For the related constraint, see latency compliance data governance framework for enterprise LLM deployment.
• This pattern also governs on-premises RBAC architecture for enterprise LLM deployment at scale.
• enterprise LLM deployment at scale covers the adjacent case.
• If this is the blocker in front of you: enterprise AI infrastructure review.