EU AI Act cost control architecture for enterprise LLM deployment — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · EU AI Act · layout walkthrough · implementation executed in CI before publish

The control that resolves this is a Pre-Flight Budget Governor, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is EU AI Act, and the question it actually asks is what risk class is this system, and where is the technical documentation. The shape below holds p95 ≤ 900ms over gRPC with mTLS, and ships with a standard-library implementation that runs as written.

The control gap

LLM spend is discovered in a monthly invoice and defended with a dashboard, so it is reported rather than controlled. Finance asks for a per-team cap and engineering cannot implement one, because nothing in the request path knows a budget exists.

What EU AI Act specifically requires

The EU AI Act regulates the SYSTEM, not the data. The first question is its risk classification, and a high-risk classification pulls in Article 11 technical documentation, Article 12 record-keeping and Article 14 human oversight — obligations that have to be designed in, not attached later.

The artifact that satisfies it is a model card plus Article 12 automatic logs. The way teams most often fail it: a system classified as limited-risk that is actually high-risk because it informs a decision about a person.

Component breakdown

1. Conservative estimator — Deliberately over-estimates; under-estimating makes the ceiling decorative.
2. Two-phase budget — Reserve the estimate, settle the actual — closes the concurrency race.
3. Rate table in config — Prices change; a stale rate in code silently misprices every chargeback.
4. Tier router — Routes to the cheapest tier that meets the quality bar for the task.
5. Chargeback ledger — Per-tenant, per-request actuals — the artifact finance reconciles against.
6. Fail-closed default — A tenant with no budget is refused, never admitted.

Working implementation — Two-phase token budget with per-tenant chargeback

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind gRPC with mTLS; the failure mode it guards against is a half-open HTTP/2 stream silently dropping the deadline. The output you hand the assessor is the Annex III classification record and model card.

"""Pre-flight token budget enforcement with per-tenant chargeback.

Why this shape: LLM spend is discovered monthly in an invoice and defended with
a dashboard, which means it is never actually controlled. The control has to run
BEFORE the call, on an estimate, with a hard ceiling per tenant and per period —
otherwise the first thing finance asks for is a cap you cannot implement.
"""
import math
import time
from dataclasses import dataclass, field


# USD per 1k tokens. Keep rates in config, not code: they change, and a stale
# rate silently misprices every chargeback downstream of it.
RATES = {
    "frontier":  {"in": 0.0030, "out": 0.0150},
    "mid":       {"in": 0.0008, "out": 0.0040},
    "small":     {"in": 0.0001, "out": 0.0004},
}


def estimate_tokens(text: str, chars_per_token: float = 3.8) -> int:
    """Deliberately a conservative OVER-estimate.

    Under-estimating defeats the purpose: the request gets admitted, the real
    usage exceeds the budget, and the ceiling is decorative."""
    return max(1, math.ceil(len(text or "") / chars_per_token))


def price(tier: str, tokens_in: int, tokens_out: int) -> float:
    r = RATES[tier]
    return (tokens_in / 1000.0) * r["in"] + (tokens_out / 1000.0) * r["out"]


class BudgetExceeded(Exception):
    def __init__(self, tenant, projected, ceiling):
        self.tenant, self.projected, self.ceiling = tenant, projected, ceiling
        super().__init__(f"{tenant}: ${projected:.4f} would exceed "
                         f"${ceiling:.2f} for the period")


@dataclass
class Budget:
    ceiling_usd: float
    period_seconds: float = 86400.0
    spent_usd: float = 0.0
    started: float = field(default_factory=time.time)
    committed: dict = field(default_factory=dict)   # request_id -> reserved USD

    def _roll(self, now):
        if now - self.started >= self.period_seconds:
            self.started, self.spent_usd = now, 0.0
            self.committed.clear()

    def reserve(self, request_id: str, projected_usd: float, now=None) -> float:
        """Two-phase: reserve the estimate, settle the actual.

        Without a reservation, N concurrent requests each see the same low
        `spent` and all get admitted — the classic race that blows a cap."""
        now = time.time() if now is None else now
        self._roll(now)
        outstanding = sum(self.committed.values())
        if self.spent_usd + outstanding + projected_usd > self.ceiling_usd:
            raise BudgetExceeded("tenant",
                                 self.spent_usd + outstanding + projected_usd,
                                 self.ceiling_usd)
        self.committed[request_id] = projected_usd
        return projected_usd

    def settle(self, request_id: str, actual_usd: float):
        self.committed.pop(request_id, None)
        self.spent_usd += actual_usd

    def release(self, request_id: str):
        self.committed.pop(request_id, None)


class Governor:
    def __init__(self):
        self.budgets: dict = {}
        self.ledger: list = []

    def set_budget(self, tenant: str, ceiling_usd: float, period_seconds=86400.0):
        self.budgets[tenant] = Budget(ceiling_usd, period_seconds)

    def admit(self, tenant: str, request_id: str, prompt: str,
              tier: str = "mid", max_output_tokens: int = 1024) -> dict:
        b = self.budgets.get(tenant)
        if b is None:
            raise KeyError(f"{tenant} has no budget: fail closed, never open")
        t_in = estimate_tokens(prompt)
        projected = price(tier, t_in, max_output_tokens)
        b.reserve(request_id, projected)
        return {"tenant": tenant, "request_id": request_id, "tier": tier,
                "tokens_in_est": t_in, "projected_usd": round(projected, 6)}

    def record(self, tenant: str, request_id: str, tier: str,
               tokens_in: int, tokens_out: int) -> dict:
        actual = price(tier, tokens_in, tokens_out)
        self.budgets[tenant].settle(request_id, actual)
        row = {"tenant": tenant, "request_id": request_id, "tier": tier,
               "tokens_in": tokens_in, "tokens_out": tokens_out,
               "actual_usd": round(actual, 6), "ts": time.time()}
        self.ledger.append(row)     # this is the chargeback record
        return row


if __name__ == "__main__":
    g = Governor()
    g.set_budget("bank-eu", ceiling_usd=0.05)
    adm = g.admit("bank-eu", "req-1", "Reconcile these 400 journal lines." * 20)
    print("admitted:", adm)
    print("settled :", g.record("bank-eu", "req-1", "mid", adm["tokens_in_est"], 310))
    try:
        for i in range(40):
            g.admit("bank-eu", f"burst-{i}", "x" * 4000, tier="frontier")
    except BudgetExceeded as e:
        print("ceiling held:", e)

The EU AI Act control, as code

Annex IV documentation completeness and Article 12 logging. This is the artifact an assessor asks for under EU AI Act specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Annex IV technical documentation completeness + Article 12 logging.

A high-risk system under the EU AI Act needs technical documentation covering
specified elements, and automatic recording of events over the system lifetime.
The failure is rarely refusal to document — it is documentation that is stale,
partial, or cannot be tied to the exact model version that produced an output.

This checks the documentation set against the Annex IV elements and refuses to
mark a release conformant while any element is missing or stale.
"""
from datetime import date, timedelta

ANNEX_IV_ELEMENTS = {
    "general_description": "intended purpose, provider, version",
    "design_specification": "architecture, algorithmic logic, design choices",
    "training_data": "provenance, scope, characteristics, labelling",
    "human_oversight": "measures enabling natural-person oversight",
    "accuracy_metrics": "accuracy, robustness and cybersecurity metrics",
    "risk_management": "risk management system per Article 9",
    "change_log": "changes made through the lifecycle",
    "post_market_plan": "post-market monitoring plan",
}

STALENESS_DAYS = 180


class TechnicalFile:
    def __init__(self, system: str, model_version: str, today: date):
        self.system, self.model_version, self.today = system, model_version, today
        self.elements: dict[str, date] = {}

    def attach(self, element: str, updated: date) -> None:
        if element not in ANNEX_IV_ELEMENTS:
            raise KeyError(f"{element} is not an Annex IV element")
        self.elements[element] = updated

    def missing(self) -> list[str]:
        return sorted(set(ANNEX_IV_ELEMENTS) - set(self.elements))

    def stale(self) -> list[str]:
        cutoff = self.today - timedelta(days=STALENESS_DAYS)
        return sorted(k for k, v in self.elements.items() if v < cutoff)

    def conformant(self) -> dict:
        missing, stale = self.missing(), self.stale()
        return {"system": self.system, "model_version": self.model_version,
                "missing": missing, "stale": stale,
                "conformant": not missing and not stale}


class EventLog:
    """Article 12: records must tie an output to the version that produced it."""

    def __init__(self):
        self.events: list[dict] = []

    def record(self, model_version: str, event: str, ref: str) -> None:
        self.events.append({"model_version": model_version,
                            "event": event, "ref": ref})

    def trace(self, ref: str) -> dict | None:
        for e in self.events:
            if e["ref"] == ref:
                return e
        return None

    def untraceable(self, known_versions: set[str]) -> list[str]:
        return [e["ref"] for e in self.events
                if e["model_version"] not in known_versions]


if __name__ == "__main__":
    today = date(2026, 10, 16)
    tf = TechnicalFile("claims-triage", "v4.2.0", today)
    for element in ANNEX_IV_ELEMENTS:
        tf.attach(element, today - timedelta(days=30))
    assert tf.conformant()["conformant"], tf.conformant()

    partial = TechnicalFile("claims-triage", "v4.3.0", today)
    for element in list(ANNEX_IV_ELEMENTS)[:5]:
        partial.attach(element, today - timedelta(days=10))
    partial.attach("change_log", today - timedelta(days=400))
    report = partial.conformant()
    assert not report["conformant"], report
    assert "post_market_plan" in report["missing"], report
    assert "change_log" in report["stale"], report

    log = EventLog()
    log.record("v4.2.0", "inference", "out-9001")
    log.record("v4.1.7", "inference", "out-8800")
    assert log.trace("out-9001")["model_version"] == "v4.2.0"
    assert log.untraceable({"v4.2.0"}) == ["out-8800"]

    print("EU AI Act Annex IV + Article 12: OK")
    print(f"  complete file conformant={tf.conformant()['conformant']}, "
          f"partial missing {len(report['missing'])}, stale {len(report['stale'])}")

The request path

Pre-Flight Budget Governor — the request path, with the decision points that must exist before the first model call:

flowchart LR
    REQ[Request] --> EST[Conservative token estimate]
    EST --> PR[Price at tier rate]
    PR --> RES{Reserve against budget<br/>spent + outstanding + projected}
    RES -->|over ceiling| REJ[429 BudgetExceeded]
    RES -->|no budget| FC[Refuse - fail closed]
    RES -->|ok| TR[Tier router<br/>cheapest tier meeting quality bar]
    TR --> M[Model call]
    M --> ST[Settle actual usage]
    ST --> LED[(Chargeback ledger)]
    LED --> FIN[Finance reconciliation]

Deployment parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Controls and acceptance parameters

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Questions to answer before approval

• What is this system's risk classification under Annex III, who decided, and what happens to that classification when the use case changes?
  The classification is recorded against its Annex III category with a named owner, and a material change of use triggers re-classification before the change ships.
• What refuses a request that would exceed this team's monthly cap, and where in the path does it run?
  The budget governor, at admission — before the model call. It prices a conservative token estimate against the tenant's remaining ceiling and refuses the request if the projection would exceed it.
• If 50 requests arrive at once against a nearly-exhausted budget, how many are admitted?
  Only those that fit. Admission counts outstanding reservations as well as settled spend, which closes the race where concurrent requests all read the same low figure and are all admitted.
• How does your chargeback figure reconcile to the provider invoice, and what is the variance?
  The chargeback ledger records per-request actuals priced from a versioned rate table with an effective date; reconcile it monthly and track the variance against the provider invoice.
• What happens to a tenant with no budget configured?
  It is refused. A missing budget fails closed — admitting an unbudgeted tenant is how the cap becomes decorative.
• Which rate table priced last month's chargeback, and when was it last updated?
  The rate table in versioned config, with its effective date recorded. A rate older than 90 days raises an alert, because a stale rate silently misprices every chargeback downstream.

Adjacent problems

• Part of enterprise llm cost control.
• For the related constraint, see EU AI Act prompt injection defence architecture for enterprise LLM.
• Where this interacts with another regime, see EU AI Act reference architecture for enterprise LLM deployment at scale.
• This pattern also governs EU AI Act model release gate and golden set evaluation for enterprise AI.
• EU AI Act vendor evaluation criteria for enterprise AI implementation applies the same mechanism under a different obligation.
• If this is the blocker in front of you: enterprise AI infrastructure review.