ISO/IEC 27001:2022 model release gate and golden set evaluation for enterprise AI — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · ISO/IEC 27001:2022 · layout diagram_first · implementation executed in CI before publish

The control that resolves this is a Frozen Golden Set Release Gate, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is ISO/IEC 27001:2022, and the question it actually asks is is this risk on the register with a named owner and a treatment. The shape below holds p95 ≤ 600ms over REST over HTTPS, and ships with a standard-library implementation that runs as written.

The blueprint

Frozen Golden Set Release Gate — the request path, with the decision points that must exist before the first model call:

flowchart LR
    CAND[Candidate model] --> SC[Score per slice]
    BASE[Baseline scores] --> CMP{Compare per slice}
    SC --> CMP
    CMP -->|slice regressed| BLOCK[Refuse release]
    CMP -->|below floor| BLOCK
    CMP -->|all slices hold| REL[Release + decision record]
    BLOCK --> REC[Record which slice and by how much]

Implementation components

1. Frozen golden set — Versioned and immutable; editing it after freeze raises.
2. Slice taxonomy — Rare-but-critical cases are their own slice, not noise in the mean.
3. Per-slice thresholds — Tolerance for regression plus an absolute floor per slice.
4. Deterministic scorer — Same inputs, same score — a flaky scorer makes the gate decorative.
5. Release gate — Refuses on any slice regression even when the aggregate improves.
6. Decision record — What was compared, which slices moved, who accepted the risk.

Deployment parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The architecture friction point

A model upgrade is approved on a demo and an aggregate score. Nobody can say which cases got worse, because the comparison is a single number against a test set that changes whenever it is inconvenient.

What ISO/IEC 27001:2022 specifically requires

ISO 27001 is a management-system standard: the auditor checks that risks are identified, owned, treated and reviewed on a cycle. A perfect technical control that is absent from the risk register is a nonconformity.

The artifact that satisfies it is risk register entries with owners, treatments and review dates. The way teams most often fail it: a technically sound control that no Statement of Applicability references, so it cannot be evidenced.

Implementation artifact — Per-slice release gate over a frozen golden set

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind REST over HTTPS; the failure mode it guards against is a retried POST creating a duplicate record. The output you hand the assessor is the Statement of Applicability extract.

"""Release gate on a frozen golden set, with per-slice thresholds.

The failure this prevents: a candidate model improves the aggregate score and
regresses badly on one slice that happens to matter. Aggregates hide that
(Simpson's paradox is not a curiosity here, it is the default outcome), so the
gate evaluates per slice and refuses on any slice regression beyond tolerance
even when the headline number improves.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Case:
    case_id: str
    slice_name: str
    prompt: str
    expected: str


@dataclass
class GoldenSet:
    """Frozen on purpose. A golden set edited to make a release pass is not a
    golden set, so mutation after freeze raises rather than warns."""
    name: str
    cases: list = field(default_factory=list)
    _frozen: bool = False

    def add(self, case: Case) -> None:
        if self._frozen:
            raise RuntimeError(f"{self.name} is frozen; create a new version")
        self.cases.append(case)

    def freeze(self) -> "GoldenSet":
        self._frozen = True
        return self

    def slices(self) -> set:
        return {c.slice_name for c in self.cases}


def score(model, golden: GoldenSet) -> dict:
    per_slice: dict[str, list] = {}
    for case in golden.cases:
        ok = model(case.prompt) == case.expected
        per_slice.setdefault(case.slice_name, []).append(ok)
    return {s: sum(v) / len(v) for s, v in per_slice.items()}


def aggregate(scores: dict, golden: GoldenSet) -> float:
    weights = {s: sum(1 for c in golden.cases if c.slice_name == s)
               for s in scores}
    total = sum(weights.values())
    return sum(scores[s] * weights[s] for s in scores) / total


class ReleaseGate:
    def __init__(self, tolerance: float = 0.02, floors: dict | None = None):
        self.tolerance = tolerance
        self.floors = floors or {}

    def decide(self, baseline: dict, candidate: dict, golden: GoldenSet) -> dict:
        regressions, floor_breaks = [], []
        for s in sorted(set(baseline) | set(candidate)):
            b, c = baseline.get(s, 0.0), candidate.get(s, 0.0)
            if b - c > self.tolerance:
                regressions.append({"slice": s, "baseline": round(b, 3),
                                    "candidate": round(c, 3),
                                    "delta": round(c - b, 3)})
            floor = self.floors.get(s)
            if floor is not None and c < floor:
                floor_breaks.append({"slice": s, "candidate": round(c, 3),
                                     "floor": floor})
        agg_b, agg_c = aggregate(baseline, golden), aggregate(candidate, golden)
        return {
            "aggregate_baseline": round(agg_b, 3),
            "aggregate_candidate": round(agg_c, 3),
            "aggregate_improved": agg_c > agg_b,
            "slice_regressions": regressions,
            "floor_breaches": floor_breaks,
            "release": not regressions and not floor_breaks,
        }


if __name__ == "__main__":
    g = GoldenSet("claims-triage-v1")
    for i in range(40):
        g.add(Case(f"c{i}", "common", f"p{i}", f"a{i}"))
    for i in range(10):
        g.add(Case(f"r{i}", "rare_high_value", f"rp{i}", f"ra{i}"))
    g.freeze()

    baseline_answers = {c.prompt: c.expected for c in g.cases}
    def baseline_model(p): return baseline_answers[p]

    # Candidate: better on the big slice, broken on the small critical one.
    def candidate_model(p):
        if p.startswith("rp"):
            return "wrong"
        return baseline_answers[p]

    base = score(baseline_model, g)
    cand = score(candidate_model, g)
    gate = ReleaseGate(tolerance=0.02, floors={"rare_high_value": 0.90})
    decision = gate.decide(base, cand, g)

    assert decision["aggregate_candidate"] >= 0.79, decision
    assert not decision["release"], "a slice regression must block the release"
    assert decision["slice_regressions"][0]["slice"] == "rare_high_value"
    assert decision["floor_breaches"], decision

    clean = gate.decide(base, base, g)
    assert clean["release"], clean

    try:
        g.add(Case("x", "common", "p", "a"))
        raise AssertionError("edited a frozen golden set")
    except RuntimeError:
        pass

    print("Evaluation release gate: OK")
    print(f"  aggregate {decision['aggregate_baseline']} -> "
          f"{decision['aggregate_candidate']}, release={decision['release']}, "
          f"slice regressions {len(decision['slice_regressions'])}")

The ISO/IEC 27001:2022 control, as code

Statement of Applicability with operating evidence. This is the artifact an assessor asks for under ISO/IEC 27001:2022 specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Annex A control to evidence mapping behind a Statement of Applicability.

ISO/IEC 27001:2022 turns on the SoA: every Annex A control is either applicable
with a justification and evidence, or excluded with a justification. The audit
failure is an applicable control whose evidence is a policy document rather
than a record that the control operated.
"""

ANNEX_A_2022 = {
    "A.5.23": "Information security for use of cloud services",
    "A.8.2": "Privileged access rights",
    "A.8.10": "Information deletion",
    "A.8.12": "Data leakage prevention",
    "A.8.15": "Logging",
    "A.8.16": "Monitoring activities",
    "A.8.24": "Use of cryptography",
    "A.8.28": "Secure coding",
}

EVIDENCE_KINDS = {"record", "configuration", "test_result"}
NON_EVIDENCE = {"policy", "intent", "roadmap"}


class StatementOfApplicability:
    def __init__(self, scope: str):
        self.scope = scope
        self.entries: dict[str, dict] = {}

    def apply(self, control: str, justification: str,
              evidence_ref: str, evidence_kind: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if evidence_kind in NON_EVIDENCE:
            raise ValueError(
                f"{control}: {evidence_kind!r} describes intent, not operation")
        if evidence_kind not in EVIDENCE_KINDS:
            raise ValueError(f"{control}: unknown evidence kind {evidence_kind!r}")
        self.entries[control] = {"applicable": True, "justification": justification,
                                 "evidence_ref": evidence_ref,
                                 "evidence_kind": evidence_kind}

    def exclude(self, control: str, justification: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if len(justification.split()) < 5:
            raise ValueError(f"{control}: exclusion needs a real justification")
        self.entries[control] = {"applicable": False, "justification": justification}

    def unaddressed(self) -> list[str]:
        return sorted(set(ANNEX_A_2022) - set(self.entries))

    def certifiable(self) -> dict:
        gaps = self.unaddressed()
        applied = [c for c, e in self.entries.items() if e["applicable"]]
        return {"scope": self.scope, "applied": sorted(applied),
                "excluded": sorted(c for c, e in self.entries.items()
                                   if not e["applicable"]),
                "unaddressed": gaps, "certifiable": not gaps}


if __name__ == "__main__":
    soa = StatementOfApplicability("LLM platform, production tenant")
    soa.apply("A.5.23", "Model inference is a cloud service",
              "cfg://gateway/allowed-providers.yaml", "configuration")
    soa.apply("A.8.2", "Operators hold privileged access to prompts",
              "rec://iam/quarterly-access-review-2026Q3", "record")
    soa.apply("A.8.10", "Retention is enforced on prompt logs",
              "test://retention/expiry-suite-2026-10", "test_result")
    soa.apply("A.8.12", "Egress to model providers can leak data",
              "cfg://dlp/egress-classifier.yaml", "configuration")
    soa.apply("A.8.15", "All inference events are logged",
              "rec://audit/chain-head-2026-10-16", "record")
    soa.apply("A.8.16", "Anomalous usage is monitored",
              "rec://siem/llm-usage-alerts", "record")
    soa.apply("A.8.24", "TLS and at-rest encryption",
              "test://crypto/tls-config-scan", "test_result")
    soa.exclude("A.8.28", "No application code is developed within this scope "
                          "boundary; the platform is configured, not coded")

    report = soa.certifiable()
    assert report["certifiable"], report
    assert len(report["applied"]) == 7 and len(report["excluded"]) == 1, report

    try:
        soa.apply("A.8.15", "we have a logging policy",
                  "doc://policies/logging.pdf", "policy")
        raise AssertionError("accepted a policy document as operating evidence")
    except ValueError:
        pass

    partial = StatementOfApplicability("partial")
    partial.apply("A.8.15", "logging", "rec://x", "record")
    assert not partial.certifiable()["certifiable"]
    assert len(partial.certifiable()["unaddressed"]) == 7

    print("ISO/IEC 27001:2022 SoA evidence mapping: OK")
    print(f"  applied {len(report['applied'])}, excluded {len(report['excluded'])}, "
          f"unaddressed {len(report['unaddressed'])}")

Risk register with acceptance criteria

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

What the auditor will ask

• Where is this control in the risk register and the Statement of Applicability, who owns it, and when was it last reviewed?
  Register entries are generated from the deployed control set, so every control has an SoA reference, a named owner and a review date that cannot silently lapse.
• Which slices did the candidate model get worse on, and by how much?
  The gate reports a per-slice delta for every slice, and the decision record keeps it. An aggregate alone cannot answer this, which is why it is not the gate.
• What stops someone editing the evaluation set until the release passes?
  The set is frozen on publish — mutation raises rather than warns, and a change requires a new version with its own identifier recorded on every subsequent decision.
• If the aggregate score improves by two points, can the release still be refused?
  Yes, and it must be. A slice regression beyond tolerance, or any breach of a named slice's floor, refuses the release regardless of the aggregate.
• Where is the record of who accepted a known regression, and on what basis?
  In the decision record attached to the release: the slices that moved, the magnitude, and the named person who accepted it. An unaccepted regression cannot ship.
• How do you know the scorer gives the same answer twice?
  The scorer is deterministic and any sampling is seeded, with the seed recorded. The same candidate against the same frozen set produces the same report.

Related architectures

• Part of enterprise ai compliance architecture.
• This pattern also governs GDPR model release gate and golden set evaluation for enterprise AI.
• SOX model release gate and golden set evaluation for enterprise AI covers the adjacent case.
• The same control appears in SOC 2 Type II model release gate and golden set evaluation for enterprise.
• Where this interacts with another regime, see EU AI Act model release gate and golden set evaluation for enterprise AI.
• If this is the blocker in front of you: enterprise AI infrastructure review.