Build vs buy evaluation criteria for enterprise AI implementation — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · SOC 2 Type II · layout control_review · implementation executed in CI before publish

The control that resolves this is a Gated Vendor Evaluation Framework, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The specific problem here is build versus buy. The shape below holds p95 ≤ 600ms over outbound webhooks with signed payloads, and ships with a standard-library implementation that runs as written.

The architecture friction point

A vendor comparison built as a weighted average lets a supplier average its way past a control it does not have. The scorecard ranks everyone, the committee cannot see which gates were failed, and the decision gets re-litigated after the contract is signed.

What makes build versus buy specifically hard

The build estimate covers the first version and the buy estimate covers three years, so the comparison is structurally biased toward building. The honest comparison includes evaluation harnesses, model upgrades, on-call and the compliance evidence that a vendor would otherwise produce.

What resolves it is a three-year fully-loaded model covering run cost and compliance effort. The failure to design against: a build decision that omits the ongoing evaluation and evidence burden.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

Risk and control matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Technical parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

What the auditor will ask

• Show me the control-execution population for the full window, and the days where it is empty.
  Every request emits an execution record, so the population is continuous by construction. A gap is an alertable incident, not something discovered during the audit.
• Over three years, who maintains the evaluation set and produces the audit evidence in the build case?
  A named team with budgeted capacity — the model includes eval maintenance, upgrade regression and evidence production, which are the costs a vendor would otherwise absorb.
• Which criteria are pass/fail, and which vendor failed one?
  The gate layer is pass/fail: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, and data residency. Gates are evaluated before any weighted scoring, so a vendor cannot average past a control it lacks.
• For each passed gate, what artifact proves it and what is its date?
  Each gate names the artifact that satisfies it — the report, the signed DPA, the architecture document — with its date. A claim is not evidence.
• Is the latency target in the signed agreement with a remedy, or on a status page?
  It must be in the signed agreement with a remedy. A status page is not a commitment; the scorecard scores the remedy, not the promise.
• What does exit look like — specifically, what can you export and has it been rehearsed?
  Export of prompts, embeddings, eval sets and fine-tunes, rehearsed during the pilot rather than promised in the contract. Portability that has never been executed is not portability.
• Was the reference deployment in the same vertical and regulatory regime as ours?
  It must be — a reference in the same vertical and the same regulatory regime, worked through a structured question set, not a vendor-selected happy customer.

Reference architecture

Gated Vendor Evaluation Framework — the request path, with the decision points that must exist before the first model call:

flowchart TB
    V[Vendor responses] --> G{Gates - pass/fail}
    G -->|any gate failed| DQ[Disqualified<br/>with named gates]
    G -->|all gates passed| W[Weighted scoring]
    W --> TCO[3-year fully loaded TCO]
    W --> REF[Reference call<br/>comparable regulated deployment]
    TCO --> R[Ranking]
    REF --> R
    R --> DR[Decision record<br/>score + reason per criterion]
    DQ --> DR
    DR --> SC[Steering committee]

The parts you have to build

1. Gate layer — Pass/fail controls: SOC 2 Type 2 currency, DPA, tenant isolation, residency.
2. Weighted layer — Ranks only vendors that cleared every gate.
3. Evidence requirement — Each gate names the artifact that satisfies it, not a claim.
4. TCO model — Fully loaded three-year: licence, run cost, internal effort.
5. Reference protocol — A structured call with a comparable regulated deployment.
6. Decision record — Ranking plus failed gates plus a written reason per score.

Reference implementation — Weighted scorecard with disqualifying gates

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind outbound webhooks with signed payloads; the failure mode it guards against is at-least-once delivery handled as exactly-once by the consumer. The output you hand the assessor is the observation-window evidence export.

"""Weighted vendor scorecard with disqualifying gates.

Why this shape: a procurement scorecard that is a weighted average lets a vendor
average its way past a control it simply does not have. Real evaluations have
two layers — GATES that are pass/fail, and weights that only ever rank the
vendors that already cleared every gate.

This produces the artifact a steering committee signs: a ranking, the gates each
vendor failed, and a written reason per score.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Criterion:
    key: str
    weight: float          # relative; normalised at scoring time
    gate: bool = False     # True -> pass/fail, cannot be averaged away
    note: str = ""


DEFAULT_CRITERIA = [
    Criterion("soc2_type2_current", 0.0, gate=True,
              note="Type 2 report inside 12 months, not a Type 1 or a roadmap"),
    Criterion("dpa_and_subprocessors", 0.0, gate=True,
              note="DPA signable as-is and a disclosed subprocessor list"),
    Criterion("tenant_isolation_evidence", 0.0, gate=True,
              note="Architectural isolation, not a WHERE clause"),
    Criterion("data_residency", 0.0, gate=True,
              note="Processing stays in the contracted region"),
    Criterion("exit_and_portability", 2.0,
              note="Export of prompts, embeddings, evals; no model lock-in"),
    Criterion("latency_slo_contractual", 3.0,
              note="p95 in the agreement with a remedy, not a status page"),
    Criterion("integration_depth", 3.0,
              note="Works against the real system of record, not a demo schema"),
    Criterion("implementation_capacity", 2.0,
              note="Named engineers, not a bench claim"),
    Criterion("tco_3yr", 3.0, note="Fully loaded: licence, run, internal effort"),
    Criterion("reference_in_our_vertical", 2.0,
              note="A reference call with a comparable regulated deployment"),
]


class Disqualified(Exception):
    pass


@dataclass
class Vendor:
    name: str
    gates: dict = field(default_factory=dict)    # key -> bool
    scores: dict = field(default_factory=dict)   # key -> 0..5
    reasons: dict = field(default_factory=dict)


def evaluate(vendors, criteria=None):
    """-> (ranked, disqualified). Gates are evaluated first, always."""
    criteria = criteria or DEFAULT_CRITERIA
    gate_keys = [c.key for c in criteria if c.gate]
    weighted = [c for c in criteria if not c.gate and c.weight > 0]
    total_w = sum(c.weight for c in weighted) or 1.0

    ranked, disqualified = [], []
    for v in vendors:
        failed = [k for k in gate_keys if not v.gates.get(k, False)]
        if failed:
            disqualified.append({"vendor": v.name, "failed_gates": failed})
            continue
        got = 0.0
        breakdown = []
        for c in weighted:
            s = float(v.scores.get(c.key, 0))
            if not 0 <= s <= 5:
                raise ValueError(f"{v.name}.{c.key}={s} outside 0..5")
            contribution = (c.weight / total_w) * (s / 5.0)
            got += contribution
            breakdown.append({"criterion": c.key, "score": s,
                              "weight": c.weight,
                              "contribution": round(contribution, 4),
                              "reason": v.reasons.get(c.key, ""),
                              "standard": c.note})
        ranked.append({"vendor": v.name, "score": round(got * 100, 1),
                       "breakdown": breakdown})
    ranked.sort(key=lambda r: -r["score"])
    return ranked, disqualified


if __name__ == "__main__":
    passing = {k: True for k in
               ("soc2_type2_current", "dpa_and_subprocessors",
                "tenant_isolation_evidence", "data_residency")}
    a = Vendor("Vendor A", gates=dict(passing),
               scores={"exit_and_portability": 4, "latency_slo_contractual": 5,
                       "integration_depth": 4, "implementation_capacity": 3,
                       "tco_3yr": 3, "reference_in_our_vertical": 5},
               reasons={"latency_slo_contractual": "p95 450ms with 10% credit"})
    b = Vendor("Vendor B", gates={**passing, "tenant_isolation_evidence": False},
               scores={k: 5 for k in
                       ("exit_and_portability", "latency_slo_contractual",
                        "integration_depth", "implementation_capacity",
                        "tco_3yr", "reference_in_our_vertical")})
    ranked, dq = evaluate([a, b])
    for r in ranked:
        print(f"{r['vendor']}: {r['score']}")
    print("disqualified despite perfect scores:", dq)

Source discussions

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• A16z partner says that the theory that we’ll vibe code everything is wrong hackernews · hn/comment · 2026-02-22
• So where are all the AI apps? hackernews · hn/comment · 2026-03-24
• A16z partner says that the theory that we’ll vibe code everything is wrong hackernews · hn/comment · 2026-02-21
• AI is killing B2B SaaS hackernews · hn/comment · 2026-02-04

Related reference architectures

• Part of enterprise ai vendor evaluation.
• This pattern also governs procurement evaluation criteria for enterprise AI implementation.
• ERP integration patterns for enterprise AI agents applies the same mechanism under a different obligation.
• Where this interacts with another regime, see air-gapped architecture for enterprise LLM deployment at scale.
• If this is the blocker in front of you: enterprise AI infrastructure review.