Procurement evaluation criteria for enterprise AI implementation in government — Reference Architecture

Reference architecture · published 2026-10-16 · government and public sector · FedRAMP Moderate · layout matrix_first · implementation executed in CI before publish

The control that resolves this is a Gated Vendor Evaluation Framework, evaluated before the request reaches a model provider rather than audited afterwards. In government and public sector the binding constraint is FedRAMP Moderate, and the question it actually asks is which NIST 800-53 control does each component satisfy. The specific problem here is RFP and structured evaluation. The shape below holds p95 ≤ 1500ms over outbound webhooks with signed payloads, and ships with a standard-library implementation that runs as written.

Technical parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Risk register with acceptance criteria

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per NARA schedule; varies by record series.

Where the rollout actually stalls

A vendor comparison built as a weighted average lets a supplier average its way past a control it does not have. The scorecard ranks everyone, the committee cannot see which gates were failed, and the decision gets re-litigated after the contract is signed.

What makes RFP and structured evaluation specifically hard

An RFP scored as a weighted average lets a vendor average its way past a control it does not have. The scoring model, not the questions, is usually what makes the evaluation indefensible.

What resolves it is a two-layer scorecard: pass/fail gates, then weights. The failure to design against: the highest-scoring vendor failing security review after award.

What FedRAMP Moderate specifically requires

FedRAMP is control-by-control. Every component must map to a specific NIST 800-53 Rev 5 control identifier with an implementation statement, and anything crossing the authorisation boundary needs its own justification in the SSP.

The artifact that satisfies it is a control implementation summary mapped to control identifiers. The way teams most often fail it: a component outside the authorisation boundary that nobody documented, found during assessment.

Modular implementation framework

1. Gate layer — Pass/fail controls: SOC 2 Type 2 currency, DPA, tenant isolation, residency.
2. Weighted layer — Ranks only vendors that cleared every gate.
3. Evidence requirement — Each gate names the artifact that satisfies it, not a claim.
4. TCO model — Fully loaded three-year: licence, run cost, internal effort.
5. Reference protocol — A structured call with a comparable regulated deployment.
6. Decision record — Ranking plus failed gates plus a written reason per score.

The blueprint

Gated Vendor Evaluation Framework — the request path, with the decision points that must exist before the first model call:

flowchart TB
    V[Vendor responses] --> G{Gates - pass/fail}
    G -->|any gate failed| DQ[Disqualified<br/>with named gates]
    G -->|all gates passed| W[Weighted scoring]
    W --> TCO[3-year fully loaded TCO]
    W --> REF[Reference call<br/>comparable regulated deployment]
    TCO --> R[Ranking]
    REF --> R
    R --> DR[Decision record<br/>score + reason per criterion]
    DQ --> DR
    DR --> SC[Steering committee]

The implementation — Weighted scorecard with disqualifying gates

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind outbound webhooks with signed payloads; the failure mode it guards against is at-least-once delivery handled as exactly-once by the consumer. The output you hand the assessor is the SSP control implementation summary.

"""Weighted vendor scorecard with disqualifying gates.

Why this shape: a procurement scorecard that is a weighted average lets a vendor
average its way past a control it simply does not have. Real evaluations have
two layers — GATES that are pass/fail, and weights that only ever rank the
vendors that already cleared every gate.

This produces the artifact a steering committee signs: a ranking, the gates each
vendor failed, and a written reason per score.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Criterion:
    key: str
    weight: float          # relative; normalised at scoring time
    gate: bool = False     # True -> pass/fail, cannot be averaged away
    note: str = ""


DEFAULT_CRITERIA = [
    Criterion("soc2_type2_current", 0.0, gate=True,
              note="Type 2 report inside 12 months, not a Type 1 or a roadmap"),
    Criterion("dpa_and_subprocessors", 0.0, gate=True,
              note="DPA signable as-is and a disclosed subprocessor list"),
    Criterion("tenant_isolation_evidence", 0.0, gate=True,
              note="Architectural isolation, not a WHERE clause"),
    Criterion("data_residency", 0.0, gate=True,
              note="Processing stays in the contracted region"),
    Criterion("exit_and_portability", 2.0,
              note="Export of prompts, embeddings, evals; no model lock-in"),
    Criterion("latency_slo_contractual", 3.0,
              note="p95 in the agreement with a remedy, not a status page"),
    Criterion("integration_depth", 3.0,
              note="Works against the real system of record, not a demo schema"),
    Criterion("implementation_capacity", 2.0,
              note="Named engineers, not a bench claim"),
    Criterion("tco_3yr", 3.0, note="Fully loaded: licence, run, internal effort"),
    Criterion("reference_in_our_vertical", 2.0,
              note="A reference call with a comparable regulated deployment"),
]


class Disqualified(Exception):
    pass


@dataclass
class Vendor:
    name: str
    gates: dict = field(default_factory=dict)    # key -> bool
    scores: dict = field(default_factory=dict)   # key -> 0..5
    reasons: dict = field(default_factory=dict)


def evaluate(vendors, criteria=None):
    """-> (ranked, disqualified). Gates are evaluated first, always."""
    criteria = criteria or DEFAULT_CRITERIA
    gate_keys = [c.key for c in criteria if c.gate]
    weighted = [c for c in criteria if not c.gate and c.weight > 0]
    total_w = sum(c.weight for c in weighted) or 1.0

    ranked, disqualified = [], []
    for v in vendors:
        failed = [k for k in gate_keys if not v.gates.get(k, False)]
        if failed:
            disqualified.append({"vendor": v.name, "failed_gates": failed})
            continue
        got = 0.0
        breakdown = []
        for c in weighted:
            s = float(v.scores.get(c.key, 0))
            if not 0 <= s <= 5:
                raise ValueError(f"{v.name}.{c.key}={s} outside 0..5")
            contribution = (c.weight / total_w) * (s / 5.0)
            got += contribution
            breakdown.append({"criterion": c.key, "score": s,
                              "weight": c.weight,
                              "contribution": round(contribution, 4),
                              "reason": v.reasons.get(c.key, ""),
                              "standard": c.note})
        ranked.append({"vendor": v.name, "score": round(got * 100, 1),
                       "breakdown": breakdown})
    ranked.sort(key=lambda r: -r["score"])
    return ranked, disqualified


if __name__ == "__main__":
    passing = {k: True for k in
               ("soc2_type2_current", "dpa_and_subprocessors",
                "tenant_isolation_evidence", "data_residency")}
    a = Vendor("Vendor A", gates=dict(passing),
               scores={"exit_and_portability": 4, "latency_slo_contractual": 5,
                       "integration_depth": 4, "implementation_capacity": 3,
                       "tco_3yr": 3, "reference_in_our_vertical": 5},
               reasons={"latency_slo_contractual": "p95 450ms with 10% credit"})
    b = Vendor("Vendor B", gates={**passing, "tenant_isolation_evidence": False},
               scores={k: 5 for k in
                       ("exit_and_portability", "latency_slo_contractual",
                        "integration_depth", "implementation_capacity",
                        "tco_3yr", "reference_in_our_vertical")})
    ranked, dq = evaluate([a, b])
    for r in ranked:
        print(f"{r['vendor']}: {r['score']}")
    print("disqualified despite perfect scores:", dq)

The FedRAMP Moderate control, as code

NIST 800-53 control map with a boundary-crossing check. This is the artifact an assessor asks for under FedRAMP Moderate specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""NIST 800-53 control satisfaction map with an authorisation-boundary check.

FedRAMP reviewers do not ask whether the system is secure. They ask which
control, by identifier, each component satisfies, and what crosses the
authorisation boundary. An external model endpoint is a boundary crossing, and
an undeclared crossing is the finding that stalls an ATO.
"""

MODERATE_BASELINE = {
    "AC-4": "Information flow enforcement",
    "AU-2": "Event logging",
    "AU-9": "Protection of audit information",
    "CA-3": "Information exchange / external connections",
    "SC-7": "Boundary protection",
    "SC-8": "Transmission confidentiality and integrity",
    "SC-28": "Protection of information at rest",
    "SI-4": "System monitoring",
}


class AuthorisationBoundary:
    def __init__(self, name: str):
        self.name = name
        self.components: dict[str, dict] = {}
        self.crossings: list[dict] = []

    def declare(self, component: str, inside: bool, controls: list[str]) -> None:
        unknown = [c for c in controls if c not in MODERATE_BASELINE]
        if unknown:
            raise KeyError(f"not in the Moderate baseline: {unknown}")
        self.components[component] = {"inside": inside, "controls": set(controls)}

    def connect(self, src: str, dst: str, declared: bool) -> None:
        for end in (src, dst):
            if end not in self.components:
                raise KeyError(f"undeclared component: {end}")
        crossing = self.components[src]["inside"] != self.components[dst]["inside"]
        self.crossings.append({"src": src, "dst": dst,
                               "boundary_crossing": crossing, "declared": declared})

    def undeclared_crossings(self) -> list[dict]:
        return [c for c in self.crossings if c["boundary_crossing"] and not c["declared"]]

    def coverage(self) -> dict:
        covered = set()
        for meta in self.components.values():
            covered |= meta["controls"]
        gaps = sorted(set(MODERATE_BASELINE) - covered)
        return {"covered": sorted(covered), "gaps": gaps,
                "ato_ready": not gaps and not self.undeclared_crossings()}


if __name__ == "__main__":
    b = AuthorisationBoundary("agency-llm-enclave")
    b.declare("gateway", True, ["AC-4", "SC-7", "SC-8"])
    b.declare("audit_store", True, ["AU-2", "AU-9", "SC-28"])
    b.declare("monitor", True, ["SI-4"])
    b.declare("external_model_api", False, ["CA-3"])

    b.connect("gateway", "audit_store", declared=True)
    b.connect("gateway", "external_model_api", declared=True)
    cov = b.coverage()
    assert cov["ato_ready"], cov
    assert cov["gaps"] == [], cov

    # The finding this exists to catch: a crossing nobody declared.
    b2 = AuthorisationBoundary("agency-llm-enclave")
    b2.declare("gateway", True, list(MODERATE_BASELINE))
    b2.declare("telemetry_sidecar", False, ["CA-3"])
    b2.connect("gateway", "telemetry_sidecar", declared=False)
    assert not b2.coverage()["ato_ready"], b2.coverage()
    assert len(b2.undeclared_crossings()) == 1, b2.undeclared_crossings()

    try:
        b2.declare("rogue", True, ["XX-1"])
        raise AssertionError("accepted a control outside the baseline")
    except KeyError:
        pass

    print("FedRAMP Moderate boundary + control map: OK")
    print(f"  controls covered {len(cov['covered'])}/{len(MODERATE_BASELINE)}, "
          f"undeclared crossings {len(b2.undeclared_crossings())}")

Questions to answer before approval

• Which NIST 800-53 control does each component satisfy, and which components sit outside the authorisation boundary?
  Each component carries its control identifiers in configuration, and the boundary diagram is generated from that configuration so it cannot drift from reality.
• Which criteria are pass/fail, and which vendor failed one?
  Gates are evaluated first and independently of score: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, residency. A failure on any of them disqualifies.
• Which criteria are pass/fail, and which vendor failed one?
  The gate layer is pass/fail: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, and data residency. Gates are evaluated before any weighted scoring, so a vendor cannot average past a control it lacks.
• For each passed gate, what artifact proves it and what is its date?
  Each gate names the artifact that satisfies it — the report, the signed DPA, the architecture document — with its date. A claim is not evidence.
• Is the latency target in the signed agreement with a remedy, or on a status page?
  It must be in the signed agreement with a remedy. A status page is not a commitment; the scorecard scores the remedy, not the promise.
• What does exit look like — specifically, what can you export and has it been rehearsed?
  Export of prompts, embeddings, eval sets and fine-tunes, rehearsed during the pilot rather than promised in the contract. Portability that has never been executed is not portability.
• Was the reference deployment in the same vertical and regulatory regime as ours?
  It must be — a reference in the same vertical and the same regulatory regime, worked through a structured question set, not a vendor-selected happy customer.

Primary sources

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• When AI Procurement Fails, What Evidence Exists? hackernews · hn/comment · 2026-01-14
• Startup Spotlight: Sourcerer wants to turn AI procurement into a supplier hackernews · hn/story · 2026-07-08
• US gov seeks AI procurement using 'full user context and data access' chatbots hackernews · hn/story · 2025-08-19
• ProcureOS – AI procurement for small businesses replacing Excel and WhatsApp hackernews · hn/story · 2026-05-17
• Albania puts AI-created 'minister' in charge of public procurement hackernews · hn/story · 2025-09-12
• China adds domestic AI chips to official procurement list for first time hackernews · hn/story · 2025-12-10
• Using government procurement for greener AI hackernews · hn/story · 2026-07-25
• Albania Appoints AI Minister Agent for Public Procurement hackernews · hn/story · 2025-09-12

See also

• Part of enterprise ai vendor evaluation.
• A different failure mode of the same component: GDPR vendor evaluation criteria for enterprise AI implementation.
• Where this interacts with another regime, see ISO/IEC 27001:2022 vendor evaluation criteria for enterprise AI.
• This pattern also governs SOC 2 Type II vendor evaluation criteria for enterprise AI implementation.
• Where this interacts with another regime, see PCI DSS 4.0 vendor evaluation criteria for enterprise AI implementation.
• If this is the blocker in front of you: enterprise AI infrastructure review.