Eval-as-a-gate
A binding deployment gate that refuses adapter activation when the candidate's eval pass-rate Wilson lower bound regresses below baseline. The old handle stays live; the audit log records why.
Why "binding" is the whole point
Most eval suites produce a number. The number gets posted in a PR comment. Someone reads it, decides it's "close enough," and ships.
Bigwave's gate is structurally different: it's wired into ReplicaManager.activate. The activation route literally cannot return success without the gate's allow. There's no "we ran the eval but shipped anyway" path.
Where it sits
┌──────────────────────────────┐
│ POST /admin/models/{id}/ │
│ adapters │ activate request
└─────────────┬─────────────────┘
│
▼
┌────────────────────────┐
│ ReplicaManager.activate│
└────────┬───────────────┘
│
▼
┌────────────────────────┐
│ GatePolicyRepo │ "is there a policy?"
└────────┬───────────────┘
│
no ▼ yes (return policy)
│ ┌────────────────────────┐
│ │ Run candidate eval │
│ │ + load baseline report │
│ └────────┬───────────────┘
│ │
│ ┌────┴────┐
│ GateAllow GateBlock
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐
│ Drain + │ │ ActivationError│
│ swap to new │ │ (422 to caller)│
│ handle │ │ Old handle │
└──────────────┘ │ preserved. │
│ Audit row │
│ recorded. │
└──────────────┘The gate decision
The gate constructs an EvalRun against the candidate and runs it against the policy's golden set. Then it compares to the baseline:
- Allow when the candidate's pass-rate Wilson lower bound is within statistical tolerance of the baseline.
- Block when the candidate's pass-rate Wilson upper bound is below the baseline's lower bound (i.e. the CIs don't overlap and the candidate is provably worse).
The decision lands in GateAllow / GateBlock Pydantic types. On block, the manager raises ActivationError("eval gate blocked activation for adapter_id='...': <reason>. report=<id>"). The admin route translates this to HTTP 422.
Why Wilson CIs, not point estimates
Comparing two pass-rates with point estimates is a bad statistical move when sample sizes are small or unequal:
- A baseline at 0.78 (n=200) and a candidate at 0.76 (n=50) — point estimate says regression, but the CIs overlap massively and the difference is noise.
- A baseline at 0.50 (n=10) and a candidate at 0.55 (n=10) — point estimate says improvement, but the CIs are huge and you've proved nothing.
Wilson 95% CIs give honest uncertainty bounds. The gate only blocks when the candidate is provably worse (CI upper < baseline CI lower) — i.e. you'd see this magnitude of regression by chance less than 5% of the time.
This is more conservative than "any drop blocks." Some real regressions slip through. The trade-off is fewer false-blocks, which keeps operators trusting the gate. A gate the team overrides constantly is worse than no gate.
Wiring it up
The lifespan ships app.state.eval_gate with a no-op policy repo by default — every workspace gets GateAllow with negligible overhead. Operators who want gates wire a real GatePolicyRepository:
from packages.eval.gate import GatePolicy, GatePolicyRepository
class MyPolicyRepo(GatePolicyRepository):
async def get(self, *, workspace, scope):
# Look up the workspace's policy; return None to skip the gate.
return GatePolicy(
workspace_id=workspace,
scope="adapter_attach",
baseline_report_id="run-baseline-...",
evaluation_dataset=open("./golden.jsonl").read(),
scorer="exact_match",
)Then on app boot:
app.state.eval_gate = EvalGate(
policy_repo=MyPolicyRepo(),
report_repo=app.state.eval_run_repo,
target_fn_factory=lifespan_target_fn_factory, # ships with bigwave
)Scorer types
The gate runs the policy's named scorer against each golden-set item. Scorer types in the codebase / on the roadmap:
| Scorer | What it checks | Wedge it serves |
|---|---|---|
exact_match | String equality (after normalisation) | Compliance baseline; brittle for free-form |
trajectory_match | Levenshtein edit distance over tool-call sequences; configurable threshold + args-strict mode | Agent eval (sequence-level) |
hallucination_probe | Refusal-rate scorer with per-probe severity weighting; 15-probe starter set across factual_void / future_prediction / private_info / fictional_attribution | Agent eval (fabrication signal) |
jailbreak_resistance | Role-retention scorer; per-attempt severity weighting; 12-attempt starter set across 6 attack families | Agent eval (role retention) |
llm_judge | Rubric-DSL-driven scoring via an injected JudgeCallable; weighted criteria with per-criterion reasoning; markdown-fence + trailing-prose tolerant JSON parsing | Tone tuning, agent-trajectory grading, qualitative regression |
DriftDetector (separate module — packages/eval/drift.py) | Windowed trajectory-distribution shift detection via PSI (default, threshold 0.20) or Jensen-Shannon divergence; top-N alerts sorted by per-bucket contribution | Agent eval (distribution drift) |
compile_pass (roadmap) | Generated code compiles | Coding assistant |
test_pass (roadmap) | Generated code passes provided tests | Coding assistant |
lm_eval_harness (roadmap) | Wrap the standard suite | General benchmarking |
The EvalScorer ABC is the extension point. Custom scorers register in packages.eval.scorers.get_scorer's _REGISTRY.
Rollback semantics
When the gate blocks, the old handle is preserved. Concretely:
- The activation method has not yet drained the in-flight requests.
- The old handle is still in
replica_manager.handles[model_id]. - The audit log records the failed activation attempt.
- Subsequent chat traffic continues against the old handle.
This is the property that makes the gate safe to wire into a CI/CD pipeline. A misconfigured candidate doesn't take production down; it just doesn't ship.
What the gate is not a substitute for
- Runtime monitoring. The gate catches regressions at activation time, not at runtime. If a tuned model is fine on the golden set but drifts in production, that's an observability concern, not a gate concern.
- Adversarial / red-team coverage. The gate runs your golden set. If your golden set doesn't exercise prompt injection, the gate won't catch a candidate that's more vulnerable to prompt injection.
- Human review. For high-stakes deploys (medical, legal, financial advice), the gate is necessary but not sufficient. Wire a human approval step in front of the activation call.
Gate refusal — what the operator sees
When the gate blocks, ReplicaManager.activate raises ActivationError and the HTTP route returns 422. Simultaneously, a gate.blocked row is written to the audit log. Here is the structured GateBlock decision that drives both:
GateBlock(
decision='block',
candidate_report_id='rpt-cand-29f1',
baseline_pass_rate=0.8950,
candidate_pass_rate=0.7320,
reason=(
"candidate pass_rate=0.7320 is below baseline pass_rate=0.8950; "
"candidate Wilson CI upper=0.8110 < baseline CI lower=0.8420 "
"(regression proven)"
),
)The candidate_report_id is a direct link to the full eval report in /admin/evals/runs/rpt-cand-29f1 — every scorer output, item-level pass/fail, and bucket breakdown. The audit row at /admin/audit links the same report via inference_log_id so operators can pivot without memorising the report ID.
The activation error surfaced to the caller:
ActivationError: eval gate blocked activation for adapter_id='ada-llm-v1':
candidate pass_rate=0.7320 is below baseline pass_rate=0.8950;
candidate Wilson CI upper=0.8110 < baseline CI lower=0.8420
(regression proven). report=rpt-cand-29f1The old adapter handle is untouched. Production traffic continues without interruption.
Reading order
- The loop — where the gate fits in the bigger picture.
- Workspace fence — gate decisions are workspace-scoped; ws-A's policy never affects ws-B.
- Compliance use case — the gate as the centerpiece of an audit-defensible deploy.
- Audit-defensibility — how the gate refusal lands in the append-only audit trail and how to query it.