Bigwave

Appearance

Agent eval & promotion gate ​

Design-partner pitch + intake. This page is the public-facing brief for one of Bigwave's two lead wedges. If you're scoping whether Bigwave fits, read this end-to-end before opening a Discussion.

The pitch ​

Your agent shipped a regression last quarter. The base-model eval was green, the staging suite passed, the human-graded spot checks looked fine. In production the agent's trajectory shifted — same first-turn answer, different sequence of tool calls — and the customer-facing failure followed. The eval that would have caught it doesn't exist in your CI because the only artefact it produced was "0.83" in a PR comment nobody is required to act on.

Bigwave's promotion gate is binding. The gate is not a report card; it is wired into the activation path. When you flip the Activate button on a candidate adapter, ReplicaManager.activate consults EvalGate first. A GateBlock decision returns 422 with the gate's reason and the old adapter handle is preserved (rollback semantics). The regressed agent literally cannot ship.

The wedge is the binding gate plus the agent-aware scorers that make the gate worth binding to: trajectory-match (how close is this run's tool-call sequence to the reference?), hallucination probes ("you should refuse to answer" prompts graded by refusal rate), and jailbreak resistance (curated break-character attempts). The gate ships today; the agent-aware scorers ship next, in collaboration with the design partner.

Ideal customer profile ​

Concrete buyer signals that say "this wedge is for you":

  • Regulated industry (legal, healthcare, financial, defense) deploying agents that touch user data, where a customer-facing agent regression is a reportable event, not just a Slack apology.
  • You've shipped at least one agent regression that a smarter CI gate would have caught — and your postmortem named "we had no binding eval" or "the eval was advisory" as the root cause.
  • Multiple agent stacks running against the same backend (chat agent, support agent, internal coding agent) where every stack needs the same activation discipline but you don't want to bolt three different gates onto three different deploy pipelines.
  • You already have a private golden set — 50 to a few hundred representative trajectories the team agrees the agent should succeed on — and you want to score against it on every adapter promotion, not once a quarter.
  • You operate on-prem or in your own VPC for data-residency reasons and need a dual-licensed (MIT OR Apache-2.0) self-hosted gate, not a vendor SaaS.

If two or more of those describe you, you're in scope. If your agents are advisory-only and a wrong answer never leaves your team's chat, this probably isn't the right wedge.

What ships today vs roadmap ​

Every "today" claim below is verifiable in the repo on main.

TodayRoadmap
EvalSuite ABC (packages/eval/suites/__init__.py) — pluggable suite contractlm-evaluation-harness adapter so partner-side eval suites import unchanged
EvalScorer ABC (packages/eval/base.py) — pluggable scorer contractPer-tool-call scoring (grade individual tool-call decisions, not just end-to-end outcomes)
hallucination_probe scorer (packages/eval/scorers/hallucination_probe.py) — refusal-rate scorer with per-probe severity weighting + 15-probe starter set across factual_void / future_prediction / private_info / fictional_attribution categoriesPublic-leaderboard mode for agents whose teams want comparative scoring
jailbreak_resistance scorer (packages/eval/scorers/jailbreak_resistance.py) — role-retention scorer with per-attempt severity weighting + 12-attempt starter set across dan / instruction_override / developer_mode / role_inversion / prefix_injection / context_poisoning families
DriftDetector (packages/eval/drift.py) — windowed trajectory-distribution shift detection with PSI (default, threshold 0.20) and Jensen-Shannon divergence (opt-in); top-N alerts sorted by per-bucket contribution; static-window comparison in v0 (operators schedule the call themselves; automated cadence + persistence is roadmap)
llm_judge scorer (packages/eval/scorers/llm_judge.py) — rubric-DSL-driven scoring via an injected judge callable; weighted criteria with per-criterion reasoning; markdown-fence + trailing-prose tolerant JSON parsing; production wires the judge to a ServingEngine via make_engine_judgeAuto-tuned baselines (the gate proposes a tighter baseline once the candidate has shipped clean for N weeks)
golden_set suite (packages/eval/suites/golden_set.py) — JSONL-driven prompts + expected outputs, Wilson 95 % CI roll-up
exact_match scorer (packages/eval/scorers/exact_match.py) — strict string match for response-level evals
trajectory_match scorer (packages/eval/scorers/trajectory_match.py) — Levenshtein edit distance over tool-call sequences, configurable threshold + args-strict mode, sequence-level not response-level
EvalGate (packages/eval/gate.py) hooked into ReplicaManager.activate — GateBlock returns 422 and the old handle is preserved
GatePolicy per-workspace (suite + baseline + Wilson CI overlap rule), persisted and hot-reloadable
Append-only audit row for every gate decision (allow / block / infrastructure-error) at /admin/audit
EvalRun orchestrator with shadow + billable=False so eval traffic doesn't pollute cost rollups
EvalReport persistence so a GateBlock decision links back to the exact run that triggered it

Two-line summary: all five agent-aware regression detectors ship today — trajectory_match (sequence-level), hallucination_probe (fabrication), jailbreak_resistance (role retention), llm_judge (qualitative rubric-DSL grading), and the DriftDetector (windowed trajectory-distribution shift via PSI / JS-divergence, in-memory v0 — operators schedule the call themselves, automated cadence + persistence is roadmap). The binding-gate plumbing — policy / report / Wilson-CI / 422-on-block / preserve-the-old-handle behaviour — is production-ready. The remaining roadmap items (lm-evaluation-harness adapter, per-tool-call scoring, public leaderboard mode, auto-tuned baselines) are conveniences, not load-bearing for the core "binding gate refuses regressed agents" claim.

What "validation working" looks like ​

The metric the design partner and Bigwave watch together, weekly:

  • Agent trajectory-match score ≥ 0.85 on the partner's private eval set, sustained over four consecutive promotion attempts.
  • At least one gate-blocked activation per month that would have shipped a regressed agent without the gate — measured by shadow-running the candidate post-block and confirming a user-visible regression.
  • Time-to-promote — from "candidate adapter trained" to "candidate adapter active in prod, gate-approved" stays under 30 minutes. The gate adds discipline, not friction.
  • Zero post-promotion rollbacks caused by the regressed-agent class of failure. (If the gate lets a regression through, that is a P0 for bigwave.)
  • Audit completeness — every gate decision (allow + block + infrastructure-error) has an AuditLogEntry linking back to the candidate adapter, the suite, and the report. The partner can answer "why was this agent allowed to ship on date X" in under 60 seconds.

If those numbers don't move in the right direction in the first 30 days, something is wrong with the wedge fit and we want to hear about it.

How to engage ​

Demo / quickstart ​

A laptop with a consumer GPU is enough to exercise the wedge end-to-end with the response-level scorer that ships today.

bash
# 1. Install bigwave with the default stack.
git clone https://github.com/bigwaveml/bigwave.git
cd bigwave
./install.sh --non-interactive

# 2. Pull a model.
ollama pull qwen2.5:7b

# 3. Confirm the runtime is up.
bw status
curl -s http://localhost:8080/healthz

# 4. Upload a golden-set JSONL — one row per (prompt, expected).
#    Schema is documented in packages/eval/suites/golden_set.py.
curl -s -X POST http://localhost:8080/admin/uploads \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  -F "file=@golden_set.jsonl" \
  -F "kind=eval_suite"

# 5. Wire a gate policy for the workspace.  Suite + scorer +
#    baseline.  GatePolicy lives at /admin/evals.
curl -s -X PUT http://localhost:8080/admin/evals/policy \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"suite":"golden_set","scorer":"exact_match",
       "baseline_pass_rate":0.80}'

# 6. Train a candidate adapter and try to activate it.
#    If pass_rate.ci_upper < baseline.ci_lower the gate
#    returns 422 GateBlock and the old adapter stays active.
curl -s -X POST http://localhost:8080/admin/adapters/$ADAPTER_ID/activate \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN"

# 7. Inspect the gate decision in the audit log.
curl -s http://localhost:8080/admin/audit?action=eval_gate_decision \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" | head -1

The end-to-end feedback-loop walk — chat → feedback → DPO → eval → promote — is documented in docs/runbooks/feedback-loop.md. The EvalGate step in that runbook is the same gate the wedge hardens.

Dual-licensed: MIT OR Apache-2.0

© 2026 bigwave contributors