Evals

Submit, list, stream, and fetch evaluation runs against the gateway's serving engine. Eval runs execute in-process via asyncio.create_task (even when BIGWAVE_RUNNER=arq) — the runner contract is fast: queue-only round-trip is sub-millisecond, the run itself drives engine.chat(handle, …) under the hood and persists a EvalReport row when terminal. See the concept page on eval-as-a-gate for the "gate adapter promotion on eval pass" workflow.

The router lives at packages/eval/admin_routes.py. Wire schemas: CreateRunRequest, CreateRunResponse, and EvalRunRecord.

Endpoints

Method	Path	Auth	Summary
POST	/admin/evals/runs	Bearer	Submit a run; returns 202 with {run_id, status: "queued"}.
GET	/admin/evals/runs	Bearer	List eval runs for the workspace, newest first.
GET	/admin/evals/runs/{run_id}	Bearer	Fetch one run's current EvalRunRecord.
GET	/admin/evals/runs/{run_id}/stream	Bearer	SSE stream of status transitions and final report.
GET	/admin/evals/reports/{report_id}	Bearer	Full EvalReport with all per-item results.
GET	/admin/evals/runs/{run_id}/report	Bearer	The report produced by run_id, when terminal+succeeded.

POST /admin/evals/runs

Body CreateRunRequest:

{
  "suite": "golden_set",
  "dataset_jsonl": "{\"prompt\":\"2+2?\",\"reference\":\"4\"}\n",
  "target_model": "llama3.2:3b",
  "target_adapter": null,
  "scorer_overrides": {}
}

suite must be "golden_set" for v0; any other value is 422. dataset_jsonl is the inline suite content; reading from a registered dataset_id lands roadmap.

Returns 202 with CreateRunResponse:

{ "run_id": "evr_01HF…", "status": "queued" }

curl -X POST -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"suite":"golden_set","dataset_jsonl":"{\"prompt\":\"2+2?\",\"reference\":\"4\"}\n","target_model":"llama3.2:3b"}' \
  http://localhost:8000/admin/evals/runs

GET /admin/evals/runs

Returns a JSON array of EvalRunRecord objects ordered by submit time (newest first), filtered to the caller's workspace.

GET /admin/evals/runs/{run_id}

Returns the current EvalRunRecord. Cross-tenant lookups return 404.

GET /admin/evals/runs/{run_id}/stream

Server-Sent Events. Frame layout:

• event: status — one frame per observed status transition (queued → running → succeeded or failed / cancelled). Data is the full EvalRunRecord JSON.
• event: item_completed — one frame per per-item result. Data is EvalItemResult. v0 replays items in batch on terminal transition.
• event: report — one terminal frame on succeeded carrying the full EvalReport.

The connection self-terminates on terminal state.

GET /admin/evals/reports/{report_id}

Returns the persisted EvalReport row directly by id. Cross-tenant returns 404.

GET /admin/evals/runs/{run_id}/report

Returns the report produced by run_id. When the run has not yet produced a report (still queued / running, or terminated as failed / cancelled with no report) the route returns 404 — the dashboard polls this on SSE drain backfill.

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  http://localhost:8000/admin/evals/runs/evr_01HF.../report

Operator notes

• Workspace fence. Every route resolves the workspace from the bearer principal; cross-tenant existence does not leak (404 on miss). See workspace fence.
• In-process execution. Eval runs always run in-process — the runner does not push them to the Arq worker even when one is wired (eval suites are seconds; the queue round-trip does not pay back at this scale). Long-running training jobs differ — see Training.
• Shutdown drain. The lifespan calls drain_eval_tasks(30s) on shutdown so a SIGTERM during an eval marks the run row failed rather than leaving it stuck running.
• 503 causes. app.state.eval_run_repo is wired in lifespan; routes return 503 with "EvalReportRepository not initialised" when missing. The default target_fn additionally requires app.state.engine and app.state.handle — those raise at run-time, not route-time.
• Promotion gate. The eval gate is consumed by /admin/models/{model_id}/adapters — failing evals can block adapter activation. See eval-as-a-gate.
• Audit. Each POST writes an eval.submit audit row with the submitted spec; see Audit.