Run an eval against a golden set

What it does

Evals measure how well a model (or adapter) performs on a test dataset using a scoring function. You submit a run specifying the model, dataset, and scorer (e.g. exact match, semantic similarity). The run executes asynchronously and produces a report with pass rate and per-row scores.

Prerequisites

• At least one dataset with 5+ rows (guide 04).
• A model to test (base model or active adapter).
• Expected eval time: 30 seconds to 5 minutes depending on dataset size and model latency.

Steps

1. Open Evals page

Click Evals in the left rail (icon grid in the left sidebar). The page shows a submit form on the left and past eval runs on the right.

2. Fill the eval form

• Model — dropdown to pick the model or active adapter to test.

• Dataset — dropdown to pick a test dataset (the golden set).

• Scorer — select the scoring function:

  • exact_match — output must be identical to expected (after casefold + whitespace normalisation).
  • trajectory_match — agent-aware: Levenshtein edit distance over the agent's tool-call sequence; configurable pass_threshold (default 0.85) + args_strict mode.
  • hallucination_probe — refusal-rate scorer for prompts the model should refuse to answer; per-probe severity weighting; a 15-probe starter set ships, operators extend per their threat model.
  • jailbreak_resistance — role-retention scorer for curated break-character attempts; ANY break_indicator regex match counts as a break; 12-attempt starter set across 6 families.
  • llm_judge — rubric-DSL-driven scoring via an injected judge callable; weighted criteria, per-criterion reasoning, tolerates markdown-fenced + trailing-prose judge output.

  Default is exact_match.

  Distribution-drift detection is exposed as a separate module (packages.eval.DriftDetector), not via the suite/scorer pipeline — operators schedule the call themselves over a window of recent trajectories.

3. Submit the run

Click Submit. A toast confirms the run_id; the list refreshes to show the new run with running status.

4. Monitor the run

The run card shows:

• Status — running (blue), succeeded (green), failed (red).
• Progress — rows processed / total rows.
• Pass rate — displayed once the run completes (e.g. "87% passing").

Click View Logs to see a real-time SSE stream of per-row results.

5. View the report

Once succeeded, click the View Report button. A drawer opens showing:

• Pass rate (headline metric).
• By-score histogram — distribution of row scores.
• Per-row results — each row with predicted output, expected output, and score.

(Direct API) Submit an eval run via curl

curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BIGWAVE_TOKEN" \
  -d '{
    "model_id": "qwen2.5:3b",
    "dataset_id": "test-set-123",
    "scorer": "exact_match"
  }' \
  "http://localhost:8080/admin/evals/runs" | jq '.run_id'

Returns 202 Accepted with the run_id.

Verify

• Submit an eval run; the Runs list updates in real-time.
• Click View Logs; per-row scores stream from the backend.
• Wait for the run to reach succeeded status.
• Click View Report; the pass rate and detailed results appear.
• Check the pass rate is >75% for a well-tuned adapter (baseline depends on dataset difficulty).

Common issues

Symptom	Cause	Fix
"Dataset not found"	dataset_id is invalid or doesn't exist	Pick a different dataset on the form
"Model not found" or "Adapter not loaded"	model_id or adapter_id is unknown	Use GET /v1/models to list available models; check that the adapter is promoted
Run stays running for >30 min	Eval process is hung or the model is very slow	Cancel and check api container logs; may need a smaller dataset or faster model
Run transitions to failed	Scorer crashed (e.g. LLM judge rate-limited, invalid scorer name)	Check logs; try a different scorer (e.g. use exact_match instead of llm_judge)
Report shows 0% pass rate	Outputs don't match expected outputs	Expected — indicates the model needs more training or the dataset is mismatched

Related

• Upload or build a dataset — datasets are the test fixtures for evals.
• Train a LoRA adapter (DPO) — test your trained adapters with evals.
• Promote an adapter to active — promote adapters with high pass rates.