Run multiple models + shadow mode

What it does

Lets you load several models concurrently into the running engine (without restarting the API) and turn on shadow mode — a fire-and-forget fan-out that runs every primary chat against a candidate model in the background, logs both responses, and links them via inference_log.shadow_of so you can A/B compare a new model on real production traffic with zero risk to users.

The whole loop is operator-driven through the bw CLI; the same admin routes power the Web UI's Catalog and Evals pages.

Prerequisites

• A running stack (the api container is healthy).

• bw available on $PATH (run ./install.sh once if not).

• The base model is already pulled — for any second model you want to serve, pull it first via the active engine, e.g.:

  bw model pull qwen2.5:0.5b

  (Ollama hosts: bw model pull is just ollama pull with an HF→engine-tag resolver. vLLM/DMR hosts: pull through your engine's native flow.)

Steps — Multi-model

1. Load a second model into the running engine

bw model load qwen2.5:0.5b

The API calls engine.load(ModelConfig(model_id)) and registers the new replica with the ReplicaManager. Idempotent — re-loading an already-loaded model is a no-op.

2. List what's loaded

bw model loaded

Returns the registered set (id + replica id + active adapter). Mirrors the data the /v1/models OpenAI catalog endpoint serves to clients, so any UI dropdown that reads /v1/models (Evals target, Chat panel, etc.) auto-populates.

3. Send a chat to a specific model

The OpenAI body's model field picks the replica. No header magic:

curl -s -X POST -H "Authorization: Bearer $BIGWAVE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Hi"}]}' \
  http://localhost:8080/v1/chat/completions | jq -r '.choices[0].message.content'

In the Web UI, both models appear in the chat panel's settings dropdown and on the Evals target-model select — so you can run the same golden-set suite against each model and compare scores side-by-side.

4. Unload a model

bw model unload qwen2.5:0.5b

Drops the handle from the manager and asks the engine to release the weights. The UI dropdowns refresh on next page load.

Steps — Shadow mode

1. Make sure the shadow target is loaded

Shadow mode requires the candidate model to be in the loaded set — the API rejects setting a shadow target that isn't registered:

bw model load qwen2.5:0.5b

2. Turn on shadow mode

bw shadow set qwen2.5:0.5b

Stores the target on app.state.shadow_model_id. Every subsequent non-streaming chat completion forks an asyncio.create_task that calls the shadow handle with the same assembled messages list (post- RAG injection) and writes the response to a sibling inference_log row tagged shadow_of=<primary_completion_id>.

The primary response is never blocked — the user sees their response at the same speed as without shadow, and shadow errors are logged-and-swallowed.

3. Send some traffic

Use the chat panel, hit /v1/chat/completions directly, or kick off an eval — anything that produces a primary inference_log row. Keep in mind:

• Streaming + agent-loop primaries do not fork shadows in v0. Send "stream": false to exercise the path.
• Tool-call primaries also skip shadow today.

4. Inspect the pair

docker compose exec -T postgres psql -U bigwave -d bigwave -c \
  "SELECT id, model_id, shadow_of, status, response_excerpt
     FROM inference_log
     WHERE id = 'chatcmpl-…' OR shadow_of = 'chatcmpl-…'
     ORDER BY ts;"

Or open the Observability page, click the primary row to expand the detail panel, then click the row tagged with the shadow's id — you'll see a yellow "Shadow lineage" callout linking back to the primary.

5. Turn shadow off

bw shadow clear

Clears app.state.shadow_model_id; subsequent primary requests stop forking. Existing shadow rows in inference_log stay (they're real data — useful for cost rollups and eval re-runs).

(Direct API) Reference

Verb	Route	Body	Effect
POST	/admin/models/load	{"model_id":"qwen2.5:0.5b"}	Load + register
DELETE	/admin/models/{model_id}	—	Unload
GET	/admin/models/loaded	—	List loaded
GET	/admin/shadow	—	Status
POST	/admin/shadow	{"model_id":"qwen2.5:0.5b"}	Set target
DELETE	/admin/shadow	—	Clear target

All require the same bearer token as the rest of the admin surface.

Verify

• bw model loaded lists ≥ 2 entries after bw model load.
• /v1/models returns the same set; the Evals target-model dropdown shows every loaded id.
• bw shadow set <id> returns {"model_id": "<id>"}.
• A primary chat completes normally; ~1–10 s later a sibling row appears in inference_log with the right shadow_of.
• The Observability detail panel for the shadow row shows a yellow "Shadow lineage" callout pointing at the primary.

Common issues

Symptom	Cause	Fix
bw model load returns 500 with "engine.load failed"	Backend doesn't recognise the id	bw model pull <id> first; for vLLM/DMR use the canonical id (Qwen/Qwen2.5-0.5B-Instruct)
bw shadow set returns 422 with "is not loaded"	The shadow target isn't in the loaded set	bw model load <id> first
Shadow rows never appear	Primary requests are streaming or use the agent loop (both skip shadow in v0)	Send a non-streaming, non-tool request to trigger the fork
Shadow latency dominates response time	The fan-out is async — if the primary feels slow, it isn't shadow's fault	Inspect Observability for the primary's latency_ms; shadow runs after the primary returns
Shadow target survives a restart but app.state.shadow_model_id is empty	The flag is in-memory only	bw shadow set <id> again after restart, or script it as a startup hook
Two models loaded but only one appears in /v1/models	The api restarted after dynamic loads	Dynamic loads are in-memory; re-issue bw model load <id> after every restart, or move the model to BIGWAVE_FALLBACK_MODELS so the lifespan picks it up

Related

• Send a chat message — primary requests are what shadow forks against.
• Inspect inferences (Observability) — the "Shadow lineage" callout lives in the detail panel.
• Run an eval — the target-model dropdown reads from the same /v1/models endpoint, so dynamically-loaded models appear automatically.
• Model catalog — browse what's available to pull before loading.

Background

ADR 0022 (docs/adr/0022-multi-model-and-shadow.md) covers the decision shape and tradeoffs in detail — why a single shadow target, why streaming skips shadow, why dynamic load instead of a restart loop.