Inspect inferences (Observability page)

What it does

Every chat completion that flows through the gateway is persisted as a row in the inference_log table. The Observability page lists those rows with model / adapter / latency / token / RAG / status columns, and the detail drawer shows the full assembled context window (system + history + RAG-injected block + current turn) plus a token-budget bar against the model's catalog context_window.

This is the single most useful page for "what did my model actually see?" debugging.

Prerequisites

• A running stack (the api container is healthy).
• At least one chat completion has gone through the gateway. If you've just installed, send one first (guide 01).

Steps

1. Open the page

Click Observability in the left rail (it's the icon-grid item roughly halfway down). The list of recent inferences renders within ~100 ms (the chunk lazy-loads on first visit).

2. Filter (optional)

Top-of-page filters:

• Model id — narrow to one model when multiple are loaded.
• Adapter id — find inferences served by a specific LoRA.
• Status — succeeded / failed / cancelled.
• Time range — 1h / 6h / 24h / 7d / all.

Click Apply to fetch. Filters are URL-encoded as query params on GET /admin/inference; you can hit the endpoint directly for the same data.

3. Open a row

Click any row. The right-side drawer slides in with five sections:

1. Context window — the headline view. Token bar fills from left to right, segmented by message role:

   • gray = system (operator's own)
   • emerald = system + RAG (injected block, when rag_kb_id is present)
   • blue = user
   • violet = assistant
   • amber = tool

   Each segment carries a hover tooltip with role + estimated tokens. Below the bar, every message in the assembled prompt renders as its own card with role badge, share %, and full content (no 1000-char truncation).

2. Request excerpt (legacy) — the old 1000-char excerpt; kept for backwards compat with rows captured before migration 0012.

3. Response — the model's reply, up to 2000 chars.

4. Metadata — model / adapter / workspace / token counts / latency / RAG kb_id + hits / agent iterations / log id.

5. RAG citations — the chunk ids the retriever returned (when rag_kb_id was set). The actual chunk text isn't stored; if you need it, the kb is still queryable via POST /admin/kb/{kb_id}/query.

4. (Direct API) Pull the same data

# List
curl -s -H "Authorization: Bearer $BIGWAVE_TOKEN" \
  "http://localhost:8080/admin/inference?limit=5" | jq

# Detail (full context-window data)
curl -s -H "Authorization: Bearer $BIGWAVE_TOKEN" \
  "http://localhost:8080/admin/inference/<id>" | jq '.request_messages, .context_window'

Verify

• Send a chat with a system prompt + user turn (guide 01).
• Refresh Observability — your new inference is the top row.
• Open it; the drawer's "Context window" section shows two cards (#1 SYSTEM and #2 USER) with the exact text you sent.

Common issues

Symptom	Cause	Fix
0 / 131,072 tokens (0.0%) despite long prompt	The model didn't return usage.prompt_tokens (Ollama doesn't always; vLLM does). The bar uses the catalog window with the model's own count — when the count is 0/null we fall back to the heuristic	Heuristic per-segment tokens still render; the headline number is just less precise
"Full context not captured for this row"	Inference predates migration 0012	New rows capture; old rows stay legacy. Run a fresh chat
401 on the page	Bearer token mismatch	cat .env | grep BIGWAVE_ADMIN_TOKEN and re-paste in the login screen
Latency column reads 49s for a 2-token reply	Model load on first request — Ollama lazy-loads the GGUF into memory	Subsequent requests reuse the loaded weights and report ms-scale latency

Related

• Send a chat message — produces the rows you'll inspect here.
• Tool calls inspector — same data filtered to rows where tool_calls is populated.
• Cost dashboard — the same inference rows, rolled up by day / model with a price table applied.