Bigwave

Appearance

Chat-agent voice consistency ​

Design-partner pitch + intake. This page is the public-facing brief for one of Bigwave's two discovery-track wedges — the harness substrate applied to a narrower vertical (mid-market support orgs), validated against buyer pull rather than declared as product direction. If a customer-facing chat agent has gone off-brand on you in production, read this end-to-end before opening a Discussion.

The pitch ​

You ship support chat at mid-market scale. A handful of agents — some hosted, some in-house — answer customer questions all day. The team has a brand-voice document. The agents read it. The agents mostly follow it. Then a base-model bump lands, or a new prompt rev goes out, or a fine-tune that was supposed to sharpen tool-calling behaviour ships, and on Tuesday a customer posts a screenshot of your support agent saying something nobody on the team would ever say. The model isn't broken. Your voice-consistency rail is.

Bigwave's harness is what that rail looks like when the buyer is a support lead, not a platform engineer. Production traffic collects per-completion feedback through POST /v1/chat/completions/{id}/feedback. That feedback funnels into a preference dataset. The DPO recipe trains a workspace- scoped adapter against that dataset. The promotion gate refuses to activate the adapter unless it beats the baseline on the eval suite. When you get to the brand-voice scorer — which is roadmap, not shipped — that gate is the place it lives.

We are explicit that this is a discovery-track wedge: the harness applied to a narrower vertical. The lead-wedge substrate (audit + fence + binding-gate + feedback / DPO loop) is what makes it possible; whether mid-market support is the right vertical to commit to is what the design-partner phase is meant to answer. If buyer pull is insufficient by Day 30 we retire the wedge and the substrate work keeps shipping anyway — the lead wedges depend on the same rails.

Ideal customer profile ​

Concrete buyer signals that say "this wedge is for you":

  • Support team running ≥ 5 chat agents in production — Intercom-/Zendesk-/Front-style deployments where customers talk to an LLM-backed bot before (or instead of) a human.
  • Brand voice already documented. Style guide exists, tone-of-voice doc exists, "we don't say it like that" rule set exists — at minimum, in a Notion doc the team agrees on. Without an explicit reference, "drift" is unmeasurable.
  • You've already had a public off-brand incident. A screenshot on social, a Slack thread that ended in a rollback, an account exec apologising to a customer for what the bot said. The team is willing to be measured against "did this happen again".
  • Multiple agent stacks running against the same backend. Not just one bot — say a public-facing support bot, an internal-knowledge bot, and a paid-tier escalation bot — that each need the same voice but currently drift independently.
  • You want a self-hosted answer. Vendor SaaS for "tone scoring" hasn't survived the InfoSec or data-residency review, or the price-per-seat model doesn't scale to your traffic.
  • You have, or are willing to build, a golden 50-message "this is what good sounds like" set. Without the golden set, neither the gate nor the eventual brand-voice scorer has ground truth to score against.

If two or more of those describe you, you're in scope. If your chat agents are internal-only and a bad answer never leaves your team's Slack, this probably isn't the right wedge.

What ships today vs roadmap ​

We are honest about this. The wedge depends heavily on roadmap work — specifically the brand-voice scorer. Every "today" claim below is verifiable in the repo on main; everything in the roadmap column is not shipped.

TodayRoadmap
Per-completion feedback collection at POST /v1/chat/completions/{id}/feedback (packages/feedback/routes.py) — thumbs / structured ratings, returns 202 with feedback_idA starter brand-voice rubric — operators currently author their own JudgeRubric tuned to their voice; we want to ship a thoughtful starter set with the design partner
Workspace-scoped feedback listing + aggregation at /admin/feedback and /admin/feedback/aggregateA/B router for canary'd voice changes — route N % of production traffic to a candidate adapter, score voice-match online, promote-or-rollback on the metric
Preference-dataset builder (packages/feedback/dataset_builder.py) — turns thumbs-up / thumbs-down feedback into dpo_pairs_v1 JSONL ready for trainingPer-completion online voice-match scoring (not just offline-eval-time) so a regression is caught the moment the adapter ships
DPO training recipe (packages/training/adapters/_unsloth_runner.py::_train_dpo) — single-GPU LoRA DPO consuming dpo_pairs_v1 JSONL (manual GPU validation)"Don't say it like that" rule-set DSL — declarative tone-policy rules (forbidden phrases, required openings, formality floor) enforced at the guardrail layer before the gate even sees the candidate
llm_judge scorer (packages/eval/scorers/llm_judge.py) — rubric-DSL scoring via injected JudgeCallable; operators define weighted criteria (factuality / relevance / tone / formality / brand-fit / etc.) per workspace, the judge model is wired via make_engine_judge against any ServingEnginePublic-leaderboard mode for support-org partners that want comparative voice-match scoring across agent stacks
Workspace-scoped AdapterRegistry + ReplicaManager so adapters route per workspace — your support workspace's voice doesn't bleed into another workspace's agentAuto-tuned baselines — once a candidate has shipped clean for N weeks, the gate proposes a tighter baseline (so voice-match doesn't silently drift downward over time)
EvalGate (packages/eval/gate.py) wired into ReplicaManager.activate — GateBlock returns 422 with the gate's reason and the old adapter handle is preserved (rollback semantics)"Voice diff" UI — given two adapters, surface the message archetypes where their voice-match scores diverge most, for human review
DriftDetector (packages/eval/drift.py) — windowed shift detection; for tone use, bucket by response signature and watch the candidate-vs-baseline distribution week-over-weekPer-customer-segment voice rubrics (the way you talk to a free-tier user is not the way you talk to an enterprise depositor) routed through the same gate
golden_set suite (packages/eval/suites/golden_set.py) — JSONL prompts + expected outputs, Wilson 95 % CI roll-up; use exact_match for response-level baselines and llm_judge with a brand-voice rubric for toneAutomated drift-detector scheduling + alert wiring (operators currently call DriftDetector.report() themselves on a window)
Append-only audit row for every gate decision, dataset upload, training submit, adapter activation at /admin/audit — the partner can answer "why did this voice change ship on date X" in under 60 seconds"Voice diff" UI — given two adapters, surface the message archetypes where their voice-match scores diverge most, for human review
End-to-end runbook (docs/runbooks/feedback-loop.md) walks chat → feedback → preference dataset → DPO → eval → promote on a single workspacePer-customer-segment voice rubrics (the way you talk to a free-tier user is not the way you talk to an enterprise depositor) routed through the same gate

Two-line summary: the loop ships, the brand-voice scorer doesn't. Feedback collection, the DPO recipe, the binding gate, and the workspace-scoped routing are all production-ready today. The brand-voice scorer that closes the wedge — turning "the gate exists" into "the gate enforces voice consistency specifically" — is the load-bearing roadmap item we want a design partner to shape. If the partner can't tolerate working against an exact_match scorer for the first few weeks while the brand-voice scorer is co-designed, this wedge is not yet ready for them.

What "validation working" looks like ​

The metric the design partner and Bigwave watch together, weekly:

  • Voice-match score ≥ 0.9 on the partner's golden 50-message set, sustained over four consecutive promotion attempts — measured by the brand-voice scorer once it lands; until then, exact-match on a curated subset stands in.
  • Zero production rollbacks caused by the off-brand class of failure in week 4 of the pilot. (If a voice-regressed adapter ships without the gate catching it once the brand- voice scorer is wired, that is a P0 for bigwave.)
  • Feedback-to-training latency — from "thumbs-down submitted by support lead" to "candidate adapter promoted with the preference signal incorporated" stays under one week. The loop closes inside the cadence of a normal sprint.
  • At least one gate-blocked activation per month that would have shipped a voice regression — measured by shadow-running the candidate post-block and confirming a voice-match score below baseline.
  • Time-to-onboard a new agent stack — adding a fresh chat bot (third-party Intercom integration, in-house ReAct agent, etc.) to the same Bigwave install, with feedback collection + gate enforcement live, takes under one hour, not days.

If those numbers don't move in the right direction in the first 30 days, something is wrong with the wedge fit and we want to hear about it.

How to engage ​

Demo / quickstart ​

A laptop with a consumer GPU is enough to exercise the wedge end-to-end with the response-level scorer that ships today — the brand-voice scorer slots into the same gate once it lands. The full operator walkthrough lives in docs/runbooks/feedback-loop.md.

bash
# 1. Install bigwave with the default stack.
git clone https://github.com/bigwaveml/bigwave.git
cd bigwave
./install.sh --non-interactive

# 2. Pull a model and confirm the runtime.
ollama pull qwen2.5:7b
bw status
curl -s http://localhost:8080/healthz

# 3. Send a chat completion.  Capture the completion id.
COMPLETION_ID=$(curl -s -X POST \
  http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $BIGWAVE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b","messages":[
        {"role":"user","content":"How do I reset my password?"}]}' \
  | jq -r '.id')

# 4. Submit per-completion feedback.  202 + feedback_id.
curl -s -X POST \
  "http://localhost:8080/v1/chat/completions/$COMPLETION_ID/feedback" \
  -H "Authorization: Bearer $BIGWAVE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"thumbs":"down","comment":"Too formal — off brand."}'

# 5. Build a DPO preference dataset from a week of feedback.
#    The dataset_builder turns thumbs into dpo_pairs_v1 JSONL.
#    See docs/runbooks/feedback-loop.md for the admin call.

# 6. Train + activate.  The gate checks the candidate adapter
#    against the workspace's golden set before it can ship.
curl -s -X POST \
  http://localhost:8080/admin/adapters/$ADAPTER_ID/activate \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN"
# → 200 if the gate passes; 422 GateBlock if it doesn't, and the
#   old adapter stays active.

# 7. Inspect the gate decision in the audit log.
curl -s \
  "http://localhost:8080/admin/audit?action=eval_gate_decision" \
  -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" | head -1

The end-to-end loop — chat → feedback → DPO → eval → promote — is documented step-by-step in docs/runbooks/feedback-loop.md. The brand-voice scorer slot, when it lands, will plug in at the same EvalScorer interface the exact_match scorer uses today.

Dual-licensed: MIT OR Apache-2.0

© 2026 bigwave contributors