Customer-support tone tuning

Capture your senior agents' edits to LLM-drafted replies as preference signal, fine-tune a model that stays on-brand under pressure, and gate deploys against tone regression.

The problem

Your support team uses an LLM for first-draft replies (Ada, Fin, Forethought, your own). Out of the box it sounds like an industry-average bot — too formal, off-brand, or missing the tone your senior agents rewrite into the response before sending. The fix today is a 3,000-word system prompt that mostly works for the first 100 turns and then drifts.

What you want: a model that learned your brand voice from your own agents' edits, with a deploy gate that refuses regressions.

The Bigwave shape

Component	Role
`ChannelAdapter` (Zendesk, Intercom — custom)	Ingests draft + agent-edited reply pairs as preference signal
`FeedbackStore`	Persists edits-as-preferences keyed by completion id
`dataset_builder` (`edit_implies_chosen` strategy)	Treats the agent's edit as the chosen response, the original LLM draft as rejected
`TrainingBackend.unsloth` (DPO recipe)	Fine-tunes the chat model
`EvalScorer` (custom: tone-match)	LLM-as-judge scorer that grades a candidate response on tone dimensions
`EvalGate`	Refuses adapter activation if tone regresses

End-to-end walkthrough

1. Define your brand voice (the golden set)

Pick 50 representative customer messages and write the exact reply you'd want a senior agent to send. These are your tone references.

jsonl

{"prompt":"My order is late and I'm furious.","reference":"I'm so sorry — that's frustrating, and you have every right to be upset. Let me find your order and get it sorted right now."}
{"prompt":"Can I cancel my subscription?","reference":"Of course — happy to help with that. Just so I have it right, are you on the monthly or annual plan?"}
{"prompt":"This product is garbage.","reference":"That's gutting to hear. Tell me what's gone wrong and I'll do whatever it takes to make it right."}

Save as tone-golden-v1.jsonl. Upload:

bash

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
     -F "file=@tone-golden-v1.jsonl" \
     "http://localhost:8000/admin/datasets?name=tone-golden-v1&kind=jsonl&source=manual"

This is the most important step. The model will be tuned toward this voice; if the references aren't on-brand, neither will the fine-tune be.

2. Implement the tone-match scorer

LLM-as-judge using your own model. The judge gets the user prompt, the candidate response, and the reference response. It scores along brand dimensions: warmth, brevity, formality, agency.

python

# packages/eval/scorers/tone_match.py
from packages.eval.base import EvalScorer
from packages.eval.types import EvalItemResult

JUDGE_PROMPT = """\
You are a brand-voice auditor. Compare the candidate response to the
reference. Both answer the same customer message. Score the candidate
on a 0-1 scale on tone fit:

- 1.0: indistinguishable in voice from the reference
- 0.5: same intent and content but different in formality / warmth /
  agency in a way the brand would not approve
- 0.0: wrong tone (too formal, too casual, off-brand, robotic)

Return only a number. No explanation.

Reference: {reference}

Candidate: {candidate}
"""

class ToneMatchScorer(EvalScorer):
    def __init__(self, judge_model_handle):
        self._handle = judge_model_handle

    def name(self) -> str:
        return "tone_match"

    async def score(self, *, item, candidate_output, latency_ms):
        prompt = JUDGE_PROMPT.format(
            reference=item.metadata["reference"],
            candidate=candidate_output.text,
        )
        # Call your judge model via the engine handle directly
        score = float(await self._judge(prompt))
        return EvalItemResult(
            item_id=item.id,
            score=score,
            passed=score >= 0.7,
            latency_ms=latency_ms,
            raw_output=candidate_output.text,
        )

3. Establish the baseline

Score your current model (or the hosted tool you're considering replacing — Bigwave can speak to it via the litellm gateway adapter once that ships, or wrap it in a small proxy today):

bash

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
     -d '{
       "suite": "golden_set",
       "dataset_jsonl": "...",
       "target_model": "qwen2.5:7b",
       "scorer_overrides": {"default": "tone_match"}
     }' \
     "http://localhost:8000/admin/evals/runs"

Note the baseline pass-rate. Anywhere from 0.4–0.7 is normal — the base model isn't on-brand yet; we'll tune it.

4. Capture agent edits

This is the wedge-defining data source. When a human agent edits an LLM-drafted reply before sending, that edit is the strongest preference signal you'll ever get — better than thumbs, better than ratings.

For Zendesk, write a small ChannelAdapter that listens to the ticket-update webhook. When the LLM draft text differs from the final agent-sent text, post a feedback row:

python

# packages/channels/adapters/zendesk/feedback.py (sketch)
async def on_ticket_update(event, store):
    """Webhook handler — Zendesk fires on every ticket comment add."""
    if event["type"] != "comment_added":
        return
    if event["comment"]["author_role"] != "agent":
        return
    completion_id = event["comment"]["metadata"].get("bigwave_completion_id")
    if not completion_id:
        return  # not an LLM-drafted comment

    draft = event["comment"]["metadata"]["bigwave_draft"]
    final = event["comment"]["plain_body"]
    if draft == final:
        return  # agent accepted as-is

    # Store the edit as a preference pair
    await store.record_edit(
        completion_id=completion_id,
        chosen=final,
        rejected=draft,
        actor=event["comment"]["author_email"],
    )

store.record_edit lands a row in the FeedbackStore with the edit captured as preference data. The dataset_builder strategy edit_implies_chosen reads these out as (chosen, rejected) pairs for DPO.

5. Wait for signal

DPO on agent edits wants ~500 pairs to produce a measurable delta. A support team handling 10K tickets/month with a 30% edit rate hits this in under a week. The Loop dashboard shows the rolling tally.

6. Build the preference dataset

bash

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
     -d '{
       "strategy": "edit_implies_chosen",
       "since": "2026-04-15T00:00:00Z",
       "model": "qwen2.5:7b",
       "min_pairs": 500
     }' \
     "http://localhost:8000/admin/feedback/build-dataset"

7. Train DPO

bash

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
     -d '{
       "base_model": "qwen2.5:7b",
       "dataset_id": "ds-edits-...",
       "recipe": "dpo",
       "hparams": {"lr": 5e-6, "beta": 0.1, "epochs": 1},
       "eval_split": 0.1
     }' \
     "http://localhost:8000/admin/training/jobs"

8. Score the candidate against the golden set + activate via gate

Wire the same tone_match scorer into the EvalGate policy:

python

GatePolicy(
    workspace_id="default",
    scope="adapter_attach",
    baseline_report_id="<the baseline run from step 3>",
    evaluation_dataset=open("tone-golden-v1.jsonl").read(),
    scorer="tone_match",
)

Now activate:

bash

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
     -d '{"adapter_id": "adp-tone-..."}' \
     "http://localhost:8000/admin/models/qwen2.5:7b/adapters"

If the tone regressed (rare but possible — DPO can overcorrect), the gate refuses with 422 and the old adapter stays live. If it improved, you're live with the tone-tuned model.

9. Run shadow-mode for the demo / soak window

For mid-market support orgs evaluating Bigwave, the safest entry point is shadow mode. Run Bigwave alongside the existing tool; both draft replies; agents see both; agents pick which to send.

The shadow-mode metrics:

Edit-distance from draft → sent reply, per tool
Time-to-send, per tool
Agent's explicit "this draft was better" votes

After two weeks, the data tells you whether the Bigwave tone-tune is better than the incumbent.

What you'll have at the end

A model that drafts replies in your specific brand voice, tuned on your senior agents' actual editing behaviour.
A binding tone-match gate that refuses deploys whose tone score regresses.
A self-host stack with no PII leaving your VPC.
A continuous improvement loop — every agent edit becomes future training signal.

What's not covered here

Multi-language tone tuning. Pick one language for the first pass; multi-language adds noise unless you train per-language adapters.
Salesforce / Freshdesk / HubSpot integrations. This guide uses Zendesk; the same pattern works for any ticketing system with a webhook for ticket-update.
Voice / call-center agents. Different product entirely; this is text-channel tuning only.
Auto-suggested-reply UI. Bigwave provides the back-end; let Zendesk/Intercom keep the agent UI surface.
Tone-judge robustness. LLM-as-judge for tone is noisy. Calibrate against human ratings on a sample before relying on the gate fully; bias toward false-allow over false-block early on.

Customer-support tone tuning ​

The problem ​

The Bigwave shape ​

End-to-end walkthrough ​

1. Define your brand voice (the golden set) ​

2. Implement the tone-match scorer ​

3. Establish the baseline ​

4. Capture agent edits ​

5. Wait for signal ​

6. Build the preference dataset ​

7. Train DPO ​

8. Score the candidate against the golden set + activate via gate ​

9. Run shadow-mode for the demo / soak window ​

What you'll have at the end ​

What's not covered here ​

See also ​

Customer-support tone tuning

The problem

The Bigwave shape

End-to-end walkthrough

1. Define your brand voice (the golden set)

2. Implement the tone-match scorer

3. Establish the baseline

4. Capture agent edits

5. Wait for signal

6. Build the preference dataset

7. Train DPO

8. Score the candidate against the golden set + activate via gate

9. Run shadow-mode for the demo / soak window

What you'll have at the end

What's not covered here

See also