Feedback aggregation

Roll up raw thumbs events into a per-(model, adapter, prompt template, channel) report, then project the underlying events into a registered dpo_pairs_v1 dataset suitable for direct DPO training. Together these endpoints power the "Feedback → DPO" half of the feedback-loop runbook and the 04 — Build a dataset how-to.

The router lives at packages/feedback/routes.py. Aggregation logic is in packages/feedback/aggregate.py.

Endpoints

Method	Path	Auth	Summary
GET	/admin/feedback/aggregate	Bearer	Workspace-scoped feedback rollup grouped by provenance.
POST	/admin/datasets/from-feedback	Bearer	Build a dpo_pairs_v1 dataset from feedback events.

GET /admin/feedback/aggregate

Query params:

Param	Type	Notes
model	string	Exact match.
adapter	string	Exact match.
prompt_template_version	string	Exact match.
channel	string	Exact match.
since / until	ISO-8601 datetime	Inclusive UTC bounds.

Filters compose with AND. The workspace is taken from the bearer principal (PRD 12 FR-7) — there is no query string for it.

Response shape: AggregateReport (packages/feedback/aggregate.py:73):

{
  "rollups": [
    {
      "model_id": "llama3.2:3b",
      "adapter_id": null,
      "prompt_template_version": "v1",
      "channel": "web",
      "up": 42,
      "down": 7,
      "total": 49
    }
  ],
  "total_events": 49,
  "workspace_id": "default",
  "since": null,
  "until": null
}

rollups is sorted by total DESC, model_id ASC so the most active bucket is first. up / down come from a SQL CASE expression so the entire rollup is one round-trip — no Python-side counting.

curl -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  "http://localhost:8000/admin/feedback/aggregate?since=2025-09-01T00:00:00Z"

POST /admin/datasets/from-feedback

Build a preference dataset from feedback rows and register it with the dataset repository. Body schema: FromFeedbackRequest (packages/feedback/routes.py:313).

{
  "since": "2025-09-01T00:00:00Z",
  "until": "2025-09-30T23:59:59Z",
  "criteria": {
    "model_id": "llama3.2:3b",
    "adapter_id": null,
    "prompt_template_version": "v1",
    "channel": "web"
  },
  "pair_strategy": "thumbs_pair",
  "balanced_negative_fraction": 0.5,
  "balanced_base_strategy": "thumbs_pair",
  "name": "support-thumbs-2025-09"
}

Field rules:

• pair_strategy — one of thumbs_pair | balanced | latest_thumbs_down (see PairStrategy).
• balanced_negative_fraction — (0, 1], applies only when pair_strategy == balanced.
• name — optional dataset label (≤128 chars); auto-generated otherwise.

Returns 201 with a BuildResult (packages/feedback/dataset_builder.py:133):

{
  "dataset": {
    "dataset_id": "ds_01HF…",
    "name": "support-thumbs-2025-09",
    "format": "dpo_pairs_v1",
    "rows": 320,
    "sha256": "…",
    "created_at": "2025-10-01T00:00:00Z"
  },
  "n_pairs": 320,
  "n_events_considered": 412,
  "n_events_skipped": 92,
  "pair_strategy": "thumbs_pair",
  "format": "dpo_pairs_v1"
}

Skipped events typically had no resolvable prompt (the corresponding inference-log row was pruned) or were the lone half of a pair the strategy could not complete.

curl -X POST -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"pair_strategy":"thumbs_pair","name":"thumbs-rollup"}' \
  http://localhost:8000/admin/datasets/from-feedback

Operator notes

• Workspace fence. Both routes filter on workspace_id from the bearer principal. Cross-tenant lookups return empty data; they never error so absence is indistinguishable from "no events".
• Empty dataset. When the criteria match zero pairs, the route still returns 201 with n_pairs=0 — the dataset row exists but has no rows. Trainers should reject zero-row datasets.
• Dataset format. Output is always dpo_pairs_v1; consume it with the Unsloth backend directly.
• Listing the raw events. GET /admin/feedback (defined in the same router) returns the unrolled-up event stream; the aggregate endpoint is the rollup. See 03 — Capture feedback for the submission surface.
• Runbook. Step-by-step end-to-end recipe in feedback-loop.md.