Training

Submit, list, stream, and cancel fine-tuning jobs against the configured TrainingBackend (Unsloth for the personal tier; Axolotl or peers for SME / enterprise). Successful runs auto-register the produced AdapterArtifact in the workspace's AdapterRegistry, where it becomes visible at /admin/adapters and activatable via /admin/models/{model_id}/adapters.

The router lives at packages/training/admin_routes.py. Wire schemas: CreateJobRequest, CreateJobResponse, TrainingJobRecord.

Endpoints

Method	Path	Auth	Summary
POST	/admin/training/jobs	Bearer	Submit a training job; returns 202 with {job_id}.
GET	/admin/training/jobs	Bearer	List jobs submitted by the workspace in this process.
GET	/admin/training/jobs/{job_id}	Bearer	Current TrainingJobRecord.
GET	/admin/training/jobs/{job_id}/stream	Bearer	SSE stream of status transitions + heartbeats.
POST	/admin/training/jobs/{job_id}:cancel	Bearer	Request cancellation; returns updated record.

POST /admin/training/jobs

Body CreateJobRequest:

{
  "base_model": "meta-llama/Llama-3.1-8B",
  "dataset_id": "ds_01HF…",
  "recipe": "sft",
  "hparams": {"lr": 2e-4, "epochs": 3},
  "eval_split": 0.1
}

recipe is a TrainingRecipe enum value (sft, dpo, …); hparams is a free-form mapping validated by the concrete backend; eval_split is in [0.0, 1.0].

Returns 202 with {"job_id": "..."}. Per PRD 04 NFR-1 the call must return within 500 ms p95 — the actual training runs asynchronously (subprocess for Unsloth; Arq worker when BIGWAVE_RUNNER=arq).

curl -X POST -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"base_model":"meta-llama/Llama-3.1-8B","dataset_id":"ds_01HF...","recipe":"sft"}' \
  http://localhost:8000/admin/training/jobs

GET /admin/training/jobs

Returns a JSON array of TrainingJobRecord for jobs submitted by the workspace in this API process. The tracker is per-process; restarts lose visibility into previous submissions for the direct backend. The Arq path persists job metadata in Redis so it survives restarts.

GET /admin/training/jobs/{job_id}

Returns the current TrainingJobRecord:

{
  "handle": {
    "job_id": "trj_01HF…",
    "backend_name": "unsloth"
  },
  "status": "succeeded",
  "artifact": {
    "id": "adp_01HF…",
    "base_model": "meta-llama/Llama-3.1-8B",
    "artifact_uri": "file:///var/lib/bigwave/adapters/adp_01HF.bin",
    "dataset_sha": "sha256:…"
  },
  "error": null
}

On observed succeeded, the route registers the artifact in the workspace AdapterRegistry (idempotent; safe to call from polling loops). Returns 404 when the job was not submitted by this workspace.

GET /admin/training/jobs/{job_id}/stream

Server-Sent Events. Frame layout:

• event: status — one frame per status transition (queued → running → succeeded or failed / cancelled). Data is the full TrainingJobRecord JSON.
• : ping heartbeats every 15 s keep idle proxies (nginx default 60 s) from severing the connection.
• A terminal event: status closes the stream. On succeeded, the artifact is registered in AdapterRegistry before the stream closes — clients observe the adapter at /admin/adapters once the stream ends.

Total polls are capped at one hour (_MAX_STREAM_POLLS = 7200 at 0.5 s cadence); a runaway connection emits a synthetic event: error and closes.

POST /admin/training/jobs/{job_id}:cancel

Idempotent — calling it on an already-terminal job is a no-op. Returns the updated TrainingJobRecord. When the backend cannot interrupt mid-step the route returns 409 Conflict. Returns 404 when the job was not submitted by this workspace.

curl -X POST -H "Authorization: Bearer $BIGWAVE_ADMIN_TOKEN" \
  http://localhost:8000/admin/training/jobs/trj_01HF...:cancel

Operator notes

• Workspace fence. Every route resolves the workspace from the bearer principal; the in-memory job tracker is keyed by (workspace, job_id) so a workspace cannot see another's jobs. See workspace fence.
• Direct vs Arq path. When app.state.job_runner is wired (set BIGWAVE_RUNNER=arq) the route enqueues an Arq function run_training_job; otherwise the route calls backend.submit directly. The wire shape and route contract are identical — only the durability of the tracker differs.
• Artifact registration. On succeeded, the produced AdapterArtifact is persisted in AdapterRegistry (idempotent on (workspace, adapter_id)). The GET and SSE paths both trigger this so non-streaming consumers still observe the new adapter at /admin/adapters.
• Cancel semantics. 409 Conflict from cancel means the backend received the signal but cannot interrupt mid-step. The job continues to terminal state on its own.
• 503 causes. app.state.training_backend and app.state.adapter_registry are wired in lifespan; routes return 503 with the missing-binding detail when either is absent.
• Audit. submit and cancel write training.submit / training.cancel audit rows. See Audit.
• Related guide. See 05 — Train an adapter for the operator- facing tour and feedback-loop.md for the end-to-end runbook.