---
layout: docs
title: Agent Orchestration
section: Pi Dash
order: 9
---

# Agent Orchestration

This page covers the cloud-side logic that turns a user's work item into a concrete run on a runner. The work happens across three Django modules:

```
pi_dash/orchestration/   ← phases, scheduling, workpad, done signals
pi_dash/prompting/       ← prompt fragments, composition, rendering, context
pi_dash/scheduler/       ← built-in scheduling hooks + signals
```

Plus the runner-side glue in `pi_dash/runner/services/` (matcher, run lifecycle, outbox, scheduler hook).

## Concept map

- **Work item** — the user-facing thing (issue/task/sub-issue) tracked in `app/`.
- **Run** — one execution attempt against a work item, persisted in `runner/models.py`. A work item can have many runs (retries, follow-ups).
- **Phase** — runs progress through ordered phases (`orchestration/agent_phases.py`). Each phase has its own prompt template and expected outputs.
- **Workpad** — the durable scratchpad attached to a run that the agent reads/writes structured artifacts into (`orchestration/workpad.py`).
- **Done signal** — explicit completion contract the agent emits to mark a phase done (`orchestration/done_signal.py`).
- **Pod** — a grouping primitive on the runner side (`runner/models.py`, `services/pod_naming.py`). Think of it as the namespace a run lives in on a particular runner.

## `orchestration/`

```
agent_phases.py   ← phase definitions + transitions
scheduling.py     ← decides when a run is eligible to dispatch
service.py        ← high-level "create a run from a work item" entrypoint
signals.py        ← Django signals fired during the run lifecycle
done_signal.py    ← parsing / validating the agent's "I'm done" frame
workpad.py        ← workpad CRUD + lifecycle
```

`service.py` is the canonical entrypoint. Most product code that wants to "kick off an agent on this work item" goes through it; it composes prompt → phase → run → persists.

## `prompting/`

```
composer.py       ← assembles a final prompt from fragments + context
context.py        ← gathers context (work item, project state, prior runs)
renderer.py       ← template render
fragments/        ← reusable prompt fragments
models.py         ← persisted prompt templates / fragment overrides
seed.py           ← seed default prompts
views.py / urls.py / serializers.py  ← admin API for editing prompts
```

Prompts are not hard-coded — they live in DB rows seeded by `seed.py`. Instance admins can override fragments from the admin UI, which is why `prompting/` ships a full REST surface.

## `scheduler/`

`scheduler/builtins/` holds the default scheduling rules; `signals.py` wires them into the run lifecycle. This is the layer that asks "is now a good time to dispatch?" — it can defer runs based on quotas, project state, manual gating, etc.

The runner-side scheduler hook (`pi_dash/runner/services/scheduler_hook.py`) is the bridge between this layer and the runner's HTTP long-poll: when a run becomes eligible, the matcher hands it to the next runner that asks.

## Runner-side dispatch services (`pi_dash/runner/services/`)

```
matcher.py            ← match an eligible run to a runner (project, capabilities, availability)
run_lifecycle.py      ← state-machine transitions (assigned → running → completed/failed)
session_service.py    ← welcome frame (incl. version advisories — see 07)
outbox.py             ← outbound message queue for cloud → runner notifications
pubsub.py             ← Redis pub/sub bridge for live status updates to web UI
chat.py               ← in-run chat messages
permissions.py        ← who can mutate which run / runner
pod_naming.py         ← derive pod names from project / runner / config
tokens.py             ← refresh/access token pair lifecycle
validation.py         ← payload validation helpers
runner_delete.py      ← runner removal — coordinates cloud + runner-side cleanup
```

## End-to-end (cloud's view)

```
User opens work item  →  triggers orchestration.service.create_run(work_item, phase)
                       ├─ prompting.composer.compose(work_item, phase, context)  → prompt text
                       ├─ persist Run row (status=pending) + workpad init
                       └─ scheduler.signals fire → eligibility set

Run becomes eligible   →  matcher.assign(run, available_runners)  → Run.assigned_to = runner

Runner long-polls      →  GET /api/v1/runner/runs/   (per-runner, see 07)
                       ←  yields the assigned Run + its prompt + workpad ref

Runner executes        →  POSTs events / approvals / outputs back
                       →  run_lifecycle transitions state
                       →  pubsub broadcasts to web UI for live progress

Agent emits done_signal → orchestration parses + validates → phase complete
                       →  advance to next phase OR finalize run
```

## Run failure taxonomy and retry policy

Every failed run stores one canonical value in `AgentRun.failure_reason`,
classified at write time by `pi_dash/runner/failure.py::classify` from the
runner protocol's `FailureReason` plus the raw error text (old runners that
only send the coarse protocol reasons still classify — the text rules fill
the gap). The stored strings are a wire contract for dashboards and API
filters: never rename one, only add.

The ticker consults `policy_for(reason)` when a run fails without yielding
(`orchestration/scheduling.py::_apply_failure_policy`):

| Policy                        | Ticker behaviour                                                                                                                                                                                                                                  |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Retry soon, free**          | Re-queue a free entry on a short backoff (`FREE_RETRY_BACKOFF_SECONDS`, 120 s). Does not spend a tick. Capped at `FREE_RETRY_MAX_ATTEMPTS` (3) consecutive tries, then the backstop stops the clock.                                              |
| **Retry on the normal clock** | Today's behaviour: keep ticking, spend a tick per retry.                                                                                                                                                                                          |
| **Needs a human**             | Stop the clock with disarm reason `failure_needs_human`; no further ticks are spent. The failure comment names the reason and the fix. Never auto-pauses — Re-tick (re-arms without granting budget), Run AI, or a state move resumes once fixed. |
| **Fresh session**             | Clear the failed run's `thread_id` so the next continuation starts a fresh session instead of resuming the one that will fail again. Ticks on the normal clock.                                                                                   |

Backstop regardless of policy: `REPEATED_FAILURE_LIMIT` (3) runs in a row
failing with the same reason stops the clock with disarm reason
`repeated_failure` (free-retry reasons get their retry allowance first, so
their backstop sits at `FREE_RETRY_MAX_ATTEMPTS + 1`). Same recovery as
`failure_needs_human`.

| Reason                                        | Side     | Policy                           |
| --------------------------------------------- | -------- | -------------------------------- |
| `workspace_setup`                             | platform | needs a human                    |
| `git_auth`                                    | platform | needs a human                    |
| `network`                                     | platform | retry soon, free                 |
| `runner_offline`                              | platform | retry soon, free                 |
| `daemon_restart`                              | platform | retry soon, free                 |
| `assign_rejected_busy`                        | platform | retry soon, free                 |
| `timeout`                                     | platform | normal clock                     |
| `max_turns`                                   | platform | normal clock                     |
| `internal`                                    | platform | normal clock                     |
| `agent_error.provider_auth_or_access`         | agent    | needs a human                    |
| `agent_error.provider_quota_limit`            | agent    | needs a human                    |
| `agent_error.provider_capacity_or_rate_limit` | agent    | normal clock                     |
| `agent_error.provider_server_error`           | agent    | normal clock                     |
| `agent_error.provider_network`                | agent    | retry soon, free                 |
| `agent_error.context_overflow`                | agent    | fresh session                    |
| `agent_error.model_not_found_or_unavailable`  | agent    | needs a human                    |
| `agent_error.missing_executable`              | agent    | needs a human                    |
| `agent_error.missing_config`                  | agent    | needs a human                    |
| `agent_error.unsupported_version`             | agent    | needs a human                    |
| `agent_error.empty_or_unparseable_output`     | agent    | normal clock                     |
| `agent_error.process_failure`                 | agent    | normal clock                     |
| `agent_error.unknown`                         | agent    | normal clock (fail-safe default) |

Classifier notes: HTTP status codes only match on digit boundaries (so
`402913 tokens` / `15290ms` / `exit status 4030` never land in a provider
bucket); pre-launch failures route platform-side by construction and never
reach the agent-text rules; capacity wording is checked before the auth
status codes because Anthropic-compatible providers use 403 for concurrency
rejections; anything unrecognised falls into the normal-clock bucket, never
the free-retry one. Refusals are out of this taxonomy — `REFUSED` +
`refusal_category` model those separately.

## Extending the system

- **New phase** — add to `orchestration/agent_phases.py` and a matching prompt template in `prompting/fragments/` + `seed.py`.
- **Custom scheduling rule** — add a rule under `scheduler/builtins/` and wire it via `scheduler/signals.py`.
- **New agent** — implement the agent trait in `runner/src/agent/` (Rust) and a matching dispatch path in `runner/src/codex/`-equivalent module. The cloud side is agent-agnostic: it only cares about the prompt and the done-signal contract.

## Where to read next

- [06 — Backend architecture](/docs/pi-dash/backend-architecture) — the modules referenced here in context
- [07 — Runner architecture](/docs/pi-dash/runner-architecture) — the runner's view of dispatch
- [08 — Cloud ↔ runner protocol](/docs/pi-dash/cloud-runner-protocol) — the wire schema runs flow over
- `pi_dash/orchestration/service.py` — the canonical entrypoint
