# How Edward tests agents: CI, evals, and CI/CD

# How Edward tests agents: CI, evals, and CI/CD

Edward's answers on testing and delivery for agent systems, in his own terms.

## Testing non-deterministic model output in CI

You stop testing strings and start testing contracts.

- Schema and parse: output must be valid JSON or a valid tool call, or the build fails.
- Invariants: required fields present, IDs exist, numbers in range, no PII in logs, citations resolve, tools called only from the allowlist.
- Golden cases with assertions rather than expected full text: "must mention X," "must not invent a policy number," "must call `get_claim` before `update_claim`."
- Multi-sample on brittle cases: run 3 to 5 times, pass if at least k of N meet the contract, and track pass-rate drift.
- Pin the model ID, prompt hash, and tool schema version. A silent model upgrade is a production incident, not a surprise.
- Snapshot traces, not just the final answer. If the answer is right and the tool path is wrong, that is a fail.

Flaky LLM tests that compare prose to a golden paragraph get deleted. They train the team to ignore CI.

## The eval harness: golden sets, graders, traces

Three layers.

**Golden set.** Versioned in git. Each case has input, context fixtures (fake tickets, fake EHR snippets), allowed tools, and assertions, split by skill: extraction, RAG groundedness, tool routing, refusal, multi-step. Healthcare and finance cases are labeled with their data class.

**Graders.** Code graders first (schema, exact fields, tool sequence, retrieval hit). LLM graders second, and only for rubric items a human would score: faithfulness, completeness, tone. A second model grades the first grader on a sample to catch judge drift.

**Traces.** Every eval run stores the full span: prompt assembly, retrieved chunks with scores, model ID, token counts, every tool request and gateway decision, final output, grader scores. If it cannot be replayed, it did not happen.

Online, live traffic is sampled into the same graders. Offline golden sets catch regressions; online sampling catches the distribution shift nobody wrote a case for.

## CI/CD for an agent system: what happens on merge

Merge is not "deploy the prompt." Merge is a bundle: prompt, tool schema, policy pack, model pin, and eval-set version.

On a pull request: unit tests on tools and the gateway (deterministic); contract evals on the golden set against the pinned model; the injection pack; a cost and latency budget check; and a diff of the prompt and tool schema in the PR so a human actually reads it.

On merge to main: build an immutable artifact (no editing the system prompt in a console); deploy to a shadow or canary with the same traces and graders; compare online scores and error rates to the previous artifact; promote if the contracts hold, with automatic rollback if refusal rate, tool-error rate, or cost spikes.

Prompts in a wiki, hot-fixed in production, is how you get an incident you cannot replay.