Test instruction following through observable evidence: start from a known commit, give a representative task, capture which decisions the agent retrieved, verify its citations, run canonical tests and deterministic policy checks, and inspect judgement-based requirements separately. Self-reported compliance is not proof.
Turn instructions into observables
An instruction such as 'respect existing architecture' cannot be tested directly. Break it into behaviours: query relevant decisions before planning, cite their IDs, avoid a known forbidden dependency, run a named command and report unresolved coverage. Each behaviour produces evidence a harness or reviewer can inspect.
Build a small evaluation set
Use real repository tasks representing normal work, an attractive but rejected approach, a missing-decision case and an explicitly superseded record. Begin every run from a clean session and fixed commit so retained conversation state does not hide weaknesses.
- Did retrieval return the expected current decision IDs?
- Did the plan cite those records without inventing content?
- Did the patch pass repository tests and objective policy checks?
- Did the agent surface ambiguity or missing authority?
Distinguish compliance layers
A passing unit test establishes only the tested behaviour. A policy check proves only its declared scope. Review is still needed for design quality and other judgement-based constraints. Report these layers separately so an evaluation score does not imply complete architectural correctness.
Use failures to improve the repository
If several agents miss the same decision, improve its discovery terms or entry-point instruction. If they find conflicting records, repair lifecycle relationships. If an objective violation repeatedly reaches review, add a deterministic check. Evaluation should strengthen project infrastructure, not merely rank models.