Benchmark project instructions and enforced decisions with a fixed repository, task set, base commits and scoring rubric. Run clean sessions under instruction-only, decision-retrieval and retrieval-plus-enforcement conditions; measure decision discovery, citation accuracy, patch validity and violations caught. Publish failures and scope, not only an aggregate score.
Define the claim before the experiment
Instructions may increase the probability that an agent behaves correctly. Deterministic enforcement should catch a declared class of violations regardless of the agent's explanation. A benchmark should measure those different claims separately rather than asking which system is universally 'better'.
Use controlled conditions
Create tasks that touch accepted decisions, tempting rejected alternatives, objective constraints and judgement-based choices. Pin the repository and tool versions. Begin each run without conversation memory, hold task wording and model settings constant, and execute enough repetitions to show variance.
- Baseline: repository code without added project guidance.
- Instructions: concise AGENTS.md or equivalent only.
- Retrieval: instructions plus cited decision discovery.
- Enforcement: retrieval plus deterministic merge checks.
Score observable outcomes
Measure whether the correct record was found, whether its status and content were cited accurately, whether the patch built and passed tests, and whether it violated the declared constraint. Track false positives and false negatives for enforcement. Have blind human review assess design dimensions that cannot be proved mechanically.
Publish a result others can challenge
Release task definitions, repository commits, configuration, raw outcomes and scoring rules where licensing permits. State sample size and limitations. Do not convert one model, repository or task distribution into a universal claim about agent governance.