The policies it ships with

People and their agents follow these when they record in a collection made from this template. Its owners can change them.

1. A node belongs in an AI-eval log when it is one of: an Eval (a test that measures some behaviour of an AI system — its task, the data it runs on, how the output is graded, and what counts as success); a Log (one recorded run of an eval, carrying its score and pass/fail verdict and the versions it ran against); or a Reference (the dataset / golden set being tested, the system under test such as a model + prompt version, the eval harness/code, or an external benchmark or report). PASS when the candidate is one of these. FAIL when it is a free-form note, a product or process artifact, or anything with no role in defining or recording an evaluation.
2. Only Eval, Log, Reference, Rule, and Principal belong in an AI-eval log. An Eval is the test definition (what is measured, on what data, how it is graded, the target); a Log is one recorded run of an eval (its score and verdict, and the versions it ran against); a Reference is the versioned dataset / golden set, the system under test (model + prompt version), the harness/code, or an external benchmark or report; a Rule captures a success criterion or release gate; a Principal is the owner — person or agent — accountable for the eval. Process steps, free-form notes, and ship decisions belong in their own collections.
3. Only these relationship edge types may be used in an AI-eval log: `supports` (a Log → the Eval it is a run of; a dataset/report Reference → the Eval it informs; an Eval → a claim it validates), `derived_from` (a Log → the system-under-test or dataset version it ran against; an Eval → a benchmark it adapts), `attributed_to` (an Eval or suite → the Principal who owns it), `constrained_by` (an Eval → a Rule stating its success criterion or release gate), `has_parent` (an Eval → the suite it belongs to), `replaces` (a new Eval version → the one it supersedes), and `relates_to` (a see-also link between related evals).
4. Every committed (`queued` or `active`) Eval declares its grading method in `criterion` — how a run's output is turned into a pass/fail or score: an exact/programmatic check, a shape/structural check, or an LLM-as-judge rubric. An eval with no grader can't be run; a `drafting` sketch may add it later.
5. Check the Eval's `prose` and `criterion`. PASS when the eval is well-formed: it states WHAT behaviour is measured, on WHAT input or dataset, HOW the output is graded, and WHAT counts as success — a specific, measurable target (e.g. `exact-match accuracy ≥ 0.9 on the 200-case golden set`, `0 responses leak PII over 10k red-team prompts`, `LLM-judge rates tone ≥ 4/5 on 100 inquiries`). FAIL with a reason when it is vague about what success means (`works well`, `good quality`), names no way to grade the output, or bundles several unrelated checks into one eval.
6. Every committed (`queued` or `active`) Log is one run of an eval: it links to that Eval with a `supports` edge. A run with no eval it belongs to is an orphaned number — there is nothing to compare it against, trend it over time, or attribute it to. A `drafting` run may defer the link.
7. Check the Log's `prose`, `outputs`, `inputs`, and `happened_at`. PASS when the run records BOTH (a) its result — the score(s) and/or the pass/fail verdict for this execution — AND (b) the versions it ran against, enough to reproduce and compare it: the system under test (model + prompt/agent version, or commit) and, where relevant, the dataset version. FAIL with a reason when it records a bare verdict with no score, or no pinned versions, so a later reader can't tell what was tested or whether a change moved the number.
8. Keep the two layers distinct: the Eval is the durable DEFINITION (what is measured, on what data, how it is graded, the target) and changes rarely; each run is a separate, append-only Log RESULT linked to it by `supports`. Never overwrite a past run to record a new one — add a new Log — so the trend over time stays visible and regressions are catchable.
9. Choose the grading method deliberately and record it in the Eval's `criterion`: prefer a code-based / programmatic check (exact match, string/shape assertion, tool-call or final-state verification) when the answer is checkable — it is fast, cheap, and deterministic; use an LLM-as-judge with a clear, empirical rubric for open-ended or subjective output, and calibrate it against human labels before trusting it at scale; reserve human grading for the gold-standard spot checks that keep the automated graders honest. Many evals combine more than one.
10. Make success criteria SMART — specific, measurable, achievable, relevant. Capture the threshold or release gate as a Rule linked to the Eval with `constrained_by` (e.g. `F1 ≥ 0.85 on the held-out set`, `< 0.1% of 10k trials flagged toxic`, `p95 latency < 200ms`), so pass/fail is judged against an explicit bar rather than a vibe. Most real evals are multidimensional — quality plus safety plus latency plus cost — so attach more than one Rule when they apply.
11. Keep the test cases as a versioned dataset: model the golden set / eval data as a Reference whose `locator` points at where it lives, and cite it from the Eval with `supports` (or `derived_from`). Version it with immutable snapshots, keep it small but ruthlessly curated, and grow it by turning real production failures and user-reported bugs into new cases.
12. Pin what was evaluated on every run: an eval measures the harness AND the model together, so record the system under test — model + version, prompt/agent version, config, and code commit — on the Log (in its `inputs`, and/or as a `derived_from` edge to a Reference for that version). Two runs are only comparable when you can see exactly what changed between them.
13. Record the metric and the aggregate score, not just a verdict: accuracy, F1, precision/recall, an LLM-judge score, or operational numbers like latency, cost, and tokens. For non-deterministic systems, run multiple trials and report the right aggregate — `pass@k` (succeeds at least once in k tries) for capability, `pass^k` (succeeds on every one of k tries) for reliability — since the two diverge sharply as k grows.
14. Distinguish capability evals from regression evals. A capability eval targets behaviour the system struggles with and starts at a low pass rate; once it passes reliably, graduate it to a regression eval kept near 100% to guard against backsliding. Say which an eval is in its `prose`, and watch for saturation — when a capability eval nears 100% it has stopped discriminating and needs harder cases.
15. Give every eval an owner — a person OR an agent — by attributing it (or its suite) to a Principal with `attributed_to`, and set that Principal's `kind` to `human` or `agent`. Eval suites rot without maintenance: someone has to read the transcripts, confirm the graders still grade correctly, retire saturated cases, and add new ones from fresh failures.
16. Group related evals into a suite: make the suite a parent Eval and link each member to it with `has_parent`, so a capability or product area reads as one set with a shared pass rate. When an eval's methodology changes, do NOT edit it in place — create the new version and link it to the old one with `replaces` (retiring the old), so past results stay interpretable against the definition that produced them.
17. Agents: read `GET /<handle>/api/authoring-contract.json` and write with `POST /<handle>/api/changesets.json`. Create each run as a Log together with its `supports` edge to the Eval (and a `derived_from` edge to the system-under-test Reference) in the same changeset, recording the score, the verdict, and the pinned model / prompt / dataset versions — rather than leaving disconnected nodes.