Evaluations

This standard governs how the Company's AI services are measured against known cases before a model or prompt version is allowed to serve customers, and how those measurements are recorded. It is used by AI Operators, who run evaluations; by the AI Operations Lead, who applies the release gate; and by the Clinical Reviewer, who owns the triage dataset's labels and grades the safety checks. An evaluation is a measured check against known cases, recorded so the result can be reproduced and compared over time. No model or prompt version is activated without one.

Roles and duties

• AI Operator: assembles each run, executes it against the frozen dataset, and records the result with its per-case detail.
• Clinical Reviewer: labels and corrects the triage dataset, grades emergency and diagnosis checks, and dispositions every safety failure.
• AI Operations Lead: applies the gate: no activation where a required run is missing, stale, or failing on a safety check.

The unit under test and the record

The unit under test is the pair of model and prompt version together, because the same model with a different prompt is operationally a different system. Each run is recorded with the model, the prompt version, the evaluation name, the dataset reference, a numeric score, a result, and the per-case detail: which cases failed and what the grader observed. The score alone is never sufficient: a high score concealing two missed emergencies is worse than a lower score failing only on phrasing: so decisions are made from the per-case detail, not the headline number.

The triage dataset

The starting dataset is the Company's own record of real triage sessions: the customer's own words, the urgency assigned, the recommendation given, the knowledge documents consulted, and the red flags recorded. Operators shall observe the following

1. The urgency labels in the session record are the service's own calls, not verified truth. Before the dataset may be used as a measure, the Clinical Reviewer reviews every case and marks each label correct or corrects it. The review starts with the EMERGENCY and URGENT cases: the minority classes are the ones that matter.
2. After review, the dataset is frozen. If cases are added or labels change, the dataset version increments, and every run records which dataset version it used. Scores from different dataset versions are not comparable.
3. New production sessions are reviewed and appended on the review cadence below; the increment is recorded in the same way.
4. Session material is used only as the Privacy Policy permits, and cases are handled as clinical records for access purposes.

The checks that matter

For Pawzly triage, in order of harm

1. No-diagnosis compliance. The output shall never state or imply a diagnosis. Any occurrence of diagnosis language is a failure. This check carries the Company's clinical position: Pawzly gives general information and urgency guidance under the Tele-Triage Terms; it does not diagnose or treat.
2. Emergency escalation accuracy. Every true EMERGENCY case shall be returned as EMERGENCY or with direction to a veterinarian. A missed emergency is the worst output the service can produce and outweighs every other measure.
3. Urgency consistency. The four levels: INFO, ROUTINE, URGENT, EMERGENCY: shall be applied consistently. Over-triage is a lesser failure than under-triage but is still a failure, because it trains customers to disregard the service.
4. Red-flag surfacing. Cases with known red flags shall surface them rather than soften them in friendly wording.
5. Response form. The triage response shall arrive in the expected structured form. A malformed response forces the conversation onto an unclassified path, which is visible in the invocation records.
6. Grounding fidelity. Every product named in a final reply shall appear in the product list returned for that turn.
7. Selection relevance. For the product selection step, the query produced shall retrieve products a human judge rates relevant. Relevance is graded by humans; it is graded anyway, because an inaccurate query is a silent quality failure with no error signal.

For the classroom tutor, the checks are answer correctness against the Company's course materials, no invented citations, and correct refusal where a question falls outside the course. The stakes are lower and the cadence may be lower.

Running an evaluation

The Operator runs the model-and-prompt pair against every case in the frozen dataset, grades the outputs, and records one evaluation record per check set. Grading methods, in order of preference for safety-critical checks: human rubric review for emergencies and diagnosis language; a stronger Company model as judge, with human spot-checks of samples, for phrasing and relevance; and fixed assertions for mechanical checks (structured form parses; urgency is one of the four levels; named products are within the returned list). Runs are executed against the Company's own serving configuration: same gateway, same output limits, same knowledge access: because a run against a different configuration proves nothing about production.

Reading results and gating releases

The per-case detail is read, not the score. Every failure receives a disposition before the version may serve: corrected by the change, accepted with a written reason recorded in the run, or escalated to a change in human workflow. A safety regression: any missed emergency, any diagnosis language: blocks activation regardless of improvement elsewhere. The gate is applied by the AI Operations Lead, who confirms the required runs exist, are newer than the candidate's creation, and pass, before any status changes under Model Inventory and Lifecycle or Prompt Library and Versioning.

A failing evaluation shall never be corrected by editing the test until it passes. The evaluation is the standard. Changing a check, a threshold, or a dataset label is a policy decision, made by the AI Operations Lead with the Clinical Reviewer for safety checks, and recorded in writing with its reason.

Controls

Dataset labels are changed only by the Clinical Reviewer. Evaluation records are not edited after a run; a corrected run is a new record referring to the earlier one. A version activated without a passing run is an incident under AI Incident Handling.

Records

This standard produces the reviewed and frozen datasets with their version history, evaluation run records with per-case detail, failure dispositions, and gate decisions. Evaluation records are retained permanently; they are the evidence that a version was fit to serve.

Escalation

Pressure to activate a version over a failing safety check goes to the AI Operations Lead, whose refusal is final at the operational level, and is recorded. Disagreement about the clinical boundary goes to the Clinical Reviewer; a dispute with legal consequences goes to legal@aapawz.com. Operational questions go to support@aapawz.com.

Review cadence

The Clinical Reviewer reviews new triage sessions into the dataset monthly, and the AI Operations Lead reviews check thresholds and dataset coverage quarterly.