# Evaluation

Banyan Sapling's evaluation characterizes classification quality, calibrated
answer coverage, and runtime reproducibility. Training and the original export,
independent reload, and API checks completed on 22 October 2026.

## Classification and answer coverage

| Partition | Cases | Raw accuracy | Answered | Unresolved |
|---|---:|---:|---:|---:|
| Combined evaluation | 1,731 | 94.22% | 779 (45.00%) | 952 (55.00%) |
| Controlled synthetic | 1,000 | 100.00% | 773 (77.30%) | 227 (22.70%) |
| ContractNLI | 731 | 86.32% | 6 (0.82%) | 725 (99.18%) |
| Additional known diagnostics | 45 | 82.22% | 1 (2.22%) | 44 (97.78%) |

Raw accuracy measures the highest-scoring class before the calibrated decision.
An answered case has a final status other than `unsure`. All 779 answered cases
in the combined evaluation matched their reference labels. There were zero
calibrated false approvals and 14 raw false approvals, where a false approval
means `supported` for a case with another reference label.

These results show a substantial domain difference. Controlled synthetic
examples have higher accuracy and answer coverage. ContractNLI has lower raw
accuracy and very low coverage under the shared calibration. The preview is
therefore most relevant to experiments in controlled policy workflows, with
contract generalization a development priority. The finite answered subset
does not establish a general correctness guarantee.

## Calibration and diagnostics

Temperature fitted to 5.171578 with a 0.5 floor. The shared nonconformity
threshold is 0.995478, determined by the harder contract partition. No empty-set
fallback was used on the evaluation or known diagnostic cases.

The five Marta/Anil insurance-claim variants all return `unsure`. For the
eligible case, the raw label is `insufficient_information` rather than
`supported`. This reflects an evidence-interpretation error as well as the
broader calibration behavior; changing the empty-set fallback alone does not
address it.

The recorded acceptance checks met their targets for zero reload drift,
paraphrase stability (3.8% flips across 500 pairs), and zero calibrated false
approvals. ContractNLI abstention was 99.18%, above the 15% target. The separate
20-pair known diagnostic set had one paraphrase flip. The release remains a
research preview for domain evaluation and integration work.

## Runtime measurements

Fresh-process comparison on 53 reference cases produced zero logit drift,
including reversed and mixed request order. HTTP parity checks passed.
Sequential pod-local HTTP latency on the H200 reference environment was
97.08 ms median and 111.33 ms p95. One eight-client request wave used serial
model execution. Sustained throughput and broader tail latency were not measured.

## Evaluation scope

The partitions include prior benchmark exposure and correlated document
claims. Results characterize this release on these data; they are not an
independent estimate of general deployment performance or a validation of
nominal conformal coverage. Subsequent development should use new representative
data, document-level splits, and separate calibration and final evaluation sets.

Machine-readable records: [evaluation](../evidence/evaluation_summary.json),
[runtime](../evidence/runtime_checks.json),
[training](../evidence/training_summary.json), and
[calibration](../evidence/calibration.json).