Evaluation
Banyan Sapling's evaluation characterizes classification quality, calibrated answer coverage, and runtime reproducibility. Training and the original export, independent reload, and API checks completed on 22 October 2026.
Classification and answer coverage
| Partition | Cases | Raw accuracy | Answered | Unresolved |
|---|---|---|---|---|
| Combined evaluation | 1,731 | 94.22% | 779 (45.00%) | 952 (55.00%) |
| Controlled synthetic | 1,000 | 100.00% | 773 (77.30%) | 227 (22.70%) |
| ContractNLI | 731 | 86.32% | 6 (0.82%) | 725 (99.18%) |
| Additional known diagnostics | 45 | 82.22% | 1 (2.22%) | 44 (97.78%) |
Raw accuracy measures the highest-scoring class before the calibrated decision.
An answered case has a final status other than unsure. All 779 answered cases
in the combined evaluation matched their reference labels. There were zero
calibrated false approvals and 14 raw false approvals, where a false approval
means supported for a case with another reference label.
These results show a substantial domain difference. Controlled synthetic examples have higher accuracy and answer coverage. ContractNLI has lower raw accuracy and very low coverage under the shared calibration. The preview is therefore most relevant to experiments in controlled policy workflows, with contract generalization a development priority. The finite answered subset does not establish a general correctness guarantee.
Calibration and diagnostics
Temperature fitted to 5.171578 with a 0.5 floor. The shared nonconformity threshold is 0.995478, determined by the harder contract partition. No empty-set fallback was used on the evaluation or known diagnostic cases.
The five Marta/Anil insurance-claim variants all return unsure. For the
eligible case, the raw label is insufficient_information rather than
supported. This reflects an evidence-interpretation error as well as the
broader calibration behavior; changing the empty-set fallback alone does not
address it.
The recorded acceptance checks met their targets for zero reload drift, paraphrase stability (3.8% flips across 500 pairs), and zero calibrated false approvals. ContractNLI abstention was 99.18%, above the 15% target. The separate 20-pair known diagnostic set had one paraphrase flip. The release remains a research preview for domain evaluation and integration work.
Runtime measurements
Fresh-process comparison on 53 reference cases produced zero logit drift, including reversed and mixed request order. HTTP parity checks passed. Sequential pod-local HTTP latency on the H200 reference environment was 97.08 ms median and 111.33 ms p95. One eight-client request wave used serial model execution. Sustained throughput and broader tail latency were not measured.
Evaluation scope
The partitions include prior benchmark exposure and correlated document claims. Results characterize this release on these data; they are not an independent estimate of general deployment performance or a validation of nominal conformal coverage. Subsequent development should use new representative data, document-level splits, and separate calibration and final evaluation sets.
Machine-readable records: evaluation, runtime, training, and calibration.