The ledger

Why anybody would publish this

Three separate results here were nearly published out of silent failures, and all three had the same shape: the failure path and the data path returned the same value, so a broken call could not be told from a real one.

• A rate over nothingA position-bias figure of 0.0%, computed across 132 comparisons that had every one come back unreadable. The numerator was zero because there was no data, and the denominator counted attempts rather than answers.
• An error that said so132 comparisons failed because the request was sent with its arguments reversed. The message printed a dictionary where a hostname belongs, and was read twice as a server outage.
• A field nobody readThree comparisons recorded as "the model could not answer", from a model that had answered in full into a field the parser did not look at. It would have made a clean, wrong result about that model's abilities.

The fix is not another check. Model calls now go through one place that returns an object which knows which of several things happened — never arrived, arrived empty, arrived and was cut off, arrived and made no sense, arrived and answered. A caller that wants the text has to have looked at which.

The first version of that got it wrong in the other direction: it treated any truncated reply as a failure. The judge puts its answer on the first line, so a reply cut off afterwards is perfectly usable; a comparison puts its answer last, so a truncated one is worthless. Deciding centrally would have decided for both. Truncation is recorded as a property and the caller's own pattern decides.

What the calls have done

This counts from when the ledger was added, not from the beginning of the archive. Most of what is published on this site was produced before anything was counting, which is the point: there is no way to go back and say how many of those calls failed.