The delivery rate went up. Deliveries didn’t.

Two teams were seeing different results.

The scenario gave us two observations that did not sit comfortably together. Procurement’s on-time dashboard had risen from roughly 76% to 91% after an AI dispatch rollout. Finance was waiting longer for invoices.

It would have been easy to write a story in which dispatch improved delivery and finance had a separate problem. We did not have enough evidence for that. The first job was to reproduce both numbers and find out exactly which orders each one included.

The test contained 3,600 synthetic purchase commitments across six months: three before the rollout and three after. Its brief defined two delivery measures, plus invoice receipt time. Those definitions made the central question testable: did more commitments arrive on time, or did the dashboard start describing a smaller population?

The two percentages were answering different questions.

The dashboard divided on-time deliveries by delivered orders. A cancelled order was absent. The business measure divided on-time deliveries by all commitments except customer-requested cancellations. A supplier cancelling an order still counted as a failed commitment.

That distinction is reasonable: a customer withdrawing an order and a supplier failing to fulfill it have different meanings. But if the dashboard drops both, supplier cancellations can make performance look better.

Dashboard rate = on-time deliveries / delivered orders
Business rate  = on-time deliveries /
                 (all commitments − customer cancellations)

We kept the on-time rule fixed: actual delivery on or before the promised date. Invoice time was averaged only over delivered orders, as the brief required. Mixing the populations would have produced a different measure and obscured the original disagreement.

For each month, the analysis retained total commitments, deliveries, on-time deliveries, both cancellation types and invoice days. The period totals could then be checked against six monthly rows rather than accepted as two isolated percentages.

The numerator stayed at 1,260.

The apparent gain came from dividing the same 1,260 on-time deliveries by fewer delivered orders. Deliveries fell by 270, matching the increase in supplier cancellations. When supplier cancellations remained in the business denominator, performance stayed at 72.16%.

Finance’s observation also held: invoice receipt took six days longer on average. That establishes a descriptive change, not its cause. The data did not show that the AI rollout caused cancellations or slower invoicing; supplier mix, process changes and timing would need further investigation.

A rounding error was enough to fail a check.

The first useful output was a recomputation, not an application. It retained exact decimal rates in structured results and rounded only in the report. That matters when a downstream check compares calculated values: a display value such as 72.16% is not the full-precision result.

The recorded one-month pilot passed after exact rates were preserved. The six-month expanded check then passed on the unchanged analysis artifact. Each pre-rollout month had 600 commitments, 420 on-time deliveries and 30 supplier cancellations; each post-rollout month had the same 600 commitments and 420 on-time deliveries, but 120 supplier cancellations.

Checking the months separately ruled out an aggregation mistake in this fixture. It did not provide a control group, remove seasonality or establish rollout causality. The report kept those questions open and proposed checking cancellation and delivery completeness before supplier-level follow-up.

The calculation passed. The whole task did not.

This is where a polished case study could give the wrong impression. Although the calculation checks passed, the autonomous run did not satisfy its full completion criteria. It reached a budget limit before final independent review.

The character count is an application budget measure, not a token bill. The intervention was a budget expansion. The evaluation correctly recorded a failed run: the expected completed state, final review and cost constraints were not met. It did not record a false completion.

This failure led to work on compacting repeated context and passing a smaller, fingerprinted candidate package to the reviewer. Replaying the old context reduced its size from about 78,700 to 39,500 characters. That local reduction was useful, but it was not proof that a new end-to-end run passed. Subsequent infrastructure and quota failures remained failures in the record.

Fix the scorecard before crediting the rollout.

For the business question, the next step is to show delivery and cancellation together, with the population stated next to the rate. A team should be able to see whether commitments were fulfilled, cancelled by the customer or cancelled by the supplier. That makes an improvement claim much harder to misread.

For a real deployment, we would first reconcile cancellation reasons and missing delivery records. Then we would examine supplier and category differences, check invoice timestamps and design a comparison that can distinguish the rollout from other changes. The current before-and-after data cannot do that on its own.

For the agent, the next step is equally concrete: repeat the full task under the original frozen budget and review requirements. The published result is a correct metric diagnosis inside a failed completion run. Keeping both facts in the same story is the point of this case.