The AI answered. Why was the backlog still growing?

What happens after the AI responds?

The support story on our homepage starts with an uncomfortable question: if the system is handling the work, why is the backlog still growing? The investigation has to distinguish a useful intermediate action from a completed customer outcome.

We built an offline replay with 20,000 support tickets and 224,260 events. The fixture includes out-of-order events, duplicates, missing fields, transfers and reopened issues. Its workflow patterns are deliberately constructed. This makes it a repeatable engineering test of measurement and state handling, rather than a claim about a customer’s production results.

In that history, 18,000 tickets had a knowledge card selected. Only 4,000 met the closure rule. Both counts were correct. Treating the first as the second was the measurement error.

Reconstruct the state before calculating the rate.

A ticket can be answered, transferred, confirmed and reopened. Counting rows or taking the last imported row loses those distinctions. The replay begins with tickets created before a fixed snapshot time and processes events in event-time order, with event ID and row ID as tie-breakers.

Each event must have a valid ID, refer to a known ticket, use a recognized event type, and occur no earlier than ticket creation. Events after the snapshot are excluded. Repeated event IDs count once. The run removed 2,220 duplicates and rejected 40 invalid events, leaving 222,000 valid unique events.

Resolution starts a new confirmation cycle. A confirmation is recognized only after resolution. A close event is recognized only after both. Reopening clears resolution, confirmation and closure, so yesterday’s confirmation cannot close today’s reopened issue.

on resolved:           resolved = true; confirmed = false; closed = false
on customer_confirmed: if resolved: confirmed = true
on closed:             if resolved and confirmed: closed = true
on reopened:           resolved = false; confirmed = false; closed = false
on transferred:        owner = none

This is a shortened description of the implemented state transitions. The input database stays read-only; its SHA-256 fingerprint is compared before and after the run. Proposed actions are written separately from the source.

One backlog contained five different states.

At the snapshot, the fixture separated into five groups of 4,000. A single backlog total would hide the next action each group actually needs.

The key distinction is between unfinished administrative closure and an unresolved customer problem. An open ticket is not proof that the customer is still waiting for an answer. Equally, an answer selected by the system is not proof that the customer’s issue is resolved.

Why we did not simply close the backlog.

Once we had identified the missing close step, the tempting shortcut was obvious: close everything that looked resolved. We did not use that rule. A resolution event without customer confirmation belongs in a different state, and an old confirmation must not survive a reopen event.

We also kept assigning an owner separate from closing a case. The former answers “who is responsible?” The latter claims a customer outcome. Combining them in one success flag would recreate the same measurement problem we were trying to remove.

A third option was to improve answer selection. The replay could not establish whether a better answer would help unresolved cases; it had no randomized routing experiment or new customer responses. What it could establish was narrower: the already-confirmed group needed a downstream close action. Improving an upstream choice would not execute that action.

The resulting action file therefore had different instructions for different states: close after confirmation, assign an owner, or request confirmation. Those instructions were proposals. No customer was contacted and no production ticket was changed.

Change one rule. Keep the history fixed.

We compared four deterministic policies against the same reconstructed states. This is a policy simulation, not a randomized customer A/B test. The denominator remains 20,000 tickets throughout.

Ownership improves under the assignment policy, while closure does not. That is a useful negative result: assigning someone to a case must not invent a resolution or a customer confirmation. The combined workflow also proposes confirmation requests, but those requests do not count as successful confirmations.

The extra closures come entirely from issues already resolved and confirmed. The simulation assumes the proposed close actions execute; it does not model staff capacity, customer response probability or confirmation expiry. Those are questions for a live pilot.

A report, a reproducible calculation and a reviewable action list.

The deliverables were a ticket-level findings file, a proposed-action list, a machine-readable summary and a report. Each proposed action retains the ticket identifier, observed state and simulation flag. A reviewer can trace an aggregate result back to its input state without changing a production record.

The recorded verification covers 16 conditions: dataset counts, event cleaning, state totals, source integrity, simulated closures, ownership and the invariant that assignment cannot create closure. All 16 passed. We reran the replay directly against the stored SQLite database while preparing this article and reproduced the state totals with an unchanged source.

This scale experiment used a deterministic implementation written with Codex; it was not a new autonomous Polaris model run. A separate, smaller model-driven closure-metric experiment is recorded with 22 model calls, frozen pilot and expanded checks, and final review. They are different experiments and should not be combined into one result.

The practical deployment question is now specific: do the real system’s resolution, confirmation, reopen and closure events have the same semantics? Establish that mapping, validate confirmation age, and review a limited action set before enabling writes. Better retrieval may still be valuable, but it cannot repair an absent downstream transition by itself.