Did AI deliver what you expected?

In a study of 2,000 CEOs in 33 countries (IBM Institute for Business Value with Oxford Economics, February to April 2025), CEOs said only 25% of their AI initiatives had delivered the expected return over the last few years.

That figure leaves a question open: expected by whom, and written down where?

In many companies, the expectation was never written down. A team approved an AI tool, the bills started to arrive, and now someone in a budget review asks whether it was worth it. Because the measuring starts after the spend, the result has nothing to compare with. Two dollars per resolved ticket might be a good result or a poor one, and you can only tell which if you decided, before the spend, what a resolved ticket should cost.

A cost per outcome means nothing until you compare it with the one you expected.

Tokens and seats measure what goes in

Cost per token and seat counts are useful numbers. Engineering teams use token prices to choose between models, procurement uses seat counts to stop paying for seats nobody uses, and each vendor's console shows its own tokens and seats well.

What both of them measure, though, is what goes in. A cheaper model can cost more per result if it needs three tries to answer a ticket. A seat used every day tells you someone is busy, but not what the work produced. And a vendor's console shows the vendor's side of the work; it does not see your help desk or your sales system.

To judge value, you need a unit the business already cares about: one finished piece of work.

Pick an outcome the business already counts

The easiest outcome to trust is one that a system you already run records for its own reasons:

• Support: a ticket resolved in the help desk, and not reopened within seven days.
• Engineering: a pull request merged in the code host.
• Sales: a deal marked won in the CRM.

A good outcome is one that someone outside the AI team already trusts, that has a clear state for success, such as resolved, merged or won, and that you can tie to the AI work by the workload, the key or the time it ran.

Write down what counts as success before you start counting, and keep it fixed once the numbers arrive, because "resolved" and "resolved and not reopened" give different answers.

None of this means reading the ticket or the code. You need the record's state and its date, and the conversation, the prompt and the output stay where they are.

Cost per successful outcome

The arithmetic itself is one division:

Cost per successful outcome = attributed spend ÷ successful outcomes.

Attributed spend is everything the workload cost in the period: every model call, every retry, every run that failed or ended with a person taking over. Successful outcomes are only the ones that meet your definition.

Keep the failures in the cost. It is tempting to make the number look better, either by dividing by all the work the AI touched or by leaving failed runs out of the spend, but both hide what the unsuccessful runs cost. With failures kept in, the number goes up when the AI fails more often, which is the signal you want.

Say how sure you are

A cost per outcome is only as good as its count of outcomes, so each figure should say how that count was established. We call this the evidence class, and it has three values:

• Observed. The count comes from the system that records the outcome, joined to the spend. You can make a decision on it.
• Declared. A person gave the count, for example "the agent resolves about 400 tickets a month." That is a good place to start, though it is not proof.
• Not measured yet. Nothing is joined yet, so no number is shown and none is guessed.

"Not measured yet" is still a useful answer, because it tells you which join to build next.

Expected first, then actual

Before you approve the spend, write down two numbers for the workload:

• the cost per successful outcome you expect, and
• what one outcome is worth to the business.

The second number is a business judgment rather than arithmetic. A support lead might set it at what it costs a person to resolve the same ticket, and because a person set it, you mark it as declared.

Then compare every month. The return is the value per outcome times the successful outcomes, set against the attributed spend. Put each workload on its own line, month by month, and you have an AI P&L, one report that finance and technology can read the same way.

An example

This is an example, not a customer, and the numbers are round so the arithmetic is easy to follow.

Picture a support team that adds an AI agent to its help desk. Before approving it, the team writes down two numbers: one resolved ticket should cost $1.50 in AI spend, and it is worth $6 to the business. The $6 is declared, because the support lead set it, while the ticket counts are observed, because they come from the help desk.

• Month 1

  Attributed spend

  $3,000

  Resolved tickets

  1,500

  Cost per resolved ticket

  $2.00

  Value at $6

  $9,000

• Month 2

  Attributed spend

  $2,400

  Resolved tickets

  1,600

  Cost per resolved ticket

  $1.50

  Value at $6

  $9,600

• Month 3

  Attributed spend

  $2,500

  Resolved tickets

  Not measured yet

  Cost per resolved ticket

  Not measured yet

  Value at $6

  Not measured yet

In month 1, the agent worked on 2,500 tickets and resolved 1,500 of them. Each resolved ticket cost $2.00, above the $1.50 the team expected. On return alone the month looked fine, with $9,000 of value against $3,000 of spend, and without the expected number the team would probably have stopped there.

Because the cost was above what they expected, they asked why. A fifth of the spend, $600, had gone to retries on tickets the agent never resolved, so the team capped retries at two. In month 2, spend was $2,400 for 1,600 resolved tickets, and the cost per ticket matched what the team expected.

In month 3, the help desk connection stopped. The spend is still known but the outcomes are not, so the cost per ticket reads "Not measured yet" until the connection works again. It would be easy to reuse month 2's count, but that gives a number that looks right and may not be.

Run it on one workload

1. 1Pick one workload with real monthly spend.
2. 2Name its outcome, and the system that records it.
3. 3Write down what counts as success, in one sentence.
4. 4Write down the cost per outcome you expect, and what one outcome is worth.
5. 5Gather all of the workload's spend for the month, retries and failures included.
6. 6Count the successful outcomes in the system that records them.
7. 7Divide, and mark the count as observed, declared or not measured yet.
8. 8Compare with what you expected, then keep, change or stop the workload.

Most of these steps are easy once the cost and the outcome sit on the same record. The one that gets skipped is the fourth, writing the expectation down before the spend. With it in place, each month tells you whether AI delivered what you expected, and if it did not, you can see why.