When not to use an LLM for a decision

Published 2026-10-15 by Jepela

If your code needs one judgment from a known set of answers, a model that writes text is often more than you need.

The pattern

Software makes many small decisions: route this ticket, flag this claim, send this mail, pick this tool. Teams often hand them to a large language model. That works, and it has three costs.

  1. The model writes its answer word by word, even when the answer is one word from a known list.
  2. The answer is text, so the code has to parse it, check it, and handle the case where it does not parse.
  3. Every call starts from zero, so what you know about the customer is sent again, and paid for again, with every request.

None of this matters when you need text. It is extra work when you need a decision.

Signs you need a decision, not text

  • The answer is one of a fixed set of options, a level on a scale, or yes or no.
  • Your code branches on the answer, and no person reads it.
  • You want a probability you can set a threshold on.
  • The same judgment runs many times a day.
  • What you know about the customer should change the answer, and you want to see how.

When most of these hold, a decision engine fits better than an LLM. Jepela takes a state and typed questions (choice, score, or noul, the probability of yes) and returns typed answers with probabilities in one call. It writes no text, so there is nothing to parse and no output to pay for.

What changes in practice

Each figure below says where it was measured:

  • Engine time: 27 ms for a three-question decision, timed on the engine's server (median of 25, 2026-09-24). The API's own work and the network come on top.
  • Tokens: a yes/no question about a 74-character support message read 47 input tokens, and three questions read 128 (through the API, 2026-09-24).
  • Cost: at $0.042 per million input tokens, that is $0.000002 and $0.0000054 per decision (arithmetic). Output is free.
  • History: on a constructed test of 60 decisions about 20 customers, sending the whole history with every request read 496 tokens per decision, and naming the customer so memory recalled its lines read 139 (cost cookbook, 2026-09-26).

The same 60 decisions were also used to choose how memory recalls lines, so that last test shows tokens, not accuracy.

Every answer also carries a confidence. Your code can act above a threshold, ask a person in the middle, and hand over below.

When an LLM is still the right tool

A decision engine is not a replacement for everything. Use a language model, or something else, when:

  • The output is text for a person: a reply, a summary, a draft.
  • The task needs several steps of reasoning. The Jepela engine reads the state once per question and answers; it is not a reasoner.
  • The possible answers are not known in advance.
  • The input is an image or audio. Jepela reads text only.
  • The input is long. jepela-english reads about 475 tokens of state (measured 2026-09-22 and 2026-09-23) and cuts the rest, and says so.

Some decisions are neither. "Stop when the loss reaches 30" is a comparison, and code does it exactly. The Jepela engine does not compare numbers: written as a memory line, that stop-loss left the stop answer at 0.22 to 0.30 when the limit was reached, and as a rule checked in code it was 0.75 (rules cookbook, 2026-09-23). Put limits like this in code, whatever model you use.

Measure on your own labels

Whatever you choose, measure it on your own cases. Send Jepela the right answer when you learn it. The quality report then shows accuracy and calibration per question and, for the accuracy you need, a threshold chosen on your labels, whether thresholds chosen that way reached that accuracy on labels they were not chosen on, and the share of decisions above them.

Calibration is fitted on your own labels. On a constructed set of 60 support messages, the English checkpoint was 83% right, and a fitted temperature lowered its calibration error from 0.17 to 0.13 (2026-09-23). Your data will differ, which is why the fit uses it. Keep a golden set of cases with known answers, and run it before every change.