Quality

Jepela tells you how often it is right on your own decisions, and tunes its confidence to them. Everything here works from labels: the right answer for a question, given after the fact.

Labels

Every decision has a decision_id. When you learn the right answer, send it:

POST /v1/feedback
{"decision_id": "dec_4b637aef332bc18f111236e0", "question": "route", "correct": "billing"}

correct is an option name for a choice, a level number counted from 0 (the lowest level) for a score, and true or false (or "yes" or "no") for a noul. The answer says whether Jepela was right, judged on the answer you were given (after your calibration, as the quality report judges it), and how many labels the question has on that decision's model (a label sent again for the same decision replaces the first and is counted once). To match labels against, Jepela keeps each decision's question ids, a hash of each question and the probabilities for 30 days, never the state or the question text. Labels and a question's own temperature belong to its id and wording: change its type, instructions or options and it starts again from zero labels. POST /v1/decisions/delete removes every record and label of your tenant, every batch with its results, and the answers the cache keeps for you (see Request options).

The quality report

GET /v1/quality?question=route&days=30

Per question, model and tier (1 for the engine's answers, 2 for those the second tier gave at quality balanced or best, which are right at other probabilities and get figures of their own): labels, accuracy, calibration_error (how far the stated probabilities are from how often they come true), brier, log_loss, reliability bands, and automate_at. Every labelled answer is scored as Jepela would give it now: the engine's own answer through the calibration in use today (GET /v1/calibration), for live labels and golden runs alike, so automate_at is on the scale of the answer_confidence you receive. Earlier versions scored golden runs without your calibration and live labels as they were given at the time, before or after a refit.

automate_at gives, for the target accuracies 0.9, 0.95 and 0.99, a probability threshold chosen on your labels, and how a threshold chosen that way did on labels it was not chosen on.

threshold is the lowest probability at which the answers at or above it, on your labels, were right often enough that the lower end of the 95% interval (Wilson) of their accuracy reaches the target. The margin keeps a short run of lucky right answers from setting the threshold, and it asks for more labels as the target rises: by that rule, even with every answer right, 0.9 needs at least 35 answers at or above the threshold, 0.95 needs 73 and 0.99 needs 381.

To check it, your labels are split into five parts, and each part is judged by a threshold chosen the same way on the other four. decisions and share are the labelled decisions at or above their part's threshold, accuracy is how often they were right, and accuracy_low is the lower end of its 95% interval; labels is how many labels were used. They estimate what a threshold chosen this way gives on decisions it has not seen: about share of them automated, right about accuracy of the time; the rest goes to a person. in_sample has the share, decisions and accuracy at or above threshold on the same labels it was chosen on; it is optimistic. status says what the check found:

reached: on labels it was not chosen on, the answers at or above a threshold chosen this way were right at least as often as the target. Automate above threshold. When accuracy_low also reaches the target, those labels show it with 95% confidence; with few decisions above the threshold it is well below accuracy.
below target: on labels it was not chosen on, they were right less often than the target. Do not automate at this target on these labels.
not checked: a threshold reaches the target on all your labels, but in none of the five parts was a decision at or above a threshold chosen on the other four (four fifths of your labels gave no threshold, or nothing in the fifth part was at or above it), so nothing was checked (decisions and share 0, accuracy null). Do not automate on it yet.
not reached: no threshold reaches the target with the margin on your labels (threshold null, share 0).
not enough labels: fewer than 50 labels, with null figures and needed.

Earlier versions of the report chose the threshold where the accuracy on your labels only just reached the target, with no margin, and gave share and decisions measured on those same labels, also with fewer than 50 labels.

Calibration per tenant

When a question has 50 labels on one model (and again at 100, 200, 400, 800, 1,600 and 3,200) Jepela fits a temperature for it on your labels and keeps it only if it lowers the calibration error and does not raise the log loss on labels it was not fitted on. The same fit sets a temperature for the question's group (same model, type and number of options), fitted on all the group's labels, never on one question's alone, which other questions of that group use until they have one of their own. POST /v1/calibration/fit does the same on demand (50 labels by default; min_labels may lower that to 20), per question and per group. For a yes/no question the fit also sets a shift of its log-odds, which corrects a question the engine leans towards yes (or towards no) on; choices and scores get a temperature only. Measured on public benchmark yes/no questions (2026-09-29), fitted on 593 of them: the other 461 went from 59.2% to 70.1% right, and their log loss from 0.697 to 0.540. A kept temperature or shift changes only the probabilities you receive, for that model only; each answer carries calibration (with its shift) and the raw_ values before it. GET /v1/calibration lists what is in use. A model here is the checkpoint that answered: jepela answers are fitted with, and use, the temperatures of jepela-english or jepela-multilingual, whichever answered (engine.checkpoint), the same ones a request naming that checkpoint uses, because the two checkpoints err differently. Temperatures fitted for jepela before 2026-10-16 pooled both; they are kept for jepela-english (where jepela-english had its own for the same question or group, the one fitted on more labels is kept).

Measured on a constructed 60-case support set (2026-09-23, through the gateway): the English checkpoint was 83% right with a calibration error of 0.17 on held-out folds, lowered to 0.13; typed-decisions 85% right, 0.34 lowered to 0.10. On that set the engine was under-confident. Your data will differ, which is the point of fitting on it.

Golden sets and regressions

A golden set is a list of cases with known answers that you keep and re-run: before changing a model, a question's wording, or a memory.

POST /v1/golden
{"cases": [{"id": "refund-1", "state": "We were charged twice.", "questions": {...}, "expected": {"route": "billing", "refund": true}}]}

POST /v1/golden/run   {"model": "jepela-english"}

A run answers every case with the run's model (default jepela) and reports accuracy per question and changed_since_last_run: every answer that differs from the previous run of the same model, so a change in behaviour shows up as a list, not a surprise. Cases may name a subject (or customer, its older name, kept as the subject), values, derive, memory, robust, windows and method, so memories, rules and how a choice is answered are tested too (earlier versions accepted a case's method and then ran it without it: send such a case again for its method to count); any other field is not kept, and the answer's warnings name it per case (unknown_field, with case), and context and pictures are refused as in a decision; a case with an existing id replaces it. A case that fails, or is not run because the credit ran out, is left out of the accuracy, is not billed, and is listed in failed with the reason. A case stored before Jepela began refusing something it holds (such as a question's depends_on) is not run: failed says what to change, and sending the case again corrected replaces it. Unlike decisions, a golden case keeps its state until you delete it (POST /v1/golden/delete with ids, or with no ids for every case; an empty list deletes nothing). Runs through the API are billed like decisions, in one request: give a large set a longer client timeout; Jepela can also run your set for you without billing, and load a large set from a file; ask. Up to 5,000 cases per tenant; a case sent again with an id you have replaces it and is not counted twice.

The loop

Decide, and keep the decision_id with your record.
Send feedback when the outcome is known.
Read quality: automate above the automate_at threshold for the accuracy your process needs when its status is reached.
Keep a golden set, and run it before every change.