Can You Trust the Result? Confidence and Evaluation

A program that prints JSON has completed an integration step. Before its decisions affect work, you need to know which inputs cause trouble, what a mistake costs, and where uncertain results go. This chapter turns a promising demo into a small, inspectable experiment.

Four different kinds of evidence

The examples in this handbook have an offline mode. It replays responses written by the authors so that readers can inspect branches, reports and tool execution without an account. Those responses were not generated by Jev. They cannot establish its accuracy, latency or price per request.

Check the run mode before interpreting a report. The label offline_fixture means authored teaching data. A real-call record should retain the returned model version, usage and elapsed request time. Combining the two modes in one performance metric turns a software exercise into a misleading model evaluation.

Probability, confidence and accuracy answer different questions

A Choice distribution shows how the model allocates probability among your options. A Score distribution refers to rubric levels; the returned score can fall between two levels. The confidence field summarizes a distribution. It is not the accuracy measured on your own dataset. Noul returns the probability of a proposition and has no equivalent separate confidence field. See the official confidence documentation for the field semantics.

Imagine an illustrative answer that splits probability between billing and technical support. The ticket might contain two requests, or your categories might overlap. Raising a threshold changes the next action, but it does not repair an unclear taxonomy. Decide whether the business needs one primary team or several labels before rewriting the question and policy.

Likewise, a Noul value of 0.05 is a strong lean toward no. It does not mean the model is only five percent certain. For a refund-request question, a low value can be a clear negative. Your application needs affirmative, negative and uncertain regions rather than treating every low number as a reason for review.

Build a small dataset with meaningful variety

For an initial experiment, a few dozen carefully checked examples can reveal useful failure modes. The number is not a certification threshold. Include straightforward inputs, multiple requests, missing information, negation, quoted speech and irrelevant material. A dataset made entirely of obvious keyword matches hides the difficult part of the task.

For example, “I do not want a refund; I need an invoice” tests negation. “Yesterday your colleague suggested a refund, but the issue is now fixed” tests time and attribution. “Can you help?” tests insufficient information. These cases expose different weaknesses, so keep their identifiers and explanations even when you change the wording.

Give every example a stable ID, a reference outcome and a short rationale. Mark disputed examples as disputed until the business rule is clarified. Do not force disagreements into apparently certain labels. For tickets, label the intended team and whether review is appropriate. For editorial checks, identify missing checklist items. For tool routing, also record whether parameters and permissions are sufficient.

Use the development split to revise questions, rubrics and thresholds. Run the holdout split after those choices are made. Once you inspect a holdout failure and tune against it, that example has become development material. Prepare new held-out examples for the next independent check.

Report accuracy and automation coverage together

The following numbers are a constructed arithmetic exercise, not a Jev benchmark. Suppose a queue contains 20 tickets. The program handles 12 automatically and sends eight to review. One of the 12 automatic routes is wrong.

• Automation coverage is 12 / 20, or 60 percent.
• Review rate is 8 / 20, or 40 percent.
• Accuracy among automatic routes is 11 / 12, about 91.7 percent.
• Incorrect automatic routes are 1 / 20, or 5 percent of all inputs.

Report all four quantities. Saying only “91.7 percent accurate” hides the review workload. Saying “95 percent avoided an automatic mistake” risks counting review as a correct prediction. A system that sends everything to a person can avoid automated mistakes while saving no review effort.

For content checks, examine false passes and false flags per item. A draft missing a required example but passing the check is a false pass. A complete draft returned for that same missing item is a false flag. Separate counts reveal which question is unclear instead of burying every failure inside one overall quality score.

A threshold expresses a business tradeoff

You can start an exercise with 0.8, but that is a teaching value rather than a universally validated recommendation. A threshold changes how much work is automated and how much is reviewed. Whether the change reduces mistakes must be measured. Refunding money, saving a draft and sorting local files do not have the same consequences.

Compare a few candidate thresholds on the same development examples. Record automatic decisions, mistaken automatic decisions and reviews for each candidate. Choose a policy consistent with the team's tolerance for mistakes and review work, then check it on examples that did not influence the choice. A confident model answer never replaces authorization, amount limits or required-field validation.

# Illustrative policy, not a validated universal threshold.
def decide(choice, confidence, allowed, threshold=0.8):
    if choice not in allowed:
        return "review"
    if confidence < threshold:
        return "review"
    return choice

This function checks an allowlist and a threshold. It is not a complete tool-execution policy. The agent routing case adds deterministic permission and parameter checks before executing any local tool.

Evaluate each language separately

A bilingual handbook does not establish equal bilingual model performance. The provider currently identifies English as the primary training language and advises testing other languages on your own material. See the model language notes.

If your application receives both Chinese and English, report results separately. Translating a failed Chinese input before trying again changes both language and wording. It does not isolate the cause of the original failure. If translation becomes part of the workflow, include its cost, information loss and latency in the evaluation.

Keep an experiment record someone else can inspect

Save the question-template revision, dataset revision, requested model name, returned model version, threshold, date, language and counts of successes, failures and reviews. Do not record the API key or unnecessary private customer material.

The jev-latest alias is convenient for learning. For reproducibility, record the version returned by the service. Re-run retained examples when the model, rubric or source of incoming data changes. The model reference explains aliases and version IDs.

Your deliverable for this chapter is a short report a colleague can understand: which inputs were tested, which calls were real, where errors occurred and which work still needs review. More decimal places cannot compensate for missing evidence.