Case study: triage a small SaaS support queue

A small SaaS operator opens an inbox containing billing disputes, login failures, invoice requests, and product questions. Before solving anything, someone must decide where each message belongs. A classifier seems useful until a customer writes: “I was charged twice, and now I cannot log in. Please help with both problems.” One label can easily hide half the request.

We will build a batch triage workflow that preserves that conflict. It reads local tickets, asks separate questions, applies a review policy, and writes an inspectable report. Its job ends with a processing recommendation. No refund is issued, no customer receives a message, and no external help desk is updated.

The completion criteria are concrete: every input has a recorded outcome; ambiguous, overlapping, and invalid inputs remain visible; and an operator can trace a recommendation back to the original response. These are useful engineering guarantees even before you know how well a model classifies your real tickets.

Get the runnable materials

Download the examples bundle and open the directory containing examples. The program uses Python's standard library. See the Python integration chapter if you need help preparing the environment.

The six records are deliberately different. T001 contains billing and login problems. T002 reports a login failure, T003 requests an invoice, and T004 asks about a feature. T005 is vague. T006 has blank text. Keep the difficult records: they test whether the workflow stops when the necessary information is absent.

These are synthetic materials written for this book, not customer records or a representative benchmark. Their references express the operating policy we chose. Another business could reasonably prioritize restoring access before investigating a payment dispute. Establish that policy before treating disagreement with a label as a model error.

The essential input shape is:

{
  "id": "T001",
  "text": "I was charged twice, and now I cannot log in. Please help with both problems.",
  "reference": {"department": "billing", "review": true}
}

Reference information belongs to evaluation. It must not become evidence supplied to the model: a classifier that sees the answer key is no longer being evaluated on the ticket.

Turn the task into separate decisions

“Handle this well” is not an executable requirement. We need a department, an assessment of operational impact, and a way to retain overlapping requests.

Billing includes charges, refunds, and invoices. Technical means an existing capability is failing. Product covers usage and capability questions. other is a real exit for inputs outside those descriptions. A request for a new login method belongs on a different path from a report that today's login is broken, even though both mention login.

Choice and Score have different response structures: one selects from named options, while the other locates an answer on ordered descriptive levels. We use both here rather than merging department and urgency into one label. Official Choice documentation, official Score documentation

Impact should describe consequences, not emotional intensity. “Please help immediately” does not establish that all customers are locked out. Nor does an angry message establish the absence of a workaround. The application must distinguish what a customer reports from what the service has independently verified.

ticket file → validate input → ask independent questions
            → validate response → apply policy → save report

The request contains the questions for one ticket together. TypeSafe supports this organization; our batch still processes tickets as separate work items. Parallel evaluation of questions should not be confused with submitting every ticket concurrently. Official speculative fan-out pattern

Run the offline path first

From the project root:

python --version
python -m examples.hello_jev tickets --mode offline
python -m json.tool examples/output/tickets.json

Use python3 throughout if that is the command your installation provides. Offline mode needs no credentials and makes no network request. It loads a response fixture associated with each sample ID, then runs the actual validation and policy code.

The report identifies this evidence as offline_fixture. This means “the authors supplied this response to exercise the application,” not “a local model classified the text.” A successful replay says nothing about Jev's accuracy, response time, or price.

Inspect the top-level mode, evidence, and policy_version. Then inspect records. Every ticket should have a visible outcome, including the blank one. Within a valid record, compare raw_response with decision: the first preserves the supplied answer; the second shows what your application did with it. An invalid record should explain its error instead of quietly disappearing.

This small inspection program uses the report's public structure:

import json
from pathlib import Path

report = json.loads(
    Path("examples/output/tickets.json").read_text(encoding="utf-8")
)
print(report["mode"], report["evidence"])
for row in report["records"]:
    print(row["id"], row["status"], row.get("decision"))

To preserve a baseline, choose a separate output file:

python -m examples.hello_jev tickets --mode offline \
  --output examples/output/tickets-baseline.json

Keep the input, question definitions, policy version, and resulting report together. Repeatedly overwriting one report makes later comparisons depend on memory.

Follow three records through the workflow

Start with T001. The customer has two problems, but the operator still needs a primary destination. The fixture supplies the model-shaped answers; the local policy retains a review flag so that assigning billing does not imply that login has been resolved.

The locally executed replay produced billing, impact_score: 1.8, priority: high, and review: true for T001. Its reasons include uncertain department, uncertain impact, and multiple issues. These are authored fixture values, not live predictions. Keeping 1.8 also checks that the application preserves fractional Scores. High priority and a need for review are independent outcomes.

Inspect department, impact_score, priority, review, and reasons inside the decision. Do not infer the full list of customer needs solely from the department distribution. A split distribution might reflect overlap, unclear category descriptions, or insufficient information. The separate multi_issue question gives the policy another signal, and the original text remains available to the reviewer.

Next inspect T003. An invoice enters billing in this example. That does not imply that invoice handling and refunds use the same procedure. We keep the category list short because this sample operator has one billing queue. If your organization has separate invoicing staff, introduce a subcategory after the first workflow is useful.

Finally inspect T006. Blank text is an input error, not a meaningful classification of other. The distinction matters: other means a valid request falls outside the taxonomy; blank means the program did not receive the required material. Asking a model to guess would conceal an upstream data problem.

Read the policy separately from the API code

The complete implementation is in the bundle. Its decision function can also be called directly with already validated answers:

from examples.hello_jev.cases import ticket_decision

# raw_response has passed validation in client.py.
decision = ticket_decision(raw_response["answers"])

The teaching policy requests review when department or impact confidence is below 0.80, the selected department's probability is below 0.80, the department is other, or multi_issue.noul is at least 0.50. An impact score of at least 1.50 marks high priority. The function returns its reasons alongside the recommendation.

Those numbers are application settings chosen to demonstrate branches. They are not calibrated service guarantees. Notice also that multi_issue is a Noul answer: its value concerns a yes/no proposition and has no separate confidence field. The Noul documentation describes that distinction.

Review is an ordinary outcome, with work attached to it. Lowering a threshold can reduce the review queue while increasing wrong automatic assignments. Raising it can hide model weaknesses behind growing manual effort. Evaluate both error and workload, and change thresholds without entangling them with HTTP transport.

Confidence itself is derived from the response distribution. It is not measured correctness on your tickets, so averaging confidence does not produce classification accuracy. Official confidence documentation

Use failures to improve the right layer

For each failure, first read the original text and write the desired outcome with a reason. If two operators disagree, resolve the business policy before editing prompts. If operators agree but the model confuses neighboring options, rewrite those descriptions and test new examples of both categories.

Do not “improve” the model by replacing an offline fixture with the answer you wanted. Fixtures are authored inputs to your application tests. The client fingerprints the state and questions; editing either produces a mismatch error and review record instead of a new inference. To test whether a description improves semantic judgment, run an actual live experiment with held-out text.

A separate failure layer is malformed output. Unknown choices, missing answers, or invalid probability values must be rejected by the client before policy evaluation. A successful HTTP response is not sufficient evidence that its body is usable. Keeping transport validation separate also makes these failures testable without a remote service.

Move to live calls deliberately

Live mode requires usable TypeSafe access and TYPESAFE_API_KEY in the environment. After setting the key securely on your machine, run:

python -m examples.hello_jev tickets --mode live \
  --output examples/output/tickets-live.json

The client posts to https://api.typesafe.ai/v1/systemone, with jev-latest as the default requested model. Record both the requested alias and any model identifier returned by the service. An unchanged alias does not establish that every experiment used the same underlying release. Request details are covered in Python integration.

Begin with the synthetic data. For authorized historical tickets, remove information unnecessary for triage before sending requests. The report itself should be handled like support data rather than placed in a public download. Authentication and network failures belong to troubleshooting; they should not be counted as incorrect department predictions.

Record evidence without overstating it

This chapter was prepared on 2026-09-20. The provided offline path exercises application behavior. No live model accuracy, latency, or billing result is claimed here. Null measurement fields remain null; do not replace them with the elapsed time of reading a fixture file.

Local verification used Python 3.12.14 and passed all 20 shared example tests. Replay produced completed for T002–T004 and review for T001, T005, and T006. This deliberately balanced teaching set does not estimate production review workload. Its response model marker is authored-fixture-not-a-model.

Run the automated checks from the project root:

python -m unittest discover -s tests -p 'test_examples.py' -v

Report wrong assignments among automatically handled tickets and the fraction of all tickets sent to review. Do not count every reviewed ticket as a correct automatic classification. Likewise, do not publish accuracy on a tiny easy subset without the coverage figure that explains how much work was excluded.

Six examples can expose broken branches; they cannot estimate production performance. When you have enough authorized historical data, reserve a set that is never used to tune questions or thresholds. Include rare categories, missing context, and overlapping requests. The evaluation chapter develops this process.

Adapt the workflow to your service

Replace the input mapping and category descriptions first, then the local review policy, and only then connect a help desk. A useful initial integration merely places the recommendation beside a human decision. Capture suggested department, final department, and the reason for any change.

These records tell you whether to fix a taxonomy, a question, or a policy threshold. They also establish a practical limit: queue organization is the completed product here. Executing refunds would require independent transaction, identity, amount, and approval checks that this tutorial has not implemented.

Continue with the content checklist case to apply the same separation between judgments and local policy to editorial work, or return to the handbook index.