Write useful Choice, Score, and Noul questions

When an answer disappoints you, first check whether another person could answer the question consistently. “Is this article good?” does not define good. “What should we do with this ticket?” combines understanding the request, assigning a team, and authorizing an action. Returning a number does not resolve those disagreements.

Start by describing what your application will do with the answer. If it returns an article for missing instructions, ask whether the draft contains instructions that meet a stated definition. If it selects a support queue, list the destinations and their boundaries first. A checkable standard often helps more than a longer prompt.

Draft a four-part question card

Add a counterexample. “You should prioritize efficiency” expresses advice but does not specify an operation the reader can carry out. That example helps you check whether your standard merely recognizes encouraging language.

A question ID identifies its answer in your code; it does not explain the judgment to the model. Even when you name an ID has_actionable_steps, put the full meaning in instructions. Question definitions

Choice: give every option a distinct job

Choice selects from a known set. Names and descriptions define the candidates, and the answer includes the selected value and probabilities over the options. Choice documentation

Consider a definition that needs improvement:

{
  "type": "choice",
  "instructions": "Which team should handle this?",
  "criteria": {
    "billing": "Billing issues",
    "technical": "Technical issues"
  }
}

Its problem is not brevity. It does not say where to put “I was charged twice and cannot log in.” Nor does it offer a sensible destination for a partnership proposal. Begin the revision by deciding the business policy. This exercise gives mixed requests a separate review destination and retains a fallback for other messages.

{
  "type": "choice",
  "instructions": "Select a queue using only the stated concerns.",
  "criteria": {
    "billing": "Only charges, invoices, or subscriptions.",
    "technical": "Only login failures or broken product features.",
    "mixed": "Both billing and technical concerns are present.",
    "other": "Neither category fits, or the message is unclear."
  }
}

These four values match the previous chapter's minimal program. Inspect other regularly. If partnership requests become common, evaluate whether a dedicated partnership category earns its place. Do not invent dozens of rarely used categories simply to eliminate a fallback rate.

Sometimes a message legitimately needs several labels. If you need to know both whether it concerns billing and whether it describes a login failure, ask two presence questions and combine them in code. A single Choice should not be expected to return two selected values. Choose a design based on whether the application needs one destination or a collection of labels.

Score: write the levels before accepting the number

Score uses ordered descriptions. Levels are numbered from zero, and the returned score weights those positions by their probabilities, so fractional values are possible. The current interface accepts two to ten levels. Score documentation

“Give this ticket a score out of ten” leaves the score undefined. This exercise measures reported workflow impact. Customer tone and account value belong to different questions.

{
  "type": "score",
  "instructions": "Rate the workflow impact stated in the message.",
  "criteria": [
    "No workflow is blocked; the issue is cosmetic or informational.",
    "A workflow is impaired, with a stated usable workaround.",
    "A core workflow is blocked, with explicitly no usable workaround."
  ]
}

Immediately test the definition against missing information: “Export failed.” That report does not establish whether a workaround exists. Check evidence sufficiency separately rather than treating an unmentioned workaround as an explicitly absent one. You can add a question about the available evidence and ignore the score when clarification is needed.

Here is arithmetic only, not a model result. With level probabilities of 0.1, 0.6, and 0.3, the score is 0×0.1 + 1×0.6 + 2×0.3 = 1.2. It is not 1.2 out of ten, and it does not express 120 percent certainty.

Different distributions can produce the same mean. All probability on level 1 and an even split between levels 0 and 2 both average to 1. Their practical implications differ. Retain the distribution and confidence when deciding on review; rounding a decimal into a neat label can conceal the distinction.

Noul: stay within visible evidence

Noul evaluates a yes/no proposition and returns the probability of yes. It does not measure the degree of a quality; use Score for a defined scale. Noul documentation

A page stating a refund deadline and a merchant honoring its policy are different propositions. Establish that boundary in the question itself. For content review, the following definition checks visible structure in a draft:

{
  "type": "noul",
  "instructions": "Does `draft.body` contain an actionable step?",
  "criteria": {
    "true": "A reader can perform a named action on a named object.",
    "false": "Only goals, slogans, or general recommendations appear."
  }
}

“Open Settings and turn off email notifications” meets this definition. “Improve the way you work” does not. Your content team may want a different standard, but the disagreement now concerns identifiable text that reviewers can examine.

Share the material; compose policy in code

Break “Can we publish?” into judgments about the intended reader, actionable steps, an example, and required sources. Each question should be answerable from the draft without guessing another answer. Questions in one request independently evaluate the shared state. A genuine dependency belongs in a subsequent step arranged by code. Multiple questions and dependencies

An original content policy for this book can be expressed as:

Empty or malformed input → stop and record the input problem
Missing required answer → review the response problem
Any required item fails → send the draft for supplementation
All required items pass → suggest editorial review

Editorial review is still a separate step from publication. A local template can list missing checklist items, but a Noul value cannot supply an explanation or quotation absent from the response. If you need sentence-level evidence, design a separate, verifiable extraction and display process. Do not assume the three judgment types return open-ended evidence text.

Revise with counterexamples and keep a record

Prepare six inputs: a clear positive, a clear negative, negation, quoted speech, mixed content, and missing information. Label them yourself before a live call. Without a key, complete the labeling and branch checks while leaving model results unfilled.

For each error, retain the input ID, question version, human expectation, actual answer, and suspected cause. Revise one boundary at a time. If one ticket persuades you to add “billing always takes priority,” revisit the technical-failure examples as well. A local exception may improve one item while changing the meaning of the whole policy.

Finally, keep several inputs out of the revision process and evaluate them afterward. If every example helped you rewrite the question, you have shown adaptation to those examples, not performance on new messages. Offline fixtures let you rehearse response handling and policy branches. Model behavior requires separate live evaluation.

Continue to Support-ticket triage to apply these questions in a complete workflow, or read Confidence and evaluation to plan the evidence you will keep.