Case study: check a Markdown draft before publication

A small content team publishes practical tutorials every week. Its recurring problem is not a shortage of advice about better writing. It is remembering the same delivery requirements under deadline: name the reader, give executable steps, show an example, and place sources beside relevant claims.

We will build a Markdown checker that applies those requirements consistently. It asks one question per checklist item and lets ordinary Python decide whether the draft enters an editorial queue or returns for revision. The readable explanation is a local template, so every sentence in the report can be traced to a field rather than an invented model rationale.

Completion means that each criterion has a visible result, missing material is distinguished from uncertainty, and an editor can return to the draft with a specific task. publish_queue is a recommendation to continue the editorial process. The example does not publish anything.

Define the scope before choosing a score

“Is this a good article?” mixes correctness, clarity, originality, usefulness, and entertainment. Even two experienced editors may apply different standards. Our first version checks four observable requirements instead.

The source check concerns visible presence and placement. It does not open the destination, establish authority, or verify that the claim is true. An article can satisfy this checklist while containing an incorrect statement. Verifying a link's availability and reading the source to compare its evidence with the claim are separate jobs.

Each requirement becomes a Noul question. That primitive returns a value for a yes/no proposition; it does not have the separate confidence field used by Choice and Score. The program reads answer['noul'], not an invented confidence attribute. Official Noul documentation

This scope also excludes AI-writing detection and SEO prediction. There is no hidden “quality” or “human-written” label in the report. A narrow result is more useful when editors know exactly what it covers.

Download and inspect the sample drafts

Get the examples bundle, then work from the directory containing examples. No third-party Python dependency is required. The Python chapter explains environment setup.

The sample IDs are complete, missing, and ambiguous. Their names describe the experiment's design, not a model's finding. All three were written for this book; they are not customer documents or a sample of professional editorial performance.

Open the drafts before running the checker. The complete draft teaches a reader to save a small ticket object and inspect it with python -m json.tool. The missing draft contains general advice and an unsupported numerical claim. The ambiguous draft provides steps and an example but links a specific command claim to a broad homepage.

Mark the four requirements yourself first. This avoids quietly changing your interpretation after seeing a model-shaped number. If you disagree with the supplied reference, write down why. A file named complete.md should not force an editor to accept it.

Understand the input and report contracts

The article and the checklist play different roles. Article text is material to evaluate; checklist text is policy. state_for puts the article under markdown. inputs_for builds the questions from checklist.json. Reference decisions remain outside the state sent for evaluation.

Markdown + checklist version
  → one Noul question per criterion
  → response validation
  → pass / missing / uncertain
  → publish_queue / revise
  → JSON report + local text template

The report retains the supplied answer in raw_response. Its decision contains checks, needs_work, and recommendation. If the model-shaped answer is an intermediate value and the program recommends revision, that is policy being applied, not an inconsistency in the record.

Every required item must pass. A beautifully identified audience cannot compensate for absent instructions. The official composite pattern separates dimensions before combining them in code; our editorial design uses required-item gates instead of a weighted score that allows one dimension to offset another. Official composite scoring pattern

This is an application choice. A different team could keep optional reminders alongside mandatory criteria, but it should state that distinction explicitly rather than burying it in a total score.

Run and read the first report

From the project root:

python --version
python -m examples.hello_jev content --mode offline
python -m json.tool examples/output/content.json

If your system uses python3, substitute it consistently. Open examples/output/content.txt in a text editor. It starts with evidence and checklist information, followed by each draft's checks and recommendation.

The following inspection code extracts the main decisions while retaining errors:

import json
from pathlib import Path

report = json.loads(
    Path("examples/output/content.json").read_text(encoding="utf-8")
)
for row in report["records"]:
    decision = row.get("decision")
    if decision:
        print(row["id"], decision["recommendation"])
        print("Needs work:", decision["needs_work"])
    else:
        print(row["id"], row.get("error"))

For a preserved baseline, provide another JSON path:

python -m examples.hello_jev content --mode offline \
  --output examples/output/content-baseline.json

The text report uses the same stem, producing content-baseline.txt. The local template displays check names, values, statuses, and recommendations. It does not add quotations from the article that the model never returned, and it does not generate replacement prose.

In the executed replay, the complete draft passed every item. The missing draft received authored values of 0.05 for steps and 0.01 for both example and sources; all three were marked missing. The ambiguous draft's sources value was 0.50 and marked uncertain. These fixture values demonstrate two reasons for the same revision recommendation: add absent material, or inspect whether a broad homepage belongs beside a specific command claim.

Offline evidence is labeled offline_fixture. The authors supplied these responses to make application behavior reproducible. A pass for the complete draft is not a recorded live Jev result. The client fingerprints the state and questions; editing either makes the fixture mismatch and produces an error/review record rather than a fresh judgment.

Inspect the decision function

Here is the policy excerpt from the bundled implementation. Use the bundle for loading, transport, validation, and writing rather than assembling a second application from fragments.

def content_decision(answers):
    checks = {}
    for key, answer in answers.items():
        value = answer['noul']
        status = 'pass' if value >= 0.85 else (
            'missing' if value <= 0.15 else 'uncertain'
        )
        checks[key] = {'value': value, 'status': status}
    needs_work = [
        key for key, check in checks.items()
        if check['status'] != 'pass'
    ]
    return {
        'recommendation': 'revise' if needs_work else 'publish_queue',
        'checks': checks,
        'needs_work': needs_work,
    }

A value of at least 0.85 passes; a value no greater than 0.15 is marked missing; values between them are uncertain. These thresholds demonstrate a policy and have not been calibrated for professional publishing.

Missing and uncertain both lead to revision, but they imply different editorial work. Missing suggests adding material. Uncertain suggests inspecting the passage and the criterion before deciding what to add. Otherwise, a writer may inflate a perfectly adequate paragraph just to satisfy an unclear rule.

Boundary behavior matters: exactly 0.85 passes and exactly 0.15 is missing. Tests should cover those boundaries rather than only obvious values such as zero and one. If an item later becomes advisory, preserve its result and change the combination policy; do not remove evidence simply to make more drafts pass.

Design a useful failure experiment

Start with the complete draft and remove one element at a time: audience description, concrete actions, worked example, or source information. Leave other content intact and record the expected changed result before evaluation. This lets you investigate whether a criterion measures what it claims to measure.

These are proposed boundary experiments, not live failures reported by this book. A real experiment needs the draft, checklist version, raw response, and an independent editorial judgment. Have a second editor label disputed examples before treating the first person's preference as objective truth.

A concrete correction cycle would be: discover that a heading is being counted as instructions; tighten the steps criterion; add a held-out heading-only draft; run a live evaluation; then recheck ordinary drafts for new false alarms. Looking only at the example that motivated the change gives you little evidence of improvement.

Never manufacture a better model result by editing a fixture. Fixture changes are appropriate for testing the policy or renderer, and should be described that way. Semantic improvements require actual calls or clearly remain untested proposals.

Version the standard as well as the draft

Changing the checklist changes what passing means. “Contains one source link” and “Every external quantitative claim has a supporting source” define different tasks. Their pass rates are not directly comparable.

The bundled checklist has version editorial-checklist-v1; the report carries it in policy_version. Write change notes such as “worked examples now require expected output,” not merely “improved prompt.” Keep a set of drafts that never participates in tuning and compare per-item results after each change.

A stricter standard can reduce the pass rate while working exactly as intended. Conversely, higher confidence or fewer revision requests may conceal a weakened definition. Read the changed criterion and the affected examples together.

Draft versions matter too. In a production integration, record a content hash or stable document revision so a reviewer knows which text the report concerns. The sample manifest links local files to IDs; it does not implement version tracking for a collaborative editor. Connecting such an editor would require that extra binding.

Run live and collect appropriate evidence

After configuring TypeSafe access and TYPESAFE_API_KEY securely in your environment:

python -m examples.hello_jev content --mode live \
  --output examples/output/content-live.json
python -m unittest discover -s tests -p 'test_examples.py' -v

The default requested model is jev-latest, and the client sends requests to https://api.typesafe.ai/v1/systemone. This chapter was prepared on 2026-09-20 without a live API key. It provides no live false-negative rate, false-positive rate, model latency, or bill. Keep offline measurements empty rather than inventing numbers.

Local checks used Python 3.12.14 and passed all 20 shared example tests. Replay placed complete in publish_queue, returned missing and ambiguous for revision, and generated the text report. The fixture model marker is authored-fixture-not-a-model; token, duration, and cost measurements remain null. Agreement with the authored references is not model accuracy.

During a live trial, retain returned model identifiers, usage, HTTP duration, failures, and the checklist version. Cost calculations belong with a dated pricing source; see estimating costs. Do not count failed authentication as an editorial misclassification.

A false negative here means an editor confirms a requirement is missing but the checker passes it. A false positive means the material satisfies the criterion but the checker sends it back. Count these per item. A single document-level error count cannot tell you whether the steps rule is too loose or the sources rule is too strict.

Small evaluations are most useful when they produce explainable failure records. They do not justify a universal article-quality percentage. The evaluation chapter explains how to keep tuning examples separate from final validation.

Put the checker into an editorial workflow

Initially, attach the report beside the draft and ask editors to accept or override each result with a short reason. Once the team agrees on the criteria, connect the recommendation to a task board. The current example has no external publishing action.

For release notes, change the checklist to affected users, changed behavior, and migration steps. For help articles, consider prerequisites, navigation, and recovery steps. The content owner should define those requirements; a generic “quality” instruction cannot substitute for that decision.

If you add a generative model to fill missing sections, recheck the resulting draft and retain human fact review. Detecting an omission and writing a correct replacement have different completion criteria. The next case examines another boundary: a model may choose a candidate tool, while the application validates parameters and permissions.