September 18, 2026

Using TypeSafe's Jev for evals

A specialized decision model for routing, classification, and eval verdicts — and how to score production traces with Jev-as-a-judge in Langfuse.

Annabell

Hassieb

TypeSafe launched Jev mid-September 2026. It makes decisions, cannot write a single sentence, and is 20 to 200x faster and 40 to 400x cheaper than frontier models according to TypeSafe.

You send a state, a string or JSON, plus typed questions. You get typed answers with probabilities. It gives you no reasoning back. Useless at other tasks.

Decisions matter a lot for AI agents and AI-powered workflows. AI decides between different categories, whether something is true or not, which tool or path to take. And in the case of evals, AI classifies or simply decides if an output is right or wrong.

Over the past years we forced this out of LLMs with structured outputs and JSON schemas, and still paid generation prices for a yes/no. Jev is deliberately built for that job instead of adapted to it. The flipside: it will not write code, summaries, or tell you why it answered the way it did.

What Jev is good for

Jev can help anywhere in your pipeline or eval harness where decisions are repeated, high volume, and the possible answers are known before the call:

• Agent and tool routing
• Document and ticket classification
• Escalation decisions
• Eval scoring, where a rubric verdict is all you need

Jev covers this surface with three question types:

• Choice picks one option from a set you define, up to 255, and returns the probability of each plus a confidence value.
• Score rates the state against ordered rubric levels and returns a probability-weighted value, the full distribution, and a confidence value. Up to 10 levels.
• Noul answers yes or no and returns the probability it is true. It carries no separate confidence field, so code that reads answer.confidence on everything will break on binaries.

Try a Jev-based sentiment classifier

Run Jev and GPT-5.6 Luna (high reasoning) on the same text. Add up to four classifications — sentiment, urgency, intent, and next action. Jev answers them in one pass; Luna makes a parallel call per classification, so its cost scales with the count. Jev returns all answers together; each Luna call appears as it finishes. Latency and estimated cost are compared below.

The shape of a Jev request

A finished agent run needs three judgments at once, and with Jev they go in one request:

{
  "model": "jev-latest",
  "state": {
    "task": "{{task}}",
    "tool_calls": "{{tool_calls}}",
    "final_output": "{{final_output}}"
  },
  "questions": {
    "needs_review": {
      "type": "noul",
      "instructions": "Does this run need a human to look at it?"
    },
    "severity": {
      "type": "score",
      "instructions": "How badly did this run go?",
      "criteria": [
        "Completed the task cleanly",
        "Completed it, but took a wasteful or confusing path",
        "Delivered a wrong or incomplete result",
        "Took a destructive or unsafe action"
      ]
    },
    "failure_mode": {
      "type": "choice",
      "instructions": "What went wrong, if anything?",
      "criteria": {
        "tool_error": "A tool returned an error or unusable output",
        "missing_context": "The agent lacked information it needed",
        "wrong_approach": "The agent chose an unsuitable strategy",
        "user_abandoned": "The user left before the task finished",
        "none": "Nothing went wrong"
      }
    }
  }
}

And what you get back:

{
  "model": "jev-1.13.0",
  "answers": {
    "needs_review": { "type": "noul", "noul": 0.88 },
    "severity": {
      "type": "score",
      "score": 1.89,
      "confidence": 0.44,
      "legend": {
        "0": "Completed the task cleanly",
        "1": "Completed it, but took a wasteful or confusing path",
        "2": "Delivered a wrong or incomplete result",
        "3": "Took a destructive or unsafe action"
      },
      "probabilities": { "0": 0.02, "1": 0.21, "2": 0.63, "3": 0.14 }
    },
    "failure_mode": {
      "type": "choice",
      "choice": "missing_context",
      "probabilities": {
        "tool_error": 0.11,
        "missing_context": 0.58,
        "wrong_approach": 0.24,
        "user_abandoned": 0.05,
        "none": 0.02
      },
      "confidence": 0.51
    }
  },
  "usage": { "input_tokens": 1840, "output_tokens": 27 }
}

needs_review comes back at 0.88, so queue it. severity lands at 1.89, just short of "delivered a wrong result". failure_mode is inconclusive: missing_context wins at 0.58, but confidence is 0.51 because wrong_approach is close behind at 0.24, and those two are genuinely hard to separate from a trace alone.

Every question is evaluated in parallel and in isolation against the same state. So adding a fourth question, or a fourteenth, barely changes response time, costs only the tokens of the question itself, and cannot degrade the answers to the others. You can ask speculatively and throw away what you do not need.

Jev's strengths for evals

The TypeSafe docs are insistent that each question must be atomic. This is very much in line with guidance on how to write good evaluators.

Jev gives you a cheap and scalable way to evaluate your application against criteria you define. Finding those criteria is still on you, but Jev forces you to think in distinct categories and yes/no decisions to define what good means. If you do not, the answer comes back with low confidence.

Let's take user disagreement as an example. The goal is to identify from a message whether the user of a chatbot is disagreeing with the reply.

With a typical LLM-as-a-judge approach you would write a prompt and force a structured output with a decision, true or false:

You are evaluating a conversation between a user and an AI assistant.
Read the conversation history and the last user message. Decide whether the
user is disagreeing with the assistant's prior response.

The user IS disagreeing if they reject, correct or challenge the assistant's
answer, say it misunderstood them, or ask it to start over. The user is NOT
disagreeing if they ask a neutral follow-up, politely clarify, debug
collaboratively, report an unrelated product problem, or express general
frustration not aimed at the assistant. If there is no prior assistant
response, the answer is false.

Judge what the user believes, not whether the assistant was actually wrong.
Return only JSON: {"disagreement": true | false}

Conversation history: {{conversation_history}}
Last user message: {{last_user_message}}

With Jev you define user disagreement as one question with a scope, and then both criteria, true and false, cleanly:

{
  "model": "jev-latest",
  "state": {
    "conversation_history": "{{conversation_history}}",
    "last_user_message": "{{last_user_message}}"
  },
  "questions": {
    "user_disagreement": {
      "type": "noul",
      "instructions": {
        "question": "Does `last_user_message` clearly communicate that the user believes the assistant's prior response, reasoning, assumption, work, or approach was mistaken or proceeding in the wrong direction?",
        "inspect": "last_user_message",
        "scope": [
          "Evaluate the user's expressed perception, not whether the assistant was objectively wrong.",
          "Use `conversation_history` only to identify the relevant prior assistant response and resolve references.",
          "If there is no prior assistant response, the answer is false."
        ]
      },
      "criteria": {
        "true": {
          "definition": "The user clearly rejects, corrects, challenges, asks to undo, or repeatedly redirects the assistant's prior response or approach.",
          "includes": [
            "Directly saying the answer, assumption, or interpretation is wrong",
            "Saying the assistant misunderstood the request",
            "Questioning why the assistant made a particular assumption",
            "Requesting that the assistant revert, restart, or abandon its approach",
            "Repeated steering that indicates the assistant is still following the wrong direction"
          ]
        },
        "false": {
          "definition": "The user does not clearly indicate that the assistant made a mistake or took the wrong approach.",
          "includes": [
            "A neutral follow-up or request for more detail",
            "A polite clarification that does not reject the prior response",
            "Collaborative debugging without criticism of the assistant's approach",
            "Reporting an external product or system problem",
            "General frustration not directed at the assistant",
            "An ambiguous reaction",
            "No prior assistant response"
          ]
        }
      }
    }
  }
}

0.93 means a 93% probability of disagreement. The cutoff is yours to define in your app context, and because you get a probability rather than a label you can run three paths instead of two: act on high confidence, send the middle band to a human, drop or flag the rest. See the Noul documentation for the full shape.

Both versions give you a true/false indication back. Jev forces you to define distinct criteria while writing your evaluator and has the potential to save you a significant amount of money and time.

Jev's early bench results

In just a few days in developers' hands, people have already benched Jev against the established models.

Early results point towards potential for time and money savings, especially over frontier models. Good Start Labs graded 6,003 rubric checks with Jev and five LLMs on identical instructions. Jev matched Claude Fable 5.1's verdict 91.5% of the time at $160 per million graded answers, against $33,000 of Fable 5.1, $400 for GPT-5.6 Luna and $1,600 for Gemini 3.8 Flash.

The open source comparison is less distinct: DeepSeek V4.1 Flash cost $260 and agreed with Fable 93.5% of the time, two points better for $100 more.

Overall, this primarily points out one thing: there are tasks and decisions, like routing, classification and eval verdicts, that do not need the frontier. Models like Jev, the rise of open source models, and the increasing demand for specialized models, are means to that end.

Where Jev is falling short

TypeSafe publishes a per-version page listing what Jev is bad at. Read it before you design an evaluator. Most of it is general: it reads literally, it cannot do arithmetic, it treats dates as text. These are the ones that bite specifically when you are using Jev to judge.

It cannot abstain. A forced binary with no unknown or needs_review option makes Jev pick the least wrong answer instead of saying it does not know. You have to think about and design an escape hatch, at least early on before calibrating your judge.

No rationale, when you actually need one. Good for eval design, bad for the individual case. Jev is not trained to generate text, so when a trace scores badly it never provides a reasoning. You debug by reading your own criteria. Anything audited or customer facing still needs a generative model on top.

Context rot, and the limit is unclear. Their docs say it plainly: "Jev suffers from context rot". Accuracy drops as the state fills with material the question does not need, which is awkward when agent traces are long and mostly irrelevant. This refocuses importance on designing the input context, much like in the old days. The models page says 64k per request and 32k for state plus the longest question, while OpenRouter lists 32K. Verify before you design around it.

Setting up Jev for production scoring in Langfuse

Jev is available as a judge directly in Langfuse. Add a TypeSafe connection, define a state and typed questions in the evaluator UI, and each question writes a score to matching observations — at up to 40 to 400x lower cost than LLM-as-a-judge on production traffic. Access runs through a waitlist, and Jev is also live on OpenRouter and the Vercel AI Gateway.