Jev Your Engines: Seven Open Decision Models vs. a Generative LLM

Previous article: When All You Have Are Decoders, Every Decision Looks Like Generation

TLDR

• Jev-style models let agents decide without generating answers. For bounded choices such as routing or tool selection, they score the available options and return a typed decision, giving agents a direct alternative to generating and parsing an answer token by token.

• Kev led on accuracy; Verdict also outperformed the LLM baseline. Kev reached 98% on both fixed routes and changing candidates. Verdict reached 89% on changing candidates, compared with 66% for LLM (baseline) Qwen3-0.6B, which reached 83% on fixed routes. JSON generation still beat the three Qwen readout configurations.

• Zero generated answer tokens, with lower observed latency for Kev and Verdict. Both generated no answer tokens, versus about 10 per request for Qwen JSON generation. Recorded median request latency was 125–130 ms for Kev and 17–18 ms for Verdict, versus 331–357 ms for the LLM baseline across the two panels.

Download the starter kit to try a decision model, explore the benchmark, and compare JSON generation with a direct readout.

Why we put decision models to the test

In Part 1, we explored something that has never sat right with us: using a language model to generate an answer token by token when the software only needs one choice. A banking customer says, “I was charged twice for the same transfer.” The next step is to choose a support route. Asking a model to write {"route": "duplicate charge"} gets us there, but it also adds text generation and parsing to a bounded decision.

The idea of scoring the available choices directly caught our attention. We wanted to know whether the open implementations could actually handle the task, and what we would gain or give up by using them. Could they choose the right route? Could they still do that when the available choices changed? Would simply prompting a small language model work just as well?

A router may have a permanent list of departments, while an agent’s available tools or actions can change with the workflow. We wanted to know whether these decision interfaces were useful in both situations. We compared seven open projects and a Qwen JSON-generation baseline on banking requests with both stable route lists and changing candidate menus. Here is what we learned from running these projects: how reliably they choose a route, what changes when the available options change, and what candidate scoring buys us over generating JSON.

Why runtime candidates change the problem

A candidate is one available choice. A candidate-based interface lets the workflow supply the routes or tools available for the current request, and the model scores that menu. The caller can change those choices without adding a permanent output slot for each route. For us, that flexibility is a major reason to explore these projects. A conventional classifier with a permanent output slot for each route needs adaptation when new routes are added.

One decision contract, three question types

The same support ticket can raise more than one bounded question. Conceptually, a decision request could look like this:

{  "state": "I was charged twice for the same transfer.",  "questions": {    "route": {      "type": "choice",      "candidates": ["duplicate charge", "transfer delay", "close account", "cash withdrawal"]    },    "escalate": {"type": "yes/no", "question": "Does this need urgent human attention?"},    "urgency": {      "type": "score",      "levels": ["low", "medium", "high"]    }  }}

Each project has its own wire format and supported question types. The shared idea is straightforward: provide state, define allowed answers, receive a typed result, and let deterministic code decide how to use it. A caller can route, apply a tested threshold, or send the case to a person.

For a first run, follow the Kev local quick start, Verdict project README, Laya quick start, OpenJev setup, AnyJev setup, Simple Jev server instructions, or NanoJev CUDA quick start. Each link goes to the project documentation retained at the revision used for the comparison. Check the hardware and package requirements before installing.

Seven open projects to try

The seven projects offer different routes into bounded decisions. The overview separates their documented setup from measured banking accuracy. We added direct Qwen JSON generation as a reference LLM baseline.

The start links go to official READMEs pinned to the tested revisions. Check their hardware and package requirements when choosing a runtime.

The projects fall into three broad groups. Trained scorers such as Kev, Verdict, Laya, and NanoJev have a model that directly scores choices. Training-free readouts such as AnyJev and Simple Jev derive scores from language-model outputs. Serving layers such as OpenJev expose a shared interface to a selected backend. The groups describe implementation choices; the comparison still belongs to each specific project, checkpoint, and setup.

How we compared them

Comparison of generative LLM routing and single-pass candidate scoring for AI decision tasks. The generative path constructs a prompt, generates answer tokens, and validates JSON. The scoring path evaluates supplied choices and returns scores for route selection or abstention without generating answer tokens. (Image by Author)

We used BANKING77, a dataset of short banking requests labeled by intent, to compare seven open projects. We also added a plain JSON-generation baseline using the same pinned Qwen3-0.6B checkpoint as AnyJev, Simple Jev, and OpenJev.

We selected twelve intents and split the test into two settings. Fixed routes means every request gets the same menu of eight banking routes. Dynamic candidates means each request gets its own shuffled menu of four possible routes: the correct route and three distractors. The dynamic requests came from four additional intents, and their distractors were sampled from the full pool of twelve.

Alongside routing accuracy, we recorded generated tokens, format failures, and request latency to see what each execution path involved. Here is how the requests were divided:

Fixed routes kept the menu unchanged. For “Could you please activate my card,” the model received these choices, in this order:

activate my cardchange pinpending card paymenttop up by card chargetop up failedverify my identitywhy verify identitywrong exchange rate for cash withdrawal

The correct route was activate my card, and the Qwen JSON baseline selected it. Every other fixed-route request received that same menu, whether the customer needed card activation, identity verification, or help with a failed top-up.

Dynamic candidates changed the menu with each request. The choices appear in the order supplied to the model:

Both requests have the same correct route: card payment not recognised. But the surrounding choices change, and that route appears at the end of one menu and the beginning of the other. It also belongs to the additional intents used for the dynamic panel, so it was absent from the fixed menu above. The model had to choose from the options supplied with the request; we did not retrain it between menus.

We sampled distractors and shuffled each dynamic menu, then gave every project the same saved message and ordered menu. This tests routing with changing choices and answer positions. To measure the effect of reordering alone, we would need to repeat the exact same message and candidate set in different orders; this benchmark does not isolate that effect.

Keeping the comparison consistent

The examples are the first 25 test records per selected intent in the pinned BANKING77 revision. Every project received the same saved candidate descriptions and order for each request, including the shuffled menus in the dynamic panel. Each project kept its own official prompt and readout policy; the JSON baseline received the same messages and menus.

We repeated each 300-request comparison twice, including the new JSON baseline. Every configuration produced the same individual predictions across its two repeats. Accuracy is the fraction of correct routes; explicit abstentions and invalid generated responses count as incorrect. Macro-F1 gives each target route equal weight.

What happened in the two panels

Can the model choose among the same routes?

For each of 200 requests, the model saw the same eight routes. Kev led with 98.0% accuracy (196 correct requests), followed by Laya at 92.0% (184). Verdict reached 86.5% and abstained on eight requests. LLM (baseline) Qwen3-0.6B reached 83.0%; the other configurations ranged from 79.5% for AnyJev to 52.0% for NanoJev.

Table 1. Fixed routes. Accuracy and macro-F1 are from either repeat; individual predictions and displayed metrics matched across both repeats.

Accuracy comparison of seven open-source Jev-style configurations and LLM (baseline) Qwen3-0.6B on 200 banking requests with the same eight routes available for every request. Kev led at 98%, followed by Laya at 92% and Verdict at 86.5%, versus 83% for the generative LLM baseline. Experiments used Colab GPU instances. (Image by Author)

What if the route list changes?

Here each request supplied four candidates, including the correct route from one of the four additional intents. Kev again led with 98.0% accuracy (98 of 100 requests). Verdict came next at 89.0% (89 requests), despite having ranked below Laya on fixed routes. Laya reached 74.0%.

Table 2. Changing candidates. Macro-F1 averages over the four target intents; a prediction of one of the distractor routes is an error.

Accuracy comparison on 100 banking requests, each supplied with four shuffled candidate routes: the correct route and three distractors. Kev reached 98% and Verdict 89%, compared with 66% for LLM (baseline) Qwen3-0.6B. This panel evaluates changing candidate menus and different target intents from the fixed-route panel. (Image by Author)

The result that stood out to us is that accepting a new candidate list and choosing well from it are separate abilities. NanoJev handled the changing-candidate interface, but its released game-trained checkpoint reached only 17% accuracy on these banking requests. Dynamic candidates make a new choice expressible; they do not guarantee that the model knows what to do with it.

The three Qwen readouts were close: 79.0–79.5% on fixed routes and 56–60% on changing candidates. Plain JSON generation reached 83% and 66%. That surprised us: removing answer generation did not improve routing accuracy for these Qwen configurations. The prompt and readout policy still matter.

The Generative Baseline: Why Not Just Prompt an LLM?

Part 1 asked why agents should generate tokens for small decisions. Here is what happened when we kept the Qwen weights fixed and asked for JSON.

The baseline uses Qwen/Qwen3-0.6B at revision c1899de289a04d12100db370d81485cdf75e47ca, exactly as AnyJev, Simple Jev, and OpenJev do here. It sees the same BANKING77 test messages and candidate menus. Thinking is disabled, decoding is greedy, the limit is 40 new tokens, and there are no retries. The required response is a single JSON object with one route field naming an allowed route ID.

Quality first: generation beat the three Qwen readout setups on both panels. It selected the correct route on 166 of 200 fixed requests and 66 of 100 changing-candidate requests under the strict contract. Verdict’s compact native model reached 89% on the changing panel, and Kev reached 98%.

Then the output contract: all nine dynamic failures were valid JSON objects wrapped in markdown fences. The strict JSON-only parser rejected those responses. This was a formatting failure, not broken JSON syntax or an off-menu route. A wrapper that strips fences would change the policy and should be measured as a separate configuration. Typed candidate output avoids this particular generated-string failure mode.

Finally the work per request: the baseline emitted 9.71 tokens on average for fixed routes and 9.96 for changing candidates, including EOS. In ordinary greedy decoding, T emitted tokens require one prefill plus T−1 decode calls. The native scorers and logit readouts emitted zero autoregressive answer tokens. AnyJev L0 still evaluated eight option rotations on the fixed panel and four on the changing panel; zero generated tokens does not imply one model pass for every implementation.

The table shows warm request measurements from repeat one. Latency is descriptive of each recorded request path; the scope section below explains the timing boundaries.

The GPU estimate assumes a reference compute rate of $2.50 per GPU-hour:

Cost ($) per 1,000 requests = mean warm seconds × 1,000 / 3,600 × GPU_USD_PER_HOUR

Evaluation Scope & Measurement Boundaries

The seven-project run and the generative follow-up used Colab GPUs at batch size one in separate sessions. Laya used bfloat16; the other configurations used float32. Existing SDK timings and HTTP round trips cover different boundaries. The generative timer includes prompt construction, tokenization, generation, decoding and validation with CUDA synchronization. The observed latency differences describe these configurations and request paths; they do not isolate an architecture-only speedup. GPU estimates exclude model loading, idle time, batching and additional production infrastructure overhead; the recorded request overhead remains included. Server-reported serialization tokens are not counted as autoregressive answer tokens.

The panels cover twelve BANKING77 intents. Four changing-panel intents were excluded from the fixed selection; their absence from model pretraining is unknown. Prompt policies, model sizes and training histories differ. This test does not cover all banking intents, varying candidate counts, long documents, multilingual behavior or production tool use. It does not validate calibration or an automation threshold. CPU, browser and integration capabilities in the overview come from project documentation. Test useful candidate descriptions and a fallback policy on held-out requests from your own workflow.

Which project should you try?

• If you want the strongest measured starting point in these two routing panels, try Kev. Its results were best here, and its README documents a local server and small-model options. Its local server is a practical entry point for testing your own routes.

• If runtime candidates matter and you want a compact native model, take a closer look at Verdict. It was second on the changing-candidate panel and documents CPU and browser paths.

• If you want a packaged Python project with integration options, consider Laya. Its Python integrations may fit your stack; start with the released English checkpoint used here.

• If you want to route the same interface through different language-model backends, explore OpenJev. In this run, its Qwen setup had the highest changing-candidate score among the three shared-Qwen approaches, by a small descriptive margin.

• If you want a training-free readout from a causal language model, compare AnyJev and Simple Jev on your prompts. Their readout procedures differ, and neither outperformed the trained scorers across both panels here.

• If you want to study or train a native dynamic-candidate architecture for game or action decisions, NanoJev is an interesting project. Its released checkpoint was a poor fit for these banking requests, so we would not choose it for banking routing based on this test.

So what does this all mean?

Open projects make the Jev-style decision interface something we can explore locally. In this small banking test, Kev was the strongest configuration on both panels. Verdict showed a stronger changing-candidate result than its fixed-route result, while NanoJev showed that a flexible candidate interface by itself does not ensure useful accuracy. Those findings give us leads to test, not a universal winner.

What we would carry into a new project is a small comparison: a native scorer, direct JSON generation, and a readout from a model we already have. Give them the same requests and menus. Measure correct routes, invalid outputs, latency and abstentions together. That makes the choice a workflow decision you can inspect.

Evidence and references

The project capabilities and setup paths below come from the pinned project documentation. Benchmark percentages, macro-F1 values, and abstention counts come from the saved comparison described above. They are different kinds of evidence.

[1] TypeSafe AI. Introducing System One Models & Jev.

[2] Palmer, J. Kev. Pinned README and local quick start.

[3] Heman10x. Verdict. Pinned README.

[4] NandhaKishorM. Laya. Pinned README and quick start.

[5] lookski. OpenJev. Pinned README; the measured server used a local Qwen backend.

[6] Zhang et al. AnyJev. Pinned README.

[7] Featherless AI. Simple Jev. Pinned README.

[8] TianyuCodings. NanoJev. Pinned README and CUDA quick start.