Benchmark decision models on hosted APIs and local GGUF

kleb commented

2026-10-31 17:56:14 +01:00

Owner

kBench had no way to compare decision models: models that read a text and answer a typed question (yes/no or multiple choice) with a probability per answer, such as TypeSafe Jev or the OpenAI Decisions API. This adds a Decision suite: pinned data, one runner, matrix cases and a site domain.

Data: four public datasets pinned by archive hash and by the hash of the exact sampled requests: BoolQ (yes/no), ANLI round 3, XNLI German and SciFact (three-way choice), 300 seeded items each. python -m decision prepare fetches and checks them without inference; text stays in the gitignored cache.
Runner: python -m decision MODEL --backend api|gguf --dataset KEY. Hosted APIs (Jev 1.14/1.13, OpenAI gpt-6-luna) read keys from the environment only and fail on model-version drift. Local models (Laya, Laya multilingual, Liquid d1-omni-600M and d1-3B, Clef Flash) run in an owned llama.cpp server through its /v1/systemone endpoint on the CPU, the Iris Xe or the RTX 4070 Ti.
Results: Decision accuracy (unanswered counts as wrong), Brier score for calibration, latency, input tokens/s and RAM. Hosted APIs are a new api engine, called from kEmbed01, so their latency includes the network.
The shared llama.cpp server class gains device and mode hooks; embedding and OCR behave as before.

Decider models are not included: llama-server cannot serve them, they need a separate decider-ai backend. No runs are included yet. Three things only a real host run can confirm: Clef's /props context check, Laya's GPU offload log line, and whether the community Laya multilingual GGUF loads. Each fails the run loudly rather than producing numbers. Hosts need llama.cpp b12020 or newer in {runtime}/llama-decision/.

Proven with the full Python (358) and site (42) tests, typecheck and a static build, plus the new page in the dev server with a temporary contract-fixture result (shown above, not committed).

kBench had no way to compare decision models: models that read a text and answer a typed question (yes/no or multiple choice) with a probability per answer, such as TypeSafe Jev or the OpenAI Decisions API. This adds a Decision suite: pinned data, one runner, matrix cases and a site domain. - **Data:** four public datasets pinned by archive hash and by the hash of the exact sampled requests: BoolQ (yes/no), ANLI round 3, XNLI German and SciFact (three-way choice), 300 seeded items each. `python -m decision prepare` fetches and checks them without inference; text stays in the gitignored cache. - **Runner:** `python -m decision MODEL --backend api|gguf --dataset KEY`. Hosted APIs (Jev 1.14/1.13, OpenAI `gpt-6-luna`) read keys from the environment only and fail on model-version drift. Local models (Laya, Laya multilingual, Liquid d1-omni-600M and d1-3B, Clef Flash) run in an owned llama.cpp server through its `/v1/systemone` endpoint on the CPU, the Iris Xe or the RTX 4070 Ti. - **Results:** Decision accuracy (unanswered counts as wrong), Brier score for calibration, latency, input tokens/s and RAM. Hosted APIs are a new `api` engine, called from kEmbed01, so their latency includes the network. - The shared llama.cpp server class gains device and mode hooks; embedding and OCR behave as before. Decider models are not included: llama-server cannot serve them, they need a separate `decider-ai` backend. No runs are included yet. Three things only a real host run can confirm: Clef's `/props` context check, Laya's GPU offload log line, and whether the community Laya multilingual GGUF loads. Each fails the run loudly rather than producing numbers. Hosts need llama.cpp b12020 or newer in `{runtime}/llama-decision/`. ![decision-fixture-brier.png](https://git.kleb.sh/attachments/ad4d4838-4fdb-4a97-8acd-1938b8b85563) Proven with the full Python (358) and site (42) tests, typecheck and a static build, plus the new page in the dev server with a temporary contract-fixture result (shown above, not committed).

decision-fixture-brier.png

56 KiB

kleb added 2 commits

2026-10-31 17:56:14 +01:00

Add the decision-model benchmark runner and pinned datasets 5c4595308c

python -m decision runs one pinned model on one pinned public dataset sample
(BoolQ, ANLI R3, XNLI German, SciFact; 300 seeded items each) through a hosted
API (TypeSafe Jev, OpenAI Decisions) or an owned llama.cpp /v1/systemone server
on the CPU, Iris Xe or CUDA. The matrix planner schedules decision cases, with
hosted APIs on the kEmbed01 client lane.

Show decision models on the site and document the suite eba8573e29

Adds the Decision domain with an API engine and a Brier score metric, the
decision evidence contract shared by Python and the site, and the protocol,
result and data-contract documentation.

kAgent commented

2026-10-31 17:56:14 +01:00

First-time contributor

kReview review

Verdict: no findings

No verified defects were found in the reviewed changes.

kReview checked only the PR diff, metadata, and supplied repository context at 1548776589. It reports only issues supported by this material.

Estimated cost: ~$0.70 total (1.5M in / 4.2k out); this run ~$0.23 (560.0k in / 1.1k out) / gpt-6.1-terra.

Run time: 1:54 total; this run 0:26.

## kReview review **Verdict:** no findings No verified defects were found in the reviewed changes. _kReview checked only the PR diff, metadata, and supplied repository context at `1548776589`. It reports only issues supported by this material._ _Estimated cost: ~$0.70 total (1.5M in / 4.2k out); this run ~$0.23 (560.0k in / 1.1k out) / gpt-6.1-terra._ _Run time: 1:54 total; this run 0:26._

kleb added 1 commit

2026-10-31 17:57:27 +01:00

Retry every hosted 5xx response except 501 7a5b8d77bf

kleb added 1 commit

2026-10-31 17:58:23 +01:00

Record CPU GPU layers and nearest-rank p95 latency in decision evidence 1548776589

kleb merged commit 43d09a7bee into main

2026-10-31 17:59:06 +01:00

kleb referenced this pull request from a commit

2026-10-31 17:59:08 +01:00

Merge pull request 'Benchmark decision models on hosted APIs and local GGUF' (#6) from benchmark/decision-models into main

Rows
Columns

Benchmark decision models on hosted APIs and local GGUF #6

kReview review