kleb benchmarks

Persistent OCR, embedding, decision-model and future benchmark suites with an Astro static dashboard at bench.kleb.sh. Repository JSON is the source of truth. There is no application server, database, runtime API, or telemetry.

Layout

data/sets/<bench>/    Datasets, model/fixture manifests and protocol definitions
data/sets/<bench>/cache/    Downloaded dataset archives (gitignored)
data/results/<bench>/  Measured runs and reference catalogues
data/results/<bench>/evidence/  Referenced sanitized evidence
packages/web/         Astro dashboard, build checks, packaging and deploy/
packages/shared/      benchlib (timing, scoring, guards, publication) and TS result schemas
packages/runners/     embedding, ocr, decision and review packages (one entrypoint each), matrix scripts
docs/                 Deployment and memory guards

The retained measurements include embedding and decision runs across multiple datasets and hardware engines. OCR and UI currently have no measured benchmark entries. Code review keeps its most complete run. Embedding supports five pinned full-corpus BEIR tasks (SciFact, ArguAna, FiQA, SCIDOCS and TREC-COVID), totaling 268,484 documents and 3,404 test queries. Cache them without inference with python -m embedding prepare; evaluations need fresh runs. The four pinned decision dataset samples are prepared with python -m decision prepare. Source records retain dataset names and revisions; the site presents their averages in one combined dataset per suite.

Build The Site

Requires Bun 1.4.3 and Node >=22.23.4. Run from the repository root:

bun install --frozen-lockfile
bun run validate
bun run test
bun run typecheck
bun run build

Bun and uv both use workspaces over packages/*. Root scripts forward to the web package; the build writes packages/web/dist/.

packages/web/dist/ contains six HTML pages, content-hashed assets, JSON/CSV downloads, UI captures, and only referenced sanitized evidence. The build fails for invalid results, duplicate IDs, inconsistent accuracy measures, missing evidence, or a mismatch between repository data and compiled pages. Astro prerenders everything; nginx supplies conditional revalidation for HTML/data and one-year immutable caching for hashed assets. No SSR cache or running Astro process is necessary.

Site domains

The site (kBench) has five domains: Embedding, OCR, Decision, UI and Code review. Embedding, OCR and Decision rank arithmetic means of the newest run per dataset for each model and engine, one metric at a time (CPU first; a suite without CPU runs opens on its first engine). Dataset names identify these groups across revisions and configurations; timestamp ties use the greatest run ID. Older runs remain archived but are excluded from site averages and their downloads. Every selected measured value has equal weight; missing values are excluded independently per metric. Run counts, source records and evidence remain available in model details and average JSON downloads. CSV downloads contain means, contributing counts and run IDs. Only RTX 5050 GPU measurements are published; RTX 4070 Ti records remain in the repository archive, excluded from pages, downloads and evidence routes. Raw static JSON/CSV downloads retain the selected original measurements. Hardware, precision, protocols and dataset coverage can differ between models; average peak memory is not a deployment memory requirement. UI and Code review show reference catalogues in data/results/ui/ui-bench.json and data/results/review/code-review.json: static facts and committed numbers from UI-Bench and kReview, with source revisions and SHA-256 hashes. They hold no invented scores. UI-Bench has no scoring rubric, so UI results are not scored. kReview target hits are out of 27 chances and are not precision or recall. packages/web/src/lib/glossary.ts holds the plain help text for every term and metric; the build test fails for a missing term.

For an optional temporary preview, run bun run preview --port 4329 in packages/web; stop it there with bunx astro preview stop. Do not replace an existing user's server.

Run Benchmarks

Engines are fixed to hosts: CPU and iGPU benchmarks run on the kEmbed01/02 VMs (Intel Core i5-12600H, Iris Xe), GPU benchmarks on root-PC (RTX 4070 Ti). Decision and OCR GPU runs currently use W-LeSt-2025 (RTX 5050, 8 GB VRAM, 32 GB RAM); --decision-gpu-host rootpc and --ocr-gpu-host rootpc retain explicit original-host profiles. Hosted decision APIs (the API engine) are called from kEmbed01, so their latency includes that host's network. Use owned runtimes, never existing model services, and never run inference on the agent VM. Each suite has one entrypoint; the backend is a flag:

uv run --no-sync python -m embedding MODEL --backend torch|gguf|openvino ...
uv run --no-sync python -m ocr MODEL --backend onnxruntime|transformers|tesserocr|gguf ...
uv run --no-sync python -m decision MODEL --backend api|gguf --dataset KEY ...
uv run --no-sync python -m review prepare|audit|run|score ...

The embedding, OCR, decision and review instructions define setup, timing boundaries, token counting, accuracy and memory. Runners and benchlib are uv workspace members, so a host needs uv sync --locked (with its extras) before uv run --no-sync.

For host runs, build a plan and run one lane per host. build-matrix.py assigns every case to its engine's host (lanes kembed01, kembed02, rootpc, and decision-only wlest; hosted APIs on kembed01, with keys from the lane's environment); run-matrix.py runs a lane's cases one after another, each in a fresh process:

uv run --no-sync python packages/runners/scripts/build-matrix.py --run-prefix r20261030a --output /external/plan.json
uv run --no-sync python packages/runners/scripts/run-matrix.py --plan /external/plan.json --lane kembed01 --run-root /external/run

Model cleanup

Plans are model-major: a model's CPU and iGPU cases share one kEmbed node, and all of its engines and datasets run back to back before the next model starts. Each case lists the downloads it leaves in the lane cache as artifacts (paths below {cache}, which is <run-root>/cache). Once the last case that uses an artifact completes and its process tree is closed, the lane deletes it, so the host holds one model's weights at a time and never downloads them twice. The lane sets HF_HUB_DISABLE_SHARED_BLOBS=1 so a Hugging Face model folder contains its bytes.

Only files the lane created are deleted. Anything present when the lane starts is kept, for example the derived BGE-Code GGUF passed with --model-file, and so is anything else in {runtime}. A path that crosses a symlink or junction is never deleted, so a linked cache is never cleaned. A run root's cache must belong to one lane at a time. status.json lists every artifact as removed (with freed bytes), absent, retained (with a reason), failed or interrupted, and each removal is printed on stdout. A failed removal stops the lane, like a failed case. By default, when a lane stops early, the downloads of its unfinished model stay in place for inspection. Delete them deliberately when you retire that run root.

For serial queues with fresh per-model roots, --cleanup-cache requires that the cache does not already exist and removes the entire owned cache at lane exit, including failed downloads, Xet metadata, remote-code modules and staged bundles. It retains results, raw evidence, logs and state, and refuses cleanup if owned process shutdown was not verified. Never share such a cache with another lane. The runner checks a 2 GiB free-disk floor before and during every case; use --min-free-disk-bytes to require a larger reserve. Queue admission must also allow space for the largest download and any staged copies before starting a model.

Default artifact cleanup does not delete the OCR fixture font, dataset archives, raw evidence, logs, Xet metadata ({cache}/huggingface/xet) or remote-code modules ({cache}/huggingface/modules). Free disk must still fit the largest single model a node runs. Direct python -m embedding or python -m ocr runs never delete models: their caches belong to the caller and may be shared with services, and a single run cannot know whether another run will use the model. Review runs call hosted models and download no weights.

On Windows the lane runner enforces a hard per-case memory cap and stops on memory pressure or failure; the native memory probe must pass before ML. Never bypass a failed guard or resume interrupted state; keep its files and use a fresh run prefix.

Progress and stalls

Every benchmark prints progress to stderr as JSON lines, never sample text:

{
  "event": "progress",
  "at": "2026-10-30T12:00:00Z",
  "phase": "inference",
  "completed": 12,
  "total": 48,
  "unit": "batches",
  "percent": 25.0,
  "elapsedSeconds": 30.2,
  "ratePerSecond": 0.397,
  "etaSeconds": 90.6,
  "status": "inference 12/48 batches 25%, 0.397 batches/s, ETA 1m31s"
}

Phases are loading, tokenization, warmup, inference and scoring for embedding, loading, warmup and inference (pages) for OCR, loading, warmup and inference (requests) for decision, and calls for review runs. A phase reports when it starts, at most every 10 seconds while it advances, and when it completes. Rate and ETA cover the current phase only.

The lane runner reads these lines. It writes the latest one, with secondsSinceProgress, to state/<lane>/status.json and prints each new status on stdout. A case fails as stalled when its count has not changed for noProgressTimeoutSeconds (1800 in the plan), or for three units at the observed rate if that is longer. A case that prints nothing at all stalls the same way.

Results are immutable. Runners publish private raw output outside the checkout, then sanitized evidence, then a complete result record. Unknown metrics remain null; they are never inferred from unrelated runs. Missing TPS for tokenizer-free OCR is intentional: pages/s is its throughput measure. Vector dimensions and vectors/s are different metrics. Website averages are derived presentation data, not new measurements or a pooled evaluation. They never replace source results, and metric definitions and units remain separate.

After reviewing a new run's evidence for public safety, validate and rebuild. Publishing is manual, not an automatic upload or CI job. Private corpora, recognized text, credentials, personal scans and raw vectors stay out of the site and Git.

Extend And Publish

To add a suite, create data/results/<suite>/*.json using RESULTS.md, keep sanitized evidence under data/results/<suite>/evidence/, and document a reproducible protocol. Suite registration is explicit, not directory auto-discovery: update the hardware loader in packages/shared/schema/results.ts, the route list in packages/web/src/pages/[suite]/index.astro, and navigation in packages/web/src/components/Shell.astro. Catalogue and review-campaign formats have separate readers and are not hardware leaderboard rows. See the data contracts and result-type index before adding a format. The default metric vocabulary covers throughput, accuracy, process memory, dimensions, vectors/s, latency, CER, WER and the decision suite's Brier score; extend the versioned contract when a new domain needs additional metrics rather than overloading existing names.

See deployment for artifact preparation, hosting, recovery, and cache verification. This repository has no CI and no automatic pushes.