Expand self-hosted decision benchmarks on dedicated hardware #7
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/decision-gpu-expansion"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The decision benchmark catalogue was missing many self-hostable models and measured CPU, Iris Xe and RTX 5050 coverage.
This adds pinned native author runtimes alongside the existing GGUF/API backends, guarded workstation scheduling, immutable sanitized measurements and sourced candidate research. It includes 176 measured decision configurations across four datasets; the latest expansion contributes 15 models and 60 GPU configurations. Failed and incompatible candidates are documented rather than published as measurements. Model services were restored after execution.
Proven with 394 Python tests, 46 Bun tests, type checking, full static build/data validation and real guarded runs on the designated hosts.
GPU ranking:
kReview review
Verdict: 2 Medium
Review found 2 findings: 2 medium. Highest priority: Record the measured precision in the evidence.
Medium (2)
Record the measured precision in the evidence ·
data/results/decision/evidence/r20261101-q5-01a1930a-igpu2-d-h2o-lightning-4b-anli-r3.json:145The evidence identifies the wrong model precision for the ANLI iGPU run. The linked result records
Q5_K_Mat lines 40–43 and links to this evidence at lines 62–65. This mismatch makes the measurement difficult to reproduce and affects all four Q5 evidence files.Suggested fix: Record
Q5_K_Min the evidence for all four Q5 runs.Support direct runs without HF_HOME ·
packages/runners/decision/native_zefan.py:59A direct
open-jev-2brun fails during startup whenHF_HOMEis unset. The native runner passes through the environment and does not set this variable; onlyrun-matrix.pysets it for matrix runs (lines 217–220). The bridge accesses it as a required key after downloading the model files.Suggested fix: Use the Hugging Face default cache location when
HF_HOMEis unset, or require and validate an explicit cache path before starting the server.This review did not check generated or vendored files:
13 excluded files
packages/runners/decision/requirements-author-cuda.lockpackages/runners/decision/requirements-jev-style-cpu.lockpackages/runners/decision/requirements-jev-style-cuda.lockpackages/runners/decision/requirements-kev-cpu.lockpackages/runners/decision/requirements-kev-cuda.lockpackages/runners/decision/requirements-label-gguf-cuda.lockpackages/runners/decision/requirements-laya-cuda.lockpackages/runners/decision/requirements-mapika-cpu.lockpackages/runners/decision/requirements-mapika-cuda.lockpackages/runners/decision/requirements-mapika-gguf-cuda.lockpackages/runners/decision/requirements-small-s1-cpu.lockpackages/runners/decision/requirements-small-s1-cuda.lockpackages/runners/decision/requirements-zefan-cuda.lockkReview checked only the PR diff, metadata, and supplied repository context at
1785557865. It reports only issues supported by this material.Estimated cost: ~$7.59 total (8.6M in / 7.0k out); this run ~$1.92 (2.2M in / 1.6k out) / gpt-6.1-terra.
Run time: 3:00 total; this run 0:35.
For “Record the measured precision in the evidence”: the four original evidence records retain the copied registry default Q8_0. Their paired results record the selected Q5_K_M measurement, and the execution plan pins the Q5 artifact. Repository policy forbids rewriting measured evidence or its published digest, so those bytes remain unchanged. The decision README now explicitly documents this metadata discrepancy and the selected artifact hash. Commit
8d9a20dfixes future evidence to use the engine-selected precision, with a Q5-vs-Q8 regression test. No quality, timing or memory values were modified.Dismissed “Return DecisionTune choice probabilities in the expected field”: the pinned author engine.py (SHA-256 e412c0c00140c4082e5df8aee4ce55210c7d5e725aca609de37634e6262ab091), lines 182-190, already returns {choice, probabilities, confidence}. The bridge preserves that object and adds type; nesting it again would break the real readout. All measured DecisionTune choice datasets answered 300/300 requests. The mocked bridge test had an inaccurate return shape; it is corrected to mirror the pinned author contract.
Deferred “Support direct runs without HF_HOME”: this expansion executes native models through the matrix runner, which explicitly assigns each lane a task-owned HF_HOME before startup. That is the measured and tested workflow. Adding fallback storage in the user-global Hugging Face cache is extra direct-CLI compatibility outside this benchmark/deployment change, and would weaken the task-owned cache convention. Direct invocation of these bridges currently requires an explicitly supplied HF_HOME. No measured run is affected.