Prepare real retrieval evaluations and organize benchmark data #4
Loading…
Reference in a new issue
No description provided.
Delete branch "benchmark/real-retrieval-data-contracts"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The existing embedding quality numbers only cover a tiny synthetic smoke test. UI and OCR benchmark entries are now cleared without removing their suites.
Embedding runners can evaluate five pinned public BEIR datasets across scientific, argument, finance and citation retrieval: 268,484 documents and 3,404 test queries. The site shows same-run nDCG@10, Recall@10, MRR@10 and Precision@1 with dataset/revision filtering. Fresh scores still require new runs on the assigned benchmark hosts; this change claims no new measurements. The portfolio is English-only, and the documented source-license caveats still apply.
Datasets now live in data/sets/ and results in data/results/. Retained measurements and evidence are byte-identical. Shared versioned evidence contracts, publication validation and storage/protocol documentation cover the previously unstructured paths. Cached archives and private corpus content are not committed. No inference or deployment was performed.
Desktop embedding quality view:
Mobile quality view:
UI suite retained with benchmark entries removed:
Proven with 319 Python tests, 45 Bun tests, typecheck, production build, desktop/mobile browser flows, 54 retained-file byte checks and local documentation-link validation.
kReview review
Verdict: no findings
No verified defects were found in the reviewed changes.
This review did not check generated or vendored files:
uv.lock.kReview checked only the PR diff, metadata, and supplied repository context at
45cf0c954c. It reports only issues supported by this material.Estimated cost: ~$0.45 (1.1M in / 1.3k out / gpt-6.1-terra).
Run time: 0:57 total; this run 0:57.