knopersikcuo/Planner
Watch
1
0
Fork
You've already forked Planner
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Memory V2 recall, recall hygiene, and offline retrieval eval #114

Merged
kleb merged 5 commits from feature/memory-v2-retrieval-eval into main 2026-08-25 22:19:24 +02:00
Conversation 1 Commits 5 Files changed 40 +5759 -179
kleb commented 2026-08-25 21:49:01 +02:00
Collaborator
Copy link

Summary

Three related tranches for the memory subsystem, driven by the 2026-08-25 full-log retrieval audit (V1 recall precision 12-30%, fixed-slot padding and recency-as-relevance as root causes).

Memory V2 recall policy (shadow by default)

  • Versioned v1/shadow/v2 automatic recall; v1 remains the default and rollback path, shadow returns V1 while diffing V2.
  • V2 turn recall is project-first, confidence-gated, capped at 3, and can abstain; startup continuity is bounded to two starred memories and one session summary.
  • Encrypted session pause/resume across all lifecycle hooks, immutable server-stamped observation origins, bounded encrypted recall traces, privacy-safe JSONL evaluation scorer.

Recall hygiene (applies to V1 immediately)

  • Junk session summaries (title-generation meta-prompts, task-notification envelopes, no-prompt and no-durable-memories stubs) are excluded from every recall slate; in the audited logs these were up to 12.6% of injected payload.
  • Recall slates deduplicate by id and by same-scope title; cross-scope same-title pairs stay for project-first plus personal fallback.
  • V1 extracted verbatim into recallPolicy.buildLegacyMemoryContext so both policies are pure and offline-invokable.

Temporary offline retrieval eval (removable after V2 promotion)

  • Refresh Dataset: stratified stored prompts plus LLM-paraphrased known-answer canaries (ambiguity and lexical-overlap rejection; generator is the workspace model, judge is a separate override).
  • Run Retrieval: faithful production turn-path runs of V1 and V2 plus a policy-neutral top-20 candidate pool.
  • Rejudge Results: detached, cancellable, session-only server job calling the configured AI provider directly; batch judging with per-batch checkpoints, fingerprint-matched idempotent resume, and a deterministic 10% reversed-order stability audit.
  • Report: precision@3, harmful selections, abstention correctness, canary recall, nDCG@5, MRR, paired V2-V1 deltas. Promotion stays a manual decision.
  • Isolated under packages/server/memory/eval, one derived encrypted record, one routes file, one settings tab; deleting it later is mechanical.

Verification

  • Full typecheck, type-aware oxlint, knip, oxfmt check.
  • 60+ targeted bun tests including a full end-to-end eval pipeline test against a mock AI provider (dataset, retrieval, judge, report through the real HTTP routes).
  • SettingsDialog vitest.
## Summary Three related tranches for the memory subsystem, driven by the 2026-08-25 full-log retrieval audit (V1 recall precision 12-30%, fixed-slot padding and recency-as-relevance as root causes). ### Memory V2 recall policy (shadow by default) - Versioned v1/shadow/v2 automatic recall; v1 remains the default and rollback path, shadow returns V1 while diffing V2. - V2 turn recall is project-first, confidence-gated, capped at 3, and can abstain; startup continuity is bounded to two starred memories and one session summary. - Encrypted session pause/resume across all lifecycle hooks, immutable server-stamped observation origins, bounded encrypted recall traces, privacy-safe JSONL evaluation scorer. ### Recall hygiene (applies to V1 immediately) - Junk session summaries (title-generation meta-prompts, task-notification envelopes, no-prompt and no-durable-memories stubs) are excluded from every recall slate; in the audited logs these were up to 12.6% of injected payload. - Recall slates deduplicate by id and by same-scope title; cross-scope same-title pairs stay for project-first plus personal fallback. - V1 extracted verbatim into recallPolicy.buildLegacyMemoryContext so both policies are pure and offline-invokable. ### Temporary offline retrieval eval (removable after V2 promotion) - Refresh Dataset: stratified stored prompts plus LLM-paraphrased known-answer canaries (ambiguity and lexical-overlap rejection; generator is the workspace model, judge is a separate override). - Run Retrieval: faithful production turn-path runs of V1 and V2 plus a policy-neutral top-20 candidate pool. - Rejudge Results: detached, cancellable, session-only server job calling the configured AI provider directly; batch judging with per-batch checkpoints, fingerprint-matched idempotent resume, and a deterministic 10% reversed-order stability audit. - Report: precision@3, harmful selections, abstention correctness, canary recall, nDCG@5, MRR, paired V2-V1 deltas. Promotion stays a manual decision. - Isolated under packages/server/memory/eval, one derived encrypted record, one routes file, one settings tab; deleting it later is mechanical. ## Verification - Full typecheck, type-aware oxlint, knip, oxfmt check. - 60+ targeted bun tests including a full end-to-end eval pipeline test against a mock AI provider (dataset, retrieval, judge, report through the real HTTP routes). - SettingsDialog vitest. 
Memory V2: versioned v1/shadow/v2 automatic recall with confidence-gated
project-first turn context, bounded startup continuity, encrypted session
pause/resume, recall traces, and a privacy-safe evaluation scorer.

Recall hygiene: junk session summaries (title-generation meta-prompts,
task-notification envelopes, empty stubs) are excluded from every recall
slate, and slates are deduplicated by id and same-scope title.

Offline retrieval eval (temporary, removable after V2 promotion): stratified
prompt+canary datasets, faithful V1/V2 retrieval runs with a policy-neutral
candidate pool, a detached cancellable judge job calling the configured AI
provider directly with checkpointed idempotent resume and a 10% reversed-order
audit, paired V1/V2 metrics, and a session-only settings tab.
kAgent commented 2026-08-25 21:49:02 +02:00
Collaborator
Copy link

kReview review

Verdict: 1 Medium

The change remains medium risk because the offline comparison still excludes session-summary evidence that production V1 injects, which can materially bias the reported V1 metrics.

Medium (1)

  • Include V1 session summaries in eval scoring · packages/server/memory/eval/retrieval.ts:81
    The eval still omits session summaries that production V1 injects, so prompts where buildLegacyMemoryContext adds a recent summary are scored as though that text was never selected; harmful or irrelevant session evidence cannot affect V1 precision or contamination. The production builder explicitly returns sessions: [...recentSessions] and renders their summaries, but this run records only v1.observations. Include V1 session-summary candidates in the judged pool and policy run (with distinct candidate identities), or otherwise score the rendered V1 context rather than observation IDs alone.
    Suggested fix: Represent selected session summaries as judge candidates and include them in V1 selections/pools, so metrics cover all evidence actually rendered by the production legacy context.

Reviewed by kReview at 65936b6567. This comment is conservative and based only on the PR diff, metadata, and supplied repository context.

Est. cost ~$12.16 total (4.7M in / 34.7k out) · this run ~$0.88 (196.2k in / 2.5k out) / gpt-5.6-sol + gpt-5.6-terra.

<!-- codex-forgejo-review --> <!-- codex-forgejo-review-head:65936b65675958855e5ea3f6d6569b2af5a13b7d --> ## kReview review **Verdict:** 1 Medium The change remains medium risk because the offline comparison still excludes session-summary evidence that production V1 injects, which can materially bias the reported V1 metrics. ### Medium (1) - **Include V1 session summaries in eval scoring** · [`packages/server/memory/eval/retrieval.ts:81`](https://git.kleb.sh/knopersikcuo/Planner/src/commit/65936b65675958855e5ea3f6d6569b2af5a13b7d/packages/server/memory/eval/retrieval.ts#L81) The eval still omits session summaries that production V1 injects, so prompts where `buildLegacyMemoryContext` adds a recent summary are scored as though that text was never selected; harmful or irrelevant session evidence cannot affect V1 precision or contamination. The production builder explicitly returns `sessions: [...recentSessions]` and renders their summaries, but this run records only `v1.observations`. Include V1 session-summary candidates in the judged pool and policy run (with distinct candidate identities), or otherwise score the rendered V1 context rather than observation IDs alone. Suggested fix: Represent selected session summaries as judge candidates and include them in V1 selections/pools, so metrics cover all evidence actually rendered by the production legacy context. _Reviewed by kReview at `65936b6567`. This comment is conservative and based only on the PR diff, metadata, and supplied repository context._ _Est. cost ~$12.16 total (4.7M in / 34.7k out) · this run ~$0.88 (196.2k in / 2.5k out) / gpt-5.6-sol + gpt-5.6-terra._ <!-- codex-forgejo-review-state:eyJoZWFkU2hhIjoiNjU5MzZiNjU2NzU5NTg4NTVlNWVhM2Y2ZDY1NjliMmFmNWExM2I3ZCIsInN1bW1hcnkiOiJUaGUgY2hhbmdlIHJlbWFpbnMgbWVkaXVtIHJpc2sgYmVjYXVzZSB0aGUgb2ZmbGluZSBjb21wYXJpc29uIHN0aWxsIGV4Y2x1ZGVzIHNlc3Npb24tc3VtbWFyeSBldmlkZW5jZSB0aGF0IHByb2R1Y3Rpb24gVjEgaW5qZWN0cywgd2hpY2ggY2FuIG1hdGVyaWFsbHkgYmlhcyB0aGUgcmVwb3J0ZWQgVjEgbWV0cmljcy4iLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJ0aXRsZSI6IkluY2x1ZGUgVjEgc2Vzc2lvbiBzdW1tYXJpZXMgaW4gZXZhbCBzY29yaW5nIiwiZmlsZSI6InBhY2thZ2VzL3NlcnZlci9tZW1vcnkvZXZhbC9yZXRyaWV2YWwudHMiLCJsaW5lIjo4MSwicXVvdGVkX3NuaXBwZXQiOiJ2MS5vYnNlcnZhdGlvbnMubWFwKChvYnNlcnZhdGlvbikgPT4gb2JzZXJ2YXRpb24uaWQpLCJ9XSwiY3VtdWxhdGl2ZUNvc3QiOnsidXNkIjoxMi4xNTY5NzUwMDAwMDAwMDMsImlucHV0VG9rZW5zIjo0NjYyMTk0LCJvdXRwdXRUb2tlbnMiOjM0Njc3LCJtb2RlbCI6ImdwdC01LjYtc29sICsgZ3B0LTUuNi10ZXJyYSJ9fQ== -->
Judge batches now retry once and fail on missing cases or candidate grades,
retrieval and judge stages reject a drifted corpus with 409, and the eval
job's abort signal reaches in-flight provider requests through proxyOllama.
The corpus fingerprint now includes the relation catalog revision, canary
recall@20 is computed over the unpadded neutral pool, and prompt strata get
reserved-slot largest-gap quotas instead of post-hoc truncation.
nDCG returned a perfect score whenever a case pool held no gaining
candidate, so a policy that selected harmful memories on exactly those
cases looked ideal. Such cases carry no ranking signal and are now
excluded from the aggregate for both policies, which share one pool per
case and stay paired.

Judge resume was keyed on the model string alone, so changing the AI
endpoint or reasoning level between a partial checkpoint and a resume
silently mixed verdicts from two judges under one reported model. The
run now records a fingerprint over endpoint, model, reasoning, judge
prompt, and judge schema, and only resumes on an exact match.
Candidate freshness was rendered from the wall clock at judge start, so
a run resumed across a day boundary judged its remaining cases from
different candidate text than the cases already recorded. Age now
derives from the retrieval run timestamp, which the resume key already
covers.
kleb merged commit 81783110a7 into main 2026-08-25 22:19:24 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
knopersikcuo/Planner!114
No description provided.