Hydration source durability: loss audit and oracle strengthening#

Current status#

PR #2049 contains the approved call-local adapter repair, committed oracles, and a patch changeset. The production base advanced to a37e69ab6aa35fd6a2a72d727de2b5168d10b1a0 before the repair was committed. The latest review receipt is authoritative for the current executable hashes, 640 passing core checks (with one existing todo), and 25 passing Chromium OPFS checks.

This document preserves a chronological audit. Earlier statements about uncommitted tests, unchanged production, no PR, old baselines, and smaller counts describe those historical stages only. Each SHA-256 binds its named stage; it is not an assertion that every historical digest matches HEAD. The final executable hashes at reviewed commit 90cbda1e5 all matched the receipt below, including the later cleanup correction to the core test hash.

Historical identity and scope: pre-fix loss audit#

At this pre-fix stage on 2026-10-06, the production baseline was 65992aacda530fde89f4c8ffa803b60cd708c425 (origin/main). Worktree branch: codex/issue-2046-red-oracles. The tests were uncommitted; the executable SHA-256 bindings below identify their reviewed content without pretending that the production commit contains these tests. Production and the lockfile were unchanged at this stage. The initial review remains an historical record of the smaller 24-core/12-browser candidate.

The user requested Field Lab's Hidden-signal recovery assay (loss-audit) and separately authorized improving the oracle. The candidate was frozen before three fresh, read-only scanners received separate sources with sibling findings hidden: S1 the oracle guide; S2 persistence contracts; S3 the existing oracle portfolio. Recovery and implementation judgment are separated below. At this stage, no Field Log, external message, production fix, commit, push, or PR had been created.

This improves enforcement of the local composition: source transactions buffered by persisted hydration must use the current scoped adapter, preserve their ordered durable obligations, settle each acceptance/receipt at its own boundary, and release that scope for ordinary/peer work. It does not claim all persistence laws universally proved or the production bug repaired.

Frozen candidate SHA-256 (paths are repository relative):

Source-by-source recovery ledger#

Pointers in this section refer to the frozen candidate or unchanged production baseline, not moving line numbers after the edits. P is the core persisted-oracle.test.ts; B is hydration-commit.opfs.ts; R is the initial review; C is src/persisted.ts in persistence-core. S is shared-logical-scheduling-oracle.test.ts; F is browser tests/shared-driver-fairness-oracle.ts; PC is browser tests/per-collection-coordinator-oracle.test.ts.

These 23 source-specific recoveries preserve overlapping support. They are not 23 distinct bugs or an instruction to restore every possible cross-product.

All scanners cautioned against turning recovery into an unrestricted Cartesian product of the persistence portfolio. The guide does not require random campaigns for this finite matrix or a second formulation without a named shared-fault hypothesis. Existing open-transaction/hydration restrictions are characterized limitations, not newly invented product laws.

Separate evaluation and implemented restoration#

The user separately requested improvements. The implementation restores the following distinctions while reusing existing primary owners:

• Individual obligations: Each receipt and each whenSyncAccepted promise is observed independently at held-read and held-durability cuts. A task boundary drains reactions while the gate remains causally held. A permanent wrong-answer test proves a fulfilled receipt cannot hide behind its sibling.
• Independent values and ordering: An immutable input edit log recomputes the last row/metadata edit. Inputs are cloned before the driver sees them. Exact transaction data, FIFO turns, sequence/row-version numbers, durable prefixes and final public/durable state are checked. put means a full source update/upsert in this core driver; the OPFS driver separately performs literal inserts, updates and deletes.
• Scope lifetime: Each controlled callback lends a fresh adapter identity. Nested calls, ordinary same-run tail commits, a second on-demand hydration, and reopen distinguish the current loan from public reentry and cached loans.
• Transaction shapes: Twelve history templates cover one/two distinct upserts, same-key replacement/delete, metadata-plus-truncate/replacement, row/collection metadata set/delete, metadata-only, truncate-only, empty, atomic multi-row, independent pre-abort, and first/second durable failure. Each runs for three phases, two scope modes and two baseline sizes: 144 histories. A populated baseline shares the source key, making precedence observable. Empty work must not allocate a durable sequence.
• Failure classes: Held first/second storage failures preserve exact cause, code/path, shared terminal error, durable prefix, suffix rejection and later same-run rejection. Publication rollback is not invented. Existing startup read-failure and incremental request-local failure witnesses now lend a distinct scoped adapter. The latter retains its successful overlapping retry.
• Acceptance before visibility: The existing causal replay witness now lends a distinct adapter, holds the optimistic handler after acceptance, checks the durable row and still-pending/absent public result, then releases publication. A repair cannot wait on the handler that awaits acceptance.
• Cleanup as an action: Four additional histories cross startup/subscription with scoped/unscheduled adapters, clean up while a source receipt is queued, reject that receipt, release the old read, restart and reject late old-control publication while fresh source work succeeds. The no-write claim is limited to work not yet issued to storage.
• Receiving host: 24 actual Chromium OPFS histories retain the original three phases, two coordinators and two baseline cases, adding single-insert versus rich transaction histories. A=11, B=22 and C=33 are separate schema adapters on the same database. A's nested work, B's cold hydration and C's ordinary source write must finish without cross-contamination. No additional SQL may enter the provider while the owning read is held. Exact metadata, schemas, rows and reopen are compared. The exact scope containing the held SELECT must exit; another scope's exit cannot satisfy that obligation.
• Causal cycle detection: Browser instrumentation matches the transaction ID passed with a live scoped adapter to its erroneous public-adapter call. Merely overlapping ordinary work cannot trigger the cycle observation.
• Failure fidelity: Browser observations are deep-copied before teardown. Every cleanup failure is retained separately, coordinator disposal cannot skip database close, and red cycles close the worker/context. A permanent browser calibration injects first and last cleanup failures and checks both diagnostics plus the intact primary observation. All setup after opening the database is inside its cleanup guard.
• Replay/provenance: The finite matrix and strengthened fixed routing witnesses now use an explicit replay guard. The versioned content hashes below bind this evidence to the changed files.

The controlled probe forwards invalid routes so the harness can release its gates and clean up. It therefore proves routing, not a simulated scheduler. The OPFS receiver supplies the real non-preemptible scope and self-cycle premise.

One attempted assertion was rejected during contract review: core truncate can establish Collection readiness at publication (collection/state.ts, truncate publication path). It cannot establish source durability. The retained test checks acceptance/receipt pendingness at that cut without inventing a stronger universal Collection-readiness law. An attempted second subset hydration in an eager Collection was also an unreached fixture path; repeated scope coverage now uses the legal on-demand path. Neither harness failure is product evidence.

Verification and hostile calibration#

The same narrow temporary scoped-adapter forwarding repair from the initial review was used only to reach the successful comparison cuts. Every calibration restored the original production file in finally.

Eight additional wrong repairs were applied independently on top of that calibration repair. All reached assertion failures; none timed out, failed setup, remained unreached, survived, or was equivalent in the tested domain:

A standalone Promise-shape probe exposed the frozen aggregate false green. The retained early-receipt calibration exercises the replacement checker. The final browser cleanup calibration supplies permanent ORC-010 evidence. Original main is itself the routing hostile subject in both controlled and actual-host tests.

The following historical session-only commands used cached tools and untracked local-source aliases. No other checkout's production build supplied test behavior. These commands are not runnable from a fresh checkout because the cache configuration and temporary typecheck project were not committed. Use the fresh-checkout instructions below to run the retained checks with repository configurations.

node node_modules/vitest/vitest.mjs run --config node_modules/.cache/issue-2046/vitest.config.ts --reporter=dot
node node_modules/vitest/vitest.mjs run --config node_modules/.cache/issue-2046/vitest.config.ts -t 'hydration source durability routing' --reporter=dot
TANSTACK_DB_ORACLE_PROPERTY=sqlite-persistence.source-fifo-order TANSTACK_DB_ORACLE_SEED=2046 TANSTACK_DB_ORACLE_PATH=0 node node_modules/vitest/vitest.mjs run --config node_modules/.cache/issue-2046/vitest.config.ts --reporter=dot
node node_modules/typescript/bin/tsc --noEmit -p packages/db-sqlite-persistence-core/tsconfig.json
node node_modules/typescript/bin/tsc --noEmit -p /private/tmp/issue-2046-audit/browser-types.json

From the browser package:

PATH="$PWD/../../node_modules/.bin:$PATH" node node_modules/@playwright/test/cli.js test --config playwright.opfs.config.ts hydration-commit-oracle.opfs.spec.ts --reporter=line --timeout=15000

Temporary logs, frozen files, calibration patch/scripts and mutation outcomes are under /private/tmp/issue-2046-loss-audit/. The assertions, laws, finite cases and verdict-critical results are retained here and in the executable files; future readers need not trust the existence of those temporary logs.

Guide conformance and remaining boundaries#

Grammar ablations: removing startup misses eager loading; removing subscription misses on-demand buffering with already-ready status; removing after-ready loses the regular route control. Removing scope distinction loses the original bug; removing default/browser coordinator loses its real workaround. Baseline zero proves no stored rows are needed; baseline one challenges source precedence. Distinct and repeated keys distinguish count/order from final membership; delete and truncate distinguish absence from stale rows; row/collection metadata and empty transactions distinguish zero mutations from zero obligations. Atomic batches check transaction granularity. Failure ordinal distinguishes preserved prefix from abandoned suffix; pre-abort distinguishes abandonment from failure of independent work. Tail, second on-demand scope and reopen distinguish current adapter loans, later ordinary use and durable storage. Bounds are 0/1 baseline, 1–3 transactions per core template, 0–3 edits per transaction, 0/1 independently pre-aborted transaction, and storage failure at ordinal 0/1. Browser rich history has six transactions plus its ordinary tail. No random-run confidence is used to extend these bounds.

The laws still have meaningful open compositions. They are retained with owners in the coverage map, not hidden by the green calibration:

• Primary persisted-history owner: dependent aborts, abort during/after publication, buffered partial-update recovery, arbitrary queued reloads, late already-issued storage success/failure across replacement, and metadata inherited from different owning reads need distinct scoped-adapter witnesses. Existing tests continue to own their unscoped semantic laws. The new cleanup witness does not promise cancellation/rollback of an already-issued write.
• Shared-driver fairness owner: K=1 and FIFO under storms already have their own models and OPFS receiver. This change proves finite peer progress and ordinary-work exclusion at a held hydrate; it does not claim the nested-source composition under arbitrary backlogs. Needed witness: nested source commit inside a hydrate while both lanes retain a K-bound-distinguishing backlog.
• Per-Collection coordinator owner: current-host schema/data isolation is now received in OPFS, but elected-owner transfer and cross-tab RPC while nested durability is pending remain with its ownership ledger.
• OPFS receiver: physical storage failure, abort/cleanup races, other browser engines and non-browser hosts need provider-supplied premises. Controlled errors in the core suite do not discharge those receiving obligations.
• Production/history: the default-coordinator repair remains open; historical package release attribution still requires a bisect. A known reachable red counterexample prevents closure regardless of the stronger test coverage.

Reviewed executable content#

Change weight against origin/main: production 0 lines; tests/fixtures/config +1705 / −220 (net +1485); documentation +559 / −0 (net +559). The larger test diff includes formatting around two existing witnesses; no production machinery was added.

Retained repair and current-base verification — 2026-10-06#

The user approved the call-local adapter repair and PR preparation after the RED audit above. The unpublished branch advanced to current origin/main, a37e69ab6aa35fd6a2a72d727de2b5168d10b1a0, before retaining the repair. That base adds Temporal persistence coverage. Its new dependency required linking an already cached temporal-polyfill into this worktree's ignored dependencies. The initial missing-import run collected no tests and is setup evidence only. No package manifest or lockfile change belongs to this repair.

SingleProcessCoordinator.requestApplyCommittedTx now accepts the interface's existing optional scopedAdapter and selects it for that call. Ordinary work continues to use the registered adapter. The coordinator does not retain the loan. This changes two production lines and removes one, for a net addition of one line. Test and contract-documentation growth is separate from that budget.

The current implementation preserves the existing hydration completion, acceptance/publication, FIFO, failure and scheduler contracts. It does not add recovery state, bypass unrelated scheduled work, or replace the default coordinator. The earlier RED-stage statements remain historical evidence.

The hostile repairs ran separately on the retained repair, and the exact repaired production file was restored in finally. None failed during setup, timed out, remained unreached, or survived. The counts refer to the 149 focused checks. Browser GREEN uses the same local source paths as RED. Cached tooling from another checkout supplies dependencies only, never the production build.

Executable SHA-256 bindings at the initial repair stage (superseded below):

Current Vitest commands add --pool=threads --pool-options.threads.maxThreads=2 --pool-options.threads.minThreads=1 to the commands above. Current browser verification uses --timeout=15000. Logs and raw prep-pr reviews are under /private/tmp/issue-2046-prep-pr/; this record preserves the verdict-critical results without requiring those temporary files.

Original issue ledger: disposition after the production repair#

This entry updates the original R1–R9 ledger without rewriting its historical verdicts. R1, R2, R3, R4 and R9 are fixed-now within the declared source hydration grammar and Chromium receiving path. R5, R6 and R7 retain the approved existing behavior and executable controls: accepted-design. R8 remains confirmed-open only for historical release attribution. The causal source change and current repair are established, but the earliest published affected version still needs a package bisect. Totals: nine raw items = five fixed-now + three accepted-design + one confirmed-open.

The scoped routing defect is repaired. The broader scoped lifecycle, fairness backlog, elected-owner/host handoff and historical-version witnesses listed in the coverage map remain separate open evidence. These bounded results do not establish universal liveness or every persistence composition.

Initial prep-pr review and cleanup oracle receipt#

The simplifier returned no proposals. Five parallel review angles checked repository instructions, obvious bugs, history, prior PR decisions and code comments. Their raw reports preserve 25 source-specific candidates, including rejected hypotheses and all nine historical review findings. No reviewer found an introduced production defect. The parent evaluated every candidate without a severity cap: one fixed-now, ten already-fixed, twelve refuted and two duplicates. The task-local append-only evaluation ledger retains the full raw claims, sources, judgments and independent cleanup score. No candidate was silently filtered or deferred. The reviewers traced the existing contracts and distinguished introduced defects from historical diagnostic limitations.

Two independent reviewers identified the same preexisting diagnostic weakness in the strengthened optimistic causal witness. Direct cleanup in finally could replace its primary error. The parent injected two sentinel failures in that actual witness: the original form reported only the cleanup error. The existing cleanupPersistedOracle helper retained the primary error and logged the separate cleanup error. Both calibration runs reached their intended error assertions without timeout. The retained test uses the helper and contains no sentinel injection. This is oracle diagnostic hardening, not another production bug or proof of a native cleanup failure. The helper's permanent calibration continues to exercise primary and cleanup diagnostics.

After this seven-line test increase, the complete persisted oracle again passed 614 tests with one existing todo. Core TypeScript and lint passed with the same 22 warnings, and the file remains formatted. Its final SHA-256 is 7156c39e8b657367aeea5491065b5836beecfeecc1f1855e344f5a4fcec36f15. The other executable hashes above remain unchanged. The eight hostile runs preceded this diagnostic-only edit and exercised the unchanged 149-case suite. Direct FIFO replay on the new base passed three checks, skipped 596, and kept one existing todo. The repair and browser fixture remained unchanged for the 25-check browser GREEN receipt.

Two review qualifications remain explicit. Exact owning-scope exit is observed for the held startup/subscription histories. The after-ready control does not establish a universal readiness-versus-scope-exit ordering. Browser reopen here creates a new Collection over retained persistence, not a physical browser or worker restart. Native failures and process restart keep their separate owners.

Reproducing the retained checks from a fresh checkout#

These commands use tracked package scripts and configurations. Run them from the repository root with the repository's supported Node.js and pnpm versions. Install dependencies and build the core package with its workspace dependencies:

pnpm install --frozen-lockfile
pnpm --filter '@tanstack/db-sqlite-persistence-core...' build

Run the complete persisted oracle through its repository Vitest configuration, with at most two workers. Coverage and integrated typechecking are disabled for this focused runtime invocation, as in the historical source-alias run:

pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest run --config vite.config.ts tests/persisted-oracle.test.ts --coverage.enabled=false --typecheck.enabled=false --pool=threads --pool-options.threads.maxThreads=2 --pool-options.threads.minThreads=1
pnpm --filter @tanstack/db-sqlite-persistence-core exec tsc --noEmit -p tsconfig.json

For only the finite routing matrix, add -t 'hydration source durability routing' to the Vitest command. To replay the existing FIFO property, prefix that same command with TANSTACK_DB_ORACLE_PROPERTY=sqlite-persistence.source-fifo-order TANSTACK_DB_ORACLE_SEED=2046 TANSTACK_DB_ORACLE_PATH=0.

Install Chromium and run the real OPFS receiver. The explicit browser channel selects the installed Playwright Chromium instead of requiring local Chrome:

pnpm --filter @tanstack/browser-db-sqlite-persistence exec playwright install --with-deps chromium
PLAYWRIGHT_CHANNEL=chromium pnpm --filter @tanstack/browser-db-sqlite-persistence exec playwright test --config playwright.opfs.config.ts hydration-commit-oracle.opfs.spec.ts --reporter=line --timeout=15000

The browser Vite configuration aliases the workspace source packages. These commands need no node_modules/.cache/issue-2046 configuration or /private/tmp project. The temporary paths elsewhere in this record identify historical session artifacts only. The reported historical counts belong to their stated source/dependency revisions, not to an unexecuted clean installation. The PR's CI separately runs the tracked package and OPFS configurations.

The documentation follow-up for CodeRabbit review 5431163123 distinguishes these reproducible entry points from the original session commands and removes nontechnical personnel commentary. No executable file, oracle law, assertion, or recorded runtime result changed.

High-effort review receipt#

This follow-up evaluates all ten items from a source-only review of 90cbda1e5, against the later documentation head 996292b31. The review found no introduced production bug. Its useful findings concern the coordinator contract, historical documentation, and oracle integrity. The runtime repair remains unchanged; this follow-up adds six lines of interface JSDoc, with no new production state, branch, fallback, or coordinator replacement.

The interface now states that wrappers must forward a supplied leader-local adapter and must not retain or serialize it. A forwarding wrapper and a loan-dropping wrapper reach the actual default coordinator and produce opposite routing observations. An explicit-adapter control also shows that this call can succeed without a registered fallback; a later ordinary call must reject before an apply until that fallback is registered. Collection construction supplies that registration. This records the existing optional-adapter design, rather than adding a new configuration restriction.

The controlled routing probe now tracks each callback's lifetime independently. Three calibration histories cover nested callbacks and overlapping callbacks with either exit order. Each distinguishes a live loan, an expired loan, reentry through the public adapter, and ordinary work after all callbacks exit. These histories describe the controlled adapter. They do not claim that the real SQLite scheduler permits overlapping hydrate callbacks.

The eight rejected-buffer histories, request-local failed-read/retry witness, and optimistic causal-replay witness each run with no scope API, a same-adapter scope, and a distinct loan. Their existing exact failure, durability, public row, acceptance, and applied-receipt comparisons are preserved. Adding a scope probe no longer substitutes for either original adapter path.

The OPFS receiver now observes both contenders at their actual scheduling entry points before taking the held-read snapshot. It continues observing peer hydrate and regular callback entry, and C's commit SQL, until A's owning callback exits. A positive count requires the C SQL observer to be reached exactly once, so a changed query cannot silently disable that comparison. Election stream-position reads deliberately bypass logical scheduling and are outside this regular-work law; treating all peer SQL as forbidden would impose a false contract. The default coordinator's tail must enter public apply; the browser coordinator's tail must enter its regular scope. Rich subscription histories join the follow-up hydrations triggered by truncate before issuing that ordinary tail. Final rows alone cannot establish correct tail routing.

Scope identity is now a callback invocation token, not a reusable adapter object or singleton active value. The actual held SELECT requires exactly one owning callback and publishes a diagnostic immediately if that premise is missing. Sixteen startup/subscription histories assert owning-scope exit. Eight after-ready controls report no held scope (null); their useful receipt, row, schema and reopen checks remain. Those controls do not prove a universal readiness-versus-scope-exit ordering.

Calibration and validation#

All runs used this worktree's source with the historical cached tooling explained above. The fresh-checkout entry points remain the tracked commands in the previous section. Temporary mutations were restored byte-for-byte. The missing-owner injection below is a deliberate harness fault, not evidence of reachable overlapping callbacks in the real host.

The original eight hostile-repair counts earlier in this record remain bound to their earlier executable revision. The new runs establish sensitivity for the newly strengthened boundaries; they do not retroactively relabel those historical runs as executions of this revision.

Review accounting and limits#

All ten review items retain separate dispositions in the task-local lossless ledger: eight fixed-now (contract prose, documentation clarity, scope probe, adapter-mode coverage, tail routing, owner diagnostics, causal preemption checks, and after-ready coverage credit), one already-fixed (personnel commentary, removed in 996292b31), and one accepted-design (explicit adapter versus registered ordinary fallback). Some claims needed qualification: the historical hashes were stage-specific and the final reviewed hashes matched; after-ready histories were not wholly vacuous; and real public A callbacks are serialized, so the hypothesized existing overlap timeout was not reproduced as product behavior. The deliberate lost-owner fault established the diagnostic weakness.

There is no deferred repair from these ten findings and no new runtime defect. The scope remains finite: the primary routing grammar, the restored failure and replay paths, and one Chromium OPFS host. Arbitrary custom coordinator wrappers must obey the stated contract; this suite cannot enforce third-party code. The separately recorded scoped lifecycle, fairness backlog, host/election handoff, native failure, and historical release-attribution gaps remain in the coverage map. A Collection reopen here is not a process or worker restart.

Current executable SHA-256 bindings for this follow-up: