Work

Certify the Ranking, Share the Read: Output Contracts, Ownership Boundaries, and Shared-Read I/O for Exact Top-k Retrieval

Year: 2026
Type: preprint
Abstract: Certifying an exact top-k answer over external-memory vector corpora combines resident interval bounds, optional intermediate sidecar refinements, and exact full-precision vectors stored on paged solid-state media. Classical threshold and selection-with-uncertainty models price item probes additively and treat top-k verification as independent per-record refinement. We show that real paged storage and retrieval APIs violate this separable picture along two fundamental axes: physical page and gap-bridged request coalescing couples acquisition prices across records, and the downstream output contract makes optimization complexity non-monotone while minimum certificate I/O cost is nondecreasing. Modeling single-batch SSD reads with affine request-plus-page tariffs (α, β) via expanded half-open intervals J_i = [βp_i, β(q_i + 1) + α) compiled in O(n log n + Λ) time, we establish a sharp output-contract trichotomy. Certifying an unordered top-k ID set uses at most 2k monotone or single-file coverage evaluations (≤ 3k two-file s–t min-cuts); certifying ordered top-k IDs admits an O(2^⌈k/2⌉) deterministic upper bound under monotone and physical single-file coverage costs (≤ 2^⌈k/2⌉ calls, or 2^⌊k/2⌋ when k = n, and ≤ 3^⌈k/2⌉ two-file min-cuts), matching a deterministic 2^⌈k/2⌉ − 1 lower bound in the general black-box monotone cost-oracle model (while fixed-k lower bounds restricted to explicit 1D physical coverage remain open); and requiring ordered (ID, exact score) pairs collapses hindsight optimization to a single global min-cut per query while imposing a +27.46% modeled physical I/O tax (2.024 vs. 1.588 modeled ms on SciFact, inflating selected raw row actions 21.1× from 1.50 to 31.63 rows/query). Keeping tariff-modeled SciFact costs strictly separated from measured O_DIRECT wall-clock latencies, we show on eight full-corpus SciFact traces that 94.97% of the 24.05% modeled gap from multi-round cascade execution (2.091 modeled ms) to the offline optimum (1.588 modeled ms) is captured simply by single-batch request coalescing of the exact same fetched rows (97.80% after deduplicating redundant half reads), with residual cross-file reallocation adding 0.69% over that coalesced deduplicated baseline—yet within single-batch planning, shared-block min-cut allocation saves 22.54% over standalone-cheapest file selection (1.588 vs. 2.050 modeled ms, 39 strict precision reversals), and an exact O(N + EL²) path–star dynamic program collapses ternary cuts from 59,049 to 1.625 cuts/query (3^m vs. 4^m on P₄^m). Across (k, L, ownership) and multi-query batch models, we prove NP-hardness for variable-k 4096 B layouts and for L = 3 arbitrary block-ownership hypergraphs at k = 1, contrasted against (L + 1)^k-conditioned nested-tree dynamic programs (7-approximate power-of-two blocks), common-order Priority Line Cover, exact separations across ex-post union, causal sequential syscall, and offline coalesced batch pricing, and 2-/8- and 4-/16-competitive online witnesses with (2 + ε)-robust metered guidance. Measured single-thread O_DIRECT experiments on 118,287-vector COCO, ArguAna, and NFCorpus diagnose when unselective host arithmetic absorbs raw byte savings (−89.47% bytes yielding −0.19% wall time) and isolate a paired same-behavior, same-I/O (12.820 MB) ablation where selective sidecar gating cuts process CPU by 33.86% and wall time by 27.05%, complemented by zero-extra-RAM joint sign-plus-projection geometry (−39.09% bytes on 128/128 queries, −13.09% wall time).
Language: English
Open Access status: green
OpenAlex ID: W7226004574
Cites:
Cited by:
Related to: 10