Summer Delta Attention: Bounded Residual Retrieval, Memory Compression, and Update Consistency
Year: 2026
Type: preprint
Abstract: Summer Delta Attention (SDA) augments a no-decay delta-rule associative state with a bounded archive of original key–value pairs. A cosine-thresholded nearest-neighbor lookup corrects the state prediction using the residual at the retrieved key. We formalize this readout, characterize its distinction from a hard switch to the retrieved value, and examine memory compression and stale-value behavior in an untrained NumPy reference. An online int8 archive with float32 state, per-row scales, and write identifiers occupies 17,408 persistent array bytes at key dimension 32, value dimension 16, and capacity 256. Across 16 fresh seeds in a single-write synthetic task, compression increases mean squared error by 1.42 × 10⁻⁵ at query noise 0.4 relative to a float32 reference. In a separate five-workload study comprising 3,600 seed-level measurements, exact stored-key overwriting substantially reduces stale retrieval under repeated value updates. However, its benefit disappears under small key drift, and improved archive management does not establish an advantage for SDA’s residual readout: a near-equal-byte, state-free archive achieves lower error in several tested conditions. These results identify update consistency, readout controls, and complete memory accounting as necessary evaluation criteria. They do not establish trained-language-model quality, state-of-the-art performance, or a general advantage over attention architectures. This is a non-peer-reviewed research preprint, version 1. The author is an independent researcher and reports no external funding or support and no conflicts of interest. AI assistance is disclosed in the paper. This record contains the paper PDF only; the source/evidence supplement is not deposited here. Project repository: https://github.com/Summer110622/SummerDeltaAttention
Author: Okamoto, hiroki
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223476027
Cites:
Cited by:
Related to: