Work

Frozen Small Language Models Pass Private Facts Better than Text

Year: 2026
Type: preprint
Abstract: Two copies of a frozen small language model share a task. One sees a private list of four people and their 7-letter codes. The other is asked for one person's code and can answer only from what the first sends. We give the sender a discrete latent channel: a message of 16 vectors, each one of 676 entries in a trained code-book, so the message carries exactly 150.4 bits. The sender's own hidden states choose the entries, and the receiver reads the vectors directly as input embeddings. Neither model is changed. We compare this channel with the best text we could find in 16 tokens (275.2 counted bits): the sender's own message, the raw list cut at 16 tokens, and five compact rewritings of the list. In Qwen3-0.6B the latent message is answered correctly .584 of the time, against .490 for the best text (the raw list) and .296 for the sender's own text (paired gap over the best text +.094, 95% CI [+.037, +.151]). Across three training seeds, the gap is +.103 [+.052, +.154]. In Qwen3-1.7B it is .659 against .490. A message chosen by the 0.6B sender, sent through the 1.7B receiver's code-book, scores .640, level with the 1.7B model talking to itself. On 500 fresh items held back from the start and scored once, the results hold: +.099 [+.046, +.152] pooled over the three 0.6B seeds, and +.151 [+.094, +.208] for the cross-model message. Four message controls (another item's message, zero vectors, random vectors, and a message built from the same list with the codes replaced) all score .000. The trained code-book places each letter where the receiver's own letter embedding points, with one code shared across the seven places. Probes show that the code read from the message appears at the same depth in the receiver as the code read from text, and that the receiver loads more of it ahead of writing.
Author: Sumedh Rasal
Language: English
Open Access status: green
OpenAlex ID: W7225915124
Cites:
Cited by:
Related to: 10