Banyan: Converting Pretrained Language Models to 1-Bit Weights with Homotopy On-Policy Distillation
Year: 2026
Type: preprint
Abstract: Status: pre-results research proposal (version 0.1). This document reports no new experimental results. A 1-bit language model stores each weight as a single sign and shares one scale across a small group of weights. At 1.125 bits per weight it needs about a fourteenth of the memory of its 16-bit parent, which matters most on devices where memory bandwidth, not arithmetic, limits generation. PrismML's Bonsai models show that a pretrained model can be converted to this format and keep most of its capability, but the method behind them is unpublished, and the one open reproduction we know of falls well short. This whitepaper proposes Banyan, an open recipe for the conversion, built on Homotopy On-Policy Distillation (HOPD). HOPD starts the student at its full-precision parent and raises quantization strength gradually, training on text the student generates itself, scored token by token by the parent. A KL-gated controller keeps each step toward binary weights only if the student recovers within a fixed budget, and rolls the step back otherwise. Neither gradual quantization nor on-policy distillation is new. Our hypothesis is that they help each other: a gradually quantized student stays a usable generator for longer, so learning from its own outputs can start at step zero instead of after a separate initialization stage. We give the full method, an implementation for whole-network binarization with tied embeddings and explicit bit accounting, and a staged plan that keeps the mechanism test apart from the strongest-baseline comparison. Kalaris Labs: https://kalarislabs.com · Code and releases: https://github.com/KalarisLabs · Author ORCID: 0009-0004-8421-0087
Author: Chowdhury, Sayan
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225336817