Phase learning-rate multiplier in a complex recurrent cell
Year: 2026
Type: preprint
Abstract: Multiplying the optimizer step of the phase parameters by 10 to 80 lowered loss onunseen text in a complex recurrent cell, at no measurable compute cost. We trained a27M-parameter model (2000 steps, ×1 to ×160) and a 101M-parameter model (4000 steps,×1 to ×320), with three seeds each.In the 100M model, mean loss on unseen English fell from 3.807 to 3.773 nats per token at×80, and on unseen Latvian from 3.330 to 3.295. Beyond ×80 the gain stopped growingreliably: at ×320 English and Latvian stayed within seed spread of ×80, the trainingmonitor was worse in all three seeds, and biomedical text kept falling while its seedsspread 3.6 times wider. No multiplier up to ×320 fell back to ×1 in any seed, so we did notreach a point of failure. In the small model, the monitor fell from 1.4931 to 1.4840 bits perbyte at ×10 and stayed flat to ×80. At ×160 its seeds spread by 0.0168 and one endedworse than its baseline.These are early-training results on one architecture, and they do not show that the gainsurvives long training. Continuing an already trained small model for 1000 steps at a lowlearning rate with ×3 or ×10 was neutral, within 0.0001 bits per byte
Author: Valbergs, Ziedonis
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225457167