Year: 2026
Type: preprint
Abstract: We ask whether tiny decoder-only language models benefit from feed-forward layers that multiply learned feature projections directly. TriPLU (Trilinear Product Linear Unit) is a product-only degree-3 FFN branch that multiplies three projected streams coordinatewise. In a character-level TinyStories 1M-byte prefix study, TriPLU reaches mean best validation loss 1.0637, versus 1.1017 for closely matched SwiGLU, 1.0780 for a degree-4 product control, and 1.1026 for a degree-2 control. In train-only Byte-BPE experiments, TriPLU also lowers validation and heldout bits per byte on TinyStories and WikiText-2 raw under low-learning-rate settings, with PMI-slice evidence compatible with gains on seen middle- and high-PMI adjacent-token pairs. Constant-learning-rate diagnostics show that product-branch normalization can reduce the high-learning-rate best-checkpoint gap, while final BPB still degrades under hot schedules. The claim is narrow: direct product FFNs can improve fixed-budget small-model loss in specific low-compute regimes, but the branch is optimization-sensitive and does not establish FLOP-normalized efficiency, scaling, or broad LLM performance.
Source: arXiv (Cornell University)
Author: Zhang, He
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7164997759
Cites:
Cited by:
Related to: