FOS Units as Neural Activations: Interpretable Nodes and a Hidden-State Boundary for Recurrent Realizations
Year: 2026
Type: article
Abstract: We explore replacing a neuron's static scalar activation (such as ReLU or tanh) with a Fast Orthogonal Search (FOS) unit: a sparse polynomial map of the unit's inputs whose terms are greedily selected against a Bayesian Information Criterion (BIC) and fitted by ridge regression. A single FOS unit is white-box — an explicit algebraic equation one can read term by term — and inherits a principled, noise-adaptive complexity control. On the Santa Fe laser benchmark its selected term count falls from 39 to 8 as observational noise rises, so its sparsity is earned under noise rather than free. We then study a tamed recurrent realization: a fixed random echo-state reservoir read out by a single FOS unit, which supplies recurrent "neurons feeding neurons" dynamics without circular training. We ask when it beats the simplest baseline — a delay-embedded FOS unit — at autonomous, free-running prediction, and find a clean, falsifiable boundary. On a signal with clean recent-lag structure (the laser), the recurrent realization is overkill: a well-embedded static unit tracks longer. On signals with hidden state — delay feedback (Mackey-Glass) or partial observation of a chaotic system (Lorenz observed through its x-coordinate alone) — the reservoir free-runs roughly 3 to 6 times longer. The advantage is therefore about hidden state in general, not a quirk of any one mechanism. Finally, we show that a free-running unit's loss of validity cannot be detected from its generated output stream alone, because the output remains statistically in-distribution; it is, however, flagged in a ground-truth-free and slightly anticipatory manner by the disagreement among an ensemble of independently-seeded units. We position the work honestly relative to the Group Method of Data Handling, cascade-correlation, and reservoir computing, and are explicit about its limits. The conditional sparsity, the no-win case on clean signals, the failure of stream-only validity monitoring, and the as-yet-unbuilt fully-recurrent (mutually-fitting) network are themselves part of the contribution: they delimit where interpretable, self-structuring units help and where they do not.
Author: Merrill, Christopher K.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7166300262