Addressing tonal weaknesses in neural audio codecs through synthetic data augmentation and balanced training
Year: 2026
Type: article
Abstract: Neural audio codecs (NACs) achieve strong perceptual quality at low bitrates but consistently struggle with harmonic-rich, sustained tonal content such as bells, glockenspiel, and piano. Prior work has used synthetic tonal augmentation mainly to improve embedding quality for evaluation; we instead treat it as a first-class training methodology that directly targets codec reconstruction fidelity. This paper presents Project Chimera, a training framework for tonal-robust NACs, instantiated as a 48 kHz codec with 32 codebooks at 30 kbps. The framework introduces three coordinated components: (1) a physically-inspired tonal synthesis generator modelling inharmonic partial structures and resonant decay beyond simple sinusoids; (2) a difficulty-aware curriculum sampling strategy that progressively increases tonal complexity during training; and (3) Exponential Moving Average (EMA) codebook stabilisation with quantisation-capacity analysis that characterises and prevents Residual Vector Quantisation (RVQ) collapse under tonal training. In limited-scale experiments on a 720-hour dataset, Project Chimera achieves 0.51 mel distance and 4.42 ViSQOL at 30 kbps on these objective metrics, outperforming the Descript Audio Codec (DAC) at comparable bitrates. A listening test with $$n=22$$ trained listeners shows subjective gains of approximately $$+0.47$$ MOS overall and $$+0.9$$ MOS on tonal material; the 95% confidence intervals for the overall, tonal, and general-music comparisons separate from DAC, while speech and environmental scores remain comparable. Per-phase checkpoints of the curriculum run show tonal fidelity, codebook entropy, and subjective quality improving monotonically as tonal complexity is staged in. The objective results come from a limited-scale training run; large-scale objective replication remains future work. Code and model weights will be released upon acceptance.
Source: Scientific Reports
Authors: Abhinav Mondal, Rishi Gupta
Institution: Manipal University Jaipur
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7203650746
FWCI:
Cites:
Cited by:
Related to: