Work

A Comparative Study of Activation Functions in Deep Learning Models

Year: 2025

Type: article

Abstract: Activation functions play a vital role in the training dynamics and generalization performance of deep learning models. This study presents a comparative analysis of ten widely used activation functions—ReLU, Sigmoid, Tanh, ELU, SELU, Softplus, Softsign, Swish, GELU, and a custom spline-based function—within a unified convolutional neural network (CNN) architecture. All models were trained and evaluated on the CIFAR-10 dataset under identical experimental settings, including fixed learning rate, batch size, number of epochs, and architecture configuration. The results show that the proposed spline activation function achieved the highest test accuracy of 71.48%, outperforming popular functions such as ReLU (67.87%) and Swish (68.33%). In contrast, traditional functions like Sigmoid exhibited significantly lower accuracy (10.00%), reaffirming known limitations in deep network contexts. These findings demonstrate the potential of adaptive, piecewise-defined activation functions to enhance model performance while maintaining competitive training efficiency. The study provides practical insights into activation function selection for image classification tasks.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.17258718

Authors: Abbaz Primbetov, Navruz Akbarov

Institutions: Tashkent University of Information Technology, University of Tashkent for Applied Sciences

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7084583715

FWCI:

Cites:

Cited by:

Related to: