Feedforward Neural Networks and the Transformer model
Year: 2026
Type: dissertation
Abstract: This thesis presents the mathematical framework underlying two central architectures in modern deep learning: the feedforward neural network and the transformer model. Beginning from first principles, the report introduces the perceptron model and builds toward the multi-layer perceptron by incorporating activation functions, layer structure, and matrix notation. The learning process is examined through parameter initialization, the cost function, the backpropagation algorithm, stochastic gradient descent, and regularization techniques including early stopping and dropout. Hyperparameter tuning via grid search is also discussed. The thesis then transitions to the transformer model, introduced in the 2017 paper "Attention Is All You Need," covering the mathematical formulation of scaled dot-product self-attention, multi-head attention, tokenization and positional encoding. The encoder and decoder modules are derived in full matrix form, detailing their respective sublayers, residual connections and layer normalization. Finally, the optimization and regularization choices of the original transformer are discussed. The aim throughout is to bridge the gap between intuitive descriptions and mathematical exposition.
Author: Brunvall, Ebba
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7205320669