Year: 2017
Type: dissertation
Abstract: In the passed decade, deep learning has achieved state-of-the-art performance for various machine learning tasks.The wide applicability of deep learning mainly arises from the ability of deep neural networks to learn useful features by themselves.By combining multiple layers in a neural network, hierarchical representations can be created with an increasing level of abstraction.However, there is one fundamental difficulty in learning deep neural networks, which is known as the vanishing gradient problem.Today, this issue has been alleviated by new activation functions and better ways to initialise weights.Although less severe, also internal covariate shift slows down learning in deep networks.Several techniques such as batch normalisation have been proposed to counter this issue by normalising the data in each layer.An alternative approach is to construct the layers in a network so that their activations have the same mean and variance as the data in the input.Networks which are capable of doing so are called Self-Normalising neural networks (SNNs) and can be constructed by enforcing certain characteristics onto the mappings that each layer induces on the moments of its input data.In this thesis, we extend the idea of analysing the moments in SNNs to the backward pass, in which we both theoretically and empirically investigate the backward dynamics.Further, we extend SNNs to networks with bias units and show that similar dynamics hold as for the weights.Additionally, we compare the performance and learning behaviour of SNNs with respect to different data normalization techniques, weight distributions for initialization and optimisers on several machine learning benchmarking data sets.We find that (1) the variance of the weights steadily increases with very small steps in networks with random errors, (2) the architecture affects how the error signal propagates back through the network and (3) that an error signal with reduced variance in lower layers can be advantageous for learning.Furthermore, our analysis reveals that SNNs perform best with whitened data, the choice of the initial weight distribution has no significant effect on the learning behaviour and that most adaptive learning rate schedules do help, although they break the conditions for self-normalisation.i First of all, I would like to thank the bioinformatics institute at the JKU that provided me access to GPUs.This saved me quite a bit of time to run all of the experiments.They also provided me with a forum to present my ideas and retrieve valuable feedback.Special thanks to Günter Klambauer who has patiently guided me through this thesis and provided new ideas .I would also like to thank the people that contribute(d) to open source projects like Tensorflow (Google) (Abadi et al., 2015), Theano (MILA) (Al-Rfou et al., 2016) and Keras (François Chollet) (Chollet, 2015).These deep learning frameworks greatly simplified the process of building and training neural networks in python.Especially Keras, which allows to switch between different lower-level learning frameworks, has been very useful to set up experiments for this thesis.Also matplotlib (Hunter, 2007) has been of great use to make the visualisations of the experiments as they are.I would also like to explicitly thank Stéfan van der Walt and Nathaniel Smith for developing virridis.Their quest for the perfect colour map has made it easier for me to focus on content rather than colour schemes.For more information on their work, I refer to https://bids.github.io/colormap/.The (free) availability of platforms as wolframalpha and the wolframcloud have been extremely useful for the more theoretical parts of this thesis.Being able to double-check most of the equations has solidified the confidence in my research and writing.Last, but not least, I would like to thank Katharina Prinz for her patience, multiple reviews and valuable feedback for this thesis.v 5.1 MNIST images . .
Author: Hoedt, Pieter-Jan
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7002186285