Work

Neuro-Evolution: Representations, Efficiency and Development

Year: 2021
Type: dissertation
Abstract: Artificial Neural Networks (ANNs) are hard to design and optimise. First, we pre-process the dataset and design and extract the features. Second, we define the topology of the network, i.e., select and parameterise the type, sequence, and connectivity of the layers. Third, we train the network, which implies selecting the learning algorithm and setting its parameters. This iterative process is time-consuming, and each choice affects the others. Moreover, to address a new task, the model must be at least reviewed. In the worst case scenario, we may have to re-start from scratch. The literature on approaches that aim at automating the generation of ANNs is vast. The focus of this Thesis is on NeuroEvolution (NE) -- the set of methods that apply Evolutionary Computation to optimise ANNs. Based on the review of the state-of-the-art, it is our perception that the vast majority of NE approaches focus on the optimisation of a particular ANN structure or the generation of solutions to a specific problem. We hypothesise that using Grammar-based Genetic Programming approaches, we can propose and develop a flexible representation to design a general-purpose framework to optimise the deployment of Deep Artificial Neural Networks (DANNs). Additionally, this representation will support time-efficient search and the use of limited computational resources. After examining the performance of Grammatical Evolution and Structured Grammatical Evolution, we conclude that these methods are not appropriate to optimise multi-layered networks. Therefore, we propose Dynamic Structured Grammatical Evolution (DSGE). DSGE is the first contribution of this Thesis and is the core of Deep Evolutionary Network Structured Representation (DENSER). DENSER is a novel grammar-based NE representation, which optimises DANNs using a two-level structure. This representation can encode all aspects of ANNs: from the layer types and sequencing to the learning strategy. It can also encode any other hyper-parameter, e.g., the data augmentation strategy. One of the main limitations of the evolutionary engine used with DENSER is the time needed to generate effective ANNs. Fast Deep Evolutionary Network Structured Representation (Fast-DENSER) uses the representation of DENSER, but to speed up search it replaces the evolutionary engine by an Evolutionary Strategy. Besides, Fast-DENSER introduces a methodology to increase the training time of each individual as generations proceed. As the individuals are enabled to train for longer, by the end of evolution, the networks are fully-trained and ready to be deployed, i.e., after the end of evolution, no further training is required. Finally, we propose incremental development to avoid restarting search from scratch each time we have to solve a new problem. Incremental development facilitates the search in target tasks by warm-starting evolution. Moreover, it allows knowledge from previously addressed source tasks to be incorporated throughout evolution. In other words, incremental development aids evolution by transferring topological information from source to target tasks.
Language: English
Open Access status: green
OpenAlex ID: W7020935239
Cites: 190
Cited by:
Related to: 10