Year: 2026
Type: article
Abstract: Evolutionary computation offers powerful optimization capabilities, but deploying population-based algorithms in cloud environments presents challenges related to state management, scalability, communication overhead, and fault tolerance. This paper presents KafkaEO, a cloud-native framework that organizes evolutionary computation through stateless services and event-driven population flows. The proposed architecture decomposes populations into micro-populations processed by independent evolutionary operators, using Apache Kafka for asynchronous communication and a lightweight controller for experiment coordination. The framework was implemented using Python, DEAP, IBM Cloud Functions, and Apache Kafka. Its performance was evaluated on selected BBOB benchmark functions across problem dimensions ranging from 2 to 40, with comparisons against the EvoSpace platform over 15 independent runs. Results indicate comparable convergence behavior, although KafkaEO incurred approximately 5–8% additional function evaluations on average. Scalability experiments involving 2–16 workers demonstrated reduced execution time, while total benchmark experiment costs remained below $15. A convolutional neural network hyperparameter optimization case study on MNIST achieved 98.7% validation accuracy and 98.5% test accuracy. These findings demonstrate the feasibility of primarily stateless, event-driven evolutionary computation for moderate-scale cloud optimization, while highlighting opportunities for improving efficiency and scaling.
DOI: 10.67231/cnx85m07
Author: Rishabh Singh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7225007425