Training LLMs to translate Mission Descriptions in Natural Language into Controller for Robot Swarms

Abstract

Collective robotics is a challenging field, particularly when designing controllers for swarm robots, where complex inter-robot interactions lead to unexpected collective behavior on the global level. Traditional methods based on evolutionary algorithms can solve individual tasks but require extensive computation, and human expertise for modeling and defining fitness functions, and typically yield controllers that do not generalize across different types of missions. For example, a controller for aggregation cannot be used on a foraging mission without modification. This master thesis explores the potential of Large Language Models (LLMs) to serve as a method for general swarm behavior generation. LLMs have already demonstrated the ability to solve a wide range of tasks in robotics already. This work explores whether LLMs can bridge the gap between individual robot actions and overall swarm behavior on the global level by generating robot controllers for swarm missions. This could have the potential to further enable natural language interaction with swarms as well as one-shot generation of valid controllers without the need for dedicated optimization. In the scope of this thesis, a dataset of natural language descriptions and robot controllers is created. Using AutoMoDe-Maple, controllers for the mission types Aggregation, Foraging, Distribution, and Con- nection are obtained. In several experiments, a 7B LLM is fine-tuned and the LLM-generated controllers are compared against the AutoMoDe baseline. Factors such as dataset size, the impact of natural versus formalized language prompts, reinforcement learning, and transferability to unseen scenarios are investi- gated. The results indicate that, while the fine-tuned LLM can replicate state-of-the-art controllers in individual cases and show promising generalization capabilities by generating viable controllers for mission types not encountered during training, their average performance does not reach that of AutoMoDe. It shows large variability regarding quality. These findings suggest, that LLMs hold potential as a general model for swarm behavior and are viable for controller generation from natural language, but further research with larger models and more diverse training datasets is necessary for more decisive and optimal results.