Year: 2023
Type: dissertation
Abstract: The purpose of this study is to utilize the generic knowledge of pre-trained Large Language Models (LLM) to enable robot interaction through natural language instruction. To demonstrate this, I have studied a plant planning and placement task implemented on a CNC based robot known as FarmBot [3]. The benefits are twofold: to interact with robots after little-to-no technical training, and to compress the amount of data required to convey complex ideas to a robot. If implemented, the reduction in learning curve would benefit the accessibility of the FarmBot. Natural Language (NL) is often too vague for a program to understand as it uses knowledge about the robot’s environment and previous human experience expressed in the form of abstract natural language descriptions. Therefore, the concept of abstraction is key in compressing a complex sequence of actions. Huge LLMs such as GPT-3 have been trained on billions of examples which enable them to associate any sentence with the nearest possible item in a database [1]. However, smaller open-source models still require fine-tuning to ground complex natural language commands that involve use of absolute or relative spatial descriptions such as, “weed plant X in the top-right corner and the opposite corner.” In this experiment, I attempt to teach a relatively small LLM model the semantics behind a mathematical constraint which represents a natural language abstraction. During inferencing, the model can place plants inside defined constraints with respect to the robot’s environment from natural language instruction. However, more general models such as GPT-4 might be the answer for more generalized constraint applications.
Author: Viloset, Michael
Language: English
Topic: Smart Agriculture and AI
Subfield: Plant Science
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7220818078