Work

Detect, segment, cluster: Apple localization for robotic harvesting in complex orchards

Year: 2025

Type: article

Abstract: Efficient robotic apple harvesting hinges on robust 3D vision systems capable of accurately identifying and localizing fruits in complex orchard environments. While localization algorithms leveraging RGBD sensors and deep learning methods exist–through extrapolating 2D detections into 3D space, they often suffer from occlusions and noisy depth data, resulting in limited real-world accuracy. To overcome these limitations, we propose a unified 3D localization framework that seamlessly integrates vision-language model (VLM)-based apple detection, instance segmentation, and unsupervised point-cloud clustering. Our method first detects and segments apples in 2D images using a foundation VLM (e.g., Grounding-DINO) and a high resolution instance segmentation model. The identified apple pixels are then projected into 3D space, where we apply Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to refine localization by filtering out noise and unreliable depth measurements. We benchmark our approach against traditional 2D-to-3D extrapolation and 3D localization techniques, demonstrating its superior robustness and precision under challenging conditions. Evaluated on a dataset of RGBD images containing 219 apples across indoor and outdoor environments, including both real and synthetic apples, our method achieves a mean absolute localization error of 7.7 mm—a 74.3% improvement over the best-performing baseline methods. Our approach has been deployed on an in-field robotic harvesting system, demonstrating real-time performance and strong robustness to occlusions and variable lighting—highlighting its practical value for autonomous fruit picking in unstructured orchard environments. • Robust 3D apple localization pipeline improves localization accuracy of occluded fruits. • Density-based clustering enables real-time noise filtering for robotic har- vesting. • Vision-language detection, segmentation, and clustering deployed on a field robot.

Source: Smart Agricultural Technology

DOI: 10.1016/j.atech.2025.101642

Authors: Siddhartha Bhattacharya, Chaaran Arunachalam, Kaixiang Zhang, Jiajia Li, Renfu Lu, Zhaojian Li

Institutions: Michigan State University, Agricultural Research Service, United States Department of Agriculture

Language: English

Topic: Smart Agriculture and AI

Subfield: Plant Science

Field: Agricultural and Biological Sciences

Domain: Life Sciences

SDG: Zero hunger

Open Access status: gold

Estimated APC paid: $1,770