Reaping the rewards of autonomous robotic harvesting: Imitation Learning of picking motions directly from video

The agricultural sector faces severe challenges; amidst strong pressures for cost saving while maintaining sustainable practices in the face of climate change, and steep labour shortages which significantly disrupt harvesting logistics, there is tremendous value to be unlocked by robotic automation. Yet, a large portion of crops is still harvested manually, requiring human workers carry out tedious, menial tasks under conditions that raise considerable health and safety risks. Harvesting requires a combination of motions such as reaching, grasping, twisting, and pulling, all under force constraints to avoid damaging the crop being harvested. This sequence can seem effortless for human expert harvesters thanks to the tremendous capabilities and physical intelligence of the human hand, but it is extremely difficult, and at times utterly impossible, to program into a robotic controller. Imitation Learning allows a robotic agent to learn how to mimic an expert in accomplishing an activity completely circumventing the need for programming explicit procedures. This thesis explores Imitation Learning pipelines for robotic harvesting. We focus on optimizing the performance of robots in the delicate task of mushroom harvesting, leveraging end-to-end learning techniques that are adaptable across various crops. The novel contributions in this work include (i) the design of efficient representations to improve action prediction accuracy while keeping computational costs low, (ii) the development of a one-shot Imitation Learning approach for mushroom harvesting enabling high success rates from a single expert demonstration combined with a small number of auxiliary trajectories that are cheap and straightforward to obtain and (iii) the introduction of a novel, end-to-end trainable Representation Learning module that produces interpretable representation, significantly boosting the transparency of the overall Imitation Learning pipelines. These advancements lay the groundwork for the next generation of robotic automation in agriculture, ultimately facilitating significant enhancements to productivity, sustainability and resilience for this strategic sector.