Planar Gaussian Splatting & Prompt-Free 3D Segmentation for Plant Canopies

An end-to-end 3D vision pipeline that integrates Planar Gaussian Splatting reconstruction with automated 3D segmentation, enabling accurate, annotation-free canopy volume estimation in real agricultural fields.

This project builds a robust 3D reconstruction and segmentation framework for plant canopies in large, unstructured field environments. We use Planar-based Gaussian Splatting Reconstruction (PGSR) as the 3D backbone and couple it with a fully automated segmentation workflow derived from Segment Any 3D Gaussians (SAGA). Together, they produce high-fidelity 3D point clouds and instance-level plant masks that support downstream phenotyping tasks such as canopy volume estimation and growth monitoring, without requiring manual labels.

PGSR + SAGA reconstruction and segmentation pipeline — End-to-end PGSR + SAGA pipeline from multi-view input to 3D canopy masks.

Methodology

The pipeline starts from multi-view RGB images and outputs per-plant 3D masks and canopy volume statistics. It is designed to be both geometrically faithful and label-efficient:

3D backbone with PGSR. Adopt PGSR as the reconstruction backbone to represent complex canopy structures using compact Gaussian primitives, preserving fine geometric details under challenging lighting and occlusions.
Detection-guided prompt generation. Run YOLO on the RGB images to obtain plant-level bounding boxes, which define regions of interest for subsequent segmentation.
HSV-based point sampling for SAGA prompts. Within each YOLO box, apply HSV-based sampling to automatically select representative points as prompts for SAGA, removing the need for human-drawn masks or manual seed points.
From 2D segments to 3D Gaussians. Project SAGA’s 2D segmentations back into the reconstructed Gaussian field to obtain consistent, instance-level 3D masks aligned with the PGSR representation.
Canopy volume estimation. Use the segmented 3D Gaussians to estimate per-plant canopy volume, providing quantitative measurements for agronomic analysis at scale.

Overall, the system forms a single pipeline: multi-view image acquisition → PGSR reconstruction → YOLO detection → HSV-based prompt generation → SAGA segmentation → 3D mask propagation → canopy volume computation.

Results & Impact

The proposed pipeline yields high-fidelity 3D reconstructions and precise instance-level canopy masks in real field conditions. Compared to the best supervised 3D baseline model, the automated 3D segmentation achieves an intersection-over-union (IoU) that is about 65% higher, while completely removing manual annotation effort.

This work shows that combining modern Gaussian Splatting methods with carefully designed, detection-guided prompting can turn foundation-model-based 3D segmentation into a practical tool for large-scale, real-world canopy phenotyping in production agriculture.