Photogrammetry-Based Zero-Shot Segmentation bridging 3D, Orthophotos and Raw Images

Preserving historical architectural monuments requires continuous monitoring to prevent deterioration from natural agents. Traditional monitoring and conservation methods involve time-consuming methods that entail manually labeling 2D photos or 3D ortho projections, which then requires label transferring back to the original 3D model or photos. This annotation process, especially when reaching stone arrangements, if performed by an expert, is time, resource, and labor-intensive and can lead to human errors and objectivity issues. This study proposes a semi-automated segmentation pipeline that combines state-of-the-art photogrammetry software, along with a robust ortho projection method, and leverages the Segment Anything Model (SAM) that supports zero-shot un-supervised image segmentation. The method enables segmentation across 2D orthophotos, 3D points clouds, and raw 2D images, offering a multimodal approach for more efficient and easy-to-generalize analysis. Tested on two different scenes, the Castle of Chambord in France and the Venetian Walls in Greece, this approach reduces manual workload, enhances segmentation propagation, and increases dataset availability for training specific segmentation models. The proposed methodology contributes to digital heritage conservation introducing an ambiguity-aware workflow transferable across different cultural heritage sites.