SOS-Match: Segmentation for Open-Set Robust Correspondence Search and Robot Localization in Unstructured Environments

Annika Thomas*1, Jouko Kinnari*2, Parker Lusk1, Kota Kondo1, Jonathan How1,
*Equal Contribution 1Massachusetts Institute of Technology 2Saab Finland,
Paper arXiv Code Data
Example Image

SOS-Match is a pipeline that provides maps less than 0.5% the size of benchmark maps and localizes up to 46x faster than feature-based approaches in unstructured environments. The front end mapping pipeline utilizes the vehicle odometry sensor along with camera images to generate compact vehicle maps from object masks. The frame alignment pipeline offsets windows and uses our data association algorithm to filter the most likely correspondences.

The Batvik dataset includes a drone flying over unstructured terrain in coastal Finland.

Abstract

We present SOS-Match, a novel framework for detecting and matching objects in unstructured environments. Our system consists of 1) a front-end mapping pipeline using a zero-shot segmentation model to extract object masks from images and track them across frames and 2) a frame alignment pipeline that uses the geometric consistency of object relationships to efficiently localize across a variety of conditions. We evaluate SOS-Match on the Batvik seasonal dataset which includes drone flights collected over a coastal plot of southern Finland during different seasons and lighting conditions. Results show that our approach is more robust to changes in lighting and appearance than classical image feature-based approaches or global descriptor methods, and it provides more viewpoint invariance than learning-based feature detection and description approaches. SOS-Match localizes within a reference map up to 46x faster than other feature-based approaches and has a map size less than 0.5% the size of the most compact other maps. SOS-Match is a promising new approach for landmark detection and correspondence search in unstructured environments that is robust to changes in lighting and appearance and is more computationally efficient than other approaches, suggesting that the geometric arrangement of segments is a valuable localization cue in unstructured environments.

Batvik Seasonal Dataset

We introduce the Batvik seasonal dataset which includes six drone flights that travel a distance of approximately 3.5 km over a coastal plot in southern Finland at an altitude approximately 100 m over ground, each following the same trajectory. The flights consist of drone images collected with a nadir-pointing camera as well as Inertial Measurement Unit (IMU) measurements, and we record autopilot output along with other telemetry data from an Ardupilot-based drone flight controller. The flight takes place over an area that contains only a few buildings, and a large part of the trajectory takes place over a forest region, as well as above sea. This dataset represents flight of an UAV over a terrain that has naturally high ambiguity. We record this flight six times over many seasonal conditions.

BibTeX

@article{thomas2024sosmatch,
  author    = {Thomas, Annika and Kinnari, Jouko and Lusk, Parker and Konda, Kota and How, Jonathan},
  title     = {SOS-Match: Segmentation for Open-Set Robust Correspondence Search and Robot Localization in Unstructured Environments},
  journal   = {arXiv preprint arXiv:2401.04791},
  year      = {2024},
}