At CVPR 2024, Niantic researchers will present “AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings”, a paper focusing on a challenging yet essential task: identifying the flat surfaces in a 3D scene from pictures taken at different angles. This work has multiple applications, such as adding digital content to the real world in augmented reality.

To identify planes in a 3D scene, AirPlanes uses a 3D mesh of the scene obtained by running Niantic’s state-of-the-art real-time depth prediction system. Once this 3D mesh is created, geometric cues can be used to identify the planes in the scene. However, geometric cues are not enough to distinguish objects which are semantically different. For example, a closed door and the wall enclosing it should be assigned different planes, because they have different semantic meanings. However, that is not possible to do from geometric cues only.

This ability to separate semantically different planes is the main contribution of AirPlanes! By relying on a deep learning model and training data which is annotated with semantic planes, the method is able to determine that objects like doors are different from their surrounding geometry and it achieves state-of-the-art performance for the task of plane estimation in indoor scenes, enabling placement of AR objects in the scene as well as providing a compressed scene representation.