FDR-Net: Foundation model-driven Depth Restoration for Transparent Object Grasping

Abstract

Transparent objects are commonly found in our surroundings and are frequently handled in automated systems such as smart factories and laboratories. Enabling robots to grasp and manipulate these objects opens up various automation possibilities. However, the unique reflection and refraction of light on the surface of transparent objects make it challenging to recognize depth information with a commercial depth camera, leading to the failure of most grasping algorithms that heavily rely on depth information. In this work, we address this challenge by transferring knowledge from the foundation model trained on a large-scale dataset to the depth restoration model. The foundation model extracts RGB features while keeping its pre-trained weights frozen. The extracted RGB features are then densely fused with the features extracted from the inaccurate depth image, and finally decoded to generate an accurate depth image. Comparative experiments with baseline methods demonstrate that our method has superior and more generalizable performance. Real robot experiments show that our method is also applicable in real environments and, when applied, improves the success rate of grasping algorithms for transparent objects.