Visual Servo Control of COVID-19 Nasopharyngeal Swab Sampling Robot

Abstract

In this study, we present a visual servo control framework for fully automated nasopharyngeal swab robots. The proposed framework incorporates a deep learning-based nostril detection with a cascade approach to reliably identify the nostrils with high accuracy in real time. In addition, a partitioned visual servoing scheme that combines image-based visual servoing with axial control is formulated for accurately positioning the sampling swabs at the nostril with a multi-DOF robot arm. As the visual servoing is designed to minimize an error between the detected nostril and the swab, it can compensate for potential errors in real operation, such as positioning error by inaccurate camera-robot calibration and kinematic error by unavoidable swab deflection. The performance of the visual servo control was tested on a head phantom model for 30 unused swabs, and then compared with a method referring to only the 3D nostril target for control. Consequently, the swabs reached the nostril target with less than an average error of 1.2 +/- 0.5 mm and a maximum error of 2.0 mm via the visual servo control, while the operation without visual feedback yielded an average error of 10.6 +/- 2.3 mm and a maximum error of 16.2 mm. The partitioned visual servoing allows the swab to rapidly converge to the nostril target within 1.0 s without control instability. Finally, the swab placement at the nostril among the entire procedure of fully automated NP swab was successfully demonstrated on a human subject via the visual servo control.