Park, Jungin Lee, Jiyoung Sohn, Kwanghoon 2024-01-12T02:46:03Z 2024-01-12T02:46:03Z 2023-11-17 2023-06 1063-6919 https://pubs.kist.re.kr/handle/201004/76431 In this paper, we efficiently transfer the surpassing representation power of the vision foundation models, such as ViT and Swin, for video understanding with only a few trainable parameters. Previous adaptation methods have simultaneously considered spatial and temporal modeling with a unified learnable module but still suffered from fully leveraging the representative capabilities of image transformers. We argue that the popular dual-path (two-stream) architecture in video models can mitigate this problem. We propose a novel DUALPATH adaptation separated into spatial and temporal adaptation paths, where a lightweight bottleneck adapter is employed in each transformer block. Especially for temporal dynamic modeling, we incorporate consecutive frames into a grid-like frameset to precisely imitate vision transformers' capability that extrapolates relationships between tokens. In addition, we extensively investigate the multiple baselines from a unified perspective in video understanding and compare them with DUALPATH. Experimental results on four action recognition benchmarks prove that pretrained image transformers with DUALPATH can be effectively generalized beyond the data domain. English IEEE COMPUTER SOC Dual-path Adaptation from Image to Video Transformers Conference 10.1109/CVPR52729.2023.00219 1 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.2203 - 2213 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2203 2213 US Vancouver, CANADA 2023-06-17 2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 001058542602052 2-s2.0-85173910084