Geospatial Domain Adaptation With Truncated Parameter-Efficient Fine-Tuning

Abstract

Parameter-efficient fine-tuning (PEFT) adapts large pretrained foundation models to downstream tasks, such as remote sensing scene classification, by learning a small set of additional parameters while keeping the pretrained parameters frozen. While PEFT offers substantial training efficiency over full fine-tuning (FT), it still incurs high inference costs due to reliance on both pretrained and task-specific parameters. To address this limitation, we propose a novel PEFT approach with model truncation, termed truncated parameter-efficient fine-tuning (TruncPEFT), enabling efficiency gains to persist during inference. Observing that predictions from final and intermediate layers often exhibit high agreement, we truncate a set of final layers and replace them with a lightweight attention module. Additionally, we introduce a token dropping strategy to mitigate interclass interference, reducing the model's sensitivity to visual similarities between different classes in remote sensing data. Extensive experiments on seven remote sensing scene classification datasets demonstrate the effectiveness of the proposed method, significantly improving training, inference, and GPU memory efficiencies while achieving comparable or even better performance than prior PEFT methods and full FT.