Token-Adaptive LoRA: Enhancing Segment Anything for Remote Sensing Imagery through Parameter-Efficient Fine- Tuning

Abstract

Abstract Adapting the Segment Anything Model (SAM) to remote sensing imagery requires domain-specific updates while retaining parameter efficiency. Existing parameter-efficient fine-tuning methods usually apply a fixed adaptation capacity to all image tokens, despite the spatial heterogeneity of remote sensing scenes. We propose Token-Adaptive LoRA, which integrates rank-differentiated LoRA experts with a Noisy Top-1 router. For each token, the router selects one expert, allowing adaptation capacity to vary across tokens while the pretrained SAM encoder weights remain frozen. We evaluate the method on TGRS-HRRSD object detection and LoveDA semantic segmentation using SAM-based downstream architectures. On TGRS-HRRSD, Token-Adaptive LoRA achieved 85.3% [email protected], improving rank-8 LoRA by 0.4 percentage points and ConvLoRA by 0.3 percentage points. The method introduced 460.82K trainable adaptation parameters and increased computation by 0.15% relative to the baseline. On LoveDA, it achieved 53.46% mIoU and 66.16% mAcc, exceeding LoRA by 0.37 and 1.44 percentage points, respectively. These results show that token-wise routing among rank-differentiated experts can improve SAM adaptation across detection and segmentation tasks with limited additional computation.