WS-SAM: Segment Anything with Sparse Prompts for Weakly-Supervised Medical Segmentation

Foundation models such as the Segment Anything Model (SAM) demonstrate strong generalization in natural image segmentation but underperform in medical imaging domains, where labeled data are limited and expensive to obtain. In this work, we propose a weakly supervised training framework that adapts SAM to medical images using only sparse point annotations. Our approach constructs foreground and background token banks from training images and generates pseudo-labels for new samples via similarity in SAM’s encoder space. We fine-tune the mask decoder while keeping the image and prompt encoders frozen. To guide training and checkpoint selection, we introduce a novel score that correlates with segmentation quality without requiring ground truth. Our method eliminates the need for dense annotations or prompts at inference time, enabling fast and automated segmentation. Experiments on microscopy datasets demonstrate that our approach significantly outperforms baselines and achieves competitive performance relative to fully supervised methods.