What can unlabelled carotid ultrasound B-mode images teach a Vision Transformer? Latent self-distillation for domain-specific representation learning
Year: 2026
Type: preprint
Abstract: Background: Large numbers of carotid ultrasound images are available without task-specific annotations. This study investigated whether self-supervised adaptation to unlabelled carotid B-mode ultrasound improves Vision Transformer (ViT) representations relative to ImageNet initialization. Methods: An ImageNet-pretrained ViT-S/16 was adapted using 2,840 unlabelled carotid B-mode images from CUBS and Orvile. We used an I-JEPA-inspired EMA-based latent self-distillation approach and a MAE-style masked-reconstruction approach for comparison. The learned representations were evaluated in three experiments. Experiment A assessed frozen representations using a CUBS grayscale proxy-classification task at 25%, 50% and 100% label fractions. Experiment B1 assessed frozen transfer to plaque segmentation on the independent Ar-PlaqSegm1 dataset, while Experiment B2 compared ImageNet and EMA-distillation initialization after supervised fine-tuning. All downstream experiments used three random seeds. Results: EMA-distillation achieved the highest mean AUROC at all label fractions: 0.866, 0.876 and 0.882, compared with 0.844, 0.858 and 0.876 for ImageNet. For frozen segmentation, mean Dice was 0.409 ± 0.022 versus 0.356 ± 0.033 for ImageNet, with a paired image-level difference of +0.0526 (95% bootstrap CI, +0.0033 to +0.1010). After fine-tuning, mean Dice increased to 0.467 ± 0.016, compared with 0.368 ± 0.028 for ImageNet. Conclusion: EMA-distillation adaptation improved transfer relative to ImageNet initialization with both frozen encoders and supervised fine-tuning. The results support further study of domain-specific self-supervised learning for carotid ultrasound. This study evaluates representation learning and transfer, not clinical diagnostic performance.
Author: Lim, Cher
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225417087