Privacy-preserving image captioning using virtual photon-limited imaging and federated learning
Year: 2026
Type: article
Abstract: • PMD-PCI virtually acquires inherently sparse images while preserving data privacy. • 224 × 224 images with over 50k photons enable accurate captioning. • ResNet101 and ViT effectively process photon-limited inputs for caption generation. • Outperforms AES and DRPE in balancing privacy and caption quality. • Federated learning achieves near-optimal accuracy in 2–4 rounds without raw data sharing. The growing demand for visual privacy in optical imaging systems has motivated the development of frameworks that can both preserve privacy and maintain utility in downstream tasks. In this work, we propose a privacy-preserving image captioning framework that integrates Poisson Multinomial Distribution-based Photon Counting Imaging (PMD-PCI) with deep learning techniques. PMD-PCI simulates photon-limited imaging conditions by generating highly sparse multispectral images, thereby inherently concealing fine visual details. These sparse representations are used as inputs to two encoder-decoder architectures, ResNet101-Transformer and ViT-Transformer, for automated caption generation. To further enhance privacy and reduce the risk of centralized data exposure, we employ federated learning, allowing model training across distributed clients without direct access to raw images. Experimental evaluations on the Flickr8k and Flickr30k datasets show that accurate captions can be generated from photon-limited images with more than 50,000 incident photons at a resolution of 224 × 224 pixels. On Flickr30k, the proposed ResNet101-Transformer achieves a BLEU-4 score of 17.32 and a CIDEr score of 27.80 at 50,000 photons in a centralized setting, demonstrating that meaningful captions can be produced even under severe optical sparsification. Compared to traditional encryption-based techniques such as Double Random Phase Encoding (DRPE) and AES, our approach provides a better trade-off between privacy and captioning performance. Furthermore, convergence analysis reveals that federated learning achieves near-optimal performance within just 3 communication rounds, significantly reducing the communication overhead required for training. The proposed framework bridges physical-layer image privacy with optical imaging and learning-based caption generation, making it suitable for secure, low-light, or resource-constrained vision systems.
Source: Results in Optics
Authors: Antoinette Deborah Martin, Inkyu Moon
Institution: Daegu Gyeongbuk Institute of Science and Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,470
Awards: G1642157022, G984383550
OpenAlex ID: W7123346566