Year: 2023
Type: article
Abstract: Satellite image time series in the optical and infrared spectrum suffer from frequent data gaps due to cloud cover, cloud shadows, and temporary sensor outages. It has been a long-standing problem of remote sensing research how to best reconstruct the missing pixel values and obtain complete, cloud-free image sequences. We approach that problem from the perspective of representation learning and develop U-TILISE, an efficient neural model that is able to implicitly capture spatio-temporal patterns of the spectral intensities, and that can therefore be trained to map a cloud-masked input sequence to a cloud-free output sequence. The model consists of a convolutionalspatial encoderthat maps each individual frame of the input sequence to a latent encoding; an attention-basedtemporal encoderthat captures dependencies between those per-frame encodings and lets them exchange information along the time dimension; and a convolutionalspatial decoderthat decodes the latent embeddings back into multi-spectral images. We experimentally evaluate the proposed model on EarthNet2021, a dataset of Sentinel-2 time series acquired all over Europe, and demonstrate its superior ability to reconstruct the missing pixels. Compared to a standard interpolation baseline, it increases the PSNR by 1.8 dB at previously seen locations and by 1.3 dB at unseen locations.
Institutions: ETH Zurich, University of Zurich
Language: English
Subfield: Media Technology
Field: Engineering
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,345
Award: G3161824013
OpenAlex ID: W4388739246