Position-error duplicates in ICOADS R3.1: identical ship reports archived at shifted positions, 35 years after COADS
Year: 2026
Type: preprint
Abstract: ICOADS Release 3.1.0 "final", the duplicate-eliminated historical product, still contains 891,847 pairs of ship reports that are identical to the minute in wind speed and direction, sea-level pressure, air temperature and sea surface temperature but are archived at different positions. Of these, 307,945 (35%) differ by an exact whole number of degrees in one coordinate (1°, 10°, 100° and so on), the signature of a single-digit keying or transmission error. These "digit twins" make up 0.26% of complete ship reports overall and peak at 4.6–5.9 per 1,000 reports in the 1970s and 1980s, when real-time radio (GTS) copies and delayed-mode logbook copies of the same observation were merged. In 57% of the pairs both copies carry the same ship identifier, yet both were retained. Counting fractional and two-coordinate offsets, about 750,000 all-fields-identical duplicates (≈0.6% of complete ship reports) survive, and in the 1970s copies with one or two altered values add about 1.4 times that number again, suggesting roughly 1.8 million duplicates (≈1.5%) in all. The problem was first reported for COADS in 1987–88. Repeating the 1988 analysis on R3.1 for the same region and decade (equatorial western Pacific, 1970–79) reproduces the spurious correlation peak at 110–120 km for every variable; away from the equator the peak splits, because a 1° longitude shift spans only 111·cos φ km. The effect on global means is negligible, but it biases spatial-correlation estimates, inflates effective sample sizes in uncertainty models, and places real observations in the wrong ocean region. Position-tolerant duplicate identification, recommended at MARCDAT-4 in 2016, would remove most of it. Files: the preprint (PDF) and the Python code used for the global pair scan, the replication of Morrissey, Lander and Maliekal (1988), the latitude test, the near-duplicate estimates and the buddy check (zip, MIT license; see README.md). Data: ICOADS R3.1.0 final (IMMA1) from NOAA NCEI.
Author: Morrissey, Mark L.
Institution: University of Oklahoma
Language: English
Subfield: Oceanography
Field: Earth and Planetary Sciences
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225638180