Get started
Work

Only Means Only: what a harmonization benchmark score averages over

Year: 2026
Type: preprint
Abstract: Image harmonization methods adjust the colour and lighting of an object pasted into a photograph so that it looks native to its new scene, and they are ranked by average error on one benchmark whose test images are real photographs with one object recoloured. One photograph can yield several such test images, and the benchmark promises that no photograph appears in both the training and the test split. We find that 1,399 photographs do, and the composites built on them are a quarter of the test set. Duplicate checks over file names or bytes miss this, because every composite is a different file. On that quarter the three methods we score end to end read two to five times lower error than on the rest, but most of the gap is composition rather than the sharing: the shared composites come almost entirely from one of the benchmark's four source collections, and inside that collection the gap falls to about one and a half times. An official score is therefore an average over two populations, mixed in a proportion the benchmark does not report. We release a test list of 5,581 composites whose construction identities do not appear in training, with four near-duplicate pairs disclosed rather than removed, adoptable without retraining at the price of one rescore per checkpoint. We leave open whether the sharing itself changes what a model learns. Contents of this record. The paper, the supplement, the LaTeX source of both, the record metadata and the licence statement. The identity matcher, the composite-locality and sensitivity checks, the CPU tests, the official iHarmony4 lists and the corrected test list are released under the MIT license at https://github.com/Buljic/only-means-only. What this record does not contain. No dataset, no images, no masks, no model checkpoints and no third-party baseline code or weights; those are described and cited by the paper and remain under their own terms. This is a preprint. It has not been peer reviewed. Licensed CC BY 4.0.
DOI: 10.5281/zenodo.23404529
Language: English
Open Access status: green
OpenAlex ID: W7222170754
Cites:
Cited by:
Related to: