Batch Updates and CDC at Scale: A Comparative Study of Iceberg and Paimon
Year: 2025
Type: conference-paper
Abstract: Business digitalization in data‑intensive sectors such as fintech, advertising, and telecommunications has shifted early-stage storage toward extract–load–transform (ELT)-centric data lakes, while the need for transactional guarantees and SQL semantics has accelerated the rise of data lakehouse architecture. This paper evaluates the practical use of open table formats within a financial analytics platform, focusing on Apache Iceberg and Apache Paimon. We report two experiments. First, we model data-mart maintenance by generating a 1 TB table (~10 billion keys) and executing staged SELECT/UPDATE operations over 10–50% of keys, exposing how table format choices affect update-heavy patterns. Second, we assess ingestion via a Change Data Capture (CDC)–Kafka–Flink–Paimon pipeline that replays 1.5 billion CDC operations per day with strong recency skew and assess its viability for long-term data-mart storage and incremental updates without intermediate databases. We also benchmark a CDC–Kafka–Spark–Parquet ingestion approach to estimate the performance gain by using the open table format. Our findings surface design trade-offs that materially influence read and write performance and operational overhead, and we derive practical guidance for selecting open table formats when building lakehouse architectures in financial settings.
Authors: Ievlev, Kirill, Surpin, Vadim
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7104467548
FWCI:
Cites:
Cited by:
Related to: