Year: 2024
Type: article
Abstract: In streaming data pipelines, append-only tables often accumulate duplicate records when ingestion occurs through Kafka and Dataflow into BigQuery. SQL views with window functions are commonly used to retrieve the latest record per primary key, but this causes full table scans and high query costs. This paper outlines the challenges and presents SQL-based optimization strategies for cost-effective BigQuery performance.
Author: Tabrez Alam
Institution: J. B. Hunt Transport Services (United States)
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7089678533
FWCI:
Cites:
Cited by:
Related to: