Study on Change Data Capture Techniques for Incremental Loading in Data Warehouse

Abstract: Incremental loading is a preferred data warehouse (DW) refresh technique in the modern world for being more efficient than Full loading. A special case in full loading where the entire content at the target is dropped and reloaded for the purpose of refresh is known as Truncate and Load. Incremental loading involves refreshing only the changed contents from the data source into the data warehouse. The gathering of these changed contents is known as Change Data Capture (CDC). This paper presents a study on different CDC techniques used in practice to aid the incremental loading. An example scenario which uses the truncate and load approach is presented and the suitability of the CDC techniques to implement incremental loading is discussed. The implementation for conducting a CDC based on the delta rules for migrating from a naive truncate and load to an effective incremental load in the example scenario is presented in this paper.

Keywords: Truncate and Load, Incremental Load, Data Warehouse, Change Data Capture, Delta, Log, Snapshot, Timestamp, Metadata.