Concepts for Data Engineers: Slowly Changing Dimension (SCD)

You probably already use it but don’t know!

In this series I’m introducing several important concepts that new Data Engineers should be aware of. The other topics I talked so far:
✅ Data Modelling
✅ CDC
✅ Idempotency
✅ ETL x ELT x EL
✅ Kappa x Lamda Data Architectures

I also have 2 series about python:
🐍 Efficient Python
🐍 Software Engineering with Python

lets get started!

Slowly Changing Dimensions (SCDs) play a crucial role in capturing and preserving historical data within a data warehouse. SCDs enable organizations to analyze trends, track changes, and make informed decisions based on the evolving nature of their data. In this article, we will delve into the different types of SCDs and provide real-world examples of data tables employing these techniques. I will be dealing in this article with the most common ones, even tough there are more SCD types.

Type 1 — Overwrite

The Type 1 SCD approach involves overwriting existing data with new values, effectively discarding the historical…