The Power of Data Vault Modeling: Enabling an Agile Data Warehouse

Challenges in building data warehouses for data modeling

Relational data models have been around for a long time, especially in operational data systems. Relational data models are often the go-to models for databases, yet they are far from perfect for every purpose. Building and maintaining a data warehouse with a relational data model can actually be a challenge. Users often run into one or more of the following drawbacks:

• Relational data model in data warehouses often cannot be adapted to the volume of changes in the source systems.
  
• Relational data models in data warehouses often require central version control of the data model, which makes them unsuitable for distributed data ownership.
  
• Relational data models lack flexibility, making it challenging to deploy the same data model across multiple domains (e.g., countries and departments).
  
• The velocity of the data ingestion process in a data warehouse with a relational data model is hampered due to a large number of transformation requirements and business rules.
  

Considering these drawbacks, a commonly favoured alternative to a relational model is the utilization of (multiple) dimensional models featuring conformed dimensions, or opting for a snowflake model. Based on our experience, the most effective approach for your data warehouse is ensemble modeling, with Data Vault as the leading ensemble modeling pattern.

How Data Vault deals with current drawbacks

Data Vault is a transformative solution that provides significant advantages to businesses across a variety of industries. While relational data models with 3rd Normal Form (3NF) have limitations when it comes to accurately reflecting real-world business logic and scenarios, Data Vault provides a powerful solution that addresses these issues head-on. Its business-driven and business-facing approach ensures that users at all levels of the organization can easily understand and engage with the system, making it a highly accessible and user-friendly option. Furthermore, unlike traditional dimensional modeling which can be slow to adapt to changing business needs, Data Vault's ability to seamlessly adapt to new business needs, data sources, and changing rules without requiring expensive re-engineering provides a powerful solution to the challenges posed by an agile and rapidly changing business environment. The ability of Data Vault to be agile stems from its decomposed concept and its ensemble, as illustrated in Figures 1 and 2. Therefore, the 3NF works best for operational systems, dimensional modeling is ideal for data marts, and ensemble modeling, with Data Vault as the leading modeling pattern, reigns supreme in the realm of data warehousing, as illustrated in Figure 1.

Figure 1: An example of Data Vault Ensemble for Data Warehouse

Staying ahead of the curve with Data Vault

Whether you’re dealing with a high volume of changes in source systems or significant deviations from initial design principles, Data Vault offers a flexible and agile approach that can help businesses to stay ahead of the curve and succeed in a rapidly evolving marketplace. Also, its unique capabilities enable businesses to achieve comprehensive data traceability, resulting in a fully auditable system. When it comes to project management, utilizing Data Vault allows for the application of agile development techniques, which reduces project risk while delivering frequent updates. Additionally, its incremental build approach enables the construction of a scalable architecture without sacrificing its core components. The incremental build approach is also made possible because Data Vault has decomposed concepts, as illustrated in Figure 2. From an architectural perspective, the parallel loading capability of Data Vault facilitates the accommodation of future expansion and growth.

Figure 2: Decomposed Concept in Data Vault vs Encapsulated Concept in 3NF or Dimensional

Limitations and workarounds of Vault modeling

While Data Vault has proven to be an effective solution for addressing the current challenges, it’s important to acknowledge and understand its potential limitations. Despite its many benefits, there are still some disadvantages that must be taken into consideration to ensure that Data Vault is the right fit for your business needs. First, the increase in semantic complexity of the underlying database architecture that’s required to support Data Vault since ideally the technical implementation of Data Vault is one to one with concepts. Hence, one should be able to recognize business terminology in the underlying database. Second, while Data Vault is optimized for write performance, it’s not specifically designed with read performance in mind, which can impact the user experience. Third, transitioning from the 3NF (or dimensional) to a Data Vault may present challenges that require substantial proficiency in Data Vault methodology within the organization. To sum up, we move away from Data Vault as modelling choice for our presentation layer to a modeling approach more suitable for consumption, in this case dimensional. However, this can be virtualized on top of the Data Vault, rather than a physical implementation.

Maximizing benefits and minimizing limitations for your data strategy

To mitigate the potential downsides of Data Vault, one common approach is the implementation of a demand-based harmonization layer on top of the existing architecture. This can help to streamline data integration and harmonization across different entities and sources. By leveraging this approach, businesses can ensure that harmonization is only performed when necessary, minimizing unnecessary overhead and improving overall efficiency. Furthermore, this approach is typically implemented on a per-track basis, allowing for granular control and flexibility in data management. Importantly, the harmonization layer is designed to be a logical rather than a physical layer, reducing complexity and enhancing overall agility. This harmonization layer can also help to improve read performance while simultaneously reducing semantic complexity for business users, resulting in a more streamlined and user-friendly system. Additionally, incorporating unique IDs within the harmonization layer can help to more effectively enforce cardinality constraints, ensuring the integrity of the data and reducing the risk of errors or inconsistencies. By leveraging these strategies, businesses can maximize the benefits of Data Vault while minimizing its potential limitations.

If you are considering the implementation of a data warehouse utilizing the Data Vault modeling approach as its core data model, we encourage you to proceed. As evidenced, there are numerous advantages associated with this approach such as historical reproducibility, data lineage & auditability, handling changes over time, and model Integrity. Moreover, you now have insights into its constraints, which will act as an invaluable reference when deciding the best way to adapt Data Vault to your organization’s specific requirements.

In our upcoming article, we will delve into the Finance and Risk data warehouse architecture, highlighting best practices from one of the leading banks in the Netherlands that has adopted Data Vault for its data warehouse. This article will also discuss its journey to identify the most suitable approach for implementing Data Vault within their organization.

Contact

If you would like to know more about Deloitte’s Data & Analytics services within Financial Services, please reach out to Yuri Jolly – Director - Risk Advisory , Ali Khalili – Senior Manager- Risk Advisory, Marit Beerepoot – Junior Manager – Risk Advisory or Audia Ariantari – Senior Consultant – Risk Advisory