Cloud platform for data mesh in the automotive industry

In most cases, all data accumulated will be converted by the IT system into a centralized ETL-pipeline (extract-transform-load). Thus, the data is stored in monolithic data warehouses or data lakes that have evolved over decades. However, this approach forces OEMs to find a solution regarding the use of the data.

Cloud Uses Cases: Automotive Industry

Challenges for automotive OEMs

In the context of growing data volumes automotive OEMs are facing numerous challenges, such as:

• Complex data sources lead to a difficult and slow integration
• Data sharing problems intensified by a silo mentality
• Lack of expertise regarding data sources in centralized data ingestion and warehousing teams
• Limited ability to address data quality issues at the source or react to changes
• Need for a more economical, effective, and efficient IT architecture as dispersed data is becoming the norm
• Lack of a domain-oriented data catalog to ease data discovery for consumers

As IDC has shown, 64.2ZB (zettabyte) of global data has been created or replicated in 2020¹. By 2025, IDC forecasts a compound annual growth rate of 23% in terms of global datasphere. But just having more data is not the key. According to IDC's "Global DataSphere Study 2021", less than 3% of the data currently created is analyzed to affect enterprise intelligence. 59% of mid- to upper-level managers responded that they are overwhelmed by the amount of available information. Nevertheless, nearly half of all respondents believe that there is not enough data for appropriate decision-making. Enterprises that overcome these challenges will be successful in the long run.

Data mesh – an architectural paradigm unlocking analytics at scale

Data mesh is a paradigm shift for creating a decentralized data architecture. Under this approach, all data is treated as a product. Specialist teams are responsible for the development and operation of data products in their domain. The concept relies on domain-driven design. Cross-functional data domain teams take on responsibility for specific data, from acquiring the data they need and processing it to making it available to the mesh as a data product. Enterprises generally do not explicitly build a data mesh, but rather create it by putting a federated governance structure in place as well as the means of exchange and discovery. Data discovery and exchange require a set of central tools and policies. 

Key benefits of a well-designed and implemented data network include creating domain-centric ownership of data sources, pipelines, and increased data quality. Data assets are offered as products in a "serve & pull", rather than a "push & ingest" model. Furthermore, the quality of the data is guaranteed because teams of specialists are responsible for the various data. And, thanks to data mesh, a faster response of the individual domain teams is possible regarding changes in the source format or quality issues. After all, there is the ability to easily scale the total number of sources and consumers.

Requirements for data mesh in the automotive industry

Regarding the design and implementation, a highly scalable infrastructure is required. In this regard, the following features should be integrated on a public cloud platform:

• Self-service capabilities
• Integration with other data lakes and interfacing applications within the organization
• Security and data protection
• Efficient DevOps tools
• Mesh governance
• Automated data pipelines
• Interoperability among data products
• Modern big data & AI capabilities 

This highlights some key features of the cloud platform for data mesh. For example, it is possible for business users to create their own reports/analyses. Thus, users are authorized to include new data sources using ingest/ETL templates. Furthermore, the use of auto-scaling mechanisms helps to easily scale with demand. Consequently, new use cases are more easily accessible by augmenting the DWH with a data lake. Also, the latest BI and Guided Analytics tools help deliver new functionality and a more attractive look and feel.

An easy integration with other applications and data lakes is made possible by standardized APIs. In addition, managed IaaS and PaaS solutions can be deployed to free up development capacity for client-specific value creation. At the same time, reliability is ensured. Finally, the cloud platform for data mesh fulfills the necessary security requirements for identity and access management, encryption, audit logging, archiving/retention, anonymization/deletion, and cyber defense.

Key features of the cloud platform for data mesh

The cloud platform for data mesh offers its users several benefits. On the one hand, the data products are structured in a modularized form. Thereby, defined interfaces support better scalability. On the other hand, the domain team with data ownership has the skills and business knowledge to maintain quality. Data product owners will be able to use federated governance in the data mesh to monitor data usage. In addition, there is an exchange of data within the enterprise at the unit layer. This allows for cross-domain data sharing based on well-defined governance policies.

Additionally, the cloud platform for data mesh offers increased discoverability of data, due to a central and domain-based data catalog. Data products are easier to maintain thanks to a code approach. Thereby, the entry barrier for development of new data products/assets is lowered. Moreover, seamless (zero-downtime) onboarding/migration of individual workloads (datasets, reports, dashboards and KPIs, and custom APIs) is possible.

Last, but not least, the cloud platform for data mesh in the automotive industry enables enterprises to foster innovation and draw resources from a large talent pool. The pay-per-use pricing structure of the public cloud platform helps OEMs reduce the total cost of storage and computing compared to legacy solutions.

Download the Deloitte Point of View “Cloud platform for data mesh in the automotive industry” and learn more.

¹ IDC: „FutureScape: Worldwide Future of Industry Ecosystems 2022 Predictions“, under: https://www.idc.com/getdoc.jsp?containerId=US47771821&pageType=PRINTFRIENDLY (accessed 17 Oct. 2022).