10 MINS READ
Industry
Retail
Omnichannel Experience & Intelligent – The Future of Retail
Highlights
- Orienting data estate to cater to domain-specific needs of the business has become a common priority for modern retailers.
- Data mesh fits well as a concept to fulfil the aspiration of domain centricity, but it requires significant organizational and process changes for true decentralization, federated governance, and value realization.
- Data ‘meshhouse’ offers a paradigm to build domain-oriented data products on top of centralized data lakehouse in a faster and cost-effective manner.
On this page
On this pageinpage
Data as a strategic asset for domain teams
Data-driven retail transformation has gained momentum in the last few years.
Most retailers have invested heavily to modernize their data estate to support business functions, giving credence to British mathematician Clive Humby’s observation made in 2006 that ‘data is the new oil’.
The value of data, however, largely depends on the contextual knowledge of data teams and, hence, in recent days, we are observing a continuous push to decentralize data estate and shift its control to business domain teams. To treat data as a strategic asset or product, retailers are focusing on making it discoverable, trustworthy, self-describing, addressable, and interoperable for better value realization.
Though data mesh fits well as a concept to address this purpose, the feasibility of setting it up for large retailers with matured data estate and a vast user base is quite challenging, if not impossible, in the short or mid-term. This is why retailers with centralized data estate and frameworks are exploring options to extend their existing data ‘lakehouse’ to cater to the needs of domain teams in a quick and cost-effective manner.
A data ‘meshhouse’ will blend the best of both the data mesh and data ‘lakehouse’ without requiring significant organizational and process changes. It can help retailers, particularly those with mature data estate, resolve the challenges posed by data mesh and ‘lakehouse’ and reap greater benefits from data.
Data ’meshhouse’, in fact, holds the potential to become the mainstay of future data platforms for retailers.
Data mesh is not good enough
Data mesh is an architectural and organizational paradigm, and not a technology or a solution that can be bought.
Data mesh, a term introduced by technologist and author Zhamak Dehghani in 2019, is based on four fundamental principles that bundle well-known concepts: domain ownership, data as a product, self- service data platform, and federated governance.
Data mesh is a decentralized approach that enables domain teams to perform cross-domain data analysis on their own. However, data practitioners have some serious concerns about it:
There is no standard tool, technology, framework to enable data mesh
It requires major organizational and process changes
Federated governance is extremely complex
Defining bounded context for domain teams is a time-consuming process
Cross-domain and common-domain use cases have performance and governance implications
Not all domains have the maturity and skillset to govern and manage their own data estate
All these make retailers apprehensive about implementing data mesh in its purest form. Those with a mature data ‘lakehouse’ also face various challenges.
Data ‘lakehouse’ challenges
A data ‘lakehouse’ is a modern platform that enables structure and schema of a data warehouse on top of low-cost and unstructured data stored in a data lake.
Over the last few years, retailers’ data estates have matured as they have transformed their modern data platform into a data ‘lakehouse’ with robust processing and management frameworks.
As many data-matured retailers have already onboarded huge volume and variety of data in their centralized cloud data estate, the focus is now shifting toward consumption use cases. That is where domain users have concerns about the quality and completeness of a data ‘lakehouse’ for analytical use cases. Some of the key concerns are:
Data ‘lakehouse’ focuses on technical capabilities and IT use cases
Trustworthiness, transparency, and discoverability of data assets are questionable due to missing context
Over-reliance on central data teams
Both data producers and data consumers are not accountable for governance of the central data estate
Functional data quality checks are more reactive than proactive
All of these are causing dissatisfaction among domain users as they perceive a central data ‘lakehouse’ and a central data team to be a roadblock to enabling data-driven business processes for their domains.
Data ‘meshhouse’ for domain orientation
Retailers are looking for options to enable data mesh principles on top of the central data ‘lakehouse’ without causing much disruption.
While both data mesh and ‘lakehouse’ have their benefits and challenges, the future of modern data platforms is expected to be a careful blend of both.
Retailers who are data matured and have invested in a centralized ‘lakehouse’ with all standards and processes will look for options to extend it further to satisfy the demands of domain teams. This is where a data ‘meshhouse’ will be of use.
A data ‘meshhouse’ provides all technical frameworks and standards of a data ‘lakehouse’ to all domain users. Data stored in a central data lake is exposed to domain users through dedicated data products. Thus, it combines the best of both a data ‘lakehouse’ and data mesh to avoid data silos and governance challenges.
Following are the key enhancements needed to establish a data ’meshhouse’:
- Domain focus:
- Domain-driven segregation within central data lakehouse by workspaces or folders supported by metadata, and access control
- Distributing central data team into domain-specific (finance, customer care, sales) DevOps teams governed jointly by both central and domain-specific data leadership
- Data as a product: Building strategic data sets with metadata, quality control, and services on top of central data ‘lakehouse’ for domain consumption.
- Self-service and governance:
- Data marketplace integrated with data catalog and data quality tools for data sharing
- Data-sharing agreement with data quality, availability, and schema certainty, along with ownership and lineage
- Reusable framework, templates, and tools along with one-click provisioning of a sandbox environment
Components of an enterprise data meshhouse in retail
Thus, data ‘meshhouse’ addresses the analytical need of domain teams by not disrupting technology and processes much. Value realization becomes much faster and more predictable through well-governed subscriptions through the data marketplace.
Data ‘meshhouse’ helps retailers to avoid challenges associated with federated governance and organizational process changes.
Let us understand the approach taken in retail to convert a central data ‘lakehouse’ into a data ‘meshhouse’ for domain teams.
A large multinational retailer built a centralized data lakehouse in Azure with multiple frameworks and templates for standardization and productivity gain. But, still, there was confusion among domain users if the central data platform team is capable of supporting their business use cases in a timely fashion.
To address this concern, the following improvement actions are considered to form a data ‘meshhouse’:
- Domain-specific DevOps teams headed by domain leads for end-to-end control. These teams were in three categories:
- Source domain teams: Liaise with enterprise data source teams like SAP, Sterling, and enterprise integration team
- Common domain teams: These teams manage master domains like sales, supply chain, customer, and finance
- Consuming domain teams: Dependent on source data domains and common data domains, along with own data sources. Examples are fashion, food, ecommerce, and marketing
- Domain-specific labeling of strategic data sets for easy discoverability and reliability
- All frameworks and templates managed by a central team and accessible to all domain teams
- A common data marketplace for data sharing among all domain teams like marketing, store operations, finance, and supply chain with well-defined data contracts
- Automated approval workflow to subscribe to any data product and associated services through the data marketplace
- Register metadata, data quality commitment, and ownership in the data catalog before publishing data products in the data marketplace
- Common collaboration platform and cadence between domain teams for better governance
All these set the foundation for a data ‘meshhouse’ for the retailer to enable the core principles of a data mesh within their central data lakehouse.
Improve analytical capabilities with data ‘meshhouse’
Retailers are looking for faster value realization from their data estate.
Retail business priorities and analytical use cases are changing rapidly. To stay ahead of competition, retailers need quick analytical enablement for domain teams without too many organizational and process changes.
Finding value out of a huge data estate without proper metadata and governance as well as unavailability of relevant skillset often becomes a big hurdle for domain-owned data teams. Data 'meshhouse’ can be a good solution to this problem as it brings in domain orientation without shifting control of data asset development to individual domain teams.
