Service
Data and Analytics
Harness the Power of Data and Analytics for better Business Outcomes
Highlights
- Data governance, the foundation that underpins the entire data lifecycle, ensures data is accurate, complete, and usable.
- An effective and automated governance system self-heals, self-cleans, and self-augments.
- AI can improve the underlying data quality, leading to robust and effective data governance.
On this page
Why data governance
High-quality, reliable data has always been the foundation of informed decision-making.
Trustworthy and interrogatable data sets enable organizations to uncover valuable insights, identify trends, and make data-driven decisions that drive business success. LLMs are prone to hallucinations, and hence, the underlying models need to be trained with intelligence that can be trusted.
Trusted data is the foundation of successful AI. Despite genAI’s compelling benefits, which include rapid data access and analysis, the underlying challenge of ensuring data origin, quality, and integrity remains a critical concern.
Data value corruption
In 2021, a tech real-estate marketplace company announced that it would use AI to estimate home values.
It started a division to directly buy homes from sellers based on AI-generated value estimates and then short-sell them based on market conditions. The value in this was minimized interaction between seller and buyer at that time when COVID was prevalent. The problem was in the estimate. The company took a massive inventory write-down, which it blamed on purchasing homes for prices higher than a reasonable sale price.
What was missing was a foundational set of master data applying to real estate information, which should have been included as part of the source data sets. A lack of adequate governance and planning led to the issue.
In hindsight, the AI model suffered a governance failure. When the housing market prices came down, its algorithms were not adjusted, and therefore, they overestimated demand and, hence, home prices.
Automated governance
AI has since evolved rapidly and, in its current state, makes available several compelling features that can be deployed across systems.
The following features enable proper governance through efficient, proactive measures:
- Self-healing (system): This is the ability of the system to detect and repair itself with minimal human intervention. The system monitors its health and performance and can identify potential problems before they cause outages or other disruptions.
- Self-cleaning (data): The system can detect and correct errors on its own. The most basic error in data is data duplicity. Data or data sets that look different from other data sets proliferate the enterprise often due to poor governance, standardization, and non-standard data acquisition processes. Effective entity resolution becomes key here. LLM-based models that run on top of current ML-based algorithms improve the accuracy of matching records and almost completely resolve entity resolution issues.
- Uniqueness of critical master data assets: Data duplication leads to inconsistencies and multiple versions of the truth. This can be solved through AI-based entity-matching techniques.
- Self-augment (data): Most organizations continue to have a siloed approach toward managing large systems such as ERP, CRM, eCommerce, supply chain, and mainframes. When they then try to combine data from different systems for, say, predictive analytics, they often end up with missing data sets, incomplete data sets, and, at times, wrong values or labels. GenAI can detect and correct errors and inconsistencies within the data. It can read entity relationship models and arrive at or suggest ‘derived attributes’ and, in some cases, attribute values. This self-augmentation ability powered by genAI will ensure that the ultimate sets of data that are fed to a larger genAI program produce results that are of a higher order of trust.
- Synthesis matching: Next-generation matching technique that addresses, for example, matching prospects to customers, matching interactions and unstructured data to customers, and discovering non-obvious relationships. Synthesis matching enables the assimilation of information across these different areas and facilitates a compelling and comprehensive view of the customer.
- Data enrichment: AI algorithms detect and correct data inconsistencies and errors. They also automate the standardization of data formats. To reduce manual efforts and labor, they can provide good descriptions of the product by analyzing the product attributes and external market. From a business perspective, they can dynamically create relationships with related products to improve cross-sells and up-sells instead of fixed relationships (refer synthesis matching). This also enables an improvement in content for search engines (Search Engine Optimization). The automation of mass data corrections with intelligent suggestions also increases productivity and reduction in time and effort. To take it a step further, they can also provide personalized content and content translation based on who is using the data.
Adequate data governance
AI can help improve the quality of underlying data that is used for decision-making and ensure compliance with industry and/or company standards.
This drastically reduces human intervention in the governance of data and saves time in manual and laborious work. Effective AI governance includes oversight mechanisms that address risks such as bias, privacy infringement, and misuse while fostering innovation and building trust.
