How AI manages and gleans insights from unstructured data  

An AI on insights

Leveraging AI to derive insights from unstructured data 

Leading-edge organizations are using AI technologies to process unstructured data to forge the path to business value. Let’s look at a few examples: 

• A manufacturing system can analyze video images to find flaws in glass, helping the company ship safer products.  
• An insurance company can scan photos of automobile damages to determine precisely what needs to be fixed and how much repairs will cost.  
• A product development organization can use transcripts of thousands of customer support calls to identify ideas for new and improved items.  

All these scenarios are in practice today, thanks to machine learning (ML) and deep learning. ML trains a machine to mimic human tasks, and it is commonly used to identify patterns across very large amounts of data. Deep learning uses multi-layered neural networks to deliver high precision and accuracy in tasks such as speech recognition, language translation, and object detection. 

These AI technologies improve with the volume of data they process, learning from them and becoming smarter over time. They also have enormous potential to unlock the value of unstructured data. This is significant, as according to a Gartner report, up to 90% of all enterprise data is either unstructured or semi-structured and most is not used to its full potential. Further, the volume of unstructured data in most organizations is estimated to be growing at 65% per year. 

“Most organizations are just scratching the surface with AI,” says Warren Barkley, Director of Product Management at Google Cloud. “Business users are trying to figure out how to operationalize it. There is a wealth of value sitting in unstructured data,” he adds. 

Consider a legal office that processes thousands of PDF contracts per year, the customer-call transcripts of a service center, or video images of a manufacturing production line. AI technologies can find what’s hidden in these data formats, analyze it, and glean insights that can lead to competitive advantage, business efficiencies, improved customer experiences, and more.

Different forms of data 

Data comes in many forms—structured and unstructured. 

Let’s try to understand the different forms of data. 

1. Structured data has predefined value types that are easy for machines to read—such as customer records in a spreadsheet with fields for name, address, ZIP code, and account number. Most business data processing, such as generating bills and payments has historically used structured data.
2. Semi-structured data has some clearly labeled elements, but much of the information is undefined. Email messages are a good example: the sender, recipient, time, and date fields are structured but most of the message is not.
3. Unstructured data includes text documents, audio and video recordings, email messages, scanned documents, and sensor readings. 

AI use cases

The primary challenge to AI adoption is making it accessible and transparent.

Making AI explainable is another important factor to be addressed because otherwise, it’s difficult for people to understand how a conclusion has been reached. Companies looking to unlock the potential of data—specifically unstructured data—with AI can focus on an applicable use case and demonstrate how ML or deep learning can solve a problem set. 

Across industries, many organizations are already doing that. Here’s how: 

• Computerized image recognition is guiding autonomous vehicles, identifying structural anomalies in buildings and bridges, and spotting wanted criminals in a crowd. It is also being used to improve safety practices on the shop floor, ensuring employees are alerted to spills or wearing appropriate protective gear.

• Automated speech recognition systems can achieve better than 95% accuracy in interpreting everyday speech. Once the speech is transcribed—such as from a call-center interaction—text-mining algorithms identify patterns that detect sentiments or yield insights into customer experiences. Companies use these insights to improve their products and ideate new ones.

• Intelligent data analysis of unstructured data has nearly limitless applications in healthcare. Take Google Cloud’s Healthcare Natural Language API and AutoML Entity Extraction for Healthcare. They are used to recognize handwritten doctors’ notes and medical records for uses such as detecting adverse drug interactions and enriching medical histories. 

• Text recognition and natural language processing are transforming office-related tasks using scanned images of printed and handwritten documents, as well as emails. This data can then be leveraged to identify patterns among customers employing social media platforms and assist compliance teams analyzing emailed documents.

Getting started

Identifying the right use case and adopting a fail-fast mentality are key.

There’s no doubt that AI solutions can help IT and business leaders make better, faster, and more-informed decisions from unstructured data to achieve enhanced business value. The key is to get started. That said, experts have some words of advice. 

“Make sure everybody in the organization has a baseline understanding of AI,” Barkley says. “That raises the bar and gets everyone thinking about how it can be applied to interesting and difficult problems.”

The first thing is to have the right use case with the right stakeholders who have literacy in AI. Also, chances are that some AI projects will fail, so adopt a fail-fast mentality. By doing so, you can move on from initial setbacks, each time gaining nuggets of learning that will help get to success and scale faster. 

By tapping into the power of unstructured data, organizations will achieve a range of extraordinary results, such as driving shorter time to market, enhancing supply chain, and improving customer insights across industries.   

A version of this article was originally published as TCS-sponsored content on CIO.com.