Skip to main content
Skip to footer

Jayashree Arunkumar

With the availability of digital data increasing, artificial intelligence (AI) plays an important role in the critical aspects of productivity improvement, cost reduction, decision making, and customer-focused development of products and services for enterprises.

The advent of deep learning techniques has improved the performance of AI solutions, but the lack of transparency in these models are obstacles in their acceptance, as well as in audit and compliance. The conventional machine learning (ML) models themselves are difficult to interpret when dealing with high dimensional data. Hence the term ‘black boxes’—as they cannot explain why the system made those decisions.

The need for explainable AI

The need for explainability, therefore, cannot be stressed enough. In the financial sector, for example, the risks associated with such black box models can be a barrier to their adoption as they cannot be trusted by consumers, including the regulators in financial services. There are several recently developed techniques for explainable AI (XAI) that provide fairly good interpretations of the models’ decisions. Even if the model is opaque, these techniques (such as, SHapley Additive exPlanations or SHAP and Local Interpretable Model-agnostic Explanations or LIME) can provide reasonable explanations based on surrogate linear models or attributions provided by the features’ shapely values.  

But is that good enough? The answer is ‘no’. Most of the explainable AI techniques prevalent today provide outputs that can only be understood and analyzed by AI experts, data scientists, and probably, ML engineers. Other stakeholders including domain experts and business users of the decisions made by the models, are often unable to comprehend these explanations, trust the outcome, and derive value from these models. If, for example, the domain experts understand why the models made these decisions, they might be able to augment or improve these models with their expertise and experience. This would make the model more trustworthy and help in assessing its bias, fairness, and risks. Likewise, a business user might want to understand specific model decisions to know and understand the reasons behind them.

Human-understandable explanations are therefore crucial to building transparency and trust in AI solutions. For instance, consider a retail bank deploying a machine learning model for home loan approval; the model would take the loan applicant’s data as input and provide a decision on loan approval. The bank would then be keen to know how the model made those decisions, which model attributes have the highest weightage and importance, and whether there is need to add or remove certain attributes to improve the performance of the model. On the other hand, the loan applicants would want to know why their loan was rejected and would demand a user-friendly interface through which they can question the model’s decision. This could be web-based visualization, or better, an engaging conversation with the model that can help the user get better insights on the outcome.

Comprehensible explanations for the common folk

What does it take to provide explanations that humans can understand? One of the key considerations for this is to mimic human thinking and provide explanations in an approximate rather than a precise way.

Consider this loan approval model—through the common prevailing methodologies in explainability techniques, the loan applicant would be informed that the top reasons for his loan rejection are his credit history (0), loan amount (XXXX), loan term (YY), and the applicant’s income (ZZZ). A more helpful output would be to inform the loan applicants that their loans were rejected because of their poor credit history and high loan amount request; their income could probably be too low for the amount applied for. The users could also be informed that if they were to reduce the loan amount or get a co-applicant with a higher income, their application was more likely to get approved. This is called ’counterfactual explanations’—providing an explanation through a hypothetical scenario that contradicts the observed values and provides the user with what it takes to change the decision. This could be done with experimenting what-if scenarios for those features that can be worked upon to flip the decision.

Communicating the results and associated explanations to the user in a human-friendly manner requires a channel like a chat interface. A chatbot could be enabled to use formatted response templates or natural language generation (NLG) techniques that can analyze the content of the explanations, derive the understanding with context, structure the narrative, and present the explanation information and recommendations in a language that humans find easy to understand.

The benefits of explaining the explainers are

  • Easy to understand explanations in human-friendly manner
  • Addresses the needs of multiple stakeholders in appropriate formats
  • Provides the recommendations in the form of counterfactuals

About the author

Jayashree Arunkumar
Jayashree Arunkumar is a technical architect working with the Incubation unit within Research and Innovation. She has significant experience in IT service management, network management, operations automation, and more recently, model operations. Her focus areas are identifying and developing new offerings to drive innovation.
Contact Contact