Explaining the explainers: Why AI needs better interpretation

AI in products and services development

Have you ever wondered why artificial intelligence (AI) solutions take the decisions that they do, and if there were any other ways to go about the problem? 

It has becoming increasingly clear that AI performance cannot always be decoded by consumers, and it is necessary to change that. With the availability of digital data increasing, AI has seeped into critical aspects of productivity improvement, cost reduction, decision-making, and customer-focused development of products and services for enterprises.

The advent of deep learning techniques has improved the performance of AI solutions, but the lack of transparency in these models are obstacles in their acceptance, as well as in audit and compliance. The conventional machine learning (ML) models themselves are difficult to interpret when dealing with high dimensional data. Hence the term ‘black boxes’—as they cannot explain why the system made those decisions.

The need for explainable AI

The need for ‘explainability’, therefore, cannot be stressed enough. In the financial sector, for example, the risks associated with such black box models can be a barrier to their adoption as they cannot be trusted by consumers, including the regulators in financial services. There are several recently developed techniques for explainable AI (XAI) that provide fairly good interpretations of the models’ decisions. Even if the model is opaque, these techniques (such as, SHapley Additive exPlanations or SHAP and Local Interpretable Model-agnostic Explanations or LIME) can provide reasonable explanations based on surrogate linear models or attributions provided by the features’ shapely values.  

But is that good enough? 

The answer is ‘no’. Most of the explainable AI techniques prevalent today provide outputs that can only be understood and analyzed by AI experts, data scientists, and probably, ML engineers. Other stakeholders including domain experts and business users of the decisions made by the models, are often unable to comprehend these explanations, trust the outcome, and derive value from these models. If, for example, the domain experts understand why the models made these decisions, they might be able to augment or improve these models with their expertise and experience. This would make the model more trustworthy and help in assessing its bias, fairness, and risks. Likewise, a business user might want to understand specific model decisions to know and understand the reasons behind them.

Human-understandable explanations are therefore crucial to building transparency and trust in AI solutions. For instance, consider a retail bank deploying a machine learning model for home loan approval; the model would take the loan applicant’s data as input and provide a decision on loan approval. The bank would then be keen to know how the model made those decisions, which model attributes have the highest weightage and importance, and whether there is need to add or remove certain attributes to improve the performance of the model. On the other hand, the loan applicants would want to know why their loan was rejected and would demand a user-friendly interface through which they can question the model’s decision. This could be web-based visualization, or better, an engaging conversation with the model that can help the user get better insights on the outcome.

Comprehensible explanations for the common folk

What does it take to provide explanations that humans can understand? One of the key considerations for this is to mimic human thinking and provide explanations in an approximate rather than a precise way.

Consider this loan approval model—through the common prevailing methodologies in explainability techniques, the loan applicant would be informed that the top reasons for his loan rejection are his credit history (0), loan amount (XXXX), loan term (YY), and the applicant’s income (ZZZ). A more helpful output would be to inform the loan applicants that their loans were rejected because of their poor credit history and high loan amount request; their income could probably be too low for the amount applied for. The users could also be informed that if they were to reduce the loan amount or get a co-applicant with a higher income, their application was more likely to get approved. This is called ’counterfactual explanations’—providing an explanation through a hypothetical scenario that contradicts the observed values and provides the user with what it takes to change the decision. This could be done with experimenting what-if scenarios for those features that can be worked upon to flip the decision.