Institutionalizing trust in AI models

A scheme to secure trust in AI systems 

A mechanism that aims to standardize the testing and governance of AI models.

The AI testing framework is built factoring certain processes, leading to APIfication that will help test the non-functional aspects in AI systems:

The framework evaluates models for their non-functional requirements based on the following aspects: 

Compliance: This covers industrial and regulatory governance aspects. It includes policies, processes, procedures, checklists, tools, and guidelines to check on the inner aspects of the framework.

Ethical AI: A reliable model should be ethical. Privacy, societal and environmental concerns, and AI governance are key aspects that are assessed. 

An AI model needs to ensure that user privacy is respected. It should avoid collecting data or using sensitive personal information for training purposes without the user’s consent. Systems are prone to model inversion attacks, and private inferencing is one such method that can prevent obtaining sensitive data. 

AI systems also assist in addressing critical environmental issues. From this perspective, one should evaluate its entire life cycle in an environment-friendly manner. To build a sustainable model, one must ensure that training consumes the minimal energy possible, as well as consider societal and social concerns while developing new methods for equitable and fair practices. 

Moreover, data collection and processing should be fair and without bias as they influence the outcomes. While there are no straightforward solutions to address it, there are ways to measure, detect, and mitigate biases. Calibration is critical, especially in clinical and financial use cases, as it measures the difference between a model’s computed output probability and accuracy.  

Trustworthy AI: The trustworthiness of AI models rests on the credence and accuracy of predictions, especially for businesses such as healthcare, banking, and insurance. Decisions based on analytical projections can have significant implications for such industries. 

Let’s take an example to understand the process involved in structuring the framework. Consider an application that takes image data, computes it, and then provides an output classifying the image. It is now essential to verify that the model is well-calibrated, ensures privacy, and is resilient. Following this, the metrics must first identify that they quantify the requirement and use non-functional testing procedures to improve the model. An API should then analyze the model to generate a report on the non-functional aspects.

Trust encompasses factors such as human-centricity and explainability. Human-centric values, principles, and perspectives are central to building AI systems, and safeguarding these is key to determining the trustworthiness of a model. 

With advancements in deep learning and artificial neural networks, data privacy and the role of fundamental rights have become critical while collecting, training data, and interpreting sensitive information. 

Explainability is the level to which humans can understand predictions and decisions made by AI models. The model should provide valid reasons behind automated decisions, as newer regulations expect the service provider to ensure this. Explainability will help uncover biases and promote fairness. It can also bring in more transparency, help identify weak spots, enable debugging, and help increase consumer trust in the system.

Robust AI: This forms the framework’s core, critical to every organization. Models must be robust to encompass factors such as resilience, calibration, bias, performance, and fairness. While adversarial training can help overcome the attacks, it can also negatively impact model performance. Adversarial defenses do not stop the attacker from trying new methods.

Raw data is susceptible to attack from insiders and outsiders. There could be situations where an attacker could gain access to an AI model and reconstruct the data. These types of attacks are known as model inversion attacks. A robust model should be able to resist intentional or unintended manipulations. A high level of security and accuracy is required even in the face of threats or changes in input. It must use several metrics for evaluation criteria instead of a single metric.