Imagine an advanced fighter aircraft is patrolling a hostile conflict area and a bogie suddenly appears on radar accelerating aggressively at them. The pilot, with the assistance of an Artificial Intelligence co-pilot, has a fraction of a second to decide what action to take – ignore, avoid, flee, bluff, or attack. The costs associated with False Positive and False Negative are substantial – a wrong decision that could potentially provoke a war or lead to the death of the pilot. What is one to do…and why?
No one less than the Defense Advanced Research Projects Agency (DARPA) and the Department of Defense (DoD) are interested in not only applying AI to decide what to do in hostile, unstable and rapidly devolving environments but also want to understand why an AI model recommended a particular action.
Welcome to the world of Explainable Artificial Intelligence (XAI).
A McKinsey Quarterly article “What AI Can And Can’t Do (Yet) For Your Business” highlights the importance of XAI in achieveing AI mass adoption:
“Explainability is not a new issue for AI systems. But it has grown along with the success and adoption of deep learning, which has given rise both to more diverse and advanced applications and to more opaqueness. Larger and more complex models make it hard to explain, in human terms, why a certain decision was reached (and even harder when it was reached in real time). This is one reason that adoption of some AI tools remains low in application areas where explainability is useful or indeed required. Furthermore, as the application of AI expands, regulatory requirements could also drive the need for more explainable AI models.”
Being able to quantify the predictive contribution of a particular variable to the effectiveness of an analytic model is a critically important for the following reasons:
Let’s drill into what DARPA is trying to achieve with their XAI initiative.
DARPA is an agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military. DARPA is soliciting proposals to help address the XAI problem. Information about the research proposals and timeline can be found at “Explainable Artificial Intelligence (XAI)”.
The target persona of XAI is an end user who depends on decisions, recommendations, or actions produced by an AI system, and therefore needs to understand the rationale for the system’s decisions. To accomplish this, DARPA is pushing the concepts of an “Explainable AI Model” and an “Explanation Interface” (see Figure 1).
Figure 1: Explainable Artificial Intelligence (XAI)
Figure 1 presents three XAI research and development challenges:
These are bold but critically-important developments if we hope to foster public confidence in the use of AI in our daily lives (e.g., autonomous vehicles, personalized medicine, robotic machines, intelligent devices, precision agriculture, smart buildings and cities).
Deep learning systems are “trained” by analyzing massive amounts of labeled data to classify the data. These models are then used to predict the probability of matches to new data sets (cats, dogs, handwriting, pedestrians, tumors, cancer, plant diseases). Deep Learning goes through three stages to create their classification models.
Step 1: Training. A Neural Network is trainedby feeding thousands, if not millions, of labeled images into the Neural Network in order to create the weights and biases across the different neurons and connections between layers of neurons that are used to classify the images (see Figure 2).
Figure 2: Training the Neural Network
Step 2: Testing.The Neural Network’s predictive effectiveness is testedby feeding new images into the Neural Network to see how effective the neural network is at classifying the new images (see Figure 3).
Figure 3: Testing Neural Network
Step 3: Learning. Learning or refining of the Neural Network model is driven by rippling learnings from the model errors in order to minimize the square of errors between expected versus actual results.
Figure 4: Learning and Refining the Model
As I covered in the blog “Neural Networks: Is Meta-learning the New Black?,” a neural network “learns” from the square of its errors using two important concepts:
In summary, we are propagating the total error backward through the connections in the network layer by layer, calculating the contribution (gradient) of each weight and bias to the total error in every layer, and then using gradient descent algorithm to optimize the weights and biases, and eventually minimizing the total error of the neural network (see Figure 5).
Figure 5: Neural Networks: Backpropagation and Gradient Descent
As powerful as Neural Networks are in analyzing and classifying data sets, the problem with a Neural Network in a world of XAI is that none of the variables and their associated weights are known, so it is impossible to explain why and how a deep learning model came up with its predictions.
Note as we discussed in the opening scenario, there are many scenarios where the costs of model failure – False Positives and False Negatives – can be significant, maybe even disastrous. Consequently, organizations must invest the time and effort to thoroughly detail the consequences of False Positives and False Negatives. See the blog “Using Confusion Matrices to Quantify the Cost of Being Wrong” for more details on understanding and quantifying the costs of False Positives and False Negatives.
Building upon our industry-leading research at the University of San Francisco on the Economic Value of Data, we are following up on that research by exploring a mathematical formula that can calculate how much of the analytic model’s predictive results can be attributed to any one individual variable. This research not only helps to further advance the XAI thinking, but also provides invaluable guidance to organizations that are trying to:
Watch this space for more details on this research project.
I’m also overjoyed by the positive feedback that I have gotten on the release of my third book: "The Art of Thinking Like A Data Scientist”. This workbook is jammed with templates, worksheets, examples and hands-on exercises to help organizations become more effective at leveraging data and analytics to power your business models. I hope that you enjoy it as well.