Second glance: A novel explainable AI to understand feature interactions in neural networks using higher-order partial derivatives

Neural networks often operate as "black boxes," making understanding how they arrive at their decisions difficult. To build trust and improve neural networks, it is essential to identify the most salient inputs and how they interact within the network. We present "Second Glance," a novel approach for performing second-order sensitivity analysis on neural networks with Rectified Linear Unit (ReLU) activations to address this. The first-order sensitivity analysis quantifies the individual influence of the input features on the model output. However, it fails to capture how features interact, potentially leading to misleading conclusions. Second-order sensitivity analysis, using second-order partial derivatives, can reveal these interactions, providing a more comprehensive understanding of the model’s inner workings.

Unfortunately, ReLU activation, a popular choice because of its efficiency, introduces zero second-order partial derivatives. To overcome this limitation, Second Glance employs a two-stage strategy. First, it trains a primary neural network with ReLU activations. Then, it trains a separate "surrogate" model using the concerned features as the input and the first-order partial derivatives obtained from the primary model as its output. In this paper, we show that the subtle second-order sensitivity analysis of the original neural network with ReLU activation function can be effectively obtained by analyzing the first-order partial derivatives of the surrogate model. We further validate the proposed method by experimenting with popular UCI bank marketing and UCI adult income datasets.