Neural Networks are black boxes. In order to understand which variables caused the Neural Network to make particular prediction, and to separate causal variables from spurious variables, a pipeline is developed (right) based on the principles of Causal Calculus.

This allows us to understand which input or intermediate variables were truly causal for the network's prediction. For example, the figure below shows how the proposed algorithm can separate causal variables from those with spurious correlations. Multiple validations are conducted on various others datasets, like the HELOC dataset for credit risk analysis.

Pearson Correlation Coefficient of input variables (top) versus Causal Importance (bottom). The variables in the green box have a positive correlation but are not causal, and are hence 'spurious' variables

Proposed workflow for Causal Inference on Neural Networks