Method for understanding machine-learning decisions based on camera data

• US 10,803,356 B2
• Filed: 04/05/2018
• Issued: 10/13/2020
• Est. Priority Date: 04/07/2017
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Martin, Charles E., Kolouri, Soheil, Hoffmann, Heiko
• Primary Examiner(s): Lee, John W
• Application Number:
  US15/946,480
• Publication Number:
  US 20180293464A1
• Time in Patent Office: 922 Days
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

Described is a system for understanding machine-learning decisions. In an unsupervised learning phase, the system extracts, from input data, concepts represented by a machine-learning (ML) model in an unsupervised manner by clustering patterns of activity of latent variables of the concepts, where the latent variables are hidden variables of the ML model. The extracted concepts are organized into a concept network by learning functional semantics among the extracted concepts. In an operational phase, a subnetwork of the concept network is generated. Nodes of the subnetwork are displayed as a set of visual images that are annotated by weights and labels, and the ML model per the weights and labels.

Claims

1. A system for understanding machine-learning (ML) decisions, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

processing a set of input data with a convolutional neural network (CNN) model, resulting in a set of latent variable activation vectors within the CNN model, wherein each latent variable activation vector in the CNN model represents a concept in the set of input data;

clustering the set of latent variable activation vectors in an unsupervised manner, resulting in a plurality of clusters of latent variable activation vectors;

computing functional semantics for each cluster, wherein the functional semantics represent relationships among concepts in the set of input data;

generating a concept network comprising a plurality of nodes and a plurality of weighted directed edges, wherein the plurality of nodes are defined by concepts and the plurality of weighted directed edges are defined by the computed functional semantics;

in an operational phase, generating a subnetwork of the concept network; and

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the CNN model per the weights and labels.

2. The system as set forth in claim 1, wherein the one or more processors further perform an operation of in the operational phase, rendering and displaying the subnetwork as a visual depiction of a set of decision pathways leading up to a decision output made by the CNN model.

3. The system as set forth in claim 2, wherein the one or more processors further perform an operation of automatically modifying the CNN model to improve correctness of the decision output based on the subnetwork, resulting in a modified CNN model.

4. The system as set forth in claim 3, wherein the one or more processors further perform an operation of controlling a physical system based on an output of an unmodified CNN model or the modified CNN model.

5. The system as set forth in claim 1, wherein in the operational phase, the one or more processors further perform operations of:

processing a single data input by the CNN model that activates latent variables of the CNN model and generates a model decision; and

associating the latent variable activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

6. The system as set forth in claim 1, wherein the functional semantics represent an extent to which concepts influence one another within the CNN model via the plurality of weighted directed edges in the concept network.

7. The system as set forth in claim 1, wherein the one or more processors further perform an operation of causing a vehicle to perform a vehicle operation based on a decision output by the CNN model.

8. The system as set forth in claim 1, wherein the device is a machine used in manufacturing.

17. The system as set forth in claim 2, wherein the decision output is an image classification.

9. A computer implemented method for understanding machine-learning (ML) decisions, the method comprising an act of:

causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of;

processing a set of input data with a convolutional neural network (CNN) model, resulting in a set of latent variable activation vectors within the CNN model, wherein each latent variable activation vector in the CNN model represents a concept in the set of input data;

clustering the set of latent variable activation vectors in an unsupervised manner, resulting in a plurality of clusters of latent variable activation vectors;

computing functional semantics for each cluster, wherein the functional semantics represent relationships among concepts in the set of input data;

generating a concept network comprising a plurality of nodes and a plurality of weighted directed edges, wherein the plurality of nodes are defined by concepts and the plurality of weighted directed edges are defined by the computed functional semantics;

in an operational phase, generating a subnetwork of the concept network;

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the CNN model per the weights and labels.

10. The method as set forth in claim 9, wherein in the operational phase, the one or more processors further perform operations of:

processing a single data input by the CNN model that activates latent variables of the CNN model and generates a model decision; and

associating the latent variable activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

11. The method as set forth in claim 9, wherein the functional semantics represent an extent to which concepts influence one another within the CNN model via the plurality of weighted directed edges in the concept network.

12. The method as set forth in claim 9, wherein the one or more processors perform an operation of causing a vehicle to perform a vehicle operation based on a decision output by the CNN model.

13. A computer program product for understanding machine-learning (ML) decisions, the computer program product comprising:

computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of;

processing a set of input data with a convolutional neural network (CNN) model, resulting in a set of latent variable activation vectors within the CNN model, wherein each latent variable activation vector in the CNN model represents a concept in the set of input data;

clustering the set of latent variable activation vectors in an unsupervised manner, resulting in a plurality of clusters of latent variable activation vectors;

computing functional semantics for each cluster, wherein the functional semantics represent relationships among concepts in the set of input data;

generating a concept network comprising a plurality of nodes and a plurality of weighted directed edges, wherein the plurality of nodes are defined by concepts and the plurality of weighted directed edges are defined by the computed functional semantics;

in an operational phase, generating a subnetwork of the concept network;

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the CNN model per the weights and labels.

14. The computer program product as set forth in claim 13, further comprising instructions for, in the operational phase, causing the one or more processors to further perform operations of:

processing a single data input by the CNN model that activates latent variables of the CNN model and generates a model decision; and

associating the latent variable activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

15. The computer program product as set forth in claim 13, wherein the functional semantics represent an extent to which concepts influence one another within the CNN model via the plurality of weighted directed edges in the concept network.

16. The computer program product as set forth in claim 13, wherein the one or more processors perform an operation of causing an autonomous vehicle to perform a vehicle operation based on a decision output by the CNN model.