METHOD FOR UNDERSTANDING MACHINE-LEARNING DECISIONS BASED ON CAMERA DATA

• US 20180293464A1
• Filed: 04/05/2018
• Published: 10/11/2018
• Est. Priority Date: 04/07/2017
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Martin, Charles E., Kolouri, Soheil, Hoffmann, Heiko
• Granted Patent: US 10,803,356 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

Described is a system for understanding machine-learning decisions. In an unsupervised learning phase, the system extracts, from input data, concepts represented by a machine-learning (ML) model in an unsupervised manner by clustering patterns of activity of latent variables of the concepts, where the latent variables are hidden variables of the ML model. The extracted concepts are organized into a concept network by learning functional semantics among the extracted concepts. In an operational phase, a subnetwork of the concept network is generated. Nodes of the subnetwork are displayed as a set of visual images that are annotated by weights and labels, and the ML model per the weights and labels.

Claims

1. A system for understanding machine-learning (ML) decisions, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

in an unsupervised learning phase, extracting from input data a plurality of concepts represented by an ML model in an unsupervised manner by clustering patterns of activity of latent variables of the concepts, wherein the latent variables are hidden variables of the ML model;

in the unsupervised learning phase, organizing the extracted concepts into a concept network by learning functional semantics among the extracted concepts;

in an operational phase, generating a subnetwork of the concept network; and

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the ML model per the weights and labels.

2. The system as set forth in claim 1, wherein the one or more processors further perform an operation of in the operational phase, rendering and displaying the subnetwork as a visual depiction of a set of decision pathways leading up to a decision output made by the ML model.

3. The system as set forth in claim 2, wherein the one or more processors further perform an operation of automatically modifying the ML model to improve correctness of the decision output based on the subnetwork, resulting in a modified ML model.

4. The system as set forth in claim 3, wherein the one or more processors further perform an operation of controlling a physical system based on an output of an unmodified ML model or the modified ML model.

5. The system as set forth in claim 1, wherein in the unsupervised learning phase, the one or more processors further perform operations of:

collecting a plurality of subsets of latent variable activations from the input data processed with the ML model, resulting in activation vectors;

clustering the activation vectors in an unsupervised manner, resulting in a set of clusters;

computing functional semantics for each cluster, wherein the functional semantics form the concept network,

wherein the concept network captures the plurality of concepts represented by the ML model with respect to the input data; and

learning the concept network.

6. The system as set forth in claim 5, wherein in the operational phase, the one or more processors further perform operations of:

processing a single data input by the ML model that activates its latent variables and generates a model decision; and

associating the activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

7. The system as set forth in claim 1, wherein each subset of a plurality of subsets of latent variable activations is generated by distinct input data.

8. The system as set forth in claim 1, wherein the functional semantics represent an extent to which the plurality of concepts influence one another within the ML model via a set of weighted directed edges in the concept network.

9. The system as set forth in claim 1, wherein the one or more processors further perform an operation of causing a vehicle to perform a vehicle operation based on a decision output by the ML model.

10. The system as set forth in claim 1, wherein the device is a machine used in manufacturing.

23. The system as set forth in claim 2, wherein the decision output is an image classification.

11. A computer implemented method for understanding machine-learning (ML) decisions, the method comprising an act of:

causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of;

in an unsupervised learning phase, extracting from input data, a plurality of concepts represented by an ML model in an unsupervised manner by clustering patterns of activity of latent variables of the concepts, wherein the latent variables are hidden variables of the ML model;

in the unsupervised learning phase, organizing the extracted concepts into a concept network by learning functional semantics among the extracted concepts;

in an operational phase, generating a subnetwork of the concept network;

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the ML model per the weights and labels.

12. The method as set forth in claim 11, wherein in the unsupervised learning phase, the one or more processors further perform operations of:

collecting a plurality of subsets of latent variable activations from the input data processed with the ML model, resulting in activation vectors;

clustering the activation vectors in an unsupervised manner, resulting in a set of clusters;

computing functional semantics for each cluster, wherein the functional semantics form the concept network,

wherein the concept network captures the plurality of concepts represented by the ML model with respect to the input data; and

learning the concept network.

13. The method as set forth in claim 12, wherein in the operational phase, the one or more processors further perform operations of:

processing a single data input by the ML model that activates its latent variables and generates a model decision; and

associating the activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

14. The method as set forth in claim 11, wherein each subset of a plurality of subsets of latent variable activations is generated by distinct input data.

15. The method as set forth in claim 11, wherein the functional semantics represent an extent to which the plurality of concepts influence one another within the ML model via a set of weighted directed edges in the concept network.

16. The method as set forth in claim 11, wherein the one or more processors perform an operation of causing a vehicle to perform a vehicle operation based on a decision output by the ML model.

17. A computer program product for understanding machine-learning (ML) decisions, the computer program product comprising:

computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of;

in an unsupervised learning phase, extracting from input data a plurality of concepts represented by an ML model in an unsupervised manner by clustering patterns of activity of latent variables of the concepts, wherein the latent variables are hidden variables of the ML model;

in the unsupervised learning phase, organizing the extracted concepts into a concept network by learning functional semantics among the extracted concepts;

in an operational phase, generating a subnetwork of the concept network;

displaying nodes of the subnetwork as a set of visual images that are annotated by weights and labels; and

refining the ML model per the weights and labels.

18. The computer program product as set forth in claim 17, further comprising instructions for, in the unsupervised learning phase, causing the one or more processors to further perform operations of:

collecting a plurality of subsets of latent variable activations from the input data processed with the ML model, resulting in activation vectors;

clustering the activation vectors in an unsupervised manner, resulting in a set of clusters;

computing functional semantics for each cluster, wherein the functional semantics form the concept network,

wherein the concept network captures the plurality of concepts represented by the ML model with respect to the input data; and

learning the concept network.

19. The computer program product as set forth in claim 18, further comprising instructions for, in the operational phase, causing the one or more processors to further perform operations of:

processing a single data input by the ML model that activates its latent variables and generates a model decision; and

associating the activation vectors with nodes in the concept network, creating the subnetwork of the concept network.

20. The computer program product as set forth in claim 17, wherein each subset of a plurality of subsets of latent variable activations is generated by distinct input data.

21. The computer program product as set forth in claim 17, wherein the functional semantics represent an extent to which the plurality of concepts influence one another within the ML model via a set of weighted directed edges in the concept network.

22. The computer program product as set forth in claim 17, wherein the one or more processors perform an operation of causing an autonomous vehicle to perform a vehicle operation based on a decision output by the ML model.