NEURAL NETWORK MEMORY COMPUTING SYSTEM AND METHOD

• US 20190325025A1
• Filed: 12/12/2018
• Published: 10/24/2019
• Est. Priority Date: 04/24/2018
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: JUNG, Ho Young, KIM, Hyun Woo, SONG, Hwa Jeon, CHUNG, Eui Sok, PARK, Jeon Gue
• Granted Patent: US 10,929,612 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/041 Abduction

Abstract

Provided are a neural network memory computing system and method. The neural network memory computing system includes a first processor configured to learn a sense-making process on the basis of sense-making multimodal training data stored in a database, receive multiple modalities, and output a sense-making result on the basis of results of the learning, and a second processor configured to generate a sense-making training set for the first processor to increase knowledge for learning a sense-making process and provide the generated sense-making training set to the first processor.

Claims

1. A neural network memory computing system for increasing knowledge for grasping a meaning of input multiple modalities, the system comprising:

a first processor configured to learn a sense-making process based on sense-making multimodal training data stored in a database, receive multiple modalities, and output a sense-making result based on results of the learning; and

a second processor configured to generate a sense-making training set for the first processor to increase knowledge for learning a sense-making process and provide the generated sense-making training set to the first processor.

2. The neural network memory computing system of claim 1, wherein the first processor embeds the sense-making multimodal training data in units of vectors having a preset magnitude and generates a sense-making training set based on the vectors.

3. The neural network memory computing system of claim 2, wherein the first processor learns a process of extracting features of visual and auditory pieces of the sense-making multimodal training data using a deep neural network (DNN), outputs final hidden-layer values of the DNN for the visual and auditory pieces of sense-making training data, and embeds the final hidden-layer values in the vectors.

4. The neural network memory computing system of claim 2, wherein, for a piece of the sense-making multimodal training data corresponding to text, the first processor learns a deep neural network (DNN)-based word vector space using a previously provided text corpus, outputs word vector values of the text, and embeds the word vector values in the vectors.

5. The neural network memory computing system of claim 2, wherein when associated information corresponding to a query and answer for sense making is not stored in a memory space, the first processor performs tagging for writing the associated information in the memory space and tags the input query as a sense-making training input and a sense-making training output to generate the sense-making training set for the query and answer.

6. The neural network memory computing system of claim 5, wherein the first processor classifies input tagged for writing as a text modality, a visual modality, and an auditory modality among multiple modalities included in the sense-making training set and stores the classified input in the memory space.

7. The neural network memory computing system of claim 6, wherein the first processor sets modalities tagged as the sense-making training input among the multiple modalities included in the sense-making training set as an input to a deep neural network (DNN), sets modalities tagged as the sense-making training output as an output of the DNN, and learns the DNN.

8. The neural network memory computing system of claim 1, wherein the first processor embeds the input multiple modalities including a query and associated information in units of vectors having a preset magnitude, repeatedly inputs the embedded multiple modalities including the query and associated information in a deep neural network (DNN), and outputs the sense-making result.

9. The neural network memory computing system of claim 8, wherein using a weight output from the DNN, the first processor updates a piece of data whose sense-making result is related to associated information corresponding to a query and answer for sense making among pieces of data in a memory space in which the associated information is stored.

10. The neural network memory computing system of claim 1, wherein the second processor samples domain information and learns a query generation deep neural network (DNN) which generates the sense-making training set using a domain query-answer set constructed based on the sampled domain information.

11. The neural network memory computing system of claim 10, wherein the second processor calls a text input corresponding to an answer and modality information whose associated information is related to the text input, converts the text input and the modality information into vectors, sets the vectors as input to the query generation DNN, sets text converted into a vector corresponding to a query as an output of the query generation DNN, and learns the query generation DNN.

12. A neural network memory computing method for increasing knowledge for grasping a meaning of input multiple modalities, the method comprising:

embedding sense-making multimodal training data stored in a database in vectors having a preset magnitude;

generating a sense-making training set based on the vectors;

learning a deep neural network (DNN) based on multiple modalities included in the generated sense-making training set; and

outputting results of grasping a meaning of input multiple modalities based on the learned DNN.

13. The neural network memory computing method of claim 12, wherein the embedding of the sense-making multimodal training data in the vectors having the preset magnitude comprises learning a process of extracting features of visual and auditory pieces of the sense-making multimodal training data using the DNN, outputting final hidden-layer values of the DNN for the visual and auditory pieces of sense-making training data, and embedding the final hidden-layer values in the vectors.

14. The neural network memory computing method of claim 12, wherein the embedding of the sense-making multimodal training data in the vectors having the preset magnitude comprises, for a piece of the sense-making multimodal training data corresponding to text, learning a DNN-based word vector space using a previously provided text corpus, outputting word vector values of the text, and embedding the word vector values in the vectors.

15. The neural network memory computing method of claim 14, wherein the generating of the sense-making training set based on the vectors comprises:

determining whether associated information corresponding to a query and answer for sense making is stored in a memory space;

when it is determined that the associated information is not stored in the memory space, performing tagging for writing the associated information in the memory space; and

tagging an input query as a sense-making training input and a sense-making training output to generate the sense-making training set for the query and answer.

16. The neural network memory computing method of claim 15, wherein the learning of the DNN based on the multiple modalities included in the generated sense-making training set comprises:

classifying input tagged for writing as a text modality, a visual modality, and an auditory modality among the multiple modalities included in the sense-making training set and storing the classified input in the memory space;

setting modalities tagged as the sense-making training input among the multiple modalities included in the sense-making training set as input to the DNN; and

setting modalities tagged as the sense-making training output as output of the DNN and learning the DNN.

17. The neural network memory computing method of claim 12, wherein the outputting of the results of grasping a meaning of the input multiple modalities based on the learned DNN comprises:

embedding the multiple modalities including an input query and associated information in units of vectors having the preset magnitude; and

repeatedly inputting the embedded multiple modalities including the query and associated information in the DNN and outputting sense-making results.

18. The neural network memory computing method of claim 17, wherein the outputting of the results of grasping a meaning of the input multiple modalities based on the learned DNN further comprises updating, using a weight output from the DNN, a piece of data whose sense-making result is related to associated information corresponding to a query and answer for sense making among pieces of data in a memory space in which the associated information is stored.

19. A neural network memory computing method for increasing knowledge for grasping a meaning of input multiple modalities, the method comprising:

sampling domain information;

learning a query generation deep neural network (DNN) using a domain query-answer set constructed based on the sampled domain information; and

generating a sense-making training set based on the query generation DNN,

wherein the learning of the query generation DNN comprises calling a text input and modality information whose associated information is related to the text input, converting the text input and the modality information into vectors, setting the vectors as input to the query generation DNN, setting text converted into a vector corresponding to a query as an output of the query generation DNN, and learning the query generation DNN.