Architecture and processes for computer learning and understanding

• US 10,599,778 B2
• Filed: 03/20/2017
• Issued: 03/24/2020
• Est. Priority Date: 06/24/2016
• Status: Active Grant

• Current Assignee: Mind Lakes, LLC
• Original Assignee: Elemental Cognition LLC
• Inventors: Ferrucci, David, Barborak, Mike, Buchanan, David, Burnham, Greg, Chu-Carroll, Jennifer, Kalyanpur, Aditya, Lally, Adam, Pacifico, Stefano, Wang, Chang
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/463,731
• Publication Number:
  US 20200034420A1
• Time in Patent Office: 1,100 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/027 Frames
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

An architecture and processes enable computer learning and developing an understanding of arbitrary natural language text through collaboration with humans in the context of joint problem solving. The architecture ingests the text and then syntactically and semantically processes the text to infer an initial understanding of the text. The initial understanding is captured in a story model of semantic and frame structures. The story model is then tested through computer generated questions that are posed to humans through interactive dialog sessions. The knowledge gleaned from the humans is used to update the story model as well as the computing system'"'"'s current world model of understanding. The process is repeated for multiple stories over time, enabling the computing system to grow in knowledge and thereby understand stories of increasingly higher reading comprehension levels.

Claims

1. A computer-implemented method, comprising:

maintaining a collection of uninstantiated semantic structures in a memory of a computing system;

receiving, by the computing system, a natural language input;

performing, by the computing system, a syntactic analysis of the natural language input to produce one or more linguistic analysis results;

instantiating one or more uninstantiated semantic structures with natural language information from the one or more linguistic analysis results to form one or more instantiated semantic structures;

creating, by the computing system, a semantic structure with the one or more instantiated semantic structures to represent the natural language input based at least in part on using the one or more linguistic analysis results and knowledge induced from large language corpora, wherein the semantic structure includes multiple generative semantic primitive (GSP) structures defined with roles that indicate one or more beliefs regarding a first understanding of the natural language input, wherein defining the roles includes mapping one or more entities in the natural language information to the roles;

generating, by the computing system, one or more structured questions to evaluate the semantic structure and the one or more beliefs;

maintaining a dependency structure for linear dialogs to identify questions that are independent from one another;

submitting, from the computing system to a separate user computing device, the one or more structured questions for presentation to a human user;

receiving, by the computing system from the separate user computing device, one or more responses indicative of input from the human user when answering the one or more structured questions; and

revising, by the computing system, the semantic structure to produce a modified semantic structure that includes a new GSP structure defined with at least one new role that indicates a new belief regarding a second understanding of the natural language input based at least in part on the one or more responses received from the separate user computing device, the modified semantic structure representing the natural language input differently than how the semantic structure before revision represented the natural language input based at least in part on the second understanding of the natural language input learned from the human user.

2. The computer-implemented method of claim 1, wherein receiving the natural language input comprises:

receiving audio containing a verbal representation of the natural language input;

generating an audio signal from the audio; and

converting the audio signal to the natural language input.

3. The computer-implemented method of claim 1, wherein the one or more linguistic analysis results comprises at least one of a syntactic parse, a predicate argument structure, an entity type, or co-references of the natural language input.

4. The computer-implemented method of claim 1, wherein creating the semantic structure comprises:

forming the multiple GSP structures using the one or more linguistic analysis results, data maintained in a current world model that expresses confidences in how accurately the multiple GSP structures represent the natural language input, and the knowledge induced from the large language corpora; and

choosing one or more of the multiple GSP structures to produce the semantic structure.

5. The computer-implemented method of claim 1, wherein creating the semantic structure comprises:

determining a particular theme or a particular context of the natural language input based at least in part on the one or more linguistic analysis results; and

selecting the one or more uninstantiated semantic structures from the collection of uninstantiated semantic structures based at least in part on the collection of uninstantiated semantic structures having an above threshold probability of being included with the semantic structure that is associated with the particular theme or the particular context of the natural language input.

6. The computer-implemented method of claim 1, wherein submitting the one or more structured questions comprises submitting different ones of the one or more structured questions to multiple user computing devices for presentations to different human users and iterating through the dependency structure based at least in part on the different ones of the one or more structured questions to identify a set of next questions.

7. A computer-implemented method, comprising:

parsing, by a computing system, a natural language story to produce a syntactic representation of the natural language story;

performing, by the computer system, a predicate argument structure (PAS) analysis on the syntactic representation of the natural language story;

assigning, by the computer system, one or more entity types to one or more words in the natural language story;

determining, by the computer system, co-reference chains in the one or more words in the natural language story;

inferring, by the computing system, one or more semantic structures as a semantic representation of the natural language story using, at least in part, the syntactic representation of the natural language story, the one or more semantic structures including generative semantic primitive (GSP) structures defined with roles, wherein the inferring includes associating a semantic structure of the one or more semantic structure with a particular theme or a particular context of the natural language story;

engaging in multiple dialog sessions with computing devices of multiple human users in parallel, by generating and submitting one or more structured questions for presentation via computer user interface to the computing devices of the multiple human users, to receive responses from the multiple human users for use by the computing system to evaluate the one or more semantic structures as an understanding of the natural language story; and

responsive to the responses from the multiple human users, revising the one or more semantic structures by defining at least one new GSP structure with at least a new role for the semantic structure, the at least one new GSP structure having an above threshold probability of being included in the semantic structure associated with the particular theme or the particular context of the natural language story.

8. The computer-implemented method of claim 7, wherein the syntactic representation comprises at least one of a syntactic parse, a predicate argument structure, an entity type, or co-references of the natural language story.

9. The computer-implemented method of claim 7, wherein the inferring comprises:

retrieving, by the computing system, one or more frame structures from a collection of frame structures in a memory, the one or more frame structures including one or more sets of GSP structures defined with predicates and roles that are relevant to a current understanding of the natural language story; and

aligning, by the computing system, entities in the natural language story to the roles defined in the one or more sets of GSP structures for the one or more frame structures.

10. The computer-implemented method of claim 7, wherein the one or more semantic structures comprise the GSP structures to facilitate inference at a sentential level and frame structures composed of multiple GSP structures to facilitate inference at a thematic level.

11. The computer-implemented method of claim 7, wherein engaging in the multiple dialog sessions comprises generating the one or more structured questions to evaluate the one or more semantic structures.

12. The computer-implemented method of claim 11, wherein generating the one or more structured questions comprises:

building a dependency structure that provides a dialog context for interpreting and answering the one or more structured questions, the dependency structure being derived from data in the one or more semantic structures; and

forming the one or more structured questions in a natural language.

13. The computer-implemented method of claim 12, wherein the dependency structure comprises a tree-type data structure with root nodes and branches to children nodes that can be logically traversed to ask opening and follow-on questions.

14. A computing system, comprising:

a datastore containing a current world model that expresses beliefs about how natural language is understood;

one or more processors; and

memory coupled to the one or more processors, the memory storing computer-executable modules comprising;

a story parsing engine to syntactically analyze a natural language story to produce linguistic analysis results;

a knowledge induction engine to induce information from large language corpora to form induced information;

a knowledge integration engine to form semantic structures that provide a semantic representation of the natural language story, the knowledge integration engine using the linguistic analysis results, information from the current world model, and the induced information to form the semantic structures, wherein the semantic structures include generative semantic primitive (GSP) structures defined with a predicate and one or more roles that indicate beliefs regarding an understanding of the natural language story, the knowledge integration engine including belief generation components to produce a probability distribution over a set of beliefs about the understanding of the natural language story; and

a dialog engine to facilitate a dialog session with a human user by generating and submitting one or more questions for presentation via a computer user interface to a computing device used by the human user and collecting one or more responses from the computing device indicative of input from the human user, wherein the dialog engine maintains a dependency structure for linear dialogs to identify questions that are independent from one another; and

wherein the knowledge integration engine revises the semantic structures by defining at least one new GSP structure that indicates a new belief regarding the understanding of the natural language story, the new belief having an above threshold probability of being relevant to the understanding of the natural language story based at least in part on the one or more responses from the human user, and wherein the knowledge integration engine updates the current world model in the datastore.

15. The computing system of claim 14, wherein the semantic structures further include frame structures composed of multiple GSP structures.

16. The computing system of claim 15, wherein the knowledge integration engine continues to revise the semantic structures until a termination condition is reached, the termination condition being a function of a confidence value that is calculated to express an extent to which the frame structures fit the semantic representation of the natural language story.

17. The computing system of claim 14, wherein the knowledge induction engine comprises a word sense disambiguator to disambiguate word senses.

18. The computing system of claim 14, wherein the knowledge integration engine comprises

a joint inference engine to calculate a marginal probability of each belief in the set of beliefs for use in constructing a story model representation of the natural language story.

19. The computing system of claim 14, further comprising:

an additional datastore to store a library of uninstantiated GSP structures, the additional datastore being part of, or accessible by, the computing system; and

a linguistic-to-GSP mapper module to select one or more uninstantiated GSP structures from the library of uninstantiated GSP structures in the additional datastore and instantiate the one or more uninstantiated GSP structures with information from the linguistic analysis results.

20. The computing system of claim 14, wherein the dialog engine facilitates multiple dialog sessions with multiple human users concurrently.